3. Running mdtmFTP server in docker environments

3.1. mdtmFTP server command syntax

mdtm-ftp-server \
  -control-interface <ip_address> \
  -data-interface <ip_address> \
  -password-file <passwd_file> \
  -p <port_num> \
  -c <server.conf>

Among those options,

  • -control-interface <ip_address> specifies a control interface for the server.

  • -data-interface <ip_address> specifies a data interface for the server.

  • -password-file <passwd_file> specifies a password file for username/password authentication.

  • -p <port_num> specifies a port that mdtmFTP server listens on.

  • -c <server.conf> specifies a configuration file to set data transfer parameters.

3.2. Required Linux capabilities for mdtmFTP server

For the purpose of performing permission checks, traditional UNIX implementations distinguish two categories of processes: privileged processes (whose effective user ID is 0, referred to as superuser or root), and unprivileged processes (whose effective UID is nonzero). Privileged processes bypass all kernel permission checks, while unprivileged processes are subject to full permission checking based on the process’s credentials (usually: effective UID, effective GID, and supplementary group list).

Starting with kernel 2.2, Linux divides the privileges traditionally associated with superuser into distinct units known as capabilities, which can be independently enabled and disabled.

Running mdtmFTP server requires four capabilities:

  • CAP_SYS_NICE, to bind threads to cores.

  • CAP_IPC_LOCK, to lock memory.

  • CAP_SYS_RESOURCE, to increase pipe capacity.

  • CAP_SYS_ADMIN, to increase the maximum number of open files.

3.3. Running mdtmFTP server in docker environments

Running mdtmFTP server in docker environments typically takes this form:

$ sudo docker run \
             [Name setting] \
             [Operation mode settings] \
             [User settings] \
             [Network settings] \
             [Volume settings] \
             [Capabilities settings] \
             [Security settings] \
             [mdtmFTP docker image] \
             [Path to mdtmFTP]mdtm-ftp-server [ARG...]

Among these settings:

  • [Name setting]. Sepcify a name for the container, e.g., --name mdtmftp-server.

  • [Opertion mode settings]. It is typically set to -ti to run the container in the foreground.

  • [User setting]. Set the user for the container, e.g., -u mdtmftp-admin.

  • [Network settings]. It is typically set to --net=host to set the host’s network stack inside the container.

  • [Volume setting]. Set up bindmount volumes that link folders from inside the container to folders on the host machine.

  • [Capabilities settings]. By default, Docker containers are unprivileged. You need to add the following Linux capabilitis to the container so that mdtmFTP server can run properly:

    • IPC_LOCK

    • SYS_NICE

    • SYS_ADMIN

    • NET_ADMIN

    • SYS_RESOURCE

  • [Security setting]. For Docker 1.10 and 1.11, --security-opt seccomp:unconfined is required to add Linux capabilities.

  • [mdtmFTP docker image]. Specify a particular docker image you want to run.

3.3.1. With username/password authentication

Here are the typical preparation steps before launching a mdtmFTP server with username/password authentication:

  • Open a new terminal and create the mdtmFTP working directory, e.g., /home/mdtmwork.

  • Create mdmtFTP configuration files in the working directory

  • Add data transfer users in the docker environment and set the appropriate permissions.

  • Save the updated docker image if necessary.

  • Create a password file, e.g., passfile, in the mdtmFTP working directory

  • Create a user account, e.g., mdtmftp-admin, to run mdtmFTP docker image in host.

  • Prepare data transfer folders in host, e.g., /data1. Set appropriate permissions to allow data transfer users to access this folder.

Then, run the mdtmFTP server. Typically, the following folders in the host will be bindmount into the container:

  • mdtmFTP workig directory in the host, e.g., /home/mdtmwork –> /mdtmwork in the container.

  • Data transfer folders in the host, e.g,, /data1 –> /data1 in the container.

$ cd /home/mdtmwork
$ sudo docker run --name mdtmftp-server \
                -it --rm \
                -u mdtmftp-admin \
                --net=host \
                -v `pwd`:/mdtmwork \
                -v /data1:/data1 \
                --cap-add=ALL --cap-add=IPC_LOCK --cap-add=SYS_NICE \
                --cap-add=SYS_ADMIN --cap-add=NET_ADMIN --cap-add=SYS_RESOURCE \
                --security-opt seccomp:unconfined \
                mdtmftp:1.1.1-xenial-updated \
                /usr/local/mdtmftp/1.1.1/sbin/mdtm-ftp-server \
                -data-interface 131.225.2.29 \
                -password-file passfile \
                -p 5050 \
                -l mdtmftp.log \
                -log-level all

If everything is all right, it then shows the outputs like,

[32]mdtmthread/mdtm_io_thread.c 459: mdtm_net_io_thread: IO thread Ready on device enp4s0f0: evenq=0x7f83b5081558
[31]mdtmthread/mdtm_io_thread.c 459: mdtm_net_io_thread: IO thread Ready on device enp4s0f0: evenq=0x7f83bd081558
[33]mdtmthread/mdtm_io_thread.c 336: mdtm_disk_io_thread: IO thread Ready on device nvme0n1: evenq=0x7f83b1081558
[34]mdtmthread/mdtm_io_thread.c 336: mdtm_disk_io_thread: IO thread Ready on device nvme0n1: evenq=0x7f83a9081558
mdtm_schedule_threads: device=enp4s0f0 cpu=0
mdtm_schedule_threads: device=enp4s0f0 cpu=1
mdtm_schedule_threads: device=nvme0n1  cpu=2
mdtm_schedule_threads: device=nvme0n1  cpu=3

And you can watch the log file in a different terminal,

$ sudo tail -f /home/mdtmwork/mdtmftp.log

3.3.2. With GSI certificate-based authentication

Here are the typical preparation steps before launching a mdtmFTP server with GSI certificate-based authentication:

  • Open a new terminal and create the mdtmFTP working directory, e.g., /home/mdtmwork.

  • Create mdmtFTP configuration files in the working directory

  • Add data transfer users in the docker environment and set the appropriate permissions.

  • Save the updated docker image if necessary.

  • Create /etc/grid-security in the host to grant access to mdtmFTP server.

  • Create a user account, e.g., mdtmftp-admin, to run mdtmFTP docker image in host.

  • Prepare data transfer folder(s) in host, e.g., /data1. Set appropriate permissions to allow data transfer users to access the folder(s).

Then, run the mdtmFTP server. Typically, the following folders in the host will be bindmount into the container:

  • mdtmFTP workig directory in the host, e.g., /home/mdtmwork –> /mdtmwork in the container.

  • /etc/grid-security in the host –> /etc/grid-security in the container.

  • Data transfer folders in the host, e.g,, /data1 –> /data1 in the container.

$ sudo docker run --name mdtmftpserver \
                     -it --rm \
                     -u mdtmftp-admin \
                     --net=host \
                     -v /etc/grid-security:/etc/grid-security \
                     -v `pwd`:/mdtmwork \
                     -v /data1:/data1 \
                     --cap-add=ALL --cap-add=IPC_LOCK --cap-add=SYS_NICE \
                     --cap-add=SYS_ADMIN --cap-add=NET_ADMIN --cap-add=SYS_RESOURCE \
                     --security-opt seccomp:unconfined \
                     mdtmftp:1.1.1-xenial-updated \
                     /usr/local/mdtmftp/1.1.1/sbin/mdtm-ftp-server \
                     -data-interface 131.225.2.29 \
                     -password-file passfile \
                     -p 5050 \
                     -l mdtmftp.log \
                     -log-level all

If everything is all right, it then shows the outputs like,

[32]mdtmthread/mdtm_io_thread.c 459: mdtm_net_io_thread: IO thread Ready on device enp4s0f0: evenq=0x7f83b5081558
[31]mdtmthread/mdtm_io_thread.c 459: mdtm_net_io_thread: IO thread Ready on device enp4s0f0: evenq=0x7f83bd081558
[33]mdtmthread/mdtm_io_thread.c 336: mdtm_disk_io_thread: IO thread Ready on device nvme0n1: evenq=0x7f83b1081558
[34]mdtmthread/mdtm_io_thread.c 336: mdtm_disk_io_thread: IO thread Ready on device nvme0n1: evenq=0x7f83a9081558
mdtm_schedule_threads: device=enp4s0f0 cpu=0
mdtm_schedule_threads: device=enp4s0f0 cpu=1
mdtm_schedule_threads: device=nvme0n1  cpu=2
mdtm_schedule_threads: device=nvme0n1  cpu=3

And you can watch the log file in a different terminal,

$ sudo tail -f /home/mdtmwork/mdtmftp.log