4. Running mdtmFTP client in docker environments

4.1. mdtmFTP client command syntax

$ mdtm-ftp-client [-ipv6] -p <parallelism> -splice <src_url> <dst_url>

Those parameters are for:

  • -ipv6 enables IPv6 for data transfer. The default is IPv4.

  • -p <parallelism> specifies the number of parallel data streams.

  • -splice turns on “splice” feature.

  • src_url specifies source URL.

  • dst_url specifies destination URL.

4.2. Running mdtmFTP client in docker environments

Running mdtmFTP client in docker environments typically takes this form:

$ sudo docker run \
   [Name setting] \
   [Operation mode settings] \
   [User settings] \
   [Network settings] \
   [Volume settings] \
   [Capabilities settings] \
   [Security settings] \
   [mdtmFTP docker image] \
   [Path to mdtmFTP]mdtm-ftp-client [ARG...]

Among these settings:

  • [Name setting]. Sepcify a name for the container, e.g., --name mdtmftp-server.

  • [Opertion mode settings]. It is typically set to -ti to run the container in the foreground.

  • [User setting]. Set the user for the container, e.g., -u mdtmftp-client.

  • [Network settings]. It is typically set to --net=host to set the host’s network stack inside the container.

  • [Volume setting]. Set up bindmount volumes that link folders from inside the container to folders on the host machine.

  • [Capabilities settings]. By default, Docker containers are unprivileged. You need to add the following Linux capabilitis to the container so that mdtmFTP server can run properly:

    • IPC_LOCK

    • SYS_NICE

    • SYS_ADMIN

    • NET_ADMIN

    • SYS_RESOURCE

  • [Security setting]. For Docker 1.10 and 1.11, --security-opt seccomp:unconfined is required to add Linux capabilities.

  • [mdtmFTP docker image]. Specify a particular docker image you want to run.

4.2.1. With username/password authentication

Here are the typical preparation steps before launching a mdtmFTP client with username/password authentication:

  • Open a new terminal and create the mdtmFTP working directory, e.g., /home/mdtmwork.

  • Create mdmtFTP configuration files in the working directory

  • Add data transfer users in the docker environment and set the appropriate permissions.

  • Save the updated docker image if necessary.

  • Create a user account, e.g., mdtmftp-client, to run mdtmFTP docker image in host.

  • Prepare data transfer folders in host, e.g., /data1. Set appropriate permissions to allow data transfer users to access this folder.

Then, run the mdtmFTP client. Typically, the following folders in the host will be bindmount into the container:

  • mdtmFTP workig directory in the host, e.g., /home/mdtmwork –> /mdtmwork in the container.

  • Data transfer folders in the host, e.g,, /data1 –> /data1 in the container.

$ cd /home/mdtmwork
$ sudo docker run --name mdtmftp-client \
      -it --rm \
      -u mdtmftp-client \
      --net=host \
      -v `pwd`:/mdtmwork \
      -v /data1:/data1 \
      --cap-add=ALL --cap-add=IPC_LOCK --cap-add=SYS_NICE \
      --cap-add=SYS_ADMIN --cap-add=NET_ADMIN --cap-add=SYS_RESOURCE \
      --security-opt seccomp:unconfined \
      mdtmftp:1.1.1-xenial-updated \
      /usr/local/mdtmftp/1.1.1/bin/mdtm-ftp-client \
      ftp://mdtmftp:123456@bde1.fnal.gov:5050/data1/linux-4.13.4/ file:///tmp/

4.2.2. With GSI certificate-based authentication

Here are the typical preparation steps before launching a mdtmFTP client with GSI certificate-based authentication:

  • Open a new terminal and create the mdtmFTP working directory, e.g., /home/mdtmwork.

  • Create mdmtFTP configuration files in the working directory

  • Add data transfer users in the docker environment and set the appropriate permissions.

  • Save the updated docker image if necessary.

  • Create a user account, e.g., mdtmftp-client, to run mdtmFTP docker image in host.

  • Create /etc/grid-security in the host to hold trusted CA certificates.

  • Create your user and proxy certificates, typically in the path /tmp, to login mdtmFTP servers.

  • Prepare data transfer folder(s) in host, e.g., /data1. Set appropriate permissions to allow data transfer users to access the folder(s).

Then, run the mdtmFTP client. Typically, the following folders in the host will be bindmount into the container:

  • mdtmFTP workig directory in the host, e.g., /home/mdtmwork –> /mdtmwork in the container.

  • /etc/grid-security in the host –> /etc/grid-security in the container.

  • /tmp in the host –> /tmp in the container.

  • Data transfer folders in the host, e.g,, /data1 –> /data1 in the container.

$ sudo docker run --name mdtmftpclient \
      -it --rm \
      -u mdtmftp-client \
      --net=host \
      -v /etc/grid-security:/etc/grid-security \
      -v `pwd`:/mdtmwork \
      -v /tmp:/tmp \
      -v /data1:/data1 \
      --cap-add=ALL --cap-add=IPC_LOCK --cap-add=SYS_NICE \
      --cap-add=SYS_ADMIN --cap-add=NET_ADMIN --cap-add=SYS_RESOURCE \
      --security-opt seccomp:unconfined \
      mdtmftp:1.1.1-xenial-updated \
      /usr/local/mdtmftp/1.1.1/bin/mdtm-ftp-client \
      gsiftp://mdtmftp:123456@bde1.fnal.gov:5050/data1/linux-4.13.4/ file:///tmp/

For more examples, see Data transfer examples: Docker environments