Running mdtmFTP client

1. Before starting data transfer, open a new terminal and create the mdtmFTP working directory /home/mdtmwork:

$ mkdir -p /home/mdtmwork && cd /home/mdtmwork
  1. Set up the credentials

mdtmFTP supports two approaches for authentication and authority between servers and clients: password and certificates.

2.1. Password

In this case, every client uses its own preset password to login to servers and each server should have legal clients’ passwords saved in a file specified in mdtmFTP server command line option -password-file. In the distributed mdtmFTP container image, there already exists a default user mdtmftp. We are about to use that default user to take on data transfer jobs.

2.2. Certificates

To use X.509 security or GSI mechanism, each user has to create its own user certificate and place the CA certificate for the host running mdtmFTP server in the path /etc/grid-security. And the protocol gsiftp should be also identified in the command line of mdtmFTP client.

  1. Running mdtmFTP client

Run the client application from the container following mdtmFTP client command syntax:

$ sudo docker run -ti --rm \
                  --net=host \
                  -v `pwd`:/mdtmwork \
                  publicregistry.fnal.gov/bigdata_express/mdtmftp:1.1.1-xenial \
                  mdtm-ftp-client [-ipv6] -p <parallelism> -splice <src_url> <dst_url>

Those parameters are for:

  • -ipv6 enables IPv6 for data transfer. The default is IPv4.
  • -p <parallelism> specifies the number of parallel data streams.
  • -splice turns on “splice” feature.
  • src_url specifies source URL.
  • dst_url specifies destination URL.

There are two modes to run mdtmFTP client applications: client-server and third party. For the client-server, the mdtmFTP client transfer data between the remote mdtmFTP server and the local system; for the third party mode, the mdtmFTP client controls data transfer job between two remote mdtmFTP servers.

3.1 Client-Server mode

For example, the below command download folder from remote system to the local path /tmp.

$ sudo docker run -ti --rm \
                  --net=host \
                  -v `pwd`:/mdtmwork \
                  publicregistry.fnal.gov/bigdata_express/mdtmftp:1.1.1-xenial \
                  /usr/local/mdtmftp/HEAD/bin/mdtm-ftp-client -vb -p 4 ftp://mdtmftp:123456@bde1.fnal.gov:5050/data1/linux-4.13.4/ file:///tmp/

3.2 Third Party mode

For example, the below command download folder from remote system bde1 to another remote system bde2 in the path /tmp.

$ sudo docker run -ti --rm \
                  --net=host \
                  -v `pwd`:/mdtmwork \
                  publicregistry.fnal.gov/bigdata_express/mdtmftp:1.1.1-xenial \
                  /usr/local/mdtmftp/HEAD/bin/mdtm-ftp-client -vb -p 4 ftp://mdtmftp:123456@bde1.fnal.gov:5050/data1/linux-4.13.4/ ftp://mdtmftp:123456@bde2.fnal.gov:5050/data2/tmp/