Data transfer examples: Docker environments

Client – Server data transfer

Step 1: Launch the server on DTN A

$ docker -v /storage_x:/storage_y \
     --net=host \
     -v `pwd`:/mdtmwork \
     --cap-add=ALL --cap-add=IPC_LOCK --cap-add=SYS_NICE \
     --cap-add=SYS_ADMIN --cap-add=NET_ADMIN --cap-add=SYS_RESOURCES \
     --security-opt seccomp:unconfined \
     --privileged \
     publicregistry.fnal.gov/bigdata_express/mdtmftp:1.1.1-xenial \
     ./mdtm-ftp-server -data-interface 131.225.2.29 -password-file \
     passfile -p 5001 &

Step 2: Launch the client on DTN B

Assuming the mdtmFTP client runs in a container.

Authentication method: username/password

Assuming user name/password: mdtmftp/123456.

  • Single file data transfer: transfer a single file from DTN A to DTN B.

    $ sudo docker run -ti --rm \
                    --net=host \
                    -v `pwd`:/mdtmwork \
                    publicregistry.fnal.gov/bigdata_express/mdtmftp:1.1.1-xenial \
                    mdtm-ftp-client -p 8 \
                    ftp://mdtmftp:123456@10.40.130.189:5001/data1/100G/file1 \
                    file:///storage/data1/tmp/
    
  • Single file data transfer: transfer a single file from DTN B to DTN A.

    $ sudo docker run -ti --rm \
                    --net=host \
                    -v `pwd`:/mdtmwork \
                    publicregistry.fnal.gov/bigdata_express/mdtmftp:1.1.1-xenial \
                    mdtm-ftp-client -p 8 \
                    file:///storage/data1/tmp/file1 \
                    ftp://mdtmftp:123456@10.40.130.189:5001/data1/tmp/
    
  • Folder data transfer: transfer a Linux folder from DTN A to DTN B.

    $ sudo docker run -ti --rm \
                    --net=host \
                    -v `pwd`:/mdtmwork \
                    publicregistry.fnal.gov/bigdata_express/mdtmftp:1.1.1-xenial \
                    mdtm-ftp-client -p 8 \
                    ftp://mdtmftp:123456@10.40.130.189:5001/data1/linux-3.18.21/ \
                    file:///storage/data1/tmp/
    
  • Folder data transfer: transfer a Linux folder from DTN B to DTN A.

    $ sudo docker run -ti --rm \
                    --net=host \
                    -v `pwd`:/mdtmwork \
                    publicregistry.fnal.gov/bigdata_express/mdtmftp:1.1.1-xenial \
                    mdtm-ftp-client -p 8 \
                    file:///storage/data1/tmp/linux-3.18.21/ \
                    ftp://mdtmftp:123456@10.40.130.189:5001/data1/tmp/
    

Authentication method: GSI certificate

  • Single file data transfer: transfer a single file from DTN A to DTN B.

    $ sudo docker run -ti --rm \
                    --net=host \
                    -v `pwd`:/mdtmwork \
                    publicregistry.fnal.gov/bigdata_express/mdtmftp:1.1.1-xenial \
                    mdtm-ftp-client -p 8 \
                    gsiftp://10.40.130.189:5001/data1/100G/file1 \
                    file:///storage/data1/tmp/
    
  • Single file data transfer: transfer a single file from DTN B to DTN A.

    $ sudo docker run -ti --rm \
                    --net=host \
                    -v `pwd`:/mdtmwork \
                    publicregistry.fnal.gov/bigdata_express/mdtmftp:1.1.1-xenial \
                    mdtm-ftp-client -p 8 \
                    file:///storage/data1/tmp/file1 \
                    gsiftp://10.40.130.189:5001/data1/tmp/
    
  • Folder data transfer: transfer a Linux folder from DTN A to DTN B.

    $ sudo docker run -ti --rm \
                    --net=host \
                    -v `pwd`:/mdtmwork \
                    publicregistry.fnal.gov/bigdata_express/mdtmftp:1.1.1-xenial \
                    mdtm-ftp-client -p 8 \
                    gsiftp://10.40.130.189:5001/data1/linux-3.18.21/ \
                    file:///storage/data1/tmp/
    
  • Folder data transfer: transfer a Linux folder from DTN B to DTN A.

    $ sudo docker run -ti --rm \
                    --net=host \
                    -v `pwd`:/mdtmwork \
                    publicregistry.fnal.gov/bigdata_express/mdtmftp:1.1.1-xenial \
                    mdtm-ftp-client -p 8 \
                    file:///storage/data1/tmp/linux-3.18.21/ \
                    gsiftp://10.40.130.189:5001/data1/tmp/
    

Third party data transfer between two remote DTNs

Step 1: Launch mdtmFTP server on DTN A

$ docker -v /storage_x:/storage_y \
    --net=host xxx:yyy \
     -v `pwd`:/mdtmwork \
     --cap-add=ALL --cap-add=IPC_LOCK --cap-add=SYS_NICE \
     --cap-add=SYS_ADMIN --cap-add=NET_ADMIN --cap-add=SYS_RESOURCES \
     --security-opt seccomp:unconfined \
     --privileged \
     publicregistry.fnal.gov/bigdata_express/mdtmftp:1.1.1-xenial \
     ./mdtm-ftp-server -data-interface 131.225.2.29 -password-file \
     passfile -p 5001 &

Step 2: Launch mdtmFTP server on DTN B

$ docker -v /storage_x:/storage_y \
     --net=host xxx:yyy \
     /bin/bash -c "cd /home/mdtmftp_server; \
       ./mdtm-ftp-server -data-interface 131.225.2.31 -password-file \
       passfile -p 5001 &

Step 3: Launch the client on DTN C

Assuming the mdtmFTP client runs in a container.

Authentication method: username/password

Assuming user name/password: mdtmftp/123456

  • Single file data transfer: transfer a single file from DTN A to DTN B.

    $ sudo docker run -ti --rm \
                    --net=host \
                    -v `pwd`:/mdtmwork \
                    publicregistry.fnal.gov/bigdata_express/mdtmftp:1.1.1-xenial \
                    mdtm-ftp-client -p 8 -vb \
                    ftp://mdtmftp:123456@131.225.2.29:5001/data1/100G/file1 \
                    ftp://mdtmftp:123456@131.225.2.31:5001/storage/data1/tmp/
    
  • Folder data transfer: transfer a Linux folder from DTN A to DTN B.

    $ sudo docker run -ti --rm \
                    --net=host \
                    -v `pwd`:/mdtmwork \
                    publicregistry.fnal.gov/bigdata_express/mdtmftp:1.1.1-xenial \
                    mdtm-ftp-client -p 8 –vb \
                    ftp://mdtmftp:123456@131.225.2.29:5001/data1/linux-3.18.21/ \
                    ftp://mdtmftp:123456@131.225.2.31:5001/storage/data1/tmp/
    

Authentication method: GSI certificate

  • Single file data transfer: transfer a single file from DTN A to DTN B.

    $ sudo docker run -ti --rm \
                    --net=host \
                    -v `pwd`:/mdtmwork \
                    publicregistry.fnal.gov/bigdata_express/mdtmftp:1.1.1-xenial \
                    mdtm-ftp-client -p 8 -vb \
                    gsiftp://131.225.2.29:5001/data1/100G/file1 \
                    gsiftp://131.225.2.31:5001/storage/data1/tmp/
    
  • Folder data transfer: transfer a Linux folder from DTN A to DTN B.

    $ sudo docker run -ti --rm \
                    --net=host \
                    -v `pwd`:/mdtmwork \
                    publicregistry.fnal.gov/bigdata_express/mdtmftp:1.1.1-xenial \
                    mdtm-ftp-client -p 8 –vb \
                    gsiftp://131.225.2.29:5001/data1/linux-3.18.21/ \
                    gsiftp://131.225.2.31:5001/storage/data1/tmp/