Data transfer examples: Docker environments¶
Client – Server data transfer¶
Step 1: Launch the server on DTN A¶
$ docker -v /storage_x:/storage_y \
--net=host \
-v `pwd`:/mdtmwork \
--cap-add=ALL --cap-add=IPC_LOCK --cap-add=SYS_NICE \
--cap-add=SYS_ADMIN --cap-add=NET_ADMIN --cap-add=SYS_RESOURCES \
--security-opt seccomp:unconfined \
--privileged \
publicregistry.fnal.gov/bigdata_express/mdtmftp:1.1.1-xenial \
./mdtm-ftp-server -data-interface 131.225.2.29 -password-file \
passfile -p 5001 &
Step 2: Launch the client on DTN B¶
Assuming the mdtmFTP client runs in a container.
Authentication method: username/password¶
Assuming user name/password: mdtmftp/123456.
Single file data transfer: transfer a single file from DTN A to DTN B.
$ sudo docker run -ti --rm \ --net=host \ -v `pwd`:/mdtmwork \ publicregistry.fnal.gov/bigdata_express/mdtmftp:1.1.1-xenial \ mdtm-ftp-client -p 8 \ ftp://mdtmftp:123456@10.40.130.189:5001/data1/100G/file1 \ file:///storage/data1/tmp/
Single file data transfer: transfer a single file from DTN B to DTN A.
$ sudo docker run -ti --rm \ --net=host \ -v `pwd`:/mdtmwork \ publicregistry.fnal.gov/bigdata_express/mdtmftp:1.1.1-xenial \ mdtm-ftp-client -p 8 \ file:///storage/data1/tmp/file1 \ ftp://mdtmftp:123456@10.40.130.189:5001/data1/tmp/
Folder data transfer: transfer a Linux folder from DTN A to DTN B.
$ sudo docker run -ti --rm \ --net=host \ -v `pwd`:/mdtmwork \ publicregistry.fnal.gov/bigdata_express/mdtmftp:1.1.1-xenial \ mdtm-ftp-client -p 8 \ ftp://mdtmftp:123456@10.40.130.189:5001/data1/linux-3.18.21/ \ file:///storage/data1/tmp/
Folder data transfer: transfer a Linux folder from DTN B to DTN A.
$ sudo docker run -ti --rm \ --net=host \ -v `pwd`:/mdtmwork \ publicregistry.fnal.gov/bigdata_express/mdtmftp:1.1.1-xenial \ mdtm-ftp-client -p 8 \ file:///storage/data1/tmp/linux-3.18.21/ \ ftp://mdtmftp:123456@10.40.130.189:5001/data1/tmp/
Authentication method: GSI certificate¶
Single file data transfer: transfer a single file from DTN A to DTN B.
$ sudo docker run -ti --rm \ --net=host \ -v `pwd`:/mdtmwork \ publicregistry.fnal.gov/bigdata_express/mdtmftp:1.1.1-xenial \ mdtm-ftp-client -p 8 \ gsiftp://10.40.130.189:5001/data1/100G/file1 \ file:///storage/data1/tmp/
Single file data transfer: transfer a single file from DTN B to DTN A.
$ sudo docker run -ti --rm \ --net=host \ -v `pwd`:/mdtmwork \ publicregistry.fnal.gov/bigdata_express/mdtmftp:1.1.1-xenial \ mdtm-ftp-client -p 8 \ file:///storage/data1/tmp/file1 \ gsiftp://10.40.130.189:5001/data1/tmp/
Folder data transfer: transfer a Linux folder from DTN A to DTN B.
$ sudo docker run -ti --rm \ --net=host \ -v `pwd`:/mdtmwork \ publicregistry.fnal.gov/bigdata_express/mdtmftp:1.1.1-xenial \ mdtm-ftp-client -p 8 \ gsiftp://10.40.130.189:5001/data1/linux-3.18.21/ \ file:///storage/data1/tmp/
Folder data transfer: transfer a Linux folder from DTN B to DTN A.
$ sudo docker run -ti --rm \ --net=host \ -v `pwd`:/mdtmwork \ publicregistry.fnal.gov/bigdata_express/mdtmftp:1.1.1-xenial \ mdtm-ftp-client -p 8 \ file:///storage/data1/tmp/linux-3.18.21/ \ gsiftp://10.40.130.189:5001/data1/tmp/
Third party data transfer between two remote DTNs¶
Step 1: Launch mdtmFTP server on DTN A¶
$ docker -v /storage_x:/storage_y \
--net=host xxx:yyy \
-v `pwd`:/mdtmwork \
--cap-add=ALL --cap-add=IPC_LOCK --cap-add=SYS_NICE \
--cap-add=SYS_ADMIN --cap-add=NET_ADMIN --cap-add=SYS_RESOURCES \
--security-opt seccomp:unconfined \
--privileged \
publicregistry.fnal.gov/bigdata_express/mdtmftp:1.1.1-xenial \
./mdtm-ftp-server -data-interface 131.225.2.29 -password-file \
passfile -p 5001 &
Step 2: Launch mdtmFTP server on DTN B¶
$ docker -v /storage_x:/storage_y \
--net=host xxx:yyy \
/bin/bash -c "cd /home/mdtmftp_server; \
./mdtm-ftp-server -data-interface 131.225.2.31 -password-file \
passfile -p 5001 &
Step 3: Launch the client on DTN C¶
Assuming the mdtmFTP client runs in a container.
Authentication method: username/password¶
Assuming user name/password: mdtmftp/123456
Single file data transfer: transfer a single file from DTN A to DTN B.
$ sudo docker run -ti --rm \ --net=host \ -v `pwd`:/mdtmwork \ publicregistry.fnal.gov/bigdata_express/mdtmftp:1.1.1-xenial \ mdtm-ftp-client -p 8 -vb \ ftp://mdtmftp:123456@131.225.2.29:5001/data1/100G/file1 \ ftp://mdtmftp:123456@131.225.2.31:5001/storage/data1/tmp/
Folder data transfer: transfer a Linux folder from DTN A to DTN B.
$ sudo docker run -ti --rm \ --net=host \ -v `pwd`:/mdtmwork \ publicregistry.fnal.gov/bigdata_express/mdtmftp:1.1.1-xenial \ mdtm-ftp-client -p 8 –vb \ ftp://mdtmftp:123456@131.225.2.29:5001/data1/linux-3.18.21/ \ ftp://mdtmftp:123456@131.225.2.31:5001/storage/data1/tmp/
Authentication method: GSI certificate¶
Single file data transfer: transfer a single file from DTN A to DTN B.
$ sudo docker run -ti --rm \ --net=host \ -v `pwd`:/mdtmwork \ publicregistry.fnal.gov/bigdata_express/mdtmftp:1.1.1-xenial \ mdtm-ftp-client -p 8 -vb \ gsiftp://131.225.2.29:5001/data1/100G/file1 \ gsiftp://131.225.2.31:5001/storage/data1/tmp/
Folder data transfer: transfer a Linux folder from DTN A to DTN B.
$ sudo docker run -ti --rm \ --net=host \ -v `pwd`:/mdtmwork \ publicregistry.fnal.gov/bigdata_express/mdtmftp:1.1.1-xenial \ mdtm-ftp-client -p 8 –vb \ gsiftp://131.225.2.29:5001/data1/linux-3.18.21/ \ gsiftp://131.225.2.31:5001/storage/data1/tmp/