mdtmFTP Docker release

About mdtmFTP

To address challenges in high performancedata movement for large-scale science, the Fermilab network research group has developed mdtmFTP, a high-performance data transfer tool to optimize data transfer on multicore platforms. mdtmFTP has a number of advanced features.

• First, it adopts a pipelined I/O design. Data transfer tasks are carried out in a pipelined manner across multiple cores. Dedicated threads are spawned to perform network and disk I/O operations in parallel.

• Second, mdtmFTP uses multicore-aware data transfer middleware (MDTM) to schedule an optimal core for each thread, based on system configuration, to optimize throughput across the underlying multicore core platform.

• Third, mdtmFTP implements a large virtual file mechanism to efficiently handle lots-of-small-files (LOSF) situations.

• Finally, mdtmFTP unitizes optimization mechanisms such as zero copy, asynchronous I/O, batch processing, and pre-allocated buffer pools, to maximize performance.

This document describes using Docker container releases of mdtmFTP. For manuals of source code or Singuarity releases of mdtmFTP, please visit the project website: http://mdtm.fnal.gov.

Docker releases of mdtmFTP are at https://hub.docker.com/r/wenji/mdtm/.

For technical details about mdtmFTP, please refer to the paper:

Liang Zhang,Wenji Wu, Phil DeMar, Eric Pouyoul: mdtmFTP and its evaluation on ESNET SDN testbed. Future Generation Comp. Syst.79:199-204 (2018)

Intended audience

This manual is intended for users and system administrators responsible for installing, running, and managing DTNs. The manual assumes familiarity with multicore and DTN concepts.

Contacts

• Wenji Wu (wenji@fnal.gov)

• Liang Zhang (liangz@fnal.gov)

• Phil DeMar (demar@fnal.gov)

Acknowledgements

mdtmFTP uses several Globus modules for rapid prototyping. We sincerely thank Globus folks at Argonne National Laboratory and University of Chicago.

Here is a list of Globus modules that mdtmFTP uses:

• GridFTP protocol module

• Globus xio module

• Globus security module

• Globus user interface

As of January 2018, Globus Toolkit is retired, and now it is succeeded by software maintained by Grid Community Forum (GCF).

Contents:

• 1. Obtaining mdtmFTP image and running container
  • 1.1. Requirements
  • 1.2. Get mdtmFTP Docker image
  • 1.3. Configuring and running mdtmFTP server container
  • 1.4. Configuring and running mdtmFTP client
• 2. Managing credentials
  • 2.1. Managing credentials: username/password authentication
  • 2.2. Managing credentials: GSI certificates
• 3. Running mdtmFTP server in docker environments
  • 3.1. mdtmFTP server command syntax
  • 3.2. Required Linux capabilities for mdtmFTP server
  • 3.3. Running mdtmFTP server in docker environments
• 4. Running mdtmFTP client in docker environments
  • 4.1. mdtmFTP client command syntax
  • 4.2. Running mdtmFTP client in docker environments
• 5. Support for IPv6
  • 5.1. IPv6-enabled network
  • 5.2. IPv6-enabled hosts
  • 5.3. mdtmFTP command line
  • 5.4. Data transfer examples
• 6. Data transfer examples: Docker environments
  • 6.1. Client–Server data transfer
  • 6.2. Third party data transfer between two remote DTNs
  • 6.3. Client–Server data transfer (IPv6)
  • 6.4. Third party data transfer between two remote hosts (IPv6)
• 7. Appendix: adding users in docker environment
• 8. Appendix: setting up X.509 security on DTN
  • 8.1. Installing CA certificates
  • 8.2. Requesting and installing host certificate
  • 8.3. Verifying trust paths
  • 8.4. What if things did not go well?
• 9. Appendix: creating server mdtmconfig.json
  • 9.1. Topology section
  • 9.2. Online section
  • 9.3. Thread section
  • 9.4. File section
  • 9.5. Manually_configured_cpus section
  • 9.6. Server section
  • 9.7. Example
• 10. Appendix: creating server mdtmconfig.xml
  • 10.1. Topology section
  • 10.2. Online section
  • 10.3. Thread section
  • 10.4. File section
• 11. Appendix: creating server.conf
• 12. Appendix: creating client mdtmconfig.json
  • 12.1. Topology section
  • 12.2. Online section
  • 12.3. Thread section
  • 12.4. File section
  • 12.5. Manually_configured_cpus section
  • 12.6. Example
• 13. Appendix: creating client mdtmconfig.xml
• 14. Appendix: recommended host CAs
  • 14.1. Using InCommon
  • 14.2. Using LetsEncrypt
  • 14.3. Using SimpleCA