mdtmFTP installation and configuration manual

About mdtmFTP

To address the high-performance challenges of data transfer in the big data era, Fermilab network research group has developed mdtmFTP, a high-performance data transfer tool for big data.

mdtmFTP has a number of advanced features.

• First, it adopts a pipelined I/O design. Data transfer tasks are carried out in a pipelined manner across multiple cores. Dedicated threads are spawned to perform network and disk I/O operations in parallel.

• Second, mdtmFTP uses multicore-aware data transfer middleware (MDTM) to schedule an optimal core for each thread, based on system configuration, to optimize throughput across the underlying multicore core platform.

• Third, mdtmFTP implements a large virtual file mechanism to efficiently handle lots-of-small-files (LOSF) situations.

• Finally, mdtmFTP unitizes optimization mechanisms such as zero copy, asynchronous I/O, batch processing, and pre-allocated buffer pools, to maximize performance.

This document describes the installation of mdtmFTP from source code release, and its basic use. For manuals of Docker or Singuarity releases of mdtmFTP, please visit the project website: http://mdtm.fnal.gov.

For technical details about mdtmFTP, please refer to the paper:

Liang Zhang,Wenji Wu, Phil DeMar, Eric Pouyoul: mdtmFTP and its evaluation on ESNET SDN testbed. Future Generation Comp. Syst.79:199-204 (2018)

Contacts

• Wenji Wu (wenji@fnal.gov)

• Liang Zhang (liangz@fnal.gov)

• Phil DeMar (demar@fnal.gov)

Intended audience

This manual is intended for users and system administrators responsible for installing, running, and managing DTNs. The manual assumes familiarity with multicore and DTN concepts.

Acknowledgements

mdtmFTP uses several Globus modules for rapid prototyping. We sincerely thank Globus folks at Argonne National Laboratory and University of Chicago.

Here is a list of Globus modules that mdtmFTP uses:

• GridFTP protocol module

• Globus xio module

• Globus security module

• Globus user interface

As of January 2018, Globus Toolkit is retired, and now it is succeeded by software maintained by Grid Community Forum (GCF).

Contents:

• 1. Obtaining and running mdtmFTP
  • 1.1. Requirements
  • 1.2. Building mdtm and mdtmFTP
  • 1.3. Configuring and running mdtmFTP server
  • 1.4. Configuring and running mdtmFTP client
• 2. Managing credentials
  • 2.1. Managing credentials: username/password authentication
  • 2.2. Managing credentials: GSI certificates
• 3. Running mdtmFTP server
  • 3.1. Running mdtmFTP server with or without root privilege
  • 3.2. Server configuration files
• 4. Running mdtmFTP client
• 5. Support for IPv6
  • 5.1. IPv6-enabled network
  • 5.2. IPv6-enabled hosts
  • 5.3. mdtmFTP command line
  • 5.4. Data transfer examples
• 6. Data transfer examples
  • 6.1. Client – Server data transfer (IPv4)
  • 6.2. Third party data transfer between two remote HOSTs (IPv4)
  • 6.3. Client – Server data transfer (IPv6)
  • 6.4. Third party data transfer between two remote hosts (IPv6)
• 7. Appendix: creating server mdtmconfig.json
  • 7.1. Topology section
  • 7.2. Online section
  • 7.3. Thread section
  • 7.4. File section
  • 7.5. Manually_configured_cpus section
  • 7.6. Server section
  • 7.7. Example
• 8. Appendix: creating server mdtmconfig.xml
  • 8.1. Topology section
  • 8.2. Online section
  • 8.3. Thread section
  • 8.4. File section
• 9. Appendix: creating server.conf
• 10. Appendix: creating client mdtmconfig.json
  • 10.1. Topology section
  • 10.2. Online section
  • 10.3. Thread section
  • 10.4. File section
  • 10.5. Manually_configured_cpus section
  • 10.6. Example
• 11. Appendix: creating client mdtmconfig.xml
• 12. Appendix: Recommended host CAs
  • 12.1. Using InCommon
  • 12.2. Using LetsEncrypt
  • 12.3. Using SimpleCA