mdtmFTP Docker release

About mdtmFTP

To address challenges in high performancedata movement for large-scale science, the Fermilab network research group has developed mdtmFTP, a high-performance data transfer tool to optimize data transfer on multicore platforms. mdtmFTP has a number of advanced features.

  • First, it adopts a pipelined I/O design. Data transfer tasks are carried out in a pipelined manner across multiple cores. Dedicated threads are spawned to perform network and disk I/O operations in parallel.
  • Second, mdtmFTP uses multicore-aware data transfer middleware (MDTM) to schedule an optimal core for each thread, based on system configuration, to optimize throughput across the underlying multicore core platform.
  • Third, mdtmFTP implements a large virtual file mechanism to efficiently handle lots-of-small-files (LOSF) situations.
  • Finally, mdtmFTP unitizes optimization mechanisms such as zero copy, asynchronous I/O, batch processing, and pre-allocated buffer pools, to maximize performance.

This document describes using Docker container releases of mdtmFTP. For manuals of source code or Singuarity releases of mdtmFTP, please visit the project website: http://mdtm.fnal.gov.

Docker releases of mdtmFTP are at https://hub.docker.com/r/wenji/mdtm/.

For technical details about mdtmFTP, please refer to the paper:

Liang Zhang,Wenji Wu, Phil DeMar, Eric Pouyoul: mdtmFTP and its evaluation on ESNET SDN testbed. Future Generation Comp. Syst.79:199-204 (2018)

Intended audience

This manual is intended for users and system administrators responsible for installing, running, and managing DTNs. The manual assumes familiarity with multicore and DTN concepts.

Contacts