The MDTM Project

Latest Releases

mdtmGUI

A web-based tool developed to monitor and manage MDTM-enabled DTNs. However, it can also be deployed to monitor and manage any networked computer systems.

Live demo

• http://mdtm-server.fnal.gov:8080

Version 1.0.1

Release Date: Oct, 2016

OSes supported

• Linux kernel 3.12 or higher

Support Features

• Online and real-time monitoring of data transfer status and progress
• Online and real-time monitoring of DTN system status and configurations
• Online and real-time monitoring of MDTM-based data tranfer tool status and configurations

Docker container release

• https://hub.docker.com/r/wenji/mdtm/
• Installation and Configuration Manual

Source code release

• mdtmGUI-1.0.1.tar.gz
• Installation and Configuration Manual

MDTM Middleware

A user-space resource sceduler that harnesses multicore parallelism to scale data movement toolkits at multicore systems.

Version 1.0.5

Release Date: January 6, 2020

OSes supported

• Linux kernel 3.12 or higher

Support Features

• NUMA system topology profiling

• Online system status monitoring
  • cpu load per core
  • memory load latency per NUMA node
    • This feature allows applications to use system memory intellignently to avoid memory hotspots.

• NUMA topology-based resource scheduling

mdtmFTP generate three types of threads: Network I/O threads, Disk I/O threads, and Management threads. The MDTM middleware will schedule Disk and Network I/O threads to cores close to I/O devices (I/O locality).

• Supporting core affinity on network and disk I/Os

• System zoning

  The MDTM middleware will partition system cores into two zones -- a MDTM-zone and a Non-MDTM-zone. A data transfer application will run in the MDTM-zone while other applications will be confined within the Non-MDTM-zone. This strategy will reduce and minimize other applications' interference to the data transfer applicaiton, thus resulting in optimium data transfer performance.

Download links

• mdtm-1.0.5.tar.gz

mdtmFTP Software Package

A high-performance data transfer tool that build upon the MDTM middleware.

Version 1.1.1

Release Date: January 6, 2020

OSes supported

• Linux kernel 3.12 or higher

Support Features

• Adotp a pipelined I/O design

  Data transfer tasks are carried out in a pipelined manner across multiple cores. Dedicated I/O trheads are spawned to performed network and disk I/O operations in parallel, with each thread pinned to a particular core.

• Implement a large virtual file mechanism to efficiently handle lots-of-small-files (LOSF) situations.

  Different from tar-based solutions, the virtual file is logically, instead of physically, created. Data is transferred on a per-dataset basis, instead of on a per-file basis.

• Use MDTM middleware servcies to fully utilize the underlying multicore system

Singularity container release

• README (Note: please use Singularity 3.0+)
• mdtmFTP Singularity image
• mdtmFTP Singularity Def

Docker container release

• FNAL Docker Repository

• publicregistry.fnal.gov
• mdtmftp/mdtmftp:1.1.1-xenial

• Instructions

1. Copy repository certificate to "/etc/docker/certs.d/publicregistry.fnal.gov"
2. Pull images
   • docker pull publicregistry.fnal.gov/mdtmftp/mdtmftp:1.1.1-xenial

• Docker Hub Repository

• https://hub.docker.com/r/wenji/mdtm/

• Docker release Manual

• mdtmFTP Docker Release Manual (html version)

Source code release

• mdtmftp-1.1.1.tar.gz (contact: wenji@fnal.gov)
• mdtmFTP Installation and Configuration Manual
• mdtmFTP Manual (html version)