Fermilab

MDTM Updates

  • LSST used mdtmFTP to demonstrate bulk data transfer at SC'18, Dallas, TX, Novbember 2018

LSST News Release: LSST 100 Gbps Network Demonstration at Supercomputing Conference 2018

  • Researchers used mdtmFTP to demonstrate bulk data transfer from KSTAR, South Korea to PPPL, USA, July 2017

Recently a demonstration was held of fast, large data transfer and near real-time analysis between the KSTAR tokamak at the National Fusion Research Institute (NFRI) and the Princeton Plasma Physics Laboratory (PPPL) in Princeton, USA. This demo was made possible by collaborative work with several researchers and personnel at NFRI and various institutions in the USA. In this demo, streaming large data transfer, with near real-time movie creation and analysis of the KSTAR Electron Cyclotron Emission imaging (ECEi) data, was performed using the I/O framework ADIOS, and comparisons made at PPPL with simulation results from the XGC1 code. Also demonstrated was large file transfer using the mdtmFTP tool, transferring 40 GB of data from NFRI servers to PPPL file systems in 2 minutes. These demonstrations were made possible utilizing an optimized network configuration at PPPL, which achieved over 8.8 Gbps (88% utilization) in throughput tests from NFRI to PPPL. This demonstration showed the feasibility of PPPL researchers connecting and participating with the KSTAR experiment remotely in near real-time, and the ability to utilize remote compute resources for large-scale data analysis of KSTAR data.

  • "Real-time scientific data streaming using Adios+mdtmFTP" demo at SC'16, Salt Lake City, Utah, 2016

We demonstrate remote data processing capability of large and high-throughput science experiment through cross-Pacific wide area networks and show how we can manage science workflow executions remotely by using ORNL ADIOS data management system and FNAL mdtmFTP data transfer system. In this demo, we show a fusion data processing workflow, called Gas Puff Imaging (GPI) analysis, to detect and trace blob movements during fusion experiment. We send GPI data streams from Singapore to Fermilab for near-real time analysis. ADIOS manages analysis workflows and mdtmFTP transports stream data.

This demo is presented at US DOE Booth, and Singapore ACRC Booth

ORNL Members: Scott Klasky, Jason Wang, Jong Youl Choi

FNAL Members: Wenji Wu, Liang Zhang, Phil DeMar

ACRC (SG) Members: Gabriel Noaje, Jonathan Low, Lim Seng

Demo slides

Demo movie .mov format

Demo movie .mp4 format

A recent press release published by Oak Ridge National Laboratory focused on a successful collaboration between ORNL and Fermilab Computing. CCD's Network Research group, with Wenji Wu as the principal investigator for the project, developed MDTM, a high performance data transfer tool, and worked with ORNL to integrate MDTM into an ORNL framework with great results.

Booth: Department of Energy (DOE) #502

Times: Wednesday, November 18, 11am-12pm, 3pm-4pm

Description: We use MDTM data transfer tools to demonstrate bulk data movement over wide area networks. We will compare MDTM with existing data transfer tools such as GridFTP and BBCP. Our purpose is to show the advantages of MDTM in fully utilizing the multi-core system resources, in particular with NUMA architecture.

Booth 1939: The Multicore-Aware Data Transfer Middleware (MDTM)

The Multicore-Aware Data Transfer Middleware (MDTM) project demonstrated how new manycore processors provided advanced features that can be exploited to design and implement a new generation of high-performance data movement tools.

The PIs reported the MDTM project progress at the PI meeting. Slides

  • Last modified
  • 02/14/2019