HPC applications spend time in three phases: computation, communication (between processes) and input/output (I/O) that read/write data to persistent storage. Tools and techniques for profiling computation and communication are widely available as most codes spend time in these phases (e.g. check out the tools provided by POP partners), but I/O is often neglected. However, with the advent of large HPC systems, I/O is now becoming a bottleneck, hence the need for optimising I/O which can only be done once the I/O characteristics have been profiled.
The I/O profiling tool Darshan, developed and maintained by the Mathematics and Computer Science division of Argonne National Laboratory, is an open-source lightweight tool for profiling I/O of MPI applications. Darshan is able to profile the following methods:
- POSIX I/O
- MPI-IO
- parallel NetCDF
- parallel HDF5
It supports to profiling application codes written in C, C++ and Fortran.
An quick overview on how to use Darshan to profile the I/O of your application is now available in our Learning Materials section.