PIERNIK is a parallel astrophysical fluid simulation code containing diverse physical modules, including advanced magnetohydrodynamical (MHD), N-body and unique momentum-dependent cosmic ray propagation algorithms. It supports adaptive mesh refinement (AMR), multigrid and multipole techniques, with parallel HDF5 data I/O. The major performance challenges are related to MPI communication in the AMR mode and to inefficient parallel I/O.

Fig 1. The output from the test case used, showing the electromagnetic field of a spiral galaxy.

The POP performance assessment of PIERNIK used an ad-hoc test-case simulating the MHD evolution of a galaxy, as shown in Figure 1. By visual analysis of the data obtained by the POP assessment, the developers noticed some unexpected structures. These were associated with code sections that had previously been assumed to be well-behaved. They were well load-balanced and with a good IPC (instructions per cycle) ratio but the problem became evident once the Paraver timelines were analysed.

In particular, there were big memory copy operations in boundary exchange routines. Once identified, the developers were able to greatly reduce the number of memory copies. These changes, along with others related to the AMR communication, have resulted in a 40% reduction in the overall execution time.

“The POP assessment has already helped us to improve code performance by focusing us on routines where big algorithmic speedups are achievable.”
-- Piernik Code Developers

-- Federico Panichi (NAG)