Today's guest post is written by Tanzima Islam, Post Doctoral Researcher at Lawrence Livermore Laboratory, and Kathryn Mohror and Martin Schulz, Computer Scientists at Lawrence Livermore Laboratory.

The latest version of the MPI Standard, MPI 3.0, includes a new interface for tools: the MPI Tools Information Interface, or "MPI_T".

MPI_T complements the existing MPI profiling interface, PMPI, and offers access to both internal performance information as well as runtime settings. It is based on the concept of typed variables that can be queried, read, and set through the MPI_T API.

In contrast to previously proposals that failed standardization, MPI_T does not make any explicit assumptions about what information an MPI implementation will provide. Instead, it allows each implementation to decide what information to expose and then provides an interface for users to query what information is available, how it is offered, and what metadata is associated with it.

As MPI implementations implement a broad spectrum of variables, this new interface can provide powerful new insights into application performance. MPI_T can also be used to self-document all configuration options. Both of these capabilities are often requested by users.

However, it comes with the usual chicken and egg problem: MPI implementations will not put in the effort to provide a wide range of variables without tools, and tools will not use the interface without implementations providing useful variables.

Fortunately, though, all three large open source MPI implementations have started adding some variables. E.g., Open MPI exposes all of its 1,000+ configuration options and MPICH provides a range of performance variables previously accessible only through internal interfaces.

We at Lawrence Livermore National Laboratory have been developing first tools that make use of MPI_T:

• varlist is a small utility that can query all variables offered by a particular MPI implementation and display them to the user for documentation or selection purposes
• Gyan, named after the Sanskrit word for knowledge, provides profiling information gathered from performance variables.

Experiments with the latter have shown imbalances in memory allocation behavior, excessive use of unexpected message queues, or changes in progress behavior between processes. Both tools are available as open source from http://scalability.llnl.gov/ (git repositories are available at GitHub).

Together, these two tools give a first glimpse at the flexibility and versatility of the new interface and demonstrate the new opportunities users can gain from the use of MPI_T.

However, the success of MPI_T will depend both on the availably of more tools exploiting these new introspection capabilities, and also on users to request such tools from the tool vendors / developers. MPI implementations are largely customer-driven; MPI implementers will provide a larger range of variables if users ask for them.

If we can achieve critical mass, the MPI_T interface will be as successful as the existing PMPI interface, and the two tools presented here will be only the first in a long line of performance, debugging, and correctness tools leveraging MPI_T.

Publication release number: LLNL-ABS-654556