Which MPI implementation to choose? Which one is better? Which one is more suitable for my application? What is the difference between MPICH, Intel MPI and OpenMPI?
Who didn’t ask one of these questions, at least when he / she first started using or developing programs on HPC systems?
In my case, as being more a HPC user than a system developer / administator, when I first started developing CFD programs to run on HPC systems, it didn’t really matter to me: I was debugging and running my programs remotely on the supercomputer. And when I wanted to setup an MPI implementation on my desktop or laptop, sothat I could continue working when I couldn’t connect to the server, any implementation would have been suitable, as long as it’s well running on my system!
But when I started having more experience with HPC systems and MPI programming, those questions made actually more sense.
In fact, each MPI implementation tries to meet different needs. Then, I found it a bit hard, and maybe somehow hasty, to give a sharp answer for those questions, since each implementation has probably his advantages / inconvenients. So, I thought it would be better to give the advantages and desadvantages of each one, the reader could then decide, depending on his usage, which one is more suitable.
1. What is MPICH and OpenMPI
Let’s start with a small presentation of each implementation.
- MPICH
MPICH is considered to be one of the most popular implementations of the MPI standard. It is a high-quality reference implementation of the latest MPI standard (currently MPI-3), and is the basis for a wide range of MPI derivatives (including IBM MPI, Cray MPI, Intel MPI and MVAPICH among others).
As for the MPI standard coverage, MPICH seems to be superior to any other MPI implementation, being the first implementation of every single release of the MPI standard since MPI-1. One can refers to the MPI-3 standard implementation status, comparing the status of the MPI-3 standard intergration on different MPI implementations, and can conclude that MPICH is by far the most complete MPI implementation up-to-date.
However, considering the MPICH implementation without its derivatives, we notice that it suffers from a huge inconvenient, since it doesn’t support InfiniBand interconnects, considered as of 2014, the most commonly used interconnect in supercomputers. Nevertheless, many of its derivatives, including Intel MPI and MVAPICH do support InfiniBand. So, if one considers MPICH as “MPICH and its derivatives”, it is definitely the most complete MPI implementations with an extremely broad network support and proprietary interconnects (including IBM, Cray, Seastar…).
One of the areas where MPICH performances used to be poor, was the process manager, with the old MPICH launcher (MPD). By now, it has been replaced, and the new process manager (developed under the Hydra framework) seems to be as good as the OpenMPI one.
- OpenMPI
Although considered to target the common case, both in terms of usage and network support, OpenMPI is also a high-quality implementation of MPI standard, with a large number of contributors worldwide, including Bull, AMD, Nvidia, Cisco Systems and the Los Alamos National Lab., among others.
For the MPI standard coverage, OpenMPI is slightly behind MPICH. Referring to the MPI-3 standard implementation status, it is clear that OpenMPI is still working on integrating the latest MPI-3 standartization. Nevertheless, it’s interesting to highlight that OpenMPI is by far the only MPI implementation that supports FORTRAN 2008 Bindings (At least up to when those lines are written), which could be interesting for people developing with FORTRAN…
The OpenMPI process manager (ORTE for Open Run-Time Environment) seems to be very good and doesn’t have any noticeable weakness.
2. Platform-based Comparison
It appears that the first difference between those implementations depends highly on the system (architecture and OS) in which it will be executed. So, after giving a small idea on the capabilities of each implementation, it is interesting to make a platform-based comparison of both implementations, wether you are working on a desktop or a cluster / supercomputer.
- Desktop
Let’s be clear from the beginning: There is no reason to think about MPI performances if you are working on a desktop / laptop! So, starting from this point, one can assume that any of the implementations is suitable. True!
However, it seems that there is a small difference between both implementations, depending on the system / OS you are using. In fact, it seems that OpenMPI is much easier to setup on a Mac system, than MPICH. I recall having hard times installing MPICH2 on my laptop with Mac OS X Leopard, back in 2009. As far as I remember, the problem seemed to originate from the process manager (MPD at that time), and now that I have read reviews about this, it does make more sense. It was alot easier and more flexible setting OpenMPI up on my Macbook.
On Linux, there is no such problem, and both implementations seem to work fine, even if MPICH seems to be more Linux-friendly.
- Cluster / Supercomputer
Here, it is a bit sensitive area, epecially to me, since I’m not a HPC administrator!
For a cluster or a supercomputer, choosing one of the implementations is more a matter of performance, since both will work just fine, but one of them will deliver better performances than the other.
The choice highly depends on the architecture and the interconnections of the HPC system. Regarding the reviews of experts, the MPICH implementation with all its derivatives, seems to have a big advantage.
With all the previous elements in mind, I suggest you to discuss this issue with your HPC system administrator. You can also run few tests and specific benchmarks in order to have a better idea on the behaviour of each implementation.
Credit: This post has been inspired by an interesting discussion on StackOverflow.
Into The Wave 