[Mpi-forum] ABI

Pavan Balaji balaji at mcs.anl.gov
Wed Dec 4 18:20:53 CST 2013

Agreed.  I should also add one more item:

 - Bug-fixes (or sorts): if we decide that our MAX_STRING_LENGTH is too short and want to increase it, that’s going to be a pain in the neck with ABI compatibility.  Everyone has to change it.  If variables in the status object have interleaved 64-bit and 32-bit objects, the compiler might pad them and waste space.  You can’t reorder them without breaking ABI.

Even if the eventual goal was for universal ABI compliance, I’d still recommend starting small with fairly similar MPI implementations, and then take the lessons learnt to other MPI implementations.

And yeah, we are not going to switch to OMPI internal implementation conventions.  These are two different design choices, each of which has its own advantages.  Isn’t that the point of having multiple implementations?  But more importantly, from a practicality standpoint, that’s a huge code churn which I can’t imagine either code base will be willing to go through.

  — Pavan

On Dec 4, 2013, at 6:01 PM, Jeff Squyres (jsquyres) <jsquyres at cisco.com> wrote:

> On Dec 4, 2013, at 4:23 PM, Jed Brown <jedbrown at mcs.anl.gov> wrote:
>> It would make a _huge_ difference for distributions if MPICH and Open
>> MPI could have a compatible ABI.  
> We, too, would be happy if MPICH adopted all of Open MPI's internal implementation conventions.  
> ...but I don't see that happening any time soon.
> I don't say this glibly; there are years of momentum and complicated engineering issues involved.  More below.
>> However, the question of integer (MPICH) versus pointer (OMPI) handles
>> is a long-standing impasse. 
> To be blunt: this totally misses the point.
> Integer vs. pointer is but one of MANY issues that prevent MPI implementations from being ABI compatible.  Let's not also forget:
> - Values of constants.  Simple integer values are usually easy to device/resolve (but don't forget that some integer values are specifically chosen on specific platforms). Sentinel values like MPI_STATUS_IGNORE are not.
> - Size/content of MPI_Status.  MPI implementations hide (different) non-public fields in MPI_Status.
> - Launcher differences.  The mpirun/mpiexec (or whatever launcher) is inherently different, with different CLI options and configuration, between different MPI implementations.
> - Run-time system differences.  Some MPI's support different run-time systems, some don't.  This is typically exposed through the launcher, but also has impact on the MPI processes themselves.  More specifically: the launcher and the MPI library (and support libraries) usually go in pairs and cannot be separated.
> ==> Note that this is a giant problem.  Many have tried to have create a unified parallel run-time over the past ~15 years.  I think it's fair to say that none have succeeded (i.e., none have taken over the world / become the one run-time system that everyone uses).
> - Library names.  libmpi.so?  libompi.so?  libmpich.so?  libmpi.a?  And so on.
> - Dependent libraries.  What else do you need to link when linking the application?  You can (usually) hide this when the MPI is a shared library, but a) not always, and b) that doesn't help when MPI is a static library.
> - Compiler ABIs.  MPI middleware cannot solve the fact that C++ and Fortran compilers cannot (and will not) agree on an ABI (for many good reasons, BTW).
> - Compiler options.  Was the MPI library compiled with -O3?  -i8?  -32 or -64?  ...?
> The fact of the matter is that the MPI API was *specifically designed with only source compatibility in mind*.  We now have nearly 20 years of momentum in different MPI implementations with different engineering design choices.  
> The term "ABI" is a catchall for many, many different issues.  People seem to think that simply switching libraries at run time is a silver bullet -- "if only I could just change my LD_LIBRARY_PATH and use a different MPI implementation, then the world would be better".  But let's also not forget that most users barely know how to use LD_LIBRARY_PATH (if at all).
> Sure, ABI between different releases of the same (MPI implementation+compiler+environment) is a Good Thing -- users want that.  And I think all major MPI implementations deliver that today.  It sure makes upgrading the MPI implementation a lot easier on the user.
> But ABI between different implementations is quite a different animal, with an entirely different set of challenges and payoffs.
> (I'm sure this post will inspire some impassioned replies :-) )
> -- 
> Jeff Squyres
> jsquyres at cisco.com
> For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/
> _______________________________________________
> mpi-forum mailing list
> mpi-forum at lists.mpi-forum.org
> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum

Pavan Balaji

More information about the mpi-forum mailing list