[mpiwg-tools] PMPI for a complete MPI wrapper

Wed Oct 4 11:22:58 CDT 2017

Hi Max, 

As Marc-André said wi4mpi was designed to avoid the recompilation phase of large applications required each time you need to change the underlying MPI implementation. Basically, wi4mpi allows to change the internal representation of all MPI type declared in the mpi.h without recompiling the application in "preload" mode. Wi4mpi provides also its own MPI interface and translate types to the underlying MPI implementation in "interface" mode. In "interface" mode, you'll have to recompile your application. Currently, wi4mpi supports bi-directional ABI conversion for  OpenMPI, MPICH, IntelMPI, MPI Spectrum, wi4mpi ABI. By the end of the year we will add the MPC ABI.

If I understand correctly what you want to do, wi4mpi can provide the glue between your API (i.e. enriched MPI types) used by the application and the underlying MPI implementation. In this case, you'll have to recompile your application but it doesn't require code modification in the application. If you want to avoid application recompilation you'll need to modify wi4mpi internals. If you have questions on wi4mpi, we should take this off the list and keep everyone else in CC.

Regards, 
Marc

-----Message d'origine-----
De : Marc-Andre Hermanns [mailto:hermanns at jara.rwth-aachen.de] 
Envoyé : mercredi 4 octobre 2017 15:51
À : mpiwg-tools at lists.mpi-forum.org; Max Sagebaum
Cc : PERACHE Marc 600952
Objet : Re: [mpiwg-tools] PMPI for a complete MPI wrapper

Hi Max,

> thanks for the fast answer. With the pmpi.h I mean a file like mpi.h
> but only containing the PMPI_ Interface. As you suggested I might try
> to create a full wrapper myself. I took a look on the wi4mpi project
> and there approach seems to create there own interface aka. mpi.h and
> then wrap this to the intel MPI or OpenMPI implementation. Due to this
> approach, they know the data types and can generate the interface.

For the wi4mpi project, Jean-Baptiste and people from CEA may be the
right people to talk to.

The wi4mpi is a bit of a special project, as it is provides a software
'glue' to make MPI implementations interchangeable. It is a way to
overcome the missing ABI (i.e., a specification of types, etc.).

Usually, the users will have to recompile their application every time
they choose a different MPI (potentially also when using a different
version of the same MPI), as values and types in the mpi.h may have
changed. For large simulation codes, this can take a long time. When
you have a translating 'glue' like wi4mpi in between, you can swap MPI
implementations via LD_PRELOAD at the start time of the application.

I don't know enough about wi4mpi to really know what their goal is:
Have a mixed MPI run (e.g., couple two codes compiled against differnt
MPIs)? Use a library compiled for one MPI together with an application
compiled against another? Just make it easier for users to link
against the right MPI? All of the above?

@Marc? Any comments on what the design goal of wi4mpi is? Does it
support other MPI implementations beside Intel-MPI and Open-MPI?

(If this discussion drifts more towards 'wi4mpi' specifics, we should
take this off the list and keep everyone else in CC)

> In my library I wanted to use a light wight wrapper. That is I wanted
> to use the original data types. With this approach I currently have
> structures like:
> 
> struct AMPI_Comm {
> // my own data;
> MPI_Comm comm; // the original object
> };
> 
> I can then simply call the pmpi functions with the stored original
> object.
> If I have a wrapper such that there is a PMPI_Comm object available, I
> could do the following:
> struct MPI_Comm {
> // my own data;
> PMPI_Comm comm; // the original object
> };
> 
> If the wrapper should use the same types from a general mpi.h, then I
> do not know the types and would need to declare something like:
> 
> hidden_mpi.c
> #include <mpi.h>
> decltype(MPI_COMM_WORLD) PMPI_COMM_WORLD = MPI_COMM_WORLD;
> 
> and then I need to use PMPI_COMM_WORLD in my library and I can
> generate a hmpi.h wich contains lines like:
> #define MPI_COMM_WORLD PMPI_COMM_WORLD
> 
> Which could be included by the user. But in order use PMPI_* in my
> library, I need to specify the symbol in a header file for which I
> need the type. In order to get the type I need to include mpi.h which
> will define MPI_COMM_WORLD and I have a name clash.

mpi.h only _declares_ the prototype. It does not define anything
(apart from CPP macros, etc.).

If you provide your own types, then you will need to declare your own
prototypes, which I would generate (see below).

> So unfortunately I see no way in providing a wrapper without writing a
> complete MPI Interface, which I would like to avoid. I might be able
> to use the wi4mpi Project and use there interface as a base for my
> implementation, which would add a dependency to my project.

For just a 'classic' wrapper, you just need to provide the definition
(implementation) of the function you want to replace, adhering to the
declared (in mpi.h) function prototype.

> A third and very ugly option would be, that I define all my types as
> void* in the interface for the user. But this disables type checking
> and I still would need to wrap from void* to references of my types.
> 
> So I might just stay in my AMPI namespace and provide a macro for the
> user to either call regular mpi functions or my wrapper functions.

If you need 'classic' wrappers for your project, you might consider
generating that code with a generator like 'wrap' [2].

As I mentioned in my other mail, writing a 'classic' wrapper is
straight forward.

Cheers,
Marc-Andre

[2] https://github.com/LLNL/wrap

> On Wed, 2017-10-04 at 11:00 +0200, Jean-Baptiste BESNARD wrote:
>> Dear Max,
>>
>> I’m not sure I completely understand what you mean by a « pmpi.h »
>> however I may have some initial elements below.
>>
>> The PMPI interface is currently targeting MPI functions only and
>> indeed some of the values you’ll find in your executable will be
>> compile time constants.
>> In fact, most MPI types/Constants are implementation dependent,
>> there is no unified ABI.
>>
>> Nonetheless, you might be able to interpret them in your wrapper
>> library in order to have them « rerouted » to your target
>> implementation.
>> I mean, knowing the value of MPI_COMM_WORLD you could rewrite it to
>> be MPI_COMM_WORLD2. 
>> And for sure you wont’t find a PMPI_COMM_WORLD.
>>
>> I can help on writing a wrapper for the whole PMPI interface. See my
>> repo here: https://github.com/besnardjb/mpi-snippets
>> There is a simple python script generating VIM snippets for MPI from
>> JSON specs, it can easily be converted to a script generating the
>> whole MPI interface.
>>
>> Eventually, an approach close to what you want to do might
>> be https://github.com/cea-hpc/wi4mpi which operates this systematic
>> handler conversion between MPI flavors, but this clearly involves
>> some rewriting.
>>
>> Hope this helps.
>>
>> Regards,
>>
>> Jean-Baptiste.
>>
>>> Le 4 oct. 2017 à 10:35, Max Sagebaum
>>> <max.sagebaum at scicomp.uni-kl.de
>>> <mailto:max.sagebaum at scicomp.uni-kl.de>> a écrit :
>>> Hello @ all,
>>>
>>> my question is concerning the PMPI specification. I hope the list
>>> is the correct place to ask.
>>>
>>> I want to write a complete wrapper for MPI. That is every define,
>>> typedef and function will be wrapped and might be completely
>>> changed. Currently I prefixed everything with AMPI_ such that no
>>> name clashes exist. But the user would need to rename every
>>> occurrence of MPI_ with AMPI_
>>>
>>> I would now like to use the PMPI definition of MPI to define my
>>> wrappers as the MPI version which then use the PMPI definitions.
>>> Unfortunately I could not find tutorials for a complete wrapper.
>>>
>>> As an example take MPI_COMM_WORLD. I made a grep on the openmpi
>>> installation on my linux machine for PMPI_COMM_WORLD but the result
>>> was empty. The definition of MPI_COMM_WORLD was 
>>> #define MPI_COMM_WORLD OMPI_PREDEFINED_GLOBAL( MPI_Comm,
>>> ompi_mpi_comm_world)
>>> without any chance to switch to PMPI_COMM_WORLD as a predefined macro.
>>>
>>> I also checked the newest source tarball of openmpi and I could not
>>> find anything for PMPI_COMM_WORLD there.
>>>
>>> In the mpi 3.0 standard on page 555 in section 14.2.1 the
>>> requirements are just listed for functions. Was the definition of
>>> the PMPI_ supplements for defines, types etc. never discussed?
>>>
>>> I would have expected, that I can just include a pmpi.h and then I
>>> would have all the PMPI_ symbols without the MPI symbols available.
>>>
>>> Do you know of any way I could make my idea work?
>>>
>>> Cheers
>>>
>>> Max
>>>
>>> -- 
>>> Max Sagebaum
>>>
>>> Chair for Scientific Computing,
>>> TU Kaiserslautern,
>>> Bldg/Geb 34, Paul-Ehrlich-Strasse,
>>> 67663 Kaiserslautern, Germany
>>>
>>> Phone: +49 (0)631 205 5638
>>> Fax:   +49 (0)631 205 3056
>>> Email: max.sagebaum at scicomp.uni-kl.de <mailto:max.sagebaum at scicomp.uni-kl.de>
>>> URL:   www.scicomp.uni-kl.de <http://www.scicomp.uni-kl.de>
>>>
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> mpiwg-tools mailing list
>>> mpiwg-tools at lists.mpi-forum.org
>>> <mailto:mpiwg-tools at lists.mpi-forum.org>
>>> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-tools
>>
>> _______________________________________________
>> mpiwg-tools mailing list
>> mpiwg-tools at lists.mpi-forum.org <mailto:mpiwg-tools at lists.mpi-forum.org>
>> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-tools
> 
> -- 
> 
> Max Sagebaum
> 
> Chair for Scientific Computing,
> TU Kaiserslautern,
> Bldg/Geb 34, Paul-Ehrlich-Strasse,
> 67663 Kaiserslautern, Germany
> 
> Phone: +49 (0)631 205 5638
> Fax:   +49 (0)631 205 3056
> Email: max.sagebaum at scicomp.uni-kl.de
> URL:   www.scicomp.uni-kl.de
> 
> 
> 
> 
> 
> 
> 
> _______________________________________________
> mpiwg-tools mailing list
> mpiwg-tools at lists.mpi-forum.org
> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-tools
> 

-- 
Marc-Andre Hermanns
Jülich Aachen Research Alliance,
High Performance Computing (JARA-HPC)
Jülich Supercomputing Centre (JSC)

Wilhelm-Johnen-Str.
52425 Jülich
Germany

Phone: +49 2461 61 2509 | +49 241 80 24381
Fax: +49 2461 80 6 99753
www.jara.org/jara-hpc
email: hermanns at jara.rwth-aachen.de