[mpiwg-tools] Mini example of single tool working with multiple MPI implementations
jbbesnard at paratools.fr
Thu Oct 22 14:57:12 CDT 2015
Hi Matthew, All,
Indeed, it is a very good idea, it should be possible to chain calls
using such opaque data-structure.
Just some random ideas here, complementary to what has already been
I consider that we want to make the tool independent from the underlying
MPI to allow "drop-in" tools.
I was thinking during the webex of/MPI_Type_get_envelope/ and
/MPI_Type_get_contents/ which provide such
abstraction for data-types (p.121 of 3.1) wouldn't it be possible to do
it for all MPI functions, packing on the stack before chaining ?
/* Internal runtime representation
passed as an opaque handler
should be a common denominator
between function footprints
(not the case here !) */
char * function_name;
void * pointers;
MPI_Status requests; ...
/* Some pointers to handle calls with arrays (Waitall, Testall, ...
no need to copy just reference here getters will know where to look) */
MPI_Status * statusses_overflow;
MPI_Status * requests_overflow;
/* QMPI Call Chain CTX */
int (*handler_chain)( void *);
/param_t/ could point to such context which would be filled prior to the
passed through the handlers until reaching the actual MPI implementation
which would be able to reference packed elements (directly).
/* The actual call at the bottom of the chain (with implementation side
__INTERNAL__MPI_Send(ctx->pointers, ctx->ints, ctx->types, ctx->ints, ctx->ints, ctx->comms)
Could be used like this in the tool, as in the example, all wrappers
would have the same footprint just a "void *"
making bindings very simple:
static void mpisend_wrapper(param_t param)
printf("START mpisend tool wrapper. Param is %s\n", (param ==
MPI_PARAM_VAL) ? "correct" : "wrong");
printf("DONE mpisend tool wrapper. Param is %s\n", (param ==
MPI_PARAM_VAL) ? "correct" : "wrong");
Stacking would allow a tool to inspect the result of a call for
elements, as in PMPI. Also the context could be used to resolve the next
function to call (QMPI chain, incrementing an integer and calling).
With the same infrastructure, it is possible to imagine that the tool
submits "generic" calls, filling a QMPI_Context
(this is why I've put a /name/ field). For example if you do a profile
and want to reduce performance data at the end,
you could do it without linking to MPI, you just need the qmpi.h header
(too much ? sort of "UMPI - Universal MPI"):
/* Allocate runtime side storage (you don't know MPI handle sizes !)*/
QMPI_Context_init( &ctx, int sizes, int sizes, ... );
/* Function to call */
QMPI_Context_set_name( &ctx, "MPI_Reduce");
/* Sendbuf */
QMPI_Context_set_pointer( &ctx, 0 /*index*/, sendbuf);
/* Recvbuf */
QMPI_Context_set_pointer( &ctx, 1 /*index*/, recvbuf);
/* Count */
QMPI_Context_set_int( &ctx, 0 /*index*/, count);
/* Datatype */
QMPI_Context_set_type( &ctx, 0 /* Index */, QMPI_CHAR );
/* OP */
QMPI_Context_set_op( &ctx, 0 /* Index */, QMPI_SUM );
/* Root */
QMPI_Context_set_int( &ctx, 1 /*index*/, 0);
/* Comm */
QMPI_Context_set_comm( &ctx, 0 /* Index */, QMPI_COMM_WORLD );
QMPI_Submit( &ctx );
/* There would be a GIANT switch behind this ! */
A small subset of data-types, Ops and Comms could be exposed as
standardized QMPI_* defines (for example a dup of COMM_WORLD per plugin
However, if the opaque handle is used, one would need to be able to
inspect /MPI_Status/, /MPI_Comm/,/MPI_Co//unt/ and /MPI_Datatype/ in an
/* Exposed interface relative to a context as you cannot expose the size
of the Comm */
int QMPI_Comm_size( QMPI_Context * ctx, int comm_index, int * size )
/* In the MPI implem */
MPI_Comm_size( ctx->coms[ comm_index], size );
... where do we stop ?
No problem for integers, you just need to know what is where
(standardized as for get_contents), idem for void *
you just need something like this on tool side (but only C types this way):
int QMPI_Context_get_envelope( QMPI_Context * ctx , int * int_count, int
* addr_count, int * datatype_count, int * comm_count... );
int QMPI_Context_get_contents( QMPI_Context * ctx , int * ints, void *
addr, ... );
If you are interested, I'll try do draft a first implementation later in
the week to see the performance overhead.
Thank you very much for the interesting discussions examples and ideas,
Le 22/10/2015 19:34, Matthew LeGendre a écrit :
> At the end of the call, Jeff asked about a simple example that
> illustrates building a tool with QMPI that works with multiple MPI
> implementations. I've attached a tarball that illustrates what I was
> This example builds a pair of libraries, libfakeopenmpi.so and
> libfakempich.so, which each export "mpisend" functions with different
> signatures as well as a pair of fake apps, app_openmpi and app_mpich,
> that use each library. It also builds a single libtool.so, which
> contains wrapper functions for each mpisend and dynamically chooses
> which one is used.
> Note that the tool library still has files built against each MPI
> implementation, but produces a single library that works with either
> On the phone I speculated that you'd need the callback registration
> functions to be ABI compatible across all MPI implementations, but
> this didn't turn up being necessary. You can get away with only the
> MPI identification function having a standard ABI. In this example I
> called the function get_mpi_implementation().
> It also turned up being very convenient if the MPI header exported the
> return value of get_mpi_implementation() though a #define (which I
> called MPI_IMPL). This allowed for an easy runtime test of whether
> the a tool implementation was compiled for the running MPI.
> Also, On the phone I speculated that getting per-compiler tools would
> be difficult. You actually could solve part of the problem by
> including a compiler name and version in the MPI_IMPL #define--most
> compilers have preprocessor macros that exports that info. Though
> this still wouldn't solve any tool incompatibilities with runtime
> libraries like libstdc++.so.
> mpiwg-tools mailing list
> mpiwg-tools at lists.mpi-forum.org
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the mpiwg-tools