[mpiwg-tools] Mini example of single tool working with multiple MPI implementations

[Jean-Baptiste BESNARD]

Hi Matthew, All,

Indeed, it is a very good idea, it should be possible to chain calls 
using such opaque data-structure.

Just some random ideas here, complementary to what has already been 
introduced:

I consider that we want to make the tool independent from the underlying 
MPI to allow "drop-in" tools.
I was thinking during the webex of/MPI_Type_get_envelope/ and 
/MPI_Type_get_contents/ which provide such
abstraction for data-types (p.121 of 3.1) wouldn't it be possible to do 
it for all MPI functions, packing on the stack before chaining ?

/* Internal runtime representation
     passed as an opaque handler
     should be a common denominator
     between function footprints
     (not the case here !) */
typedef struct
{
     char * function_name;
     int ints[16];
     void * pointers[8];
     MPI_Comm comms[2];
     MPI_Datatype types[10];
     MPI_Status statuses[10];
MPI_Status requests[10]; ...
     /* Some pointers to handle calls with arrays (Waitall, Testall, ... 
no need to copy just reference here getters will know where to look) */
     MPI_Status * statusses_overflow;
     MPI_Status * requests_overflow;
     /* QMPI Call Chain CTX */
     int depth_in_the_chain;
     int (*handler_chain[10])( void *);
}QMPI_Context;

/param_t/ could point to such context which would be filled prior to the 
call and
passed through the handlers until reaching the actual MPI implementation
which would be able to reference packed elements (directly).

/* The actual call at the bottom of the chain (with implementation side 
knowledge) */

__INTERNAL__MPI_Send(ctx->pointers[0], ctx->ints[0], ctx->types[0], ctx->ints[1], ctx->ints[2], ctx->comms[0])

Could be used like this in the tool, as in the example, all wrappers 
would have the same footprint just a "void *"
making bindings very simple:

static void mpisend_wrapper(param_t param)
{

    printf("START mpisend tool wrapper. Param is %s\n", (param == 
MPI_PARAM_VAL) ? "correct" : "wrong");
QMPI_Wrapper_chain(param);
printf("DONE mpisend tool wrapper. Param is %s\n", (param == 
MPI_PARAM_VAL) ? "correct" : "wrong");
}

Stacking would allow a tool to inspect the result of a call for 
functions outputting
elements, as in PMPI. Also the context could be used to resolve the next 
function to call (QMPI chain, incrementing an integer and calling).

With the same infrastructure, it is possible to imagine that the tool 
submits "generic" calls, filling a QMPI_Context
(this is why I've put a /name/ field). For example if you do a profile 
and want to reduce performance data at the end,
you could do it without linking to MPI, you just need the qmpi.h header 
(too much ? sort of "UMPI - Universal MPI"):

/* Allocate runtime side storage (you don't know MPI handle sizes !)*/
QMPI_Context_init( &ctx, int sizes, int sizes, ... );
/* Function to call */
QMPI_Context_set_name( &ctx, "MPI_Reduce");
/* Sendbuf */
QMPI_Context_set_pointer( &ctx, 0 /*index*/, sendbuf);
/* Recvbuf */
QMPI_Context_set_pointer( &ctx, 1 /*index*/, recvbuf);
/* Count */
QMPI_Context_set_int( &ctx, 0 /*index*/, count);
/* Datatype */
QMPI_Context_set_type( &ctx, 0 /* Index */, QMPI_CHAR );
/* OP */
QMPI_Context_set_op( &ctx, 0 /* Index */, QMPI_SUM );
/* Root */
QMPI_Context_set_int( &ctx, 1 /*index*/, 0);
/* Comm */
QMPI_Context_set_comm( &ctx, 0 /* Index */, QMPI_COMM_WORLD );
QMPI_Submit( &ctx );
/* There would be a GIANT switch behind this ! */

A small subset of data-types, Ops and Comms could be exposed as 
standardized QMPI_* defines (for example a dup of COMM_WORLD per plugin 
QMPI_COMM_PLUGIN).

However, if the opaque handle is used, one would need to be able to 
inspect /MPI_Status/, /MPI_Comm/,/MPI_Co//unt/ and /MPI_Datatype/ in an 
abstract manner:

/* Exposed interface relative to a context as you cannot expose the size 
of the Comm */
int QMPI_Comm_size( QMPI_Context * ctx, int comm_index, int * size )
{
      /* In the MPI implem */
      MPI_Comm_size( ctx->coms[ comm_index], size );
}

... where do we stop ?

No problem for integers, you just need to know what is where 
(standardized as for get_contents), idem for void *
you just need something like this on tool side (but only C types this way):

int QMPI_Context_get_envelope( QMPI_Context * ctx , int * int_count, int 
* addr_count, int * datatype_count, int * comm_count... );
int QMPI_Context_get_contents( QMPI_Context * ctx , int * ints, void * 
addr, ... );

If you are interested, I'll try do draft a first implementation later in 
the week to see the performance overhead.

Thank you very much for the interesting discussions examples and ideas,

Jean-Baptiste.

Le 22/10/2015 19:34, Matthew LeGendre a écrit :
>
> At the end of the call, Jeff asked about a simple example that 
> illustrates building a tool with QMPI that works with multiple MPI 
> implementations. I've attached a tarball that illustrates what I was 
> picturing.
>
> This example builds a pair of libraries, libfakeopenmpi.so and 
> libfakempich.so, which each export "mpisend" functions with different 
> signatures as well as a pair of fake apps, app_openmpi and app_mpich, 
> that use each library.  It also builds a single libtool.so, which 
> contains wrapper functions for each mpisend and dynamically chooses 
> which one is used.
>
> Note that the tool library still has files built against each MPI 
> implementation, but produces a single library that works with either 
> implementation.
>
> On the phone I speculated that you'd need the callback registration 
> functions to be ABI compatible across all MPI implementations, but 
> this didn't turn up being necessary.  You can get away with only the 
> MPI identification function having a standard ABI.  In this example I 
> called the function get_mpi_implementation().
>
> It also turned up being very convenient if the MPI header exported the 
> return value of get_mpi_implementation() though a #define (which I 
> called MPI_IMPL).  This allowed for an easy runtime test of whether 
> the a tool implementation was compiled for the running MPI.
>
> Also, On the phone I speculated that getting per-compiler tools would 
> be difficult.  You actually could solve part of the problem by 
> including a compiler name and version in the MPI_IMPL #define--most 
> compilers have preprocessor macros that exports that info.  Though 
> this still wouldn't solve any tool incompatibilities with runtime 
> libraries like libstdc++.so.
>
> -Matt
>
>
>
>
> _______________________________________________
> mpiwg-tools mailing list
> mpiwg-tools at lists.mpi-forum.org
> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpiwg-tools

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mpi-forum.org/pipermail/mpiwg-tools/attachments/20151022/b76bdb7b/attachment-0001.html>