[Mpi3-tools] Tools WG webex: tomorrow!

[David Goodell (dgoodell)]

On Jul 2, 2013, at 4:54 AM, George Bosilca <bosilca at icl.utk.edu> wrote:

> On Jul 1, 2013, at 20:37 , David Goodell (dgoodell) <dgoodell at cisco.com> wrote:
> 
>> I'm fairly sure that MPICH interprets the standard to mean invoke comm_delete_attr_fn at actual object destruction time, which may be some time after MPI_COMM_FREE returns.
>> 
>> I believe that the "at destruction time" interpretation is the most useful one, even if it's not clearly mandated nor universally implemented.
> 
> I have to disagree as I do believe that once there is no way to access an object, and as a side-effect the attributes attached to it, there is absolutely no reason to delay their release. 

A tool or stacked MPI library may have some outstanding resources associated with a communication operation (e.g., MPI_Isend) and/or communicator.  The "free time" interpretation requires much more careful interception and emulation of MPI semantics than the "destruction time" interpretation does.

> Maybe a even stronger case why the destruction of the attributes should be done when MPI_*_Free is called is the behavior imposed on MPI_COMM_SELF. The standard states at page 363 line 14 "When MPI_FINALIZE is called, it will first execute the equivalent of an MPI_COMM_FREE on MPI_COMM_SELF. Thus will cause the delete callback function to be executed on all keys associated with MPI_COMM_SELF, in the reverse other that they were set on MPI_COMM_SELF. … The 'freeing' of MPI_COMM_SELF occurs before any other parts of MPI are affected".

But MPI_Finalize is a special case of communicator destruction, since it's an invalid MPI program to have outstanding MPI communication operations at finalize time.  In other words, this is an invalid MPI program AIUI:

----8<----
MPI_Request req;

MPI_Init(...);
MPI_Isend(..., MPI_COMM_SELF, &req);
MPI_Finalize();
MPI_Wait(&req, MPI_STATUS_IGNORE);
----8<----

But this is completely valid:
----8<----
MPI_Request req;
MPI_Comm comm;

MPI_Init(...);
MPI_Comm_dup(MPI_COMM_SELF, &comm);
MPI_Isend(..., comm, &req);
MPI_Comm_free(&comm);
MPI_Wait(&req, MPI_STATUS_IGNORE);
MPI_Finalize();
----8<----

The standard is not written in a way that clearly specifies semantics in the general case.  Most of the explanatory text that it includes assumes that the "pending communication operation" case won't happen, which leads to ambiguity about behavior when it does.

> Of course one can have a special code-path for the case mentioned above, but a consistent behavior over all calls (equivalent or not) to a particular MPI function is more suitable.

Given a valid user program, I think that you do not need a special code path for finalize, irrespective of which implementation choice you make.

-Dave