<html><head><meta http-equiv="Content-Type" content="text/html; charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class=""><div style="margin: 0px;" class=""><font face="Helvetica" class="" style="font-stretch: normal; line-height: normal;">Hi all,</font></div><div class=""><br class=""></div><div class="">I took another pass at updating the QMPI API based on our discussion in the working group today. I wasn’t going to post this on the pull request yet since I haven’t yet updated the implementation, but I wanted to start circulating it to get additional feedback. Please let me know if you think I misunderstood something or you think something should change.</div><div class=""><br class=""></div><div class=""><div class="">Thanks,</div><div class="">Wes</div></div><div style="margin: 0px;" class=""><font face="Helvetica" class="" style="font-stretch: normal; font-size: 21px; line-height: normal;"><b class=""><br class=""></b></font></div><div style="margin: 0px;" class=""><font face="Helvetica" class="" style="font-stretch: normal; font-size: 21px; line-height: normal;"><b class="">QMPI API</b></font></div><div style="margin: 0px; font-stretch: normal; font-size: 14px; line-height: normal; min-height: 17px;" class=""><font class="" style="font-variant-ligatures: no-common-ligatures;"></font><br class=""></div><div style="margin: 0px;" class=""><font face="Helvetica" class="" style="font-stretch: normal; font-size: 15px; line-height: normal;"><b class="">Tool Loading</b></font></div><div style="margin: 0px; font-stretch: normal; font-size: 14px; line-height: normal; min-height: 17px;" class=""><font class="" style="font-variant-ligatures: no-common-ligatures;"></font><br class=""></div><div style="margin: 0px;" class=""><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-variant-ligatures: no-common-ligatures;">The tool should use a system specific option to initialize the library when it's loaded. GCC and other compiler use the</font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;"> </font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; font-variant-ligatures: no-common-ligatures;">__attribute__((constructor</font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;">)</font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-variant-ligatures: no-common-ligatures;"> trick to accomplish this. The same can be used during finalization (if simple interception of</font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;"> </font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; font-variant-ligatures: no-common-ligatures;">MPI_FINALIZ</font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;">E</font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-variant-ligatures: no-common-ligatures;"> is insufficient.</font></div><div style="margin: 0px; font-stretch: normal; font-size: 14px; line-height: normal; min-height: 17px;" class=""><font class="" style="font-variant-ligatures: no-common-ligatures;"></font><br class=""></div><div style="margin: 0px;" class=""><font face="Helvetica" class="" style="font-stretch: normal; font-size: 15px; line-height: normal;"><b class="">Registration</b></font></div><div style="margin: 0px; font-stretch: normal; font-size: 14px; line-height: normal; min-height: 17px;" class=""><font class="" style="font-variant-ligatures: no-common-ligatures;"></font><br class=""></div><div style="margin: 0px;" class=""><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-variant-ligatures: no-common-ligatures;">When the tool is loaded, it needs to be registered with the MPI library. This happens with this function:</font></div><div style="margin: 0px; font-stretch: normal; font-size: 14px; line-height: normal; min-height: 17px;" class=""><font class="" style="font-variant-ligatures: no-common-ligatures;"></font><br class=""></div><div style="margin: 0px;" class=""><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; font-variant-ligatures: no-common-ligatures;">int QMPI_Register_tool_name(char * tool_name, void (* init_function_ptr)(int tool_id));</font></div><div style="margin: 0px; font-stretch: normal; font-size: 14px; line-height: normal; min-height: 17px;" class=""><font class="" style="letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;"></font><br class=""></div><div style="margin: 0px;" class=""><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; font-variant-ligatures: no-common-ligatures;">tool_nam</font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;">e</font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-variant-ligatures: no-common-ligatures;"> - [IN] Character string - A unique string representing the name of the tool</font></div><div style="margin: 0px;" class=""><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; font-variant-ligatures: no-common-ligatures;">init_function_pt</font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;">r</font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-variant-ligatures: no-common-ligatures;"> - [IN] Function pointer - Pointer to a function that MPI will called before MPI is initialized</font></div><div style="margin: 0px; font-stretch: normal; font-size: 14px; line-height: normal; min-height: 17px;" class=""><font class="" style="font-variant-ligatures: no-common-ligatures;"></font><br class=""></div><div style="margin: 0px;" class=""><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-variant-ligatures: no-common-ligatures;">This function allows a tool to “announce” itself to MPICH, but does not insert a tool into the QMPI call stack. The</font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;"> </font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; font-variant-ligatures: no-common-ligatures;">tool_nam</font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;">e</font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-variant-ligatures: no-common-ligatures;"> provided needs to match the one provided in the environment variable describing the tool order (described later). At some point before MPI is initialized, it will call all of the callback functions provided via</font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;"> </font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; font-variant-ligatures: no-common-ligatures;">init_function_pt</font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;">r</font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-variant-ligatures: no-common-ligatures;">. These functions may be called at any point between the calling of</font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;"> </font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; font-variant-ligatures: no-common-ligatures;">QMPI_Register_too</font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;">l</font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-variant-ligatures: no-common-ligatures;"> and executing</font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;"> </font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; font-variant-ligatures: no-common-ligatures;">MPI_Ini</font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;">t</font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-variant-ligatures: no-common-ligatures;">. For MPICH, this will probably be implemented in the wrapper code that gets executed between when an application calls</font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;"> </font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; font-variant-ligatures: no-common-ligatures;">MPI_Ini</font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;">t</font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-variant-ligatures: no-common-ligatures;"> and when the QMPI tools that intercept</font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;"> </font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; font-variant-ligatures: no-common-ligatures;">MPI_Ini</font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;">t</font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-variant-ligatures: no-common-ligatures;"> are called.</font></div><div style="margin: 0px; font-stretch: normal; font-size: 14px; line-height: normal; min-height: 17px;" class=""><font class="" style="font-variant-ligatures: no-common-ligatures;"></font><br class=""></div><div style="margin: 0px;" class=""><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-variant-ligatures: no-common-ligatures;">The callback function pointer looks like this:</font></div><div style="margin: 0px; font-stretch: normal; font-size: 14px; line-height: normal; min-height: 17px;" class=""><font class="" style="font-variant-ligatures: no-common-ligatures;"></font><br class=""></div><div style="margin: 0px;" class=""><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; font-variant-ligatures: no-common-ligatures;">void init_function_ptr(int tool_id);</font></div><div style="margin: 0px; font-stretch: normal; font-size: 14px; line-height: normal; min-height: 17px;" class=""><font class="" style="letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;"></font><br class=""></div><div style="margin: 0px;" class=""><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; font-variant-ligatures: no-common-ligatures;">tool_i</font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;">d</font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-variant-ligatures: no-common-ligatures;"> - [OUT] Integer - A unique identifier for the tool</font></div><div style="margin: 0px; font-stretch: normal; font-size: 14px; line-height: normal; min-height: 17px;" class=""><font class="" style="font-variant-ligatures: no-common-ligatures;"></font><br class=""></div><div style="margin: 0px;" class=""><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-variant-ligatures: no-common-ligatures;">This callback function is called one time for each instance of the tool that was requested via the environment variable specifying the order and quantity of tools. Each instance of the tool receives a unique</font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;"> </font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; font-variant-ligatures: no-common-ligatures;">tool_i</font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;">d</font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-variant-ligatures: no-common-ligatures;">. These tool IDs are not necessarily monotonically increasing and cannot be assumed to do so.</font></div><div style="margin: 0px; font-stretch: normal; font-size: 14px; line-height: normal; min-height: 17px;" class=""><font class="" style="font-variant-ligatures: no-common-ligatures;"></font><br class=""></div><div style="margin: 0px;" class=""><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-variant-ligatures: no-common-ligatures;">Inside this callback function, the tool should register any tool-specific storage it will to use as well as any MPI functions it plans to intercept. This happens through the following two functions.</font></div><div style="margin: 0px; font-stretch: normal; font-size: 14px; line-height: normal; min-height: 17px;" class=""><font class="" style="font-variant-ligatures: no-common-ligatures;"></font><br class=""></div><div style="margin: 0px;" class=""><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; font-variant-ligatures: no-common-ligatures;">int QMPI_Register_tool_storage(int tool_id, void *tool_storage);</font></div><div style="margin: 0px; font-stretch: normal; font-size: 14px; line-height: normal; min-height: 17px;" class=""><font class="" style="letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;"></font><br class=""></div><div style="margin: 0px;" class=""><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; font-variant-ligatures: no-common-ligatures;">tool_i</font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;">d</font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-variant-ligatures: no-common-ligatures;"> - [IN] Integer - A unique identifier for the tool</font></div><div style="margin: 0px;" class=""><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; font-variant-ligatures: no-common-ligatures;">tool_storag</font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;">e</font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-variant-ligatures: no-common-ligatures;"> - [IN] Address - A pointer to an address where the tool has data being stored for its particular instance</font></div><div style="margin: 0px; font-stretch: normal; font-size: 14px; line-height: normal; min-height: 17px;" class=""><font class="" style="font-variant-ligatures: no-common-ligatures;"></font><br class=""></div><div style="margin: 0px;" class=""><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-variant-ligatures: no-common-ligatures;">This function allows the tool to provide a storage object </font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;">(</font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; font-variant-ligatures: no-common-ligatures;">tool_storag</font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;">e</font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-variant-ligatures: no-common-ligatures;">) which lets the tool store information specific to its instance of itself (in case there are multiple copies of a tool). The specific instance of the tool is identified with the</font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;"> </font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; font-variant-ligatures: no-common-ligatures;">tool_i</font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;">d</font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-variant-ligatures: no-common-ligatures;"> value and should be matched with the one returned by the callback function that the tool registered with MPI.</font></div><div style="margin: 0px; font-stretch: normal; font-size: 14px; line-height: normal; min-height: 17px;" class=""><font class="" style="font-variant-ligatures: no-common-ligatures;"></font><br class=""></div><div style="margin: 0px;" class=""><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; font-variant-ligatures: no-common-ligatures;">int QMPI_Register_tool_function(int tool_id, enum QMPI_Functions_enum function_enum, void (* function_ptr)(void))</font></div><div style="margin: 0px; font-stretch: normal; font-size: 14px; line-height: normal; min-height: 17px;" class=""><font class="" style="letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;"></font><br class=""></div><div style="margin: 0px;" class=""><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; font-variant-ligatures: no-common-ligatures;">tool_i</font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;">d</font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-variant-ligatures: no-common-ligatures;"> - [IN] Integer - A unique identifier for the tool</font></div><div style="margin: 0px;" class=""><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; font-variant-ligatures: no-common-ligatures;">function_enu</font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;">m</font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-variant-ligatures: no-common-ligatures;"> - [IN] Enum value - An enumerated value specifying which MPI function is being registered for interception.</font></div><div style="margin: 0px;" class=""><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; font-variant-ligatures: no-common-ligatures;">function_pt</font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;">r</font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-variant-ligatures: no-common-ligatures;"> - [IN] Function pointer - A pointer to a function that should be called when an application calls the specified function.</font></div><div style="margin: 0px; font-stretch: normal; font-size: 14px; line-height: normal; min-height: 17px;" class=""><font class="" style="font-variant-ligatures: no-common-ligatures;"></font><br class=""></div><div style="margin: 0px;" class=""><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-variant-ligatures: no-common-ligatures;">This function accepts the parameter</font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;"> </font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; font-variant-ligatures: no-common-ligatures;">tool_i</font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;">d</font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-variant-ligatures: no-common-ligatures;"> as the specific instance of a tool and should be matched with the one returned by the callback function that the tool registered with MPI.</font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;"> </font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; font-variant-ligatures: no-common-ligatures;">function_enu</font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;">m</font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-variant-ligatures: no-common-ligatures;"> is used to specify which MPI function is being registered. The enum values are defined in the enum</font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;"> </font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; font-variant-ligatures: no-common-ligatures;">QMPI_Functions_enu</font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;">m</font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-variant-ligatures: no-common-ligatures;"> in</font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;"> </font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; font-variant-ligatures: no-common-ligatures;">mpi.</font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;">h</font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-variant-ligatures: no-common-ligatures;"> and generally are of this form:</font></div><div style="margin: 0px; font-stretch: normal; font-size: 14px; line-height: normal; min-height: 17px;" class=""><font class="" style="font-variant-ligatures: no-common-ligatures;"></font><br class=""></div><div style="margin: 0px;" class=""><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; font-variant-ligatures: no-common-ligatures;">enum QMPI_Functions_enum {</font></div><div style="margin: 0px;" class=""><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; font-variant-ligatures: no-common-ligatures;"> <span class="Apple-converted-space"> </span>MPI_ABORT_T,</font></div><div style="margin: 0px;" class=""><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; font-variant-ligatures: no-common-ligatures;"> <span class="Apple-converted-space"> </span>MPI_ACCUMULATE_T,</font></div><div style="margin: 0px;" class=""><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; font-variant-ligatures: no-common-ligatures;"> <span class="Apple-converted-space"> </span>... snip ...</font></div><div style="margin: 0px;" class=""><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; font-variant-ligatures: no-common-ligatures;"> <span class="Apple-converted-space"> </span>MPI_WTIME_T,</font></div><div style="margin: 0px;" class=""><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; font-variant-ligatures: no-common-ligatures;"> <span class="Apple-converted-space"> </span>MPI_LAST_FUNC</font></div><div style="margin: 0px;" class=""><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; font-variant-ligatures: no-common-ligatures;">};</font></div><div style="margin: 0px; font-stretch: normal; font-size: 14px; line-height: normal; min-height: 17px;" class=""><font class="" style="letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;"></font><br class=""></div><div style="margin: 0px;" class=""><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; font-variant-ligatures: no-common-ligatures;">function_pt</font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;">r</font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-variant-ligatures: no-common-ligatures;"> should be a pointer to the tool's interception function pointer. This function pointer will be the same signature as the MPI function, with an additional</font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;"> </font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; font-variant-ligatures: no-common-ligatures;">QMPI_Contex</font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;">t</font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-variant-ligatures: no-common-ligatures;"> argument in the first position as a way for tools to query information about the specific MPI procedure being intercepted when the callback functions are called. The type signature for each MPI function is defined in</font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;"> </font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; font-variant-ligatures: no-common-ligatures;">mpi.</font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;">h</font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-variant-ligatures: no-common-ligatures;"> in the form of </font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;"> </font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; font-variant-ligatures: no-common-ligatures;">QMPI_<function_name>_</font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;">t</font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-variant-ligatures: no-common-ligatures;"> (e.g.,</font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;"> </font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; font-variant-ligatures: no-common-ligatures;">QMPI_Send_t(QMPI_Context context, const void *buf, int count, MPI_Datatype datatype, int dest, int tag, MPI_Comm comm</font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;">)</font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-variant-ligatures: no-common-ligatures;">).</font></div><div style="margin: 0px; font-stretch: normal; font-size: 14px; line-height: normal; min-height: 17px;" class=""><font class="" style="font-variant-ligatures: no-common-ligatures;"></font><br class=""></div><div style="margin: 0px;" class=""><font face="Helvetica" class="" style="font-stretch: normal; font-size: 15px; line-height: normal;"><b class="">Interception</b></font></div><div style="margin: 0px; font-stretch: normal; font-size: 14px; line-height: normal; min-height: 17px;" class=""><font class="" style="font-variant-ligatures: no-common-ligatures;"></font><br class=""></div><div style="margin: 0px;" class=""><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-variant-ligatures: no-common-ligatures;">When the tool's interception function is called, it can get any tool-specific information from the context object. The fields in the context object can be retrieved with these functions:</font></div><div style="margin: 0px; font-stretch: normal; font-size: 14px; line-height: normal; min-height: 17px;" class=""><font class="" style="font-variant-ligatures: no-common-ligatures;"></font><br class=""></div><div style="margin: 0px;" class=""><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; font-variant-ligatures: no-common-ligatures;">QMPI_Context_get_storage(QMPI_Context context, int tool_id, void **storage);</font></div><div style="margin: 0px; font-stretch: normal; font-size: 14px; line-height: normal; min-height: 17px;" class=""><font class="" style="letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;"></font><br class=""></div><div style="margin: 0px;" class=""><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; font-variant-ligatures: no-common-ligatures;">contex</font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;">t</font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-variant-ligatures: no-common-ligatures;"> - [IN] Opaque handle - A handle to a context object containing tool and function-specific information</font></div><div style="margin: 0px;" class=""><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; font-variant-ligatures: no-common-ligatures;">tool_i</font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;">d</font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-variant-ligatures: no-common-ligatures;"> - [IN] Integer - A unique identifier for the tool</font></div><div style="margin: 0px;" class=""><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; font-variant-ligatures: no-common-ligatures;">storag</font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;">e</font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-variant-ligatures: no-common-ligatures;"> - [OUT] Address - A pointer to an address where the tool has data being stored for its particular instance</font></div><div style="margin: 0px; font-stretch: normal; font-size: 14px; line-height: normal; min-height: 17px;" class=""><font class="" style="font-variant-ligatures: no-common-ligatures;"></font><br class=""></div><div style="margin: 0px;" class=""><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-variant-ligatures: no-common-ligatures;">Get the back storage object the tool provided to</font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;"> </font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; font-variant-ligatures: no-common-ligatures;">QMPI_Register_tool_storag</font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;">e</font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-variant-ligatures: no-common-ligatures;"> that matches</font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;"> </font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; font-variant-ligatures: no-common-ligatures;">tool_i</font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;">d</font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-variant-ligatures: no-common-ligatures;">.</font></div><div style="margin: 0px; font-stretch: normal; font-size: 14px; line-height: normal; min-height: 17px;" class=""><font class="" style="font-variant-ligatures: no-common-ligatures;"></font><br class=""></div><div style="margin: 0px;" class=""><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; font-variant-ligatures: no-common-ligatures;">QMPI_Context_get_calling_address(QMPI_Context context, int tool_id, void *calling_address);</font></div><div style="margin: 0px; font-stretch: normal; font-size: 14px; line-height: normal; min-height: 17px;" class=""><font class="" style="letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;"></font><br class=""></div><div style="margin: 0px;" class=""><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; font-variant-ligatures: no-common-ligatures;">contex</font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;">t</font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-variant-ligatures: no-common-ligatures;"> - [IN] Opaque handle - A handle to a context object containing tool and function-specific information</font></div><div style="margin: 0px;" class=""><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; font-variant-ligatures: no-common-ligatures;">tool_i</font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;">d</font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-variant-ligatures: no-common-ligatures;"> - [IN] Integer - A unique identifier for the tool</font></div><div style="margin: 0px;" class=""><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; font-variant-ligatures: no-common-ligatures;">calling_addres</font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;">s</font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-variant-ligatures: no-common-ligatures;"> - [OUT] Address - The address of the application the called the MPI function.</font></div><div style="margin: 0px; font-stretch: normal; font-size: 14px; line-height: normal; min-height: 17px;" class=""><font class="" style="font-variant-ligatures: no-common-ligatures;"></font><br class=""></div><div style="margin: 0px;" class=""><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-variant-ligatures: no-common-ligatures;">The the address of the point where the application originally called the MPI function before entering the QMPI tool stack. If both a PMPI and QMPI tool are registered and active at the same time, the address will be that of the PMPI function, rather than the application.</font></div><div style="margin: 0px; font-stretch: normal; font-size: 14px; line-height: normal; min-height: 17px;" class=""><font class="" style="font-variant-ligatures: no-common-ligatures;"></font><br class=""></div><div style="margin: 0px;" class=""><font face="Helvetica" class="" style="font-stretch: normal; font-size: 15px; line-height: normal;"><b class="">Calling Additional QMPI Tools</b></font></div><div style="margin: 0px; font-stretch: normal; font-size: 14px; line-height: normal; min-height: 17px;" class=""><font class="" style="font-variant-ligatures: no-common-ligatures;"></font><br class=""></div><div style="margin: 0px;" class=""><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-variant-ligatures: no-common-ligatures;">When the tool is done with its own interception, it should call this function to determine the next tool in the QMPI stack (which the tool itself is responsible for calling):</font></div><div style="margin: 0px; font-stretch: normal; font-size: 14px; line-height: normal; min-height: 17px;" class=""><font class="" style="font-variant-ligatures: no-common-ligatures;"></font><br class=""></div><div style="margin: 0px;" class=""><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; font-variant-ligatures: no-common-ligatures;">int QMPI_Get_function(int tool_id, enum QMPI_Functions_enum function_enum, void (** function_ptr)(void), QMPI_Context * next_tool_context);</font></div><div style="margin: 0px; font-stretch: normal; font-size: 14px; line-height: normal; min-height: 17px;" class=""><font class="" style="letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;"></font><br class=""></div><div style="margin: 0px;" class=""><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; font-variant-ligatures: no-common-ligatures;">tool_i</font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;">d</font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-variant-ligatures: no-common-ligatures;"> - [IN] Integer - A unique identifier for the tool</font></div><div style="margin: 0px;" class=""><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; font-variant-ligatures: no-common-ligatures;">function_enu</font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;">m</font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-variant-ligatures: no-common-ligatures;"> - [IN] Enum value - An enumerated value specifying which MPI function is being registered for interception.</font></div><div style="margin: 0px;" class=""><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; font-variant-ligatures: no-common-ligatures;">function_pt</font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;">r</font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-variant-ligatures: no-common-ligatures;"> - [OUT] Function pointer - A pointer to a function that should be called when an application calls the specified function.</font></div><div style="margin: 0px;" class=""><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; font-variant-ligatures: no-common-ligatures;">next_tool_contex</font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;">t</font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-variant-ligatures: no-common-ligatures;"> - [OUT] Opaque handle - A handle to a context object containing tool and function-specific information</font></div><div style="margin: 0px; font-stretch: normal; font-size: 14px; line-height: normal; min-height: 17px;" class=""><font class="" style="font-variant-ligatures: no-common-ligatures;"></font><br class=""></div><div style="margin: 0px;" class=""><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-variant-ligatures: no-common-ligatures;">This function also uses</font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;"> </font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; font-variant-ligatures: no-common-ligatures;">tool_i</font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;">d</font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-variant-ligatures: no-common-ligatures;"> and</font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;"> </font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; font-variant-ligatures: no-common-ligatures;">function_enu</font><font face="Courier" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-family: Courier; letter-spacing: 3.2px; font-variant-ligatures: no-common-ligatures;">m</font><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-variant-ligatures: no-common-ligatures;"> to determine the current tool's ID and the enum value for the function being queried. It also provides a function pointer in a similar format to the registration function. Finally, the function returns the context object of the next tool in the call stack which should be used when calling the returned function.</font></div><div class=""><font face="Helvetica" class="" style="font-stretch: normal; font-size: 14px; line-height: normal; font-variant-ligatures: no-common-ligatures;"><br class=""></font></div></body></html>