[Mpi3-tools] Current Version of the MPIR document

[Dong H. Ahn]

Hi Martin, John and all,

I really like the idea of publishing this as an official document 
through the MPI forum. I've helped putting together several machine 
RFPs, and I've always been hoping to have this kind of document to point 
to describe this required interface.

Here are some questions and comments:

- Section 6.2
      I recently worked with a SLURM developer to implement 
MPIR_partial_attach_ok support within SLURM and pointed a draft of this 
document to him. In in regard to "When an implementation uses a 
synchronization technique that does not use the MPIR_debug_gate, and 
does not require the tool to attach to and continue the MPI process, it 
should define the symbol MPIR_partial_attach_ok (§9.13) in the starter 
process, and avoid defining MPIR_debug_gate in the MPI processes."

      We found that this "avoid defining MPIR_debug_gate in the MPI 
processes" is too strict. Resource management software layers like SLURM 
would want to support MPIR_partial_attach_ok without having to modify 
MPI binaries which may already define MPIR_debug_gate. But "undefining" 
an already defined symbol like this within an MPI binary isn't trivial. 
Could you consider changing this requirement to "avoid *using* 
MPIR_debug_gate in the MPI processes?" This is to say having the 
definition of MPIR_debug_gate within MPI is OK as far as it's not *used* 
for the synchronization?

      With respect to this consideration, though, I think it would also 
be good to be clearer about the relationship between 
MPIR_partial_attach_ok and MPIR_debug_gate described in Section 9.13 as 
well. In particular, the requirement,

      "The tool may choose to ignore the presence of the 
MPIR_partial_attach_ok symbol and acquire all MPI rank processes. The 
presence of this symbol does not prevent the tool from using the MPIR 
synchronization technique to acquire all of the processes, if it so 
chooses, because setting the MPIR_debug_gate variable (if present) is 
harmless."

      is confusing. Shouldn't setting the MPIR_debug_gate variable" is 
NOOP under this condition? If so, how could the tool acquire the 
processes using this technique?

- Section 7.1
      In this case, a potential race condition can occur as the tool 
cannot precisely control when those daemons are launched especially 
under a synchronization scheme like BlueGene's: The job may start 
running before daemons are launched. To minimize this kind of problem, 
can we be explicit about the requirement as to when the daemons should 
be launched in relation to when the job is loaded and run? On BlueGene, 
if the job is loaded and run before daemons are launched onto the I/O 
nodes and issue (pre)ATTACH commands to the compute node processes, the 
daemons won't be able to acquire the processes before the processes run 
away from their initial instructions.


- Section 9.2
      I like that this document encourages the MPI implementation to 
"share the host and executable name character strings across
multiple process descriptor entries whenever possible." The section 
states that this would improve tools' performance at scale. But our 
experience has been that it would improve RM's scalability as RMs 
themselves often consist of distributed components and the bigger 
strings means larger communication overheads within RM's 
inter-communication. And the impact gets magnified at extreme scale. So, 
can you consider saying to the effect that "not sharing these character 
strings have been a major scalability problem within the RMs themselves 
as well."


- Section 9.13
      In addition to the requirement that I mentioned in relation to 
Section 6.2 above,  can you be more explicit about "the tool need only 
release the starter process to release the whole MPI job, which can 
therefore be run without requiring the tool to acquire all of the MPI 
processes included in the MPI job." Perhaps, the tool need only to 
release the starter process from MPIR_Breakpoint?

Best,
Dong

On 6/14/2010 9:27 AM, Martin Schulz wrote:
> Hi all,
>
> Attached is the latest and updated version of the MPIR document, which John
> DelSignore put together. The intent is still to publish this through the MPI forum
> as an official document. The details for this are still tbd. and Jeff will lead a
> discussion on this topic during the forum this week.
>
> We don't have a tools WG meeting scheduled for meeting, but if you have
> any comments or feedback (on the document or how we should publish it),
> please post it to the list. If necessary or useful, we can also dedicate one
> of the upcoming tools TelCons for this.
>
> Thanks!
>
> Martin
>
> PS: Feel free to distribute the document further, in particular to tool and
> MPI developers.
>
>
>
>