<html><head></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; color: rgb(0, 0, 0); font-size: 14px; font-family: Calibri, sans-serif; "><div>I generally agree with Bill, although I think that Jeff's original proposal did a better job at that than the discussion which followed. That is, I think you could talk about state changing vs. non-state changing calls and perhaps localize which states change per-call and end up with a reasonable result. This might be worth doing anyway, in terms of Jeff H's original problem, where he needs to query the thread level to determine if he can query the thread level. Since that's clearly not a state changing operation, we could perhaps redefine Thread_single in a way that works for Jeff H (but probably not Jeff S)?</div><div><br></div><div>Brian</div><div><br></div><span id="OLK_SRC_BODY_SECTION"><div><div>On 3/25/13 1:08 PM, "William Gropp" <<a href="mailto:wgropp@illinois.edu">wgropp@illinois.edu</a>> wrote:</div></div><div><br></div><blockquote id="MAC_OUTLOOK_ATTRIBUTION_BLOCKQUOTE" style="BORDER-LEFT: #b5c4df 5 solid; PADDING:0 0 0 5; MARGIN:0 0 0 5;"><div><meta http-equiv="Content-Type" content="text/html; charset=utf-8"><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; ">
We need to be careful here. There is nothing about MPI_THREAD_MULTIPLE that requires a global lock or even locks at all. There is great freedom in the implementation, and it is also important to note that the MPI implementation is not required to enforce
any thread-related restrictions; i.e., it is not required to detect or correct for erroneous programs. It can be interesting to consider *semantics* that might offer an implementation additional opportunities (e.g., single reader/single writer). Whatever
we do, we should not be referring to global locks, as that is not an MPI concept.
<div><br></div><div>Bill</div><div><br><div><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div><div style="font-size: 12px; ">William Gropp</div><div style="font-size: 12px; ">Director, Parallel Computing Institute</div><div style="font-size: 12px; ">Deputy Director for Research</div><div style="font-size: 12px; ">Institute for Advanced Computing Applications and Technologies</div></div></div></span><span class="Apple-style-span" style="font-size: 12px; ">Thomas M. Siebel Chair in Computer Science</span><span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; font-size: medium; "><div style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><div><div style="font-size: 12px; ">University of Illinois Urbana-Champaign</div></div><div><br></div></div></span><br class="Apple-interchange-newline"></span><br class="Apple-interchange-newline"></div><br><div><div>On Mar 25, 2013, at 12:57 PM, Jeff Hammond wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div><blockquote type="cite"><blockquote type="cite">but I realized that it also addresses one of the primary challenges<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">to upping the requirement of thread-support in the standard, since<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">MPI_THREAD_PER_OBJECT is one possible way to achieve<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">MPI_THREAD_MULTIPLE-like behavior without requiring locks in<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite">performance-critical functions.<br></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote><blockquote type="cite">Yes, it could be. I would be happy if this indeed solved two problems. I'd be happy to assist you in fleshing out this proposal. Let me know how I can help.<br></blockquote><br>
My primary concerns are:<br><br>
- To what extent is per-object locking going to lead to better<br>
performance on the pertinent networks? I know how MPICH+PAMI works,<br>
but it is critical to get feedback from e.g. Intel, Cray, etc. about<br>
the possible problems that may exist. If you could drive the<br>
discussion from the Infiniband perspective, that would be great.<br><br>
- While MPI_Comm, MPI_Win and MPI_File are the most obvious examples,<br>
at least to me, where per-object locking can be used to achieve<br>
concurrency, I want to be sure that this makes sense. Are there<br>
aspects of the MPI standard that have nonlocal effects and therefore<br>
corrupt the clean separation of state implied by<br>
MPI_THREAD_PER_OBJECT? If so, does this actually matter? What I mean<br>
is that, if a function isn't performance critical, it seems<br>
sufficient, at least to me, to specify global-lock (global within the<br>
process, of course) semantics.<br><br>
- When functions take multiple objects as arguments, what are the<br>
concurrency rules? When I imagined this idea, it seems<br>
straightforward to me, but I certainly did not consider all ~500 MPI<br>
functions, particularly in parts of the standard that I don't<br>
currently use.<br><br>
- Is per-object locking actually the right model? Maybe it is better<br>
to talk in terms of a single-writer, multiple-reader requirement, i.e.<br>
that the per-object serialization is only required when the object<br>
state will be changed. MPI_Comm is the most complicated situation<br>
since I am assuming that message-queues will be per-communicator, and<br>
thus any call that takes an MPI_Comm argument and drives progress will<br>
potentially modify that communicator. We need to make the semantics<br>
of this very clear. On the other hand, I think that MPI_Info,<br>
MPI_Datatype, MPI_Group, and possibly other objects are pretty clear<br>
regarding the need for mutual exclusion only when the object state is<br>
being changed.<br><br>
Best,<br><br>
Jeff<br><br><br><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">-----Original Message-----<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">From: <a href="mailto:mpi3-hybridpm-bounces@lists.mpi-forum.org">
mpi3-hybridpm-bounces@lists.mpi-forum.org</a><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">[<a href="mailto:mpi3-hybridpm-">mailto:mpi3-hybridpm-</a> <a href="mailto:bounces@lists.mpi-forum.org">
bounces@lists.mpi-forum.org</a>] On Behalf Of Jeff<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Hammond<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Sent: Monday, March 25, 2013 10:08 AM<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">To: <a href="mailto:mpi3-hybridpm@lists.mpi-forum.org">mpi3-hybridpm@lists.mpi-forum.org</a><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Subject: Re: [Mpi3-hybridpm] Threading homeworking / next telecon<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">I finally wrote up another ticket that was my attempt at a more<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">holistic proposal for thread-safety. It's a bit raw, so please try<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">to extract the essence of it before poking at specific flaws.<br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><a href="https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/373">https://svn.mpi-forum.org/trac/mpi-forum-web/ticket/373</a><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><br></blockquote></blockquote></blockquote></blockquote><blockquote type="cite"><blockquote type="cite"><blockquote type="cite"><blockquote type="cite">Jeff<br></blockquote></blockquote></blockquote></blockquote><br>
-- <br>
Jeff Hammond<br>
Argonne Leadership Computing Facility<br>
University of Chicago Computation Institute<br><a href="mailto:jhammond@alcf.anl.gov">jhammond@alcf.anl.gov</a> / (630) 252-5381<br><a href="http://www.linkedin.com/in/jeffhammond">http://www.linkedin.com/in/jeffhammond</a><br>
<a href="https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond">https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond</a><br>
_______________________________________________<br>
Mpi3-hybridpm mailing list<br>
<a href="mailto:Mpi3-hybridpm@lists.mpi-forum.org">Mpi3-hybridpm@lists.mpi-forum.org</a><br>
<a href="http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm">http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm</a><br></div></blockquote></div><br></div></div></div></blockquote></span><div><br></div><div><div><div><br></div><div><div><div><div style="font-family: Consolas; font-size: medium; ">--</div><div style="font-family: Consolas; font-size: medium; "> Brian W. Barrett</div><div style="font-family: Consolas; font-size: medium; "> Scalable System Software Group</div><div style="font-family: Consolas; font-size: medium; "> Sandia National Laboratories</div></div></div></div></div></div></body></html>