<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.apple-style-span
{mso-style-name:apple-style-span;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">This is interesting. It might be useful for implementers if the app could inform the MPI library that in its usage model, per-communicator queues might lead
to a performance benefit. Such as in the case of many threads (among others).<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Info key? Assert?<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Sayantan<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> mpi3-hybridpm-bounces@lists.mpi-forum.org [mailto:mpi3-hybridpm-bounces@lists.mpi-forum.org]
<b>On Behalf Of </b>William Gropp<br>
<b>Sent:</b> Monday, March 25, 2013 2:24 PM<br>
<b>To:</b> mpi3-hybridpm@lists.mpi-forum.org<br>
<b>Subject:</b> Re: [Mpi3-hybridpm] [EXTERNAL] Re: Threading homeworking / next telecon<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">An implementation is free to use separate queues for each communicator; some of us have discussed this in the past, in part to permit use of lock-free structures for the queue updates, particularly as this is the only place there are no
wild cards, ever. I believe that this is within the existing semantics. It even has benefits for single threaded execution, since the communicator matching is done once, rather than in every query on the queue.<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">In terms of progress, the standard is deliberately vague on the details, and thus I don't believe we have the requirement that you quote. And some of the other interpretations of progress would not be helped by any thread-safety restriction.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Bill<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif";color:black">William Gropp<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif";color:black">Director, Parallel Computing Institute<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif";color:black">Deputy Director for Research<o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif";color:black">Institute for Advanced Computing Applications and Technologies<o:p></o:p></span></p>
</div>
</div>
</div>
<p class="MsoNormal"><span class="apple-style-span"><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif";color:black">Thomas M. Siebel Chair in Computer Science</span></span><span class="apple-style-span"><span style="font-size:13.5pt"><o:p></o:p></span></span></p>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-size:9.0pt;font-family:"Helvetica","sans-serif";color:black">University of Illinois Urbana-Champaign</span><span style="font-size:9.0pt"><o:p></o:p></span></p>
</div>
</div>
<div>
<p class="MsoNormal"><span style="font-size:13.5pt;font-family:"Helvetica","sans-serif";color:black"><o:p> </o:p></span></p>
</div>
</div>
<p class="MsoNormal"><span style="font-size:13.5pt;font-family:"Helvetica","sans-serif";color:black"><br>
<br>
</span><o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Mar 25, 2013, at 4:15 PM, Jeff Hammond wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal"><br>
<br>
<o:p></o:p></p>
<div>
<p class="MsoNormal">On Mon, Mar 25, 2013 at 3:17 PM, William Gropp <<a href="mailto:wgropp@illinois.edu">wgropp@illinois.edu</a>> wrote:<br>
<br>
<o:p></o:p></p>
<p class="MsoNormal">I was only addressing the issue of calling the thread level routines before<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">knowing what thread level you had.<o:p></o:p></p>
</blockquote>
<p class="MsoNormal"><br>
Okay, sorry, I cannot tell which tickets people are referring to since<br>
I have a bunch of different ones right now.<br>
<br>
<br>
<o:p></o:p></p>
<p class="MsoNormal">I'm not sure what you are looking for. In the case of MPI_THREAD_MULTIPLE,<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">an implementation can provide significant concurrency today without any<o:p></o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">change in the MPI standard - that's a major reason for that table (more to<o:p></o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">the point - this table is meant as a guide for not using locks). Can you<o:p></o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">give me an example of something that the current MPI semantics prohibits<o:p></o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">that you'd like to achieve with MPI_THREAD_PER_OBJECT?<o:p></o:p></p>
</blockquote>
<p class="MsoNormal"><br>
It is my understanding of the progress requirements that any call to<br>
MPI must make progress on all MPI operations. This means that two<br>
threads calling e.g. MPI_Recv must walk all of the message queues. If<br>
a thread needs to modify any queue because it matches, then this must<br>
be done in a thread-safe way, which presumably requires something<br>
resembling mutual exclusion or transactions. If a call to MPI_Recv<br>
only had to make progress on its own communicator, then two threads<br>
calling MPI_Recv on two different communicators would (1) only have to<br>
walk the message queue associated with that communicator and (2)<br>
nothing resembling mutual exclusion is required for the thread to<br>
update the message queue in the event that matching occurs.<br>
<br>
Forgive me if I've got some of the details wrong. If I've got all of<br>
the details and the big picture wrong, then I'll think about it more.<br>
<br>
Jeff<br>
<br>
<br>
<o:p></o:p></p>
<p class="MsoNormal">On Mar 25, 2013, at 2:53 PM, Jeff Hammond wrote:<o:p></o:p></p>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><o:p> </o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">That doesn't do much for me in terms of enabling greater concurrency<o:p></o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">in performance-critical operations.<o:p></o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><o:p> </o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">I'd like to propose that we try to make all of "Access Only", "Update<o:p></o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">RefCount", "Read of List" and "None" thread safe in all cases. All of<o:p></o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">these are read-only except for "Update RefCount", but this can be done<o:p></o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">with atomics. I am assuming that concurrent reads are only permitted<o:p></o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">to happen after the writing calls on the object have completed. This<o:p></o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">is the essence of MPI_THREAD_PER_OBJECT.<o:p></o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><o:p> </o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">Jeff<o:p></o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><o:p> </o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><o:p> </o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><o:p> </o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">_______________________________________________<o:p></o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal">Mpi3-hybridpm mailing list<o:p></o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><a href="mailto:Mpi3-hybridpm@lists.mpi-forum.org">Mpi3-hybridpm@lists.mpi-forum.org</a><o:p></o:p></p>
</blockquote>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<p class="MsoNormal"><a href="http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm">http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm</a><o:p></o:p></p>
</blockquote>
<p class="MsoNormal"><br>
<br>
<br>
-- <br>
Jeff Hammond<br>
Argonne Leadership Computing Facility<br>
University of Chicago Computation Institute<br>
<a href="mailto:jhammond@alcf.anl.gov">jhammond@alcf.anl.gov</a> / (630) 252-5381<br>
<a href="http://www.linkedin.com/in/jeffhammond">http://www.linkedin.com/in/jeffhammond</a><br>
<a href="https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond">https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond</a><br>
_______________________________________________<br>
Mpi3-hybridpm mailing list<br>
<a href="mailto:Mpi3-hybridpm@lists.mpi-forum.org">Mpi3-hybridpm@lists.mpi-forum.org</a><br>
<a href="http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm">http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm</a><o:p></o:p></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</body>
</html>