<html><body>
<p><font size="2" face="sans-serif">Yes, the idea about the for loop at the bottom was that a parallel copy is faster. Yes, it eliminates the need for any additional synchronization. I guess it's about lines of code, I guess it could be "memcpy(...); #pragma omp barrier" but it seems like this way provokes more thinking about parallelism.</font><br>
<br>
<font size="2" face="sans-serif">Re: "i" being private: in the "#pragma omp for" construct the loop-control variable is implicitly made private by OpenMP. </font><br>
<br>
<font size="2" face="sans-serif">Re: "oldval = newval" having races: there is only one value for "newval", so each thread will redundantly copy the same value over the first thread's copy. the value is safe since no thread can update newval until all threads have completed the for loop. No thread can test the value(s) until all threads have completed not only the MPI_Team_leave but also the "for" loop that copies the array. Making them private complicates the example by requiring more synchronization in order to share/compute the single value between all threads.</font><br>
<br>
<font size="2" face="sans-serif">Re: remove INFO "nobreak": my take from previous readings is that this was needed in the example. I am reluctant to remove it and face another set-back.</font><br>
<br>
<br>
<br>
<font size="2" face="sans-serif">Q: What is "DIM" in the work for-loop? should that be "count" (COUNT) - the size of the sendbuf array? I've changed it to COUNT and ran a compile. Aside from no declaration for "i" it was fine. I added "i" inside the while loop, which eliminates concerns about a problem if shared.</font><br>
<br>
<font size="2" face="sans-serif">Here's what I currently have for the example:</font><br>
<br>
<font size="2" face="sans-serif">MPI_Team team;</font><br>
<font size="2" face="sans-serif">MPI_Info info;</font><br>
<font size="2" face="sans-serif">double oldval = 0.0, newval = 9.9e99; </font><br>
<font size="2" face="sans-serif">double tolerance = 1.0e-6; </font><br>
<font size="2" face="sans-serif">double sendbuf[COUNT] = { 0.0 };</font><br>
<font size="2" face="sans-serif">double recvbuf[COUNT] = { 0.0 };</font><br>
<font size="2" face="sans-serif">MPI_Info_create(&info);</font><br>
<font size="2" face="sans-serif">MPI_Info_set(info, "nobreak", "true");</font><br>
<font size="2" face="sans-serif">MPI_Team_create(omp_get_thread_limit(), info, &team); </font><br>
<font size="2" face="sans-serif">MPI_Info_free(&info);</font><br>
<font size="2" face="sans-serif">#pragma omp parallel num_threads(omp_get_thread_limit())</font><br>
<font size="2" face="sans-serif">{</font><br>
<font size="2" face="sans-serif"> while (abs(newval - oldval) > tolerance) {</font><br>
<font size="2" face="sans-serif"> double myval = 0.0;</font><br>
<font size="2" face="sans-serif"> int i;</font><br>
<font size="2" face="sans-serif"> oldval = newval; </font><br>
<br>
<font size="2" face="sans-serif"># pragma omp for </font><br>
<font size="2" face="sans-serif"> for (i = 0; i < COUNT; i++) {</font><br>
<font size="2" face="sans-serif"> myval += do_work(i, sendbuf);</font><br>
<font size="2" face="sans-serif"> }</font><br>
<br>
<font size="2" face="sans-serif"># pragma omp critical</font><br>
<font size="2" face="sans-serif"> {</font><br>
<font size="2" face="sans-serif"> newval += myval;</font><br>
<font size="2" face="sans-serif"> }</font><br>
<br>
<font size="2" face="sans-serif"> MPI_Team_join(omp_get_num_threads(), team);</font><br>
<font size="2" face="sans-serif"> /* this barrier is not required but helps ensure</font><br>
<font size="2" face="sans-serif"> * all threads arrive before the MPI_Allreduce begins */</font><br>
<font size="2" face="sans-serif"># pragma omp barrier </font><br>
<font size="2" face="sans-serif"># pragma omp master</font><br>
<font size="2" face="sans-serif"> {</font><br>
<font size="2" face="sans-serif"> MPI_Allreduce(sendbuf, recvbuf, COUNT, MPI_DOUBLE,</font><br>
<font size="2" face="sans-serif"> MPI_SUM, MPI_COMM_WORLD);</font><br>
<font size="2" face="sans-serif"> }</font><br>
<font size="2" face="sans-serif"> /* The remaining threads directly go to MPI_Team_leave */</font><br>
<font size="2" face="sans-serif"> MPI_Team_leave(team);</font><br>
<br>
<font size="2" face="sans-serif"># pragma omp for </font><br>
<font size="2" face="sans-serif"> for (i = 0; i < COUNT; i++) {</font><br>
<font size="2" face="sans-serif"> sendbuf[i] = recvbuf[i];</font><br>
<font size="2" face="sans-serif"> }</font><br>
<font size="2" face="sans-serif"> }</font><br>
<font size="2" face="sans-serif">}</font><br>
<font size="2" face="sans-serif">MPI_Team_free(&team);</font><br>
<br>
<font size="2" face="sans-serif">_______________________________________________<br>
Douglas Miller BlueGene Messaging Development<br>
IBM Corp., Rochester, MN USA Bldg 030-2 A401<br>
dougmill@us.ibm.com Douglas Miller/Rochester/IBM</font><br>
<br>
<img width="16" height="16" src="cid:1__=09BBF306DFFC0F808f9e8a93df938@us.ibm.com" border="0" alt="Inactive hide details for Jim Dinan ---01/30/2012 01:58:19 PM---Jim Dinan <dinan@mcs.anl.gov>"><font size="2" color="#424282" face="sans-serif">Jim Dinan ---01/30/2012 01:58:19 PM---Jim Dinan <dinan@mcs.anl.gov></font><br>
<br>
<table width="100%" border="0" cellspacing="0" cellpadding="0">
<tr valign="top"><td style="background-image:url(cid:2__=09BBF306DFFC0F808f9e8a93df938@us.ibm.com); background-repeat: no-repeat; " width="40%">
<ul style="padding-left: 72pt"><font size="1" face="sans-serif"><b>Jim Dinan <dinan@mcs.anl.gov></b></font><font size="1" face="sans-serif"> </font><br>
<font size="1" face="sans-serif">Sent by: mpi3-hybridpm-bounces@lists.mpi-forum.org</font>
<p><font size="1" face="sans-serif">01/30/2012 01:56 PM</font>
<table border="1">
<tr valign="top"><td width="168" bgcolor="#FFFFFF"><div align="center"><font size="1" face="sans-serif">Please respond to<br>
mpi3-hybridpm@lists.mpi-forum.org</font></div></td></tr>
</table>
</ul>
</td><td width="60%">
<table width="100%" border="0" cellspacing="0" cellpadding="0">
<tr valign="top"><td width="1%"><img width="58" height="1" src="cid:3__=09BBF306DFFC0F808f9e8a93df938@us.ibm.com" border="0" alt=""><br>
<div align="right"><font size="1" face="sans-serif">To</font></div></td><td width="100%"><img width="1" height="1" src="cid:3__=09BBF306DFFC0F808f9e8a93df938@us.ibm.com" border="0" alt=""><br>
<ul style="padding-left: 7pt"><font size="1" face="sans-serif">mpi3-hybridpm@lists.mpi-forum.org, </font></ul>
</td></tr>
<tr valign="top"><td width="1%"><img width="58" height="1" src="cid:3__=09BBF306DFFC0F808f9e8a93df938@us.ibm.com" border="0" alt=""><br>
<div align="right"><font size="1" face="sans-serif">cc</font></div></td><td width="100%"><img width="1" height="1" src="cid:3__=09BBF306DFFC0F808f9e8a93df938@us.ibm.com" border="0" alt=""><br>
</td></tr>
<tr valign="top"><td width="1%"><img width="58" height="1" src="cid:3__=09BBF306DFFC0F808f9e8a93df938@us.ibm.com" border="0" alt=""><br>
<div align="right"><font size="1" face="sans-serif">Subject</font></div></td><td width="100%"><img width="1" height="1" src="cid:3__=09BBF306DFFC0F808f9e8a93df938@us.ibm.com" border="0" alt=""><br>
<ul style="padding-left: 7pt"><font size="1" face="sans-serif">Re: [Mpi3-hybridpm] Ticket 217 proposal document updated</font></ul>
</td></tr>
</table>
<table border="0" cellspacing="0" cellpadding="0">
<tr valign="top"><td width="58"><img width="1" height="1" src="cid:3__=09BBF306DFFC0F808f9e8a93df938@us.ibm.com" border="0" alt=""></td><td width="336"><img width="1" height="1" src="cid:3__=09BBF306DFFC0F808f9e8a93df938@us.ibm.com" border="0" alt=""></td></tr>
</table>
</td></tr>
</table>
<br>
<tt><font size="2">Hi Doug,<br>
<br>
I think we can probably get rid of the barrier, it seems to suggest a <br>
stronger requirement (sync-first) for helper threads. MPI could <br>
implement this behavior internally in leave if it's required by the <br>
implementation.<br>
<br>
This copy code also seems a bit verbose to me:<br>
<br>
# pragma omp for<br>
for (i = 0; i < count; i++) {<br>
sendbuf[i] = recvbuf[i];<br>
}<br>
<br>
Is the idea that this would be faster than a serial memcpy? We could <br>
simplify the example by just calling memcpy. I guess we would need to <br>
add a barrier, though.<br>
<br>
"i" needs to be made thread private (declare in parallel region?).<br>
<br>
"double myval = 0.0" should be moved to the top of the block for C89 <br>
compliance.<br>
<br>
oldval and newval are shared, "oldval = newval" will have <br>
race/consistency problems. Should these also be thread private?<br>
<br>
"nobreak" adds a lot of lines of code and not a lot of information to <br>
the example. I think it would be better to point out in a caption that <br>
you could pass the "nobreak" info key and shorten the example by leaving <br>
out all the Info wrangling code.<br>
<br>
Suggest s/count/COUNT/ so that it looks like a constant in the array <br>
declarations.<br>
<br>
Have you tried compiling/running this with no-op/barrier implementations <br>
of the helper threads functions?<br>
<br>
~Jim.<br>
<br>
On 1/30/12 1:32 PM, Douglas Miller wrote:<br>
> The "#pragma omp for" does a barrier at the end of the block (since we<br>
> did not specify "nowait"), so I don't think it is required.<br>
><br>
><br>
> _______________________________________________<br>
> Douglas Miller BlueGene Messaging Development<br>
> IBM Corp., Rochester, MN USA Bldg 030-2 A401<br>
> dougmill@us.ibm.com Douglas Miller/Rochester/IBM<br>
><br>
> Inactive hide details for Jim Dinan ---01/30/2012 12:44:47 PM---Jim<br>
> Dinan <dinan@mcs.anl.gov>Jim Dinan ---01/30/2012 12:44:47 PM---Jim Dinan<br>
> <dinan@mcs.anl.gov><br>
><br>
> *Jim Dinan <dinan@mcs.anl.gov>*<br>
> Sent by: mpi3-hybridpm-bounces@lists.mpi-forum.org<br>
><br>
> 01/30/2012 12:38 PM<br>
> Please respond to<br>
> mpi3-hybridpm@lists.mpi-forum.org<br>
><br>
> <br>
><br>
> To<br>
> <br>
><br>
> mpi3-hybridpm@lists.mpi-forum.org,<br>
><br>
><br>
> cc<br>
> <br>
><br>
> Subject<br>
> <br>
><br>
> Re: [Mpi3-hybridpm] Ticket 217 proposal document updated<br>
><br>
> <br>
><br>
><br>
> Hi Doug,<br>
><br>
> In Example 12.3, the "#omp barrier" should be required to ensure that<br>
> all threads have finished updating sendbuf and reading recvbuf before<br>
> the Allreduce.<br>
><br>
> ~Jim.<br>
><br>
> On 1/30/12 8:27 AM, Douglas Miller wrote:<br>
> > I've updated the PDF in Ticket 217 based on latest comments. Please<br>
> > review the text and send me comments. This is an attempt to make the<br>
> > concept easier to follow by describing in terms of a "team epoch".<br>
> ><br>
> > thanks,<br>
> > _______________________________________________<br>
> > Douglas Miller BlueGene Messaging Development<br>
> > IBM Corp., Rochester, MN USA Bldg 030-2 A401<br>
> > dougmill@us.ibm.com Douglas Miller/Rochester/IBM<br>
> ><br>
> ><br>
> > _______________________________________________<br>
> > Mpi3-hybridpm mailing list<br>
> > Mpi3-hybridpm@lists.mpi-forum.org<br>
> > </font></tt><tt><font size="2"><a href="http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm">http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm</a></font></tt><tt><font size="2"><br>
> _______________________________________________<br>
> Mpi3-hybridpm mailing list<br>
> Mpi3-hybridpm@lists.mpi-forum.org<br>
> </font></tt><tt><font size="2"><a href="http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm">http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm</a></font></tt><tt><font size="2"><br>
><br>
><br>
><br>
><br>
> _______________________________________________<br>
> Mpi3-hybridpm mailing list<br>
> Mpi3-hybridpm@lists.mpi-forum.org<br>
> </font></tt><tt><font size="2"><a href="http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm">http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm</a></font></tt><tt><font size="2"><br>
_______________________________________________<br>
Mpi3-hybridpm mailing list<br>
Mpi3-hybridpm@lists.mpi-forum.org<br>
</font></tt><tt><font size="2"><a href="http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm">http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-hybridpm</a></font></tt><tt><font size="2"><br>
<br>
</font></tt><br>
</body></html>