<html>
<head>
<style>
.hmmessage P
{
margin:0px;
padding:0px
}
body.hmmessage
{
font-size: 10pt;
font-family:Verdana
}
</style>
</head>
<body class='hmmessage'>
I forgot to include an important parameter (communicator) in the psuedo interface below:<br><span class="EC_Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"><span style="font-size: 10pt; font-family: Verdana,sans-serif;">Create_memobj_collective(IN user_ptr, IN size, IN communicator, OUT mem_obj)<br><br>In addition to this Bill suggested Bind interface (something similar to MDBind in portals) that would help reduce latency for commonly re-used RMAs.<br><br><br></span></span><span style="font-size: 10pt; font-family: 'Verdana','sans-serif';">Vinod Tipparaju ^ http://ft.ornl.gov/~vinod ^ 1-865-241-1802</span><br><br><br><br><hr id="stopSpelling">From: wgropp@illinois.edu<br>To: mpi3-rma@lists.mpi-forum.org<br>Date: Thu, 3 Sep 2009 13:20:38 -0500<br>Subject: Re: [Mpi3-rma] MPI3 RMA Design Goals<br><br>Design goal one allows collective creation of objects; its there because many important algorithms don't have collective (over MPI_COMM_WORLD) allocation semantics, and a design that *requires* collective creation of memory objects will also limit the use of the interface.<div><br></div><div>Bill</div><div><br><div><div>On Sep 3, 2009, at 10:49 AM, Underwood, Keith D wrote:</div><br class="EC_Apple-interchange-newline"><blockquote><span class="EC_Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"><div lang="EN-US"><div class="EC_Section1"><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><span style="font-size: 11pt; font-family: Calibri,sans-serif; color: rgb(31, 73, 125);">My commentary was on the design goals… if we<span class="EC_Apple-converted-space"> </span><b><i>allow</i></b><span class="EC_Apple-converted-space"> </span>collective creation of memory objects, and design goal #1 simply says we don’t<span class="EC_Apple-converted-space"> </span><b><i>require</i></b><span class="EC_Apple-converted-space"> </span>it, that may be ok. Design goal #1 could be interpreted to mean that you wouldn’t have collective creation in the semantic at all. Do you really anticipate one data type for an object that is either collectively or non-collectively created? </span></div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><span style="font-size: 11pt; font-family: Calibri,sans-serif; color: rgb(31, 73, 125);"> </span></div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><span style="font-size: 11pt; font-family: Calibri,sans-serif; color: rgb(31, 73, 125);">I strongly disagree with your assertion that you can communicate with no cache misses for the non-collectively allocated memory object. In a non-collectively allocated case, you will have to keep an array of these on every process, right? i.e. one for every process you are communicating with? Randomly indexing that array is going to pound on your cache.</span></div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><span style="font-size: 11pt; font-family: Calibri,sans-serif; color: rgb(31, 73, 125);"> </span></div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><span style="font-size: 11pt; font-family: Calibri,sans-serif; color: rgb(31, 73, 125);">We need to make sure that we don’t ignore the overhead having multiple communicators and heterogeneity. Yes, I think there are ways around this, but we should at least consider what is practical and likely rather than just what is possible.</span></div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><span style="font-size: 11pt; font-family: Calibri,sans-serif; color: rgb(31, 73, 125);"> </span></div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><span style="font-size: 11pt; font-family: Calibri,sans-serif; color: rgb(31, 73, 125);">Keith</span></div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><span style="font-size: 11pt; font-family: Calibri,sans-serif; color: rgb(31, 73, 125);"> </span></div><div style="border-style: none none none solid; border-left: 1.5pt solid blue; padding: 0in 0in 0in 4pt;"><div><div style="border-style: solid none none; border-top: 1pt solid rgb(181, 196, 223); padding: 3pt 0in 0in;"><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><b><span style="font-size: 10pt; font-family: Tahoma,sans-serif;">From:</span></b><span style="font-size: 10pt; font-family: Tahoma,sans-serif;"><span class="EC_Apple-converted-space"> </span><a href="mailto:mpi3-rma-bounces@lists.mpi-forum.org" style="color: blue; text-decoration: underline;">mpi3-rma-bounces@lists.mpi-forum.org</a><span class="EC_Apple-converted-space"> </span>[<a href="mailto:mpi3-rma-bounces@lists.mpi-forum.org" style="color: blue; text-decoration: underline;">mailto:mpi3-rma-bounces@lists.mpi-forum.org</a>]<span class="EC_Apple-converted-space"> </span><b>On Behalf Of<span class="EC_Apple-converted-space"> </span></b>Vinod tipparaju<br><b>Sent:</b><span class="EC_Apple-converted-space"> </span>Tuesday, September 01, 2009 10:15 AM<br><b>To:</b><span class="EC_Apple-converted-space"> </span>MPI 3.0 Remote Memory Access working group<br><b>Subject:</b><span class="EC_Apple-converted-space"> </span>Re: [Mpi3-rma] MPI3 RMA Design Goals</span></div></div></div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"> </div><div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><span style="font-size: 10pt; font-family: Verdana,sans-serif;"> </span></div></div><div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><span style="font-size: 10pt; font-family: Verdana,sans-serif;">You are correct in trying to look at the best possible case and estimating cache-misses/performance-bottlenecks. However, personally don't see any difference between this and shmem. When you cannot really allocate symmetric memory underneath, the amount of bookkeeping is same in both cases. When there is no heterogeneity, the check for this can be disabled at MPI startup. When there is heterogeneity we cannot compare with shmem.</span></div></div><div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><span style="font-size: 10pt; font-family: Verdana,sans-serif;"> </span></div></div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><span style="font-size: 10pt; font-family: Verdana,sans-serif;">I cannot imagine not having symmetric/collective memory object creation to support these RMA interfaces, I think it is a must-have. Sorry I have only been saying we should have these interfaces but haven't given any example for this yet. Given how many times this same issue is coming up, I will do it now.</span></div><div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><span style="font-size: 10pt; font-family: Verdana,sans-serif;"> </span></div></div><div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><span style="font-size: 10pt; font-family: Verdana,sans-serif;">Consider the creation interfaces:</span></div></div><div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><span style="font-size: 10pt; font-family: Verdana,sans-serif;">Create_memobj(IN user_ptr, IN size, OUT mem_obj)</span></div></div><div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><span style="font-size: 10pt; font-family: Verdana,sans-serif;">Create_memobj_collective(user_ptr, size, OUT mem_obj)</span></div></div><div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><span style="font-size: 10pt; font-family: Verdana,sans-serif;">Assign_memobj(IN/OUT mem_obj, IN user_address, IN size) </span></div><div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><span style="font-size: 10pt; font-family: Verdana,sans-serif;"> </span></div></div><div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><span style="font-size: 10pt; font-family: Verdana,sans-serif;">There will be more details on how a mem object which is a result of create_memobj on process A will be exchanged with process B. When it is exchanged explicitly, the heterogeneity information can be created at process B. </span></div></div><div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><span style="font-size: 10pt; font-family: Verdana,sans-serif;"> </span></div></div><div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><span style="font-size: 10pt; font-family: Verdana,sans-serif;">Now take the example with symmetric object:</span></div></div><div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><span style="font-size: 10pt; font-family: Verdana,sans-serif;"> </span></div></div><div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><span style="font-size: 10pt; font-family: Verdana,sans-serif;">Process A </span></div></div><div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><span style="font-size: 10pt; font-family: Verdana,sans-serif;"> </span></div></div><div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><span style="font-size: 10pt; font-family: Verdana,sans-serif;">myptr = allocate(mysize);</span></div></div><div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><span style="font-size: 10pt; font-family: Verdana,sans-serif;">Create_memobj_collective(myptr,mysize, all_obj);</span></div></div><div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><span style="font-size: 10pt; font-family: Verdana,sans-serif;">Do all kinds of RMA_Xfers</span></div></div><div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><span style="font-size: 10pt; font-family: Verdana,sans-serif;"> </span></div></div><div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><span style="font-size: 10pt; font-family: Verdana,sans-serif;">and an example without symmetric object:</span></div></div><div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><span style="font-size: 10pt; font-family: Verdana,sans-serif;"> </span></div></div><div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><span style="font-size: 10pt; font-family: Verdana,sans-serif;">myptr = allocate(mysize);</span></div></div><div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><span style="font-size: 10pt; font-family: Verdana,sans-serif;">Create_memobj(myptr,mysize,my_obj);</span></div></div><div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><span style="font-size: 10pt; font-family: Verdana,sans-serif;"> ----exchange objects here----</span></div></div><div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><span style="font-size: 10pt; font-family: Verdana,sans-serif;">do all kinds of RAM_Xfers</span></div></div><div><div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><span style="font-size: 10pt; font-family: Verdana,sans-serif;"> </span></div></div><div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><span style="font-size: 10pt; font-family: Verdana,sans-serif;">In both cases, I can see being able to communicate without any cache misses for mem_obj.</span></div></div><div><p class="EC_MsoNormal" style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 12pt;"><span style="font-size: 10pt; font-family: Verdana,sans-serif;"><br>Vinod Tipparaju ^<span class="EC_Apple-converted-space"> </span><a href="http://ft.ornl.gov/%7Evinod" style="color: blue; text-decoration: underline;">http://ft.ornl.gov/~vinod</a><span class="EC_Apple-converted-space"> </span>^ 1-865-241-1802<br><br><br></span></p><div class="EC_MsoNormal" style="margin-right: 0in; margin-bottom: 0.0001pt; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; text-align: center;" align="center"><span style="font-size: 10pt; font-family: Verdana,sans-serif;"><hr id="EC_stopSpelling" align="center" size="2" width="100%"></span></div><p class="EC_MsoNormal" style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 12pt;"><span style="font-size: 10pt; font-family: Verdana,sans-serif;">From:<span class="EC_Apple-converted-space"> </span><a href="mailto:keith.d.underwood@intel.com" style="color: blue; text-decoration: underline;">keith.d.underwood@intel.com</a><br>To:<span class="EC_Apple-converted-space"> </span><a href="mailto:mpi3-rma@lists.mpi-forum.org" style="color: blue; text-decoration: underline;">mpi3-rma@lists.mpi-forum.org</a><br>Date: Tue, 1 Sep 2009 09:07:41 -0600<br>Subject: Re: [Mpi3-rma] MPI3 RMA Design Goals</span></p><div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><span style="font-size: 11pt; font-family: Calibri,sans-serif; color: rgb(31, 73, 125);">If we take the SINGLE_RMA_INTERFACE_DRAFT_PROPOSAL as an example, and combine it with the draft design goal #1:<span class="EC_Apple-converted-space"> </span></span><span style="font-size: 10pt; font-family: Verdana,sans-serif;">In order to support RMA to arbitrary locations, no constraints on memory, such as symmetric allocation or collective window creation, can be required</span></div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><span style="font-size: 10pt; font-family: Verdana,sans-serif;"> </span></div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><span style="font-size: 10pt; font-family: Verdana,sans-serif;">We get an interesting view on how difficult it can be to get “close to the metal”. So, for MPI_RMA_xfer, we have to assume that the user has some array of target_mem data items. That means the sequence of steps in user space is:</span></div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><span style="font-size: 10pt; font-family: Verdana,sans-serif;"> </span></div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><span style="font-size: 10pt; font-family: Verdana,sans-serif;">target_mem = ranks[dest];</span></div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><span style="font-size: 10pt; font-family: Verdana,sans-serif;">MPI_RMA_xfer(… target_mem, dest…);</span></div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><span style="font-size: 10pt; font-family: Verdana,sans-serif;"> </span></div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><span style="font-size: 10pt; font-family: Verdana,sans-serif;">If we assume that the message sizes are small and the destinations randomly selected and the machine is large… every access to ranks is a cache miss, and we cannot prevent that by providing fancy hardware. This actually leads me to believe that we may need to reconsider design goal #1, or at least clarify what it means in a way that makes the access more efficient.</span></div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><span style="font-size: 10pt; font-family: Verdana,sans-serif;"> </span></div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><span style="font-size: 10pt; font-family: Verdana,sans-serif;">MPI_RMA_xfer itself is no picnic either. If we take the draft design goal #5: The RMA model must support non-cache-coherent and heterogeneous environments, then MPI is required to maintain a data structure for every rank (ok, it has to do this anyway, but we are trying to get close to the metal) and do a lookup into that data structure with every MPI_RMA_xfer to find out if the target is heterogeneous relative to the target rank – another cache miss. Now, nominally, since this is inside MPI, a lower layer could absorb that check… or, a given MPI could refuse to support heterogeneity or… but, you get the idea. </span></div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><span style="font-size: 10pt; font-family: Verdana,sans-serif;"> </span></div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><span style="font-size: 10pt; font-family: Verdana,sans-serif;">So, we’ve got two cache line loads for every transfer. One in the application and one in the MPI library. One is impossible to move to the hardware and the other is simply very difficult to move. </span></div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><span style="font-size: 10pt; font-family: Verdana,sans-serif;"> </span></div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><span style="font-size: 10pt; font-family: Verdana,sans-serif;">For a contrast, look at SHMEM. Assume homogeneous, only one communicator context, and hardware mapping of ranks to physical locations. A shmem_put() of a short item could literally be turned into a few instructions and a processor store (on machines that supported such things). Personally, I think we will have done well if we can get to the point that a reasonable hardware implementation can get MPI RMA to within 2x of a reasonable SHMEM implementation. I think we have a long way to go to get there.</span></div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><span style="font-size: 10pt; font-family: Verdana,sans-serif;"> </span></div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><span style="font-size: 10pt; font-family: Verdana,sans-serif;">Keith</span></div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><span style="font-size: 11pt; font-family: Calibri,sans-serif; color: rgb(31, 73, 125);"> </span><span style="font-size: 10pt; font-family: Verdana,sans-serif;"></span></div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><span style="font-size: 11pt; font-family: Calibri,sans-serif; color: rgb(31, 73, 125);"> </span><span style="font-size: 10pt; font-family: Verdana,sans-serif;"></span></div><div style="border-style: none none none solid; border-left: 1.5pt solid blue; padding: 0in 0in 0in 4pt;"><div><div style="border-style: solid none none; border-top: 1pt solid rgb(181, 196, 223); padding: 3pt 0in 0in;"><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><b><span style="font-size: 10pt; font-family: Tahoma,sans-serif;">From:</span></b><span style="font-size: 10pt; font-family: Tahoma,sans-serif;"><span class="EC_Apple-converted-space"> </span><a href="mailto:mpi3-rma-bounces@lists.mpi-forum.org" style="color: blue; text-decoration: underline;">mpi3-rma-bounces@lists.mpi-forum.org</a><span class="EC_Apple-converted-space"> </span>[<a href="mailto:mpi3-rma-bounces@lists.mpi-forum.org" style="color: blue; text-decoration: underline;">mailto:mpi3-rma-bounces@lists.mpi-forum.org</a>]<span class="EC_Apple-converted-space"> </span><b>On Behalf Of<span class="EC_Apple-converted-space"> </span></b>Vinod tipparaju<br><b>Sent:</b><span class="EC_Apple-converted-space"> </span>Tuesday, September 01, 2009 5:23 AM<br><b>To:</b><span class="EC_Apple-converted-space"> </span>MPI 3.0 Remote Memory Access working group<br><b>Subject:</b><span class="EC_Apple-converted-space"> </span>Re: [Mpi3-rma] MPI3 RMA Design Goals</span><span style="font-size: 10pt; font-family: Verdana,sans-serif;"></span></div></div></div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><span style="font-size: 10pt; font-family: Verdana,sans-serif;"> </span></div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><span style="font-size: 10pt; font-family: Verdana,sans-serif;">Good points! RMA interfaces should do nothing to prevent utilizing a high message rate (or low overhead communication) that the underlying hardware may offer. To ensure this happens, there should always be a unrestricted path (lets call it this for now, people have called it a "thin layer", "direct access") to the network. </span></div><div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><span style="font-size: 10pt; font-family: Verdana,sans-serif;"> </span></div></div><div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><span style="font-size: 10pt; font-family: Verdana,sans-serif;">This means, despite the fact the the RMA interface has features that abstract out complexity by providing useful characteristics such as ordering and atomicity, it (the RMA interface) should always have this unrestricted, close to the heart of the hardware, path. To achieve this, the unrestricted path should not require any book keeping (from implementation perspective) in relation to the feature-rich path or vice-versa. </span></div></div><div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><span style="font-size: 10pt; font-family: Verdana,sans-serif;"> </span></div></div><div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><span style="font-size: 10pt; font-family: Verdana,sans-serif;">I believe this is what we have demonstrated with the example interfaces hence the null set isn't the case here :-). I will distribute an example implementation very soon so people can get a feel.</span></div></div><div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><span style="font-size: 10pt; font-family: Verdana,sans-serif;"> </span></div></div><div><div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><span style="font-size: 10pt; font-family: Verdana,sans-serif;">---</span></div></div><div><div style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman',serif; margin-bottom: 0.0001pt;"><span style="font-size: 10pt; font-family: Verdana,sans-serif;">Vinod Tipparaju ^<span class="EC_Apple-converted-space"> </span><a href="http://ft.ornl.gov/%7Evinod" style="color: blue; text-decoration: underline;">http://ft.ornl.gov/~vinod</a><span class="EC_Apple-converted-space"> </span>^ 1-865-241-1802<br><br><br><br>> From:<span class="EC_Apple-converted-space"> </span><a href="mailto:keith.d.underwood@intel.com" style="color: blue; text-decoration: underline;">keith.d.underwood@intel.com</a><br>> To:<span class="EC_Apple-converted-space"> </span><a href="mailto:mpi3-rma@lists.mpi-forum.org" style="color: blue; text-decoration: underline;">mpi3-rma@lists.mpi-forum.org</a><br>> Date: Mon, 31 Aug 2009 16:17:28 -0600<br>> Subject: Re: [Mpi3-rma] MPI3 RMA Design Goals<br>><span class="EC_Apple-converted-space"> </span><br>> There has been stunning silence since this email, so I will go ahead and toss out a thought...<br>><span class="EC_Apple-converted-space"> </span><br>> In the draft design goals, I don't see two issues that I see as key. The first is "support for high message rate/low overhead communications to random targets". As best I can tell, this is one of the key places were the existing one-sided operations are perceived as falling down for existing customers of SHMEM/PGAS. The second is "elimination of the access epoch requirement". This one may be, um, more controversial, but I believe it is part and parcel with the first one. That is, the first one is not that valuable if the programming model requires an excessive amount of access epoch opens and closes just to force the global visibility of the operations. Unfortunately, the intersection of this solution space with the solution space for the current draft design goal #5 (support non-cache-coherent and heterogeneous environments) may be the null set... I will hold out hope that this isn't the case ;-)<br>><span class="EC_Apple-converted-space"> </span><br>> Keith<span class="EC_Apple-converted-space"> </span><br>><span class="EC_Apple-converted-space"> </span><br>> > -----Original Message-----<br>> > From: <a href="mailto:mpi3-rma-bounces@lists.mpi-forum.org">mpi3-rma-bounces@lists.mpi-forum.org</a> [<a href="mailto:mpi3-rma-" style="color: blue; text-decoration: underline;">mailto:mpi3-rma-</a><br>> ><span class="EC_Apple-converted-space"> </span><a href="mailto:bounces@lists.mpi-forum.org" style="color: blue; text-decoration: underline;">bounces@lists.mpi-forum.org</a>] On Behalf Of William Gropp<br>> > Sent: Wednesday, August 05, 2009 12:37 PM<br>> > To:<span class="EC_Apple-converted-space"> </span><a href="mailto:mpi3-rma@lists.mpi-forum.org" style="color: blue; text-decoration: underline;">mpi3-rma@lists.mpi-forum.org</a><br>> > Subject: [Mpi3-rma] MPI3 RMA Design Goals<br>> ><span class="EC_Apple-converted-space"> </span><br>> > I've added versions of the RMA design goals that we discussed at the<br>> > Forum meeting last week to the wiki page for our group (<br>> ><span class="EC_Apple-converted-space"> </span><a href="https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/RmaWikiPage" style="color: blue; text-decoration: underline;">https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/RmaWikiPage</a><br>> > ). This is a draft; lets discuss these. Also, feel free to add to<br>> > the discussion, particularly in the background section.<br>> ><span class="EC_Apple-converted-space"> </span><br>> > Bill<br>> ><span class="EC_Apple-converted-space"> </span><br>> > William Gropp<br>> > Deputy Director for Research<br>> > Institute for Advanced Computing Applications and Technologies<br>> > Paul and Cynthia Saylor Professor of Computer Science<br>> > University of Illinois Urbana-Champaign<br>> ><span class="EC_Apple-converted-space"> </span><br>> ><span class="EC_Apple-converted-space"> </span><br>> ><span class="EC_Apple-converted-space"> </span><br>> ><span class="EC_Apple-converted-space"> </span><br>> > _______________________________________________<br>> > mpi3-rma mailing list<br>> ><span class="EC_Apple-converted-space"> </span><a href="mailto:mpi3-rma@lists.mpi-forum.org" style="color: blue; text-decoration: underline;">mpi3-rma@lists.mpi-forum.org</a><br>> ><span class="EC_Apple-converted-space"> </span><a href="http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma" style="color: blue; text-decoration: underline;">http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma</a><br>><span class="EC_Apple-converted-space"> </span><br>> _______________________________________________<br>> mpi3-rma mailing list<br>><span class="EC_Apple-converted-space"> </span><a href="mailto:mpi3-rma@lists.mpi-forum.org" style="color: blue; text-decoration: underline;">mpi3-rma@lists.mpi-forum.org</a><br>><span class="EC_Apple-converted-space"> </span><a href="http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma" style="color: blue; text-decoration: underline;">http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma</a></span></div></div></div></div></div></div></div></div></div><span><ATT00001.txt></span></div></div></span></blockquote></div><br><div> <span class="EC_Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"><div style="word-wrap: break-word;"><span class="EC_Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px;"><div style="word-wrap: break-word;"><div>William Gropp</div><div>Deputy Director for Research</div><div>Institute for Advanced Computing Applications and Technologies</div><div>Paul and Cynthia Saylor Professor of Computer Science</div><div>University of Illinois Urbana-Champaign</div><div><br class="EC_khtml-block-placeholder"></div></div><br class="EC_Apple-interchange-newline"></span></div></span><br class="EC_Apple-interchange-newline"> </div><br></div></body>
</html>