<html>
<head>
<style>
.hmmessage P
{
margin:0px;
padding:0px
}
body.hmmessage
{
font-size: 10pt;
font-family:Verdana
}
</style>
</head>
<body class='hmmessage'>
<div><br></div>I am OK with having two interfaces, one which is WRT comm_world and only supports MPI_BYTE and the other a more general one.<br><br><font class="Apple-style-span" face="Verdana, sans-serif"><br></font><div><font class="Apple-style-span" face="Verdana, sans-serif">Thanks,</font></div><div><font class="Apple-style-span" face="Verdana, sans-serif">Vinod.<br></font><br><br><br><hr id="stopSpelling">From: keith.d.underwood@intel.com<br>To: mpi3-rma@lists.mpi-forum.org<br>Date: Wed, 16 Sep 2009 11:08:51 -0600<br>Subject: Re: [Mpi3-rma] non-contiguous support in RMA & one-sided pack/unpack (?)<br><br>
<style>
.ExternalClass p.EC_MsoNormal, .ExternalClass li.EC_MsoNormal, .ExternalClass div.EC_MsoNormal
{margin-bottom:.0001pt;font-size:12.0pt;font-family:'Times New Roman','serif';}
.ExternalClass a:link, .ExternalClass span.EC_MsoHyperlink
{color:blue;text-decoration:underline;}
.ExternalClass a:visited, .ExternalClass span.EC_MsoHyperlinkFollowed
{color:purple;text-decoration:underline;}
.ExternalClass p
{margin-right:0in;margin-left:0in;font-size:12.0pt;font-family:'Times New Roman','serif';}
.ExternalClass span.EC_EmailStyle18
{font-family:'Calibri','sans-serif';color:#1F497D;}
.ExternalClass .EC_MsoChpDefault
{font-size:10.0pt;}
@page Section1
{size:8.5in 11.0in;}
.ExternalClass div.EC_Section1
{page:Section1;}
</style>
<div class="EC_Section1">
<p class="EC_MsoNormal"><span style="font-size:11.0pt;font-family:'Calibri','sans-serif';color:#1F497D">But, going back to Bill’s point: performance across
a range of platforms is key. While you can’t have a function for
every usage (well, you can, but it would get cumbersome at some point), it may
be important to have a few levels of specialization in the API. E.g. you
could have two variants:</span></p>
<p class="EC_MsoNormal"><span style="font-size:11.0pt;font-family:'Calibri','sans-serif';color:#1F497D"> </span></p>
<p class="EC_MsoNormal"><span style="font-size:11.0pt;font-family:'Calibri','sans-serif';color:#1F497D">MPI_Fast_RMA_xfer(): no data types, no communicators, etc.</span></p>
<p class="EC_MsoNormal"><span style="font-size:11.0pt;font-family:'Calibri','sans-serif';color:#1F497D">MPI_Slow_RMA_xfer(): include the kitchen sink.</span></p>
<p class="EC_MsoNormal"><span style="font-size:11.0pt;font-family:'Calibri','sans-serif';color:#1F497D"> </span></p>
<p class="EC_MsoNormal"><span style="font-size:11.0pt;font-family:'Calibri','sans-serif';color:#1F497D">Yes, the naming is a little tongue in cheek ;-)</span></p>
<p class="EC_MsoNormal"><span style="font-size:11.0pt;font-family:'Calibri','sans-serif';color:#1F497D"> </span></p>
<p class="EC_MsoNormal"><span style="font-size:11.0pt;font-family:'Calibri','sans-serif';color:#1F497D">Keith</span></p>
<p class="EC_MsoNormal"><span style="font-size:11.0pt;font-family:'Calibri','sans-serif';color:#1F497D"> </span></p>
<div style="border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt">
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="EC_MsoNormal"><b><span style="font-size:10.0pt;font-family:'Tahoma','sans-serif'">From:</span></b><span style="font-size:10.0pt;font-family:'Tahoma','sans-serif'">
mpi3-rma-bounces@lists.mpi-forum.org
[mailto:mpi3-rma-bounces@lists.mpi-forum.org] <b>On Behalf Of </b>Vinod
tipparaju<br>
<b>Sent:</b> Wednesday, September 16, 2009 10:58 AM<br>
<b>To:</b> MPI 3.0 Remote Memory Access working group<br>
<b>Subject:</b> Re: [Mpi3-rma] non-contiguous support in RMA & one-sided
pack/unpack (?)</span></p>
</div>
</div>
<p class="EC_MsoNormal"> </p>
<p class="EC_MsoNormal"><span style="font-size:10.0pt;font-family:'Verdana','sans-serif'">"else"
condition is non-trivial when<br>
> one considers the latency of the BlueGene/P interconnect, the memory<br>
> latency, the clock rate and the absence of dynamic communication<br>
> threads. Forgive me for being specific to one machine, but it is the<br>
> benchmark data I have obtained that inspires me to comment on these<br>
> issues</span></p>
<div>
<p class="EC_MsoNormal"><span style="font-size:10.0pt;font-family:'Verdana','sans-serif'"> </span></p>
</div>
<div>
<p class="EC_MsoNormal"><span style="font-size:10.0pt;font-family:'Verdana','sans-serif'">You
have a valid point, but, as you already note, BG is an exception. It is a
"slowed down" exception. In most cases, latency associated with
branching will have to continue to be insignificant (by 2 orders of magnitude)
WRT to network latency. Hence what needs to be done for non-contiguous data WRT
and in the network is more important than a mere branch. What the network does
or doesn't do to support non-contiguous data transfers is critical. </span></p>
</div>
<div>
<p class="EC_MsoNormal"><span style="font-size:10.0pt;font-family:'Verdana','sans-serif'"> </span></p>
</div>
<div>
<p class="EC_MsoNormal" style="margin-bottom:12.0pt"><span style="font-size:10.0pt;font-family:'Verdana','sans-serif'">The input/guidance forum gives hence is
more important when taken during the network design -- many things may be fixed
in an implementations but one cannot fix what a network lack here.</span></p>
</div>
<div>
<p class="EC_MsoNormal"><span style="font-size:10.0pt;font-family:'Verdana','sans-serif'">Basically,
you believe having two calls instead of one will help because you believe
you can help the implementation by giving hints on what you need from this
particular RMA transfer. I added attributes for a similar reason. Usages are
many, we cannot have an interface for each usage. We can however use attributes
to give hints/orders that may help implementations utilize networks better or
know when to give up. We can also support features such as Binding (discussed
in the last meeting) to help reduce latency. </span></p>
</div>
<div>
<p class="EC_MsoNormal"><span style="font-size:10.0pt;font-family:'Verdana','sans-serif'"> </span></p>
</div>
<div>
<p class="EC_MsoNormal"><span style="font-size:10.0pt;font-family:'Verdana','sans-serif'">Thanks,</span></p>
</div>
<div>
<p class="EC_MsoNormal"><span style="font-size:10.0pt;font-family:'Verdana','sans-serif'">Vinod.</span></p>
</div>
<div>
<p class="EC_MsoNormal"><span style="font-size:10.0pt;font-family:'Verdana','sans-serif'"><br>
<br>
<br>
> Date: Wed, 16 Sep 2009 09:28:31 -0500<br>
> From: jeff.science@gmail.com<br>
> To: mpi3-rma@lists.mpi-forum.org<br>
> Subject: Re: [Mpi3-rma] non-contiguous support in RMA & one-sided
pack/unpack (?)<br>
> <br>
> > 1) MPI is implemented as a library. There is no connection between
the<br>
> > analysis the compiler can do and the MPI implementation so what can
be<br>
> > "known" by the compiler cannot be communicated to the
implementation.<br>
> <br>
> Yes, I am well aware of this. What I mean is that I, the programmer,<br>
> can choose to use in my source code the function or function arguments<br>
> which invoke the fastest correct call, thus preventing MPI from<br>
> proceeding through all the logic necessary to decipher the most<br>
> general case.<br>
> <br>
> For example, if "xfer" contained a flag just for<br>
> contiguous/non-contiguous, I could do the following in my code:<br>
> <br>
> ======================================<br>
> /* get row */<br>
> ( invoke xfer_get with contig flag set and using the simple datatype<br>
> for a contiguous vector of floats)<br>
> <br>
> /* get column */<br>
> ( invoke xfer_get with noncontig flag set and using derived datatype<br>
> for the column vector of a matrix of floats)<br>
> ======================================<br>
> <br>
> knowing that MPI can implement "xfer" like this:<br>
> <br>
> ======================================<br>
> xfer_get ( ... ){<br>
> if (contig){<br>
> low_level_comm_api_get(...)<br>
> } else {<br>
> process_datatype(...) /* figure out how many contiguous segments<br>
> are in this */<br>
> #ifdef (FAST_INJECTION_RATE)<br>
> (for all contig segments in datatype){<br>
> low_level_comm_api_get(...,datatype,...)<br>
> }<br>
> #elseif (SLOW_INJECTION_RATE)<br>
> active_message(invoke_remote_pack(datatype,packed_buffer))<br>
> low_level_comm_api_get(...,packed_buffer,...)<br>
> local_unpack(packed_buffer,datatype)<br>
> #endif<br>
> }<br>
> ======================================<br>
> <br>
> If you look at ARMCI or the MPI-2 one-sided implementation over DCMF,<br>
> the overhead associated with the "else" condition is non-trivial
when<br>
> one considers the latency of the BlueGene/P interconnect, the memory<br>
> latency, the clock rate and the absence of dynamic communication<br>
> threads. Forgive me for being specific to one machine, but it is the<br>
> benchmark data I have obtained that inspires me to comment on these<br>
> issues.<br>
> <br>
> On machines with capped message injection rates, implementing the<br>
> "else" condition might be best done in a completely different
manner<br>
> with one-sided packing, which is why I brought that up in the first<br>
> place. Of course, the overhead of packing is large and thus should be<br>
> reserved exclusively for the case where hardware prefers it, and then<br>
> only for "xfer" calls where it is absolutely necessary,
otherwise it<br>
> should be ignored completely.<br>
> <br>
> > 2) There is no requirement in MPI that all tasks of an application
run the<br>
> > same executable so even if one piece of source code that will be used
in an<br>
> > application is free of complex datatypes, only the human being who
knows if<br>
> > other pieces of code will be used by some tasks can make the
judgement that<br>
> > there will be no calls with complex datatypes by any participating
task.<br>
> <br>
> While this may be true for MPI ("no requirement that all tasks of an<br>
> application run the same executable"), it may not be true for all<br>
> systems on which MPI is to be run. Hence, I do not think this is a<br>
> useful solution.<br>
> <br>
> It is exactly my point that the human knows which calls require<br>
> complex datatypes and should be able to make calls to "xfer" in
light<br>
> of this knowledge. Currently the "xfer" API does not support<br>
> bypassing complex datatypes and it is my contention that, in some<br>
> cases, doing so will improve performance markedly.<br>
> <br>
> Best,<br>
> <br>
> Jeff<br>
> <br>
> -- <br>
> Jeff Hammond<br>
> Argonne Leadership Computing Facility<br>
> jhammond@mcs.anl.gov / (630) 252-5381<br>
> http://www.linkedin.com/in/jeffhammond<br>
> http://home.uchicago.edu/~jhammond/<br>
> _______________________________________________<br>
> mpi3-rma mailing list<br>
> mpi3-rma@lists.mpi-forum.org<br>
> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma</span></p>
</div>
</div>
</div></div></body>
</html>