<html><body>
<p><tt>Hi,</tt><br>
<br>
<tt>Jeff Hammond wrote:</tt><br>
<tt> Active messages are very appealing to me but I'm not aware of a<br>
library implementation (F77 and C compatible) that runs on BlueGene/P,<br>
Cray XT, Infiniband and Myrinet. Can you point me to one?</tt><br>
<br>
<tt>IBM has since 1997 an Active Messaging library for the SP/Power4,5&6 clusters (SP switch, SP switch2, HPS and Infiniband networked clusters). It is called Low Level API (LAPI for short). It supports F77 (old interfaces), F95 (newer interfaces), C and C++ bindings. This library was initially written to support NWChem/GA/ARMCI on SP cluster for PNNL.</tt><br>
<tt><a href="http://publib.boulder.ibm.com/epubs/pdf/a2279368.pdf">http://publib.boulder.ibm.com/epubs/pdf/a2279368.pdf</a> (latest document).</tt><br>
<br>
<tt>GA/ARMCI, GASNet and IBM's Parallel Environment MPI are coded to LAPI on IBM's Power clusters (AIX/Linux). </tt><br>
<tt>They all use LAPI's Reliable Active Messaging layer and and some of LAPI's one-sided Put/Get/RMW functions.</tt><br>
<tt>LAPI library exploits RDMA, striping across multiple adapters, failover/recovery for the upper protocols plus other things.</tt><br>
<br>
<tt>On BG/P there is DCMF (Deep Computing Messaging Framework) library that supports active messaging. I believe it is limited to C and C++ bindings. There also, GA/ARMCI, GASNet and MPICH uses DCMF to do the messaging on BG/P.</tt><br>
<br>
<br>
<tt>best regards,</tt><br>
<br>
-- chulho<br>
<br>
PE & Communication Protocols Architect (STSM)<br>
2455 South Road, P963/7T, Poughkeepsie, NY 12601<br>
Poughkeepsie HPC Development Lab<br>
(845) 433-7863 T/L 8 - 293-7863 <br>
chulho@us.ibm.com<br>
<br>
<img width="16" height="16" src="cid:1__=0ABBFF31DFC13D5C8f9e8a93df938@us.ibm.com" border="0" alt="Inactive hide details for Jeff Hammond ---04/24/2009 08:43:28 AM---Greg, > The only significant public one-sided app that I've "><font color="#424282">Jeff Hammond ---04/24/2009 08:43:28 AM---Greg, > The only significant public one-sided app that I've run into is</font><br>
<br>
<table width="100%" border="0" cellspacing="0" cellpadding="0">
<tr valign="top"><td width="1%"><img width="96" height="1" src="cid:2__=0ABBFF31DFC13D5C8f9e8a93df938@us.ibm.com" border="0" alt=""><br>
<font size="2" color="#5F5F5F">From:</font></td><td width="100%"><img width="1" height="1" src="cid:2__=0ABBFF31DFC13D5C8f9e8a93df938@us.ibm.com" border="0" alt=""><br>
<font size="2">Jeff Hammond <jeff.science@gmail.com></font></td></tr>
<tr valign="top"><td width="1%"><img width="96" height="1" src="cid:2__=0ABBFF31DFC13D5C8f9e8a93df938@us.ibm.com" border="0" alt=""><br>
<font size="2" color="#5F5F5F">To:</font></td><td width="100%"><img width="1" height="1" src="cid:2__=0ABBFF31DFC13D5C8f9e8a93df938@us.ibm.com" border="0" alt=""><br>
<font size="2">Main MPI Forum mailing list <mpi-forum@lists.mpi-forum.org></font></td></tr>
<tr valign="top"><td width="1%"><img width="96" height="1" src="cid:2__=0ABBFF31DFC13D5C8f9e8a93df938@us.ibm.com" border="0" alt=""><br>
<font size="2" color="#5F5F5F">Date:</font></td><td width="100%"><img width="1" height="1" src="cid:2__=0ABBFF31DFC13D5C8f9e8a93df938@us.ibm.com" border="0" alt=""><br>
<font size="2">04/24/2009 08:43 AM</font></td></tr>
<tr valign="top"><td width="1%"><img width="96" height="1" src="cid:2__=0ABBFF31DFC13D5C8f9e8a93df938@us.ibm.com" border="0" alt=""><br>
<font size="2" color="#5F5F5F">Subject:</font></td><td width="100%"><img width="1" height="1" src="cid:2__=0ABBFF31DFC13D5C8f9e8a93df938@us.ibm.com" border="0" alt=""><br>
<font size="2">Re: [Mpi-forum] MPI One-Sided Communication</font></td></tr>
<tr valign="top"><td width="1%"><img width="96" height="1" src="cid:2__=0ABBFF31DFC13D5C8f9e8a93df938@us.ibm.com" border="0" alt=""><br>
<font size="2" color="#5F5F5F">Sent by:</font></td><td width="100%"><img width="1" height="1" src="cid:2__=0ABBFF31DFC13D5C8f9e8a93df938@us.ibm.com" border="0" alt=""><br>
<font size="2">mpi-forum-bounces@lists.mpi-forum.org</font></td></tr>
</table>
<hr width="100%" size="2" align="left" noshade style="color:#8091A5; "><br>
<br>
<br>
<tt>Greg,<br>
<br>
> The only significant public one-sided app that I've run into is<br>
> NWCHEM, which uses a non-MPI one-sided library. And in that case,<br>
> PathScale did an implementation on top of 2-sided MPI-1 send/recv, and<br>
> it scaled better than competing real one-sided hardware. (Check out my<br>
> old whitepaper for numbers.)<br>
<br>
You're comparing ("scaled better than") apples ("an implementation on<br>
top of 2-sided MPI-1 send/recv") and oranges ("real one-sided<br>
hardware"). Please post the whitepaper (I cannot find it online) and<br>
clarify what you mean here.<br>
<br>
Did Pathscale implement Global Arrays on top of MPI-1 instead of ARMCI<br>
and found that NWChem scaled better in this context? NWChem has many<br>
different modules, some of which can be scaled fine with a<br>
message-passing (MP) model whereas others cannot. It took a decade<br>
after NWChem first did it with GA for anyone to demonstrate scalable<br>
quantum chemical many-body theories using MPI and then only with a<br>
pseudo-compiler and new programming language (ACES3 + SIAL if anyone<br>
cares). One-sided provides a much better programming model than MP<br>
for complex algorithms that show up in quantum chemistry, which is<br>
why GA not MPI is the basis for every existing large-scale code of<br>
this kind.<br>
<br>
If MPI-1 was better than ARMCI for the "meat and potatoes" in NWChem,<br>
NWChem would be using it instead.<br>
<br>
> Now that I'm not an HPC guy anymore, I'll note that the distributed<br>
> database that I'm doing is completely implemented in terms of active messages.<br>
<br>
Active messages are very appealing to me but I'm not aware of a<br>
library implementation (F77 and C compatible) that runs on BlueGene/P,<br>
Cray XT, Infiniband and Myrinet. Can you point me to one?<br>
<br>
> They're easier to use corrrectly than real one-sided<br>
> communications, which are hard because programmers screw up<br>
> synchronization & double-buffering.<br>
<br>
If one-sided is evaluated at the level of Global Arrays rather than<br>
ARMCI by itself, those issues disappear. Perhaps that's an unfair<br>
comparison but direct use of ARMCI is rare whereas GA is relatively<br>
popular.<br>
<br>
Jeff<br>
<br>
-- <br>
Jeff Hammond<br>
The University of Chicago<br>
</tt><tt><a href="http://home.uchicago.edu/~jhammond/">http://home.uchicago.edu/~jhammond/</a></tt><tt><br>
_______________________________________________<br>
mpi-forum mailing list<br>
mpi-forum@lists.mpi-forum.org<br>
</tt><tt><a href="http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum">http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum</a></tt><tt><br>
</tt><br>
<br>
</body></html>