[Mpi-forum] MPI One-Sided Communication

[Chulho Kim]

Hi,

Jeff Hammond wrote:
      Active messages are very appealing to me but I'm not aware of a
      library implementation (F77 and C compatible) that runs on
BlueGene/P,
      Cray XT, Infiniband and Myrinet.  Can you point me to one?

IBM has since 1997 an Active Messaging library for the SP/Power4,5&6
clusters (SP switch, SP switch2, HPS and Infiniband networked clusters).
It is called Low Level API (LAPI for short).  It supports F77 (old
interfaces), F95 (newer interfaces), C and C++ bindings.  This library was
initially written to support NWChem/GA/ARMCI on SP cluster for PNNL.
http://publib.boulder.ibm.com/epubs/pdf/a2279368.pdf  (latest document).

GA/ARMCI, GASNet and IBM's Parallel Environment MPI are coded to LAPI on
IBM's Power clusters (AIX/Linux).
They all use LAPI's Reliable Active Messaging layer and and some of LAPI's
one-sided Put/Get/RMW functions.
LAPI library exploits RDMA, striping across multiple adapters,
failover/recovery for the upper protocols plus other things.

On BG/P there is DCMF (Deep Computing Messaging Framework) library that
supports active messaging.  I believe it is limited to C and C++ bindings.
There also, GA/ARMCI, GASNet and MPICH uses DCMF to do the messaging on
BG/P.


best regards,

-- chulho

PE & Communication Protocols Architect (STSM)
2455 South Road, P963/7T, Poughkeepsie, NY 12601
Poughkeepsie HPC Development Lab
(845) 433-7863  T/L 8 - 293-7863
chulho at us.ibm.com


                                                                                                                   
  From:       Jeff Hammond <jeff.science at gmail.com>                                                                
                                                                                                                   
  To:         Main MPI Forum mailing list <mpi-forum at lists.mpi-forum.org>                                          
                                                                                                                   
  Date:       04/24/2009 08:43 AM                                                                                  
                                                                                                                   
  Subject:    Re: [Mpi-forum] MPI One-Sided Communication                                                          
                                                                                                                   
  Sent by:    mpi-forum-bounces at lists.mpi-forum.org                                                                
                                                                                                                   





Greg,

> The only significant public one-sided app that I've run into is
> NWCHEM, which uses a non-MPI one-sided library. And in that case,
> PathScale did an implementation on top of 2-sided MPI-1 send/recv, and
> it scaled better than competing real one-sided hardware. (Check out my
> old whitepaper for numbers.)

You're comparing ("scaled better than") apples ("an implementation on
top of 2-sided MPI-1 send/recv") and oranges ("real one-sided
hardware").  Please post the whitepaper (I cannot find it online) and
clarify what you mean here.

Did Pathscale implement Global Arrays on top of MPI-1 instead of ARMCI
and found that NWChem scaled better in this context?  NWChem has many
different modules, some of which can be scaled fine with a
message-passing (MP) model whereas others cannot.  It took a decade
after NWChem first did it with GA for anyone to demonstrate scalable
quantum chemical many-body theories using MPI and then only with a
pseudo-compiler and new programming language (ACES3 + SIAL if anyone
cares).  One-sided provides a much better programming model than MP
for  complex algorithms that show up in quantum chemistry, which is
why GA not MPI is the basis for every existing large-scale code of
this kind.

If MPI-1 was better than ARMCI for the "meat and potatoes" in NWChem,
NWChem would be using it instead.

> Now that I'm not an HPC guy anymore, I'll note that the distributed
> database that I'm doing is completely implemented in terms of active
messages.

Active messages are very appealing to me but I'm not aware of a
library implementation (F77 and C compatible) that runs on BlueGene/P,
Cray XT, Infiniband and Myrinet.  Can you point me to one?

> They're easier to use corrrectly than real one-sided
> communications, which are hard because programmers screw up
> synchronization & double-buffering.

If one-sided is evaluated at the level of Global Arrays rather than
ARMCI by itself, those issues disappear.  Perhaps that's an unfair
comparison but direct use of ARMCI is rare whereas GA is relatively
popular.

Jeff

--
Jeff Hammond
The University of Chicago
http://home.uchicago.edu/~jhammond/
_______________________________________________
mpi-forum mailing list
mpi-forum at lists.mpi-forum.org
http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-forum

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mpi-forum.org/pipermail/mpi-forum/attachments/20090424/1d48e008/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: graycol.gif
Type: image/gif
Size: 105 bytes
Desc: not available
URL: <http://lists.mpi-forum.org/pipermail/mpi-forum/attachments/20090424/1d48e008/attachment-0002.gif>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: ecblank.gif
Type: image/gif
Size: 45 bytes
Desc: not available
URL: <http://lists.mpi-forum.org/pipermail/mpi-forum/attachments/20090424/1d48e008/attachment-0003.gif>