[MPI3 Fortran] Argument data sizes

[Jeff Squyres]

On Sep 18, 2008, at 11:39 AM, Bill Long wrote:

>> I said that MPI's *count* arguments are C int's.  I never said that  
>> MPI's *address* specifications were C int's (indeed, in C, they're  
>> not -- they're of type MPI_Aint, which is a type that is large  
>> enough to hold an address-sized integer).
>>
>> The *count* (and other similar parameters) is how many [typed]  
>> items you want to send / receive / etc.  Right now, MPI defines  
>> counts as int's in C.
>
> Actually, this is what I'm talking about.  In particular, the size  
> of a single array, which in a modern machine can easily be larger  
> that 4 billion elements.  The limits on array sizes and buffer sizes  
> are directly tied to the hardware addressing limitations of the  
> system.  The idea that counts should be limited to 32 bits is an  
> archaic remnant from the era when machine addresses were 32 bits.


I don't follow Cray hardware -- do you actually have machines that  
have more than 32GB per core?  If my math is right, 4^32 * 8 = 32GB  
would be the actual size of a 4B element array of doubles.

Remember three key facts:

- MPI's model is effectively distributed memory
- MPI's counts are *elements*, not *bytes*
- MPI datatypes are composable

Most MPI apps that I have seen with huge arrays like this are actually  
representing multiple dimensions -- so they usually make a datatype  
for one row and use that as a unit to compose larger datatypes (e.g.,  
a plane, a 3D space, ...etc.).  So most apps send "1 array" (or "N  
rows" or ...), not 2B (or 4B or ...) individual elements.

This is at least part of the discussion in the Forum about this issue  
(I mentioned this in a prior mail as well): since MPI datatypes are  
composable, applications can easily smash the 2B int count limitation  
anyway.  As Alexander S. noted, however, there are other issues as well.

-- 
Jeff Squyres
Cisco Systems