[MPI3 Fortran] Argument data sizes

[Bill Long]

Jeff Squyres wrote:
> On Sep 18, 2008, at 11:39 AM, Bill Long wrote:
>
>>> I said that MPI's *count* arguments are C int's.  I never said that 
>>> MPI's *address* specifications were C int's (indeed, in C, they're 
>>> not -- they're of type MPI_Aint, which is a type that is large 
>>> enough to hold an address-sized integer).
>>>
>>> The *count* (and other similar parameters) is how many [typed] items 
>>> you want to send / receive / etc.  Right now, MPI defines counts as 
>>> int's in C.
>>
>> Actually, this is what I'm talking about.  In particular, the size of 
>> a single array, which in a modern machine can easily be larger that 4 
>> billion elements.  The limits on array sizes and buffer sizes are 
>> directly tied to the hardware addressing limitations of the system.  
>> The idea that counts should be limited to 32 bits is an archaic 
>> remnant from the era when machine addresses were 32 bits.
>
>
> I don't follow Cray hardware -- do you actually have machines that 
> have more than 32GB per core?  If my math is right, 4^32 * 8 = 32GB 
> would be the actual size of a 4B element array of doubles.

Yes, but you don't really need that.  It is not uncommon to use OpenMP 
within a local SMP domain and use MPI between such domains.  Local SMP 
domains of more than 32GB are not hard to find.  Only 16GB is needed for 
an array with 4-byte elements.  PC's have that much memory these days.  
If the intention is to make a standard relevant going forward, the 
target should be clear.

Cheers,
Bill


>
> Remember three key facts:
>
> - MPI's model is effectively distributed memory
> - MPI's counts are *elements*, not *bytes*
> - MPI datatypes are composable
>
> Most MPI apps that I have seen with huge arrays like this are actually 
> representing multiple dimensions -- so they usually make a datatype 
> for one row and use that as a unit to compose larger datatypes (e.g., 
> a plane, a 3D space, ...etc.).  So most apps send "1 array" (or "N 
> rows" or ...), not 2B (or 4B or ...) individual elements.
>
> This is at least part of the discussion in the Forum about this issue 
> (I mentioned this in a prior mail as well): since MPI datatypes are 
> composable, applications can easily smash the 2B int count limitation 
> anyway.  As Alexander S. noted, however, there are other issues as well.
>

-- 
Bill Long                                   longb at cray.com
Fortran Technical Support    &              voice: 651-605-9024
Bioinformatics Software Development         fax:   651-605-9142
Cray Inc., 1340 Mendota Heights Rd., Mendota Heights, MN, 55120