[Mpiwg-large-counts] Large Count - the principles for counts, sizes, and byte and nonbyte displacements

[Jeff Squyres (jsquyres)]

On Oct 24, 2019, at 2:19 AM, Rolf Rabenseifner <rabenseifner at hlrs.de> wrote:
> 
>> Does MPI support segmented address spaces or not?
> 
> When  I understand correctly then MPI is and was always prepared for segmented address spaces as long as all integer types are signed integers and the segmentation is done through some higher order bits.


As I was pondering your replies from yesterday and thinking about how to respond, this is exactly the same point that I got to.

I think what I have been failing to elucidate is that this is not how modern segmented address spaces work.  It's not sufficient to have a few bits to specify which segment to use.  Instead, you typically need three things:

1. The type of memory segment being described (which may be just a few bits).  In reality, this "type" may correlate to which API you need to use to access this memory (e.g., main RAM, RDMA, accelerator memory, persistent memory, other IO memory, ...etc.).
2. A context of some kind (usually more than a small number of bits -- perhaps 32 or 64 bits).
3. A pointer (usually 32 or 64 bits).

For example, if I'm accessing IO memory of some flavor, I probably need some kind of context/handle that points to a descriptor with information about my data segment on the IO peripheral in question (e.g., a NIC), and then a pointer for the specific place that I want in that specific memory.  I use this context/handle + pointer with some underlying API to get/set that data.

I think that's the disconnect here: the prior thinking about MPI_Aint may have assumed "it's either main RAM or it's XYZ memory, and the rest of the bits are enough to uniquely describe the address."  But that doesn't seem sufficient any more to describe the evolution of complex, segmented memory systems (and be able to use their supporting APIs).

Meaning: perhaps a pure integer MPI_Aint may have been sufficient in the past.  But I'm not sure that's useful any more.

-- 
Jeff Squyres
jsquyres at cisco.com