[Mpi3-hybridpm] Shared memory segments proposal

[James Dinan]

Hi Bill,

The problem with favoring one, but allowing both is that the 
representation for pointers changes from symmetric to non-symmetric 
allocations.  In the non-symmetric case, direct pointers are not valid 
on every process so you have to use displacements instead.  In a linked 
list, for example, you'd store a displacement in next instead of the 
direct next pointer:

elem_t *next = current->next + ((uint8_t*) segment_base);

versus symmetric:

elem_t *next = current->next;

This means that next has a different type in symmetric vs non-symmetric 
and will be dereferenced differently - it can get messy supporting both 
pointers and displacements in the same code.  Portable programs would 
most likely go for the least common denominator and assume non-symmetric 
so any effort we invest in supporting symmetric allocation would be wasted.

This isn't a huge deal to me, you can easily put together a few macros 
to make non-symmetric allocations easier to live with.  I'm willing to 
pass on this if we don't feel we can guarantee that it will always be 
symmetric.  This just popped out as a potential opportunity for us to 
improve on one of the hassles of conventional shared segments.

  ~Jim.

PS- CC'd back to the Hybrid PM list, I think Ron and others might be 
interested in the discussion.

On 11/6/10 10:45 AM, William Gropp wrote:
> Agreed, though having a hint for the implementation would be helpful.
> There are cases where it may be very hard to deliver the guarantee, as
> it depends on the allocation history of each process. For CAF-like
> applications with no allocated data, providing symmetric allocation is
> (usually) easy. When applications have been running a dynamic mesh
> refinement application for a few hours, it could get hard :) . Thus,
> even having a requirement might not work unless your address space is so
> big (including the capabilities of your paging and swap) that you can
> always find unallocated addresses in common. More precisely, we can't
> require that symmetric allocations work, so the application must either
> be prepared to accept non-symmetric or failure from the allocator.
> Adding a hint requesting symmetric allocation and an advice to
> implementors stating that high-quality implementations are expected to
> pay attention to the hint might be sufficient.
>
> Bill
>
> On Nov 6, 2010, at 10:37 AM, Pavan Balaji wrote:
>
>> Bill,
>>
>> On 11/06/2010 08:45 AM, William Gropp wrote:
>>> On the specific question, one could use an info hint to request
>>> symmetric allocation.
>>
>> I believe Jim is looking for a "guarantee" on symmetric allocation,
>> which the info argument cannot give as the MPI implementation is allowed
>> to ignore it.
>>
>> It's possible for the application to check if all the base pointers
>> returned are symmetric or not, and decide what to do after that. But it
>> cannot rely on the fact that it'll always get a symmetric allocation
>> (unless we change the function prototype and add a parameter for it).
>>
>> -- Pavan