[Mpi3-hybridpm] Shared memory segments proposal

[William Gropp]

I understand the issue, but the problem is what you want to do if the  
symmetric allocation isn't feasible - do you permit the allocation or  
do you fail?  One option would be an info that required symmetric  
allocation or failure; this wouldn't prevent the use of shared memory  
when symmetric allocation wasn't feasible (modulo the discussion about  
whether info is always a hint or can be a requirement).

Bill

On Nov 6, 2010, at 11:18 AM, James Dinan wrote:

> Hi Bill,
>
> The problem with favoring one, but allowing both is that the
> representation for pointers changes from symmetric to non-symmetric
> allocations.  In the non-symmetric case, direct pointers are not valid
> on every process so you have to use displacements instead.  In a  
> linked
> list, for example, you'd store a displacement in next instead of the
> direct next pointer:
>
> elem_t *next = current->next + ((uint8_t*) segment_base);
>
> versus symmetric:
>
> elem_t *next = current->next;
>
> This means that next has a different type in symmetric vs non- 
> symmetric
> and will be dereferenced differently - it can get messy supporting  
> both
> pointers and displacements in the same code.  Portable programs would
> most likely go for the least common denominator and assume non- 
> symmetric
> so any effort we invest in supporting symmetric allocation would be  
> wasted.
>
> This isn't a huge deal to me, you can easily put together a few macros
> to make non-symmetric allocations easier to live with.  I'm willing to
> pass on this if we don't feel we can guarantee that it will always be
> symmetric.  This just popped out as a potential opportunity for us to
> improve on one of the hassles of conventional shared segments.
>
>  ~Jim.
>
> PS- CC'd back to the Hybrid PM list, I think Ron and others might be
> interested in the discussion.
>
> On 11/6/10 10:45 AM, William Gropp wrote:
>> Agreed, though having a hint for the implementation would be helpful.
>> There are cases where it may be very hard to deliver the guarantee,  
>> as
>> it depends on the allocation history of each process. For CAF-like
>> applications with no allocated data, providing symmetric allocation  
>> is
>> (usually) easy. When applications have been running a dynamic mesh
>> refinement application for a few hours, it could get hard :) . Thus,
>> even having a requirement might not work unless your address space  
>> is so
>> big (including the capabilities of your paging and swap) that you can
>> always find unallocated addresses in common. More precisely, we can't
>> require that symmetric allocations work, so the application must  
>> either
>> be prepared to accept non-symmetric or failure from the allocator.
>> Adding a hint requesting symmetric allocation and an advice to
>> implementors stating that high-quality implementations are expected  
>> to
>> pay attention to the hint might be sufficient.
>>
>> Bill
>>
>> On Nov 6, 2010, at 10:37 AM, Pavan Balaji wrote:
>>
>>> Bill,
>>>
>>> On 11/06/2010 08:45 AM, William Gropp wrote:
>>>> On the specific question, one could use an info hint to request
>>>> symmetric allocation.
>>>
>>> I believe Jim is looking for a "guarantee" on symmetric allocation,
>>> which the info argument cannot give as the MPI implementation is  
>>> allowed
>>> to ignore it.
>>>
>>> It's possible for the application to check if all the base pointers
>>> returned are symmetric or not, and decide what to do after that.  
>>> But it
>>> cannot rely on the fact that it'll always get a symmetric allocation
>>> (unless we change the function prototype and add a parameter for  
>>> it).
>>>
>>> -- Pavan

William Gropp
Deputy Director for Research
Institute for Advanced Computing Applications and Technologies
Paul and Cynthia Saylor Professor of Computer Science
University of Illinois Urbana-Champaign