[Mpi3-hybridpm] Shared memory segments proposal

[William Gropp]

Hi James,

Standards have to be careful about what they guarantee.  For example,  
no where does Fortran guarantee that you can open more than one file  
(at least through F90), but most users do not choose to abandon  
Fortran because they need to access more than one file.  And we  
clearly cannot guarantee symmetric allocation of large amounts of  
memory in all cases, since there is no guarantee that it is even  
possible (presumably, the current systems return a null pointer when  
they can't provide symmetric memory).  Many of the "guarantees" that  
users assume do not, in fact, exist.

The question really is: is there a good chance that an implementation  
would choose not to make users happy?  In the MPI Forum process, we  
consider this a quality of implementation issue, and rely on the  
market to ensure that the implementations perform well.  In this case,  
I believe an advice to implementors that symmetric allocation should  
be provided if possible should be sufficient.  The other option is to  
require symmetric or failure, even if shared memory is available.  If  
that's preferable, then the allocation routine should have a mandatory  
assert that indicates whether symmetric memory is a requirement.  The  
one thing that we can't guarantee is that symmetric memory allocation  
will always succeed.

Bill

On Nov 9, 2010, at 10:11 AM, James Dinan wrote:

> Hi Bill,
>
> What you're suggesting is of course the right thing to do from the MPI
> library's point of view.  I'm coming at it from the other side as an  
> MPI
> user.  If I want to write robust and portable code and there's a good
> chance that shared segments will be non-symmetric on some systems  
> then I
> should do the extra work of coding for non-symmetric allocation  
> since it
> will work everywhere.  So, my feeling is that if we can't guarantee
> symmetry 99% of the time then there is no point in offering it since
> good programmers will not want to use it and others who do might be
> shooting themselves in the foot on portability.
>
> I've been thinking of symmetric allocation as a convenience rather  
> than
> performance property.  However, if I merge threads and bring in the
> discussion about synchronization - will symmetric allocation allow  
> us to
> do more efficient locking?  System V uses semaphores which are system
> calls.  If we have non-symmetric mappings of the same data, will all
> architectures (e.g. VIVT caches) be able to perform atomic  
> operations on
> that memory?  So, if we can do a symmetric mapping will this allow  
> us to
> do something more efficient a la pthreads mutexes?
>
>  ~Jim.
>
> On 11/08/2010 08:11 AM, William Gropp wrote:
>> I understand the issue, but the problem is what you want to do if the
>> symmetric allocation isn't feasible - do you permit the allocation  
>> or do
>> you fail? One option would be an info that required symmetric  
>> allocation
>> or failure; this wouldn't prevent the use of shared memory when
>> symmetric allocation wasn't feasible (modulo the discussion about
>> whether info is always a hint or can be a requirement).
>>
>> Bill
>>
>> On Nov 6, 2010, at 11:18 AM, James Dinan wrote:
>>
>>> Hi Bill,
>>>
>>> The problem with favoring one, but allowing both is that the
>>> representation for pointers changes from symmetric to non-symmetric
>>> allocations. In the non-symmetric case, direct pointers are not  
>>> valid
>>> on every process so you have to use displacements instead. In a  
>>> linked
>>> list, for example, you'd store a displacement in next instead of the
>>> direct next pointer:
>>>
>>> elem_t *next = current->next + ((uint8_t*) segment_base);
>>>
>>> versus symmetric:
>>>
>>> elem_t *next = current->next;
>>>
>>> This means that next has a different type in symmetric vs non- 
>>> symmetric
>>> and will be dereferenced differently - it can get messy supporting  
>>> both
>>> pointers and displacements in the same code. Portable programs would
>>> most likely go for the least common denominator and assume non- 
>>> symmetric
>>> so any effort we invest in supporting symmetric allocation would be
>>> wasted.
>>>
>>> This isn't a huge deal to me, you can easily put together a few  
>>> macros
>>> to make non-symmetric allocations easier to live with. I'm willing  
>>> to
>>> pass on this if we don't feel we can guarantee that it will always  
>>> be
>>> symmetric. This just popped out as a potential opportunity for us to
>>> improve on one of the hassles of conventional shared segments.
>>>
>>> ~Jim.
>>>
>>> PS- CC'd back to the Hybrid PM list, I think Ron and others might be
>>> interested in the discussion.
>>>
>>> On 11/6/10 10:45 AM, William Gropp wrote:
>>>> Agreed, though having a hint for the implementation would be  
>>>> helpful.
>>>> There are cases where it may be very hard to deliver the  
>>>> guarantee, as
>>>> it depends on the allocation history of each process. For CAF-like
>>>> applications with no allocated data, providing symmetric  
>>>> allocation is
>>>> (usually) easy. When applications have been running a dynamic mesh
>>>> refinement application for a few hours, it could get hard :) .  
>>>> Thus,
>>>> even having a requirement might not work unless your address  
>>>> space is so
>>>> big (including the capabilities of your paging and swap) that you  
>>>> can
>>>> always find unallocated addresses in common. More precisely, we  
>>>> can't
>>>> require that symmetric allocations work, so the application must  
>>>> either
>>>> be prepared to accept non-symmetric or failure from the allocator.
>>>> Adding a hint requesting symmetric allocation and an advice to
>>>> implementors stating that high-quality implementations are  
>>>> expected to
>>>> pay attention to the hint might be sufficient.
>>>>
>>>> Bill
>>>>
>>>> On Nov 6, 2010, at 10:37 AM, Pavan Balaji wrote:
>>>>
>>>>> Bill,
>>>>>
>>>>> On 11/06/2010 08:45 AM, William Gropp wrote:
>>>>>> On the specific question, one could use an info hint to request
>>>>>> symmetric allocation.
>>>>>
>>>>> I believe Jim is looking for a "guarantee" on symmetric  
>>>>> allocation,
>>>>> which the info argument cannot give as the MPI implementation is
>>>>> allowed
>>>>> to ignore it.
>>>>>
>>>>> It's possible for the application to check if all the base  
>>>>> pointers
>>>>> returned are symmetric or not, and decide what to do after that.  
>>>>> But it
>>>>> cannot rely on the fact that it'll always get a symmetric  
>>>>> allocation
>>>>> (unless we change the function prototype and add a parameter for  
>>>>> it).
>>>>>
>>>>> -- Pavan
>>
>> William Gropp
>> Deputy Director for Research
>> Institute for Advanced Computing Applications and Technologies
>> Paul and Cynthia Saylor Professor of Computer Science
>> University of Illinois Urbana-Champaign
>>
>>
>>
>>
>

William Gropp
Deputy Director for Research
Institute for Advanced Computing Applications and Technologies
Paul and Cynthia Saylor Professor of Computer Science
University of Illinois Urbana-Champaign