[Mpi3-hybridpm] Shared memory segments proposal

[James Dinan]

Hi Bill,

What you're suggesting is of course the right thing to do from the MPI 
library's point of view.  I'm coming at it from the other side as an MPI 
user.  If I want to write robust and portable code and there's a good 
chance that shared segments will be non-symmetric on some systems then I 
should do the extra work of coding for non-symmetric allocation since it 
will work everywhere.  So, my feeling is that if we can't guarantee 
symmetry 99% of the time then there is no point in offering it since 
good programmers will not want to use it and others who do might be 
shooting themselves in the foot on portability.

I've been thinking of symmetric allocation as a convenience rather than 
performance property.  However, if I merge threads and bring in the 
discussion about synchronization - will symmetric allocation allow us to 
do more efficient locking?  System V uses semaphores which are system 
calls.  If we have non-symmetric mappings of the same data, will all 
architectures (e.g. VIVT caches) be able to perform atomic operations on 
that memory?  So, if we can do a symmetric mapping will this allow us to 
do something more efficient a la pthreads mutexes?

  ~Jim.

On 11/08/2010 08:11 AM, William Gropp wrote:
> I understand the issue, but the problem is what you want to do if the
> symmetric allocation isn't feasible - do you permit the allocation or do
> you fail? One option would be an info that required symmetric allocation
> or failure; this wouldn't prevent the use of shared memory when
> symmetric allocation wasn't feasible (modulo the discussion about
> whether info is always a hint or can be a requirement).
>
> Bill
>
> On Nov 6, 2010, at 11:18 AM, James Dinan wrote:
>
>> Hi Bill,
>>
>> The problem with favoring one, but allowing both is that the
>> representation for pointers changes from symmetric to non-symmetric
>> allocations. In the non-symmetric case, direct pointers are not valid
>> on every process so you have to use displacements instead. In a linked
>> list, for example, you'd store a displacement in next instead of the
>> direct next pointer:
>>
>> elem_t *next = current->next + ((uint8_t*) segment_base);
>>
>> versus symmetric:
>>
>> elem_t *next = current->next;
>>
>> This means that next has a different type in symmetric vs non-symmetric
>> and will be dereferenced differently - it can get messy supporting both
>> pointers and displacements in the same code. Portable programs would
>> most likely go for the least common denominator and assume non-symmetric
>> so any effort we invest in supporting symmetric allocation would be
>> wasted.
>>
>> This isn't a huge deal to me, you can easily put together a few macros
>> to make non-symmetric allocations easier to live with. I'm willing to
>> pass on this if we don't feel we can guarantee that it will always be
>> symmetric. This just popped out as a potential opportunity for us to
>> improve on one of the hassles of conventional shared segments.
>>
>> ~Jim.
>>
>> PS- CC'd back to the Hybrid PM list, I think Ron and others might be
>> interested in the discussion.
>>
>> On 11/6/10 10:45 AM, William Gropp wrote:
>>> Agreed, though having a hint for the implementation would be helpful.
>>> There are cases where it may be very hard to deliver the guarantee, as
>>> it depends on the allocation history of each process. For CAF-like
>>> applications with no allocated data, providing symmetric allocation is
>>> (usually) easy. When applications have been running a dynamic mesh
>>> refinement application for a few hours, it could get hard :) . Thus,
>>> even having a requirement might not work unless your address space is so
>>> big (including the capabilities of your paging and swap) that you can
>>> always find unallocated addresses in common. More precisely, we can't
>>> require that symmetric allocations work, so the application must either
>>> be prepared to accept non-symmetric or failure from the allocator.
>>> Adding a hint requesting symmetric allocation and an advice to
>>> implementors stating that high-quality implementations are expected to
>>> pay attention to the hint might be sufficient.
>>>
>>> Bill
>>>
>>> On Nov 6, 2010, at 10:37 AM, Pavan Balaji wrote:
>>>
>>>> Bill,
>>>>
>>>> On 11/06/2010 08:45 AM, William Gropp wrote:
>>>>> On the specific question, one could use an info hint to request
>>>>> symmetric allocation.
>>>>
>>>> I believe Jim is looking for a "guarantee" on symmetric allocation,
>>>> which the info argument cannot give as the MPI implementation is
>>>> allowed
>>>> to ignore it.
>>>>
>>>> It's possible for the application to check if all the base pointers
>>>> returned are symmetric or not, and decide what to do after that. But it
>>>> cannot rely on the fact that it'll always get a symmetric allocation
>>>> (unless we change the function prototype and add a parameter for it).
>>>>
>>>> -- Pavan
>
> William Gropp
> Deputy Director for Research
> Institute for Advanced Computing Applications and Technologies
> Paul and Cynthia Saylor Professor of Computer Science
> University of Illinois Urbana-Champaign
>
>
>
>