[Mpi3-hybridpm] Shared memory segments proposal
wgropp at illinois.edu
Mon Nov 8 08:11:17 CST 2010
I understand the issue, but the problem is what you want to do if the
symmetric allocation isn't feasible - do you permit the allocation or
do you fail? One option would be an info that required symmetric
allocation or failure; this wouldn't prevent the use of shared memory
when symmetric allocation wasn't feasible (modulo the discussion about
whether info is always a hint or can be a requirement).
On Nov 6, 2010, at 11:18 AM, James Dinan wrote:
> Hi Bill,
> The problem with favoring one, but allowing both is that the
> representation for pointers changes from symmetric to non-symmetric
> allocations. In the non-symmetric case, direct pointers are not valid
> on every process so you have to use displacements instead. In a
> list, for example, you'd store a displacement in next instead of the
> direct next pointer:
> elem_t *next = current->next + ((uint8_t*) segment_base);
> versus symmetric:
> elem_t *next = current->next;
> This means that next has a different type in symmetric vs non-
> and will be dereferenced differently - it can get messy supporting
> pointers and displacements in the same code. Portable programs would
> most likely go for the least common denominator and assume non-
> so any effort we invest in supporting symmetric allocation would be
> This isn't a huge deal to me, you can easily put together a few macros
> to make non-symmetric allocations easier to live with. I'm willing to
> pass on this if we don't feel we can guarantee that it will always be
> symmetric. This just popped out as a potential opportunity for us to
> improve on one of the hassles of conventional shared segments.
> PS- CC'd back to the Hybrid PM list, I think Ron and others might be
> interested in the discussion.
> On 11/6/10 10:45 AM, William Gropp wrote:
>> Agreed, though having a hint for the implementation would be helpful.
>> There are cases where it may be very hard to deliver the guarantee,
>> it depends on the allocation history of each process. For CAF-like
>> applications with no allocated data, providing symmetric allocation
>> (usually) easy. When applications have been running a dynamic mesh
>> refinement application for a few hours, it could get hard :) . Thus,
>> even having a requirement might not work unless your address space
>> is so
>> big (including the capabilities of your paging and swap) that you can
>> always find unallocated addresses in common. More precisely, we can't
>> require that symmetric allocations work, so the application must
>> be prepared to accept non-symmetric or failure from the allocator.
>> Adding a hint requesting symmetric allocation and an advice to
>> implementors stating that high-quality implementations are expected
>> pay attention to the hint might be sufficient.
>> On Nov 6, 2010, at 10:37 AM, Pavan Balaji wrote:
>>> On 11/06/2010 08:45 AM, William Gropp wrote:
>>>> On the specific question, one could use an info hint to request
>>>> symmetric allocation.
>>> I believe Jim is looking for a "guarantee" on symmetric allocation,
>>> which the info argument cannot give as the MPI implementation is
>>> to ignore it.
>>> It's possible for the application to check if all the base pointers
>>> returned are symmetric or not, and decide what to do after that.
>>> But it
>>> cannot rely on the fact that it'll always get a symmetric allocation
>>> (unless we change the function prototype and add a parameter for
>>> -- Pavan
Deputy Director for Research
Institute for Advanced Computing Applications and Technologies
Paul and Cynthia Saylor Professor of Computer Science
University of Illinois Urbana-Champaign
More information about the mpiwg-hybridpm