[Mpi3-rma] Available size and number of shared memory windows with MPI_WIN_ALLOCATE_SHARED

Tue Jun 4 01:49:40 CDT 2013

I'm not familiar with any formal limits, it's just a practical issue
of what the OS can do; I don't know what you're OS configuration is so
it's impossible to answer any questions along these lines.  Sys5 has
some issues i.e. limits but I don't care to find out what they are
because POSIX shm is so much better (and standardized)(and modern).

As an example of a restrictive platform, on BGQ, the POSIX interface
is perfectly supported but the user has to specific the size at job
launch and the default is 64 MB.  MPI+OpenMP has no such restrictions,
obviously, and is fully dynamic, unlike POSIX.  Alternatively, you can
skip POSIX and do interprocess load-store (at the expense of memory
protection), which I suppose is how MPI_WIN_ALLOCATE_SHARED could be
implemented.

In the general case, if you use MPI+Pthreads where you would use
Pthreads+nothing like processes+POSIX shm, you'd have private stacks
just like processes, but they'd have be statically allocated (at
pthread_create), perhaps with a lower default than OS processes (via
ulimit -s).  Is this the programming model you're trying to achieve
(private by default ala processes rather than public/shared by default
ala threads)?

If you are thinking of MPI+MPI instead of MPI+OpenMP because you don't
like OpenMP, why don't you try Pthreads (yes, you'll need a Fortran
wrapper but there's at least one on the Internet already)?  I suppose
Pthreads puts you in MPI_THREAD_MULTIPLE instead of MPI_THREAD_SINGLE
as with MPI+MPI, but the performance differences there are almost
entirely due to the horrible abuse of fat locks in most/all non-BGQ
implementations rather than something fundamental.

The other way to get rid of OpenMP is to use F08 coarrays within the
node, but I'm sure you know about that.

I guess I just don't understand what you problem you're trying to solve.

Jeff

On Tue, Jun 4, 2013 at 12:32 AM, Rolf Rabenseifner <rabenseifner at hlrs.de> wrote:
> Jeff and all,
>
> I'm not familiar with Limits of POSIX shared memory.
>
> Are there significant limits?
> Or is it possible to use most of the physical memory as shared memory?
>
> Best regards
> Rolf
>
> ----- Original Message -----
>> From: "Jeff Hammond" <jeff.science at gmail.com>
>> To: "MPI 3.0 Remote Memory Access working group" <mpi3-rma at lists.mpi-forum.org>
>> Sent: Monday, June 3, 2013 10:42:37 PM
>> Subject: Re: [Mpi3-rma] Available size and number of shared memory windows with MPI_WIN_ALLOCATE_SHARED
>> This is totally platform specific. MPICH on BGQ is different than on
>> Linux, for example. This question is thus mostly unanswerable. Your
>> best bet is to consider the POSIX shared memory spec and assume that
>> MPI uses it.
>>
>> Jeff
>>
>> Sent from my iPhone
>>
>> On Jun 3, 2013, at 3:14 PM, Rolf Rabenseifner <rabenseifner at hlrs.de>
>> wrote:
>>
>> > Dear implementers of MPI-3.0 MPI_WIN_ALLOCATE_SHARED,
>> >
>> > my question is not about the Interface, but on the implementations:
>> >
>> > If a set of MPI processes within a shared memory node and
>> > Communicator
>> > wants to allocate a shared memory window:
>> >
>> > - Are there similar limits as if a set of threads wants to allocate
>> >   memory and use it as global Memory by all threads, i.e.,
>> >   is it possible to allocate most physical
>> >   memory with MPI_WIN_ALLOCATE_SHARED?
>> >
>> > - Is there an additional restriction on the number of shared memory
>> >   windows?
>> >
>> > - Is there an additional restriction if one process defines the
>> >   whole size as window size and all other processes within
>> >   the SMP node use size=0?
>> >
>> > - Do you know other restrictions if I want to substitute
>> >   OpenMP (not OpenMPI!) by shared memory MPI programming,
>> >   i.e. hybrid MPI+MPI instead of hybrid MPI+OpenMP?
>> >
>> > Is your answer generally valid for mpich and OpenMPI and must
>> > I expect some additional restrictions on some vendor's MPIs?
>> >
>> > It is about the real implementations, e.g. in mpich and OpenMPI,
>> > but I expect that our MPI-3 RMA working group knows the answer.
>> >
>> > Best regards
>> > Rolf
>> >
>> >
>> > --
>> > Dr. Rolf Rabenseifner . . . . . . . . . .. email
>> > rabenseifner at hlrs.de
>> > High Performance Computing Center (HLRS) . phone
>> > ++49(0)711/685-65530
>> > University of Stuttgart . . . . . . . . .. fax ++49(0)711 /
>> > 685-65832
>> > Head of Dpmt Parallel Computing . . .
>> > www.hlrs.de/people/rabenseifner
>> > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room
>> > 1.307)
>> > _______________________________________________
>> > mpi3-rma mailing list
>> > mpi3-rma at lists.mpi-forum.org
>> > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma
>> _______________________________________________
>> mpi3-rma mailing list
>> mpi3-rma at lists.mpi-forum.org
>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma
>
> --
> Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner at hlrs.de
> High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530
> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832
> Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner
> Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)
> _______________________________________________
> mpi3-rma mailing list
> mpi3-rma at lists.mpi-forum.org
> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma

-- 
Jeff Hammond
Argonne Leadership Computing Facility
University of Chicago Computation Institute
jhammond at alcf.anl.gov / (630) 252-5381
http://www.linkedin.com/in/jeffhammond
https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond
ALCF docs: http://www.alcf.anl.gov/user-guides