[Mpi3-rma] Available size and number of shared memory windows with MPI_WIN_ALLOCATE_SHARED

Tue Jun 4 03:40:22 CDT 2013

Jeff,

> I guess I just don't understand what you problem you're trying to solve.

My problem is that I want to understand how the new combination of
MPI_COMM_SPLIT_TYPE and MPI_WIN_ALLOCATE_SHARED can be used
and which restrictions there are.

In principle, to stay within one programming model is attractive.
Therefore, I want to learn what are the limits of MPI_WIN_ALLOCATE_SHARED.

It is not the question whether I do not like OpenMP.
There are senarios with restrictions that do not allow OpenMP,
e.g., the use of non-thread-safe library routines.
Therefore, I'm looking for general alternatives to MPI+OpenMP
and pure MPI on clusters of SMP nodes.

Best regards
Rolf  

----- Original Message -----
> From: "Jeff Hammond" <jhammond at alcf.anl.gov>
> To: "MPI 3.0 Remote Memory Access working group" <mpi3-rma at lists.mpi-forum.org>
> Sent: Tuesday, June 4, 2013 8:49:40 AM
> Subject: Re: [Mpi3-rma] Available size and number of shared memory windows with MPI_WIN_ALLOCATE_SHARED
> I'm not familiar with any formal limits, it's just a practical issue
> of what the OS can do; I don't know what you're OS configuration is so
> it's impossible to answer any questions along these lines. Sys5 has
> some issues i.e. limits but I don't care to find out what they are
> because POSIX shm is so much better (and standardized)(and modern).
> 
> As an example of a restrictive platform, on BGQ, the POSIX interface
> is perfectly supported but the user has to specific the size at job
> launch and the default is 64 MB. MPI+OpenMP has no such restrictions,
> obviously, and is fully dynamic, unlike POSIX. Alternatively, you can
> skip POSIX and do interprocess load-store (at the expense of memory
> protection), which I suppose is how MPI_WIN_ALLOCATE_SHARED could be
> implemented.
> 
> In the general case, if you use MPI+Pthreads where you would use
> Pthreads+nothing like processes+POSIX shm, you'd have private stacks
> just like processes, but they'd have be statically allocated (at
> pthread_create), perhaps with a lower default than OS processes (via
> ulimit -s). Is this the programming model you're trying to achieve
> (private by default ala processes rather than public/shared by default
> ala threads)?
> 
> If you are thinking of MPI+MPI instead of MPI+OpenMP because you don't
> like OpenMP, why don't you try Pthreads (yes, you'll need a Fortran
> wrapper but there's at least one on the Internet already)? I suppose
> Pthreads puts you in MPI_THREAD_MULTIPLE instead of MPI_THREAD_SINGLE
> as with MPI+MPI, but the performance differences there are almost
> entirely due to the horrible abuse of fat locks in most/all non-BGQ
> implementations rather than something fundamental.
> 
> The other way to get rid of OpenMP is to use F08 coarrays within the
> node, but I'm sure you know about that.
> 
> I guess I just don't understand what you problem you're trying to
> solve.
> 
> Jeff
> 
> On Tue, Jun 4, 2013 at 12:32 AM, Rolf Rabenseifner
> <rabenseifner at hlrs.de> wrote:
> > Jeff and all,
> >
> > I'm not familiar with Limits of POSIX shared memory.
> >
> > Are there significant limits?
> > Or is it possible to use most of the physical memory as shared
> > memory?
> >
> > Best regards
> > Rolf
> >
> > ----- Original Message -----
> >> From: "Jeff Hammond" <jeff.science at gmail.com>
> >> To: "MPI 3.0 Remote Memory Access working group"
> >> <mpi3-rma at lists.mpi-forum.org>
> >> Sent: Monday, June 3, 2013 10:42:37 PM
> >> Subject: Re: [Mpi3-rma] Available size and number of shared memory
> >> windows with MPI_WIN_ALLOCATE_SHARED
> >> This is totally platform specific. MPICH on BGQ is different than
> >> on
> >> Linux, for example. This question is thus mostly unanswerable. Your
> >> best bet is to consider the POSIX shared memory spec and assume
> >> that
> >> MPI uses it.
> >>
> >> Jeff
> >>
> >> Sent from my iPhone
> >>
> >> On Jun 3, 2013, at 3:14 PM, Rolf Rabenseifner
> >> <rabenseifner at hlrs.de>
> >> wrote:
> >>
> >> > Dear implementers of MPI-3.0 MPI_WIN_ALLOCATE_SHARED,
> >> >
> >> > my question is not about the Interface, but on the
> >> > implementations:
> >> >
> >> > If a set of MPI processes within a shared memory node and
> >> > Communicator
> >> > wants to allocate a shared memory window:
> >> >
> >> > - Are there similar limits as if a set of threads wants to
> >> > allocate
> >> >   memory and use it as global Memory by all threads, i.e.,
> >> >   is it possible to allocate most physical
> >> >   memory with MPI_WIN_ALLOCATE_SHARED?
> >> >
> >> > - Is there an additional restriction on the number of shared
> >> > memory
> >> >   windows?
> >> >
> >> > - Is there an additional restriction if one process defines the
> >> >   whole size as window size and all other processes within
> >> >   the SMP node use size=0?
> >> >
> >> > - Do you know other restrictions if I want to substitute
> >> >   OpenMP (not OpenMPI!) by shared memory MPI programming,
> >> >   i.e. hybrid MPI+MPI instead of hybrid MPI+OpenMP?
> >> >
> >> > Is your answer generally valid for mpich and OpenMPI and must
> >> > I expect some additional restrictions on some vendor's MPIs?
> >> >
> >> > It is about the real implementations, e.g. in mpich and OpenMPI,
> >> > but I expect that our MPI-3 RMA working group knows the answer.
> >> >
> >> > Best regards
> >> > Rolf
> >> >
> >> >
> >> > --
> >> > Dr. Rolf Rabenseifner . . . . . . . . . .. email
> >> > rabenseifner at hlrs.de
> >> > High Performance Computing Center (HLRS) . phone
> >> > ++49(0)711/685-65530
> >> > University of Stuttgart . . . . . . . . .. fax ++49(0)711 /
> >> > 685-65832
> >> > Head of Dpmt Parallel Computing . . .
> >> > www.hlrs.de/people/rabenseifner
> >> > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room
> >> > 1.307)
> >> > _______________________________________________
> >> > mpi3-rma mailing list
> >> > mpi3-rma at lists.mpi-forum.org
> >> > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma
> >> _______________________________________________
> >> mpi3-rma mailing list
> >> mpi3-rma at lists.mpi-forum.org
> >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma
> >
> > --
> > Dr. Rolf Rabenseifner . . . . . . . . . .. email
> > rabenseifner at hlrs.de
> > High Performance Computing Center (HLRS) . phone
> > ++49(0)711/685-65530
> > University of Stuttgart . . . . . . . . .. fax ++49(0)711 /
> > 685-65832
> > Head of Dpmt Parallel Computing . . .
> > www.hlrs.de/people/rabenseifner
> > Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room
> > 1.307)
> > _______________________________________________
> > mpi3-rma mailing list
> > mpi3-rma at lists.mpi-forum.org
> > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma
> 
> 
> 
> --
> Jeff Hammond
> Argonne Leadership Computing Facility
> University of Chicago Computation Institute
> jhammond at alcf.anl.gov / (630) 252-5381
> http://www.linkedin.com/in/jeffhammond
> https://wiki.alcf.anl.gov/parts/index.php/User:Jhammond
> ALCF docs: http://www.alcf.anl.gov/user-guides
> _______________________________________________
> mpi3-rma mailing list
> mpi3-rma at lists.mpi-forum.org
> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-rma

-- 
Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner at hlrs.de
High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530
University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832
Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner
Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)