Subject: RE: [Fwd: Re: Helper threads proposal] |
From: James Beyer <beyerj@cray.com> |
Date: Mon, 14 Mar 2011 12:38:32 -0500 |
To: Howard Pritchard <howardp@cray.com> |
Here are a couple of comments. 1) Up to section 12.4.4 everything made sense and was useful. 2) Section 12.4.4 looks really useful for the pthread programmer. 3) Examples 12.3 and 12.4 are bad, omp_get_num_threads, should never be used outside of a parallel region, the value returned is going to be one. The rationale behind section 12.4.4 might be better explained using pthreads. However, even for OpenMP codes this makes sense assuming that there really is parallel work available in the MPI calls. One of the biggest problems we see with hybrid codes is load balancing, in the threads. I don't know enough about the MPI library to know if there is parallelism to exploit in the calls, but if there is, this definitely one way to exploit that parallelism. Having said this, I agree, that this solution seems geared heavily towards systems that do not support nested parallel regions, or that do not have a mechanism to idle threads. After everything is said and done. I would recommend something like this over nested parallelism just because it give the programmer more control over how hardware resources are allocated to threads. Every time a library creates a thread it seems like it screws something up in the users code, at least that has been my experience in the last few years. The OpenMP aware solutions to this problem do not help codes that are written using pure pthreads, however, a system written, like this one, that works with pthreads, should work equally well with most openmp implementations. There are a few OpenMP implementations based on user threads which may behave differently. I hope this helps, and is not to late. james
mailto:howardp@cray.com] Sent: Monday, February 28, 2011 2:17 PM To: James Beyer Subject: Re: [Fwd: Re: Helper threads proposal] Hi Jim, 2-4 weeks would be fine. The next MPI forum meeting is last week of March. Its not urgent. I just think I'm outside my area of expertise to do more than say something doesn't seem right - which is basically what I said at the last MPI forum meeting. I was also suspicious because it does seem to be ibm/bluegene motivated. Howard James Beyer wrote:-----Original Message----- From: Howard Pritchard [Ok, how soon do you need this? jamesmailto:howardp@cray.com] Sent: Monday, February 28, 2011 2:04 PM To: James Beyer Cc: David Oehmke; Gary Elsesser; Terry Greyzck Subject: Re: [Fwd: Re: Helper threads proposal] Hi James, I think maybe it would be better if one of you could take a little time to review the proposed extensions to mpi to allow for mpi/openmp interop improvements. You can start at page 15 thru page 18. I'd really appreciate it. I've attached the changes document. Thanks, Howard James Beyer wrote:-----Original Message----- From: Howard Pritchard [<This is a bit of a ramble so please bear with me.> Ok, so why not use openmp tasks? This allows the system to usewhatever
resources it can find in a parallel region to speed-up any computation. This would immediately address codes like:if (my_thread == 0) { MPI_Send(big_message); } #pragma omp barrierIf MPI_Send contained tasks, then any thread parked in the omp barrier
would help with the MPI_Send. If there are no threads to help, then
the
work gets done by the main thread. There are a couple of problems with this approach. First task overhead is not insignificant, so the programmer would likely only want to generate tasks when there arelikely
to be helper threads. Second, the programmer probably does not want to have to link in the OpenMP runtime every time they run an MPI programif
they are not building a hybrid application.
The nice things about tasks is that the programmer could write
something
like, assuming non-colliding sends:
if ( my_thread==0) { MPI_Send(big_message1); } else if ( my_thread == 1 ) { MPI_Send(big_message2); } #pragma omp barrier Now all of the threads will work together to get both messages sent asfast as possible, however that is done.
Or if they really want to use old school openmp worksharing
constructs,
they will simply need only one additional argument to the routines to
tell
the library if the call should workshare or not, this is probably a new interface.There is just one caveat, because OpenMP does not have an ABI, once an
MPI library contains any OpenMP it will have to be compiled the
compiler
that is used for the users program.
I think what they want is something like: MPI_Send(): Old call MPI_Send_omp_workshare(): New call which contains an OpenMPworksharing construct.
So, from what I can determine there are two reasons why MPI might want
to expose something to the programmer:
1) Parallelism status: This is handled by the openmp runtime, int
omp_in_parallel();
2) worksharing status: Is the parallel team in a worksharing
construct?
This is not queryable.
3) redundancy status: Is the call redundant or not, ie worksharing
is
allowed. Only the programmer knows this.
Number 1 allows the library to determine if it should try to start a
new
parallel region.
Number 2 would allow the library to try to reuse a time for openmp
loop
constructs.
Number 3 tells the library if the library can try to us worksharing,
see
number 2.
MPI_Send can easily handle creating new parallel regions when
possible.
However, a new interface is definitely needed if they want to provide
the
programmer with the controls required to make optimal use of an
existing
parallel team.
Yes a system that uses soft-threads could possible use nested
parallelism to extract the parallelism desired, however, there is only
one
system that I know of that uses soft-threads and it is a research
system.
So, if MPI wants to play really nice with OpenMP and all other
threading
systems, it would actually be useful for it to provide worksharing versions of routines. These routines should not be written to use any particular system but should rather be written in a generic way toallow
for speedups whenever multiple threads call into the routine. This
would
also keep the system more portable.
Jamesmailto:howardp@cray.com] Sent: Monday, February 28, 2011 12:18 PM To: David Oehmke Cc: James Beyer; Gary Elsesser; Terry Greyzck Subject: Re: [Fwd: Re: Helper threads proposal] Hi David, They are wanting to provide 'hooks' for the app developer to tell MPI that it can use some of its omp threads within MPI. For example if (my_thread == 0) { MPI_Send(big message); } else { call_new_mpi_3_function_to_mpi_i_can_be_invoked_in_a omp_loop_inside_mpi } #pragma omp barrier That's the gist of it. The current status which they want to deal with better is if (my_thread == 0) { MPI_Send(big_message); } #pragma omp barrier where most of the threads are parked in the barrier instead of potentially helping out in the MPI implementation. Actually, I think the idea is that somehow the MPI can get control of the omp runtime generated threads and use them like pthreads inside the MPI implementation. At the forum, I asked them if they'd consulted with any compiler groups (or the OpenMP folks) and they said no. It seemed to me that a smart openmp runtime could handle this without these hooks. For example, rather than having a simple 1-thread per physical execution unit (core), the openmp runtime would have multiple threads in a pool. It would track how many execution units were currently in use when it hit a omp region. If there were available execution units then the loop would be run in parallel using either parked threads, or newly created threads. In the above case, threads that had hit the barrier and gotten parked (after some time of spin waiting), would release their hold on their execution units, allowing other parallel regions in the code invoked by my_thread == 0 to run in parallel. I didn't say all this, but figured a really smart omp runtime could do this sort of thing if requested by the user. I was sceptical once they said they'd not consulted with compiler people before coming up with this idea - which I do think is motivated by IBM. Howard David Oehmke wrote:-----Original Message----- From: Howard Pritchard [mailto:tdg@cray.com] Sent: Monday, February 28, 2011 11:14 AM To: Howard Pritchard Cc: James Beyer; David Oehmke; Gary Elsesser Subject: Re: [Fwd: Re: Helper threads proposal] Hi Howard - I am indeed forwarding this to our OpenMP experts for theirCan you give us a summary of what they are proposing? David -----Original Message----- From: Terry Greyzck [comments...
-- Terry On 2/28/2011 11:07 AM, Howard Pritchard wrote:Hi Terry, If you have time, or maybe the omp expert in your group could answer some questions from Pavan (argonne). At the latest MPI forum meeting some folks were proposing a sort of 'helper' set of functions to 'help' an omp compiler parallelize both an app and certain functions within an mpi compiler. My argument was that a good omp compiler supporting nested parallelism would not need what they were proposing. I think the interface was partly motivated by some investigations on ibm bluegene using ibm's compiler. You can respond to me and then I'll feed this back to the mpi forum discussion group. Thanks, Howard-- Howard Pritchard Software Engineering Cray, Inc.-- Howard Pritchard Software Engineering Cray, Inc.-- Howard Pritchard Software Engineering Cray, Inc.