[mpiwg-p2p] Ordering of P2P messages in multithreaded applications

Joachim Protze protze at itc.rwth-aachen.de
Thu Nov 15 12:01:46 CST 2018


Hi Dan,

I look at this from the perspective of an correctness/debugging tool. 
Therefore I tried to understand which application behavior should be 
flagged yellow/red.
As I understand now, expecting anything around this paragraph can lead 
to portability issues and should be reported to the programmer :)

Thanks
Joachi,

On 11/15/18 6:57 PM, HOLMES Daniel wrote:
> Hi Joachim,
> 
> That is technically correct (although, as Jeff points out, not at all 
> ideal). In practice, you will probably get the behaviour you expect 
> (i.e. ordering) because of the debate over the meaning of this text - as 
> demonstrated by the comment from Pavan.
> 
> Disambiguating this paragraph is on the to-do list for the 
> point-to-point working group. Thanks for bringing this up again and 
> thereby raising its priority for us.
> 
> Cheers,
> Dan.
>> Dr Daniel Holmes PhD
> Applications Consultant in HPC Research
> d.holmes at epcc.ed.ac.uk <mailto:d.holmes at epcc.ed.ac.uk>
> Phone: +44 (0) 131 651 3465
> Mobile: +44 (0) 7940 524 088
> Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, 
> EH8 9BT
>> The University of Edinburgh is a charitable body, registered in 
> Scotland, with registration number SC005336.
>> 
>> On 15 Nov 2018, at 11:19, Joachim Protze via mpiwg-p2p 
>> <mpiwg-p2p at lists.mpi-forum.org <mailto:mpiwg-p2p at lists.mpi-forum.org>> 
>> wrote:
>>
>> I think, I was once again confused by "may not" in the cited 
>> paragraph. As a non-native speaker this hits me from time to time.
>>
>> So if I understand it right now, the paragraph says that even if the 
>> threading semantics provide an ordering, the operations are still 
>> logically concurrent and have no ordering.
>>
>> Thanks
>> Joachim
>>
>> On 11/15/18 6:09 PM, Jeff Hammond wrote:
>>> Dan has convinced me that the MPI standard is terrible and, while my 
>>> original interpretation is what we want and which is consistent with 
>>> the principle of least surprise, it is not guaranteed by the 
>>> following text.
>>> Per our discussion, there are a few options:
>>> 1) make all MPI_Send logically concurrent, even on a single thread. 
>>>  this will break stuff and make people sad.
>>> 2) force MPI to order injection <somehow>, which might for some 
>>> implementations to add more memory ordering on the send path than 
>>> they want, particularly if they do not have a TSO memory model.
>>> 3) add something like MPI_Win_sync that logically orders sends from 
>>> multiple threads explicitly.
>>> 4) add MPI_THREAD_SERIALIZED_WITH_EXTRA_SAUCE that does the 
>>> equivalent of 2 or 3 and thus doesn't cause a performance regression 
>>> in MPI_THREAD_SERIALIZED.
>>> Jeff
>>> /If a process has a single thread of execution, then any two 
>>> communications executed by this process are ordered. On the other 
>>> hand, if the process is multithreaded, then the semantics of thread 
>>> execution may not define a relative order between two send operations 
>>> executed by two distinct threads. The operations are logically 
>>> concurrent, even if one physically precedes the other. In such a 
>>> case, the two messages sent can be received in any order. Similarly, 
>>> if two receive operations that are logically concurrent receive two 
>>> successively sent messages, then the two messages can match the two 
>>> receives in either order. /
>>> On Thu, Nov 15, 2018 at 10:55 AM Balaji, Pavan via mpiwg-p2p 
>>> <mpiwg-p2p at lists.mpi-forum.org 
>>> <mailto:mpiwg-p2p at lists.mpi-forum.org><mailto:mpiwg-p2p at lists.mpi-forum.org>> 
>>> wrote:
>>>    Dan,
>>>    The matching *is* ordered in this case.  So the program will print 0
>>>    followed by 1.
>>>    MPI does not order delivery of the actual data, but the first
>>>    message is guaranteed to go into the first buffer.  If the second
>>>    message ends up going first, the MPI implementation will need to
>>>    buffer it.
>>>       — Pavan
>>>    Sent from my iPhone
>>>    On Nov 15, 2018, at 7:56 AM, HOLMES Daniel via mpiwg-p2p
>>>    <mpiwg-p2p at lists.mpi-forum.org <mailto:mpiwg-p2p at lists.mpi-forum.org>
>>>    <mailto:mpiwg-p2p at lists.mpi-forum.org>> wrote:
>>>>    Hi Joachim,
>>>>
>>>>    There is no guarantee of ordering between the two sends because
>>>>    they are logically concurrent. If they were issued on the same
>>>>    thread then MPI guarantees delivery order will be identical to the
>>>>    sequential issuing order.
>>>>
>>>>    Many MPI libraries are very likely to deliver these messages "in
>>>>    order”, that is, the first one to be called chronologically at the
>>>>    sender process is likely to leave first and therefore likely to
>>>>    arrive first. Interleaving execution of the sending threads may
>>>>    change the issuing order on the network and out-of-order networks
>>>>    may change the order of arrival.
>>>>
>>>>    On the other hand, if an MPI implementation is internally using
>>>>    sequence numbers (or a similar mechanism) to enforce ordering for
>>>>    the same-thread case, then it may also (incidentally) reconstruct
>>>>    the issuing order for this case. However, you cannot rely on this
>>>>    behaviour being portable from system to system or from MPI library
>>>>    to MPI library.
>>>>
>>>>    If you wish to enforce a particular ordering of these messages,
>>>>    then you can use tags to differentiate each from the other. There
>>>>    is an argument for always using tags in this type of situation to
>>>>    increase program readability.
>>>>
>>>>    Cheers,
>>>>    Dan.
>>>>    —
>>>>    Dr Daniel Holmes PhD
>>>>    Applications Consultant in HPC Research
>>>> d.holmes at epcc.ed.ac.uk 
>>>> <mailto:d.holmes at epcc.ed.ac.uk><mailto:d.holmes at epcc.ed.ac.uk>
>>>>    Phone: +44 (0) 131 651 3465
>>>>    Mobile: +44 (0) 7940 524 088
>>>>    Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area,
>>>>    Edinburgh, EH8 9BT
>>>>    —
>>>>    The University of Edinburgh is a charitable body, registered in
>>>>    Scotland, with registration number SC005336.
>>>>    —
>>>>
>>>>>    On 15 Nov 2018, at 04:16, Joachim Protze via mpiwg-p2p
>>>>>    <mpiwg-p2p at lists.mpi-forum.org 
>>>>> <mailto:mpiwg-p2p at lists.mpi-forum.org>
>>>>>    <mailto:mpiwg-p2p at lists.mpi-forum.org>> wrote:
>>>>>
>>>>>    Hi all,
>>>>>
>>>>>    I have a question on the "Semantics of Point-to-Point
>>>>>    Communication" in a multithreaded context.
>>>>>
>>>>>    For me the situation for the code below is not clear, especially
>>>>>    with respect to the paragraph in MPI-3.1 p.41, l.10-17 :
>>>>>
>>>>>
>>>>>    void test(int rank) {
>>>>>     int msg = 0;
>>>>>     if (rank == 0) {
>>>>>    #pragma omp parallel num_threads(2)
>>>>>    #pragma omp critical
>>>>>       {
>>>>>         MPI_Send(&msg, 1, MPI_INT, 1, 42, MPI_COMM_WORLD);
>>>>>         msg++;
>>>>>       }
>>>>>     } else if (rank == 1) {
>>>>>       MPI_Recv(&msg, 1, MPI_INT, 0, 42, MPI_COMM_WORLD,
>>>>>    MPI_STATUS_IGNORE);
>>>>>       printf("Received %i\n", msg);
>>>>>       MPI_Recv(&msg, 1, MPI_INT, 0, 42, MPI_COMM_WORLD,
>>>>>    MPI_STATUS_IGNORE);
>>>>>       printf("Received %i\n", msg);
>>>>>     }
>>>>>    }
>>>>>
>>>>>    Two threads on the first process send a message, the first thread
>>>>>    sends 0, the second thread send 1. From OpenMP semantics, the
>>>>>    first send happens before the second send.
>>>>>
>>>>>    Is there a guarantee, that the other process receives the 0 first?
>>>>>
>>>>>    Thanks,
>>>>>    Joachim
>>>>>
>>>>>
>>>>>    --     Dipl.-Inf. Joachim Protze
>>>>>
>>>>>    IT Center
>>>>>    Group: High Performance Computing
>>>>>    Division: Computational Science and Engineering
>>>>>    RWTH Aachen University
>>>>>    Seffenter Weg 23
>>>>>    D 52074  Aachen (Germany)
>>>>>    Tel: +49 241 80- 24765
>>>>>    Fax: +49 241 80-624765
>>>>> protze at itc.rwth-aachen.de 
>>>>> <mailto:protze at itc.rwth-aachen.de><mailto:protze at itc.rwth-aachen.de>
>>>>> www.itc.rwth-aachen.de 
>>>>> <http://www.itc.rwth-aachen.de/><http://www.itc.rwth-aachen.de 
>>>>> <http://www.itc.rwth-aachen.de/>>
>>>>>
>>>>>    _______________________________________________
>>>>>    mpiwg-p2p mailing list
>>>>> mpiwg-p2p at lists.mpi-forum.org 
>>>>> <mailto:mpiwg-p2p at lists.mpi-forum.org><mailto:mpiwg-p2p at lists.mpi-forum.org>
>>>>> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-p2p
>>>>
>>>>    The University of Edinburgh is a charitable body, registered in
>>>>    Scotland, with registration number SC005336.
>>>>    _______________________________________________
>>>>    mpiwg-p2p mailing list
>>>> mpiwg-p2p at lists.mpi-forum.org 
>>>> <mailto:mpiwg-p2p at lists.mpi-forum.org><mailto:mpiwg-p2p at lists.mpi-forum.org>
>>>> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-p2p
>>>    _______________________________________________
>>>    mpiwg-p2p mailing list
>>> mpiwg-p2p at lists.mpi-forum.org 
>>> <mailto:mpiwg-p2p at lists.mpi-forum.org><mailto:mpiwg-p2p at lists.mpi-forum.org>
>>> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-p2p
>>> --
>>> Jeff Hammond
>>> jeff.science at gmail.com 
>>> <mailto:jeff.science at gmail.com><mailto:jeff.science at gmail.com>
>>> http://jeffhammond.github.io/
>>
>>
>> --
>> Dipl.-Inf. Joachim Protze
>>
>> IT Center
>> Group: High Performance Computing
>> Division: Computational Science and Engineering
>> RWTH Aachen University
>> Seffenter Weg 23
>> D 52074  Aachen (Germany)
>> Tel: +49 241 80- 24765
>> Fax: +49 241 80-624765
>> protze at itc.rwth-aachen.de <mailto:protze at itc.rwth-aachen.de>
>> www.itc.rwth-aachen.de <http://www.itc.rwth-aachen.de/>
>>
>> _______________________________________________
>> mpiwg-p2p mailing list
>> mpiwg-p2p at lists.mpi-forum.org <mailto:mpiwg-p2p at lists.mpi-forum.org>
>> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-p2p
> 
> 
> The University of Edinburgh is a charitable body, registered in
> Scotland, with registration number SC005336.
> 


-- 
Dipl.-Inf. Joachim Protze

IT Center
Group: High Performance Computing
Division: Computational Science and Engineering
RWTH Aachen University
Seffenter Weg 23
D 52074  Aachen (Germany)
Tel: +49 241 80- 24765
Fax: +49 241 80-624765
protze at itc.rwth-aachen.de
www.itc.rwth-aachen.de

-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4915 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.mpi-forum.org/pipermail/mpiwg-p2p/attachments/20181115/985d68f6/attachment-0001.p7s>


More information about the mpiwg-p2p mailing list