[mpiwg-p2p] Ordering of P2P messages in multithreaded applications
Joachim Protze
protze at itc.rwth-aachen.de
Thu Nov 15 12:01:46 CST 2018
Hi Dan,
I look at this from the perspective of an correctness/debugging tool.
Therefore I tried to understand which application behavior should be
flagged yellow/red.
As I understand now, expecting anything around this paragraph can lead
to portability issues and should be reported to the programmer :)
Thanks
Joachi,
On 11/15/18 6:57 PM, HOLMES Daniel wrote:
> Hi Joachim,
>
> That is technically correct (although, as Jeff points out, not at all
> ideal). In practice, you will probably get the behaviour you expect
> (i.e. ordering) because of the debate over the meaning of this text - as
> demonstrated by the comment from Pavan.
>
> Disambiguating this paragraph is on the to-do list for the
> point-to-point working group. Thanks for bringing this up again and
> thereby raising its priority for us.
>
> Cheers,
> Dan.
> —
> Dr Daniel Holmes PhD
> Applications Consultant in HPC Research
> d.holmes at epcc.ed.ac.uk <mailto:d.holmes at epcc.ed.ac.uk>
> Phone: +44 (0) 131 651 3465
> Mobile: +44 (0) 7940 524 088
> Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh,
> EH8 9BT
> —
> The University of Edinburgh is a charitable body, registered in
> Scotland, with registration number SC005336.
> —
>
>> On 15 Nov 2018, at 11:19, Joachim Protze via mpiwg-p2p
>> <mpiwg-p2p at lists.mpi-forum.org <mailto:mpiwg-p2p at lists.mpi-forum.org>>
>> wrote:
>>
>> I think, I was once again confused by "may not" in the cited
>> paragraph. As a non-native speaker this hits me from time to time.
>>
>> So if I understand it right now, the paragraph says that even if the
>> threading semantics provide an ordering, the operations are still
>> logically concurrent and have no ordering.
>>
>> Thanks
>> Joachim
>>
>> On 11/15/18 6:09 PM, Jeff Hammond wrote:
>>> Dan has convinced me that the MPI standard is terrible and, while my
>>> original interpretation is what we want and which is consistent with
>>> the principle of least surprise, it is not guaranteed by the
>>> following text.
>>> Per our discussion, there are a few options:
>>> 1) make all MPI_Send logically concurrent, even on a single thread.
>>> this will break stuff and make people sad.
>>> 2) force MPI to order injection <somehow>, which might for some
>>> implementations to add more memory ordering on the send path than
>>> they want, particularly if they do not have a TSO memory model.
>>> 3) add something like MPI_Win_sync that logically orders sends from
>>> multiple threads explicitly.
>>> 4) add MPI_THREAD_SERIALIZED_WITH_EXTRA_SAUCE that does the
>>> equivalent of 2 or 3 and thus doesn't cause a performance regression
>>> in MPI_THREAD_SERIALIZED.
>>> Jeff
>>> /If a process has a single thread of execution, then any two
>>> communications executed by this process are ordered. On the other
>>> hand, if the process is multithreaded, then the semantics of thread
>>> execution may not define a relative order between two send operations
>>> executed by two distinct threads. The operations are logically
>>> concurrent, even if one physically precedes the other. In such a
>>> case, the two messages sent can be received in any order. Similarly,
>>> if two receive operations that are logically concurrent receive two
>>> successively sent messages, then the two messages can match the two
>>> receives in either order. /
>>> On Thu, Nov 15, 2018 at 10:55 AM Balaji, Pavan via mpiwg-p2p
>>> <mpiwg-p2p at lists.mpi-forum.org
>>> <mailto:mpiwg-p2p at lists.mpi-forum.org><mailto:mpiwg-p2p at lists.mpi-forum.org>>
>>> wrote:
>>> Dan,
>>> The matching *is* ordered in this case. So the program will print 0
>>> followed by 1.
>>> MPI does not order delivery of the actual data, but the first
>>> message is guaranteed to go into the first buffer. If the second
>>> message ends up going first, the MPI implementation will need to
>>> buffer it.
>>> — Pavan
>>> Sent from my iPhone
>>> On Nov 15, 2018, at 7:56 AM, HOLMES Daniel via mpiwg-p2p
>>> <mpiwg-p2p at lists.mpi-forum.org <mailto:mpiwg-p2p at lists.mpi-forum.org>
>>> <mailto:mpiwg-p2p at lists.mpi-forum.org>> wrote:
>>>> Hi Joachim,
>>>>
>>>> There is no guarantee of ordering between the two sends because
>>>> they are logically concurrent. If they were issued on the same
>>>> thread then MPI guarantees delivery order will be identical to the
>>>> sequential issuing order.
>>>>
>>>> Many MPI libraries are very likely to deliver these messages "in
>>>> order”, that is, the first one to be called chronologically at the
>>>> sender process is likely to leave first and therefore likely to
>>>> arrive first. Interleaving execution of the sending threads may
>>>> change the issuing order on the network and out-of-order networks
>>>> may change the order of arrival.
>>>>
>>>> On the other hand, if an MPI implementation is internally using
>>>> sequence numbers (or a similar mechanism) to enforce ordering for
>>>> the same-thread case, then it may also (incidentally) reconstruct
>>>> the issuing order for this case. However, you cannot rely on this
>>>> behaviour being portable from system to system or from MPI library
>>>> to MPI library.
>>>>
>>>> If you wish to enforce a particular ordering of these messages,
>>>> then you can use tags to differentiate each from the other. There
>>>> is an argument for always using tags in this type of situation to
>>>> increase program readability.
>>>>
>>>> Cheers,
>>>> Dan.
>>>> —
>>>> Dr Daniel Holmes PhD
>>>> Applications Consultant in HPC Research
>>>> d.holmes at epcc.ed.ac.uk
>>>> <mailto:d.holmes at epcc.ed.ac.uk><mailto:d.holmes at epcc.ed.ac.uk>
>>>> Phone: +44 (0) 131 651 3465
>>>> Mobile: +44 (0) 7940 524 088
>>>> Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area,
>>>> Edinburgh, EH8 9BT
>>>> —
>>>> The University of Edinburgh is a charitable body, registered in
>>>> Scotland, with registration number SC005336.
>>>> —
>>>>
>>>>> On 15 Nov 2018, at 04:16, Joachim Protze via mpiwg-p2p
>>>>> <mpiwg-p2p at lists.mpi-forum.org
>>>>> <mailto:mpiwg-p2p at lists.mpi-forum.org>
>>>>> <mailto:mpiwg-p2p at lists.mpi-forum.org>> wrote:
>>>>>
>>>>> Hi all,
>>>>>
>>>>> I have a question on the "Semantics of Point-to-Point
>>>>> Communication" in a multithreaded context.
>>>>>
>>>>> For me the situation for the code below is not clear, especially
>>>>> with respect to the paragraph in MPI-3.1 p.41, l.10-17 :
>>>>>
>>>>>
>>>>> void test(int rank) {
>>>>> int msg = 0;
>>>>> if (rank == 0) {
>>>>> #pragma omp parallel num_threads(2)
>>>>> #pragma omp critical
>>>>> {
>>>>> MPI_Send(&msg, 1, MPI_INT, 1, 42, MPI_COMM_WORLD);
>>>>> msg++;
>>>>> }
>>>>> } else if (rank == 1) {
>>>>> MPI_Recv(&msg, 1, MPI_INT, 0, 42, MPI_COMM_WORLD,
>>>>> MPI_STATUS_IGNORE);
>>>>> printf("Received %i\n", msg);
>>>>> MPI_Recv(&msg, 1, MPI_INT, 0, 42, MPI_COMM_WORLD,
>>>>> MPI_STATUS_IGNORE);
>>>>> printf("Received %i\n", msg);
>>>>> }
>>>>> }
>>>>>
>>>>> Two threads on the first process send a message, the first thread
>>>>> sends 0, the second thread send 1. From OpenMP semantics, the
>>>>> first send happens before the second send.
>>>>>
>>>>> Is there a guarantee, that the other process receives the 0 first?
>>>>>
>>>>> Thanks,
>>>>> Joachim
>>>>>
>>>>>
>>>>> -- Dipl.-Inf. Joachim Protze
>>>>>
>>>>> IT Center
>>>>> Group: High Performance Computing
>>>>> Division: Computational Science and Engineering
>>>>> RWTH Aachen University
>>>>> Seffenter Weg 23
>>>>> D 52074 Aachen (Germany)
>>>>> Tel: +49 241 80- 24765
>>>>> Fax: +49 241 80-624765
>>>>> protze at itc.rwth-aachen.de
>>>>> <mailto:protze at itc.rwth-aachen.de><mailto:protze at itc.rwth-aachen.de>
>>>>> www.itc.rwth-aachen.de
>>>>> <http://www.itc.rwth-aachen.de/><http://www.itc.rwth-aachen.de
>>>>> <http://www.itc.rwth-aachen.de/>>
>>>>>
>>>>> _______________________________________________
>>>>> mpiwg-p2p mailing list
>>>>> mpiwg-p2p at lists.mpi-forum.org
>>>>> <mailto:mpiwg-p2p at lists.mpi-forum.org><mailto:mpiwg-p2p at lists.mpi-forum.org>
>>>>> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-p2p
>>>>
>>>> The University of Edinburgh is a charitable body, registered in
>>>> Scotland, with registration number SC005336.
>>>> _______________________________________________
>>>> mpiwg-p2p mailing list
>>>> mpiwg-p2p at lists.mpi-forum.org
>>>> <mailto:mpiwg-p2p at lists.mpi-forum.org><mailto:mpiwg-p2p at lists.mpi-forum.org>
>>>> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-p2p
>>> _______________________________________________
>>> mpiwg-p2p mailing list
>>> mpiwg-p2p at lists.mpi-forum.org
>>> <mailto:mpiwg-p2p at lists.mpi-forum.org><mailto:mpiwg-p2p at lists.mpi-forum.org>
>>> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-p2p
>>> --
>>> Jeff Hammond
>>> jeff.science at gmail.com
>>> <mailto:jeff.science at gmail.com><mailto:jeff.science at gmail.com>
>>> http://jeffhammond.github.io/
>>
>>
>> --
>> Dipl.-Inf. Joachim Protze
>>
>> IT Center
>> Group: High Performance Computing
>> Division: Computational Science and Engineering
>> RWTH Aachen University
>> Seffenter Weg 23
>> D 52074 Aachen (Germany)
>> Tel: +49 241 80- 24765
>> Fax: +49 241 80-624765
>> protze at itc.rwth-aachen.de <mailto:protze at itc.rwth-aachen.de>
>> www.itc.rwth-aachen.de <http://www.itc.rwth-aachen.de/>
>>
>> _______________________________________________
>> mpiwg-p2p mailing list
>> mpiwg-p2p at lists.mpi-forum.org <mailto:mpiwg-p2p at lists.mpi-forum.org>
>> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-p2p
>
>
> The University of Edinburgh is a charitable body, registered in
> Scotland, with registration number SC005336.
>
--
Dipl.-Inf. Joachim Protze
IT Center
Group: High Performance Computing
Division: Computational Science and Engineering
RWTH Aachen University
Seffenter Weg 23
D 52074 Aachen (Germany)
Tel: +49 241 80- 24765
Fax: +49 241 80-624765
protze at itc.rwth-aachen.de
www.itc.rwth-aachen.de
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 4915 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.mpi-forum.org/pipermail/mpiwg-p2p/attachments/20181115/985d68f6/attachment-0001.p7s>
More information about the mpiwg-p2p
mailing list