[mpiwg-p2p] Ordering of P2P messages in multithreaded applications

HOLMES Daniel d.holmes at epcc.ed.ac.uk
Thu Nov 15 12:56:42 CST 2018


Hi Joachim,

In more recent changes to MPI, the MPI Forum is trying to be consistent in its usage of terms like “shall” “can” “may” and so on. Specifically, we have adopted the meanings set out here:
https://www.iec.ch/standardsdev/resources/draftingpublications/directives/principles/verbal_forms.htm
For existing text, usage of such words is not as clear-cut.

Your re-wording is one way to interpret the text as written. It is also possible to read the “may not” sentence as one example (sufficient but not necessary) of when this might happen or become relevant. Unfortunately, neither of these interpretations state clearly what happens in the converse situation, i.e. when the thread execution *does* define a relative order (as in your example OpenMP code).

Cheers,
Dan.
—
Dr Daniel Holmes PhD
Applications Consultant in HPC Research
d.holmes at epcc.ed.ac.uk<mailto:d.holmes at epcc.ed.ac.uk>
Phone: +44 (0) 131 651 3465
Mobile: +44 (0) 7940 524 088
Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area, Edinburgh, EH8 9BT
—
The University of Edinburgh is a charitable body, registered in Scotland, with registration number SC005336.
—

On 15 Nov 2018, at 11:57, Joachim Protze via mpiwg-p2p <mpiwg-p2p at lists.mpi-forum.org<mailto:mpiwg-p2p at lists.mpi-forum.org>> wrote:

I guess the paragraph is even more confusing than I thought. As I understand after reading some English grammar pages (oxford/cambridge), "may not" can be prohibitive like "shall not"/"must not"/"cannot" or optional like "might not".

Searching the MPI standard for use of "may not", it is most often used in the optional sense.

From this perspective, I interpreted the paragraph as:

... if the process is multithreaded and the semantics of thread execution do not define a relative order between two send operations executed by two distinct threads, then the operations are logically concurrent, even if one physically precedes the other.


So, probably replacing "may not" by "cannot" would better meet the intention?

Best
Joachim


On 11/15/18 6:19 PM, Joachim Protze wrote:
I think, I was once again confused by "may not" in the cited paragraph. As a non-native speaker this hits me from time to time.
So if I understand it right now, the paragraph says that even if the threading semantics provide an ordering, the operations are still logically concurrent and have no ordering.
Thanks
Joachim
On 11/15/18 6:09 PM, Jeff Hammond wrote:
Dan has convinced me that the MPI standard is terrible and, while my original interpretation is what we want and which is consistent with the principle of least surprise, it is not guaranteed by the following text.

Per our discussion, there are a few options:
1) make all MPI_Send logically concurrent, even on a single thread. this will break stuff and make people sad.
2) force MPI to order injection <somehow>, which might for some implementations to add more memory ordering on the send path than they want, particularly if they do not have a TSO memory model.
3) add something like MPI_Win_sync that logically orders sends from multiple threads explicitly.
4) add MPI_THREAD_SERIALIZED_WITH_EXTRA_SAUCE that does the equivalent of 2 or 3 and thus doesn't cause a performance regression in MPI_THREAD_SERIALIZED.

Jeff

/If a process has a single thread of execution, then any two communications executed by this process are ordered. On the other hand, if the process is multithreaded, then the semantics of thread execution may not define a relative order between two send operations executed by two distinct threads. The operations are logically concurrent, even if one physically precedes the other. In such a case, the two messages sent can be received in any order. Similarly, if two receive operations that are logically concurrent receive two successively sent messages, then the two messages can match the two receives in either order. /


On Thu, Nov 15, 2018 at 10:55 AM Balaji, Pavan via mpiwg-p2p <mpiwg-p2p at lists.mpi-forum.org<mailto:mpiwg-p2p at lists.mpi-forum.org> <mailto:mpiwg-p2p at lists.mpi-forum.org>> wrote:

    Dan,

    The matching *is* ordered in this case.  So the program will print 0
    followed by 1.

    MPI does not order delivery of the actual data, but the first
    message is guaranteed to go into the first buffer.  If the second
    message ends up going first, the MPI implementation will need to
    buffer it.

       — Pavan

    Sent from my iPhone

    On Nov 15, 2018, at 7:56 AM, HOLMES Daniel via mpiwg-p2p
    <mpiwg-p2p at lists.mpi-forum.org<mailto:mpiwg-p2p at lists.mpi-forum.org>
    <mailto:mpiwg-p2p at lists.mpi-forum.org>> wrote:

    Hi Joachim,

    There is no guarantee of ordering between the two sends because
    they are logically concurrent. If they were issued on the same
    thread then MPI guarantees delivery order will be identical to the
    sequential issuing order.

    Many MPI libraries are very likely to deliver these messages "in
    order”, that is, the first one to be called chronologically at the
    sender process is likely to leave first and therefore likely to
    arrive first. Interleaving execution of the sending threads may
    change the issuing order on the network and out-of-order networks
    may change the order of arrival.

    On the other hand, if an MPI implementation is internally using
    sequence numbers (or a similar mechanism) to enforce ordering for
    the same-thread case, then it may also (incidentally) reconstruct
    the issuing order for this case. However, you cannot rely on this
    behaviour being portable from system to system or from MPI library
    to MPI library.

    If you wish to enforce a particular ordering of these messages,
    then you can use tags to differentiate each from the other. There
    is an argument for always using tags in this type of situation to
    increase program readability.

    Cheers,
    Dan.
    —
    Dr Daniel Holmes PhD
    Applications Consultant in HPC Research
    d.holmes at epcc.ed.ac.uk<mailto:d.holmes at epcc.ed.ac.uk> <mailto:d.holmes at epcc.ed.ac.uk>
    Phone: +44 (0) 131 651 3465
    Mobile: +44 (0) 7940 524 088
    Address: Room 2.09, Bayes Centre, 47 Potterrow, Central Area,
    Edinburgh, EH8 9BT
    —
    The University of Edinburgh is a charitable body, registered in
    Scotland, with registration number SC005336.
    —

    On 15 Nov 2018, at 04:16, Joachim Protze via mpiwg-p2p
    <mpiwg-p2p at lists.mpi-forum.org<mailto:mpiwg-p2p at lists.mpi-forum.org>
    <mailto:mpiwg-p2p at lists.mpi-forum.org>> wrote:

    Hi all,

    I have a question on the "Semantics of Point-to-Point
    Communication" in a multithreaded context.

    For me the situation for the code below is not clear, especially
    with respect to the paragraph in MPI-3.1 p.41, l.10-17 :


    void test(int rank) {
     int msg = 0;
     if (rank == 0) {
    #pragma omp parallel num_threads(2)
    #pragma omp critical
       {
         MPI_Send(&msg, 1, MPI_INT, 1, 42, MPI_COMM_WORLD);
         msg++;
       }
     } else if (rank == 1) {
       MPI_Recv(&msg, 1, MPI_INT, 0, 42, MPI_COMM_WORLD,
    MPI_STATUS_IGNORE);
       printf("Received %i\n", msg);
       MPI_Recv(&msg, 1, MPI_INT, 0, 42, MPI_COMM_WORLD,
    MPI_STATUS_IGNORE);
       printf("Received %i\n", msg);
     }
    }

    Two threads on the first process send a message, the first thread
    sends 0, the second thread send 1. From OpenMP semantics, the
    first send happens before the second send.

    Is there a guarantee, that the other process receives the 0 first?

    Thanks,
    Joachim


    --     Dipl.-Inf. Joachim Protze

    IT Center
    Group: High Performance Computing
    Division: Computational Science and Engineering
    RWTH Aachen University
    Seffenter Weg 23
    D 52074  Aachen (Germany)
    Tel: +49 241 80- 24765
    Fax: +49 241 80-624765
    protze at itc.rwth-aachen.de<mailto:protze at itc.rwth-aachen.de> <mailto:protze at itc.rwth-aachen.de>
    www.itc.rwth-aachen.de<http://www.itc.rwth-aachen.de> <http://www.itc.rwth-aachen.de>

    _______________________________________________
    mpiwg-p2p mailing list
    mpiwg-p2p at lists.mpi-forum.org<mailto:mpiwg-p2p at lists.mpi-forum.org> <mailto:mpiwg-p2p at lists.mpi-forum.org>
    https://lists.mpi-forum.org/mailman/listinfo/mpiwg-p2p

    The University of Edinburgh is a charitable body, registered in
    Scotland, with registration number SC005336.
    _______________________________________________
    mpiwg-p2p mailing list
    mpiwg-p2p at lists.mpi-forum.org<mailto:mpiwg-p2p at lists.mpi-forum.org> <mailto:mpiwg-p2p at lists.mpi-forum.org>
    https://lists.mpi-forum.org/mailman/listinfo/mpiwg-p2p
    _______________________________________________
    mpiwg-p2p mailing list
    mpiwg-p2p at lists.mpi-forum.org<mailto:mpiwg-p2p at lists.mpi-forum.org> <mailto:mpiwg-p2p at lists.mpi-forum.org>
    https://lists.mpi-forum.org/mailman/listinfo/mpiwg-p2p



--
Jeff Hammond
jeff.science at gmail.com<mailto:jeff.science at gmail.com> <mailto:jeff.science at gmail.com>
http://jeffhammond.github.io/


--
Dipl.-Inf. Joachim Protze

IT Center
Group: High Performance Computing
Division: Computational Science and Engineering
RWTH Aachen University
Seffenter Weg 23
D 52074  Aachen (Germany)
Tel: +49 241 80- 24765
Fax: +49 241 80-624765
protze at itc.rwth-aachen.de<mailto:protze at itc.rwth-aachen.de>
www.itc.rwth-aachen.de

_______________________________________________
mpiwg-p2p mailing list
mpiwg-p2p at lists.mpi-forum.org
https://lists.mpi-forum.org/mailman/listinfo/mpiwg-p2p

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mpi-forum.org/pipermail/mpiwg-p2p/attachments/20181115/07820a8c/attachment-0001.html>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: not available
URL: <http://lists.mpi-forum.org/pipermail/mpiwg-p2p/attachments/20181115/07820a8c/attachment-0001.ksh>


More information about the mpiwg-p2p mailing list