[mpiwg-coll] backward communication in neighborhood collectives

[Jed Brown]

For ghost value updates in domain decomposition PDE solvers (usually the
most performance-sensitive use of VecScatter), the graph will usually be
symmetric.  It is often nonsymmetric for multilevel restriction and
prolongation and for some field-split methods in multiphysics
applications, in which case I think you'll want to make two neighborhood
communicators rather than the union of the forward and reverse/transpose
operations with some empty messages.

Junchao Zhang <jczhang at mcs.anl.gov> writes:

> Dan,
>   Using your approach, I have to specify zero messages in arguments to
> MPI_NEIGHBOR_ALLTOALLV,
> because I only do communication in one direction in one time.  Also, in my
> graph, there may have edges A->B and B->A. They may have different weights.
> Using your approach, it is quite confusing.
>   One uses MPI_NEIGHBOR_ALLTOALLV to have better performance.  We have to
> keep it in mind.
>
> --Junchao Zhang
>
> On Mon, Jul 23, 2018 at 9:59 AM, HOLMES Daniel <d.holmes at epcc.ed.ac.uk>
> wrote:
>
>> Hi Junchao Zhang,
>>
>> You are correct - the neighbourhood collectives send messages to (and
>> receive messages from) only neighbouring MPI processes, as defined by the
>> virtual topology. However, it is possible to use point-to-point,
>> single-sided, or normal collective functions to send (and receive) messages
>> between *any* MPI processes in a communicator, irrespective of the
>> presence/structure of a virtual topology.
>>
>> For your use-case (if I understand it correctly), you should specify a
>> symmetric graph - each process specifies to MPI_DIST_GRAPH_CREATE_ADJACENT
>> all the processes it wishes to communicate with (both sending and
>> receiving) during subsequent calls to MPI_NEIGHBOR_ALLTOALLV.
>>
>> Cheers,
>> Dan.
>> —
>> Dr Daniel Holmes PhD
>> Applications Consultant in HPC Research
>> d.holmes at epcc.ed.ac.uk
>> Phone: +44 (0) 131 651 3465
>> Mobile: +44 (0) 7940 524 088
>> Address: Room 3415, JCMB, The King’s Buildings, Edinburgh, EH9 3FD
>> —
>> The University of Edinburgh is a charitable body, registered in Scotland,
>> with registration number SC005336.
>> —
>>
>> On 23 Jul 2018, at 15:45, Junchao Zhang <jczhang at mcs.anl.gov> wrote:
>>
>> Hello, Dan,
>>  I am interested in MPI_Neighbor_alltoallv. From its arguments, I do not
>> see one can send a message to a neighbor not specified in the graph created
>> by for example, MPI_Dist_graph_create_adjacent.
>>
>> --Junchao Zhang
>>
>> On Mon, Jul 23, 2018 at 5:51 AM, HOLMES Daniel <d.holmes at epcc.ed.ac.uk>
>> wrote:
>>
>>> Hi Junchao Zhang,
>>>
>>> My understanding of the current API for MPI-3.1 is that:
>>>
>>> 1) the virtual topology does not actually restrict communication via the
>>> communicator to the edges specified in the topology - messages can be sent
>>> along any edge in either direction, and even between pairs of processes for
>>> which no edge was specified.
>>>
>>> 2) the virtual topology can be specified as a symmetric graph - for every
>>> ‘forward’ edge (e.g. from A to B), the ‘backward’ edge (i.e. from B to A)
>>> can be included as well.
>>>
>>> 3) there is already language in the MPI Standard regarding how MPI
>>> handles symmetric and non-symmetric graph topologies for neighbourhood
>>> collective operations.
>>>
>>> Thus, there is no need to create two distributed graph topology
>>> communicators to achieve ‘forward and backward communication along the
>>> edges’.
>>>
>>> Cheers,
>>> Dan.
>>> —
>>> Dr Daniel Holmes PhD
>>> Applications Consultant in HPC Research
>>> d.holmes at epcc.ed.ac.uk
>>> Phone: +44 (0) 131 651 3465
>>> Mobile: +44 (0) 7940 524 088
>>> Address: Room 3415, JCMB, The King’s Buildings, Edinburgh, EH9 3FD
>>> —
>>> The University of Edinburgh is a charitable body, registered in Scotland,
>>> with registration number SC005336.
>>> —
>>>
>>> On 15 Jul 2018, at 10:42, Anthony Skjellum <tony at runtimecomputing.com>
>>> wrote:
>>>
>>> Hi, I just saw this.
>>>
>>> We definitely need to consider this concern.
>>>
>>> I also need to go review the APIs.
>>>
>>> Thanks,
>>> Tony
>>>
>>>
>>> On Sat, Jul 14, 2018 at 12:27 AM, Junchao Zhang <jczhang at mcs.anl.gov>
>>> wrote:
>>>
>>>>  I want to try MPI neighborhood collectives. I have a communication
>>>> graph and want to do both forward and backward communication along the
>>>> edges. With current APIs, it looks I need to create two comm_dist_graphs.
>>>> It is a waste since MPI implementations can do similar optimizations for
>>>> the two.
>>>>  Should MPI support this scenario?  Thanks.
>>>>
>>>> --Junchao Zhang
>>>>
>>>> _______________________________________________
>>>> mpiwg-coll mailing list
>>>> mpiwg-coll at lists.mpi-forum.org
>>>> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-coll
>>>>
>>>>
>>>
>>>
>>> --
>>> Tony Skjellum, PhD
>>> RunTime Computing Solutions, LLC
>>> tony at runtimecomputing.com
>>> direct: +1-205-918-7514
>>> cell: +1-205-807-4968
>>> _______________________________________________
>>> mpiwg-coll mailing list
>>> mpiwg-coll at lists.mpi-forum.org
>>> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-coll
>>>
>>>
>>>
>>> The University of Edinburgh is a charitable body, registered in
>>> Scotland, with registration number SC005336.
>>>
>>> _______________________________________________
>>> mpiwg-coll mailing list
>>> mpiwg-coll at lists.mpi-forum.org
>>> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-coll
>>>
>>>
>> _______________________________________________
>> mpiwg-coll mailing list
>> mpiwg-coll at lists.mpi-forum.org
>> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-coll
>>
>>
>>
>> The University of Edinburgh is a charitable body, registered in
>> Scotland, with registration number SC005336.
>>
>> _______________________________________________
>> mpiwg-coll mailing list
>> mpiwg-coll at lists.mpi-forum.org
>> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-coll
>>
>>
> _______________________________________________
> mpiwg-coll mailing list
> mpiwg-coll at lists.mpi-forum.org
> https://lists.mpi-forum.org/mailman/listinfo/mpiwg-coll