<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Exchange Server">
<!-- converted from rtf -->
<style><!-- .EmailQuote { margin-left: 1pt; padding-left: 4pt; border-left: #800000 2px solid; } --></style>
</head>
<body>
<font face="Calibri, sans-serif" size="2">
<div>Hi all, here’s the draft code for scenario 2, please take a look.</div>
<div>(I still need to beef up the text, but the entire code and concepts are there)</div>
<div><font face="Times New Roman, serif"> </font></div>
<div style="margin-top: 24pt; "><font face="Cambria, serif" size="4" color="#365F91"><b>Scenario 2: Master-Leader-Workers</b></font></div>
<div style="margin-top: 10pt; "><font face="Cambria, serif" size="3" color="#4F81BD"><b>New </b><b>MPI API</b><b>’</b><b>s</b><b> in this scenario</b></font></div>
<div><font face="Times New Roman, serif"> </font></div>
<div><font face="Courier New, monospace"><b>MPI_Is_restored_rank(</b><b>int* restored</b><b>);</b></font></div>
<div>Indicates if this rank has been restored or is the first start of this process. Can be defined as a generation counter instead of a flag, however for this scenario only the flag version was needed.</div>
<div><font face="Times New Roman, serif"> </font></div>
<div><font face="Courier New, monospace"><b>MPI_Comm_rejoin(</b><b>const char* </b><b>name</b><b>, </b><b>MPI_Comm* newcomm</b><b>);</b></font></div>
<div>This interface recreates the communicator context from a named saved context. It does not restore error handles or any of the attributes (possibly including the comm name). I does restore all the necessary context for the MPI library to continue and work
with this communicator such as, contextid, comm rank to world rank mapping and other information that is impl dependent.</div>
<div><font face="Times New Roman, serif"> </font></div>
<div><font face="Courier New, monospace"><b>MPI_Comm_save(</b><b>MPI_Comm comm</b><b>, </b><b>const char* name</b><b>);</b></font></div>
<div>This interface saves the communicator context under specific name. it is a collective operation and all ranks should call this interface with the same name and communicator. The reason to make this a collective operation is logical and practical. However
from a theoretical pov it does not have to be a collective call. The reasons are, (1) all ranks have to save the communicator context to be able to restore if one fails, saving by only some ranks in the comm has no meaning. (2) a lot of the data is shared between
the ranks (like the rank mapping) saving a single object scales much better than saving an object per rank. (3) different objects would possibly require different name for each rank which seems like a complication of the programming model.</div>
<div><font face="Times New Roman, serif"> </font></div>
<div style="margin-top: 10pt; "><font face="Cambria, serif" size="3" color="#4F81BD"><b>Description</b></font></div>
<div style="margin-top: 10pt; "><font face="Cambria, serif" color="#4F81BD"><b>Helper code</b></font></div>
<div>This code is used but the master, leader and worker to establish the group communicator. The master has to participate as it is part of the initial split of comm world.</div>
<div><font face="Times New Roman, serif"> </font></div>
<div><font face="Courier New, monospace" size="2" color="#0070C0"><b>const</b><font color="#000000"> </font><font color="#000000"><b>int</b></font><font color="#000000"> x_group_size = 10;</font></font></div>
<div><font face="Courier New, monospace" size="2"> </font></div>
<div><font face="Courier New, monospace" size="2" color="#0070C0"><b>MPI_Comm</b><font color="#000000"> get_workers_group_comm()</font></font></div>
<div><font face="Courier New, monospace" size="2">{</font></div>
<div><font face="Courier New, monospace" size="2"> <font color="#0070C0"><b>MPI_Comm</b></font> comm;</font></div>
<div><font face="Courier New, monospace" size="2"> <font color="#0070C0"><b>int</b></font> restored = 0;</font></div>
<div><font face="Courier New, monospace" size="2"> MPI_Is_restored_rank(&restored); </font></div>
<div><font face="Courier New, monospace" size="2"> </font></div>
<div><font face="Courier New, monospace" size="2"> <font color="#FF0000"><b>if</b></font>(restored)</font></div>
<div><font face="Courier New, monospace" size="2"> {</font></div>
<div><font face="Courier New, monospace" size="2" color="#808080"> //</font></div>
<div><font face="Courier New, monospace" size="2" color="#808080"> // Get an already saved communicator context</font></div>
<div><font face="Courier New, monospace" size="2" color="#808080"> // N.B. need to define the failure semantic.</font></div>
<div><font face="Courier New, monospace" size="2" color="#808080"> //</font></div>
<div><font face="Courier New, monospace" size="2"> MPI_Comm_rejoin("workers-group-comm", &comm);</font></div>
<div><font face="Courier New, monospace" size="2"> }</font></div>
<div><font face="Courier New, monospace" size="2"> <font color="#FF0000"><b>else</b></font></font></div>
<div><font face="Courier New, monospace" size="2"> {</font></div>
<div><font face="Courier New, monospace" size="2"> MPI_Comm_rank(MPI_COMM_WORLD, &rank);</font></div>
<div><font face="Courier New, monospace" size="2"> int color = (rank - 1) / x_group_size;</font></div>
<div><font face="Courier New, monospace" size="2"> int key = (rank - 1) % x_group_size;</font></div>
<div><font face="Courier New, monospace" size="2"> MPI_Comm_split(MPI_COMM_WORLD, color, key, &comm);</font></div>
<div><font face="Courier New, monospace" size="2"> </font></div>
<div><font face="Courier New, monospace" size="2"> </font></div>
<div><font face="Courier New, monospace" size="2" color="#808080"> //</font></div>
<div><font face="Courier New, monospace" size="2" color="#808080"> // Save the communicator context to be able to restore it later.</font></div>
<div><font face="Courier New, monospace" size="2" color="#808080"> // note that communicator attributes and error handler are not saved.</font></div>
<div><font face="Courier New, monospace" size="2" color="#808080"> //</font></div>
<div><font face="Courier New, monospace" size="2"> MPI_Comm_save(comm, "workers-group-comm");</font></div>
<div><font face="Courier New, monospace" size="2"> </font></div>
<div><font face="Courier New, monospace" size="2" color="#808080"> //</font></div>
<div><font face="Courier New, monospace" size="2" color="#808080"> // N.B. The barrier here guarantees that the comm was saved for all ranks and</font></div>
<div><font face="Courier New, monospace" size="2" color="#808080"> // that recovery is posible. </font></div>
<div><font face="Courier New, monospace" size="2" color="#808080"> //</font></div>
<div><font face="Courier New, monospace" size="2"> MPI_Barrier(MPI_COMM_WORLD);</font></div>
<div><font face="Courier New, monospace" size="2"> }</font></div>
<div><font face="Courier New, monospace" size="2">}</font></div>
<div><font face="Times New Roman, serif"> </font></div>
<div style="margin-top: 10pt; "><font face="Cambria, serif" size="3" color="#4F81BD"><b>Worker</b></font></div>
<div>The worker code is very similar to the scenario 1 worker code, except that its using the group communicator.</div>
<div><font face="Times New Roman, serif"> </font></div>
<div><font face="Times New Roman, serif"> </font></div>
<div><font face="Courier New, monospace" size="2" color="#808080">//</font></div>
<div><font face="Courier New, monospace" size="2" color="#808080">// Assumption: fatal errors on comm x would abort only the group of processes that are part of comm x</font></div>
<div><font face="Courier New, monospace" size="2" color="#808080">//</font></div>
<div><font face="Courier New, monospace" size="2" color="#808080">//</font></div>
<div><font face="Times New Roman, serif"> </font></div>
<div><font face="Courier New, monospace" size="2" color="#0070C0"><b>void</b><font color="#000000"> worker()</font></font></div>
<div><font face="Courier New, monospace" size="2">{</font></div>
<div><font face="Courier New, monospace" size="2"> MPI_Init(0, 0);</font></div>
<div><font face="Courier New, monospace" size="2"> MPI_Comm comm = get_workers_group_comm();</font></div>
<div><font face="Courier New, monospace" size="2"> </font></div>
<div><font face="Courier New, monospace" size="2"> <font color="#FF0000"><b>for</b></font>(;;)</font></div>
<div><font face="Courier New, monospace" size="2"> {</font></div>
<div><font face="Courier New, monospace" size="2"> MPI_Recv(src=0, &query, comm);</font></div>
<div><font face="Courier New, monospace" size="2"> <font color="#FF0000"><b>if</b></font>(is_done_message(query))</font></div>
<div><font face="Courier New, monospace" size="2"> <font color="#FF0000"><b>break</b></font>;</font></div>
<div><font face="Courier New, monospace" size="2"> </font></div>
<div><font face="Courier New, monospace" size="2"> process_query(&query, &answer);</font></div>
<div><font face="Courier New, monospace" size="2"> </font></div>
<div><font face="Courier New, monospace" size="2"> MPI_Send(dst=0, &answer, comm);</font></div>
<div><font face="Courier New, monospace" size="2"> }</font></div>
<div><font face="Courier New, monospace" size="2"> </font></div>
<div><font face="Courier New, monospace" size="2"> MPI_Finalize()</font></div>
<div><font face="Courier New, monospace" size="2">}</font></div>
<div><font face="Courier New, monospace" size="2"> </font></div>
<div><font face="Courier New, monospace" size="2"> </font></div>
<div style="margin-top: 10pt; "><font face="Cambria, serif" size="3" color="#4F81BD"><b>Leader</b></font></div>
<div>The leader code is the glue between the master and the workers, it redistribute the worker and reassemble the answers to send back to the master.</div>
<div>It is responsible for restarting the master and the workers in case of a failure.</div>
<div><font face="Times New Roman, serif"> </font></div>
<div><font face="Times New Roman, serif"> </font></div>
<div><font face="Courier New, monospace" size="2" color="#0070C0"><b>void</b><font color="#000000"> leader()</font></font></div>
<div><font face="Courier New, monospace" size="2">{</font></div>
<div><font face="Courier New, monospace" size="2"> MPI_Init(0, 0)</font></div>
<div><font face="Courier New, monospace" size="2"> MPI_Comm comm = get_workers_group_comm();</font></div>
<div><font face="Courier New, monospace" size="2"> </font></div>
<div><font face="Courier New, monospace" size="2"> int my_group_size;</font></div>
<div><font face="Courier New, monospace" size="2"> MPI_Comm_size(comm, &my_group_size);</font></div>
<div><font face="Courier New, monospace" size="2"> </font></div>
<div><font face="Courier New, monospace" size="2" color="#808080"> //</font></div>
<div><font face="Courier New, monospace" size="2" color="#808080"> // Set the error handler on comm world to 'errors return' to be able to handle the</font></div>
<div><font face="Courier New, monospace" size="2" color="#808080"> // errors and restart the master or workers if they fail. Set the error handler</font></div>
<div><font face="Courier New, monospace" size="2" color="#808080"> // *after* creating the group communicator, to avoid inheriting the error handler.</font></div>
<div><font face="Courier New, monospace" size="2" color="#808080"> //</font></div>
<div><font face="Courier New, monospace" size="2"> <span style="background-color:#FFFF00">MPI_Comm_set_errhandler(MPI_COMM_WORLD, MPI_ERRORS_RETURN);</span></font></div>
<div><font face="Times New Roman, serif"> </font></div>
<div><font face="Courier New, monospace" size="2" color="#808080"> //</font></div>
<div><font face="Courier New, monospace" size="2" color="#808080"> // process requests coming from the master, until the *done* message arrives</font></div>
<div><font face="Courier New, monospace" size="2" color="#808080"> //</font></div>
<div><font face="Courier New, monospace" size="2"> <font color="#FF0000"><b>for</b></font>(;;)</font></div>
<div><font face="Courier New, monospace" size="2"> {</font></div>
<div><font face="Courier New, monospace" size="2"> QueryMessage query;</font></div>
<div><font face="Courier New, monospace" size="2"> rc = MPI_Recv(src=0, &query, MPI_COMM_WORLD);</font></div>
<div><font face="Courier New, monospace" size="2"> <font color="#FF0000"><b>if</b></font>(rc != MPI_SUCCESS)</font></div>
<div><font face="Courier New, monospace" size="2"> {</font></div>
<div><font face="Courier New, monospace" size="2" color="#808080"> //</font></div>
<div><font face="Courier New, monospace" size="2" color="#808080"> // Communication with the master failed, try to restart it</font></div>
<div><font face="Courier New, monospace" size="2" color="#808080"> //</font></div>
<div><font face="Courier New, monospace" size="2"> <span style="background-color:#FFFF00">rc = MPI_Comm_Restart_rank(MPI_COMM_WORLD, 0);</span></font></div>
<div><font face="Courier New, monospace" size="2"><span style="background-color:#FFFF00"> </span><font color="#FF0000"><span style="background-color:#FFFF00"><b>if</b></span></font><span style="background-color:#FFFF00">(rc == MPI_SUCCESS)</span></font></div>
<div><font face="Courier New, monospace" size="2"><span style="background-color:#FFFF00"> </span><font color="#FF0000"><span style="background-color:#FFFF00"><b>continue</b></span></font><span style="background-color:#FFFF00">;</span></font></div>
<div><font face="Times New Roman, serif"> </font></div>
<div><font face="Courier New, monospace" size="2" color="#808080"> //</font></div>
<div><font face="Courier New, monospace" size="2" color="#808080"> // Could not restart the master, maybe other leaders could; abort this group</font></div>
<div><font face="Courier New, monospace" size="2" color="#808080"> // and expect the master to restart this leader.</font></div>
<div><font face="Courier New, monospace" size="2" color="#808080"> //</font></div>
<div><font face="Courier New, monospace" size="2"> <span style="background-color:#FFFF00">MPI_Abort(comm, 1);</span></font></div>
<div><font face="Courier New, monospace" size="2"> }</font></div>
<div><font face="Courier New, monospace" size="2"> </font></div>
<div><font face="Courier New, monospace" size="2"> </font></div>
<div><font face="Courier New, monospace" size="2"> <font color="#FF0000"><b>for</b></font>(<font color="#0070C0"><b>int</b></font> i = 1; i < my_group_size; i++)</font></div>
<div><font face="Courier New, monospace" size="2"> {</font></div>
<div><font face="Courier New, monospace" size="2"> <font color="#0070C0"><b>WorkerQuery</b></font> worker_query;</font></div>
<div><font face="Courier New, monospace" size="2"> compose_worker_query(query, i, &worker_query);</font></div>
<div><font face="Courier New, monospace" size="2"> <font color="#FF0000"><b>for</b></font>(;;)</font></div>
<div><font face="Courier New, monospace" size="2"> {</font></div>
<div><font face="Courier New, monospace" size="2"> rc = MPI_Send(dst=i, worker_query, comm);</font></div>
<div><font face="Courier New, monospace" size="2"> <font color="#FF0000"><b>if</b></font>(rc == MPI_SUCCESS)</font></div>
<div><font face="Courier New, monospace" size="2"> <font color="#FF0000"><b>break</b></font>;</font></div>
<div><font face="Courier New, monospace" size="2"> </font></div>
<div><font face="Courier New, monospace" size="2" color="#808080"> //</font></div>
<div><font face="Courier New, monospace" size="2" color="#808080"> // Cannot communicate with this worker; try to restart it; if successful</font></div>
<div><font face="Courier New, monospace" size="2" color="#808080"> // send the work item again.</font></div>
<div><font face="Courier New, monospace" size="2" color="#808080"> //</font></div>
<div><font face="Courier New, monospace" size="2"> <span style="background-color:#FFFF00">rc = MPI_Comm_Restart_rank(comm, i);</span></font></div>
<div><font face="Courier New, monospace" size="2"><span style="background-color:#FFFF00"> </span><font color="#FF0000"><span style="background-color:#FFFF00"><b>if</b></span></font><span style="background-color:#FFFF00">(rc == MPI_SUCCESS)</span></font></div>
<div><font face="Courier New, monospace" size="2"><span style="background-color:#FFFF00"> </span><font color="#FF0000"><span style="background-color:#FFFF00"><b>continue</b></span></font><span style="background-color:#FFFF00">;</span></font></div>
<div><font face="Times New Roman, serif"> </font></div>
<div><font face="Courier New, monospace" size="2" color="#808080"> //</font></div>
<div><font face="Courier New, monospace" size="2" color="#808080"> // Could not start the worker; abort this group and expect the master to</font></div>
<div><font face="Courier New, monospace" size="2" color="#808080"> // restart it.</font></div>
<div><font face="Courier New, monospace" size="2" color="#808080"> //</font></div>
<div><font face="Courier New, monospace" size="2"> <span style="background-color:#FFFF00">MPI_Abort(comm, 2);</span></font></div>
<div><font face="Courier New, monospace" size="2"> }</font></div>
<div><font face="Courier New, monospace" size="2"> }</font></div>
<div><font face="Courier New, monospace" size="2"> </font></div>
<div><font face="Courier New, monospace" size="2"> <font color="#FF0000"><b>if</b></font>(is_done_message(query))</font></div>
<div><font face="Courier New, monospace" size="2"> <font color="#FF0000"><b>break</b></font>;</font></div>
<div><font face="Courier New, monospace" size="2"> </font></div>
<div><font face="Courier New, monospace" size="2"> <font color="#0070C0"><b>AnswerMessage</b></font> answer;</font></div>
<div><font face="Courier New, monospace" size="2"> <font color="#FF0000"><b>for</b></font>(<font color="#0070C0"><b>int</b></font> i = 1; i < my_group_size; i++)</font></div>
<div><font face="Courier New, monospace" size="2"> {</font></div>
<div><font face="Courier New, monospace" size="2"> <font color="#0070C0"><b>WorkerAns</b></font><font color="#0070C0"><b>w</b></font><font color="#0070C0"><b>er</b></font> worker_anser;</font></div>
<div><font face="Courier New, monospace" size="2"> rc = MPI_Recv(src=i, worker_answer, comm);</font></div>
<div><font face="Courier New, monospace" size="2"> <font color="#FF0000"><b>if</b></font>(rc !- MPI_SUCCESS)</font></div>
<div><font face="Courier New, monospace" size="2"> {</font></div>
<div><font face="Courier New, monospace" size="2" color="#808080"> //</font></div>
<div><font face="Courier New, monospace" size="2" color="#808080"> // Cannot receive the answer from this worker, and we can't really retry</font></div>
<div><font face="Courier New, monospace" size="2" color="#808080"> // abort this group and expect the master to restart it.</font></div>
<div><font face="Courier New, monospace" size="2" color="#808080"> //</font></div>
<div><font face="Courier New, monospace" size="2"> <span style="background-color:#FFFF00">MPI_Abort(comm, 3);</span></font></div>
<div><font face="Courier New, monospace" size="2"> }</font></div>
<div><font face="Courier New, monospace" size="2"> agregate_worker_result(worker_answer, i, &answer);</font></div>
<div><font face="Courier New, monospace" size="2"> }</font></div>
<div><font face="Courier New, monospace" size="2"> </font></div>
<div><font face="Courier New, monospace" size="2" color="#808080"> //</font></div>
<div><font face="Courier New, monospace" size="2" color="#808080"> // Send the result to the master; no need to check for error here, we will detect</font></div>
<div><font face="Courier New, monospace" size="2" color="#808080"> // it when with the next receive.</font></div>
<div><font face="Courier New, monospace" size="2" color="#808080"> //</font></div>
<div><font face="Courier New, monospace" size="2"> MPI_Send(dst=0, &answer, MPI_COMM_WORLD);</font></div>
<div><font face="Courier New, monospace" size="2"> }</font></div>
<div><font face="Courier New, monospace" size="2"> </font></div>
<div><font face="Courier New, monospace" size="2"> MPI_Finalize()</font></div>
<div><font face="Courier New, monospace" size="2">}</font></div>
<div><font face="Courier New, monospace" size="2"> </font></div>
<div style="margin-top: 10pt; "><font face="Cambria, serif" size="3" color="#4F81BD"><b>Master</b></font></div>
<div>The master code is very similar to scenario 1. It’s the master responsibility to restart the leaders in case of a failure.</div>
<div><font face="Times New Roman, serif"> </font></div>
<div><font face="Courier New, monospace" size="2" color="#0070C0"><b>int</b><font color="#000000"> leaderof(int group)</font></font></div>
<div><font face="Courier New, monospace" size="2">{</font></div>
<div><font face="Courier New, monospace" size="2"> <font color="#FF0000"><b>return</b></font> (group - 1) * x_group_size + 1;</font></div>
<div><font face="Courier New, monospace" size="2">}</font></div>
<div><font face="Courier New, monospace" size="2"> </font></div>
<div><font face="Courier New, monospace" size="2"> </font></div>
<div><font face="Courier New, monospace" size="2" color="#0070C0"><b>void</b><font color="#000000"> master()</font></font></div>
<div><font face="Courier New, monospace" size="2">{</font></div>
<div><font face="Courier New, monospace" size="2"> MPI_Init()</font></div>
<div><font face="Courier New, monospace" size="2"> </font></div>
<div><font face="Courier New, monospace" size="2" color="#808080"> //</font></div>
<div><font face="Courier New, monospace" size="2" color="#808080"> // The master does not need a workers group comm, but in needs to participate in the</font></div>
<div><font face="Courier New, monospace" size="2" color="#808080"> // creation of that comm;</font></div>
<div><font face="Courier New, monospace" size="2" color="#808080"> //</font></div>
<div><font face="Courier New, monospace" size="2"> get_workers_group_comm()</font></div>
<div><font face="Courier New, monospace" size="2"> </font></div>
<div><font face="Courier New, monospace" size="2"> <span style="background-color:#FFFF00">MPI_Comm_set_errhandler(MPI_COMM_WORLD, MPI_ERRORS_RETURN);</span></font></div>
<div><font face="Courier New, monospace" size="2"> MPI_Comm_size(MPI_COMM_WORLD, &world_size);</font></div>
<div><font face="Courier New, monospace" size="2"> </font></div>
<div><font face="Courier New, monospace" size="2" color="#808080"> //</font></div>
<div><font face="Courier New, monospace" size="2" color="#808080"> // include the master as group 0</font></div>
<div><font face="Courier New, monospace" size="2" color="#808080"> //</font></div>
<div><font face="Courier New, monospace" size="2"> <font color="#0070C0"><b>int</b></font> ngroups = (world_size - 1) / x_group_size + 1 + 1;</font></div>
<div><font face="Courier New, monospace" size="2"> <font color="#0070C0"><b>MPI_Request</b></font> r[ngroups] = MPI_REQUEST_NULL;</font></div>
<div><font face="Courier New, monospace" size="2"> <font color="#0070C0"><b>QueryMessage</b></font> q[ngroups];</font></div>
<div><font face="Courier New, monospace" size="2"> <font color="#0070C0"><b>AnswerMessage</b></font> a[ngroups];</font></div>
<div><font face="Courier New, monospace" size="2"> <font color="#0070C0"><b>int</b></font> active_groups = 0;</font></div>
<div><font face="Courier New, monospace" size="2"> </font></div>
<div><font face="Courier New, monospace" size="2"> <font color="#0070C0"><span style="background-color:#FFFF00"><b>bool</b></span></font><span style="background-color:#FFFF00"> restarting[ngroups] = </span><font color="#FF0000"><span style="background-color:#FFFF00"><b>false</b></span></font><span style="background-color:#FFFF00">;</span></font></div>
<div><font face="Times New Roman, serif"> </font></div>
<div><font face="Courier New, monospace" size="2" color="#808080"> //</font></div>
<div><font face="Courier New, monospace" size="2" color="#808080"> // Phase 1: send initial requests</font></div>
<div><font face="Courier New, monospace" size="2" color="#808080"> //</font></div>
<div><font face="Courier New, monospace" size="2"> <font color="#FF0000"><b>for</b></font>(<font color="#0070C0"><b>int</b></font> i = 1; i < ngroups; i++)</font></div>
<div><font face="Courier New, monospace" size="2"> {</font></div>
<div><font face="Courier New, monospace" size="2"> <font color="#FF0000"><b>if</b></font>(get_next_query(stream, &q[i]) == eof)</font></div>
<div><font face="Courier New, monospace" size="2"> <font color="#FF0000"><b>break</b></font>;</font></div>
<div><font face="Courier New, monospace" size="2"> </font></div>
<div><font face="Courier New, monospace" size="2"> active_groups++;</font></div>
<div><font face="Courier New, monospace" size="2"> MPI_Send(dest=leaderof(i);, &q[i], MPI_COMM_WORLD);</font></div>
<div><font face="Courier New, monospace" size="2"> rc = MPI_Irecv(src=leaderof(i), buffer=&a[i], request=&r[i], MPI_COMM_WORLD)</font></div>
<div><font face="Courier New, monospace" size="2"> <font color="#FF0000"><b>if</b></font>(rc != MPI_SUCCESS)</font></div>
<div><font face="Courier New, monospace" size="2"> {</font></div>
<div><font face="Courier New, monospace" size="2"> <span style="background-color:#FFFF00">restart_leader(i, restarting, q, a, r, stream);</span></font></div>
<div><font face="Courier New, monospace" size="2"> }</font></div>
<div><font face="Courier New, monospace" size="2"> }</font></div>
<div><font face="Courier New, monospace" size="2"> </font></div>
<div><font face="Courier New, monospace" size="2" color="#808080"> //</font></div>
<div><font face="Courier New, monospace" size="2" color="#808080"> // Phase 2: finalize any unnecessary ranks</font></div>
<div><font face="Courier New, monospace" size="2" color="#808080"> //</font></div>
<div><font face="Courier New, monospace" size="2"> <font color="#FF0000"><b>for</b></font>(<font color="#0070C0"><b>int</b></font> i = active_groups + 1; i < ngroups; i++)</font></div>
<div><font face="Courier New, monospace" size="2"> {</font></div>
<div><font face="Courier New, monospace" size="2"> MPI_Send(dest=leaderof(i), &done_msg, MPI_COMM_WORLD);</font></div>
<div><font face="Courier New, monospace" size="2"> }</font></div>
<div><font face="Courier New, monospace" size="2"> </font></div>
<div><font face="Courier New, monospace" size="2" color="#808080"> //</font></div>
<div><font face="Courier New, monospace" size="2" color="#808080"> // The progress engine. Get answers; send new requests and handle</font></div>
<div><font face="Courier New, monospace" size="2" color="#808080"> // process restarts</font></div>
<div><font face="Courier New, monospace" size="2" color="#808080"> //</font></div>
<div><font face="Courier New, monospace" size="2"> while(active_groups != 0)</font></div>
<div><font face="Courier New, monospace" size="2"> {</font></div>
<div><font face="Courier New, monospace" size="2"> rc = MPI_Waitany(ngroups, r, &i, MPI_STATUS_IGNORE);</font></div>
<div><font face="Courier New, monospace" size="2"> </font></div>
<div><font face="Courier New, monospace" size="2"> <font color="#FF0000"><b>if</b></font>(rc != MPI_SUCCESS)</font></div>
<div><font face="Courier New, monospace" size="2"> {</font></div>
<div><font face="Courier New, monospace" size="2"> <font color="#FF0000"><span style="background-color:#FFFF00"><b>if</b></span></font><span style="background-color:#FFFF00">(!restarting[i])</span></font></div>
<div><font face="Courier New, monospace" size="2"><span style="background-color:#FFFF00"> {</span></font></div>
<div><font face="Courier New, monospace" size="2"><span style="background-color:#FFFF00"> restart_leader(i, restarting, q, a, r, stream)</span></font></div>
<div><font face="Courier New, monospace" size="2"><span style="background-color:#FFFF00"> }</span></font></div>
<div><font face="Courier New, monospace" size="2"><span style="background-color:#FFFF00"> </span><font color="#FF0000"><span style="background-color:#FFFF00"><b>else</b></span></font></font></div>
<div><font face="Courier New, monospace" size="2"><span style="background-color:#FFFF00"> {</span></font></div>
<div><font face="Courier New, monospace" size="2"><span style="background-color:#FFFF00"> active_groups--;</span></font></div>
<div><font face="Courier New, monospace" size="2"><span style="background-color:#FFFF00"> }</span></font></div>
<div><font face="Courier New, monospace" size="2"><span style="background-color:#FFFF00"> </span><font color="#FF0000"><span style="background-color:#FFFF00"><b>continue</b></span></font><span style="background-color:#FFFF00">;</span></font></div>
<div><font face="Courier New, monospace" size="2"> }</font></div>
<div><font face="Courier New, monospace" size="2"> </font></div>
<div><font face="Courier New, monospace" size="2"> <span style="background-color:#FFFF00">restarting[i] = false;</span></font></div>
<div><font face="Times New Roman, serif"> </font></div>
<div><font face="Courier New, monospace" size="2"> process_answer(&a[i]);</font></div>
<div><font face="Courier New, monospace" size="2"> </font></div>
<div><font face="Courier New, monospace" size="2"> <font color="#FF0000"><b>if</b></font>(get_next_input(stream, &q[i]) == eof)</font></div>
<div><font face="Courier New, monospace" size="2"> {</font></div>
<div><font face="Courier New, monospace" size="2"> active_groups--;</font></div>
<div><font face="Courier New, monospace" size="2"> MPI_Send(dest=leaderof(i), &done_msg)</font></div>
<div><font face="Courier New, monospace" size="2"> {</font></div>
<div><font face="Courier New, monospace" size="2"> <font color="#FF0000"><b>else</b></font></font></div>
<div><font face="Courier New, monospace" size="2"> {</font></div>
<div><font face="Courier New, monospace" size="2"> MPI_Send(dest=leaderof(i), &q[i])</font></div>
<div><font face="Courier New, monospace" size="2"> rc = MPI_Irecv(src=leaderof(i), buffer=&a[i], request=&r[i], MPI_COMM_WORLD)</font></div>
<div><font face="Courier New, monospace" size="2"> <font color="#FF0000"><b>if</b></font>(rc != MPI_SUCCESS)</font></div>
<div><font face="Courier New, monospace" size="2"> {</font></div>
<div><font face="Courier New, monospace" size="2"> restart_leader(i, restarting, q, a, r, stream);</font></div>
<div><font face="Courier New, monospace" size="2"> }</font></div>
<div><font face="Courier New, monospace" size="2"> }</font></div>
<div><font face="Courier New, monospace" size="2"> }</font></div>
<div><font face="Courier New, monospace" size="2"> </font></div>
<div><font face="Courier New, monospace" size="2"> MPI_Finalize();</font></div>
<div><font face="Courier New, monospace" size="2">}</font></div>
<div><font face="Courier New, monospace" size="2"> </font></div>
<div><font face="Courier New, monospace" size="2"> </font></div>
<div><font face="Courier New, monospace" size="2">void restart_leader(int i, int restarting[], Query q[], Answer q[], MPI_Request r[], Stream stream)</font></div>
<div><font face="Courier New, monospace" size="2">{</font></div>
<div><font face="Courier New, monospace" size="2"> restarting[i] = <font color="#FF0000"><b>true</b></font>;</font></div>
<div><font face="Courier New, monospace" size="2"> push_query_back(stream, &q[i]);</font></div>
<div><font face="Courier New, monospace" size="2"> MPI_Comm_Irestart_rank(MPI_COMM_WORLD, leaderof(i), &r[i]);</font></div>
<div><font face="Courier New, monospace" size="2">}</font></div>
<div><font face="Times New Roman, serif"> </font></div>
<div><font face="Times New Roman, serif"> </font></div>
<div>Thanks,</div>
<div>.Erez</div>
<div><font face="Times New Roman, serif"> </font></div>
</font>
</body>
</html>