[Mpi-forum] MPI - write to same binary file. Each process shows huge difference in time intervals to complete

[Catherine Jenifer Rajam Rajendran]

Hi All,

I am trying to write in the same binary file using MPI. I set the offset
for each process in the beginning as per the rank. Then the following code
snippet in C runs. All MPI process executes and computes the value and it
writes to the exact offset as set.

The problem I am facing is, say, out of 32 Process, one process is executed
in 2 hours. Rest of the process keeps running for more than 24 hours, The
thing is, it computes the values as expected but it takes so much time. It
seems like a deadlock situation, each process waits for some resource. But,
I am not sharing/communicating between the processes. I am just using
MPI_File_write_at to write at a specific location in the binary file.

I need to mention that each process computes huge amount of data so storing
it temporarily seemed inappropriate. I want to write the output in single
file as number of processes is increased depending on input data. Number of
computations are evenly distributed to all process. So, why does process
takes different time interval to finish its job?!

for(i=1;i<=limit;i++)
{
    for(j=i+1;j<=limit;j++)
    {
        if(my_rank == step%num_cpus)
        {
            Calc = Calculation();
            buf[0] = (double)Calc;
            MPI_File_write_at(outFile, OUT_ofst, buf, 1, MPI_DOUBLE,
&status);
            Calc = 0.0;
            OUT_ofst += num_cpus*MPI_File_write_at(sizeof(double));
            count++;
        }
        step++;
    }
}

I am new to MPI and I guess people must have had similar issues while
executing in MPI. Can anyone help me out please! I can provide more details
if needed.

Thanks,
Catherine
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.mpi-forum.org/pipermail/mpi-forum/attachments/20180710/01a7bccd/attachment.html>