[Mpi-forum] Fwd: MPI_Pack max buffer size
ftillier at microsoft.com
Wed Oct 3 12:53:11 CDT 2012
Jeff Hammond wrote on Wed, 3 Oct 2012 at 06:58:22
> Unlike many other cases where communicating >2 GB can be done with a
> datatype workaround, MPI_PACK/UNPACK seem particularly vulnerable to
> the 32-bit integer representation issue.
Yeah, MPI_(UN)PACK are a mess. The buffer size being represented by an int is wrong (should be MPI_Aint) and the position being in bytes (thus should also be MPI_Aint). However, if you make position MPI_Aint, you can't pass it in as count to MPI_SEND anymore. The logical conclusion is then that position is a count, but then what is the datatype? I would suggest the datatype is MPI_PACKED since that is what gets passed to MPI_SEND, but the standard talks about bytes, so someone else might think MPI_BYTE.
> I don't know whether or not this has been discussed before (and
> apologize if my message is redundant). Is there a proposal for
> MPI_(UN)PACK_COUNT yet?
This was deliberate, and we spent a bit of time discussing it in the past. The primary reason to not address pack/unpack is that you really shouldn't pack/unpack > 2GB buffers. If anything, you would get better parallelism by chunking and interleaving packing and sending if you really must pack. You'd probably be better off still creating a datatype to represent your data and avoid the copying inherent in packing/unpacking. There are of course exceptions, e.g. fast CPU + slow network where compressing the data can yield gains.
In any case, getting the little bit of count in that we did was a 3 year effort, and we had to minimize the scope as much as possible. I don't personally plan on proposing updated pack/unpack routines, and I'd much rather see us define extended datatype constructors.
> ---------- Forwarded message ----------
> From: Christoph Sprenger <csprenger at wetafx.co.nz>
> Date: Wed, Oct 3, 2012 at 3:10 AM
> Subject: [mpich-discuss] MPI_Pack max buffer size
> To: mpich-discuss at mcs.anl.gov
> just wanted to double check if there is a way to Pack/Unpack bigger
> char buffers than numeric_limits<int>::max() without tokenizing the
> message into multiple chunks ?
> seems like one of my messages is hitting these bounds unfortunately.
> _______________________________________________ mpich-discuss mailing
> list mpich-discuss at mcs.anl.gov To manage subscription options or
> unsubscribe: https://lists.mcs.anl.gov/mailman/listinfo/mpich-discuss
More information about the mpi-forum