From wgropp at [hidden] Thu Jul 3 13:15:37 2008 From: wgropp at [hidden] (William Gropp) Date: Thu, 3 Jul 2008 13:15:37 -0500 Subject: [Mpi-22] MPI 2.2 Activities Message-ID: Now that MPI 2.1 is done, MPI 2.2 is kicking into high gear. We will be using the 2.2 wiki ( http://svn.mpi-forum.org/trac/mpi-forum-web/ wiki/MpiTwoTwoWikiPage ) to discuss and prepare items; please see the (amended) presentation from this week's MPI Forum meeting for more details about the process. Also see the link "Web page of MPI 2.2 errata items" ; if you submitted an item to either the old MPI errata list or to the 2.1 process that was deferred to 2.2, please check and see if it is still valid and appropriate for MPI 2.2, and if so, prepare a wiki page for it (we may use tickets instead of wiki pages - the difference should be small to anyone preparing an entry). It the entry isn't relevant to 2.2, please let me know so that I can remove it from the pending items. Thanks! Bill William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign * -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsquyres at [hidden] Mon Jul 7 14:29:40 2008 From: jsquyres at [hidden] (Jeff Squyres) Date: Mon, 7 Jul 2008 15:29:40 -0400 Subject: [Mpi-22] 2 new proposals Message-ID: I have submitted two new 2.2 proposals with the help of George Bosilca: https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/mpi22/TYPE_GET_CONTENTS https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/mpi22/RepeatedNeighborGraphComms These are issues that came up as side-effects of MPI-2.1; I brought both of them up at the Menlo Park Forum meeting. -- Jeff Squyres Cisco Systems From jsquyres at [hidden] Tue Jul 8 13:08:18 2008 From: jsquyres at [hidden] (Jeff Squyres) Date: Tue, 8 Jul 2008 14:08:18 -0400 Subject: [Mpi-22] MPI Forum wiki Message-ID: <563A08F2-709C-44C5-9A13-695608430C51@cisco.com> Policy change: all anonymous edits on the MPI Forum wiki have been disabled (the server got spam-bombed). You must register for an account and then login to be able to edit wiki pages, tickets, etc. For uninteresting reasons, the wiki we're using is not capable of registering users, so you have to use a secondary system to register for an account: https://svn.mpi-forum.org/wiki/start Once you get a username/password on that system, you can use that username/password to logon to the real MPI Forum wiki: https://svn.mpi-forum.org/trac/mpi-forum-web/wiki (this information is on the front page of the wiki as well) ***NOTE 1: It is *STRONGLY* encouraged that you visit the "Preferences" link in the top-right of the MPI Forum Wiki to set your e-mail address. ***NOTE 2: We copied all existing MPI Forum Subversion usernames/ passwords to the wiki system. Hence, if you were involved in the MPI-2.1 editing effort and have a Subversion account, you do not need to re-register; you can simply use your existing Subversion username and password to login to the wiki. Finally, note that Subversion and the wiki are two separate systems; changing your password on one of them will not be reflected on the other. Many thanks to Indiana University for donating system administrator time and server resources to get all this stuff setup. -- Jeff Squyres Cisco Systems From jsquyres at [hidden] Fri Jul 11 13:36:38 2008 From: jsquyres at [hidden] (Jeff Squyres) Date: Fri, 11 Jul 2008 14:36:38 -0400 Subject: [Mpi-22] New proposal: Remove MPI::Fint Message-ID: The MPI::Fint type is included in a table in A.1.1 in MPI-2.1 (and 2.0), but it looks like its inclusion is a mistake. The issue is pretty simple; see the following for details: https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/mpi22/MPI%3A%3AFint -- Jeff Squyres Cisco Systems From erezh at [hidden] Mon Jul 14 16:50:16 2008 From: erezh at [hidden] (Erez Haba) Date: Mon, 14 Jul 2008 14:50:16 -0700 Subject: [Mpi-22] MPI 2.2 proposal: resolving MPI_Request_free issues Message-ID: <6B68D01C00C9994A8E150183E62A119E730A0AA9F2@NA-EXMSG-C105.redmond.corp.microsoft.com> The set of issues related to MPI_Request_free have come up in the last MPI forum meeting. I've added an MPI 2.2 proposal for resolving those issues. See: https://svn.mpi-forum.org/trac/mpi-forum-web/wiki/mpi22/MPI_Request_free Please comment. Thanks, .Erez MPI_Request_free issues Author: Erez Haba Description: The MPI_Request_free mechanism was provided for reasons of performance and convenience on the sending side. However there are number of issues with its definitions and the advice to use text that conflicts with many implementations. Issue #1: Advice to user quote: "Once a request is freed by a call to MPI_REQUEST_FREE, it is not possible to check for the successful completion of the associated communication with calls to MPI_WAIT or MPI_TEST. Also, if an error occurs subsequently during the communication, an error code cannot be returned to the user - such an error must be treated as fatal." This is the only place in the MPI standard that mandates an error to be FATAL, regardless of the user settings. This is truly unrecoverable because the user can not associate the error with the failed send and cannot recover after MPI_Request_free was called. This poses a problem for a fault-tolerance implementation as it must handle this failure without the ability to notify the user for the specific error for the lack of context. Issue #2: Advice to user quote: "Questions arise as to how one knows when the operations have completed when using MPI_REQUEST_FREE. Depending on the program logic, there may be other ways in which the program knows that certain operations have completed and this makes usage of MPI_REQUEST_FREE practical. For example, an active send request could be freed when the logic of the program is such that the receiver sends a reply to the message sent - the arrival of the reply informs the sender that the send has completed and the send buffer can be reused." The suggestion to reuse (free) the buffer once a reply arrived seems straight forward, however it is a naïve one which might lead to access violation at best or worse data corruption. When zero copy is being use the local interconnect resource manager might still be using the send buffer even though a reply message was received. One alternative to enable that programming paradigm is always to copy the buffer (or the tail of the buffer) when using MPI_Isend. This will prevent any misuse of the user buffer. Consider the following examples: Example 1: TCP interconnect Rank 0 on node A sends message x to rank 1 on node B using the following sequence, MPI_Isend(buffer) MPI_Request_free Upon receive message x, rank 1 sends message y back to rank 0; when rank 0 receive message y it frees the buffer with the following sequence MPI_Recv free(buffer) This would result in access violation (seg fault) as the TCP stack still tries to touch buffer after it was freed. This would happen because although node B sends back message y, it did not piggyback the TCP acknowledgment seq numbers back with message x. As a result message y was consumed by the application and buffer was freed. Hence if the TCP stack on node A tries to resend buffer tail resulting in access violation. (note that node A sent message x using zcopy) Example 2: TCP interconnect (2 connections) To make it easier to understand think about the above same problem, but now there are two TCP connections each to only deliver messages in one direction and TCP acknowledgment in the other. This setting decouples the reply message from the TCP acknowledgment and makes the previous example easier to competence. Example 3: RMA interconnect (using RMA write) In this case rank 0 on node A issue its MPI_Isend using RDMA Write. The receiver is polling on memory detecting that the write complete and sends a reply back. The reply message bypasses the hardware acknowledgment that the write was successfully complete. Rank 0 processes it and the app frees the memory which disrupts the DMA on node A and causes the send to fail. Proposed Solution 3 proposals from the least restrictive to the most restrictive: Solution #1: Remove the advice to user to reuse the buffer once a reply has arrived. There is no safe way to reuse the buffer (free), overwrite is somewhat safer. Solution #2: Remove the advice to user altogether, disallow the usage pattern of freeing active requests. Only inactive requests are allowed to be freed. (i.e., not started). Solution #3: Deprecate MPI_Request_free. Users can always use MPI_Wait to complete the request. Recommendation: Use solution #2, as users still need to free requests if they are not used; e.g., the app called MPI_Send_init but never got the start that request; hence the request still need to be freed. History Calling MPI_Request_free was added for convenience (the user does not have to remember to free the request later) and performance (freeing the request is not on the critical path when the reply message arrives and require processing). Impact on Implementations (assuming solution 2) High quality implementations would detect a call to free an active request and return an error. Impact on Applications / Users Applications must refrain freeing the request before it's complete. Therefore, requiring the application to track the request handle to completion. (this potentially breaks existing applicaitons) Alternative Solutions Entry for the Change Log Using MPI_Request_free to free an active request has been disallowed. * -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsquyres at [hidden] Mon Jul 14 20:42:36 2008 From: jsquyres at [hidden] (Jeff Squyres) Date: Mon, 14 Jul 2008 21:42:36 -0400 Subject: [Mpi-22] [Mpi-forum] MPI 2.2 proposal: resolving MPI_Request_free issues In-Reply-To: <6B68D01C00C9994A8E150183E62A119E730A0AA9F2@NA-EXMSG-C105.redmond.corp.microsoft.com> Message-ID: On Jul 14, 2008, at 5:50 PM, Erez Haba wrote: > Issue #1: > Advice to user quote: > > “Once a request is freed by a call to MPI_REQUEST_FREE, it is not > possible to check for the successful completion of the associated > communication with calls to MPI_WAIT or MPI_TEST. Also, if an error > occurs subsequently during the communication, an error code cannot > be returned to the user — such an error must be treated as fatal.” > > This is the only place in the MPI standard that mandates an error to > be FATAL, regardless of the user settings. This is truly > unrecoverable because the user can not associate the error with the > failed send and cannot recover after MPI_Request_free was called. > This poses a problem for a fault-tolerance implementation as it must > handle this failure without the ability to notify the user for the > specific error for the lack of context. I'm not sure I agree with this premise. If you need this functionality, then you shouldn't be using MPI_REQUEST_FREE. Logically speaking, if you want to associate a specific error with a specific communication request, then you must have something to tie the error *to*. In this case, it's a request -- but the application has explicitly stated that it no longer cares about the request. Therefore: if you care about the request, don't free it. > Issue #2: > Advice to user quote: > > “Questions arise as to how one knows when the operations have > completed when using [snip] > causes the send to fail. I don't quite understand examples 1 and 2 (how would they cause segv's in the TCP stack). It is permissible to (pseudocode): while (bytes_to_send > 0) { rc = write(fd, buffer, bytes_to_send); if (rc > 0) { buffer += rc; bytes_to_send -= rc; } else { ...error... } } free(buffer); regardless of what the receiver does. I'm not a kernel guy; does updating TCP sequence numbers also interact with the payload buffer? FWIW: I can see the RMA interconnect example much easier. You can imagine a scenario where a sender successfully sends and the receiver successfully receives, but the hardware ACK from the receiver gets lost. The receiver then sends an MPI message back to the sender, but the sender is still in the middle of a retransmit timeout (while waiting for the hardware ACK that was lost). In this case, the user app may free the buffer too soon, resulting in a segv (or some other lion, tiger, or bear) when the sending hardware tries to retransmit. Don't get me wrong; I'm not a fan of MPI_REQUEST_FREE either. :-) > Proposed Solution > 3 proposals from the least restrictive to the most restrictive: > > Solution #1: > Remove the advice to user to reuse the buffer once a reply has > arrived. There is no safe way to reuse the buffer (free), overwrite > is somewhat safer. > > Solution #2: > Remove the advice to user altogether, disallow the usage pattern of > freeing active requests. Only inactive requests are allowed to be > freed. (i.e., not started). > > Solution #3: > Deprecate MPI_Request_free. Users can always use MPI_Wait to > complete the request. > > Recommendation: > Use solution #2, as users still need to free requests if they are > not used; e.g., the app called MPI_Send_init but never got the start > that request; hence the request still need to be freed. I'm not an apps guy, but I thought there were real world apps out there that use MPI_REQUEST_FREE. So #2 would break real apps -- but I have no idea how many. Perhaps an alternative solution would be: Solution #4: Deprecate MPI_REQUEST_FREE so that it can actually be removed someday. State that requests that are freed can never know when it is safe to free the corresponding buffer (thus making MPI_REQUEST_FREE so unattractive that its use tapers off, and also making it perfectly permissible for an MPI implementation to segv if a user frees a buffer associated with a REQUEST_FREE'd request --- such as the scenarios described above -- because that would be an erroneous program :-) ). -- Jeff Squyres Cisco Systems From ritzdorf at [hidden] Tue Jul 15 05:11:28 2008 From: ritzdorf at [hidden] (Hubert Ritzdorf) Date: Tue, 15 Jul 2008 12:11:28 +0200 Subject: [Mpi-22] [Mpi-forum] MPI 2.2 proposal: resolving MPI_Request_free issues In-Reply-To: Message-ID: <487C77D0.2020200@it.neclab.eu> Jeff Squyres wrote: > On Jul 14, 2008, at 5:50 PM, Erez Haba wrote: > >> Issue #1: >> Advice to user quote: >> >> “Once a request is freed by a call to MPI_REQUEST_FREE, it is not >> possible to check for the successful completion of the associated >> communication with calls to MPI_WAIT or MPI_TEST. Also, if an error >> occurs subsequently during the communication, an error code cannot be >> returned to the user — such an error must be treated as fatal.” >> >> This is the only place in the MPI standard that mandates an error to >> be FATAL, regardless of the user settings. This is truly >> unrecoverable because the user can not associate the error with the >> failed send and cannot recover after MPI_Request_free was called. >> This poses a problem for a fault-tolerance implementation as it must >> handle this failure without the ability to notify the user for the >> specific error for the lack of context. > > I'm not sure I agree with this premise. If you need this > functionality, then you shouldn't be using MPI_REQUEST_FREE. > Logically speaking, if you want to associate a specific error with a > specific communication request, then you must have something to tie > the error *to*. In this case, it's a request -- but the application > has explicitly stated that it no longer cares about the request. > > Therefore: if you care about the request, don't free it. > >> Issue #2: >> Advice to user quote: >> >> “Questions arise as to how one knows when the operations have >> completed when using [snip] >> causes the send to fail. > > I don't quite understand examples 1 and 2 (how would they cause segv's > in the TCP stack). It is permissible to (pseudocode): > > while (bytes_to_send > 0) { > rc = write(fd, buffer, bytes_to_send); > if (rc > 0) { > buffer += rc; > bytes_to_send -= rc; > } else { > ...error... > } > } > free(buffer); > > regardless of what the receiver does. I'm not a kernel guy; does > updating TCP sequence numbers also interact with the payload buffer? > > FWIW: I can see the RMA interconnect example much easier. You can > imagine a scenario where a sender successfully sends and the receiver > successfully receives, but the hardware ACK from the receiver gets > lost. The receiver then sends an MPI message back to the sender, but > the sender is still in the middle of a retransmit timeout (while > waiting for the hardware ACK that was lost). In this case, the user > app may free the buffer too soon, resulting in a segv (or some other > lion, tiger, or bear) when the sending hardware tries to retransmit. > > Don't get me wrong; I'm not a fan of MPI_REQUEST_FREE either. :-) > >> Proposed Solution >> 3 proposals from the least restrictive to the most restrictive: >> >> Solution #1: >> Remove the advice to user to reuse the buffer once a reply has >> arrived. There is no safe way to reuse the buffer (free), overwrite >> is somewhat safer. >> >> Solution #2: >> Remove the advice to user altogether, disallow the usage pattern of >> freeing active requests. Only inactive requests are allowed to be >> freed. (i.e., not started). >> >> Solution #3: >> Deprecate MPI_Request_free. Users can always use MPI_Wait to complete >> the request. >> >> Recommendation: >> Use solution #2, as users still need to free requests if they are not >> used; e.g., the app called MPI_Send_init but never got the start that >> request; hence the request still need to be freed. > > > I'm not an apps guy, but I thought there were real world apps out > there that use MPI_REQUEST_FREE. So #2 would break real apps -- but I > have no idea how many. MPI_Request_free() is used in application programs. For example, it is the easiest (and portable) way to send a non-blocking acknowledge to a destination process. MPI_Isend (buf, 0, MPI_INT, dest, TAG_ACK, comm, lrequest) MPI_Request_free (lrequest) I know applications which are transferring (long living) buffers to destination processes in broker/request parts of application and using MPI_Request_free(). By the way, I think that Issue 2 is independent on MPI_Request_free(). "/the arrival of the reply informs the sender that the send has completed and the send buffer can be reused"/ is not necessarily related to the usage of MPI_Request_free(). // > > Perhaps an alternative solution would be: > > Solution #4: Deprecate MPI_REQUEST_FREE so that it can actually be > removed someday. State that requests that are freed can never know > when it is safe to free the corresponding buffer (thus making > MPI_REQUEST_FREE so unattractive that its use tapers off, and also > making it perfectly permissible for an MPI implementation to segv if a > user frees a buffer associated with a REQUEST_FREE'd request --- such > as the scenarios described above -- because that would be an erroneous > program :-) ). > MPI_Request_free() is used, provides a required functionality and should not be deprecated. Freeing user buffers which are still in use by an interconnect is another topic. What's about the receive buffer ? The receive buffer could also already be freed or re-used in another way directly after the receive was finished. Does the retransmit to the receive buffer not cause corresponding problems ? I think that the receive cannot return before the transfer is finished and the user buffers can be reused. Hubert * -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3245 bytes Desc: S/MIME Cryptographic Signature URL: From jsquyres at [hidden] Tue Jul 15 06:02:19 2008 From: jsquyres at [hidden] (Jeff Squyres) Date: Tue, 15 Jul 2008 07:02:19 -0400 Subject: [Mpi-22] [Mpi-forum] MPI 2.2 proposal: resolving MPI_Request_free issues In-Reply-To: <487C77D0.2020200@it.neclab.eu> Message-ID: <9CA2E993-741B-44F1-AFE9-55332AE93FA5@cisco.com> On Jul 15, 2008, at 6:11 AM, Hubert Ritzdorf wrote: > Freeing user buffers which are still in use by an interconnect is > another topic. What's about the receive buffer ? The receive buffer > could also already be freed or re-used in another way directly after > the receive was finished. That should be fine. > Does the retransmit to the receive buffer not cause corresponding > problems ? I would think that the hardware should recognize it as a duplicate message and drop it (therefore not causing a problem with the receive buffer). After all, in my example, only the receiver's ACK was lost -- the message was otherwise already successfully received. > I think that the receive cannot return before the transfer is > finished and the user buffers can be reused. Agreed. -- Jeff Squyres Cisco Systems From erezh at [hidden] Tue Jul 15 14:08:35 2008 From: erezh at [hidden] (Erez Haba) Date: Tue, 15 Jul 2008 12:08:35 -0700 Subject: [Mpi-22] [Mpi-forum] MPI 2.2 proposal: resolving MPI_Request_free issues In-Reply-To: <487C77D0.2020200@it.neclab.eu> Message-ID: <6B68D01C00C9994A8E150183E62A119E730A0AAE0D@NA-EXMSG-C105.redmond.corp.microsoft.com> See inline -----Original Message----- From: mpi-22-bounces_at_[hidden] [mailto:mpi-22-bounces_at_[hidden]] On Behalf Of Hubert Ritzdorf Sent: Tuesday, July 15, 2008 3:11 AM To: MPI 2.2 Subject: Re: [Mpi-22] [Mpi-forum] MPI 2.2 proposal: resolving MPI_Request_free issues Jeff Squyres wrote: > On Jul 14, 2008, at 5:50 PM, Erez Haba wrote: > >> Issue #1: >> Advice to user quote: >> >> "Once a request is freed by a call to MPI_REQUEST_FREE, it is not >> possible to check for the successful completion of the associated >> communication with calls to MPI_WAIT or MPI_TEST. Also, if an error >> occurs subsequently during the communication, an error code cannot be >> returned to the user - such an error must be treated as fatal." >> >> This is the only place in the MPI standard that mandates an error to >> be FATAL, regardless of the user settings. This is truly >> unrecoverable because the user can not associate the error with the >> failed send and cannot recover after MPI_Request_free was called. >> This poses a problem for a fault-tolerance implementation as it must >> handle this failure without the ability to notify the user for the >> specific error for the lack of context. > > I'm not sure I agree with this premise. If you need this > functionality, then you shouldn't be using MPI_REQUEST_FREE. > Logically speaking, if you want to associate a specific error with a > specific communication request, then you must have something to tie > the error *to*. In this case, it's a request -- but the application > has explicitly stated that it no longer cares about the request. > > Therefore: if you care about the request, don't free it. > [erezh] The points I am making are. 1. This is the only location in the standard where it mandates the error to be FATAL 2. using this function creates a problem for fault-tolerance implementations. (i.e., how would the FT code would know if to make the MPI_Isend FT? the request free can happen any time in the future) >> Issue #2: >> Advice to user quote: >> >> "Questions arise as to how one knows when the operations have >> completed when using [snip] >> causes the send to fail. > > I don't quite understand examples 1 and 2 (how would they cause segv's > in the TCP stack). It is permissible to (pseudocode): > > while (bytes_to_send > 0) { > rc = write(fd, buffer, bytes_to_send); > if (rc > 0) { > buffer += rc; > bytes_to_send -= rc; > } else { > ...error... > } > } > free(buffer); > > regardless of what the receiver does. I'm not a kernel guy; does > updating TCP sequence numbers also interact with the payload buffer? [erezh] it will never happen with your code above; but you are not using async zcopy. The pattern in windows is to use overlapped send (write) which is still active when the function returns, and is the most efficient way to send your buffer. I know it's possible with Linux but I don't' have the exact pattern. > > FWIW: I can see the RMA interconnect example much easier. You can > imagine a scenario where a sender successfully sends and the receiver > successfully receives, but the hardware ACK from the receiver gets > lost. The receiver then sends an MPI message back to the sender, but > the sender is still in the middle of a retransmit timeout (while > waiting for the hardware ACK that was lost). In this case, the user > app may free the buffer too soon, resulting in a segv (or some other > lion, tiger, or bear) when the sending hardware tries to retransmit. > [erezh] Correct; this is the scenario I was describing with the RDMA write. > Don't get me wrong; I'm not a fan of MPI_REQUEST_FREE either. :-) > >> Proposed Solution >> 3 proposals from the least restrictive to the most restrictive: >> >> Solution #1: >> Remove the advice to user to reuse the buffer once a reply has >> arrived. There is no safe way to reuse the buffer (free), overwrite >> is somewhat safer. >> >> Solution #2: >> Remove the advice to user altogether, disallow the usage pattern of >> freeing active requests. Only inactive requests are allowed to be >> freed. (i.e., not started). >> >> Solution #3: >> Deprecate MPI_Request_free. Users can always use MPI_Wait to complete >> the request. >> >> Recommendation: >> Use solution #2, as users still need to free requests if they are not >> used; e.g., the app called MPI_Send_init but never got the start that >> request; hence the request still need to be freed. > > > I'm not an apps guy, but I thought there were real world apps out > there that use MPI_REQUEST_FREE. So #2 would break real apps -- but I > have no idea how many. [erezh] correct, it possibly can break existing applications as mentioned in the "impact on user" section. > MPI_Request_free() is used in application programs. For example, it is > the easiest (and portable) way to send a non-blocking acknowledge > to a destination process. [erezh] See Adam Moody proposal for MPI_REQUEST_IGNORE; I think it's safer than the current pattern > > MPI_Isend (buf, 0, MPI_INT, dest, TAG_ACK, comm, lrequest) > MPI_Request_free (lrequest) > > I know applications which are transferring (long living) buffers to > destination processes in broker/request parts of application > and using MPI_Request_free(). [erezh] longed lived buffers is okay (at least they don't free it); how do they know when it's safe to update the send buffer? (there is an issue of correctness here) > By the way, I think that Issue 2 is independent on MPI_Request_free(). > "/the arrival of the reply informs the sender that the send has > completed and the send buffer can be reused"/ is not necessarily related > to the usage of MPI_Request_free(). > // [erezh] I agree; however this pattern is suggested only in the MPI_Request_free() advice to users. > > Perhaps an alternative solution would be: > > Solution #4: Deprecate MPI_REQUEST_FREE so that it can actually be > removed someday. State that requests that are freed can never know > when it is safe to free the corresponding buffer (thus making > MPI_REQUEST_FREE so unattractive that its use tapers off, and also > making it perfectly permissible for an MPI implementation to segv if a > user frees a buffer associated with a REQUEST_FREE'd request --- such > as the scenarios described above -- because that would be an erroneous > program :-) ). > > MPI_Request_free() is used, provides a required functionality and should > not be deprecated. > > Hubert From jsquyres at [hidden] Tue Jul 15 21:11:23 2008 From: jsquyres at [hidden] (Jeff Squyres) Date: Tue, 15 Jul 2008 22:11:23 -0400 Subject: [Mpi-22] MPI-2.1 ambiguity? Message-ID: <473A5EDB-722C-43DF-8F2C-C88970D67F7F@cisco.com> I will file a proposal about this if there is a problem, but I wanted to ask the group before I did so. In the changelog for MPI-2.1: 2. Section 3.2.2 on page 27, Section 16.1.6 on page 453, and Annex A.1 on page 491. MPI_LONG_LONG_INT, MPI_LONG_LONG (as synonym), MPI_UNSIGNED_LONG_LONG, MPI_SIGNED_CHAR, and MPI_WCHAR are moved from optional to official and they are therefore defined for all three language bindings. Note that it says that these C datatype names must be in all three language bindings. Are C MPI datatypes supposed to be available in Fortran? I looked in mpif.h for Open MPI, MPICH2, HP MPI, and Intel MPI, and I didn't see C MPI datatypes in there (e.g., MPI_INT and MPI_FLOAT). So: - is there language somewhere in MPI that says that C datatypes are not supposed to be in mpif.h? (and therefore the MPI-2.1 changelog is wrong) or - are all 4 of the above MPI's wrong for not including the C datatypes in mpif.h? -- Jeff Squyres Cisco Systems From ritzdorf at [hidden] Wed Jul 16 08:18:07 2008 From: ritzdorf at [hidden] (Hubert Ritzdorf) Date: Wed, 16 Jul 2008 15:18:07 +0200 Subject: [Mpi-22] MPI-2.1 ambiguity? In-Reply-To: <473A5EDB-722C-43DF-8F2C-C88970D67F7F@cisco.com> Message-ID: <487DF50F.60504@it.neclab.eu> Hi, when I remember correctly, it was decided to provide the C MPI datatypes in Fortran since the Fortran datatypes are supported in C and C++. Hubert Jeff Squyres wrote: > I will file a proposal about this if there is a problem, but I wanted > to ask the group before I did so. In the changelog for MPI-2.1: > > 2. Section 3.2.2 on page 27, Section 16.1.6 on page 453, and Annex A.1 > on page 491. > MPI_LONG_LONG_INT, MPI_LONG_LONG (as synonym), MPI_UNSIGNED_LONG_LONG, > MPI_SIGNED_CHAR, and MPI_WCHAR are moved from optional to official and they > are therefore defined for all three language bindings. > > Note that it says that these C datatype names must be in all three > language bindings. > > Are C MPI datatypes supposed to be available in Fortran? I looked in > mpif.h for Open MPI, MPICH2, HP MPI, and Intel MPI, and I didn't see C > MPI datatypes in there (e.g., MPI_INT and MPI_FLOAT). > > So: > > - is there language somewhere in MPI that says that C datatypes are > not supposed to be in mpif.h? (and therefore the MPI-2.1 changelog is > wrong) > > or > > - are all 4 of the above MPI's wrong for not including the C datatypes > in mpif.h? > > --Jeff Squyres > Cisco Systems > > > _______________________________________________ > mpi-22 mailing list > mpi-22_at_[hidden] > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22 > * -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3245 bytes Desc: S/MIME Cryptographic Signature URL: From wgropp at [hidden] Wed Jul 16 10:24:32 2008 From: wgropp at [hidden] (William Gropp) Date: Wed, 16 Jul 2008 10:24:32 -0500 Subject: [Mpi-22] MPI-2.1 ambiguity? In-Reply-To: <487DF50F.60504@it.neclab.eu> Message-ID: <55CFCBF9-360B-4C47-8CC6-1DD9BDD50A8B@illinois.edu> One piece of relevant text in MPI 2.0 is in section 4.12.6 (in MPI-2.0) says that "All predefined datatypes can be used in datatype constructors in any language." However, the example uses MPI_Type_f2c ton convert the Fortran handle to the C version before passing it into the MPI_Type_create_struct routine, so this text could be interpreted as allowing the use of those datatypes through the handle conversion mechanism without requiring them to be defined in each languages header file. Presumably the requirement is that they be predefined with the value that you'd get from the handle conversion function? Bill On Jul 16, 2008, at 8:18 AM, Hubert Ritzdorf wrote: > Hi, > > when I remember correctly, it was decided to provide the C MPI > datatypes > in Fortran since > the Fortran datatypes are supported in C and C++. > > Hubert > > Jeff Squyres wrote: >> I will file a proposal about this if there is a problem, but I wanted >> to ask the group before I did so. In the changelog for MPI-2.1: >> >> 2. Section 3.2.2 on page 27, Section 16.1.6 on page 453, and Annex >> A.1 >> on page 491. >> MPI_LONG_LONG_INT, MPI_LONG_LONG (as synonym), >> MPI_UNSIGNED_LONG_LONG, >> MPI_SIGNED_CHAR, and MPI_WCHAR are moved from optional to official >> and they >> are therefore defined for all three language bindings. >> >> Note that it says that these C datatype names must be in all three >> language bindings. >> >> Are C MPI datatypes supposed to be available in Fortran? I looked in >> mpif.h for Open MPI, MPICH2, HP MPI, and Intel MPI, and I didn't >> see C >> MPI datatypes in there (e.g., MPI_INT and MPI_FLOAT). >> >> So: >> >> - is there language somewhere in MPI that says that C datatypes are >> not supposed to be in mpif.h? (and therefore the MPI-2.1 changelog is >> wrong) >> >> or >> >> - are all 4 of the above MPI's wrong for not including the C >> datatypes >> in mpif.h? >> >> --Jeff Squyres >> Cisco Systems >> >> >> _______________________________________________ >> mpi-22 mailing list >> mpi-22_at_[hidden] >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22 >> >> William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign * -------------- next part -------------- An HTML attachment was scrubbed... URL: From ritzdorf at [hidden] Wed Jul 16 11:57:00 2008 From: ritzdorf at [hidden] (Hubert Ritzdorf) Date: Wed, 16 Jul 2008 18:57:00 +0200 Subject: [Mpi-22] MPI-2.1 ambiguity? In-Reply-To: <55CFCBF9-360B-4C47-8CC6-1DD9BDD50A8B@illinois.edu> Message-ID: <487E285C.4000003@it.neclab.eu> What's about the C++ datatypes for complex ? MPI_COMPLEX and MPI_Double_Complex are already defined for Fortran datatypes. The complex Fortran datatypes within C++ have a special name in order to avoid a name collision. Hubert William Gropp wrote: > One piece of relevant text in MPI 2.0 is in section 4.12.6 (in > MPI-2.0) says that "All predefined datatypes can be used in datatype > constructors in any language." However, the example uses MPI_Type_f2c > ton convert the Fortran handle to the C version before passing it into > the MPI_Type_create_struct routine, so this text could be interpreted > as allowing the use of those datatypes through the handle conversion > mechanism without requiring them to be defined in each languages > header file. Presumably the requirement is that they be predefined > with the value that you'd get from the handle conversion function? > > Bill > > On Jul 16, 2008, at 8:18 AM, Hubert Ritzdorf wrote: > >> Hi, >> >> when I remember correctly, it was decided to provide the C MPI datatypes >> in Fortran since >> the Fortran datatypes are supported in C and C++. >> >> Hubert >> >> Jeff Squyres wrote: >>> I will file a proposal about this if there is a problem, but I wanted >>> to ask the group before I did so. In the changelog for MPI-2.1: >>> >>> 2. Section 3.2.2 on page 27, Section 16.1.6 on page 453, and Annex A.1 >>> on page 491. >>> MPI_LONG_LONG_INT, MPI_LONG_LONG (as synonym), MPI_UNSIGNED_LONG_LONG, >>> MPI_SIGNED_CHAR, and MPI_WCHAR are moved from optional to official and >>> they >>> are therefore defined for all three language bindings. >>> >>> Note that it says that these C datatype names must be in all three >>> language bindings. >>> >>> Are C MPI datatypes supposed to be available in Fortran? I looked in >>> mpif.h for Open MPI, MPICH2, HP MPI, and Intel MPI, and I didn't see C >>> MPI datatypes in there (e.g., MPI_INT and MPI_FLOAT). >>> >>> So: >>> >>> - is there language somewhere in MPI that says that C datatypes are >>> not supposed to be in mpif.h? (and therefore the MPI-2.1 changelog is >>> wrong) >>> >>> or >>> >>> - are all 4 of the above MPI's wrong for not including the C datatypes >>> in mpif.h? >>> >>> --Jeff Squyres >>> Cisco Systems >>> >>> >>> _______________________________________________ >>> mpi-22 mailing list >>> mpi-22_at_[hidden] >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22 >>> >>> > > William Gropp > Paul and Cynthia Saylor Professor of Computer Science > University of Illinois Urbana-Champaign > > > > ------------------------------------------------------------------------ > > _______________________________________________ > mpi-22 mailing list > mpi-22_at_[hidden] > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22 > * -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3245 bytes Desc: S/MIME Cryptographic Signature URL: From keith.d.underwood at [hidden] Wed Jul 16 18:31:42 2008 From: keith.d.underwood at [hidden] (Underwood, Keith D) Date: Wed, 16 Jul 2008 18:31:42 -0500 Subject: [Mpi-22] [Mpi-forum] MPI 2.2 proposal:resolving MPI_Request_free issues In-Reply-To: <6B68D01C00C9994A8E150183E62A119E730A0AAE0D@NA-EXMSG-C105.redmond.corp.microsoft.com> Message-ID: <1BABDF0B0BF60D428576BCC7646C0A8F01DEA91A@rrsmsx411.amr.corp.intel.com> >> I don't quite understand examples 1 and 2 (how would they cause segv's >> in the TCP stack). It is permissible to (pseudocode): >> >> while (bytes_to_send > 0) { >> rc = write(fd, buffer, bytes_to_send); >> if (rc > 0) { >> buffer += rc; >> bytes_to_send -= rc; >> } else { >> ...error... >> } >> } >> free(buffer); >> >> regardless of what the receiver does. I'm not a kernel guy; does >> updating TCP sequence numbers also interact with the payload buffer? > >[erezh] it will never happen with your code above; but you are not using >async zcopy. >The pattern in windows is to use overlapped send (write) which is still >active when the function returns, and is the most efficient way to send >your buffer. I know it's possible with Linux but I don't' have the exact >pattern. So, how does this work for other users of zero copy? How do they know when the send buffer is truly free? >> FWIW: I can see the RMA interconnect example much easier. You can >> imagine a scenario where a sender successfully sends and the receiver >> successfully receives, but the hardware ACK from the receiver gets >> lost. The receiver then sends an MPI message back to the sender, but >> the sender is still in the middle of a retransmit timeout (while >> waiting for the hardware ACK that was lost). In this case, the user >> app may free the buffer too soon, resulting in a segv (or some other >> lion, tiger, or bear) when the sending hardware tries to retransmit. >> > >[erezh] Correct; this is the scenario I was describing with the RDMA write. It would be interesting to see exactly what the error mode here is. Retransmitting corrupted data should be ok, since a correctly delivered message means that the retransmit must be dropped. I suppose that if the NIC speaks virtual addresses and the free actually results in a trap to the kernel that unmaps the pages, then the NIC could retransmit and find that there isn't a valid page table entry... >> MPI_Request_free() is used in application programs. For example, it is >> the easiest (and portable) way to send a non-blocking acknowledge >> to a destination process. > >[erezh] See Adam Moody proposal for MPI_REQUEST_IGNORE; I think it's safer >than the current pattern > >> >> MPI_Isend (buf, 0, MPI_INT, dest, TAG_ACK, comm, lrequest) >> MPI_Request_free (lrequest) Hrm. This example doesn't seem to trigger Erez's concern... That's a zero length buffer, right? Solution #5: Change the advice to users - "...the arrival of the reply informs the sender that the send has completed and the send buffer can be overwritten. If the buffer will ever be freed, the application should call MPI_Wait or MPI_Cancel instead of MPI_Request_free." If you have received your application level ack from the remote side, MPI_Cancel should give you the equivalent functionality of MPI_Request_free - allowing you to release the request without having to wait for local completion. I'm not entirely sure WHY you would want to do this, but... ;-) Keith From jsquyres at [hidden] Wed Jul 16 20:59:23 2008 From: jsquyres at [hidden] (Jeff Squyres) Date: Wed, 16 Jul 2008 21:59:23 -0400 Subject: [Mpi-22] [Mpi-forum] MPI 2.2 proposal:resolving MPI_Request_free issues In-Reply-To: <1BABDF0B0BF60D428576BCC7646C0A8F01DEA91A@rrsmsx411.amr.corp.intel.com> Message-ID: <9604850B-F646-4CBA-A91F-FF3FF442B72F@cisco.com> On Jul 16, 2008, at 7:31 PM, Underwood, Keith D wrote: >>> [erezh] Correct; this is the scenario I was describing with the RDMA > write. > > It would be interesting to see exactly what the error mode here is. > Retransmitting corrupted data should be ok, since a correctly > delivered > message means that the retransmit must be dropped. I suppose that if > the NIC speaks virtual addresses and the free actually results in a > trap > to the kernel that unmaps the pages, then the NIC could retransmit and > find that there isn't a valid page table entry... 'zactly. And then the local completion entry would be a failure -- so the sender would [erroneously] think that the message had failed to be delivered. > Solution #5: Change the advice to users - "...the arrival of the > reply > informs the sender that the send has completed and the send buffer can > be overwritten. If the buffer will ever be freed, the application > should call MPI_Wait or MPI_Cancel instead of MPI_Request_free." That seems like a good compromise. -- Jeff Squyres Cisco Systems From jsquyres at [hidden] Wed Jul 16 21:05:31 2008 From: jsquyres at [hidden] (Jeff Squyres) Date: Wed, 16 Jul 2008 22:05:31 -0400 Subject: [Mpi-22] MPI-2.1 ambiguity? In-Reply-To: <55CFCBF9-360B-4C47-8CC6-1DD9BDD50A8B@illinois.edu> Message-ID: <21AA2D28-CD37-4241-97BA-83536A51B3A2@cisco.com> On Jul 16, 2008, at 11:24 AM, William Gropp wrote: > One piece of relevant text in MPI 2.0 is in section 4.12.6 (in > MPI-2.0) says that "All predefined datatypes can be used in datatype > constructors in any language." However, the example uses > MPI_Type_f2c ton convert the Fortran handle to the C version before > passing it into the MPI_Type_create_struct routine, so this text > could be interpreted as allowing the use of those datatypes through > the handle conversion mechanism without requiring them to be defined > in each languages header file. Presumably the requirement is that > they be predefined with the value that you'd get from the handle > conversion function? The text in Annex B is stronger than that, though. It specifically states "they are therefore defined for all three language bindings" -- is that a mistake? >>> 2. Section 3.2.2 on page 27, Section 16.1.6 on page 453, and Annex >>> A.1 >>> on page 491. >>> MPI_LONG_LONG_INT, MPI_LONG_LONG (as synonym), >>> MPI_UNSIGNED_LONG_LONG, >>> MPI_SIGNED_CHAR, and MPI_WCHAR are moved from optional to official >>> and they >>> are therefore defined for all three language bindings. -- Jeff Squyres Cisco Systems From jsquyres at [hidden] Wed Jul 16 21:05:54 2008 From: jsquyres at [hidden] (Jeff Squyres) Date: Wed, 16 Jul 2008 22:05:54 -0400 Subject: [Mpi-22] MPI-2.1 ambiguity? In-Reply-To: <487E285C.4000003@it.neclab.eu> Message-ID: What does NEC MPI do -- do you have MPI_INT and MPI_FLOAT (etc.) in mpif.h? On Jul 16, 2008, at 12:57 PM, Hubert Ritzdorf wrote: > What's about the C++ datatypes for complex ? > MPI_COMPLEX and MPI_Double_Complex are already defined for Fortran > datatypes. The complex Fortran datatypes within C++ have a special > name in order to avoid a name collision. > > Hubert > > William Gropp wrote: >> One piece of relevant text in MPI 2.0 is in section 4.12.6 (in >> MPI-2.0) says that "All predefined datatypes can be used in >> datatype constructors in any language." However, the example uses >> MPI_Type_f2c ton convert the Fortran handle to the C version before >> passing it into the MPI_Type_create_struct routine, so this text >> could be interpreted as allowing the use of those datatypes through >> the handle conversion mechanism without requiring them to be >> defined in each languages header file. Presumably the requirement >> is that they be predefined with the value that you'd get from the >> handle conversion function? >> >> Bill >> >> On Jul 16, 2008, at 8:18 AM, Hubert Ritzdorf wrote: >> >>> Hi, >>> >>> when I remember correctly, it was decided to provide the C MPI >>> datatypes >>> in Fortran since >>> the Fortran datatypes are supported in C and C++. >>> >>> Hubert >>> >>> Jeff Squyres wrote: >>>> I will file a proposal about this if there is a problem, but I >>>> wanted >>>> to ask the group before I did so. In the changelog for MPI-2.1: >>>> >>>> 2. Section 3.2.2 on page 27, Section 16.1.6 on page 453, and >>>> Annex A.1 >>>> on page 491. >>>> MPI_LONG_LONG_INT, MPI_LONG_LONG (as synonym), >>>> MPI_UNSIGNED_LONG_LONG, >>>> MPI_SIGNED_CHAR, and MPI_WCHAR are moved from optional to official >>>> and they >>>> are therefore defined for all three language bindings. >>>> >>>> Note that it says that these C datatype names must be in all three >>>> language bindings. >>>> >>>> Are C MPI datatypes supposed to be available in Fortran? I >>>> looked in >>>> mpif.h for Open MPI, MPICH2, HP MPI, and Intel MPI, and I didn't >>>> see C >>>> MPI datatypes in there (e.g., MPI_INT and MPI_FLOAT). >>>> >>>> So: >>>> >>>> - is there language somewhere in MPI that says that C datatypes are >>>> not supposed to be in mpif.h? (and therefore the MPI-2.1 >>>> changelog is >>>> wrong) >>>> >>>> or >>>> >>>> - are all 4 of the above MPI's wrong for not including the C >>>> datatypes >>>> in mpif.h? >>>> >>>> --Jeff Squyres >>>> Cisco Systems >>>> >>>> >>>> _______________________________________________ >>>> mpi-22 mailing list >>>> mpi-22_at_[hidden] >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22 >>>> >>>> >> >> William Gropp >> Paul and Cynthia Saylor Professor of Computer Science >> University of Illinois Urbana-Champaign >> >> >> >> ------------------------------------------------------------------------ >> >> _______________________________________________ >> mpi-22 mailing list >> mpi-22_at_[hidden] >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22 >> > _______________________________________________ > mpi-22 mailing list > mpi-22_at_[hidden] > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22 -- Jeff Squyres Cisco Systems From ritzdorf at [hidden] Thu Jul 17 03:20:28 2008 From: ritzdorf at [hidden] (Hubert Ritzdorf) Date: Thu, 17 Jul 2008 10:20:28 +0200 Subject: [Mpi-22] MPI-2.1 ambiguity? In-Reply-To: Message-ID: <487F00CC.5090405@it.neclab.eu> No, we didn't have them. The request to provide all C datatypes in Fortran was specified in one of the last meetings. When I remember correctly, it was specified in the last meeting in Chicago or the meeting before (I think it was specified together with item 33 of the change log). I implemented it together with the changes for MPI 2.1 since I assumed that it was contained also in the change log. After your mail, I recognized that it is only implicitly mentioned by item 33. Hubert Jeff Squyres wrote: > What does NEC MPI do -- do you have MPI_INT and MPI_FLOAT (etc.) in > mpif.h? > > > On Jul 16, 2008, at 12:57 PM, Hubert Ritzdorf wrote: > >> What's about the C++ datatypes for complex ? >> MPI_COMPLEX and MPI_Double_Complex are already defined for Fortran >> datatypes. The complex Fortran datatypes within C++ have a special >> name in order to avoid a name collision. >> >> Hubert >> >> William Gropp wrote: >>> One piece of relevant text in MPI 2.0 is in section 4.12.6 (in >>> MPI-2.0) says that "All predefined datatypes can be used in datatype >>> constructors in any language." However, the example uses >>> MPI_Type_f2c ton convert the Fortran handle to the C version before >>> passing it into the MPI_Type_create_struct routine, so this text >>> could be interpreted as allowing the use of those datatypes through >>> the handle conversion mechanism without requiring them to be defined >>> in each languages header file. Presumably the requirement is that >>> they be predefined with the value that you'd get from the handle >>> conversion function? >>> >>> Bill >>> >>> On Jul 16, 2008, at 8:18 AM, Hubert Ritzdorf wrote: >>> >>>> Hi, >>>> >>>> when I remember correctly, it was decided to provide the C MPI >>>> datatypes >>>> in Fortran since >>>> the Fortran datatypes are supported in C and C++. >>>> >>>> Hubert >>>> >>>> Jeff Squyres wrote: >>>>> I will file a proposal about this if there is a problem, but I wanted >>>>> to ask the group before I did so. In the changelog for MPI-2.1: >>>>> >>>>> 2. Section 3.2.2 on page 27, Section 16.1.6 on page 453, and Annex >>>>> A.1 >>>>> on page 491. >>>>> MPI_LONG_LONG_INT, MPI_LONG_LONG (as synonym), >>>>> MPI_UNSIGNED_LONG_LONG, >>>>> MPI_SIGNED_CHAR, and MPI_WCHAR are moved from optional to official >>>>> and they >>>>> are therefore defined for all three language bindings. >>>>> >>>>> Note that it says that these C datatype names must be in all three >>>>> language bindings. >>>>> >>>>> Are C MPI datatypes supposed to be available in Fortran? I looked in >>>>> mpif.h for Open MPI, MPICH2, HP MPI, and Intel MPI, and I didn't >>>>> see C >>>>> MPI datatypes in there (e.g., MPI_INT and MPI_FLOAT). >>>>> >>>>> So: >>>>> >>>>> - is there language somewhere in MPI that says that C datatypes are >>>>> not supposed to be in mpif.h? (and therefore the MPI-2.1 changelog is >>>>> wrong) >>>>> >>>>> or >>>>> >>>>> - are all 4 of the above MPI's wrong for not including the C >>>>> datatypes >>>>> in mpif.h? >>>>> >>>>> --Jeff Squyres >>>>> Cisco Systems >>>>> >>>>> >>>>> _______________________________________________ >>>>> mpi-22 mailing list >>>>> mpi-22_at_[hidden] >>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22 >>>>> >>>>> >>> >>> William Gropp >>> Paul and Cynthia Saylor Professor of Computer Science >>> University of Illinois Urbana-Champaign >>> >>> >>> >>> ------------------------------------------------------------------------ >>> >>> >>> _______________________________________________ >>> mpi-22 mailing list >>> mpi-22_at_[hidden] >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22 >>> >> _______________________________________________ >> mpi-22 mailing list >> mpi-22_at_[hidden] >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22 > > > --Jeff Squyres > Cisco Systems > > > _______________________________________________ > mpi-22 mailing list > mpi-22_at_[hidden] > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22 > * -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3245 bytes Desc: S/MIME Cryptographic Signature URL: From rabenseifner at [hidden] Thu Jul 17 04:03:39 2008 From: rabenseifner at [hidden] (Rolf Rabenseifner) Date: Thu, 17 Jul 2008 11:03:39 +0200 Subject: [Mpi-22] MPI-2.1 ambiguity? In-Reply-To: <487F00CC.5090405@it.neclab.eu> Message-ID: Hi all, The argument-chain was: >>> William Gropp wrote: >>>> One piece of relevant text in MPI 2.0 is in section 4.12.6 (in >>>> MPI-2.0) says that "All predefined datatypes can be used in >>>> datatype constructors in any language." [....] This text is supported also by: MPI-2.0 Chap. 10.1.6 page 276 and MPI-2.0 Annex B.2, page 344 define for all Fortran predefined datatype handles as C++ counter-part. In the meeting, there was a consensus (?) that it is a bug if an MPI implementation does not support, e.g. MPI_INT in the mpif.h Fortran header file. (This bug can be found, e.g., in MPICH-2 and will be fixed, when I understood correctly) The value (not the name!!) of MPI_INT in mpif.h and mpi.h may be different, as stated already in MPI-2.0, Sect. 4.12.9. MPI_INT from C mpi.h cannot be used in a Fortran CALL MPI_SEND(...). You need to convert, or you must use MPI_INT from Fortran mpif.h. (I.e. one may distiguish between mixed languages in one MPI executable [addressed by MPI-2.0 Section 4.12.9], and sending a message from a Fortran MPI executable to a C MPI executable and vice versa [see MPI 2.0 Section 4.12.6].) Because this is already "clearly" defined in MPI-2.0, you will not find any remark about this in the Change-Log of MPI-2.1. MPI-2.2 may make it more clear in the revisiting of datatypes. Best regards Rolf On Thu, 17 Jul 2008 10:20:28 +0200 Hubert Ritzdorf wrote: > No, we didn't have them. The request to provide all C datatypes in >Fortran was specified in one of the last meetings. When I remember > correctly, it was specified in the last meeting in Chicago or > the meeting before (I think it was specified together with item 33 > of the change log). I implemented it together with the changes > for MPI 2.1 since I assumed that it was contained also in the > change log. After your mail, I recognized that it is only implicitly > mentioned by item 33. > > Hubert > > Jeff Squyres wrote: >> What does NEC MPI do -- do you have MPI_INT and MPI_FLOAT (etc.) in >> mpif.h? >> >> >> On Jul 16, 2008, at 12:57 PM, Hubert Ritzdorf wrote: >> >>> What's about the C++ datatypes for complex ? >>> MPI_COMPLEX and MPI_Double_Complex are already defined for Fortran >>> datatypes. The complex Fortran datatypes within C++ have a special >>> name in order to avoid a name collision. >>> >>> Hubert >>> >>> William Gropp wrote: >>>> One piece of relevant text in MPI 2.0 is in section 4.12.6 (in >>>> MPI-2.0) says that "All predefined datatypes can be used in datatype >>>> constructors in any language." However, the example uses >>>> MPI_Type_f2c ton convert the Fortran handle to the C version before >>>> passing it into the MPI_Type_create_struct routine, so this text >>>> could be interpreted as allowing the use of those datatypes through >>>> the handle conversion mechanism without requiring them to be defined >>>> in each languages header file. Presumably the requirement is that >>>> they be predefined with the value that you'd get from the handle >>>> conversion function? >>>> >>>> Bill >>>> >>>> On Jul 16, 2008, at 8:18 AM, Hubert Ritzdorf wrote: >>>> >>>>> Hi, >>>>> >>>>> when I remember correctly, it was decided to provide the C MPI >>>>> datatypes >>>>> in Fortran since >>>>> the Fortran datatypes are supported in C and C++. >>>>> >>>>> Hubert >>>>> >>>>> Jeff Squyres wrote: >>>>>> I will file a proposal about this if there is a problem, but I >>>>>>wanted >>>>>> to ask the group before I did so. In the changelog for MPI-2.1: >>>>>> >>>>>> 2. Section 3.2.2 on page 27, Section 16.1.6 on page 453, and Annex >>>>>> A.1 >>>>>> on page 491. >>>>>> MPI_LONG_LONG_INT, MPI_LONG_LONG (as synonym), >>>>>> MPI_UNSIGNED_LONG_LONG, >>>>>> MPI_SIGNED_CHAR, and MPI_WCHAR are moved from optional to official >>>>>> and they >>>>>> are therefore defined for all three language bindings. >>>>>> >>>>>> Note that it says that these C datatype names must be in all three >>>>>> language bindings. >>>>>> >>>>>> Are C MPI datatypes supposed to be available in Fortran? I looked >>>>>>in >>>>>> mpif.h for Open MPI, MPICH2, HP MPI, and Intel MPI, and I didn't >>>>>> see C >>>>>> MPI datatypes in there (e.g., MPI_INT and MPI_FLOAT). >>>>>> >>>>>> So: >>>>>> >>>>>> - is there language somewhere in MPI that says that C datatypes are >>>>>> not supposed to be in mpif.h? (and therefore the MPI-2.1 changelog >>>>>>is >>>>>> wrong) >>>>>> >>>>>> or >>>>>> >>>>>> - are all 4 of the above MPI's wrong for not including the C >>>>>> datatypes >>>>>> in mpif.h? >>>>>> >>>>>> --Jeff Squyres >>>>>> Cisco Systems >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> mpi-22 mailing list >>>>>> mpi-22_at_[hidden] >>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22 >>>>>> >>>>>> >>>> >>>> William Gropp >>>> Paul and Cynthia Saylor Professor of Computer Science >>>> University of Illinois Urbana-Champaign >>>> >>>> >>>> >>>> ------------------------------------------------------------------------ >>>> >>>> >>>> _______________________________________________ >>>> mpi-22 mailing list >>>> mpi-22_at_[hidden] >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22 >>>> >>> _______________________________________________ >>> mpi-22 mailing list >>> mpi-22_at_[hidden] >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22 >> >> >> --Jeff Squyres >> Cisco Systems >> >> >> _______________________________________________ >> mpi-22 mailing list >> mpi-22_at_[hidden] >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22 >> Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner_at_[hidden] High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30) From wgropp at [hidden] Thu Jul 17 09:03:35 2008 From: wgropp at [hidden] (William Gropp) Date: Thu, 17 Jul 2008 09:03:35 -0500 Subject: [Mpi-22] MPI-2.1 ambiguity? In-Reply-To: Message-ID: <5F5AC4BC-3C12-4B18-B0D0-26961FAFD4F1@illinois.edu> To make this perfectly clear, should the standard state that the values of the predefined datatypes from another language are exactly the same as the value produced by the handle conversion function? E.g., in Fortran, the value of MPI_INT is what you would get from MPI_TYPE_C2F( MPI_INT ) (where MPI_INT is the C version). It should also state that just because Fortran defines MPI_LONG_LONG, that does not imply that Fortran support an integer type that corresponds to the C long long type. Bill On Jul 17, 2008, at 4:03 AM, Rolf Rabenseifner wrote: > Hi all, > > The argument-chain was: > >>>> William Gropp wrote: >>>>> One piece of relevant text in MPI 2.0 is in section 4.12.6 (in >>>>> MPI-2.0) says that "All predefined datatypes can be used in >>>>> datatype constructors in any language." > [....] > > This text is supported also by: > MPI-2.0 Chap. 10.1.6 page 276 and > MPI-2.0 Annex B.2, page 344 define for all Fortran predefined > datatype handles as C++ counter-part. > > In the meeting, there was a consensus (?) that it is a bug > if an MPI implementation does not support, e.g. MPI_INT > in the mpif.h Fortran header file. > (This bug can be found, e.g., in MPICH-2 and will be fixed, > when I understood correctly) > > The value (not the name!!) of MPI_INT in mpif.h and mpi.h may be > different, as stated already in MPI-2.0, Sect. 4.12.9. > MPI_INT from C mpi.h cannot be used in a Fortran CALL MPI_SEND(...). > You need to convert, or you must use MPI_INT from Fortran mpif.h. > > (I.e. one may distiguish between mixed languages in > one MPI executable [addressed by MPI-2.0 Section 4.12.9], > and sending a message from a Fortran MPI executable to a C MPI > executable and vice versa [see MPI 2.0 Section 4.12.6].) > > Because this is already "clearly" defined in MPI-2.0, > you will not find any remark about this in the Change-Log > of MPI-2.1. > > MPI-2.2 may make it more clear in the revisiting of datatypes. > > Best regards > Rolf > > On Thu, 17 Jul 2008 10:20:28 +0200 > Hubert Ritzdorf wrote: >> No, we didn't have them. The request to provide all C datatypes in >> Fortran was specified in one of the last meetings. When I remember >> correctly, it was specified in the last meeting in Chicago or >> the meeting before (I think it was specified together with item 33 >> of the change log). I implemented it together with the changes >> for MPI 2.1 since I assumed that it was contained also in the >> change log. After your mail, I recognized that it is only implicitly >> mentioned by item 33. >> >> Hubert >> >> Jeff Squyres wrote: >>> What does NEC MPI do -- do you have MPI_INT and MPI_FLOAT (etc.) in >>> mpif.h? >>> >>> >>> On Jul 16, 2008, at 12:57 PM, Hubert Ritzdorf wrote: >>> >>>> What's about the C++ datatypes for complex ? >>>> MPI_COMPLEX and MPI_Double_Complex are already defined for Fortran >>>> datatypes. The complex Fortran datatypes within C++ have a >>>> special >>>> name in order to avoid a name collision. >>>> >>>> Hubert >>>> >>>> William Gropp wrote: >>>>> One piece of relevant text in MPI 2.0 is in section 4.12.6 (in >>>>> MPI-2.0) says that "All predefined datatypes can be used in >>>>> datatype >>>>> constructors in any language." However, the example uses >>>>> MPI_Type_f2c ton convert the Fortran handle to the C version >>>>> before >>>>> passing it into the MPI_Type_create_struct routine, so this text >>>>> could be interpreted as allowing the use of those datatypes >>>>> through >>>>> the handle conversion mechanism without requiring them to be >>>>> defined >>>>> in each languages header file. Presumably the requirement is that >>>>> they be predefined with the value that you'd get from the handle >>>>> conversion function? >>>>> >>>>> Bill >>>>> >>>>> On Jul 16, 2008, at 8:18 AM, Hubert Ritzdorf wrote: >>>>> >>>>>> Hi, >>>>>> >>>>>> when I remember correctly, it was decided to provide the C MPI >>>>>> datatypes >>>>>> in Fortran since >>>>>> the Fortran datatypes are supported in C and C++. >>>>>> >>>>>> Hubert >>>>>> >>>>>> Jeff Squyres wrote: >>>>>>> I will file a proposal about this if there is a problem, but I >>>>>>> wanted >>>>>>> to ask the group before I did so. In the changelog for MPI-2.1: >>>>>>> >>>>>>> 2. Section 3.2.2 on page 27, Section 16.1.6 on page 453, and >>>>>>> Annex >>>>>>> A.1 >>>>>>> on page 491. >>>>>>> MPI_LONG_LONG_INT, MPI_LONG_LONG (as synonym), >>>>>>> MPI_UNSIGNED_LONG_LONG, >>>>>>> MPI_SIGNED_CHAR, and MPI_WCHAR are moved from optional to >>>>>>> official >>>>>>> and they >>>>>>> are therefore defined for all three language bindings. >>>>>>> >>>>>>> Note that it says that these C datatype names must be in all >>>>>>> three >>>>>>> language bindings. >>>>>>> >>>>>>> Are C MPI datatypes supposed to be available in Fortran? I >>>>>>> looked >>>>>>> in >>>>>>> mpif.h for Open MPI, MPICH2, HP MPI, and Intel MPI, and I didn't >>>>>>> see C >>>>>>> MPI datatypes in there (e.g., MPI_INT and MPI_FLOAT). >>>>>>> >>>>>>> So: >>>>>>> >>>>>>> - is there language somewhere in MPI that says that C >>>>>>> datatypes are >>>>>>> not supposed to be in mpif.h? (and therefore the MPI-2.1 >>>>>>> changelog >>>>>>> is >>>>>>> wrong) >>>>>>> >>>>>>> or >>>>>>> >>>>>>> - are all 4 of the above MPI's wrong for not including the C >>>>>>> datatypes >>>>>>> in mpif.h? >>>>>>> >>>>>>> --Jeff Squyres >>>>>>> Cisco Systems >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> mpi-22 mailing list >>>>>>> mpi-22_at_[hidden] >>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22 >>>>>>> >>>>>>> >>>>> >>>>> William Gropp >>>>> Paul and Cynthia Saylor Professor of Computer Science >>>>> University of Illinois Urbana-Champaign >>>>> >>>>> >>>>> >>>>> ------------------------------------------------------------------ >>>>> ------ >>>>> >>>>> >>>>> _______________________________________________ >>>>> mpi-22 mailing list >>>>> mpi-22_at_[hidden] >>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22 >>>>> >>>> _______________________________________________ >>>> mpi-22 mailing list >>>> mpi-22_at_[hidden] >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22 >>> >>> >>> --Jeff Squyres >>> Cisco Systems >>> >>> >>> _______________________________________________ >>> mpi-22 mailing list >>> mpi-22_at_[hidden] >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22 >>> > > Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner_at_[hidden] > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 > Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner > Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30) > _______________________________________________ > mpi-22 mailing list > mpi-22_at_[hidden] > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22 William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign * -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsquyres at [hidden] Thu Jul 17 09:34:22 2008 From: jsquyres at [hidden] (Jeff Squyres) Date: Thu, 17 Jul 2008 10:34:22 -0400 Subject: [Mpi-22] MPI-2.1 ambiguity? In-Reply-To: <5F5AC4BC-3C12-4B18-B0D0-26961FAFD4F1@illinois.edu> Message-ID: On Jul 17, 2008, at 10:03 AM, William Gropp wrote: > To make this perfectly clear, should the standard state that the > values of the predefined datatypes from another language are exactly > the same as the value produced by the handle conversion function? > E.g., in Fortran, the value of MPI_INT is what you would get from > MPI_TYPE_C2F( MPI_INT ) (where MPI_INT is the C version). It > should also state that just because Fortran defines MPI_LONG_LONG, > that does not imply that Fortran support an integer type that > corresponds to the C long long type. I think I agree with the idea (you can use MPI_INT in Fortran just like you would use MPI_INT in C just like you would use MPI::INT in C+ +), but your exact idea doesn't seem to work for C++. For example, what about MPI::INTEGER? I had assumed that the only use of MPI_INT in Fortran would be used for Fortran to pass a buffer to an MPI function that was pre-filled with int's in C (and other similar cases like this). Is that consistent with what you're suggesting? -- Jeff Squyres Cisco Systems From rabenseifner at [hidden] Thu Jul 17 10:01:23 2008 From: rabenseifner at [hidden] (Rolf Rabenseifner) Date: Thu, 17 Jul 2008 17:01:23 +0200 Subject: [Mpi-22] MPI-2.1 ambiguity? In-Reply-To: <5F5AC4BC-3C12-4B18-B0D0-26961FAFD4F1@illinois.edu> Message-ID: My primary oal is: We should make clarifications which state that each predefined handle must be available in every language binding. People may be confused by current wording in several places: - MPI-2.1 (June 23, 2008) Section 16.3.6 MPI Opaque Objects, page 483, lines 46-47: "All predefined datatypes can be used in datatype constructors in any language." - MPI-2.1 (June 23, 2008) Section 16.3.9 Constants, page 488, lines 33-35: "MPI constants have the same value in all languages, unless specified otherwise. This does not apply to constant handles (MPI_INT, MPI_COMM_WORLD, MPI_ERRORS_RETURN, MPI_SUM, etc.) These handles need to be converted, as explained in Section 16.3.4." - MPI-2.1 (June 23, 2008) Annex A.1.1 Defined Constants, page 491, lines 22-23: "The C and Fortran name is listed in the left column and the C++ name is listed in the middle or right column." Proposal: ========= MPI-2.1 p. 488, lines 33-35 reads: "MPI constants have the same value in all languages, unless specified otherwise. This does not apply to constant handles (MPI_INT, MPI_COMM_WORLD, MPI_ERRORS_RETURN, MPI_SUM, etc.) These handles need to be converted, as explained in Section 16.3.4." but should read: "MPI constants have the same value in all languages, unless specified otherwise. This does not apply to constant handles (MPI_INT, MPI_COMM_WORLD, MPI_ERRORS_RETURN, MPI_SUM, etc.) Although these handles (e.g. predefined datatypes) must exist in all language bindings, their values may be different in each language binding. If one wants to use a constant handle from one language binding (e.g., MPI_DOUBLE from mpi.h) in a call to an MPI function in another language binding (e.g., in Fortran MPI_SEND), then these handles need to be converted, as explained in Section 16.3.4." Does this help? Best regards Rolf On Thu, 17 Jul 2008 09:03:35 -0500 William Gropp wrote: > To make this perfectly clear, should the standard state that the > values of the predefined datatypes from another language are exactly > the same as the value produced by the handle conversion function? > E.g., in Fortran, the value of MPI_INT is what you would get from > MPI_TYPE_C2F( MPI_INT ) (where MPI_INT is the C version). It >should also state that just because Fortran defines MPI_LONG_LONG, >that does not imply that Fortran support an integer type that >corresponds to the C long long type. > > Bill > > On Jul 17, 2008, at 4:03 AM, Rolf Rabenseifner wrote: > >> Hi all, >> >> The argument-chain was: >> >>>>> William Gropp wrote: >>>>>> One piece of relevant text in MPI 2.0 is in section 4.12.6 (in >>>>>> MPI-2.0) says that "All predefined datatypes can be used in >>>>>> datatype constructors in any language." >> [....] >> >> This text is supported also by: >> MPI-2.0 Chap. 10.1.6 page 276 and >> MPI-2.0 Annex B.2, page 344 define for all Fortran predefined >> datatype handles as C++ counter-part. >> >> In the meeting, there was a consensus (?) that it is a bug >> if an MPI implementation does not support, e.g. MPI_INT >> in the mpif.h Fortran header file. >> (This bug can be found, e.g., in MPICH-2 and will be fixed, >> when I understood correctly) >> >> The value (not the name!!) of MPI_INT in mpif.h and mpi.h may be >> different, as stated already in MPI-2.0, Sect. 4.12.9. >> MPI_INT from C mpi.h cannot be used in a Fortran CALL MPI_SEND(...). >> You need to convert, or you must use MPI_INT from Fortran mpif.h. >> >> (I.e. one may distiguish between mixed languages in >> one MPI executable [addressed by MPI-2.0 Section 4.12.9], >> and sending a message from a Fortran MPI executable to a C MPI >> executable and vice versa [see MPI 2.0 Section 4.12.6].) >> >> Because this is already "clearly" defined in MPI-2.0, >> you will not find any remark about this in the Change-Log >> of MPI-2.1. >> >> MPI-2.2 may make it more clear in the revisiting of datatypes. >> >> Best regards >> Rolf >> >> On Thu, 17 Jul 2008 10:20:28 +0200 >> Hubert Ritzdorf wrote: >>> No, we didn't have them. The request to provide all C datatypes in >>> Fortran was specified in one of the last meetings. When I remember >>> correctly, it was specified in the last meeting in Chicago or >>> the meeting before (I think it was specified together with item 33 >>> of the change log). I implemented it together with the changes >>> for MPI 2.1 since I assumed that it was contained also in the >>> change log. After your mail, I recognized that it is only implicitly >>> mentioned by item 33. >>> >>> Hubert >>> >>> Jeff Squyres wrote: >>>> What does NEC MPI do -- do you have MPI_INT and MPI_FLOAT (etc.) in >>>> mpif.h? >>>> >>>> >>>> On Jul 16, 2008, at 12:57 PM, Hubert Ritzdorf wrote: >>>> >>>>> What's about the C++ datatypes for complex ? >>>>> MPI_COMPLEX and MPI_Double_Complex are already defined for Fortran >>>>> datatypes. The complex Fortran datatypes within C++ have a >>>>> special >>>>> name in order to avoid a name collision. >>>>> >>>>> Hubert >>>>> >>>>> William Gropp wrote: >>>>>> One piece of relevant text in MPI 2.0 is in section 4.12.6 (in >>>>>> MPI-2.0) says that "All predefined datatypes can be used in >>>>>> datatype >>>>>> constructors in any language." However, the example uses >>>>>> MPI_Type_f2c ton convert the Fortran handle to the C version >>>>>> before >>>>>> passing it into the MPI_Type_create_struct routine, so this text >>>>>> could be interpreted as allowing the use of those datatypes >>>>>> through >>>>>> the handle conversion mechanism without requiring them to be >>>>>> defined >>>>>> in each languages header file. Presumably the requirement is that >>>>>> they be predefined with the value that you'd get from the handle >>>>>> conversion function? >>>>>> >>>>>> Bill >>>>>> >>>>>> On Jul 16, 2008, at 8:18 AM, Hubert Ritzdorf wrote: >>>>>> >>>>>>> Hi, >>>>>>> >>>>>>> when I remember correctly, it was decided to provide the C MPI >>>>>>> datatypes >>>>>>> in Fortran since >>>>>>> the Fortran datatypes are supported in C and C++. >>>>>>> >>>>>>> Hubert >>>>>>> >>>>>>> Jeff Squyres wrote: >>>>>>>> I will file a proposal about this if there is a problem, but I >>>>>>>> wanted >>>>>>>> to ask the group before I did so. In the changelog for MPI-2.1: >>>>>>>> >>>>>>>> 2. Section 3.2.2 on page 27, Section 16.1.6 on page 453, and >>>>>>>> Annex >>>>>>>> A.1 >>>>>>>> on page 491. >>>>>>>> MPI_LONG_LONG_INT, MPI_LONG_LONG (as synonym), >>>>>>>> MPI_UNSIGNED_LONG_LONG, >>>>>>>> MPI_SIGNED_CHAR, and MPI_WCHAR are moved from optional to >>>>>>>> official >>>>>>>> and they >>>>>>>> are therefore defined for all three language bindings. >>>>>>>> >>>>>>>> Note that it says that these C datatype names must be in all >>>>>>>> three >>>>>>>> language bindings. >>>>>>>> >>>>>>>> Are C MPI datatypes supposed to be available in Fortran? I >>>>>>>> looked >>>>>>>> in >>>>>>>> mpif.h for Open MPI, MPICH2, HP MPI, and Intel MPI, and I didn't >>>>>>>> see C >>>>>>>> MPI datatypes in there (e.g., MPI_INT and MPI_FLOAT). >>>>>>>> >>>>>>>> So: >>>>>>>> >>>>>>>> - is there language somewhere in MPI that says that C >>>>>>>> datatypes are >>>>>>>> not supposed to be in mpif.h? (and therefore the MPI-2.1 >>>>>>>> changelog >>>>>>>> is >>>>>>>> wrong) >>>>>>>> >>>>>>>> or >>>>>>>> >>>>>>>> - are all 4 of the above MPI's wrong for not including the C >>>>>>>> datatypes >>>>>>>> in mpif.h? >>>>>>>> >>>>>>>> --Jeff Squyres >>>>>>>> Cisco Systems >>>>>>>> >>>>>>>> >>>>>>>> _______________________________________________ >>>>>>>> mpi-22 mailing list >>>>>>>> mpi-22_at_[hidden] >>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22 >>>>>>>> >>>>>>>> >>>>>> >>>>>> William Gropp >>>>>> Paul and Cynthia Saylor Professor of Computer Science >>>>>> University of Illinois Urbana-Champaign >>>>>> >>>>>> >>>>>> >>>>>> ------------------------------------------------------------------ >>>>>> ------ >>>>>> >>>>>> >>>>>> _______________________________________________ >>>>>> mpi-22 mailing list >>>>>> mpi-22_at_[hidden] >>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22 >>>>>> >>>>> _______________________________________________ >>>>> mpi-22 mailing list >>>>> mpi-22_at_[hidden] >>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22 >>>> >>>> >>>> --Jeff Squyres >>>> Cisco Systems >>>> >>>> >>>> _______________________________________________ >>>> mpi-22 mailing list >>>> mpi-22_at_[hidden] >>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22 >>>> >> >> Dr. Rolf Rabenseifner . . . . . . . . . .. email >>rabenseifner_at_[hidden] >> High Performance Computing Center (HLRS) . phone >>++49(0)711/685-65530 >> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / >>685-65832 >> Head of Dpmt Parallel Computing . . . >>www.hlrs.de/people/rabenseifner >> Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30) >> _______________________________________________ >> mpi-22 mailing list >> mpi-22_at_[hidden] >> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22 > > William Gropp > Paul and Cynthia Saylor Professor of Computer Science > University of Illinois Urbana-Champaign > > > Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner_at_[hidden] High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30) From treumann at [hidden] Thu Jul 17 10:04:50 2008 From: treumann at [hidden] (Richard Treumann) Date: Thu, 17 Jul 2008 11:04:50 -0400 Subject: [Mpi-22] MPI-2.1 ambiguity? In-Reply-To: <5F5AC4BC-3C12-4B18-B0D0-26961FAFD4F1@illinois.edu> Message-ID: To me this debate seems trivial but perhaps that is because for IBM MPI the value of a predefined datatype handle is a 4 byte integer constant and MPI_INT is a different constant than MPI_INTEGER. Putting MPI_INT in mpif.h is easy and it has been there since day one. MPI_TYPE_C2F( MPI_INT )has no work to do. Are there MPI implementations that do something that makes it complicated to satisfy the following statements? 1) All predefined datatype handles must be defined in all languages (C, C++ and Fortran) 2) The value of a C handle as defined in mpif.h must be that same as is returned when MPI_TYPE_C2F( handle ) is used in C 3) Having a C handle in mpif.h does not imply that the Fortran compiler supports a corresponding Fortran datatype It could be argued that point 3 is too obvious to mention. Dick Treumann - MPI Team/TCEM IBM Systems & Technology Group Dept 0lva / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 Tele (845) 433-7846 Fax (845) 433-8363 mpi-22-bounces_at_[hidden] wrote on 07/17/2008 10:03:35 AM: > To make this perfectly clear, should the standard state that the > values of the predefined datatypes from another language are exactly > the same as the value produced by the handle conversion function? >  E.g., in Fortran, the value of MPI_INT is what you would get from > MPI_TYPE_C2F( MPI_INT )  (where MPI_INT is the C version).  It > should also state that just because Fortran defines MPI_LONG_LONG, > that does not imply that Fortran support an integer type that > corresponds to the C long long type. > > Bill > > On Jul 17, 2008, at 4:03 AM, Rolf Rabenseifner wrote: > > Hi all, > > The argument-chain was: > > William Gropp wrote: > One piece of relevant text in MPI 2.0 is in section 4.12.6 (in > MPI-2.0) says that "All predefined datatypes can be used in > datatype constructors in any language." > [....] > > This text is supported also by: > MPI-2.0 Chap. 10.1.6 page 276 and > MPI-2.0 Annex B.2, page 344 define for all Fortran predefined > datatype handles as C++ counter-part. > > In the meeting, there was a consensus (?) that it is a bug > if an MPI implementation does not support, e.g. MPI_INT > in the mpif.h Fortran header file. > (This bug can be found, e.g., in MPICH-2 and will be fixed, > when I understood correctly) > > The value (not the name!!) of MPI_INT in mpif.h and mpi.h may be > different, as stated already in MPI-2.0, Sect. 4.12.9. > MPI_INT from C mpi.h cannot be used in a Fortran CALL MPI_SEND(...). > You need to convert, or you must use MPI_INT from Fortran mpif.h. > > (I.e. one may distiguish between mixed languages in > one MPI executable [addressed by MPI-2.0 Section 4.12.9], > and sending a message from a Fortran MPI executable to a C MPI > executable and vice versa [see MPI 2.0 Section 4.12.6].) > > Because this is already "clearly" defined in MPI-2.0, > you will not find any remark about this in the Change-Log > of MPI-2.1. > > MPI-2.2 may make it more clear in the revisiting of datatypes. > > Best regards > Rolf > > On Thu, 17 Jul 2008 10:20:28 +0200 >   Hubert Ritzdorf wrote: > No, we didn't have them. The request to provide all C datatypes in > Fortran was specified in one of the last meetings. When I remember > correctly, it was specified in the last meeting in Chicago or > the meeting before (I think it was specified together with item 33 > of the change log). I implemented it together with the changes > for MPI 2.1 since I assumed that it was contained also in the > change log. After your mail, I recognized that it is only implicitly > mentioned by item 33. > > Hubert > > Jeff Squyres wrote: > What does NEC MPI do -- do you have MPI_INT and MPI_FLOAT (etc.) in > mpif.h? > > On Jul 16, 2008, at 12:57 PM, Hubert Ritzdorf wrote: > > What's about the C++ datatypes for complex ? > MPI_COMPLEX and MPI_Double_Complex are already defined for Fortran > datatypes. The complex Fortran datatypes  within C++  have a special > name in order to avoid a name collision. > > Hubert > > William Gropp wrote: > One piece of relevant text in MPI 2.0 is in section 4.12.6 (in > MPI-2.0) says that "All predefined datatypes can be used in datatype > constructors in any language."  However, the example uses > MPI_Type_f2c ton convert the Fortran handle to the C version before > passing it into the MPI_Type_create_struct routine, so this text > could be interpreted as allowing the use of those datatypes through > the handle conversion mechanism without requiring them to be defined > in each languages header file.  Presumably the requirement is that > they be predefined with the value that you'd get from the handle > conversion function? > > Bill > > On Jul 16, 2008, at 8:18 AM, Hubert Ritzdorf wrote: > > Hi, > > when I remember correctly, it was decided to provide the C MPI > datatypes > in Fortran since > the Fortran datatypes are supported in C and C++. > > Hubert > > Jeff Squyres wrote: > I will file a proposal about this if there is a problem, but I > wanted > to ask the group before I did so.  In the changelog for MPI-2.1: > > 2. Section 3.2.2 on page 27, Section 16.1.6 on page 453, and Annex > A.1 > on page 491. > MPI_LONG_LONG_INT, MPI_LONG_LONG (as synonym), > MPI_UNSIGNED_LONG_LONG, > MPI_SIGNED_CHAR, and MPI_WCHAR are moved from optional to official > and they > are therefore defined for all three language bindings. > > Note that it says that these C datatype names must be in all three > language bindings. > > Are C MPI datatypes supposed to be available in Fortran?  I looked > in > mpif.h for Open MPI, MPICH2, HP MPI, and Intel MPI, and I didn't > see C > MPI datatypes in there (e.g., MPI_INT and MPI_FLOAT). > > So: > > - is there language somewhere in MPI that says that C datatypes are > not supposed to be in mpif.h? (and therefore the MPI-2.1 changelog > is > wrong) > > or > > - are all 4 of the above MPI's wrong for not including the C > datatypes > in mpif.h? > > --Jeff Squyres > Cisco Systems > > _______________________________________________ > mpi-22 mailing list > mpi-22_at_[hidden] > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22 > > > > William Gropp > Paul and Cynthia Saylor Professor of Computer Science > University of Illinois Urbana-Champaign > > ------------------------------------------------------------------------ > > _______________________________________________ > mpi-22 mailing list > mpi-22_at_[hidden] > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22 > > _______________________________________________ > mpi-22 mailing list > mpi-22_at_[hidden] > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22 > > --Jeff Squyres > Cisco Systems > > _______________________________________________ > mpi-22 mailing list > mpi-22_at_[hidden] > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22 > > Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner_at_[hidden] > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 > Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner > Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30) > _______________________________________________ > mpi-22 mailing list > mpi-22_at_[hidden] > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22 > > William Gropp > Paul and Cynthia Saylor Professor of Computer Science > University of Illinois Urbana-Champaign > > _______________________________________________ > mpi-22 mailing list > mpi-22_at_[hidden] > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22 * -------------- next part -------------- An HTML attachment was scrubbed... URL: From wgropp at [hidden] Thu Jul 17 11:41:55 2008 From: wgropp at [hidden] (William Gropp) Date: Thu, 17 Jul 2008 11:41:55 -0500 Subject: [Mpi-22] MPI-2.1 ambiguity? In-Reply-To: Message-ID: I don't think that text will work. Specifically, what is the type of MPI_INT in Fortran? It may be something other than a Fortran INTEGER. So, either these values need to already be converted into the language-of-the-headerfile, or we need a new datatype for Fortran that can contain a C or a C++ Datatype (e.g., something like INTEGER (KIND=MPI_DATATYPE_C_HANDLE) ). That's why I think that they must be presented by the implementation with the values that you would get by taking making use of the conversion functions. Of course, an application must then be careful not to apply MPI_Type_c2f to the Fortran version of MPI_INT ... There's another option. Drop the requirement that all of the predefined datatypes are predefined in all languages, but specify that they may be used (with the appropriate handle conversion) as input to any routine where it makes sense. For datatypes, this includes the datatype construction routines but not the communication routines (don't use MPI_INT when sending INTEGERs with Fortran). This is what the Opaque Objects text expects. Bill On Jul 17, 2008, at 10:01 AM, Rolf Rabenseifner wrote: > My primary oal is: > We should make clarifications which state that > each predefined handle must be available in every > language binding. > People may be confused by current wording in several places: > > - MPI-2.1 (June 23, 2008) Section 16.3.6 MPI Opaque Objects, > page 483, lines 46-47: > "All predefined datatypes can be used in datatype constructors > in any language." > > - MPI-2.1 (June 23, 2008) Section 16.3.9 Constants, > page 488, lines 33-35: > "MPI constants have the same value in all languages, > unless specified otherwise. This does not > apply to constant handles (MPI_INT, MPI_COMM_WORLD, > MPI_ERRORS_RETURN, MPI_SUM, etc.) These handles need > to be converted, as explained in Section 16.3.4." > > - MPI-2.1 (June 23, 2008) Annex A.1.1 Defined Constants, > page 491, lines 22-23: > "The C and Fortran name is listed in the left column and the > C++ name is listed in the middle or right column." > > Proposal: > ========= > > MPI-2.1 p. 488, lines 33-35 reads: > "MPI constants have the same value in all languages, > unless specified otherwise. This does not > apply to constant handles (MPI_INT, MPI_COMM_WORLD, > MPI_ERRORS_RETURN, MPI_SUM, etc.) These handles need > to be converted, as explained in Section 16.3.4." > > but should read: > "MPI constants have the same value in all languages, > unless specified otherwise. This does not > apply to constant handles (MPI_INT, MPI_COMM_WORLD, > MPI_ERRORS_RETURN, MPI_SUM, etc.) > Although these handles (e.g. predefined datatypes) > must exist in all language bindings, their values > may be different in each language binding. > If one wants to use a constant handle from one language binding > (e.g., MPI_DOUBLE from mpi.h) in a call to an MPI function in > another language binding (e.g., in Fortran MPI_SEND), then > these handles need > to be converted, as explained in Section 16.3.4." > > Does this help? > > Best regards > Rolf > > > On Thu, 17 Jul 2008 09:03:35 -0500 > William Gropp wrote: >> To make this perfectly clear, should the standard state that the >> values of the predefined datatypes from another language are exactly >> the same as the value produced by the handle conversion function? >> E.g., in Fortran, the value of MPI_INT is what you would get from >> MPI_TYPE_C2F( MPI_INT ) (where MPI_INT is the C version). It >> should also state that just because Fortran defines MPI_LONG_LONG, >> that does not imply that Fortran support an integer type that >> corresponds to the C long long type. >> >> Bill >> >> On Jul 17, 2008, at 4:03 AM, Rolf Rabenseifner wrote: >> >>> Hi all, >>> >>> The argument-chain was: >>> >>>>>> William Gropp wrote: >>>>>>> One piece of relevant text in MPI 2.0 is in section 4.12.6 (in >>>>>>> MPI-2.0) says that "All predefined datatypes can be used in >>>>>>> datatype constructors in any language." >>> [....] >>> >>> This text is supported also by: >>> MPI-2.0 Chap. 10.1.6 page 276 and >>> MPI-2.0 Annex B.2, page 344 define for all Fortran predefined >>> datatype handles as C++ counter-part. >>> >>> In the meeting, there was a consensus (?) that it is a bug >>> if an MPI implementation does not support, e.g. MPI_INT >>> in the mpif.h Fortran header file. >>> (This bug can be found, e.g., in MPICH-2 and will be fixed, >>> when I understood correctly) >>> >>> The value (not the name!!) of MPI_INT in mpif.h and mpi.h may be >>> different, as stated already in MPI-2.0, Sect. 4.12.9. >>> MPI_INT from C mpi.h cannot be used in a Fortran CALL MPI_SEND(...). >>> You need to convert, or you must use MPI_INT from Fortran mpif.h. >>> >>> (I.e. one may distiguish between mixed languages in >>> one MPI executable [addressed by MPI-2.0 Section 4.12.9], >>> and sending a message from a Fortran MPI executable to a C MPI >>> executable and vice versa [see MPI 2.0 Section 4.12.6].) >>> >>> Because this is already "clearly" defined in MPI-2.0, >>> you will not find any remark about this in the Change-Log >>> of MPI-2.1. >>> >>> MPI-2.2 may make it more clear in the revisiting of datatypes. >>> >>> Best regards >>> Rolf >>> >>> On Thu, 17 Jul 2008 10:20:28 +0200 >>> Hubert Ritzdorf wrote: >>>> No, we didn't have them. The request to provide all C datatypes in >>>> Fortran was specified in one of the last meetings. When I remember >>>> correctly, it was specified in the last meeting in Chicago or >>>> the meeting before (I think it was specified together with item 33 >>>> of the change log). I implemented it together with the changes >>>> for MPI 2.1 since I assumed that it was contained also in the >>>> change log. After your mail, I recognized that it is only >>>> implicitly >>>> mentioned by item 33. >>>> >>>> Hubert >>>> >>>> Jeff Squyres wrote: >>>>> What does NEC MPI do -- do you have MPI_INT and MPI_FLOAT >>>>> (etc.) in >>>>> mpif.h? >>>>> >>>>> >>>>> On Jul 16, 2008, at 12:57 PM, Hubert Ritzdorf wrote: >>>>> >>>>>> What's about the C++ datatypes for complex ? >>>>>> MPI_COMPLEX and MPI_Double_Complex are already defined for >>>>>> Fortran >>>>>> datatypes. The complex Fortran datatypes within C++ have a >>>>>> special >>>>>> name in order to avoid a name collision. >>>>>> >>>>>> Hubert >>>>>> >>>>>> William Gropp wrote: >>>>>>> One piece of relevant text in MPI 2.0 is in section 4.12.6 (in >>>>>>> MPI-2.0) says that "All predefined datatypes can be used in >>>>>>> datatype >>>>>>> constructors in any language." However, the example uses >>>>>>> MPI_Type_f2c ton convert the Fortran handle to the C version >>>>>>> before >>>>>>> passing it into the MPI_Type_create_struct routine, so this text >>>>>>> could be interpreted as allowing the use of those datatypes >>>>>>> through >>>>>>> the handle conversion mechanism without requiring them to be >>>>>>> defined >>>>>>> in each languages header file. Presumably the requirement is >>>>>>> that >>>>>>> they be predefined with the value that you'd get from the handle >>>>>>> conversion function? >>>>>>> >>>>>>> Bill >>>>>>> >>>>>>> On Jul 16, 2008, at 8:18 AM, Hubert Ritzdorf wrote: >>>>>>> >>>>>>>> Hi, >>>>>>>> >>>>>>>> when I remember correctly, it was decided to provide the C MPI >>>>>>>> datatypes >>>>>>>> in Fortran since >>>>>>>> the Fortran datatypes are supported in C and C++. >>>>>>>> >>>>>>>> Hubert >>>>>>>> >>>>>>>> Jeff Squyres wrote: >>>>>>>>> I will file a proposal about this if there is a problem, but I >>>>>>>>> wanted >>>>>>>>> to ask the group before I did so. In the changelog for >>>>>>>>> MPI-2.1: >>>>>>>>> >>>>>>>>> 2. Section 3.2.2 on page 27, Section 16.1.6 on page 453, and >>>>>>>>> Annex >>>>>>>>> A.1 >>>>>>>>> on page 491. >>>>>>>>> MPI_LONG_LONG_INT, MPI_LONG_LONG (as synonym), >>>>>>>>> MPI_UNSIGNED_LONG_LONG, >>>>>>>>> MPI_SIGNED_CHAR, and MPI_WCHAR are moved from optional to >>>>>>>>> official >>>>>>>>> and they >>>>>>>>> are therefore defined for all three language bindings. >>>>>>>>> >>>>>>>>> Note that it says that these C datatype names must be in all >>>>>>>>> three >>>>>>>>> language bindings. >>>>>>>>> >>>>>>>>> Are C MPI datatypes supposed to be available in Fortran? I >>>>>>>>> looked >>>>>>>>> in >>>>>>>>> mpif.h for Open MPI, MPICH2, HP MPI, and Intel MPI, and I >>>>>>>>> didn't >>>>>>>>> see C >>>>>>>>> MPI datatypes in there (e.g., MPI_INT and MPI_FLOAT). >>>>>>>>> >>>>>>>>> So: >>>>>>>>> >>>>>>>>> - is there language somewhere in MPI that says that C >>>>>>>>> datatypes are >>>>>>>>> not supposed to be in mpif.h? (and therefore the MPI-2.1 >>>>>>>>> changelog >>>>>>>>> is >>>>>>>>> wrong) >>>>>>>>> >>>>>>>>> or >>>>>>>>> >>>>>>>>> - are all 4 of the above MPI's wrong for not including the C >>>>>>>>> datatypes >>>>>>>>> in mpif.h? >>>>>>>>> >>>>>>>>> --Jeff Squyres >>>>>>>>> Cisco Systems >>>>>>>>> >>>>>>>>> >>>>>>>>> _______________________________________________ >>>>>>>>> mpi-22 mailing list >>>>>>>>> mpi-22_at_[hidden] >>>>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22 >>>>>>>>> >>>>>>>>> >>>>>>> >>>>>>> William Gropp >>>>>>> Paul and Cynthia Saylor Professor of Computer Science >>>>>>> University of Illinois Urbana-Champaign >>>>>>> >>>>>>> >>>>>>> >>>>>>> ---------------------------------------------------------------- >>>>>>> -- >>>>>>> ------ >>>>>>> >>>>>>> >>>>>>> _______________________________________________ >>>>>>> mpi-22 mailing list >>>>>>> mpi-22_at_[hidden] >>>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22 >>>>>>> >>>>>> _______________________________________________ >>>>>> mpi-22 mailing list >>>>>> mpi-22_at_[hidden] >>>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22 >>>>> >>>>> >>>>> --Jeff Squyres >>>>> Cisco Systems >>>>> >>>>> >>>>> _______________________________________________ >>>>> mpi-22 mailing list >>>>> mpi-22_at_[hidden] >>>>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22 >>>>> >>> >>> Dr. Rolf Rabenseifner . . . . . . . . . .. email >>> rabenseifner_at_[hidden] >>> High Performance Computing Center (HLRS) . phone >>> ++49(0)711/685-65530 >>> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / >>> 685-65832 >>> Head of Dpmt Parallel Computing . . . >>> www.hlrs.de/people/rabenseifner >>> Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30) >>> _______________________________________________ >>> mpi-22 mailing list >>> mpi-22_at_[hidden] >>> http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22 >> >> William Gropp >> Paul and Cynthia Saylor Professor of Computer Science >> University of Illinois Urbana-Champaign >> >> >> > > Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner_at_[hidden] > High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530 > University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832 > Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner > Nobelstr. 19, D-70550 Stuttgart, Germany . (Office: Allmandring 30) > _______________________________________________ > mpi-22 mailing list > mpi-22_at_[hidden] > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22 William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign * -------------- next part -------------- An HTML attachment was scrubbed... URL: From erezh at [hidden] Thu Jul 17 13:01:24 2008 From: erezh at [hidden] (Erez Haba) Date: Thu, 17 Jul 2008 11:01:24 -0700 Subject: [Mpi-22] [Mpi-forum] MPI 2.2 proposal: resolving MPI_Request_free issues In-Reply-To: <1BABDF0B0BF60D428576BCC7646C0A8F01DEA91A@rrsmsx411.amr.corp.intel.com> Message-ID: <6B68D01C00C9994A8E150183E62A119E730A151C57@NA-EXMSG-C105.redmond.corp.microsoft.com> -----Original Message----- From: mpi-22-bounces_at_[hidden] [mailto:mpi-22-bounces_at_[hidden]] On Behalf Of Underwood, Keith D Sent: Wednesday, July 16, 2008 4:32 PM To: MPI 2.2 Subject: Re: [Mpi-22] [Mpi-forum] MPI 2.2 proposal:resolving MPI_Request_free issues >> I don't quite understand examples 1 and 2 (how would they cause segv's >> in the TCP stack). It is permissible to (pseudocode): >> >> while (bytes_to_send > 0) { >> rc = write(fd, buffer, bytes_to_send); >> if (rc > 0) { >> buffer += rc; >> bytes_to_send -= rc; >> } else { >> ...error... >> } >> } >> free(buffer); >> >> regardless of what the receiver does. I'm not a kernel guy; does >> updating TCP sequence numbers also interact with the payload buffer? > >[erezh] it will never happen with your code above; but you are not using >async zcopy. >The pattern in windows is to use overlapped send (write) which is still >active when the function returns, and is the most efficient way to send >your buffer. I know it's possible with Linux but I don't' have the exact >pattern. > [Keith] > So, how does this work for other users of zero copy? How do they know > when the send buffer is truly free? [erezh] you get a notification from your local resource manager; on windows with the overlapped mechanism you have few ways to get notified. 1) The overlapped event get set, the thread can wait on it. 2) a completion port notification that the thread can dequeue. 3) an APC (similar to unix signal) get queued to the thread (thread need to be in an alertable state) From treumann at [hidden] Thu Jul 17 13:51:10 2008 From: treumann at [hidden] (Richard Treumann) Date: Thu, 17 Jul 2008 14:51:10 -0400 Subject: [Mpi-22] [Mpi-forum] MPI 2.2 proposal: resolving MPI_Request_free issues In-Reply-To: <6B68D01C00C9994A8E150183E62A119E730A151C57@NA-EXMSG-C105.redmond.corp.microsoft.com> Message-ID: On any MPI implementation the author of this application sequence: MPI_ISEND compute MPI_WAIT modify or free buffer will hope that the MPI_WAIT takes as little time as possible. The standard tells the user he can safely "reuse" the send buffer when the MPI_WAIT returns. If there are things which must happen before it is safe to free the buffer that do not need to happen before it is safe to modify its content, must an MPI implementation with the kind of issues Erez is suggesting delay the return from MPI_WAIT until both modify and free are safe? I suppose so - but it is unfortunate to need to delay the return from MPI_WAIT when it is safe for the user to do what he most likely has in mind (modify). As someone else pointed out, once the destination receive has finished, any packet retransmissions that are caused by ACK loss can harmlessly contain modified data. The send buffer does not need to be protected from updates. The destination will simply discard these packets them anyway. Also, in most systems support for malloc/free, the free() call does not change the ability of the adapter or VMM to modify the status of the address range. The address range that had been exposed to application use before the free() is simply added back to the heap and stays there until another malloc needs it. Doing a store to a memory range after a free() is likely to cause trouble but doing a load from ii is normally harmless. (I suppose someone will now tell me about a system where "free(ptr); x=*ptr;" causes segfaults. It is not the norm) Dick Dick Treumann - MPI Team/TCEM IBM Systems & Technology Group Dept 0lva / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 Tele (845) 433-7846 Fax (845) 433-8363 mpi-22-bounces_at_[hidden] wrote on 07/17/2008 02:01:24 PM: > > > -----Original Message----- > From: mpi-22-bounces_at_[hidden] [mailto:mpi-22- > bounces_at_[hidden]] On Behalf Of Underwood, Keith D > Sent: Wednesday, July 16, 2008 4:32 PM > To: MPI 2.2 > Subject: Re: [Mpi-22] [Mpi-forum] MPI 2.2 proposal:resolving > MPI_Request_free issues > > >> I don't quite understand examples 1 and 2 (how would they cause segv's > >> in the TCP stack). It is permissible to (pseudocode): > >> > >> while (bytes_to_send > 0) { > >> rc = write(fd, buffer, bytes_to_send); > >> if (rc > 0) { > >> buffer += rc; > >> bytes_to_send -= rc; > >> } else { > >> ...error... > >> } > >> } > >> free(buffer); > >> > >> regardless of what the receiver does. I'm not a kernel guy; does > >> updating TCP sequence numbers also interact with the payload buffer? > > > >[erezh] it will never happen with your code above; but you are not using > >async zcopy. > >The pattern in windows is to use overlapped send (write) which is still > >active when the function returns, and is the most efficient way to send > >your buffer. I know it's possible with Linux but I don't' have the exact > >pattern. > > > [Keith] > > So, how does this work for other users of zero copy? How do they know > > when the send buffer is truly free? > > [erezh] you get a notification from your local resource manager; on > windows with the overlapped mechanism you have few ways to get > notified. 1) The overlapped event get set, the thread can wait on > it. 2) a completion port notification that the thread can dequeue. > 3) an APC (similar to unix signal) get queued to the thread (thread > need to be in an alertable state) > > _______________________________________________ > mpi-22 mailing list > mpi-22_at_[hidden] > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22 * -------------- next part -------------- An HTML attachment was scrubbed... URL: From lindahl at [hidden] Thu Jul 17 14:11:30 2008 From: lindahl at [hidden] (Greg Lindahl) Date: Thu, 17 Jul 2008 12:11:30 -0700 Subject: [Mpi-22] [Mpi-forum] MPI 2.2 proposal: resolving MPI_Request_free issues In-Reply-To: Message-ID: <20080717191128.GA21009@bx9.net> On Thu, Jul 17, 2008 at 02:51:10PM -0400, Richard Treumann wrote: > Also, in most systems support for malloc/free, the free() call does not > change the ability of the adapter or VMM to modify the status of the > address range. The address range that had been exposed to application use > before the free() is simply added back to the heap and stays there until > another malloc needs it. glibc issues individual mmap()s for large allocations. These are returned to the OS on free. Also, if the top of the heap of small allocations is all free (rare due to fragmentation), it will sbrk backwards. Adaptors which have to maintain virt->phys mappings all struggle with this, in one way or another. I'm finding this whole discussion confusing. On all of the interconnects that I'm familiar with, freeing and modifying the buffer are both unsafe before the wait, and safe after. There's no free lunch. Am I not thinking of some important example? -- greg From erezh at [hidden] Thu Jul 17 14:40:55 2008 From: erezh at [hidden] (Erez Haba) Date: Thu, 17 Jul 2008 12:40:55 -0700 Subject: [Mpi-22] [Mpi-forum] MPI 2.2 proposal: resolving MPI_Request_free issues In-Reply-To: <9604850B-F646-4CBA-A91F-FF3FF442B72F@cisco.com> Message-ID: <6B68D01C00C9994A8E150183E62A119E730A151CEA@NA-EXMSG-C105.redmond.corp.microsoft.com> The problem with solution #5 is that it still might be incorrect. It is okay to read the buffer while the local interconnect is still using it but it's not always the case with write. For example, it is okay for the interconnect to probe the memory again before releasing it. It can use probe for write (which is odd, but can happen). This probe is usually implemented as read-write back, if the probe happens as the same time the user is updating the data; the user data might get corrupted. I know it is farfetched, but I'd be extremely cautions recommending the user to write the buffer before the local resource "owner" (which is the interconnect) relinquish it back. As a solution I would put text that discourage the use for MPI_Request_free for active request; and recommend using Adam Moody proposal for MPI_REQUEST_IGNORE. (we might not be able to make this an error case in 2.2 to keep backward compatibility) The text specifically would say that writing or freeing the buffer after calling MPI_Request_free is unsafe. Thanks, .Erez -----Original Message----- From: mpi-22-bounces_at_[hidden] [mailto:mpi-22-bounces_at_[hidden]] On Behalf Of Jeff Squyres Sent: Wednesday, July 16, 2008 6:59 PM To: MPI 2.2 Subject: Re: [Mpi-22] [Mpi-forum] MPI 2.2 proposal:resolving MPI_Request_free issues On Jul 16, 2008, at 7:31 PM, Underwood, Keith D wrote: >>> [erezh] Correct; this is the scenario I was describing with the RDMA write. > > It would be interesting to see exactly what the error mode here is. > Retransmitting corrupted data should be ok, since a correctly > delivered > message means that the retransmit must be dropped. I suppose that if > the NIC speaks virtual addresses and the free actually results in a > trap > to the kernel that unmaps the pages, then the NIC could retransmit and > find that there isn't a valid page table entry... 'zactly. And then the local completion entry would be a failure -- so the sender would [erroneously] think that the message had failed to be delivered. > Solution #5: Change the advice to users - "...the arrival of the > reply > informs the sender that the send has completed and the send buffer can > be overwritten. If the buffer will ever be freed, the application > should call MPI_Wait or MPI_Cancel instead of MPI_Request_free." That seems like a good compromise. -- Jeff Squyres Cisco Systems _______________________________________________ mpi-22 mailing list mpi-22_at_[hidden] http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22 From treumann at [hidden] Thu Jul 17 14:46:59 2008 From: treumann at [hidden] (Richard Treumann) Date: Thu, 17 Jul 2008 15:46:59 -0400 Subject: [Mpi-22] [Mpi-forum] MPI 2.2 proposal: resolving MPI_Request_free issues In-Reply-To: <20080717191128.GA21009@bx9.net> Message-ID: Sorry - the discussion is in the context of MPI_REQUEST_FREE. MPI_ISEND(question_buffer ..... request) MPI_REQUEST_FREE(request) MPI_RECV( reply_buffer) free(question_buffer) Where the MPI_RECV is for a reply that will come back from the task the ISEND went to after the ISEND message is received and the completion of the send side MPI_RECV is taken as sufficient for calling free on the ISEND buffer Dick Treumann - MPI Team/TCEM IBM Systems & Technology Group Dept 0lva / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 Tele (845) 433-7846 Fax (845) 433-8363 mpi-22-bounces_at_[hidden] wrote on 07/17/2008 03:11:30 PM: > On Thu, Jul 17, 2008 at 02:51:10PM -0400, Richard Treumann wrote: > > > Also, in most systems support for malloc/free, the free() call does not > > change the ability of the adapter or VMM to modify the status of the > > address range. The address range that had been exposed to application use > > before the free() is simply added back to the heap and stays there until > > another malloc needs it. > > glibc issues individual mmap()s for large allocations. These are > returned to the OS on free. Also, if the top of the heap of small > allocations is all free (rare due to fragmentation), it will sbrk > backwards. Adaptors which have to maintain virt->phys mappings all > struggle with this, in one way or another. > > I'm finding this whole discussion confusing. On all of the > interconnects that I'm familiar with, freeing and modifying the buffer > are both unsafe before the wait, and safe after. There's no free > lunch. Am I not thinking of some important example? > > -- greg > > > _______________________________________________ > mpi-22 mailing list > mpi-22_at_[hidden] > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22 * -------------- next part -------------- An HTML attachment was scrubbed... URL: From erezh at [hidden] Thu Jul 17 15:33:41 2008 From: erezh at [hidden] (Erez Haba) Date: Thu, 17 Jul 2008 13:33:41 -0700 Subject: [Mpi-22] [Mpi-forum] MPI 2.2 proposal: resolving MPI_Request_free issues In-Reply-To: <20080717191128.GA21009@bx9.net> Message-ID: <6B68D01C00C9994A8E150183E62A119E730A151D46@NA-EXMSG-C105.redmond.corp.microsoft.com> I agree with you 100% On windows pinning down the memory for registration cache is enabled by having a callback from the OS when the memory is actually freed (returned back to the OS); thus enabling the implementation of a safe registration cache with no segv or memory leak. .Erez -----Original Message----- From: mpi-22-bounces_at_[hidden] [mailto:mpi-22-bounces_at_[hidden]] On Behalf Of Greg Lindahl Sent: Thursday, July 17, 2008 12:12 PM To: mpi-22_at_[hidden] Subject: Re: [Mpi-22] [Mpi-forum] MPI 2.2 proposal: resolving MPI_Request_free issues On Thu, Jul 17, 2008 at 02:51:10PM -0400, Richard Treumann wrote: > Also, in most systems support for malloc/free, the free() call does not > change the ability of the adapter or VMM to modify the status of the > address range. The address range that had been exposed to application use > before the free() is simply added back to the heap and stays there until > another malloc needs it. glibc issues individual mmap()s for large allocations. These are returned to the OS on free. Also, if the top of the heap of small allocations is all free (rare due to fragmentation), it will sbrk backwards. Adaptors which have to maintain virt->phys mappings all struggle with this, in one way or another. I'm finding this whole discussion confusing. On all of the interconnects that I'm familiar with, freeing and modifying the buffer are both unsafe before the wait, and safe after. There's no free lunch. Am I not thinking of some important example? -- greg _______________________________________________ mpi-22 mailing list mpi-22_at_[hidden] http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22 From treumann at [hidden] Thu Jul 17 15:57:22 2008 From: treumann at [hidden] (Richard Treumann) Date: Thu, 17 Jul 2008 16:57:22 -0400 Subject: [Mpi-22] [Mpi-forum] MPI 2.2 proposal: resolving MPI_Request_free issues In-Reply-To: <6B68D01C00C9994A8E150183E62A119E730A151CEA@NA-EXMSG-C105.redmond.corp.microsoft.com> Message-ID: How is this different from saying MPI_REQUEST_FREE was a mistake. The behavior of an application that uses this call is undefined. Of course we could add MPI_NO_REQUEST_FREE as an MPI_Init time assertion and say that any MPI implementation optimization that create the problems Erez describes are only valid when the application has asserted MPI_NO_REQUEST_FREE. Dick Treumann - MPI Team/TCEM IBM Systems & Technology Group Dept 0lva / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 Tele (845) 433-7846 Fax (845) 433-8363 mpi-22-bounces_at_[hidden] wrote on 07/17/2008 03:40:55 PM: > > The text specifically would say that writing or freeing the buffer > after calling MPI_Request_free is unsafe. > > Thanks, > .Erez > * -------------- next part -------------- An HTML attachment was scrubbed... URL: From erezh at [hidden] Thu Jul 17 16:11:02 2008 From: erezh at [hidden] (Erez Haba) Date: Thu, 17 Jul 2008 14:11:02 -0700 Subject: [Mpi-22] [Mpi-forum] MPI 2.2 proposal: resolving MPI_Request_free issues In-Reply-To: Message-ID: <6B68D01C00C9994A8E150183E62A119E730A151D7C@NA-EXMSG-C105.redmond.corp.microsoft.com> It is not different for active requests; I think it was a mistake to define it the way it is. You can still use MPI_Request_free to free requests that are allocated but not started; which is why I think we should keep it, but change the semantic. From: mpi-22-bounces_at_[hidden] [mailto:mpi-22-bounces_at_[hidden]] On Behalf Of Richard Treumann Sent: Thursday, July 17, 2008 1:57 PM To: MPI 2.2 Subject: Re: [Mpi-22] [Mpi-forum] MPI 2.2 proposal: resolving MPI_Request_free issues How is this different from saying MPI_REQUEST_FREE was a mistake. The behavior of an application that uses this call is undefined. Of course we could add MPI_NO_REQUEST_FREE as an MPI_Init time assertion and say that any MPI implementation optimization that create the problems Erez describes are only valid when the application has asserted MPI_NO_REQUEST_FREE. Dick Treumann - MPI Team/TCEM IBM Systems & Technology Group Dept 0lva / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 Tele (845) 433-7846 Fax (845) 433-8363 mpi-22-bounces_at_[hidden] wrote on 07/17/2008 03:40:55 PM: > > The text specifically would say that writing or freeing the buffer > after calling MPI_Request_free is unsafe. > > Thanks, > .Erez > * -------------- next part -------------- An HTML attachment was scrubbed... URL: From lindahl at [hidden] Thu Jul 17 16:14:42 2008 From: lindahl at [hidden] (Greg Lindahl) Date: Thu, 17 Jul 2008 14:14:42 -0700 Subject: [Mpi-22] [Mpi-forum] MPI 2.2 proposal: resolving MPI_Request_free issues In-Reply-To: Message-ID: <20080717211441.GA31804@bx9.net> On Thu, Jul 17, 2008 at 03:46:59PM -0400, Richard Treumann wrote: > Sorry - the discussion is in the context of MPI_REQUEST_FREE. > > MPI_ISEND(question_buffer ..... request) > MPI_REQUEST_FREE(request) > MPI_RECV( reply_buffer) > free(question_buffer) Perhaps an implementer would dodge this bullet by having MPI_REQUEST_FREE do a MPI_CANCEL before freeing anything? That's assuming you have MPI_CANCEL on a send do the right thing. Thus, we turn this new can of worms into an existing can of worms. Voila! The example above strikes me as being undefined behavior in any case. I don't think we would guarantee that the message got out at all, and even if the remote side thinks the message is complete, that doesn't mean that the sender agrees the message is complete, without calling WAIT. -- greg From treumann at [hidden] Thu Jul 17 16:29:36 2008 From: treumann at [hidden] (Richard Treumann) Date: Thu, 17 Jul 2008 17:29:36 -0400 Subject: [Mpi-22] [Mpi-forum] MPI 2.2 proposal: resolving MPI_Request_free issues In-Reply-To: <20080717211441.GA31804@bx9.net> Message-ID: But the MPI standard says this is valid and can be used to avoid the need to keep track of ISEND request handles. It is clear that if the matching RECV has completed then the unaltered content of the ISEND buffer is no longer needed. The RECV cannot have completed unless the message got out. Erez is pointing out a case where the semantic stipulated by the MPI standard back before RDMA adapters were available becomes flawed with the way RDMA adapters now work. The content of the ISEND buffer is irrelevant once the destination RECV has completed but some other things about the ISEND buffer are still in limbo Dick Treumann - MPI Team/TCEM IBM Systems & Technology Group Dept 0lva / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 Tele (845) 433-7846 Fax (845) 433-8363 mpi-22-bounces_at_[hidden] wrote on 07/17/2008 05:14:42 PM: > On Thu, Jul 17, 2008 at 03:46:59PM -0400, Richard Treumann wrote: > > Sorry - the discussion is in the context of MPI_REQUEST_FREE. > > > > MPI_ISEND(question_buffer ..... request) > > MPI_REQUEST_FREE(request) > > MPI_RECV( reply_buffer) > > free(question_buffer) > > Perhaps an implementer would dodge this bullet by having > MPI_REQUEST_FREE do a MPI_CANCEL before freeing anything? That's > assuming you have MPI_CANCEL on a send do the right thing. > > Thus, we turn this new can of worms into an existing can of > worms. Voila! > > The example above strikes me as being undefined behavior in any case. > I don't think we would guarantee that the message got out at all, and > even if the remote side thinks the message is complete, that doesn't > mean that the sender agrees the message is complete, without calling > WAIT. > > -- greg > > _______________________________________________ > mpi-22 mailing list > mpi-22_at_[hidden] > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22 * -------------- next part -------------- An HTML attachment was scrubbed... URL: From lindahl at [hidden] Thu Jul 17 16:37:12 2008 From: lindahl at [hidden] (Greg Lindahl) Date: Thu, 17 Jul 2008 14:37:12 -0700 Subject: [Mpi-22] [Mpi-forum] MPI 2.2 proposal: resolving MPI_Request_free issues In-Reply-To: Message-ID: <20080717213711.GB31804@bx9.net> On Thu, Jul 17, 2008 at 05:29:36PM -0400, Richard Treumann wrote: > It is clear that if the matching > RECV has completed then the unaltered content of the ISEND buffer is no > longer needed. This isn't clear to me. There's certainly no promise of that in the MPI-1 standard text on the web. That's why I said I thought it should be undefined. This is "spooky action at a distance" for me. We shouldn't encourage it. And I still think it's the same can of worms as MPI_Cancel. If MPI_Cancel works, then an RDMA-using MPI can simple call it at the start of the request_free call. -- greg From keith.d.underwood at [hidden] Thu Jul 17 22:39:32 2008 From: keith.d.underwood at [hidden] (Underwood, Keith D) Date: Thu, 17 Jul 2008 22:39:32 -0500 Subject: [Mpi-22] [Mpi-forum] MPI 2.2 proposal:resolving MPI_Request_free issues In-Reply-To: <20080717213711.GB31804@bx9.net> Message-ID: <1BABDF0B0BF60D428576BCC7646C0A8F01DEAC8D@rrsmsx411.amr.corp.intel.com> >And I still think it's the same can of worms as MPI_Cancel. If >MPI_Cancel works, then an RDMA-using MPI can simple call it at the >start of the request_free call. That was kind of my point as well, but I think I was too subtle, as it seems people took me a little too seriously with that suggestion :-) Keith From keith.d.underwood at [hidden] Thu Jul 17 22:48:19 2008 From: keith.d.underwood at [hidden] (Underwood, Keith D) Date: Thu, 17 Jul 2008 22:48:19 -0500 Subject: [Mpi-22] [Mpi-forum] MPI 2.2 proposal: resolving MPI_Request_free issues In-Reply-To: Message-ID: <1BABDF0B0BF60D428576BCC7646C0A8F01DEAC8F@rrsmsx411.amr.corp.intel.com> > But the MPI standard says this is valid and can be used to avoid the need > to keep track of ISEND request handles. It is clear that if the matching > RECV has completed then the unaltered content of the ISEND buffer is no > longer needed. The RECV cannot have completed unless the message got out. I tend to agree with this sentiment, but the MPI standard has long had a prohibition on touching the transmit buffer, because you never know what the implementation is using that buffer for. So... anything that ever had a reliability protocol and did zero copy has had this issue. > Erez is pointing out a case where the semantic stipulated by the MPI > standard back before RDMA adapters were available becomes flawed with the > way RDMA adapters now work. The content of the ISEND buffer is irrelevant > once the destination RECV has completed but some other things about the > ISEND buffer are still in limbo I don't believe this is new. Anything that ever provided a reason that you couldn't touch the send buffer likely had a similar issue. Keith From ritzdorf at [hidden] Fri Jul 18 07:06:54 2008 From: ritzdorf at [hidden] (Hubert Ritzdorf) Date: Fri, 18 Jul 2008 14:06:54 +0200 Subject: [Mpi-22] MPI-2.1 ambiguity? In-Reply-To: Message-ID: <4880875E.8050403@it.neclab.eu> Rolf Rabenseifner wrote: > My primary oal is: > We should make clarifications which state that > each predefined handle must be available in every > language binding. > People may be confused by current wording in several places: > > - MPI-2.1 (June 23, 2008) Section 16.3.6 MPI Opaque Objects, > page 483, lines 46-47: > "All predefined datatypes can be used in datatype constructors > in any language." > > Does this mean, that the C++ datatypes BOOL, COMPLEX, DOUBLE_COMPLEX and LONG_DOUBLE_COMPLEX have to be available in C and Fortran ? If yes, what are the names of them in C and Fortran ? Or is the statement in Annex 1.1 (MPI 2.1, page 495, line 1) "C++-Only Named Predefined Datatypes" indicating, that they are not available in C and Fortran versions of MPI 2.1 ? In this case, the sentence above should be corrected. Hubert * -------------- next part -------------- A non-text attachment was scrubbed... Name: smime.p7s Type: application/x-pkcs7-signature Size: 3245 bytes Desc: S/MIME Cryptographic Signature URL: From jsquyres at [hidden] Fri Jul 18 13:02:41 2008 From: jsquyres at [hidden] (Jeff Squyres) Date: Fri, 18 Jul 2008 14:02:41 -0400 Subject: [Mpi-22] MPI-2.1 ambiguity? In-Reply-To: Message-ID: On Jul 17, 2008, at 12:41 PM, William Gropp wrote: > There's another option. Drop the requirement that all of the > predefined datatypes are predefined in all languages, but specify > that they may be used (with the appropriate handle conversion) as > input to any routine where it makes sense. Just curious: if you drop the requirement for direct access to all the predefined datatypes in all languages, why do you need to specify anything else? MPI-2.1:16.3.4 (transfer of handles -- the *f2c/*c2f stuff) seems to be sufficient. Is there a need to say something more? Further, if we drop the requirement, do we need a "high quality implementation" note about making all predefined handles available in all languages for MPI programmer convenience? I.e., what reasonable expectation should an MPI programmer have (particularly for already- existing codes) about accessing language A's predefined handles in language B? Should they *always* call C to do the translation? > For datatypes, this includes the datatype construction routines but > not the communication routines (don't use MPI_INT when sending > INTEGERs with Fortran). This is what the Opaque Objects text expects. To me, dropping the requirement sounds like we're just making the user's life hard -- a Fortran library may have to introduce C code into their project, and then figure out all the cross-language compiling and linking stuff (which is not for the meek). It might be a whole lot simpler to specifically define which predefined handles need to be available in each language. The spectrum ranges from "all languages' predefined handles must be available in every language" to "only that language A's predefined handles must be available in language A". Suggestions? -- Jeff Squyres Cisco Systems From wgropp at [hidden] Fri Jul 18 13:21:49 2008 From: wgropp at [hidden] (William Gropp) Date: Fri, 18 Jul 2008 13:21:49 -0500 Subject: [Mpi-22] MPI-2.1 ambiguity? In-Reply-To: Message-ID: <7AA28189-1393-42C2-B330-68A672A89CBD@illinois.edu> Notes inline. On Jul 18, 2008, at 1:02 PM, Jeff Squyres wrote: > On Jul 17, 2008, at 12:41 PM, William Gropp wrote: > >> There's another option. Drop the requirement that all of the >> predefined datatypes are predefined in all languages, but specify >> that they may be used (with the appropriate handle conversion) as >> input to any routine where it makes sense. > > Just curious: if you drop the requirement for direct access to all the > predefined datatypes in all languages, why do you need to specify > anything else? MPI-2.1:16.3.4 (transfer of handles -- the *f2c/*c2f > stuff) seems to be sufficient. Is there a need to say something more? The implicit assumption you are making is that the existence of the handle transfer functions means that the handles can be used everywhere. You can certainly argue that nothing else is needed, but MPI 2.0 felt that it was important to make it clear that this applied to the datatype routines, even if those datatypes do not exist in the calling language. In other words, you can't have different versions of MPI_Type_xxx that only know about the predefined datatypes in the calling language. Without this clarification, we could be having that argument instead :) > > Further, if we drop the requirement, do we need a "high quality > implementation" note about making all predefined handles available in > all languages for MPI programmer convenience? I.e., what reasonable > expectation should an MPI programmer have (particularly for already- > existing codes) about accessing language A's predefined handles in > language B? Should they *always* call C to do the translation? > >> For datatypes, this includes the datatype construction routines but >> not the communication routines (don't use MPI_INT when sending >> INTEGERs with Fortran). This is what the Opaque Objects text >> expects. > > To me, dropping the requirement sounds like we're just making the > user's life hard -- a Fortran library may have to introduce C code > into their project, and then figure out all the cross-language > compiling and linking stuff (which is not for the meek). > > It might be a whole lot simpler to specifically define which > predefined handles need to be available in each language. The > spectrum ranges from "all languages' predefined handles must be > available in every language" to "only that language A's predefined > handles must be available in language A". Suggestions? I'm in favor of explicitly specifying the values and names (because of the need to use alternate names when their might be a conflict, as in the MPI::F_COMPLEX case), and that these have the exact same value as if the handle transfer function was used. Bill > > -- > Jeff Squyres > Cisco Systems > > _______________________________________________ > mpi-22 mailing list > mpi-22_at_[hidden] > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22 William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign * -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsquyres at [hidden] Fri Jul 18 13:36:32 2008 From: jsquyres at [hidden] (Jeff Squyres) Date: Fri, 18 Jul 2008 14:36:32 -0400 Subject: [Mpi-22] MPI-2.1 ambiguity? In-Reply-To: <7AA28189-1393-42C2-B330-68A672A89CBD@illinois.edu> Message-ID: <69D6D088-C9C2-414B-9933-C5FE8B3005D3@cisco.com> On Jul 18, 2008, at 2:21 PM, William Gropp wrote: >> It might be a whole lot simpler to specifically define which >> predefined handles need to be available in each language. The >> spectrum ranges from "all languages' predefined handles must be >> available in every language" to "only that language A's predefined >> handles must be available in language A". Suggestions? > > I'm in favor of explicitly specifying the values and names (because > of the need to use alternate names when their might be a conflict, > as in the MPI::F_COMPLEX case), and that these have the exact same > value as if the handle transfer function was used. Do we really need to mandate that they have the same value, or just exhibit the same behavior? Specifically does it matter if this code hits branch A or B? MPI_Fint ftype; ftype = MPI_Type_c2f(MPI_INT); if (MPI_INT == MPI_Type_f2c(ftype)) { //branch A } else { //branch B } Or is it simpler to say that the return from an operation like the above must behave exactly like the original, regardless of value? -- Jeff Squyres Cisco Systems From treumann at [hidden] Fri Jul 18 13:57:19 2008 From: treumann at [hidden] (Richard Treumann) Date: Fri, 18 Jul 2008 14:57:19 -0400 Subject: [Mpi-22] [Mpi-forum] MPI 2.2 proposal: resolving MPI_Request_free issues In-Reply-To: <1BABDF0B0BF60D428576BCC7646C0A8F01DEAC8F@rrsmsx411.amr.corp.intel.com> Message-ID: The semantic provided by using MPI_REQUEST_FREE on an active send request is clearly useful and is clearly mandated by MPI 1.1. It seems to have been a mistake to assume it would always be practical to support this symantic in MPI implementations and we can discuss removing the guarentee or making it optional via MPI_INIT time assertion. I do not see how we can suggest the intent was ever vague. USEFULNESS: The semantic is useful for what amounts to a home rolled get. Say all tasks know that at some point in each step they may need to provide data to other tasks. They do not know what tasks will want which data so they post an IRECV for any incoming data pull commands and TEST or they do IPROBEs and post RECVs when it comes time to accept data pulls. Each RECVed message at a get target represents some remote task describing the data it wants. The task that sent the data pull command (the get origin) posts a receive suitable for the data it requested. The data cannot be sent from the get target until the RECV of the pull command is complete so once the get origin completes the RECV of the data it requested it can be sure the current content of the send buffer with the pull command is obsolete. By immediately after MPI_ISEND, using MPI_REQUEST_FREE on the request handle from the data pull command, the application is spared the need to keep track of that handle until after the data comes in. Immediately calling MPI_WAIT on the ISEND is safe and removes the need to keep track of the handle but delays the application. Saving the request handle to call MPI_WAIT after the pulled data comes in will give good performance because by the time the data has come in the MPI_WAIT should normally be very fast. It does require keeping track of all data pull handles though. There is no magic in reaching a conclusion at the get side that receiving the pulled data means the content of the data pull command buffer is no longer needed. The problem is that the arrival of the reply may not always guarantee that other attributes of the buffer status are complete. MPI 1.1 statement: [] Advice to users. Once a request is freed by a call to MPI_REQUEST_FREE, it is not possible to check for the successful completion of the associated communication with calls to MPI_WAIT or MPI_TEST. Also, if an error occurs subsequently during the communication, an error code cannot be returned to the user --- such an error must be treated as fatal. Questions arise as to how one knows when the operations have completed when using MPI_REQUEST_FREE. Depending on the program logic, there may be other ways in which the program knows that certain operations have completed and this makes usage of MPI_REQUEST_FREE practical. For example, an active send request could be freed when the logic of the program is such that the receiver sends a reply to the message sent --- the arrival of the reply informs the sender that the send has completed and the send buffer can be reused. An active receive request should never be freed as the receiver will have no way to verify that the receive has completed and the receive buffer can be reused. ( End of advice to users.) Dick Dick Treumann - MPI Team/TCEM IBM Systems & Technology Group Dept 0lva / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 Tele (845) 433-7846 Fax (845) 433-8363 * -------------- next part -------------- An HTML attachment was scrubbed... URL: From wgropp at [hidden] Fri Jul 18 14:08:36 2008 From: wgropp at [hidden] (William Gropp) Date: Fri, 18 Jul 2008 14:08:36 -0500 Subject: [Mpi-22] MPI-2.1 ambiguity? In-Reply-To: <69D6D088-C9C2-414B-9933-C5FE8B3005D3@cisco.com> Message-ID: <76C6E6B2-AC6C-40FE-A9B7-2183314BFA4A@illinois.edu> Any behavior other than branch A violates the principle of least surprise. Bill On Jul 18, 2008, at 1:36 PM, Jeff Squyres wrote: > On Jul 18, 2008, at 2:21 PM, William Gropp wrote: > >>> It might be a whole lot simpler to specifically define which >>> predefined handles need to be available in each language. The >>> spectrum ranges from "all languages' predefined handles must be >>> available in every language" to "only that language A's predefined >>> handles must be available in language A". Suggestions? >> >> I'm in favor of explicitly specifying the values and names (because >> of the need to use alternate names when their might be a conflict, >> as in the MPI::F_COMPLEX case), and that these have the exact same >> value as if the handle transfer function was used. > > Do we really need to mandate that they have the same value, or just > exhibit the same behavior? Specifically does it matter if this code > hits branch A or B? > > MPI_Fint ftype; > ftype = MPI_Type_c2f(MPI_INT); > if (MPI_INT == MPI_Type_f2c(ftype)) { > //branch A > } else { > //branch B > } > > Or is it simpler to say that the return from an operation like the > above must behave exactly like the original, regardless of value? > > -- > Jeff Squyres > Cisco Systems > > _______________________________________________ > mpi-22 mailing list > mpi-22_at_[hidden] > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22 William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign * -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsquyres at [hidden] Fri Jul 18 15:16:33 2008 From: jsquyres at [hidden] (Jeff Squyres) Date: Fri, 18 Jul 2008 16:16:33 -0400 Subject: [Mpi-22] MPI-2.1 ambiguity? In-Reply-To: <76C6E6B2-AC6C-40FE-A9B7-2183314BFA4A@illinois.edu> Message-ID: <6AF50A9E-801F-4E83-97FF-98A1796BE27F@cisco.com> On Jul 18, 2008, at 3:08 PM, William Gropp wrote: > Any behavior other than branch A violates the principle of least > surprise. I assume that most MPI implementations will have the values be equal just by how their f2c/c2f engines work (OMPI will). I was just wondering if it's necessary to *mandate* that behavior. Do apps really check handle equality? If so, I agree that anything other than path A violates the law of least astonishment. I assumed that not many apps do this, but I probably don't have a good basis for reference. -- Jeff Squyres Cisco Systems From wgropp at [hidden] Fri Jul 18 15:33:28 2008 From: wgropp at [hidden] (William Gropp) Date: Fri, 18 Jul 2008 15:33:28 -0500 Subject: [Mpi-22] MPI-2.1 ambiguity? In-Reply-To: <6AF50A9E-801F-4E83-97FF-98A1796BE27F@cisco.com> Message-ID: <030D35D5-7FFA-4395-943D-080481BAB807@illinois.edu> Its quite common to see code that isn't even valid for MPI, such as a switch statement using predefined datatypes. As there isn't any want to check handle similarity for datatypes, we really need to require equality. As you note, this doesn't affect any implementations. Bill On Jul 18, 2008, at 3:16 PM, Jeff Squyres wrote: > On Jul 18, 2008, at 3:08 PM, William Gropp wrote: > >> Any behavior other than branch A violates the principle of least >> surprise. > > I assume that most MPI implementations will have the values be equal > just by how their f2c/c2f engines work (OMPI will). I was just > wondering if it's necessary to *mandate* that behavior. > > Do apps really check handle equality? If so, I agree that anything > other than path A violates the law of least astonishment. I assumed > that not many apps do this, but I probably don't have a good basis for > reference. > > -- > Jeff Squyres > Cisco Systems > > _______________________________________________ > mpi-22 mailing list > mpi-22_at_[hidden] > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi-22 William Gropp Paul and Cynthia Saylor Professor of Computer Science University of Illinois Urbana-Champaign * -------------- next part -------------- An HTML attachment was scrubbed... URL: From lindahl at [hidden] Mon Jul 21 22:15:14 2008 From: lindahl at [hidden] (Greg Lindahl) Date: Mon, 21 Jul 2008 20:15:14 -0700 Subject: [Mpi-22] [Mpi-forum] MPI 2.2 proposal: resolving MPI_Request_free issues In-Reply-To: Message-ID: <20080722031513.GA14415@bx9.net> On Fri, Jul 18, 2008 at 02:57:19PM -0400, Richard Treumann wrote: > The semantic provided by using MPI_REQUEST_FREE on an active send request > is clearly useful and is clearly mandated by MPI 1.1. My apologies, I wasn't reading the right part of the standard. It is clear and it is mandated, and it does restrict implementations in a way that seems odd. The sending side might have some cleanup work before it thinks the send is complete, yet the recieving side might conclude that the message is complete before the sender does. That's the 'spooky action at a distance' problem I was referring to. Now, the reason I was confused? When you google MPI_REQUEST_FREE, it lands you on the MPICH-1 manpage entry for MPI_REQUEST_FREE, which lacks most of the details of section 3.7.3 on the standard. At the least the manpages ought to refer to/link to the standard, or else they could be replaced by (for example) the OpenMPI manpages, which are much more complete. This doesn't excuse my carelessness, of course, but I'm probably not the only idiot googling routine names expecting complete info. -- greg