From alexander.supalov at [hidden] Wed Mar 5 08:38:09 2008 From: alexander.supalov at [hidden] (Supalov, Alexander) Date: Wed, 5 Mar 2008 14:38:09 -0000 Subject: [Mpi3-subsetting] Subsetting slides for F2F meeting: please comment by EOD March 6, 2008 Message-ID: <5ECAB1304A8B5B4CB3F9D6C01E4E21A2011CC335@swsmsx413.ger.corp.intel.com> Hi everybody, Here come the promised draft slides for the subsetting presentation at the MPI Forum. I put this into our corporate template, I hope everybody is OK with this. Your comments will be most appreciated. I would like to finalize the slides on Friday, so please send your feedback by EOD your time on Thursday, March 6, 2008. Sorry for the message size. I cannot upload this file to the Wiki due to file size limitation there. I work this with Rich in background. Best regards. Alexander -- Dr Alexander Supalov Intel GmbH Hermuelheimer Strasse 8a 50321 Bruehl, Germany Phone: +49 2232 209034 Mobile: +49 173 511 8735 Fax: +49 2232 209029 --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. * -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: MPI-3_subsetting.ppt Type: application/vnd.ms-powerpoint Size: 425472 bytes Desc: MPI-3 subsetting.ppt URL: From treumann at [hidden] Wed Mar 5 10:46:42 2008 From: treumann at [hidden] (Richard Treumann) Date: Wed, 5 Mar 2008 11:46:42 -0500 Subject: [Mpi3-subsetting] Subsetting slides In-Reply-To: Message-ID: One argument for subsetting is that MPI is somehow not well suited to massive master-worker applications. It does seem reasonable to me that pure master-worker applications have little use for many of MPI's features. Most communication may be point to point because workers are not expected to cooperate or communicate among themselves. Groups, topologies, new communicators, MPI-IO etc are not very useful. In the simple case, the master hands each worker a piece of work to do and the worker runs on its own until the work is done. As a worker finishes, it sends an answer to the master and waits to be handed another piece of work. Some pieces of work are brief and some are not so workers are not synched with each other and the master does not know who will have an answer next. My impression is that the biggest problem with MPI for this model is that with one master and many workers, it is not hard to make an application that can survive the loss of one or more workers. The master just hands out the lost work assignment again. MPI is not an ideal fit because the loss of one member of MPI_COMM_WORLD will probably bring down the job. A simple subset will not resolve this. An Fault Tolerant subset could be considered if this is an important rationale for a subset. Also, lots of people seem to like the idea of getting rid of MPI_ANY_SOURCE in a subset. I am one of these people. MPI_ANY_SOURCE is a problem. The difficulty is that the master in a master worker algorithm is likely to find MPI_ANY_SOURCE and MPI_PROBE useful. Dick Dick Treumann - MPI Team/TCEM IBM Systems & Technology Group Dept 0lva / MS P963 -- 2455 South Road -- Poughkeepsie, NY 12601 Tele (845) 433-7846 Fax (845) 433-8363 * -------------- next part -------------- An HTML attachment was scrubbed... URL: From rbarrett at [hidden] Wed Mar 5 11:26:05 2008 From: rbarrett at [hidden] (Richard Barrett) Date: Wed, 05 Mar 2008 12:26:05 -0500 Subject: [Mpi3-subsetting] mpi3-subsetting Digest, Vol 2, Issue 3 In-Reply-To: Message-ID: > Also, lots of people seem to like the idea of getting rid of MPI_ANY_SOURCE > in a subset. I am one of these people. MPI_ANY_SOURCE is a problem. The > difficulty is that the master in a master worker algorithm is likely to > find MPI_ANY_SOURCE and MPI_PROBE useful. This issue captures a (the?) key decision that this group must make imo. Is the goal of the sub-setting to provide a performance boost, or to address the "mpi is too big" concern? In some sense the concerns overlap, but here is a big example of where they often don't. Many of the applications I've worked with use it in setting up communication (esp. in amr codes). And many of these people asked for an mpi-subset. Interesting to note that for many of these codes, actual performance of the send/recv is _not_ an issue - the work associated with preparing and managing messages for those sorts of codes, as well as the work imbalances, overwhelm the performance issues. (But note these code teams may _say_ mpi performance is an issue, but a complete view of their communication requirements does not support that claim.) However, they also require MPI_ALL_REDUCE, and at the peta-scale, so performance there is critical. MPI_ANY_SOURCE is found in many libraries I use, but library developers seem to make significant use of mpi functionality, without complaint. All this said, admittedly counter-claims and examples may be produced. However, as Dick stated for the Master/worker model, a large segment of users "require" MPI_ANY_SOURCE" somewhere in their code. I'm not saying a decision needs to be reached quickly, but I do believe we should keep this issue clearly in view in order to keep the discussions on-track. Richard -- Richard Barrett Future Technologies Group, Computer Science and Mathematics Division, and Scientific Computing Group, National Center for Computational Science Oak Ridge National Laboratory http://ft.ornl.gov/~rbarrett From keller at [hidden] Wed Mar 5 14:14:59 2008 From: keller at [hidden] (Rainer Keller) Date: Wed, 5 Mar 2008 21:14:59 +0100 Subject: [Mpi3-subsetting] MPI3: Proposal to remove PMPI-Requirement Message-ID: <200803052115.00366.keller@hlrs.de> Dear Alexander, dear all, at the previous Chicago meeting, some of us (Rich Graham, Jeff Squyres, Hubert Ritzdorf) have been discussing about the MPI-Standard's requirement to provide PMPI-functions for each MPI-Call into the library (except MPI_Wtime, MPI_Wtick). This is in my eyes a limitation: there could be some gains for the common case of large-scale applications: - Not using PMPI-based tools (dynamically loaded in) - Not using MPI_PROC_NULL - Not using MPI_STATUS_IGNORE So, if this proposal would fit Your definition of MPI3-Subsets, I would like to discuss the removal of the PMPI-function requirement, allowing inlining of MPI-functions and, based on that, pre-processor hints, e.g. MPI_HINT_NO_THREADS or MPI_HINT_NO_ANY_SOURCE, so the compiler might be able to eliminate several if-statements in the fast-path, allowing inlining, allowing dead code elimination. (of course, there is a whole lot of further issues involved, here -- please see the last section in the document). I have included some timings with NetPipe on Open MPI using these hints (further information section 4 -- Ompi Patch availble upon request, as its work in progress). Please take the timings with a grain of salt. I would like to get better figures on a more well-defined cluster environment. Any comments would be welcome. With best regards, Rainer -- ---------------------------------------------------------------- Dipl.-Inf. Rainer Keller http://www.hlrs.de/people/keller HLRS Tel: ++49 (0)711-685 6 5858 Nobelstrasse 19 Fax: ++49 (0)711-685 6 5832 70550 Stuttgart email: keller_at_[hidden] Germany AIM/Skype:rusraink * -------------- next part -------------- A non-text attachment was scrubbed... Name: proposal_mpi_header_hints.pdf Type: application/pdf Size: 105749 bytes Desc: proposal_mpi_header_hints.pdf URL: From alexander.supalov at [hidden] Wed Mar 5 15:03:44 2008 From: alexander.supalov at [hidden] (Supalov, Alexander) Date: Wed, 5 Mar 2008 21:03:44 -0000 Subject: [Mpi3-subsetting] MPI3: Proposal to remove PMPI-Requirement In-Reply-To: <200803052115.00366.keller@hlrs.de> Message-ID: <5ECAB1304A8B5B4CB3F9D6C01E4E21A2011CC578@swsmsx413.ger.corp.intel.com> Hi, Thanks. Interesting. I wonder in what relation this will stand with the ABI activity. Best regards. Alexander -----Original Message----- From: mpi3-subsetting-bounces_at_[hidden] [mailto:mpi3-subsetting-bounces_at_[hidden]] On Behalf Of Rainer Keller Sent: Wednesday, March 05, 2008 9:15 PM To: MPI3-Subsetting_at_[hidden] Subject: [Mpi3-subsetting] MPI3: Proposal to remove PMPI-Requirement Dear Alexander, dear all, at the previous Chicago meeting, some of us (Rich Graham, Jeff Squyres, Hubert Ritzdorf) have been discussing about the MPI-Standard's requirement to provide PMPI-functions for each MPI-Call into the library (except MPI_Wtime, MPI_Wtick). This is in my eyes a limitation: there could be some gains for the common case of large-scale applications: - Not using PMPI-based tools (dynamically loaded in) - Not using MPI_PROC_NULL - Not using MPI_STATUS_IGNORE So, if this proposal would fit Your definition of MPI3-Subsets, I would like to discuss the removal of the PMPI-function requirement, allowing inlining of MPI-functions and, based on that, pre-processor hints, e.g. MPI_HINT_NO_THREADS or MPI_HINT_NO_ANY_SOURCE, so the compiler might be able to eliminate several if-statements in the fast-path, allowing inlining, allowing dead code elimination. (of course, there is a whole lot of further issues involved, here -- please see the last section in the document). I have included some timings with NetPipe on Open MPI using these hints (further information section 4 -- Ompi Patch availble upon request, as its work in progress). Please take the timings with a grain of salt. I would like to get better figures on a more well-defined cluster environment. Any comments would be welcome. With best regards, Rainer -- ---------------------------------------------------------------- Dipl.-Inf. Rainer Keller http://www.hlrs.de/people/keller HLRS Tel: ++49 (0)711-685 6 5858 Nobelstrasse 19 Fax: ++49 (0)711-685 6 5832 70550 Stuttgart email: keller_at_[hidden] Germany AIM/Skype:rusraink --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. From alexander.supalov at [hidden] Thu Mar 6 09:27:29 2008 From: alexander.supalov at [hidden] (Supalov, Alexander) Date: Thu, 6 Mar 2008 15:27:29 -0000 Subject: [Mpi3-subsetting] Subsetting slides for F2F meeting: please comment by EOD March 6, 2008 In-Reply-To: <[Mpi3-subsetting] Subsetting slides for F2F meeting: please comment by EOD March 6, 2008> Message-ID: <5ECAB1304A8B5B4CB3F9D6C01E4E21A2011F4601@swsmsx413.ger.corp.intel.com> Hi everybody, Thanks for your comments. Now the slides are also in the Wiki, see http://svn.mpi-forum.org/trac/mpi-forum-web/attachment/wiki/SubsettingWi kiPage/MPI-3%20subsetting.ppt . Note that I'm going to incorporate your feedback and remove the backup in the final version - we'll only have 5 minutes to present the WG results to the Forum. Best regards. Alexander ________________________________ From: Supalov, Alexander Sent: Wednesday, March 05, 2008 3:38 PM To: 'mpi3-subsetting_at_[hidden]' Subject: Subsetting slides for F2F meeting: please comment by EOD March 6, 2008 Hi everybody, Here come the promised draft slides for the subsetting presentation at the MPI Forum. I put this into our corporate template, I hope everybody is OK with this. Your comments will be most appreciated. I would like to finalize the slides on Friday, so please send your feedback by EOD your time on Thursday, March 6, 2008. Sorry for the message size. I cannot upload this file to the Wiki due to file size limitation there. I work this with Rich in background. Best regards. Alexander -- Dr Alexander Supalov Intel GmbH Hermuelheimer Strasse 8a 50321 Bruehl, Germany Phone: +49 2232 209034 Mobile: +49 173 511 8735 Fax: +49 2232 209029 --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. * -------------- next part -------------- An HTML attachment was scrubbed... URL: From rlgraham at [hidden] Thu Mar 6 09:56:31 2008 From: rlgraham at [hidden] (Richard Graham) Date: Thu, 06 Mar 2008 10:56:31 -0500 Subject: [Mpi3-subsetting] Subsetting slides for F2F meeting: please comment by EOD March 6, 2008 In-Reply-To: <5ECAB1304A8B5B4CB3F9D6C01E4E21A2011F4601@swsmsx413.ger.corp.intel.com> Message-ID: Thanks a lot for taking on the task of herding cats. Here are some comments: - Growing complexity does not have to mean worse performance. It really depends on where the complexity resides. - We have not at all agreed to formal recognition of optional parts of the standard. As a matter of fact, on the call there were more opinions expressed in favor of use of a subset in a certain instant, not optional sections. We may at the end conclude that for some features optional is the best way, but have not gone there yet. - Optimization potential: don¹t get into too much detail. I am not convinced that all that is listed there is actually correct ­ implementation can do (and do) a lot to eliminate some of the things listed in the bulleted list. - What did you have in mind with ³mutually incompatible families² ? Rich On 3/6/08 10:27 AM, "Supalov, Alexander" wrote: > Hi everybody, > > Thanks for your comments. Now the slides are also in the Wiki, see > http://svn.mpi-forum.org/trac/mpi-forum-web/attachment/wiki/SubsettingWikiPage > /MPI-3%20subsetting.ppt . > > Note that I'm going to incorporate your feedback and remove the backup in the > final version - we'll only have 5 minutes to present the WG results to the > Forum. > > Best regards. > > Alexander > > > From: Supalov, Alexander > Sent: Wednesday, March 05, 2008 3:38 PM > To: 'mpi3-subsetting_at_[hidden]' > Subject: Subsetting slides for F2F meeting: please comment by EOD March 6, > 2008 > > Hi everybody, > > Here come the promised draft slides for the subsetting presentation at the MPI > Forum. I put this into our corporate template, I hope everybody is OK with > this. > > Your comments will be most appreciated. I would like to finalize the slides on > Friday, so please send your feedback by EOD your time on Thursday, March 6, > 2008. > > Sorry for the message size. I cannot upload this file to the Wiki due to file > size limitation there. I work this with Rich in background. > > Best regards. > > Alexander > > -- > Dr Alexander Supalov > Intel GmbH > Hermuelheimer Strasse 8a > 50321 Bruehl, Germany > Phone: +49 2232 209034 > Mobile: +49 173 511 8735 > Fax: +49 2232 209029 > > --------------------------------------------------------------------- > Intel GmbH > Dornacher Strasse 1 > 85622 Feldkirchen/Muenchen Germany > Sitz der Gesellschaft: Feldkirchen bei Muenchen > Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer > Registergericht: Muenchen HRB 47456 Ust.-IdNr. > VAT Registration No.: DE129385895 > Citibank Frankfurt (BLZ 502 109 00) 600119052 > > This e-mail and any attachments may contain confidential material for > the sole use of the intended recipient(s). Any review or distribution > by others is strictly prohibited. If you are not the intended > recipient, please contact the sender and delete all copies. > > > _______________________________________________ > mpi3-subsetting mailing list > mpi3-subsetting_at_[hidden] > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-subsetting * -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.supalov at [hidden] Thu Mar 6 10:16:33 2008 From: alexander.supalov at [hidden] (Supalov, Alexander) Date: Thu, 6 Mar 2008 16:16:33 -0000 Subject: [Mpi3-subsetting] Subsetting slides for F2F meeting: please comment by EOD March 6, 2008 In-Reply-To: Message-ID: <5ECAB1304A8B5B4CB3F9D6C01E4E21A2011F4660@swsmsx413.ger.corp.intel.com> Thanks. Details inside. ________________________________ From: mpi3-subsetting-bounces_at_[hidden] [mailto:mpi3-subsetting-bounces_at_[hidden]] On Behalf Of Richard Graham Sent: Thursday, March 06, 2008 4:57 PM To: MPI 3.0 Sub-setting working group Subject: Re: [Mpi3-subsetting] Subsetting slides for F2F meeting: please comment by EOD March 6, 2008 Thanks a lot for taking on the task of herding cats. Here are some comments: - Growing complexity does not have to mean worse performance. It really depends on where the complexity resides. I'll restore "may" here. - We have not at all agreed to formal recognition of optional parts of the standard. As a matter of fact, on the call there were more opinions expressed in favor of use of a subset in a certain instant, not optional sections. We may at the end conclude that for some features optional is the best way, but have not gone there yet. OK, I'll turn this into suggestion or drop it. - Optimization potential: don't get into too much detail. I am not convinced that all that is listed there is actually correct - implementation can do (and do) a lot to eliminate some of the things listed in the bulleted list. I'll consider this. - What did you have in mind with "mutually incompatible families" ? Several stable combinations of subsets may emerge that will cover different application areas well, but they (the subsets) will be mutually incompatible between each other. Rich On 3/6/08 10:27 AM, "Supalov, Alexander" wrote: Hi everybody, Thanks for your comments. Now the slides are also in the Wiki, see http://svn.mpi-forum.org/trac/mpi-forum-web/attachment/wiki/SubsettingWi kiPage/MPI-3%20subsetting.ppt . Note that I'm going to incorporate your feedback and remove the backup in the final version - we'll only have 5 minutes to present the WG results to the Forum. Best regards. Alexander ________________________________ From: Supalov, Alexander Sent: Wednesday, March 05, 2008 3:38 PM To: 'mpi3-subsetting_at_[hidden]' Subject: Subsetting slides for F2F meeting: please comment by EOD March 6, 2008 Hi everybody, Here come the promised draft slides for the subsetting presentation at the MPI Forum. I put this into our corporate template, I hope everybody is OK with this. Your comments will be most appreciated. I would like to finalize the slides on Friday, so please send your feedback by EOD your time on Thursday, March 6, 2008. Sorry for the message size. I cannot upload this file to the Wiki due to file size limitation there. I work this with Rich in background. Best regards. Alexander -- Dr Alexander Supalov Intel GmbH Hermuelheimer Strasse 8a 50321 Bruehl, Germany Phone: +49 2232 209034 Mobile: +49 173 511 8735 Fax: +49 2232 209029 --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. ________________________________ _______________________________________________ mpi3-subsetting mailing list mpi3-subsetting_at_[hidden] http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-subsetting --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. * -------------- next part -------------- An HTML attachment was scrubbed... URL: From bronis at [hidden] Thu Mar 6 10:58:37 2008 From: bronis at [hidden] (Bronis R. de Supinski) Date: Thu, 6 Mar 2008 08:58:37 -0800 (PST) Subject: [Mpi3-subsetting] MPI3: Proposal to remove PMPI-Requirement In-Reply-To: <200803052115.00366.keller@hlrs.de> Message-ID: Rainer: There are some significant problems with this proposal. The premise that the profiling interface is only used early in the development is not correct. The PMPI interface is useful at almost every stage of the application's life time. One of the things that makes it successful is that it is always available and does not require recompilation. Making it optional would destroy that. I can tell you that one of the biggest problems for OpenMP has been the lack of a profiling interface. We recently had one sanctioned by the OpenMP ARB but that took a lot of effort since it is very hard to add it in after the fact. Your proposal would not be as drastic as eliminating the interface but the effect would be almost as catastrophic. Perhaps just as importantly to the general need of the interface, the idea that inlining of MPI will provide substantial benefits is inaccurate. MPI is a library interface. Many implementations are distributed only in binary form; unless you are proposing link time inlining (possible but not without its pitfalls), they will not gain any benefit from this. Even if we consider only open source MPI implementations, the VAST majority of users do NOT want to compile the MPI library. The possible performance gain is far too small for them to take on this task. A properly designed subsetting approach will NOT try to partition the profiling interface into its own subset. Instead, the interface must be partitioned into different subsets, with each PMPI function in the same subset as the corresponding MPI function. Bronis On Wed, 5 Mar 2008, Rainer Keller wrote: > Dear Alexander, dear all, > at the previous Chicago meeting, some of us (Rich Graham, Jeff Squyres, Hubert > Ritzdorf) have been discussing about the MPI-Standard's requirement to > provide PMPI-functions for each MPI-Call into the library (except MPI_Wtime, > MPI_Wtick). > > This is in my eyes a limitation: there could be some gains for the common case > of large-scale applications: > - Not using PMPI-based tools (dynamically loaded in) > - Not using MPI_PROC_NULL > - Not using MPI_STATUS_IGNORE > > So, if this proposal would fit Your definition of MPI3-Subsets, I would like > to discuss the removal of the PMPI-function requirement, allowing inlining of > MPI-functions and, based on that, pre-processor hints, e.g. > MPI_HINT_NO_THREADS or MPI_HINT_NO_ANY_SOURCE, so the compiler might be able > to eliminate several if-statements in the fast-path, allowing inlining, > allowing dead code elimination. > (of course, there is a whole lot of further issues involved, here -- please > see the last section in the document). > > I have included some timings with NetPipe on Open MPI using these hints > (further information section 4 -- Ompi Patch availble upon request, as its > work in progress). Please take the timings with a grain of salt. I would like > to get better figures on a more well-defined cluster environment. > > Any comments would be welcome. > > With best regards, > Rainer > -- > ---------------------------------------------------------------- > Dipl.-Inf. Rainer Keller http://www.hlrs.de/people/keller > HLRS Tel: ++49 (0)711-685 6 5858 > Nobelstrasse 19 Fax: ++49 (0)711-685 6 5832 > 70550 Stuttgart email: keller_at_[hidden] > Germany AIM/Skype:rusraink > From alexander.supalov at [hidden] Thu Mar 6 11:24:01 2008 From: alexander.supalov at [hidden] (Supalov, Alexander) Date: Thu, 6 Mar 2008 17:24:01 -0000 Subject: [Mpi3-subsetting] Subsetting slides for F2F meeting: please comment by EOD March 6, 2008 In-Reply-To: Message-ID: <5ECAB1304A8B5B4CB3F9D6C01E4E21A2011F470E@swsmsx413.ger.corp.intel.com> Wiki updated accordingly. ________________________________ From: Supalov, Alexander Sent: Thursday, March 06, 2008 5:17 PM To: 'MPI 3.0 Sub-setting working group' Subject: RE: [Mpi3-subsetting] Subsetting slides for F2F meeting: please comment by EOD March 6, 2008 Thanks. Details inside. ________________________________ From: mpi3-subsetting-bounces_at_[hidden] [mailto:mpi3-subsetting-bounces_at_[hidden]] On Behalf Of Richard Graham Sent: Thursday, March 06, 2008 4:57 PM To: MPI 3.0 Sub-setting working group Subject: Re: [Mpi3-subsetting] Subsetting slides for F2F meeting: please comment by EOD March 6, 2008 Thanks a lot for taking on the task of herding cats. Here are some comments: - Growing complexity does not have to mean worse performance. It really depends on where the complexity resides. I'll restore "may" here. - We have not at all agreed to formal recognition of optional parts of the standard. As a matter of fact, on the call there were more opinions expressed in favor of use of a subset in a certain instant, not optional sections. We may at the end conclude that for some features optional is the best way, but have not gone there yet. OK, I'll turn this into suggestion or drop it. - Optimization potential: don't get into too much detail. I am not convinced that all that is listed there is actually correct - implementation can do (and do) a lot to eliminate some of the things listed in the bulleted list. I'll consider this. - What did you have in mind with "mutually incompatible families" ? Several stable combinations of subsets may emerge that will cover different application areas well, but they (the subsets) will be mutually incompatible between each other. Rich On 3/6/08 10:27 AM, "Supalov, Alexander" wrote: Hi everybody, Thanks for your comments. Now the slides are also in the Wiki, see http://svn.mpi-forum.org/trac/mpi-forum-web/attachment/wiki/SubsettingWi kiPage/MPI-3%20subsetting.ppt . Note that I'm going to incorporate your feedback and remove the backup in the final version - we'll only have 5 minutes to present the WG results to the Forum. Best regards. Alexander ________________________________ From: Supalov, Alexander Sent: Wednesday, March 05, 2008 3:38 PM To: 'mpi3-subsetting_at_[hidden]' Subject: Subsetting slides for F2F meeting: please comment by EOD March 6, 2008 Hi everybody, Here come the promised draft slides for the subsetting presentation at the MPI Forum. I put this into our corporate template, I hope everybody is OK with this. Your comments will be most appreciated. I would like to finalize the slides on Friday, so please send your feedback by EOD your time on Thursday, March 6, 2008. Sorry for the message size. I cannot upload this file to the Wiki due to file size limitation there. I work this with Rich in background. Best regards. Alexander -- Dr Alexander Supalov Intel GmbH Hermuelheimer Strasse 8a 50321 Bruehl, Germany Phone: +49 2232 209034 Mobile: +49 173 511 8735 Fax: +49 2232 209029 --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. ________________________________ _______________________________________________ mpi3-subsetting mailing list mpi3-subsetting_at_[hidden] http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-subsetting --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. * -------------- next part -------------- An HTML attachment was scrubbed... URL: From Dries.Kimpe at [hidden] Thu Mar 6 11:34:07 2008 From: Dries.Kimpe at [hidden] (Dries Kimpe) Date: Thu, 6 Mar 2008 18:34:07 +0100 Subject: [Mpi3-subsetting] MPI3: Proposal to remove PMPI-Requirement In-Reply-To: Message-ID: <20080306173407.GA30010@mhdmobile.lan> * Bronis R. de Supinski [2008-03-06 08:58:37]: > Perhaps just as importantly to the general need of the > interface, the idea that inlining of MPI will provide > substantial benefits is inaccurate. MPI is a library > interface. Many implementations are distributed only in > binary form; unless you are proposing link time inlining > (possible but not without its pitfalls), they will not gain > any benefit from this. Even if we consider only open source > MPI implementations, the VAST majority of users do NOT > want to compile the MPI library. The possible performance > gain is far too small for them to take on this task. This might not be completely true. Since there is no ABI right now, every MPI application needs to be recompiled before it can be used. Even though the MPI library might be compiled already, the C header will contain inline functions (they need to be in the header to have an effect) and will thus be inlined into the application. The MPI library itself can choose how far it goes with this inlining. (For example, all of MPI_Send / MPI_Recv could be inlined if it wishes to do so). So it is NOT needed to recompile the MPI library, only the functions that are part of mpi.h (and any headers it might include) will be recompiled, and this is transparent to the application. The real issue is that an ABI and inline functions are not compatible, unless the implementation of the inlined functions themselves is considered part of the ABI. (this means all of the structures they use, all of the functions they call, ...) Dries * -------------- next part -------------- A non-text attachment was scrubbed... Name: 01-part Type: application/pgp-signature Size: 190 bytes Desc: not available URL: From bronis at [hidden] Thu Mar 6 13:37:38 2008 From: bronis at [hidden] (Bronis R. de Supinski) Date: Thu, 6 Mar 2008 11:37:38 -0800 (PST) Subject: [Mpi3-subsetting] MPI3: Proposal to remove PMPI-Requirement In-Reply-To: <20080306173407.GA30010@mhdmobile.lan> Message-ID: Dries: While the question of an ABI is also important, it does not really address the primary reason that recompiling is bad. My issue is not that the applications have to be compiled the first time. It is that recompiling to use the profiling interface is an excessively high barrier for many applications. Also, while I see your point that only a small set of stuff needs to be placed into mpi.h, that does not address the real question: What is the actual application level benefit? Why do I care if I can get MPI_Comm_size or MPI_Comm_rank inlined? Good programming practice does not call them frequently enough for any performance gain to matter. Alternatively, suppose I can squash some stuff out of the MPI_Send call path. Will it really matter to real applications? It seems unlikely, particularly relative to the cost of lost flexibility for understanding communication behavior and performance. Bronis On Thu, 6 Mar 2008, Dries Kimpe wrote: > > * Bronis R. de Supinski [2008-03-06 08:58:37]: > > > Perhaps just as importantly to the general need of the > > interface, the idea that inlining of MPI will provide > > substantial benefits is inaccurate. MPI is a library > > interface. Many implementations are distributed only in > > binary form; unless you are proposing link time inlining > > (possible but not without its pitfalls), they will not gain > > any benefit from this. Even if we consider only open source > > MPI implementations, the VAST majority of users do NOT > > want to compile the MPI library. The possible performance > > gain is far too small for them to take on this task. > > This might not be completely true. Since there is no ABI right now, every > MPI application needs to be recompiled before it can be used. Even though > the MPI library might be compiled already, the C header will contain > inline functions (they need to be in the header to have an effect) and > will thus be inlined into the application. The MPI library itself can > choose how far it goes with this inlining. (For example, all of MPI_Send > / MPI_Recv could be inlined if it wishes to do so). > > So it is NOT needed to recompile the MPI library, only the functions that > are part of mpi.h (and any headers it might include) will be recompiled, > and this is transparent to the application. > > The real issue is that an ABI and inline functions are not compatible, > unless the implementation of the inlined functions themselves is > considered part of the ABI. (this means all of the structures they use, > all of the functions they call, ...) > > Dries > > From jsquyres at [hidden] Fri Mar 7 06:39:23 2008 From: jsquyres at [hidden] (Jeff Squyres) Date: Fri, 7 Mar 2008 07:39:23 -0500 Subject: [Mpi3-subsetting] MPI3: Proposal to remove PMPI-Requirement In-Reply-To: Message-ID: <8EF7AF86-2A7E-4DE2-82D1-B9266BD2E4DC@cisco.com> On Mar 6, 2008, at 11:58 AM, Bronis R. de Supinski wrote: > There are some significant problems with this proposal. > The premise that the profiling interface is only used > early in the development is not correct. The PMPI interface > is useful at almost every stage of the application's life > time. One of the things that makes it successful is that > it is always available and does not require recompilation. > Making it optional would destroy that. I'm sure that that is true for some apps. But there are also wide classes of apps where Rainer's statements *are* true (that PMPI is only used in the development lifecycle, which is relatively short compared to the apps' production lifetime). > A properly designed subsetting approach will NOT try to > partition the profiling interface into its own subset. > Instead, the interface must be partitioned into different > subsets, with each PMPI function in the same subset as the > corresponding MPI function. I'm not sure that I understand your rationale here (perhaps we can discuss more in Chicago?). In my view of the world, at least one way to effect subsetting could be something that is dynamically loaded (or not) at run-time. Hence, you could have PMPI or not, depending on options given to mpirun (for example). I'm obviously waving my hands a bit here, but that's the premise I was assuming (details TBD, of course). From your later mail: > Also, while I see your point that only a small set of > stuff needs to be placed into mpi.h, that does not > address the real question: What is the actual application > level benefit? Why do I care if I can get MPI_Comm_size > or MPI_Comm_rank inlined? Good programming practice > does not call them frequently enough for any performance > gain to matter. Alternatively, suppose I can squash some > stuff out of the MPI_Send call path. Will it really matter > to real applications? It seems unlikely, particularly > relative to the cost of lost flexibility for understanding > communication behavior and performance. Rainer's proposal came from joint discussions with NEC where (correct me if I'm wrong, Hubert) select functions *are* [optionally?] inlined in mpi.h because the cost of function calls on their platform is so high. Additionally, I'm always wary of the term "real applications" -- what a "real" application is to a US DOE lab is very different than what a "real application" is to, say, someone in the oil and gas industry. :-) The scale difference alone is enough to make the latency gains by inlining select MPI functions important (meaning: non- US-DOE-labs tend to run at [much] smaller scale; fabric latency may be constrained to a single switch, and therefore *can* see benefit from inlining). As I understand Rainer's proposal, the inlining aspects also predicated on apps that enter production and are never changed. Perhaps this is not a pattern used much at DOE labs, but it is a pattern I see with many customers. -- Jeff Squyres Cisco Systems From bronis at [hidden] Fri Mar 7 10:20:35 2008 From: bronis at [hidden] (Bronis R. de Supinski) Date: Fri, 7 Mar 2008 08:20:35 -0800 (PST) Subject: [Mpi3-subsetting] MPI3: Proposal to remove PMPI-Requirement In-Reply-To: <8EF7AF86-2A7E-4DE2-82D1-B9266BD2E4DC@cisco.com> Message-ID: Jeff: Re: [snip] > > A properly designed subsetting approach will NOT try to > > partition the profiling interface into its own subset. > > Instead, the interface must be partitioned into different > > subsets, with each PMPI function in the same subset as the > > corresponding MPI function. > > I'm not sure that I understand your rationale here (perhaps we can > discuss more in Chicago?). Sure, we can discuss it in Chicago. However, The rationale is simple enough. The user of a subset wants to be able to profile the functionality in a given subset. > In my view of the world, at least one way to effect subsetting could > be something that is dynamically loaded (or not) at run-time. Hence, > you could have PMPI or not, depending on options given to mpirun (for > example). I'm obviously waving my hands a bit here, but that's the > premise I was assuming (details TBD, of course). I have no objection to your premise. However, this proposal is specifically designed to prevent dynamic expansion of the feature set to include the PMPI calls (or at least to doing so effectively since there are no MPI calls between which the tool could interpose). In fact, the ability to change the subset dynamically is beyond what I was requiring (but is desirable). All I stated was that the expansion should not require recompilation in order to include the profiling interface for what is included in the subset. If you (or anyone else) can design a mechanism that allows using the macros that Rainer wants to enable without precluding that interposition dynamically then I offer no opposition to the idea. However, I would also suggest that such a mechanism does not need any change to the standard; it is already compliant. > From your later mail: > > > Also, while I see your point that only a small set of > > stuff needs to be placed into mpi.h, that does not > > address the real question: What is the actual application > > level benefit? Why do I care if I can get MPI_Comm_size > > or MPI_Comm_rank inlined? Good programming practice > > does not call them frequently enough for any performance > > gain to matter. Alternatively, suppose I can squash some > > stuff out of the MPI_Send call path. Will it really matter > > to real applications? It seems unlikely, particularly > > relative to the cost of lost flexibility for understanding > > communication behavior and performance. > > > Rainer's proposal came from joint discussions with NEC where (correct > me if I'm wrong, Hubert) select functions *are* [optionally?] inlined > in mpi.h because the cost of function calls on their platform is so > high. While one can argue whether the standard should avoid precluding the exploitation of as yet unbuilt clever hardware mechanisms, it seems pretty clear that it is not the job of the standard to accommodate poorly designed hardware. If the cost of function calls is so excessively high that ONE prevents effective use of the hardware then the vendor needs to reconsider their design. In any event, it appears that you are stating that a solution was found: offer a non-compliant alternative implementation. This is a far more appropriate solution in this case. Since I don't believe Rainer was proposing that the PMPI interface should be optional (just somehow not included in a base subset), the implementor still must support it. That means no implementation effort is saved by saying they could have a compliant implementation that does not always support the interface (but always offers the option to include that support). The implementor is always free to offer non-compliant alternatives; it is up the users whether they want to build with it. > Additionally, I'm always wary of the term "real applications" -- what > a "real" application is to a US DOE lab is very different than what a > "real application" is to, say, someone in the oil and gas > industry. :-) The scale difference alone is enough to make the > latency gains by inlining select MPI functions important (meaning: non- > US-DOE-labs tend to run at [much] smaller scale; fabric latency may be > constrained to a single switch, and therefore *can* see benefit from > inlining). > > As I understand Rainer's proposal, the inlining aspects also > predicated on apps that enter production and are never changed. > Perhaps this is not a pattern used much at DOE labs, but it is a > pattern I see with many customers. I'm not going to get into a pissing match with you over what applications do what. You seem to almost be using "US DOE lab" pejoratively. What I can say without the slightest hesitation is that recompilation is often a significant barrier to tool acceptance for application users and those users are NOT limited to one site or even one type of site (yes, I know users who are not at "DOE labs"). My opposition to changes that would force tools to require recompilation is borne from experience. Bronis From keller at [hidden] Fri Mar 7 14:44:55 2008 From: keller at [hidden] (Rainer Keller) Date: Fri, 7 Mar 2008 21:44:55 +0100 Subject: [Mpi3-subsetting] MPI3: Proposal to remove PMPI-Requirement In-Reply-To: Message-ID: <200803072144.55962.keller@hlrs.de> Hello Bronis, thanks for Your comments. On Thursday 06 March 2008 17:58, Bronis R. de Supinski wrote: > There are some significant problems with this proposal. > The premise that the profiling interface is only used > early in the development is not correct. The PMPI interface > is useful at almost every stage of the application's life > time. One of the things that makes it successful is that > it is always available and does not require recompilation. > Making it optional would destroy that. This would not make it optional. It still is required by the standard. However, if the user wishes to do so, he may ask the implementation to inline functions (and on top of that specify some characterstics of his/her app). > I can tell you that one of the biggest problems for OpenMP > has been the lack of a profiling interface. We recently > had one sanctioned by the OpenMP ARB but that took a lot > of effort since it is very hard to add it in after the fact. > Your proposal would not be as drastic as eliminating the > interface but the effect would be almost as catastrophic. Hmm, I do follow Your argument on OpenMP that is a real issue... However fail to see, why it's so dramatic in our case (see above). > Perhaps just as importantly to the general need of the > interface, the idea that inlining of MPI will provide > substantial benefits is inaccurate. MPI is a library > interface. Many implementations are distributed only in > binary form; They do provide at least one header file, now. They could provide other header files. They could provide obviouscated header files which call into specialized functions, that do not check for any_source ,-] > unless you are proposing link time inlining (possible but not without its > pitfalls), No. > they will not gain any benefit from this. Even if we consider only open > source MPI implementations, the VAST majority of users do NOT > want to compile the MPI library. The possible performance > gain is far too small for them to take on this task. Errr, they do not need to compile the MPI library at all? They just #define MPI_HINT_INLINE #include "mpi.h" and the magic happens. > A properly designed subsetting approach will NOT try to > partition the profiling interface into its own subset. Yeah, that was probably the biggest misunderstanding. Please see the other mail ,-] > Instead, the interface must be partitioned into different > subsets, with each PMPI function in the same subset as the > corresponding MPI function. Yes. Thanks, Rainer > On Wed, 5 Mar 2008, Rainer Keller wrote: > > Dear Alexander, dear all, > > at the previous Chicago meeting, some of us (Rich Graham, Jeff Squyres, > > Hubert Ritzdorf) have been discussing about the MPI-Standard's > > requirement to provide PMPI-functions for each MPI-Call into the library > > (except MPI_Wtime, MPI_Wtick). > > > > This is in my eyes a limitation: there could be some gains for the common > > case of large-scale applications: > > - Not using PMPI-based tools (dynamically loaded in) > > - Not using MPI_PROC_NULL > > - Not using MPI_STATUS_IGNORE > > > > So, if this proposal would fit Your definition of MPI3-Subsets, I would > > like to discuss the removal of the PMPI-function requirement, allowing > > inlining of MPI-functions and, based on that, pre-processor hints, e.g. > > MPI_HINT_NO_THREADS or MPI_HINT_NO_ANY_SOURCE, so the compiler might be > > able to eliminate several if-statements in the fast-path, allowing > > inlining, allowing dead code elimination. > > (of course, there is a whole lot of further issues involved, here -- > > please see the last section in the document). > > > > I have included some timings with NetPipe on Open MPI using these hints > > (further information section 4 -- Ompi Patch availble upon request, as > > its work in progress). Please take the timings with a grain of salt. I > > would like to get better figures on a more well-defined cluster > > environment. > > > > Any comments would be welcome. > > > > With best regards, > > Rainer > > -- > > ---------------------------------------------------------------- > > Dipl.-Inf. Rainer Keller http://www.hlrs.de/people/keller > > HLRS Tel: ++49 (0)711-685 6 5858 > > Nobelstrasse 19 Fax: ++49 (0)711-685 6 5832 > > 70550 Stuttgart email: keller_at_[hidden] > > Germany AIM/Skype:rusraink > > _______________________________________________ > mpi3-subsetting mailing list > mpi3-subsetting_at_[hidden] > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-subsetting -- ---------------------------------------------------------------- Dipl.-Inf. Rainer Keller http://www.hlrs.de/people/keller HLRS Tel: ++49 (0)711-685 6 5858 Nobelstrasse 19 Fax: ++49 (0)711-685 6 5832 70550 Stuttgart email: keller_at_[hidden] Germany AIM/Skype:rusraink From keller at [hidden] Fri Mar 7 14:48:39 2008 From: keller at [hidden] (Rainer Keller) Date: Fri, 7 Mar 2008 21:48:39 +0100 Subject: [Mpi3-subsetting] MPI3: Proposal to remove PMPI-Requirement In-Reply-To: Message-ID: <200803072148.39493.keller@hlrs.de> Hello Bronis, the idea was not to eliminate PMPI per se. But rather allow in the standard a well-defined approach, with which function calls (due to the PMPI-requirement) can be macroized and additionally allowing the user to pass more information to the library. This should enable the compiler to do "a better job"... - The users that do not want it, do not specify the pre-processor flag and may happily use PMPI-based tools. - The users that do care, may specify it -- and depending on the implementation/hardware may see performance improvements. If the MPI-implementation does not yet process these macros, well so be it. Therefore I see this approach as the last step before deployment of an application. The proposed approach is just for discussion and by no means was meant to be solution for "the subset" -- however it may be a starting point and pre-processor based information a valid way, that goes hand-in-hand with other (MPI_Info-based?) subset-approaches... Thanks, Rainer On Friday 07 March 2008 17:20, Bronis R. de Supinski wrote: > > > A properly designed subsetting approach will NOT try to > > > partition the profiling interface into its own subset. > > > Instead, the interface must be partitioned into different > > > subsets, with each PMPI function in the same subset as the > > > corresponding MPI function. > > > > I'm not sure that I understand your rationale here (perhaps we can > > discuss more in Chicago?). > > Sure, we can discuss it in Chicago. However, The rationale is > simple enough. The user of a subset wants to be able to profile > the functionality in a given subset. > > > In my view of the world, at least one way to effect subsetting could > > be something that is dynamically loaded (or not) at run-time. Hence, > > you could have PMPI or not, depending on options given to mpirun (for > > example). I'm obviously waving my hands a bit here, but that's the > > premise I was assuming (details TBD, of course). > > I have no objection to your premise. However, this proposal > is specifically designed to prevent dynamic expansion of the > feature set to include the PMPI calls (or at least to doing > so effectively since there are no MPI calls between which > the tool could interpose). In fact, the ability to change > the subset dynamically is beyond what I was requiring (but > is desirable). All I stated was that the expansion should > not require recompilation in order to include the profiling > interface for what is included in the subset. > > If you (or anyone else) can design a mechanism that allows > using the macros that Rainer wants to enable without precluding > that interposition dynamically then I offer no opposition to > the idea. However, I would also suggest that such a mechanism > does not need any change to the standard; it is already compliant. > > > From your later mail: > > > Also, while I see your point that only a small set of > > > stuff needs to be placed into mpi.h, that does not > > > address the real question: What is the actual application > > > level benefit? Why do I care if I can get MPI_Comm_size > > > or MPI_Comm_rank inlined? Good programming practice > > > does not call them frequently enough for any performance > > > gain to matter. Alternatively, suppose I can squash some > > > stuff out of the MPI_Send call path. Will it really matter > > > to real applications? It seems unlikely, particularly > > > relative to the cost of lost flexibility for understanding > > > communication behavior and performance. > > > > Rainer's proposal came from joint discussions with NEC where (correct > > me if I'm wrong, Hubert) select functions *are* [optionally?] inlined > > in mpi.h because the cost of function calls on their platform is so > > high. > > While one can argue whether the standard should avoid precluding > the exploitation of as yet unbuilt clever hardware mechanisms, > it seems pretty clear that it is not the job of the standard to > accommodate poorly designed hardware. If the cost of function > calls is so excessively high that ONE prevents effective use of > the hardware then the vendor needs to reconsider their design. > > In any event, it appears that you are stating that a solution was > found: offer a non-compliant alternative implementation. This is > a far more appropriate solution in this case. Since I don't believe > Rainer was proposing that the PMPI interface should be optional > (just somehow not included in a base subset), the implementor > still must support it. That means no implementation effort is > saved by saying they could have a compliant implementation that > does not always support the interface (but always offers the > option to include that support). The implementor is always free > to offer non-compliant alternatives; it is up the users whether > they want to build with it. > > > Additionally, I'm always wary of the term "real applications" -- what > > a "real" application is to a US DOE lab is very different than what a > > "real application" is to, say, someone in the oil and gas > > industry. :-) The scale difference alone is enough to make the > > latency gains by inlining select MPI functions important (meaning: non- > > US-DOE-labs tend to run at [much] smaller scale; fabric latency may be > > constrained to a single switch, and therefore *can* see benefit from > > inlining). > > > > As I understand Rainer's proposal, the inlining aspects also > > predicated on apps that enter production and are never changed. > > Perhaps this is not a pattern used much at DOE labs, but it is a > > pattern I see with many customers. > > I'm not going to get into a pissing match with you over what > applications do what. You seem to almost be using "US DOE lab" > pejoratively. What I can say without the slightest hesitation > is that recompilation is often a significant barrier to tool > acceptance for application users and those users are NOT > limited to one site or even one type of site (yes, I know users > who are not at "DOE labs"). > > My opposition to changes that would force tools to require > recompilation is borne from experience. > > Bronis > _______________________________________________ > mpi3-subsetting mailing list > mpi3-subsetting_at_[hidden] > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-subsetting -- ---------------------------------------------------------------- Dipl.-Inf. Rainer Keller http://www.hlrs.de/people/keller HLRS Tel: ++49 (0)711-685 6 5858 Nobelstrasse 19 Fax: ++49 (0)711-685 6 5832 70550 Stuttgart email: keller_at_[hidden] Germany AIM/Skype:rusraink From bronis at [hidden] Fri Mar 7 15:35:33 2008 From: bronis at [hidden] (Bronis R. de Supinski) Date: Fri, 7 Mar 2008 13:35:33 -0800 (PST) Subject: [Mpi3-subsetting] MPI3: Proposal to remove PMPI-Requirement In-Reply-To: <200803072144.55962.keller@hlrs.de> Message-ID: Rainer: There are two kinds of optional. In the paragraph you were responding to, I meant that the user would be explicitly required to include the profiling interface. This requirement is generally a bad idea. Recompilation is a huge barrier to tool adoption. As I have tried to make clear in my comments, requiring recompilation is not a good solution. You would have to demonstrate that the performance benefits are significant across a range of architectures and implementations to justify it. I do not believe that you can demonstrate that since most implementations would eventually make some function call. Eliminating the API level call would not provide much benefit in that case; you could instead inline everything below the API and not make anything public. That solution comes at no cost. Alternatively, linking to a non-standard library is reasonable solution for the rare case when it does make a difference (and the code is not constantly in development, as is the case for MOST large scale applications). Bronis On Fri, 7 Mar 2008, Rainer Keller wrote: > Hello Bronis, > thanks for Your comments. > > On Thursday 06 March 2008 17:58, Bronis R. de Supinski wrote: > > There are some significant problems with this proposal. > > The premise that the profiling interface is only used > > early in the development is not correct. The PMPI interface > > is useful at almost every stage of the application's life > > time. One of the things that makes it successful is that > > it is always available and does not require recompilation. > > Making it optional would destroy that. > This would not make it optional. It still is required by the standard. > However, if the user wishes to do so, he may ask the implementation to inline > functions (and on top of that specify some characterstics of his/her app). > > > > I can tell you that one of the biggest problems for OpenMP > > has been the lack of a profiling interface. We recently > > had one sanctioned by the OpenMP ARB but that took a lot > > of effort since it is very hard to add it in after the fact. > > Your proposal would not be as drastic as eliminating the > > interface but the effect would be almost as catastrophic. > Hmm, I do follow Your argument on OpenMP that is a real issue... > However fail to see, why it's so dramatic in our case (see above). > > > Perhaps just as importantly to the general need of the > > interface, the idea that inlining of MPI will provide > > substantial benefits is inaccurate. MPI is a library > > interface. Many implementations are distributed only in > > binary form; > They do provide at least one header file, now. > They could provide other header files. > They could provide obviouscated header files which call into specialized > functions, that do not check for any_source ,-] > > > > unless you are proposing link time inlining (possible but not without its > > pitfalls), > No. > > > they will not gain any benefit from this. Even if we consider only open > > source MPI implementations, the VAST majority of users do NOT > > want to compile the MPI library. The possible performance > > gain is far too small for them to take on this task. > Errr, they do not need to compile the MPI library at all? > They just > #define MPI_HINT_INLINE > #include "mpi.h" > > and the magic happens. > > > A properly designed subsetting approach will NOT try to > > partition the profiling interface into its own subset. > Yeah, that was probably the biggest misunderstanding. Please see the other > mail ,-] > > > Instead, the interface must be partitioned into different > > subsets, with each PMPI function in the same subset as the > > corresponding MPI function. > Yes. > > > Thanks, > Rainer > > > > On Wed, 5 Mar 2008, Rainer Keller wrote: > > > Dear Alexander, dear all, > > > at the previous Chicago meeting, some of us (Rich Graham, Jeff Squyres, > > > Hubert Ritzdorf) have been discussing about the MPI-Standard's > > > requirement to provide PMPI-functions for each MPI-Call into the library > > > (except MPI_Wtime, MPI_Wtick). > > > > > > This is in my eyes a limitation: there could be some gains for the common > > > case of large-scale applications: > > > - Not using PMPI-based tools (dynamically loaded in) > > > - Not using MPI_PROC_NULL > > > - Not using MPI_STATUS_IGNORE > > > > > > So, if this proposal would fit Your definition of MPI3-Subsets, I would > > > like to discuss the removal of the PMPI-function requirement, allowing > > > inlining of MPI-functions and, based on that, pre-processor hints, e.g. > > > MPI_HINT_NO_THREADS or MPI_HINT_NO_ANY_SOURCE, so the compiler might be > > > able to eliminate several if-statements in the fast-path, allowing > > > inlining, allowing dead code elimination. > > > (of course, there is a whole lot of further issues involved, here -- > > > please see the last section in the document). > > > > > > I have included some timings with NetPipe on Open MPI using these hints > > > (further information section 4 -- Ompi Patch availble upon request, as > > > its work in progress). Please take the timings with a grain of salt. I > > > would like to get better figures on a more well-defined cluster > > > environment. > > > > > > Any comments would be welcome. > > > > > > With best regards, > > > Rainer > > > -- > > > ---------------------------------------------------------------- > > > Dipl.-Inf. Rainer Keller http://www.hlrs.de/people/keller > > > HLRS Tel: ++49 (0)711-685 6 5858 > > > Nobelstrasse 19 Fax: ++49 (0)711-685 6 5832 > > > 70550 Stuttgart email: keller_at_[hidden] > > > Germany AIM/Skype:rusraink > > > > _______________________________________________ > > mpi3-subsetting mailing list > > mpi3-subsetting_at_[hidden] > > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-subsetting > > -- > ---------------------------------------------------------------- > Dipl.-Inf. Rainer Keller http://www.hlrs.de/people/keller > HLRS Tel: ++49 (0)711-685 6 5858 > Nobelstrasse 19 Fax: ++49 (0)711-685 6 5832 > 70550 Stuttgart email: keller_at_[hidden] > Germany AIM/Skype:rusraink > From alexander.supalov at [hidden] Sat Mar 8 13:00:10 2008 From: alexander.supalov at [hidden] (Supalov, Alexander) Date: Sat, 8 Mar 2008 19:00:10 -0000 Subject: [Mpi3-subsetting] agenda items for March 11 WG meeting? Message-ID: <5ECAB1304A8B5B4CB3F9D6C01E4E21A2011F4E8F@swsmsx413.ger.corp.intel.com> Hi everybody, Please send me any agenda items you want to cover when we meet 5:45 pm - 7:15 pm on March 11. At the moment I'd propose the following: - Opens - Scope and direction of the WG - Standard modularization to better fit different application areas - 2.Performance advantages for reasonable module combinations - 3.Influence upon the overall shape of the MPI-3 standard - Discussion - Optional: really optional (may be not present) or selectable (are present but may be unused)? - Performance penalty for unused subsets: implementation matter or standard choice? - PMPI: subset or not? - - Next steps - Getting started on the proposal - Division of labor Best regards. Alexander -- Dr Alexander Supalov Intel GmbH Hermuelheimer Strasse 8a 50321 Bruehl, Germany Phone: +49 2232 209034 Mobile: +49 173 511 8735 Fax: +49 2232 209029 --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. * -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsquyres at [hidden] Sun Mar 9 17:25:36 2008 From: jsquyres at [hidden] (Jeff Squyres) Date: Sun, 9 Mar 2008 17:25:36 -0500 Subject: [Mpi3-subsetting] MPI3: Proposal to remove PMPI-Requirement In-Reply-To: Message-ID: <5781C272-28E1-4FE5-8783-9E8C1AABE95F@cisco.com> On Mar 7, 2008, at 10:20 AM, Bronis R. de Supinski wrote: >> Additionally, I'm always wary of the term "real applications" -- what >> a "real" application is to a US DOE lab is very different than what a >> "real application" is to, say, someone in the oil and gas >> industry. :-) The scale difference alone is enough to make the >> latency gains by inlining select MPI functions important (meaning: >> non- >> US-DOE-labs tend to run at [much] smaller scale; fabric latency may >> be >> constrained to a single switch, and therefore *can* see benefit from >> inlining). >> >> As I understand Rainer's proposal, the inlining aspects also >> predicated on apps that enter production and are never changed. >> Perhaps this is not a pattern used much at DOE labs, but it is a >> pattern I see with many customers. > > I'm not going to get into a pissing match with you over what > applications do what. You seem to almost be using "US DOE lab" > pejoratively. Not at all. All I'm trying to say is that there are many different types of apps out there and many different usage scenarios. What is appropriate for one kind of app run in one kind of scenario is not necessarily appropriate for another. > What I can say without the slightest hesitation > is that recompilation is often a significant barrier to tool > acceptance for application users and those users are NOT > limited to one site or even one type of site (yes, I know users > who are not at "DOE labs"). > > My opposition to changes that would force tools to require > recompilation is borne from experience. I agree with that point. I, too, have some reservations against macro- based optimizations, but I would think that if someone *chooses* to use them, they they have implicitly agreed to forego some classes of tools and other kinds of features that would be incompatible with said optimizations. -- Jeff Squyres Cisco Systems From alexander.supalov at [hidden] Fri Mar 28 11:16:17 2008 From: alexander.supalov at [hidden] (Supalov, Alexander) Date: Fri, 28 Mar 2008 16:16:17 -0000 Subject: [Mpi3-subsetting] Subsetting WG telecon: dates/time, proposal draft to write Message-ID: <5ECAB1304A8B5B4CB3F9D6C01E4E21A201309F37@swsmsx413.ger.corp.intel.com> Hi everybody, It's time for us to meet again virtually and discuss how to move forward on the subsetting. Here's a list of dates/times that are free in my schedule in the coming two weeks. Please let me know whether you can make any of the following slots: Wednesday, April 2, 8:00 am PDT/10:00 CDT/12:00 pm EDT/17:00 CET yes/no/plan B Wednesday, April 2, 9:00 am PDT/11:00 CDT/ 1:00 pm EDT/18:00 CET yes/no/plan B Thursday, April 3, 9:00 am PDT/11:00 CDT/ 1:00 pm EDT/18:00 CET yes/no/plan B Wednesday, April 9, 8:00 am PDT/10:00 CDT/12:00 pm EDT/17:00 CET yes/no/plan B Wednesday, April 9, 9:00 am PDT/11:00 CDT/ 1:00 pm EDT/18:00 CET yes/no/plan B Thursday, April 10, 9:00 am PDT/11:00 CDT/ 1:00 pm EDT/18:00 CET yes/no/plan B I hope that the transition to summer time is over next week, and we can count on the usual time difference as indicated above. A discussion of the subsetting relation to the structure of the MPI-3 standard is likely to come up again at the April Forum meeting, so we better start writing something up prior to the meeting. I will put together a first draft by early next week for starters and will ask everybody to contribute. Best regards. Alexander -- Dr Alexander Supalov Intel GmbH Hermuelheimer Strasse 8a 50321 Bruehl, Germany Phone: +49 2232 209034 Mobile: +49 173 511 8735 Fax: +49 2232 209029 --------------------------------------------------------------------- Intel GmbH Dornacher Strasse 1 85622 Feldkirchen/Muenchen Germany Sitz der Gesellschaft: Feldkirchen bei Muenchen Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer Registergericht: Muenchen HRB 47456 Ust.-IdNr. VAT Registration No.: DE129385895 Citibank Frankfurt (BLZ 502 109 00) 600119052 This e-mail and any attachments may contain confidential material for the sole use of the intended recipient(s). Any review or distribution by others is strictly prohibited. If you are not the intended recipient, please contact the sender and delete all copies. * -------------- next part -------------- An HTML attachment was scrubbed... URL: From jsquyres at [hidden] Mon Mar 31 07:00:58 2008 From: jsquyres at [hidden] (Jeff Squyres) Date: Mon, 31 Mar 2008 08:00:58 -0400 Subject: [Mpi3-subsetting] Subsetting WG telecon: dates/time, proposal draft to write In-Reply-To: <5ECAB1304A8B5B4CB3F9D6C01E4E21A201309F37@swsmsx413.ger.corp.intel.com> Message-ID: <46FF6458-5810-4C4F-AB0D-ABBAC7760F70@cisco.com> On Mar 28, 2008, at 12:16 PM, Supalov, Alexander wrote: > Hi everybody, > > It's time for us to meet again virtually and discuss how to move > forward on the subsetting. Here's a list of dates/times that are > free in my schedule in the coming two weeks. Please let me know > whether you can make any of the following slots: > > Wednesday, April 2, 8:00 am PDT/10:00 CDT/12:00 pm EDT/17:00 > CET yes/no/plan B > Wednesday, April 2, 9:00 am PDT/11:00 CDT/ 1:00 pm EDT/18:00 > CET yes/no/plan B > Thursday, April 3, 9:00 am PDT/11:00 CDT/ 1:00 pm EDT/18:00 > CET yes/no/plan B All of the above are ok with me. > Wednesday, April 9, 8:00 am PDT/10:00 CDT/12:00 pm EDT/17:00 > CET yes/no/plan B Difficult for me, but do-able if necessary. > Wednesday, April 9, 9:00 am PDT/11:00 CDT/ 1:00 pm EDT/18:00 > CET yes/no/plan B I have a conflict. > Thursday, April 10, 9:00 am PDT/11:00 CDT/ 1:00 pm EDT/18:00 > CET yes/no/plan B Difficult but do-able if necessary. -- Jeff Squyres Cisco Systems From rlgraham at [hidden] Mon Mar 31 08:20:30 2008 From: rlgraham at [hidden] (Richard Graham) Date: Mon, 31 Mar 2008 09:20:30 -0400 Subject: [Mpi3-subsetting] Subsetting WG telecon: dates/time, proposal draft to write In-Reply-To: <5ECAB1304A8B5B4CB3F9D6C01E4E21A201309F37@swsmsx413.ger.corp.intel.com> Message-ID: April 2nd and 3rd will not work for me. Either the 9th or the 10th WILL work for me. Thanks, Rich On 3/28/08 12:16 PM, "Supalov, Alexander" wrote: > Hi everybody, > > It's time for us to meet again virtually and discuss how to move forward on > the subsetting. Here's a list of dates/times that are free in my schedule in > the coming two weeks. Please let me know whether you can make any of the > following slots: > > Wednesday, April 2, 8:00 am PDT/10:00 CDT/12:00 pm EDT/17:00 CET > yes/no/plan B > Wednesday, April 2, 9:00 am PDT/11:00 CDT/ 1:00 pm EDT/18:00 CET > yes/no/plan B > Thursday, April 3, 9:00 am PDT/11:00 CDT/ 1:00 pm EDT/18:00 CET > yes/no/plan B > Wednesday, April 9, 8:00 am PDT/10:00 CDT/12:00 pm EDT/17:00 CET > yes/no/plan B > Wednesday, April 9, 9:00 am PDT/11:00 CDT/ 1:00 pm EDT/18:00 CET > yes/no/plan B > Thursday, April 10, 9:00 am PDT/11:00 CDT/ 1:00 pm EDT/18:00 CET > yes/no/plan B > > I hope that the transition to summer time is over next week, and we can count > on the usual time difference as indicated above. > > A discussion of the subsetting relation to the structure of the MPI-3 standard > is likely to come up again at the April Forum meeting, so we better start > writing something up prior to the meeting. I will put together a first draft > by early next week for starters and will ask everybody to contribute. > > Best regards. > > Alexander > > -- > Dr Alexander Supalov > Intel GmbH > Hermuelheimer Strasse 8a > 50321 Bruehl, Germany > Phone: +49 2232 209034 > Mobile: +49 173 511 8735 > Fax: +49 2232 209029 > > --------------------------------------------------------------------- > Intel GmbH > Dornacher Strasse 1 > 85622 Feldkirchen/Muenchen Germany > Sitz der Gesellschaft: Feldkirchen bei Muenchen > Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer > Registergericht: Muenchen HRB 47456 Ust.-IdNr. > VAT Registration No.: DE129385895 > Citibank Frankfurt (BLZ 502 109 00) 600119052 > > This e-mail and any attachments may contain confidential material for > the sole use of the intended recipient(s). Any review or distribution > by others is strictly prohibited. If you are not the intended > recipient, please contact the sender and delete all copies. > > > _______________________________________________ > mpi3-subsetting mailing list > mpi3-subsetting_at_[hidden] > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-subsetting * -------------- next part -------------- An HTML attachment was scrubbed... URL: From Terry.Dontje at [hidden] Mon Mar 31 08:26:43 2008 From: Terry.Dontje at [hidden] (Terry Dontje) Date: Mon, 31 Mar 2008 09:26:43 -0400 Subject: [Mpi3-subsetting] Subsetting WG telecon: dates/time, proposal draft to write In-Reply-To: <5ECAB1304A8B5B4CB3F9D6C01E4E21A201309F37@swsmsx413.ger.corp.intel.com> Message-ID: <47F0E693.5000709@sun.com> Supalov, Alexander wrote: > Hi everybody, > > It's time for us to meet again virtually and discuss how to move > forward on the subsetting. Here's a list of dates/times that are free > in my schedule in the coming two weeks. Please let me know whether you > can make any of the following slots: > > Wednesday, April 2, 8:00 am PDT/10:00 CDT/12:00 pm EDT/17:00 CET > yes/no/plan B Yes > Wednesday, April 2, 9:00 am PDT/11:00 CDT/ 1:00 pm EDT/18:00 CET > yes/no/plan B Yes > Thursday, April 3, 9:00 am PDT/11:00 CDT/ 1:00 pm EDT/18:00 > CET yes/no/plan B Yes > Wednesday, April 9, 8:00 am PDT/10:00 CDT/12:00 pm EDT/17:00 CET > yes/no/plan B Conflict but possibly doable > Wednesday, April 9, 9:00 am PDT/11:00 CDT/ 1:00 pm EDT/18:00 CET > yes/no/plan B Conflict but possibly doable > Thursday, April 10, 9:00 am PDT/11:00 CDT/ 1:00 pm EDT/18:00 > CET yes/no/plan B No > > I hope that the transition to summer time is over next week, and we > can count on the usual time difference as indicated above. > > A discussion of the subsetting relation to the structure of the MPI-3 > standard is likely to come up again at the April Forum meeting, so we > better start writing something up prior to the meeting. I will put > together a first draft by early next week for starters and will ask > everybody to contribute. > > Best regards. > > Alexander > > -- > Dr Alexander Supalov > Intel GmbH > Hermuelheimer Strasse 8a > 50321 Bruehl, Germany > Phone: +49 2232 209034 > Mobile: +49/ /173 511 8735 > Fax: +49 2232 209029 > > --------------------------------------------------------------------- > Intel GmbH > Dornacher Strasse 1 > 85622 Feldkirchen/Muenchen Germany > Sitz der Gesellschaft: Feldkirchen bei Muenchen > Geschaeftsfuehrer: Douglas Lusk, Peter Gleissner, Hannes Schwaderer > Registergericht: Muenchen HRB 47456 Ust.-IdNr. > VAT Registration No.: DE129385895 > Citibank Frankfurt (BLZ 502 109 00) 600119052 > > This e-mail and any attachments may contain confidential material for > the sole use of the intended recipient(s). Any review or distribution > by others is strictly prohibited. If you are not the intended > recipient, please contact the sender and delete all copies. > > ------------------------------------------------------------------------ > > _______________________________________________ > mpi3-subsetting mailing list > mpi3-subsetting_at_[hidden] > http://lists.mpi-forum.org/mailman/listinfo.cgi/mpi3-subsetting > From keller at [hidden] Mon Mar 31 08:26:08 2008 From: keller at [hidden] (Rainer Keller) Date: Mon, 31 Mar 2008 15:26:08 +0200 Subject: [Mpi3-subsetting] Subsetting WG telecon: dates/time, proposal draft to write In-Reply-To: <5ECAB1304A8B5B4CB3F9D6C01E4E21A201309F37@swsmsx413.ger.corp.intel.com> Message-ID: <200803311526.08678.keller@hlrs.de> Dear Alexander, On Friday 28 March 2008 17:16, Supalov, Alexander wrote: > It's time for us to meet again virtually and discuss how to move forward > on the subsetting. Here's a list of dates/times that are free in my > schedule in the coming two weeks. Please let me know whether you can > make any of the following slots: > > Wednesday, April 2, 8:00 am PDT/10:00 CDT/12:00 pm EDT/17:00 CET > yes/no/plan B > Wednesday, April 2, 9:00 am PDT/11:00 CDT/ 1:00 pm EDT/18:00 CET > yes/no/plan B > Thursday, April 3, 9:00 am PDT/11:00 CDT/ 1:00 pm EDT/18:00 CET > yes/no/plan B These would be fine. > Wednesday, April 9, 8:00 am PDT/10:00 CDT/12:00 pm EDT/17:00 CET > yes/no/plan B > Wednesday, April 9, 9:00 am PDT/11:00 CDT/ 1:00 pm EDT/18:00 CET > yes/no/plan B > Thursday, April 10, 9:00 am PDT/11:00 CDT/ 1:00 pm EDT/18:00 CET > yes/no/plan B April 10 would be plan B but doable. Thanks, Rainer -- ---------------------------------------------------------------- Dipl.-Inf. Rainer Keller http://www.hlrs.de/people/keller HLRS Tel: ++49 (0)711-685 6 5858 Nobelstrasse 19 Fax: ++49 (0)711-685 6 5832 70550 Stuttgart email: keller_at_[hidden] Germany AIM/Skype:rusraink