[MPI3 Fortran] [Mpi-comments] MPI 3.0: Fortran 2008 interface - issue with the LOGICAL kind

[Bill Long]

On 3/25/13 3:28 PM, Hubert Ritzdorf wrote:
> I vote for Option 2.
>
> Option 3 is not possible for an errata. This is a severe change in the standard.
> Your modification of Option 3 makes the standard more difficult (we have again mixing of
> BIND(C) and non BIND(C) functions) and I can't see a real application advantage.
>
> To your items:
> Item 2: You have to perform also additional work in the wrappers
>               (for example memory allocation and transformation for sources
>                compiled with -i8; additional error checks)

There is a trivial amount of work in the wrapper - this is intentional. 
  Here is a simple  example from the mpi_f08 module:

subroutine MPI_Abort(comm, errorcode, ierror)
      use,intrinsic :: iso_c_binding, only: c_int
      use :: mpi_f08_types, only: MPI_Comm
      use :: mpi_C_interfaces_nobuf, only:  C_Comm, MPI_Abort_C
      TYPE(MPI_Comm),      INTENT(IN)  :: comm
      INTEGER,             INTENT(IN)  :: errorcode
      INTEGER, OPTIONAL,   INTENT(OUT) :: ierror
      integer(C_Comm) :: comm_c
      integer(c_int)  :: errorcode_c
      integer(c_int)  :: res

      comm_c = comm%MPI_VAL
      errorcode_c = errorcode
      res = MPI_Abort_C (comm_c, errorcode_c)
      if (present(ierror)) ierror = res

end subroutine MPI_Abort

The local xxx_c variables are the ones for the C routine; the ones 
without the _c are the Fortran dummy arguments.   I have not encountered 
a case where memory allocation is needed, at least yet.   To accommodate 
-i8,  you only recompile the modules with -i8. There is no change to the 
module source.  This is the intent of the design.  The three assignment 
statements are, in effect, 'casts' between the Fortran and C types.  If 
there is no difference, the compiler could simplify the executable code.

In this particular case, the code following the MPI_Abort call to C 
might not be necessary, since I would expect the routine to not return. 
  But the same pattern is used elsewhere to deal with the error return 
value.



> Item 3: Support of PMPI version of Fortran is not really an issue.
> Item 4b: The tools have to support routines compiled with mpi.h and old
>                  objects.

That is true.  For that mode, the text corresponding text from the mpi 
module (with INTEGER handle for comm and the use of mpi_f08_types 
removed) is extracted to a file without the module wrapping.  Compiling 
that file will result in a routine with an external symbol of mpi_abort_ 
  (or something similar, depending on the conventions of the compiler). 
   That version would be called only in the case that the user included 
mpif.h.  A particularly aggressive compiler that defers inlining until 
the link step might even inline this case.

Cheers,
Bill

>
> Hubert
>
> ________________________________________
> From: Bill Long [longb at cray.com]
> Sent: Monday, March 25, 2013 7:35 PM
> To: Craig Rasmussen
> Cc: Rolf Rabenseifner; MPI-3 Fortran working group; Jeff Squyres (jsquyres); Hubert Ritzdorf; Martin Schulz; Tobias Burnus
> Subject: Re: [MPI3 Fortran] [Mpi-comments] MPI 3.0: Fortran     2008 interface - issue with the LOGICAL kind
>
> On 3/25/13 12:50 PM, Craig Rasmussen wrote:
>> My summary of the three options:
>>
>> 1. Absolute minimal change, remove BIND(C) only for routines with
>> LOGICAL arguments.  Tools will need to support underscore name mangling
>> forever (even when TS29113 is implemented by compilers) but only on
>> routines with LOGICAL arguments.
>>
>> 2. Remove BIND(C) from all routines.  Tools will need to support
>> underscore name mangling for all routines.
>>
>> 3. Remove BIND(C) from all routines.  Plus allow Fortran implementations
>> to substitute the C PMPI interface standard.
>>
>> Is this a fair summary?
>>
>> My vote:
>>
>> I'm strongly in favor of option 3.  It is the only option that allows
>> the possibility of underscore name mangling disappearing in the future.
>>    I think most (if not all) MPI Fortran implementations will eventually
>> evolve to thin wrappers calling the C API.  There are several advantages
>> with this implementation:
>>       1. The thin wrappers can be put in a module.
>>       2. Compilers may be able to inline the wrappers for very efficient
>> implementations.
>>       3. Fortran implementations are much simpler because they won't have
>> to directly support PMPI in Fortran.
>>       4. Since the Fortran implementation is substantially less complex,
>> the costs associated with testing, supporting, and maintaining a Fortran
>> implementation will also be reduced.
>>       4. If all MPI implementations eventually choose to call the C API,
>> then the tools community eventually will not have to support underscore
>> name mangling, thus reducing their costs as well.
>
> I like option 3 as well.  It uses the C interoperability feature in
> Fortran the way it was intended/designed.   I would add to the above:
>
> 6. You can isolate the issues of Fortran types (compiled with -i8,
> meaning of LOGICAL, character arguments with len > 1) into the Fortran
> wrappers so that the C library writers do not have to know about these
> issues.   Simplifies the maintenance and testing issues for the C
> library code.
>
>
> I would modify option 3 slightly. The implementer might want to write
> some routine ONLY in Fortran.  The routines that convert the status
> formats between the Fortran and the C forms comes to mind.  In that
> case, BIND(C) would be required, and it would specify the C binding name
> as the external linker name. This would be needed so that C codes could
> call the routine.
>
>
> Cheers,
> Bill
>
>
>>
>> Craig Rasmussen
>> CAS Scientific Programmer
>> rasmus at cas.uoregon.edu <mailto:rasmus at cas.uoregon.edu>
>>
>>
>>
>>
>> On Mar 25, 2013, at 10:11 AM, Rolf Rabenseifner wrote:
>>
>>> We (Jeff, Bill L., Craig, Hubert, Martin, Tobias, and me (Rolf)) had
>>> a telecon today and the outcome are following options:
>>>
>>> Option 1
>>> - #364 as it currently is, as a minimal solution for
>>>   the LOGICAL problem:
>>>     Pro: It is consistent between Fortran and MPI.
>>>          I.e., it is the minimal solution that we must do.
>>>     Con: It has a mix of BIND(C) and non-BIND(C) routine groups
>>>          which is ugly for the future.
>>>     Remark: non-BIND(C) will stay there because the callbacks
>>>             need non-BIND(C) in the moment and this cannot be
>>>             changed in the future by nice new concepts from
>>>             the Fortran language standardization.
>>>
>>> Option 2
>>> - #364 plus changing MPI-3.0 page 605 lines 24-30 and
>>>   page 605 line 47 - page 606 line 6 by also using non-BIND(C),
>>>   but other linker names:
>>>    MPI_Send_f08d (in mpi_f08 with TS29113 quality, i.e.,
>>>                   buffers are done with C-descriptors and
>>>                   not with void* pointers.)
>>>    MPI_Send_fd (in the mpi module or mpif.h if
>>>                 MPI_SUBARRAYS_SUPPORTED=.TRUE. is also done
>>>                 there, i.e., with the descriptors.)
>>>
>>>     Pro: It is consistent between Fortran and MPI
>>>          and it is consistent within MPI, i.e., all
>>>          functionality is done with non-BIND(C) routines.
>>>          BIND(C) can be still used internally to
>>>          call MPI_Send_cdesc and MPI_Isend_cdesc.
>>>          To be fair, implementatiosn will be based mainly
>>>          on this TS 29113 feature, but only internally.
>>>     Con: No BIND(C) on the official MPI interface.
>>>          We urged Fortran to put some features into TS 29113,
>>>          but we will not use these features, because they
>>>          are not needed on this internal interface.
>>>
>>> Option 3:
>>> - Option 2 plus the following rule:
>>>    - The Fortran PMPI interface can be substituted by a
>>>      wrapper based sulution that internally calls the full
>>>      MPI C interface.
>>>    - In this case, the Fortran interface should be hidden
>>>      to the tools.
>>>    - For the buffer routines with the new cdesc descriptor,
>>>      new interfaces are added for the tools people,
>>>      e.g., MPI_Send_cdesc and MPI_Isend_cdesc.
>>>
>>>    Pro: I'm not sure whether there is a real pro?
>>>
>>>         Bill, can you add the pros, it is your option.
>>>
>>>    Con: An additional new interface for the tools folk.
>>>         It is not backward compatible for users who really
>>>         intercept the Fortran MPI interface.
>>>
>>> My favorite in the moment, i.e. at the starting of
>>> the pro-con-discussion: Option 2.
>>>
>>> Best regards
>>> Rolf
>>>
>>>
>>>
>>> ----- Original Message -----
>>>> From: "Rolf Rabenseifner" <rabenseifner at hlrs.de
>>>> <mailto:rabenseifner at hlrs.de>>
>>>> To: longb at cray.com <mailto:longb at cray.com>
>>>> Cc: "Jeff Squyres (jsquyres)" <jsquyres at cisco.com
>>>> <mailto:jsquyres at cisco.com>>, "Craig E Rasmussen"
>>>> <rasmus at cas.uoregon.edu <mailto:rasmus at cas.uoregon.edu>>, "Hubert
>>>> Ritzdorf"
>>>> <Hubert.Ritzdorf at emea.nec.com <mailto:Hubert.Ritzdorf at emea.nec.com>>,
>>>> "Martin Schulz" <schulzm at llnl.gov <mailto:schulzm at llnl.gov>>, "Tobias
>>>> Burnus" <burnus at net-b.de <mailto:burnus at net-b.de>>
>>>> Sent: Monday, March 25, 2013 12:21:39 PM
>>>> Subject: Re: [MPI3 Fortran] [Mpi-comments] MPI 3.0: Fortran 2008
>>>> interface - issue with the LOGICAL kind
>>>> Yes, this freedom (1)-(4) in Bill's email is important.
>>>>
>>>> In our discussion, we forgot one additional reason why we defined
>>>> it as it is written in MPI-3.0 Section 17.1.5, page 605:
>>>>
>>>> - It should be possible to implement the interfaces with
>>>> these non-standard Fortran-void-* directives,
>>>> and later on,
>>>> to substitute this implementation by new ones
>>>> that are based on TS 29113.
>>>> - When doing this substitution, it should be save for
>>>> applications when the application is not re-compiled.
>>>> - More than that, it should be possible to combine older
>>>> .o files with newly TS 29113 based .o files.
>>>> - For this, we decided, that the implementation has to switch
>>>> from non-BIND(C) to BIND(C) at least in the MPI_ISEND
>>>> routine group when switching to MPI_SUBARRAYS_SUPPORTED==.TRUE.
>>>> - The real trick is that the linker name must be changed.
>>>> - This all is valid for both, the new mpi_f08 module,
>>>> and the old mpi module.
>>>>
>>>> A second issue in our discussion is always, which changes
>>>> we want to do to the mpi_f08 MPI-3.0 user interface.
>>>> I want to remember that this user interface is defined
>>>> according following rules.
>>>> If a user modifies his source to use everywhere mpi_f08,
>>>> he/she has to modify:
>>>> - He/she has to substitute the "include 'mpif.h'" or "use mpi"
>>>> statements by "use mpi_f08".
>>>> - He/she has to modify all handle and status declarations.
>>>> - In the executable statements, he/she has to substitute all
>>>> status(MPI_TAG), status(MPI_SOURCE), status(MPI_ERROR)
>>>> accesses into
>>>> status%MPI_TAG, status%MPI_SOURCE, status%MPI_ERROR.
>>>> - Callback routines need more effort due to the
>>>> abstract interface definitions.
>>>> This is touched by ticket #364, i.e.,
>>>> the MPI-3.0 BIND(C) of callbacks had to be removed.
>>>> - Changes in the interfaces are only
>>>> -- all handles use the new handle types
>>>> and in
>>>> -- MPI_User_function,
>>>> -- MPI_Datarep_conversion_function,
>>>> the buffers are done with TYPE(C_TR), VALUE.
>>>>
>>>> This should be all.
>>>> All other user correct code should directly compile with the new
>>>> mpi_f08 module.
>>>> We should not change these things, except the necessary
>>>> errata in #364.
>>>>
>>>> I see that we have some freedom only in the area,
>>>> what we do internally (together with linker names) and with PMPI,
>>>> but not in the user interface.
>>>> If we add additional user interfaces (e.g. overloading the LOGICAL
>>>> with INTEGER arguments)
>>>> we would only make it more complicate.
>>>>
>>>> Best regards
>>>> Rolf
>>>>
>>>>
>>>> ----- Original Message -----
>>>>> From: "Bill Long" <longb at cray.com <mailto:longb at cray.com>>
>>>>> To: "Rolf Rabenseifner" <rabenseifner at hlrs.de
>>>>> <mailto:rabenseifner at hlrs.de>>
>>>>> Cc: "Jeff Squyres (jsquyres)" <jsquyres at cisco.com
>>>>> <mailto:jsquyres at cisco.com>>, "Craig E
>>>>> Rasmussen" <rasmus at cas.uoregon.edu <mailto:rasmus at cas.uoregon.edu>>,
>>>>> "Hubert Ritzdorf"
>>>>> <Hubert.Ritzdorf at emea.nec.com
>>>>> <mailto:Hubert.Ritzdorf at emea.nec.com>>, "Martin Schulz"
>>>>> <schulzm at llnl.gov <mailto:schulzm at llnl.gov>>,
>>>>> "Tobias Burnus" <burnus at net-b.de <mailto:burnus at net-b.de>>
>>>>> Sent: Sunday, March 24, 2013 5:59:38 PM
>>>>> Subject: Re: [MPI3 Fortran] [Mpi-comments] MPI 3.0: Fortran 2008
>>>>> interface - issue with the LOGICAL kind
>>>>> This time is OK for me. I'll look for the webex email.
>>>>>
>>>>> I would like to note that the various options discussed are not
>>>>> necessarily "all or nothing" propositions. An implementation could
>>>>> use
>>>>> the thin wrappers in a module approach to eliminate the bad Fortran
>>>>> types problems for most of the MPI routines, but also have some
>>>>> routines
>>>>> implemented in a different way, such as (1) all in Fortran that is
>>>>> callable from C, (2) all in C that is called through an interface
>>>>> directly from Fortran, (3) as a fat wrapper in Fortran, or (4) a
>>>>> completely replicated routine for the Fortran implementation. There
>>>>> is
>>>>> no reason that all of the routines have to be the same method - in
>>>>> fact
>>>>> probably good arguments against that in some cases. We just need to
>>>>> have
>>>>> the rules such that the tools have some known name to intercept.
>>>>> Along
>>>>> the lines of "If implementation method X is chosen for routine
>>>>> MPI_xxx,
>>>>> then these are the linker name requirements:.....".
>>>>>
>>>>> Cheers,
>>>>> Bill
>>>>>
>>>>>
>>>>>
>>>>> On 3/23/13 2:11 AM, Rolf Rabenseifner wrote:
>>>>>> Dear Jeff, Craig, Bill, Hubert, Martin, and Tobias,
>>>>>>
>>>>>> I expect that this is the total group that discusses the
>>>>>> Fortran LOGICAL etc. problem.
>>>>>> We learnt that the solutions also touch the PMPI concept.
>>>>>> There was a telecon between Craig, Jeff and Bill with the proposal
>>>>>> at the end of this email.
>>>>>> I like to understand this proposal and therefore I asked Jeff
>>>>>> for explanations. I would like to open this for the whole group.
>>>>>>
>>>>>>> I have meetings at 9am and 1pm is eastern on Monday. I can talk
>>>>>>> any
>>>>>>> other time.
>>>>>>> Craig prob can't attend because he's 3 hours earlier, but that's
>>>>>>> prob ok.
>>>>>>
>>>>>> Monday March 25,
>>>>>>   - 4 PM Central European Time CET
>>>>>>   = 11 AM Eastern Daylight Time EST/EDT
>>>>>>   = 8 AM Pacific Time
>>>>>>
>>>>>> I hope these relations are correct. Would this work for you?
>>>>>>
>>>>>> At least for Jeff and me, because I want to understand
>>>>>> what they discussed with their proposal.
>>>>>>
>>>>>> Jeff, can you set up a webex?
>>>>>>
>>>>>> Best regards
>>>>>> Rolf
>>>>>>
>>>>>> ----- Original Message -----
>>>>>>> From: "Jeff Squyres (jsquyres)" <jsquyres at cisco.com
>>>>>>> <mailto:jsquyres at cisco.com>>
>>>>>>> To: "Rolf Rabenseifner" <rabenseifner at hlrs.de
>>>>>>> <mailto:rabenseifner at hlrs.de>>
>>>>>>> Cc: "Craig E Rasmussen" <rasmus at cas.uoregon.edu
>>>>>>> <mailto:rasmus at cas.uoregon.edu>>
>>>>>>> Sent: Friday, March 22, 2013 8:01:23 PM
>>>>>>> Subject: Re: [MPI3 Fortran] [Mpi-comments] MPI 3.0: Fortran 2008
>>>>>>> interface - issue with the LOGICAL kind
>>>>>>> Btw, I'm not sure how this is not the least surprise.
>>>>>>>
>>>>>>> We can't change the params to not be logical, yes we must still
>>>>>>> allow
>>>>>>> for calling to c. So the overloaded solution is the only one.
>>>>>>>
>>>>>>> And the new Integer param signature must have a new bind name so
>>>>>>> as
>>>>>>> to
>>>>>>> did ambiguity it from the one we published in MPI3. See my
>>>>>>> previous
>>>>>>> mail about this.
>>>>>>>
>>>>>>> I have meetings at 9am and 1pm is eastern on Monday. I can talk
>>>>>>> any
>>>>>>> other time.
>>>>>>>
>>>>>>> Craig prob can't attend because he's 3 hours earlier, but that's
>>>>>>> prob
>>>>>>> ok.
>>>>>>>
>>>>>>> Sent from my phone. No type good.
>>>>>>>
>>>>>>> On Mar 22, 2013, at 8:51 AM, "Rolf Rabenseifner"
>>>>>>> <rabenseifner at hlrs.de <mailto:rabenseifner at hlrs.de>> wrote:
>>>>>>>
>>>>>>>> Jeff,
>>>>>>>>
>>>>>>>> can I call you by phone?
>>>>>>>> I would like to talk to you as the binding responsible and MPI
>>>>>>>> chair.
>>>>>>>>
>>>>>>>>> 2. Replace the current binding names for MPI functions with
>>>>>>>>> LOGICAL
>>>>>>>>> dummy arguments with a different name and replacing LOGICAL
>>>>>>>>> type
>>>>>>>>> with
>>>>>>>>> default INTEGER arguments. The name must be changed so as to
>>>>>>>>> not
>>>>>>>> ...
>>>>>>>> This is a hard step and far away from "last surprise".
>>>>>>>>
>>>>>>>> I would like to learn how it came out as a result of your
>>>>>>>> telecon,
>>>>>>>> before I try to go the next step (e.g. email answer).
>>>>>>>>
>>>>>>>> Best regards
>>>>>>>> Rolf
>>>>>>>>
>>>>>>>> ----- Original Message -----
>>>>>>>>> From: "Craig Rasmussen" <rasmus at cas.uoregon.edu
>>>>>>>>> <mailto:rasmus at cas.uoregon.edu>>
>>>>>>>>> To: "MPI-3 Fortran working group"
>>>>>>>>> <mpi3-fortran at lists.mpi-forum.org
>>>>>>>>> <mailto:mpi3-fortran at lists.mpi-forum.org>>
>>>>>>>>> Cc: "Hubert Ritzdorf" <Hubert.Ritzdorf at emea.nec.com
>>>>>>>>> <mailto:Hubert.Ritzdorf at emea.nec.com>>, "Martin
>>>>>>>>> Schulz" <schulzm at llnl.gov <mailto:schulzm at llnl.gov>>
>>>>>>>>> Sent: Friday, March 22, 2013 4:29:36 PM
>>>>>>>>> Subject: Re: [MPI3 Fortran] [Mpi-comments] MPI 3.0: Fortran
>>>>>>>>> 2008
>>>>>>>>> interface - issue with the LOGICAL kind
>>>>>>>>>
>>>>>>>>> Bill Long, Jeff Squyres, and I had a phone conversation
>>>>>>>>> yesterday
>>>>>>>>> and came up with a straw man proposal. We believe it fixes the
>>>>>>>>> LOGICAL argument problem while satisfying three major goals:
>>>>>>>>>
>>>>>>>>> 1. Backwards compatibility.
>>>>>>>>> 2. Flexibility so that efficient implementations can be
>>>>>>>>> produced
>>>>>>>>> on
>>>>>>>>>     a variety of hardware.
>>>>>>>>> 3. Not dramatically increase the complexity for the tools
>>>>>>>>>     community.
>>>>>>>>> 4. Ensure that the changes won't be seen by ordinary Fortran
>>>>>>>>> MPI
>>>>>>>>>     users.
>>>>>>>>>
>>>>>>>>> The straw man proposal is:
>>>>>>>>>
>>>>>>>>> 1. Remove the current binding names for MPI functions with
>>>>>>>>> LOGICAL
>>>>>>>>> dummy arguments (. This must be done because these interfaces
>>>>>>>>> break
>>>>>>>>> the Fortran standard.
>>>>>>>>> 2. Replace the current binding names for MPI functions with
>>>>>>>>> LOGICAL
>>>>>>>>> dummy arguments with a different name and replacing LOGICAL
>>>>>>>>> type
>>>>>>>>> with
>>>>>>>>> default INTEGER arguments. The name must be changed so as to
>>>>>>>>> not
>>>>>>>>> conflict with the current name with LOGICAL arguments. For
>>>>>>>>> example,
>>>>>>>>> the binding name "MPI_Test_f08" will be replaced by
>>>>>>>>> "MPI_Test_int_f08"
>>>>>>>>> (exact replacement name open for discussion). The Fortran
>>>>>>>>> implementation of the function MPI_Test must call
>>>>>>>>> MPI_Test_int_f08
>>>>>>>>> as soon as possible for the tools implementers.
>>>>>>>>> 3. Implementation of the Fortran PMPI calling convention will
>>>>>>>>> be
>>>>>>>>> made optional (we think it already it). If an MPI implementer
>>>>>>>>> doesn't
>>>>>>>>> provide the MPI_F08 binding names, then it must call the C MPI
>>>>>>>>> implementation in a thin Fortran wrapper so that the tools
>>>>>>>>> implementers can intercept the call in C. An MPI implementation
>>>>>>>>> shall not provide/call both the MPI_F08 binding name and the C
>>>>>>>>> name.
>>>>>>>>>
>>>>>>>>> Option 2 allows some MPI vendors to implement MPI functions
>>>>>>>>> entirely within Fortran (using the MPI_f08 binding names),
>>>>>>>>> if this is more efficient.
>>>>>>>>>
>>>>>>>>> Option 3 allows some MPI implementations to place Fortran
>>>>>>>>> wrappers
>>>>>>>>> inside modules where they may be inlined by the compiler for
>>>>>>>>> efficiency.
>>>>>>>>>
>>>>>>>>> Craig Rasmussen
>>>>>>>>> CAS Scientific Programmer
>>>>>>>>> rasmus at cas.uoregon.edu <mailto:rasmus at cas.uoregon.edu>
>>>>>>
>>>>>
>>>>> --
>>>>> Bill Long longb at cray.com <mailto:longb at cray.com>
>>>>> Fortran Technical Support & voice: 651-605-9024
>>>>> Bioinformatics Software Development fax: 651-605-9142
>>>>> Cray Inc./Cray Plaza, Suite 210/380 Jackson St./St. Paul, MN 55101
>>>>
>>>> --
>>>> Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner at hlrs.de
>>>> <mailto:rabenseifner at hlrs.de>
>>>> High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530
>>>> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832
>>>> Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner
>>>> <http://www.hlrs.de/people/rabenseifner>
>>>> Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)
>>>
>>> --
>>> Dr. Rolf Rabenseifner . . . . . . . . . .. email rabenseifner at hlrs.de
>>> <mailto:rabenseifner at hlrs.de>
>>> High Performance Computing Center (HLRS) . phone ++49(0)711/685-65530
>>> University of Stuttgart . . . . . . . . .. fax ++49(0)711 / 685-65832
>>> Head of Dpmt Parallel Computing . . . www.hlrs.de/people/rabenseifner
>>> <http://www.hlrs.de/people/rabenseifner>
>>> Nobelstr. 19, D-70550 Stuttgart, Germany . . . . (Office: Room 1.307)
>>
>
> --
> Bill Long                                           longb at cray.com
> Fortran Technical Support    &                 voice: 651-605-9024
> Bioinformatics Software Development            fax:   651-605-9142
> Cray Inc./Cray Plaza, Suite 210/380 Jackson St./St. Paul, MN 55101
>
>
>
>
>   Click https://www.mailcontrol.com/sr/PSMHhYrWn2nGX2PQPOmvUmmLQnadxIEhPqNHPbHRv8fTzNoj9ICXcB470Jlue7Tvj5KdIm2MzNwfmF8UmgH1OQ==  to report this email as spam.
>

-- 
Bill Long                                           longb at cray.com
Fortran Technical Support    &                 voice: 651-605-9024
Bioinformatics Software Development            fax:   651-605-9142
Cray Inc./Cray Plaza, Suite 210/380 Jackson St./St. Paul, MN 55101