# $Id: README.IBM 1741 2014-08-23 16:13:13Z wkliao $

1. Building PnetCDF on BGQ
2. Building PnetCDF on BGP
3. Building PnetCDF on BGL
4. Building PnetCDF on UCAR BlueFire
5. Building PnetCDF on IBM SP

-------------------------------------------------------------------------------
1. Building PnetCDF on BGQ
-------------------------------------------------------------------------------

Building for BGQ is not so different from BGP or BGL: front end node is still a
cross compile host for the back end.    (as of this writing, the "log in io
nodes", a.k.a lion nodes, are not on line for the Mira BGQ machine)

Be sure to run configure with the --build and --host flags to put it in "cross
compile mode".  This will make configure use compile-only tests, instead of the
usual compile-and-run tests (running tests on the BGP login node won't work as
the compute nodes are totally different.  This will change if/when lion
nodes are available).

./configure --host powerpc64-bgq-linux --build ppc64-redhat-linux \
            MPICC=mpicc MPICXX=mpicxx MPIF77=mpif77 MPIF90=mpif90

Note if running make is slow, try parallel make, e.g.

    make -j8

It's possible to build PnetCDF with the IBM xl compilers:

./configure --host powerpc64-bgq-linux --build ppc64-redhat-linux \
            --with-mpi=/bgsys/drivers/ppcfloor/comm/xl \
            MPICC=mpixlc MPICXX=mpixlcxx MPIF77=mpixlf77 MPIF90=mpixlf90 \
            FFLAGS="-qsuppress=cmpmsg" FCFLAGS="-qsuppress=cmpmsg"

Fortran flag "-qsuppress=cmpmsg" can suppresses XLF compile informational
messages that report compilation progress and a successful completion.


Another possibility that uses thread-safe XL compilers is:

./configure --host powerpc64-bgq-linux --build ppc64-redhat-linux \
            --with-mpi=/bgsys/drivers/ppcfloor/comm/xl \
            MPICC=mpixlc_r    MPICXX=mpixlcxx_r \
            MPIF77=mpixlf77_r MPIF90=mpixlf90_r \
            CFLAGS=-O2 CXXFLAGS=-O2 FFLAGS=-O2 FCFLAGS=-O2 \
            --prefix=/path/to/install


As far as we know, there are no issues with PnetCDF on BGQ, but should you
find any, email parallel-netcdf@mcs.anl.gov

----
Note on compiling Fortran 90 programs on Intrepid @ ANL
----
https://www.alcf.anl.gov/user-guides/bgp-faqs-compiling-and-linking#how-do-i-use-xlf-and-"use-mpi"

If you attempt to do a "use mpi" from XL Fortran, and build with the default
compiler wrappers, you will encounter this error:

  XL Fortran90 error: Module symbol file for module mpi is in a format not
recognized by this compiler

The cause of the error is that the mpi.mod referenced in the "default" software
stack (/bgsys/drivers/ppcfloor/comm/default) is built with GNU Fortran.

A work-around to the problem using mpif.h instead of "use mpi.mod", e.g.:

program test
include "mpif.h"

... your MPI stuff ...

end program test

The proper solution is to use an alternate software stack with MPI built using
the XL compilers instead of the GNU compilers, thus containing a Fortran mpi
module (mpi.mod) generated by XLF. This software stack was introduced in
V1R4M1. You may build with it using the wrappers in comm/xl:

 /bgsys/drivers/ppcfloor/comm/xl/bin/mpixlf90


----
* Note on running PnetCDF testing programs on Mira/Cetus @ ALCF, ANL
----
Mira and Cetus are IBM BlueGene Q, a cross-compile system. Hence, one cannot
run PnetCDF test programs on the login nodes. Test programs must run on the
compute nodes through submitting a batch job. An example batch script file is
provided in cobalt.script. The submission command for allocating 8 node with 90
minutes is given below. Note the testing requires at least 8 MPI processes and
in this example the command requests 8 nodes and the script runs 1 MPI process
per node.

qsub -A YOUR_ACCOUNT -n 8 -t 30 --mode script ./cobalt.script

Running "make testing" can take a while (more than one hour on Mira/Cetus).
This is because the testing performs many small read/write requests and under
BlueGene's I/O forwarding architecture, local file-system caching is not
available on the compute nodes.

Running "make ptest" will take approximately 10 minutes.

-------------------------------------------------------------------------------
2. Building PnetCDF on BGP
-------------------------------------------------------------------------------

Building for BGP is not so different from BGL: front end node is still a cross
compile host for the back end.  

Be sure to run configure with the --build and --host flags to put it in "cross
compile mode".  This will make configure use compile-only tests, instead of the
usual compile-and-run tests (running tests on the bgp login node won't work as
the compute nodes are totally different).

There is one run-time check for MPI-IO support of resized types. Unfortunately
we have to test for this with a runtime test, but you can set the environment
variable "ac_cv_MPI_TYPE_RESIZED_WORKS" . 

./configure --host powerpc-bgp-linux --build powerpc64-suse-linux \
            --with-mpi=/bgsys/drivers/ppcfloor/comm

It's possible to build PnetCDF with the IBM xl compilers:

./configure --host powerpc-bgp-linux --build powerpc64-suse-linux \
            --with-mpi=/bgsys/drivers/ppcfloor/comm/xl \
            MPICC=mpixlc MPICXX=mpixlcxx MPIF77=mpixlf77 MPIF90=mpixlf90 \
            FFLAGS="-qsuppress=cmpmsg" FCFLAGS="-qsuppress=cmpmsg"

Fortran flag "-qsuppress=cmpmsg" can suppresses XLF compile informational
messages that report compilation progress and a successful completion.

As far as we know, there are no issues with PnetCDF on BGP, but should you
find any, email parallel-netcdf@mcs.anl.gov

----
Note on compiling Fortran 90 programs on Intrepid @ ANL
----
https://www.alcf.anl.gov/user-guides/bgp-faqs-compiling-and-linking#how-do-i-use-xlf-and-"use-mpi"

If you attempt to do a "use mpi" from XL Fortran, and build with the default
compiler wrappers, you will encounter this error:

  XL Fortran90 error: Module symbol file for module mpi is in a format not
recognized by this compiler

The cause of the error is that the mpi.mod referenced in the "default" software
stack (/bgsys/drivers/ppcfloor/comm/default) is built with GNU Fortran.

A work-around to the problem using mpif.h instead of "use mpi.mod", e.g.:

program test
include "mpif.h"

... your MPI stuff ...

end program test

The proper solution is to use an alternate software stack with MPI built using
the XL compilers instead of the GNU compilers, thus containing a Fortran mpi
module (mpi.mod) generated by XLF. This software stack was introduced in
V1R4M1. You may build with it using the wrappers in comm/xl:

 /bgsys/drivers/ppcfloor/comm/xl/bin/mpixlf90


-------------------------------------------------------------------------------
3. Building PnetCDF on BGL
-------------------------------------------------------------------------------

Be sure to run configure with the --build and --host flags to put it in "cross
compile mode".  This will make configure use compile-only tests, instead of the
usual compile-and-run tests (running tests on the bgl login node won't work as
the compute nodes are totally different).

configure --build powerpc32-unknown-gnu --host powerpc-suse-linux  \
          --with-mpi=/bgl/BlueLight/ppcfloor/bglsys/

It's possible to build PnetCDF with the IBM xl compilers, but you have to set
quite a few environment variables

export CC=blrts_xlc
export MPICC=blrts_xlc
export CXX=blrts_xlC
export MPICXX=blrts_xlC
export FC=blrts_xlf
export F77=blrts_xlf
export MPIF77=blrts_xlf

export CFLAGS="-I/bgl/BlueLight/ppcfloor/bglsys/include"
export LIBS="-L/bgl/BlueLight/ppcfloor/bglsys/lib -lmpich.rts -lmsglayer.rts -ldevices.rts -lrts.rts -ldevices.rts -lrts.rts"

configure --build powerpc32-unknown-gnu --host powerpc-suse-linux

Several early versions of IBM's MPI-IO implementation would segfault under
certain workloads.  If you are running driver version "V1R3M0_240_2006-060623"
or newer, the segfault issue should be resolved.  If you are running an older
driver, read on:

When built against some older BlueGene drivers,  nc_test does not run
completely without setting a special environment variable, hitting a seg fault
deep inside ROMIO.  We first noticed this in IBM's "Driver 202" MPI and also in
"V1R1M1_253_2005-051003" and "V1R2M1_020_2006-060110"  We have told IBM
developers about the problem.  Code that makes use of the ncmpi_get_vara_*_all
or ncmpi_put_vara_*_all routines will likely trigger a seg fault.  IBM has
provided a workaround:  if your code seg-faults, try setting the
"BGLMPIO_TUNEBLOCKING" environment variable to 0.  With this environment
variable set, nc_tests runs to completion and passes.  For one real-world
example, the FLASH-IO benchmark with 8 processors sees a 5% performance hit
when writing out plotfiles.  That increases to 22% with 16 processors.   Again, upgrading to the latest BlueGene drivers should fix this issue.



-------------------------------------------------------------------------------
4. Building PnetCDF on UCAR BlueFire
-------------------------------------------------------------------------------

- the nc_test and nf_test tests use M4 to generate code.  AIX-m4 won't
  generate correct code, so use gnu M4.

- We had a report of a build failure when building the 'ncmpigen' utility.  The
  system 'bison' tool (bison-1.875) did not produce output files, but when the
  user installed a new version of gnu bison, everything worked fine.   

- the PnetCDF code is slowly taking on more and more c99 features.  

On BlueFire, the commands I used look like this:

$ module add m4-1.4.14

$ configure --prefix=/path/to/install \
        CFLAGS=-qlanglvl=stdc99 \
        CC=xlc FC=xlf F77=xlf F90=xlf90 \
        MPICC=mpcc_r MPIF77=mpxlf_r MPIF90=mpxlf90_r CXX=xlC MPICXX=mpCC_r


-------------------------------------------------------------------------------
5. Building PnetCDF on IBM SP
-------------------------------------------------------------------------------
# John Tannahill <tannahill1@llnl.gov> reported success building
# parallel-netcdf on the 'seaborg' cluster (an IBM-SP at NERSC) and Tyce
# Mclarty <mclarty3@llnl.gov> reported success on LLNL's 'frost' cluster by
# setting these environment variables:

   setenv MPICC  mpcc_r
   setenv MPIF77 mpxlf_r
   setenv F77    xlf
   setenv FC     xlf
   setenv CC     xlc
   setenv CXX    xlC

# after setting these variables, configure/make/make install should "just work"


# We also successfully tested 64-bit mode on 'DataStar' (an IBM-SP at SDSC)
# with environment variables:

   setenv OBJECT_MODE 64
   setenv MPICC  mpcc_r
   setenv MPIF77 mpxlf_r
   setenv F77    xlf
   setenv FC     xlf
   setenv CC     xlc
   setenv CXX    xlC
   setenv CFLAGS -q64
   setenv FFLAGS -q64
   setenv F90FLAGS -q64
   setenv CXXFLAGS -q64
