#
# Copyright (C) 2017, Northwestern University and Argonne National Laboratory
# See COPYRIGHT notice in top-level directory.
#
# $Id$

===============================================================================
    Cray XC40
===============================================================================

* Building PnetCDF on the Cray XC40 (tested on Cori @ NERSC and Theta @ ALCF)
  http://www.nersc.gov/systems/cori/
  https://www.alcf.anl.gov/theta

    ./configure --prefix=/path/to/install \
                cross_compiling="yes" \
                MPICC=cc MPICXX=CC MPIF77=ftn MPIF90=ftn \
                CFLAGS="" CXXFLAGS="" FFLAGS="" FCFLAGS=""

  On Cori, when using Intel-based compilers, NERSC recommends the default level
  of optimization, i.e. no optimization arguments to the compiler. See section
  "Recommended flags" for Intel compilers at
  http://www.nersc.gov/users/computational-systems/cori/programming/compiling-codes-on-cori/

  * On April 29, 2019, The default gcc module version 7.0.3 is loaded on Cori,
    which causes `make` process to fail with an error message of
    /usr/bin/ld: attempted static link of dynamic object `/opt/gcc/7.3.0/snos/lib/../lib64/libstdc++.so'
    This problem can be resolved by unloading the new gcc module before running
    the configure command, i.e.
        module unload gcc/7.3.0

* Note that the default setting of PnetCDF is to build static libraries only.
  To also build shared libraries, add "--enable-shared LDFLAGS=-dynamic" to the
  configure command.

    ./configure --prefix=/path/to/install \
                cross_compiling="yes" \
                MPICC=cc MPICXX=CC MPIF77=ftn MPIF90=ftn \
                CFLAGS="" CXXFLAGS="" FFLAGS="" FCFLAGS="" \
                --enable-shared LDFLAGS=-dynamic

  See further information about "Shared and Dynamic Libraries" from the
  following NERSC web site:
  http://www.nersc.gov/users/computational-systems/edison/running-jobs/shared-and-dynamic-libraries/


* Note on compiling utility programs ncoffsets and cdfdiff.
  Error messages as shown below may appear, when the environment is set to use
  Intel-based compilers (i.e. PrgEnv-intel, default on Cray XC40).

  /usr/bin/gcc -o ncoffsets src/utils/ncoffsets/ncoffsets.c
  In file included from /usr/include/inttypes.h:27:0,
                   from src/utils/ncoffsets/ncoffsets.c:16:
  /theta-archive/intel/compilers_and_libraries_2019.5.281/linux/compiler/include/stdint.h:43:54: error: missing binary operator before token "("
       defined(__has_include_next) && __has_include_next(<stdint.h>)
                                                        ^
  There are two ways to resolve this problem.
  1. Add "SEQ_CC=icc" to your make command line, i.e.
     make SEQ_CC=icc
  2. Unset environment variable CPATH whose value conflicts with the gcc
     include directories, i.e. run command:
     make SEQ_CC=gcc CPATH=

* Note on using compile command-line option "-fast".
  According to the NERSC URL below, the flag "-no-ipo" must be used together
  with flag "-fast" to build an application.
  http://www.nersc.gov/users/software/compilers/intel-fortran-c-and-c/

  In addition, option "-fast" implies "-static". Thus please use "-O2" flags
  when building PnetCDF shared libraries.

* Note on running "make check" or "make ptest".
  Cori is a cross-compile environment. Jobs must be run through a batch
  scheduler. At the time of this writing, the schedule is Slurm. Below
  configure command line adds 3 variables in preparation for job submission.

    ./configure --prefix=/path/to/install \
                cross_compiling="yes" \
                MPICC=cc MPICXX=CC MPIF77=ftn MPIF90=ftn \
                CFLAGS="" CXXFLAGS="" FFLAGS="" FCFLAGS="" \
                TESTMPIRUN="srun -n NP" TESTSEQRUN="srun -n 1" \
                TESTOUTDIR="$SCRATCH"

  A sample batch script shown below allocates 4 compute nodes. "make ptest"
  requires at least 4 MPI processes. The number can be changed to use less
  number of nodes. Variable TESTOUTDIR should be set to the file directory
  for storing the test output files.
  #!/bin/bash -l
  #SBATCH -p debug
  #SBATCH -N 4
  #SBATCH -t 00:20:00
  #SBATCH -L SCRATCH
  #SBATCH -C haswell
  make ptest

* Note on using GNU compilers
  To swap to the GNU programming environment from default, run command below
  first:
        module swap PrgEnv-intel PrgEnv-gnu

  We found that libtool does not work well with Cray's GNU-based compiler
  wrappers, e.g. cc, CC, and ftn when building static-only library. If you
  must use GNU compilers, please add "LDFLAGS=-all-static" to the make
  command line, i.e.

    ./configure --prefix=/path/to/install \
                cross_compiling="yes" \
                MPICC=cc MPICXX=CC MPIF77=ftn MPIF90=ftn \
                CFLAGS="" CXXFLAGS="" FFLAGS="" FCFLAGS=""

  Then add the following LDFLAGS to the make command line, i.e.

                make       LDFLAGS=-all-static
                make tests LDFLAGS=-all-static
                make check LDFLAGS=-all-static

  Without adding LDFLAGS, you may encounter the following error messages.
    libtool: warning: '/opt/gcc/6.3.0/snos/lib/gcc/x86_64-suse-linux/6.3.0/../../../../lib64/libgfortran.la' seems to be moved
    libtool: warning: '/opt/gcc/6.3.0/snos/lib/gcc/x86_64-suse-linux/6.3.0/../../../../lib64/libquadmath.la' seems to be moved
    libtool: warning: '/opt/gcc/6.3.0/snos/lib/gcc/x86_64-suse-linux/6.3.0/../../../../lib64/libstdc++.la' seems to be moved
    /usr/bin/ld: attempted static link of dynamic object `/opt/gcc/6.3.0/snos/lib/../lib64/libgfortran.so'
    collect2: error: ld returned 1 exit status

  * Building shared libraries:

  For 1.9.1 and later, follow the same instructions for the Intel-based
  compilers above.

  For 1.9.0, use command below.
    ./configure --prefix=/path/to/install \
                cross_compiling="yes" \
                MPICC=cc MPICXX=CC MPIF77=ftn MPIF90=ftn \
                CFLAGS="" CXXFLAGS="" FFLAGS="" FCFLAGS="" \
                --enable-shared LDFLAGS=-dynamic

  and then run

                make

* Note on using Cray compilers (i.e. craycc, crayCC, crayftn)
  libtool fails to work with Cray compiler based wrappers. See discussion in
  https://trac.mpich.org/projects/mpich/ticket/1909 and
  http://lists.gnu.org/archive/html/libtool/2015-04/msg00003.html
  Before libtool fixes this, PnetCDF 1.9.0 does not support Cray-based
  compilers.

===============================================================================
    Cray XC30
===============================================================================

Building PnetCDF on the Cray XC30 (tested on Edison @ NERSC)
http://www.nersc.gov/systems/edison-cray-xc30/

The building recipe for Edison is the same as Cori.

===============================================================================
    Cray XE6
===============================================================================

Building PnetCDF on the Cray XE6 (tested on Hopper @ NERSC)
http://www.nersc.gov/systems/hopper-cray-xe6/

./configure --prefix=/path/to/install \
            --with-mpi=/path/to/mpi/implementation \
            CFLAGS=-fast CXXFLAGS=-fast FFLAGS=-fast FCFLAGS=-fast


The configure command above works for PGI, GNU, and Intel
compilers, i.e. when one of the module load commands below is used:

    module load PrgEnv-pgi
    module load PrgEnv-gnu
    module load PrgEnv-intel


For Pathscale compilers, i.e.
    module load PrgEnv-pathscale
    use command below:
    ./configure --prefix=/path/to/install \
                --with-mpi=/path/to/mpi/implementation \
                CFLAGS=-Ofast CXXFLAGS=-Ofast FFLAGS=-Ofast FCFLAGS=-Ofast


For Cray compilers, i.e.
    module load PrgEnv-cray
    use command below:
    ./configure --prefix=/path/to/install \
                --with-mpi=/path/to/mpi/implementation \
                CFLAGS=-O2 CXXFLAGS=-O2 FFLAGS=-O2 FCFLAGS="-O2 -emf" \
                LDFLAGS=-Wl,-z,muldefs

    Check crayftn man page for using option "-emf" in FCFLAGS:
    to creates .mod files to hold module and allows the creation
    of lower-case module .mod file names.

    Option "-Wl,-z,muldefs" in LDFLAGS is to get around the
    error of multiple definitions of `tc_version', etc.


===============================================================================
    Cray X1
===============================================================================

2 May 2005

I (Rob Latham) performed the following steps to get PnetCDF to build on the
Cray X1 at Oak Ridge (phoenix.ccs.ornl.gov).  Note that out-of-tree (or VPATH)
builds do not work for the Fortran interface as of 1.0.0-pre2, but we will try
to address this issue in a future release.

prompt:$ module load mpt
prompt:$ export CC=cc
prompt:$ export FC=ftn
prompt:$ export MPIF77=$FC
prompt:$ export MPICC=$CC
prompt:$ export FFLAGS="-eh"
prompt:$ ./configure --prefix=/path/to/install
# note: configure takes a fairly long time.
prompt:$ make

The "nc_test" test will exhaust the available MPI datatypes on the X1.  Your
application might see this error:

   MPI has run out of internal datatype entries.
   Please set the environment variable MPI_TYPE_MAX for additional space.
   The current value of MPI_TYPE_MAX is 2098

I did as asked and nc_test completed with MPI_TYPE_MAX set to 4096

If you run on the login node, expect to see a lot of these messages:

  Process [nc_test] 89345 generated trap, but has signal 8 held or ignored
        epc 0x1219bb4 ra 0x1219b94 badvaddr 0x40004f0004000020

The messages don't *appear* to impact the program results, and additionally do
not show up if you submit the job to PBS.

Fortran codes should use '-eh' so that the Cray ftn compiler will use 1 byte
for int*1 and 2 bytes for int*2.  Otherwise, our Fortran bindings will pass
incorrect values to the C routines.

