# The protocol file generated by protogen.py ver. 1.0 on 2013-06-27 12:54
# AUTHOR: Magda S aka Magic Magg magg dot gatech at gmail dot com
# DATE: 2013-06-27


DESCRIPTION
===========
This is a test case for Flexpath. The idea of the test: The writer writes a
1D array of NX double elements. The array is written as ADIOS global array
so each process writes in a global space, but in its own place.

The writer writes the 1D array and the reader reads the 1D array of doubles.

There might be as many writers as you wish, and there might be as many readers
as you wish. However, each rank reads its own rank. The reader knows how many 
writers were there so if its rank is higher then it quits.

The test can work with two modes (use '-t' option with the appropriate mode):

1. MPI/ADIOS_READ_METHOD_BP
2. FLEXPATH/ADIOS_READ_METHOD_FLEXPATH

BUILD
=======
# you need to set the environment variables as Makefile uses those locations 
# to locate libraries and headers

export ADIOS_ROOT=/rock/opt/adios/git-dbg
export MXML_ROOT=/rock/opt/mxml/2.7
export MPI_ROOT=/rock/opt/openmpi/1.6.3
export EVPATH_ROOT=/rock/opt/evpath

# in certain cases you might need the lustre directory (e.g., on kraken)
export LUSTRE_ROOT=/opt/cray/lustre-cray_ss_s/default


# build 
$ make -f Makefile.generic

# should remove all unnecessary exec files 
$ make -f Makefile.generic clean

RUN
=====
# you should remove text file remnants from Flexpath _read_ready.txt, _info_writer.txt
$ make -f Makefile.generic clean_test

$ mpirun -np 2 ./writer -t flx
$ mpirun -np 2 ./reader -t flx

See Makefile for other options or add '-h' option to the reader or writer

NOTES
=======
2013-09-04 - readme might be outdated

2013-07-17 - tested on my local laptop (MPI and FLEXPATH)


OUTDATED (2013-07-08)
======================
I will update later. 

RUNNING ON KRAKEN
=================
This runs one reader per node and one writer per node.

aprun -n 1 -N 1 ./arrays_read &
aprun -n 1 -N 1 ./arrays_write

You should be able to run the example with as many readers and writers as you wish.

Example PBS script
------------------
#!/bin/bash
#PBS -l walltime=00:05:00,size=24
#PBS -A UT-TENN0033

date

echo "nodefile="
cat $PBS_NODEFILE
echo "=end nodefile"

# make sure you have all modules loaded
module use ~smagg/opt/modulefiles
module load mag-mxml-2.7/kraken-gnu
module list

cd $PBS_O_WORKDIR

# on Kraken the path needs to be specified precisely 
# ./writer might cause a strange error
aprun -n 24 -N 12 /lustre/scratch/smagg/writer &
sleep 20
aprun -n 24 -N 12 /lustre/scratch/smagg/reader 

date

# eof


TROUBLESHOOTING
===============

There might be text files left; they should be removed for the next run.

# EOF
