Mars : Version 9.

06

Mars_Agrif2_V9.06
“MARS ENVIRONMENT’s manual”

Valérie GARNIER, Sébastien THEETTEN

lfremer

(easier to consult this documentation activating slide (diaporama)

in order to show links )

1
 Mars : Version 9.06

MARS environment
Sommaire (click on each chapter to go directly in)
 Why this MARS environment
 What is the MARS environment
 Setting up MARS environment
 MARS requirements
 List of useful commands
 How to implement a new configuration
 How to update a configuration
 MARS environment : SUMMARY
lfremer

 Choice of CPP Keys

 How to prepare parametrization before running
 How to run a job
 Run a job : SUMMARY
 How to decompose domain for MPI purpose
 How to use a test case
 How to set a new test case
 How to modify MARS - coding rules 2
 Mars : Version 9.06

Why this MARS environment (1)

Frequently observed problems

 Difficulty in following MARS evolutions led to different MARS codes !

 Numerical developments and configurations must be performed following a few
usual rules, otherwise they get lost

MARS is used through a working environment based on C-shell scripts

Goals
lfremer

 Management of MARS versions

 Easier tracking of configuration changes
 Easier keeping up to date of configurations
 Direct visualization of user’s modifications
 Sharing of configurations
 Saving of disk space

3
 Mars : Version 9.06

Why this MARS environment (2)

Super-users
 management and distribution (V. Garnier)
 developments / evolutions (F. Dumas, V. Garnier, B. Thouvenin)
 teaching and user’s help (S. Theetten)

MARS code
The complete MARS code (sources, namelist, environment, documentation and netcdf
libraries) is located in a directory defined by the environment variable $HOMEMARS
lfremer

Users
 Run the code for personal configuration or use test cases
 Improve the code (numerical developments…)
 Report encountered problems or needs
 Follow code improvements, keep configurations up to date

4
Retour au Sommaire Mars : Version 9.06

Reference MARS environment (3)

Reference

MARS environment is originally derived from utility named

Drakkar Config Manager and developped by J. M. Molines and
S. Theetten.

Similar environments are used with MARS code, NEMO code

lfremer

(DRAKKAR projects and followings) and SYMPHONIE code

(soon).

5
 Mars : Version 9.06

What is the MARS environment (1)

3 directories (user) 3 scripts (user)

MARS_CONFIG : mkconfdir :
compilation / code modification create the directories of a configuration

COMPILE_MARS : makefile :
compilation directory, manage the configuration
(choice of rank, test case, previous configurations),
lfremer

where the user must not go

(otherwise user’s changes might be lost)
compile the code by calling :
RUN_MARS :
run simulation, input and ouput files Makefile.linux
(or Makefile.caparmor)

6
Retour au Sommaire Mars : Version 9.06

What is the MARS environment (2)

The originial code (reference code) is placed in the

Reference directory $HOMEMARS/.. (super user)

which contains :
 MARS sources,
 Scripts of MARS environment,
 Makefile for the compilation,
 Input files as namelists and *.dat (river.dat, outflow.dat…)
 Documentation : manuals, faq
 Module example (fortran)
lfremer

 Libraries (netcdf, io_netcdf)

 Code rules
$HOMEMARS/../

$HOMEMARS/

LIB EXAMPLES DOC TOOLS Mars_Agrif2

Netcdf, input files, Manual, bathy, Reference
io_netcdf.. Makefile.. faq connect, … Sources code

7
 Mars : Version 9.06

MARS environment : setting up (1)

1/ Inside your .cshrc,
add following environment variables and paths :
#------------------------------ Directory where is the reference code
#  MARS DEVELOPMENT ENVIRONMENT
#-----------------------------
setenv HOMEMARS /export/home11/mars/CODE_MARS/CODE_MARS_V9/V9.06/Mars_Agrif2
setenv UDIR /export/home/${USER}/MARS/MARS_CONFIG
setenv CDIR /export/home/${USER}/MARS/COMPILE_MARS
setenv RDIR /export/home/${USER}/MARS/RUN_MARS
Site gforge useful to update
setenv HTTPSVNROOT https://forge.ifremer.fr/svn/mars3d your MARS version
lfremer

setenv extranet_login yours Set your extranet_login name

setenv PATH ${PATH}:$HOMEMARS/../TOOLS/MARSENV

alias mkconfdir  "$HOMEMARS/../TOOLS/MARSENV/mkconfdir_caparmor“
alias mkconfdir_bisc  "$HOMEMARS/../TOOLS/MARSENV/mkconfdir_caparmor_bisc“

Allow you to use commands

useful to the setting up MARS
2/ type : source .cshrc
8
 Mars : Version 9.06

MARS environment : setting up (2) (bash use)

If bash use, add following environmental variables and paths :

1/ create a new file (env_MARS_V9.06 for instance) :

#------------------------------
#  MARS DEVELOPMENT ENVIRONMENT
#------------------------------

export HOMEMARS = /export/home11/mars/CODE_MARS/CODE_MARS_V9/V9.06/Mars_Agrif2

export UDIR=/export/home/${USER}/MARS/MARS_CONFIG
export CDIR=/export/home/${USER}/MARS/COMPILE_MARS
export RDIR=/export/home/${USER}/MARS/RUN_MARS
lfremer

alias mkconfdir=‘$HOMEMARS/../TOOLS/MARSENV/mkconfdir_linux’
alias getfile=‘$HOMEMARS/../TOOLS/MARSENV/getfile’
alias update=‘$HOMEMARS/../TOOLS/MARSENV/update’

2/ type : source env_MARS_V9.06

9
 Mars : Version 9.06

MARS environment : setting up (3)

3/ create the 3 directories necessary for MARS environment :

# UDIR :
type : mkdir /export/home/${USER}/MARS/MARS_CONFIG

# CDIR :
type : mkdir /export/home/${USER}/MARS/COMPILE_MARS
lfremer

# RDIR :
type : mkdir /export/home/${USER}/MARS/RUN_MARS

# CODE_MARS (if the reference MARS code is locally saved – not for IFREMER users)
type : mkdir /export/home/${USER}/MARS/CODE_MARS

10
 Mars : Version 9.06

MARS environment : setting up (4)

Important remark
 meant for research institutes, private societies or labtop use
 IFREMER users have no need to duplicate the code under their account

Step local setting up (not for caparmor users)

 Get the code from mars3d projet under forge with the following command :
svn –username $extranet_login checkout $HTTPSVNROOT/tags/V9.06
 cd V9.06/EXAMPLES
lfremer

 vi Makefile.linux : modify library paths of netcdf, io_netcdf.

 (if make use, replace all ‘gmake’ by ‘make’ in makefile_env_linux or link
gmake to make in your linux environment)

Rq 1 : changes inside Makefile.linux (and makefile_env_linux) are the only

ones allowed in CODE_MARS directory
Rq 2 : no help from PHYSED if the source code has been modified

11
 Mars : Version 9.06

MARS environment : setting up (5)

Important remarks
 MARS requires fortran compiler, netcdf4 library
(netcdf4 requires hdf5 and zlib librairies)
 MARS results are easily visualized with ferret (free) or
matlab…
 Library NCO is useful for files management
lfremer

For users working on caparmor :

Return to sommaire

Informations on Fortran, netcdf4, netcdf, click here

12
 Mars : Version 9.06

MARS environment : local setting up (6)

Fortran compiler setting up

1. Download free Intel Fortran compiler from
http://www.intel.com/cd/software/products/asmo-na/eng/index.h
tm
(a - register at « Free Non Commercial
Download » to receive a number authorization by mail; b –
choose ‘Intel Fortran Compiler Profressional Edition for linux’)
2. Installation from file /export/home/l_fc_p_10.X.XXX.tar.gz
lfremer

3. If library stcd++ is missing : yum install libstdc++.so.5

4. Set compiler environment : source
/opt/intel/fce/10.X.XXX/bin/ifortvars.csh

13
 Mars : Version 9.06

MARS environment : local setting up (7)

Netcdf4 (1/3)
We assume the netcdf4 library is installed under /export/home/logiciel/netcdf4

zlib library

1. Create installation directory

 cd export/home/logiciel/netcdf4 ; mkdir install ; cd install
2. Download of zlib-1.2.5.tar from http://zlib.net/ (get the sources and not the binaries)
3. bunzip2 zlib-1.2.5.tar.bz2 ; tar xvf zlib-1.2.5.tar
4. Installation
lfremer

sequential or OMP purpose MPI purpose

Module purge Module purge
module load intel-comp/11.1.073 (path for intel module load intel-comp/11.1.073
fortran and c++) module load intel-mpi/4.0.0.028 (path mpi)
export export NC4DIR=/export/home/logiciels/netcdf4/lib/mpi
NC4DIR=/export/home/logiciels/netcdf4/lib/seq
cd /export/home/logiciels/netcdf4/install/zlib-1.2.5 cd /export/home/logiciels/netcdf4/install/zlib-1.2.5
./configure --prefix=$NC4DIR ./configure --prefix=$NC4DIR
make check install make check install

14
 Mars : Version 9.06

MARS environment : local setting up (8)

Netcdf4 (2/3)
We assume the netcdf4 library is installed under /export/home/logiciel/netcdf4

hdf5 library

1. Go to installation directory
 cd export/home/logiciel/netcdf4/install
2. Download http://www.hdfgroup.org/ftp/HDF5/prev-releases/hdf5-1.8.6/src/hdf5-
1.8.6.tar.gz (get the sources and not the binaries)
3. gunzip file ; tar xvf file
4. Installation
lfremer

sequential or OMP purpose MPI purpose

Module purge Module purge
module load gfortran (path for gfortran and c++) module load gfortran
module load gfortran-mpi (path mpi)
export NC4DIR=/export/home/logiciels/netcdf4/lib/seq export NC4DIR=/export/home/logiciels/netcdf4/lib/mpi
cd /export/home/logiciels/netcdf4/install/hdf5-1.8.6 cd /export/home/logiciels/netcdf4/install/hdf5-1.8.6
./configure --prefix=$NC4DIR --with-zlib=$NC4DIR -- ./configure --prefix=$NC4DIR --with-zlib=$NC4DIR --
disable-shared CC=gcc CPP='gcc -E' disable-shared --enable-parallel CC=mpicc CPP='mpicc -E'
make make
make install make install
15
 Mars : Version 9.06

MARS environment : local setting up (9)

Netcdf4 (3/3)
We assume the netcdf4 library is installed under /export/home/logiciel/netcdf4

netcdf4 library
1. Go to the installation directory (cd export/home/logiciel/netcdf4/install)
2. Download of netcdf-4.1.2.tar.gz from http://www.unidata.ucar.edu/downloads/netcdf/netcdf-
4_1_2/index.jsp (get the sources and not the binaries)
3. gunzip file ; tar xvf file
4. Installation

sequential or OMP purpose MPI purpose

Module purge Module purge
module load gfortran (path for gfortran and c++) module load gfortran
lfremer

module load gfortran-mpi (path mpi)

export NC4DIR=/export/home/logiciels/netcdf4/lib/seq export NC4DIR=/export/home/logiciels/netcdf4/lib/mpi
cd /export/home/logiciels/netcdf4/install/netcdf-4.1.2 cd /export/home/logiciels/netcdf4/install/netcdf-4.1.2
export LDFLAGS="-lm"  (access to a mathematics
library)
./configure --prefix=$NC4DIR --with-hdf5=$NC4DIR -- ./configure --prefix=$NC4DIR --with-hdf5=$NC4DIR --with-
with-zlib=$NC4DIR --enable-ncgen4 --enable-netcdf-4 zlib=$NC4DIR --enable-ncgen4 --enable-netcdf-4 --
--disable-shared --disable-cxx --disable-dap disable-shared --disable-cxx --disable-dap CPPFLAGS='-
FC=gfortran CC=gcc CPP='gcc -E' DNDEBUG -DpgiFortran' FC=mpif90 CC=mpicc
CPP='mpicc -E'
make make
make check install make check install
16
 Mars : Version 9.06

MARS environment : local setting up (10)

IO_NETCDF library (to be compiled only if used by software other than MARS)

Libionc4 is an interface between NetCDF4 and MARS (or other softwares). This library is developed by
IFREMER/PHYSED and supplied with MARS under $HOMEMARS/../LIB/libionc4. It is a module of
MARS software.

1. cd $HOMEMARS/../LIB/libionc4/src
 ln -s $HOMEMARS/INC/comionc4.F90
 ln -s $HOMEMARS/PHYS/OUT/ionc4.F90
2. Prepare directories
 cd ..
 ./install.csh (2 directories are created intel-seq and intel-mpi; the will store the library and the include files)
3. edit Makefile and modify
lfremer

sequential or OMP purpose MPI purpose

FC = ifort FC = mpiifort
CPPFLAGS = CPPFLAGS = -Dkey_MPI_2D
NETCDF4 = /home1/caparmor/rramel/netcdf4.1.2 NETCDF4 = /home1/caparmor/rramel/netcdf4.1.2_par
OUTDIR = ./intel-seq OUTDIR = ./intel-mpi
FFLAGS required compilation options FFLAGS required compilation options

4. create the library :

 gmake clean ; gmake

Library libionc4.a is under $HOMEMARS/../LIB/libionc4/intel-XXX/lib

module files comionc4.mod and ionc4.mod are under $HOMEMARS/../LIB/libionc4/intel-XXX/inc

17
 Mars : Version 9.06

MARS environment : local setting up (11)

File connexion

This fortran code developed by IFREMER/PHYSED to

gather MPI output files is supplied with MARS under
$HOMEMARS/../TOOLS/CONNECT
lfremer

1. cd $HOMEMARS/../TOOLS/connect
2. Modify the library paths in compile_cap
3. Compile_cap

The connect.exe is created under the directory

$HOMEMARS/../TOOLS/connect. It will be automatically copied
under $RDIR/CONF/CONF-CASE[/rank_X]
18
 Mars : Version 9.06

MARS environment : local setting up (12)

Domain cutting out executable (for MPI purpose)

This fortran code has been developed by IFREMER/RIC to divide

the regional model in X sub-domains. It is available in
$HOMEMARS/../TOOLS/MPI2D_DOMAIN

1. cd $HOMEMARS/../TOOLS/MPI2D_DOMAIN
2. Modify the library paths in makefile
3. make
lfremer

The ./run executable is created under the directory

$HOMEMARS/../TOOLS/MPI2D_DOMAIN.

A user manual is available from http://wwz.ifremer.fr/dyneco/Moyens-

Outils/Logiciels/MARS/Documentation

19
Retour au Sommaire Mars : Version 9.06

MARS environment : local setting up (13)

FERRET

1. Download files from

http://ferret.pmel.noaa.gov/static/Downloads/
2. Follow instructions of installation

NCO tools : yum install nco

lfremer

nedit editor : yum install nedit

20
 Mars : Version 9.06

MARS environment : commands (1/2)

Useful commands

 mkconfdir : creates all directories required for the implementation of a

configuration (for more details of what mkconfdir does, click here)
 gmake install : sets the reference code into the directory of compilation
(for more details, click here)
 gmake : compiles with user’s routines
(for more details, click here)
lfremer

 gmake clean : cleans *.o and *.f

 gmake cleaninst : empties the directory of compilation (useful when you want to
remove a user’s routine)
 gmake update ["OLDREV=XXX" "NEWREV=YYY"] : sets the reference code into the
directory of compilation and updates user’s routines from revision XXX to revision
YYY
All these commands are launched from the directory MARS_CONFIG/conf/conf-
case, location where the user prepares the code he will run.
21
Retour au Sommaire Mars : Version 9.06

MARS environment : commands (2/2)

Useful commands

 getfile WORK/routine.F90 : allows the user to modify the reference code

source (for more details, click here)

 rmfile routine.F90 : remove a file from MARS_CONFIG/CONF/CONF-CASE.

(for more details, click here)

 cmpfile routine.F90 : Easy way for making either a diff between a file in the
lfremer

MARS_CONFIG/CONF/CONF-CASE directory and the corresponding file in the

HOMEMARS directory. (for more details, click here)

All these commands are launched from the directory MARS_CONFIG/conf/conf-

case, location where the user prepares the code he will run.

22
 MARS environment Mars : Version 9.06

mkconfdir command
creates all directories required for the implementation of a configuration

Usage : mkconfdir CONF CASE (number of ranks)

 without rank (click here)

 with rank (click here)

lfremer

23
 Return Liste commands Mars : Version 9.06

Return new config

MARS environment
mkconfdir command (without rank)
mkconfdir CONF CASE
(CONF for geographic configuration – CASE for one case of this configuration)
 1. Create in $UDIR $UDIR/CONF/CONF-CASE

create empty directories INC PHYS TRAJ BIOLO SEDIM SWAN AGRIF

makefile The makefile file manages the MARS environment and

copy the makefile and calls to Makefile.caparmor for compilation

Makefile.caparmor Makefile. where user chooses cpp keys and

Makefile.caparmor files
compiling option
lfremer

 2. Create in $CDIR
$CDIR/WCONF-CASE
Create empty directory

 3. Create in $RDIR
$RDIR/CONF/CONF-CASE $RDIR/CONF/inputs
Create directories – default namelist files paramain.txt, paraspec.txt,
- examples of files *.dat
paracom.txt, parasubs.txt, parasedim.txt, parabiolo.txt
to give the formats of the files
– output.dat file that allows the user to define the output files managing rivers, outflows,
variables, trajectories…
– script and batchs to run the executable
+ example of
– batch_connect and connect.exe files to gather MPI output files
(l_out_nc4par=.false.)
head.CONF file
24
 Return Liste commands Mars : Version 9.06

Return new config

MARS environment
mkconfdir command (with rank)
mkconfdir CONF CASE 3 (if X=3 ranks)
(CONF for geographic configuration – CASE for one case of this configuration)
$UDIR/CONF/CONF-CASE
 1. Create in $UDIR
INC PHYS TRAJ BIOLO SEDIM SWAN AGRIF
create empty directories
makefile_rankX The makefile file manages the MARS environment and
copy the makefile and calls to Makefile.caparmor for compilation

Makefile.caparmor_rankX Makefile. where user chooses cpp keys and

Makefile.caparmor files compiling option

 2. Create in $CDIR
lfremer

Create empty directory $CDIR/WCONF-CASE

 3. Create in $RDIR
$RDIR/CONF/CONF-CASE
Create directories $RDIR/CONF/inputs
– default namelist files paramain.txt, paracom.txt - examples of files *.dat
to give the formats of the files
$RDIR/CONF/CONF-CASE /rank_0,
– default rank_1
namelist files et paraspec.txt,
rank_2 parasubs.txt….managing rivers, outflows,
each contains : variables, trajectories…
– output.dat file allows the user to define the output files
– script and batchs to run the executable + example of
– batch_connect and connect.exe files to gather MPI head.CONF
output files file
(l_out_nc4par=.false.) 25
 Return Liste commands Mars : Version 9.06

Return new config MARS environment

gmake install command

The setting up (user’s modification of the code, compiling)

of the configuration is done from directory $UDIR/CONF/CONF-CASE

Usage : cd $UDIR/CONF/CONF-CASE
gmake install

This command copies the reference MARS code in $CDIR/WCONF-CASE

lfremer

And creates a link to the directory WORK in $UDIR/CONF/CONF-CASE

CODE_MARS COMPILE_MARS/WCONF-CASE
Reference ($HOMEMARS) Reference code

All routines are available

from WORK directory MARS_CONFIG/CONF/CONF-CASE/WORK
But NEVER go or work directly
In WORK : all MARS routines – as link to $CDIR
under this WORK directory

26
 Return Liste commands Mars : Version 9.06

Return new config MARS environment

Execution : gmake
This command prepares the code, compiles it and copy the executable to
$RDIR
Usage : cd $UDIR/CONF/CONF-CASE
gmake
Copy modified user’s routines in $CDIR

MARS_CONFIG/CONF/CONF-CASE COMPILE_MARS/WCONF-CASE
Reference code
User’s routines + modified user’s routines
lfremer

smallf90 smallf90 +executable exe

Creates smallf90 directory Compiles the code to get the executable

(exe => mars_exe in $RDIR)
smallf90 is a link $CDIR,
NEVER work under this directory smallf90 RUN_MARS/CONF-CASE/CASE(rank*)
In smallf90, the user can see all routines Executable (mars_exe)
used when compiling
(after pre-processing)
27
 Return Liste commands Mars : Version 9.06

Return new config MARS environment

Execution : modification of the reference code source

Usage : cd $UDIR/CONF/CONF-CASE
getfile WORK/routine.F90

 gets a routine and saves it at the correct location in

MARS_CONFIG/CONF/CONF-CASE.

Then, this routine can be modified by the user directly from the
directory MARS_CONFIG/CONF/CONF-CASE.
lfremer

(links are automatically made)

After the modification of the routine by the user

You have to compile the code again by gmake
which takes into account the modified routine

28
Return Liste commands Mars : Version 9.06
MARS environment
remove a file modified by user
and back to reference
Usage : cd $UDIR/CONF/CONF-CASE
rmfile routine.F90

 Remove a file from MARS_CONFIG/CONF/CONF-CASE

 Restore the removed file from the REFERENCES
(HOMEMARS,REF_CONFIG),
lfremer

After asking for confirmation (twice),

it deletes the local link to routine.F90
and moves the file from its actual position to the .trash directory
(this directory is created if it doesn't exist).
It is the user responsability to empty the .trash directory !

29
Return Liste commands Mars : Version 9.06
MARS environment
visualize differences between user’s routine and
reference code
Usage : cd $UDIR/CONF/CONF-CASE
cmpfile routine.F90

Easy way for making either a diff between a file in the

MARS_CONFIG/CONF/CONF-CASE directory and the corresponding file in
the HOMEMARS directory.
lfremer

 Default option launches tkdiff wheter -t option produces a diff -bic listing.
 It can be used as template for personnal improvement.

30
Retour au Sommaire Mars : Version 9.06

MARS environment
new configuration

 New configuration without rank

 New configuration with ranks
 New configuration using (copying) an other configuration (PREV_CONFIG)
 New configuration recovering an other configuration updated by an
other user (REF_CONFIG) WITHOUT RANK
 New configuration recovering an other configuration updated by an
lfremer

other user (REF_CONFIG) WITH RANKS

 Usage of PREV_CONFIG and REF_CONFIG
 Before compilation : choice of CPP Keys (list)
 After compilation and before running

31
Return new configuration list Mars : Version 9.06

MARS environment
new configuration without rank
The rank_0 does not supply the rank_1 with open boundary conditions

 ssh -X -l username caparmor (connection to caparmor )

 mkconfdir MENOR V9.06 (create directories for your configuration
 (here CONF= MENOR, CASE= V9.06)

 cd … (follow the screen directive) (go in the directory $UDIR/CONFIG/CASE)

 gmake install (install the code for your configuration)

 vi Makefile.caparmor and choice of CPP keys, compilation option
lfremer

 getfile WORK/parameters.F90 and change specific parameters to your

configuration
 gmake (compilation)
 modification of the code by the user :
 getfile WORK/routine.F90
 modification of routine.F90
 gmake

32
Return new configuration list Mars : Version 9.06

MARS environment
new configuration with ranks
The rank_0 supplies the rank_1 with open boundary conditions (SSH)

 ssh -X -l username caparmor (connection to caparmor )

 mkconfdir MENOR V9.06 4 (for ranks 0, 1 ,2 et 3) (create directories for your
configuration (here CONF= MENOR, CASE= V9.06, 4 ranks)
 cd … (follow the screen directive) (go in the directory $UDIR/CONFIG/CASE)

 gmake install (install the code for your configuration)

 vi Makefile.caparmor_rankX and choice of CPP keys, compilation option for
each rank
lfremer

 getfile WORK/parameters.F90_rank0
 cp INC/parameters.F90_rank0 INC/parameters.F90_rank1,
 cp INC/parameters.F90_rank0 INC/parameters.F90_rank2 ...
 modification of INC/parameters.F90_rank* (no=, nrac=,imax=,jmax=,kmax=)

 vi makefile and change rank=0,1 or 2 in order to compile the code for this rank
 gmake clean ; gmake (compilation)

33
 Mars : Version 9.06

MARS environment
why variables REF_CONFIG et PREV_CONFIG (1/2)
PREV_CONFIG (with gmake copyconfig)
 During the evolution of a configuration, the creation of a new configuration
CONF-CASE and the use of “gmake copyconfig”
 allows to get exactly the same code as in configuration PREV_CONFIG (‘gmake
copyconfig’ command copies user’s routines and Makefile.linux(or .caparmor) from
directory PREV_CONFIG to $UDIR/CONF/CONF-CASE)
 allows to keep a saving of configuration PREV_CONFIG
 This command is used when updating a configuration with a new version of the
reference code (the user get directly the routines he may have to update)
lfremer

REF_CONFIG
 Useful when several users use the same configuration
 Only one user is in charge of updating the REF_CONFIG configuration
 At each evolution of the reference MARS code, other users have to update their
owns routines only
 Routines of directory REF_CONFIG are directly copied from REF_CONFIG directory
to the compiling directory : the user does not see them in $UDIR/CONF/CONF-CASE
but he uses them

34
Return new configuration list Mars : Version 9.06

MARS environment
Remarks on variables REF_CONFIG et PREV_CONFIG
(2/2)
PREV_CONFIG and REF_CONFIG
 The variables PREV_CONFIG and REF_CONFIG allow to manage the
code but not the input files : they have an influence in directory
$UDIR/CONF/CONF-CASE only
 The user must copy namelists paraspec.txt, paramain.txt, paracom.txt,
parasubs.txt, parasedim.txt and parabiolo.txt (+ output.dat file and mpi.txt)
relative to the configuration PREV_CONFIG or REF_CONFIG
lfremer

PREV_CONFIG
 Makefile.linux files is copied from PREV_CONFIG directory –> update the
path in INCDIR and CPP variables

REF_CONFIG
 Makefile.linux is not changed –> the user has to change it in order to use
the same cpp keys and the same compiling options as the ones used in
REF_CONFIG directory
35
Return new configuration list Mars : Version 9.06

MARS environment
Recovery of a previous configuration
use of variable PREV_CONFIG in makefile
 ssh -X -l username caparmor (connection to caparmor )
 mkconfdir MENOR V9.06 [3] (3 if ranks) (create directories for your configuration
(here CONF= MENOR, CASE= V9.06, 3 ranks)
 cd … (follow the screen directive) (go in the directory $UDIR/CONFIG/CASE)
 gmake install (install the code for your configuration)
 vi makefile : PREV_CONFIG =/export/home11/mars/CONFIG/MENOR/MARS/MARS_CONFIG/MENOR/MENOR-
V9.06 (choose the name of the configuration you want to copy … without any blank at the end of the line)

 gmake copyconfig (Rk : all user’s routines of MENOR-V9.06 are copied to the configuration).
 vi Makefile.caparmor[_rankX]
lfremer

 update the file (if new cpp key or routines have been introduced in a new version of the reference
code ) (compare with Makefile.caparmor_ref)
 vi makefile and change rank=0,1 or 2 in order to compile the code for this rank
 (gmake clean ) ; gmake (compilation)

(To make sure you have the same code, compare smallf90 directory with
PREV_CONFIG/smallf90 directory)

Click for summary of what PREV_CONFIG is doing 36

Return new configuration list Mars : Version 9.06

MARS environment (PREV_CONFIG summary)

$HOMEMARS
mkconfdir CONF CASE (rank)
gmake install (reference code)
gmake copyconfig
gmake
OLD_CONFIG
(configuration to be used)

$UDIR/CONF/CONF-CASE
makefile
Makefile.caparmor[_rankX] $CDIR/WCONF-CASE
tree of the reference code WORK link
reference code
lfremer

PREV_CONFIG = OLD_CONFIG (makefile) OLD_CONFIG routines

user’s routines of OLD_CONFIG user’s routines
Makefile.caparmor[_rankX] link compilation
smallf90
user’s routines
$RDIR/CONF/CONF-CASE

mars_exe
Verification :
diff WORK OLD_CONFIG/WORK ; diff smallf90[_rankX] OLD_CONFIG/ smallf90[_rankX]
37
Return new configuration list Mars : Version 9.06

MARS environment
Recovery of a previous configuration (without rank)
use of variable REF_CONFIG in makefile
 ssh -X -l username caparmor (connection to caparmor )
 mkconfdir MENOR V9.06 (create directories for your configuration
(here CONF= MENOR, CASE= V9.06)
 cd … (follow the screen directive) (go in the directory $UDIR/CONFIG/CASE)
 vi makefile : REF_CONFIG =/export/home11/mars/CONFIG/MENOR/MARS/MARS_CONFIG/MENOR/MENOR-V9.06 (choice
the name of the configuration you wants to copy … without blank at the end of the line)

Rk : all user’s routines of MENOR-V9.06 are compiled but they are not copied in
your directory $UDIR/CONF/CONF-CASE.
NEVER ERASE DIRECTORY REF_CONFIG !!!!!
 vi Makefile.caparmor
lfremer

 Choose the same compiling options and cpp keys… as those in REF_CONFIG/Makefile.caparmor
 If ranks :
 cp REF_CONFIG/INC/parameters.F90_rank* INC/.
 If test case :
 cp REF_CONFIG/INC/parameters.F90_casXXX INC/.
 gmake install (install the code for your configuration)
 gmake (compilation)
(To make sure you have the same code, compare smallf90 directory with REF_CONFIG/smallf90
directory)

38
Return new configuration list Mars : Version 9.06

MARS environment
Recovery of a previous configuration (with rank)
use of variable REF_CONFIG in makefile
 ssh -X -l username caparmor (connection to caparmor )
 mkconfdir MENOR V9.06 2 (create directories for your configuration
(here CONF= MENOR, CASE= V9.06, 2 ranks)
 cd … (follow the screen directive) (go in the directory $UDIR/CONFIG/CASE)
 vi makefile : REF_CONFIG= /export/home1/vgarnier/MARS_CONFIG/calv/calv-V9.06
 (choice the name of the configuration you wants to copy … without blank at the end of the line)

Rk : all user’s routines of MENOR-V9.06 are compiled but they are not copied in
your directory $UDIR/CONF/CONF-CASE.
NEVER ERASE THE REF_CONFIG DIRECTORY !!!!!
lfremer

 get routines parameters.F90 of each rank

 cp /export/home1/vgarnier/MARS_CONFIG/calv/calv-V9.06/INC/parameters.F90_rank* INC/.

 vi Makefile.caparmor_rankX
 Choose the same compiling options and cpp keys… as those in
REF_CONFIG/Makefile.caparmor
 gmake install (install the code for your configuration)
 vi makefile (change RANK =) ; gmake clean; gmake (compilation of each rank)

39
Return new configuration list Mars : Version 9.06

MARS environment (REF_CONFIG summary)

$HOMEMARS
mkconfdir CONF CASE (rank)
gmake install (reference code)
gmake
OLD_CONFIG
(configuration to be used)

$UDIR/CONF/CONF-CASE
makefile
Makefile.caparmor[_rankX] $CDIR/WCONF-CASE
tree of the reference code reference code
lfremer

REF_CONFIG = OLD_CONFIG (makefile) OLD_CONFIG routines

cp OLD_CONFIG/Makefile_caparmor[_rankX] . user’s routines
(modify INCDIR,CPP)
cp OLD_CONFIG/parameters.F90_rank* INC/.

user’s routines
$RDIR/CONF/CONF-CASE

mars_exe
Verification :
diff WORK OLD_CONFIG/WORK ; diff smallf90[_rankX] OLD_CONFIG/ smallf90[_rankX]
40
 Mars : Version 9.06
MARS environment : Update of a configuration (1)

 change the number of reference code version

in .cshrc.caparmor (or env_MARS_VX.XX)

setenv HOMEMARS /export/home11/mars/CODE_MARS/CODE_MARS_V9/V9.06/Mars_Agrif2

 Verify you have in .cshrc.caparmor :

setenv HTTPSVNROOT https://forge.ifremer.fr/svn/mars3d
setenv extranet_login yours (Set your extranet_login name)
lfremer

 Run .cshrc.caparmor typing :

source .cshrc.caparmor
 

41
 Mars : Version 9.06

MARS environment : Update of a configuration (2)

 create a new configuration :
mkconfdir MENOR V9.06 (CONFIG=MENOR, new case or new version : V9.06)
cd ... (follow the screen directive)
gmake install (install the code for your configuration – useless if you update with subversion)
vi makefile :
PREV_CONFIG = /export/home1/vgarnier/MARS_CONFIG/MENOR/MENOR-old
gmake copyconfig
vi Makefile.caparmor[_rankX] : update if new cpp key or routines have been introduced in
the new version (compare to Makefile.caparmor_ref)
 
lfremer

WARNING : No gmake at this step

update user's routines

without SubVersioN use click here
or
with SubversioN use: click here
42
 Mars : Version 9.06

MARS environment : Update of a configuration (2)

[without SubVersioN use]
 
update user's routines without SubVersioN use
a - notice which user’s routines has been modified in reference MARS code
(the list of modified routines is published at each new version – or
compare the routines between the old and the new $HOMEMARS)
 
b – visualize the modifications introduced by the user
vimdiff WORK/routine.F90 $HOMEMARS/../Mars_Agrif2_old/...../routine.F90
lfremer

c - visualize the modifications introduced into the new reference version in

comparison to the old reference version
vimdiff $HOMEMARS/../Mars_Agrif2_XXX/...../routine.F90
$HOMEMARS/../Mars_Agrif2_old/...../routine.F90
 

43
 Mars : Version 9.06

MARS environment : Update of a configuration (3)

[without SubVersioN use]
 
d - 2 cases

1st case : the number of modifications introduced by the user is small

---> getfile WORK/routine.F90
(get the routine of the new reference code because the code has not
been compiled yet)
---> introduce user’s modifications
 
lfremer

2nd case : the number of modifications introduced by the user is larger that
the number of modifications introduced into the new reference code
---> introduce modifications of the new reference code into the user’s
routine
 
 choice CPP Keys (in Makefile_caparmor) then gmake
 cd $RDIR/CONF/CASE (click here to continue)
44
 Mars : Version 9.06

MARS environment : Update of a configuration (2’)

[with SubVersion]
   install and update user's routines
gmake update "OLDREV=XXX" "NEWREV=YYY"
(check the number into $HOMEMARS/../DOC/list_version -FIRST COLUMN- )

  [vi $HOMEMARS/../DOC/list_version to get XXX YYY relative to the old and new versions]
this command updates all the routines of the current tree from an old revision XXX to the current one YYY
using SubVersioN
(you need a client of SubVersioN installed on your machine. Available under caparmor for all the users)

   resolve conflicts for routines described by “C”

a – open 3 windows
lfremer

b – windows 1 : visualize modifications introduced by the user in old version

Setenv HOMEMARS /export/home11/mars/CODE_MARS/V7.71 (only in this window)
Cmpfile routine.F90
b – windows 2 : visualize modifications introduced into the new reference version by comparison to the old reference
version
vimdiff $HOMEMARS/...../routine.F90 /export/home11/mars/CODE_MARS/V7.71/Mars_Agrif2/…/routine.F90
c – windows 3 : correct user’s changes taking care of the modifications added in the reference code. Delete warning
comments relative to the merge
(<<<<.working ; >>>>.merge.right.rXXX ; >>>>>>.merge.left.rXXX)
Cmpfile routine.F90
 

 choice CPP Keys (in Makefile_caparmor) then gmake

cd $RDIR/CONF/CASE 45
Return new configuration list Mars : Version 9.06

MARS environment : Update of a configuration (4)

  update files paramain.txt… under $RDIR/CONF/CONF-CASE

 Run the new configuration and the old one in sequential with no
optimization option de compilation (FLAG = -r8 –O0)

The update is finished and secured

 If you get exactly the same results (all digits) when you compare files named “listing”,
lfremer

files that contain maximum velocities at different time steps,

 Otherwise, if you can explain differences.

Consequently, you need to get exactly the same results before introducing the
corrections or evolutions that modify results. These corrections/evolutions are listed
in the file list_modif_V6 supplied at each new version.

46
 Mars : Version 9.06

MARS environment (summary)

mkconfdir CONF CASE (rank); gmake install; gmake; getfile WORK/casvortex.F90

$UDIR/CONF/CONF-CASE
makefile
Makefile.caparmor[_rankX]
tree of the reference code INC; PHYS/*; BIOLO; TRAJ; SEDIM; SWAN; ..
WORK/all the routines of the reference code (link to $CDIR)
Version (contains used $HOMEMARS)
(edit makefile and choose the RANK or test case)
(edit Makefile.caparmor[_rankX] and choose CPP keys and compiling options)
smallf90[_rankX] (link to $CDIR)
lfremer

PHYS/KTEST/casvortex.F90
casvortex.F90 (link to PHYS/KTEST/casvortex.F90)
edit and modify casvortex.F90, then compile

$CDIR/WCONF-CASE
copy of the entire reference code
WORK contains all the routine (links to PHYS/*…)
smallf90[_rankX] contains all the compiled routines after preprocessing
47
Retour au Sommaire Mars : Version 9.06

MARS environment (summary)

$RDIR/CONF/CONF-CASE
namelists : paramain.txt, paraspec.txt, paracom.txt,
parasubs.txt,parasedim.txt,parabiolo.txt
output.dat
batch_seq, batch_omp, runmpi_connect (which uses batch_mpi and
batch_connect(bydate) ), runmpt_connect…
mpi.txt (for MPI purpose)
mars_exe
lfremer

Before running, the user has to :

 define all the parameters in the different namelist (para*)
 under inputs directory, define head.CONF
 under inputs directory, add the input files (like bathymetry, initial condition,
open boundary conditions…)
 under inputs directory, define all default .dat files read by the simulation (ex
river.dat)

48
 Mars : Version 9.06

Before compilation – choice of CPP keys (1/5)

In Makefile.caparmor : add CPP_keys in
CPPFLAGS =

FORMAT BATHY FILE TO READ

-Dkey_bathy_final to read definite bathymetry in netCDF format file (no more changes)

TIDES - choice of harmonic components

-Dkey_tide_schwid schwiderski model (1980), modele de pg Islande-Portugal (5 km) (on progress)
-Dkey_tide_fes2004 FES 2004
-Dkey_tide_shom CST
-Dkey_tide_lagoon lagoons of Banuyls and Marseilles
-Dkey_tide_lagoon_marseille tide for Marseilles
-Dkey_tide_lagoon_banyuls tide for Banyuls
lfremer

MOMENTUM NON LINEAR TERMS

default centered
-Dkey_dyn_adv_quick QUICK
-Dkey_dyn_adv_quickest QUICKEST
-Dkey_dyn_adv_linear no nonlinear terms

PRESSURE GRADIENT
default pressure jacobian
-Dkey_dyn_pg_djcs Shchepetkin et al. 2004

WET DRYING
-Dkey_dyn_wetdry_fct use of Flux Corrected Transport to correct ssh and velocity estimates and
keep positive water depths 49
 Mars : Version 9.06

Before compilation – choice of CPP keys (2/5)

In Makefile.caparmor : add CPP_keys in
CPPFLAGS =

TRACER ADVECTION
(xy direction)
default Quick / Euler-Upwind scheme
-Dkey_tssub_adv_ultimatequickest quickest scheme (xy direction) + ultimate limiter
-Dkey_tssub_adv_ultimatequickestmacho quickest scheme (xy direction) + ultimate limiter
+ multidimensional extension macho
-Dkey_tssub_adv_fourthorder fourth order centred scheme (xy direction)
lfremer

(along z)
default centered scheme
-Dkey_tssub_wquickupwind QUICK scheme along the vertical (tracer advection)
-Dkey_tssub_wcompact Upwind Compact and Conservative scheme along the vertical

TRACER DIFFUSION
default along sigma
-Dkey_ts_diffh_geo along geopotential
-Dkey_ts_diffh_rho along isopycnes

50
 Mars : Version 9.06

Before compilation – choice of CPP keys (3/5)

In Makefile.caparmor : add CPP_keys in
CPPFLAGS =

HEAT FLUX (radiative fluxes - long and short waves)

-Dkey_sflx_solar_luyten Luyten et al, 1992
-Dkey_sflx_solar_gill Gill, 1982
-Dkey_sflx_ir_swimbank Swimbank (Agoumi thesis)

HEAT TURBULENT FLUXES (latent and sensible fluxes and wind stresses)
-Dkey_sflx_turb_fairall Fairall et al.,2003
-Dkey_sflx_turb_ayina Ayina et Bentamy, 2007
-Dkey_sflx_turb_large Large and Yeager, 2004
lfremer

-Dkey_sflx_turb_luyten Luyten et al, 1992

-Dkey_turb_default POL, used for NOMADS 2 program

ROTATED GRID
-Dkey_grid_rotated to use a rotated grid
-Dkey_tide_saverotatedobc to save 2D obc for a rotated child rank

AGRIF
-Dkey_agrif use both keys at the same time
-Dkey_agrif -DAGRIF_MPI -Dkey_MPI_2D use these 4 keys at the same time to run with MPI

51
 Mars : Version 9.06

Before compilation – choice of CPP keys (4/5)

In Makefile.caparmor : add CPP_keys in
CPPFLAGS =

TRACER
-Dkey_substance
-Dkey_subs_part_eqsubdt to save time for substance with settling velocity

+ SEDIMENTOLOGY
--Dkey_substance -Dkey_sedim -Dkey_sedim_mudsand to use mud/sand sediment module
-Dkey_substance -Dkey_sedim -Dkey_sedim_mixsed to use mixed sediment module

+ BIOGEOCHEMISTRY AND CONTAMINANTS

lfremer

-Dkey_substance -Dkey_contaminant
-Dkey_sedim -Dkey_sedim_mudsand add these keys if some contaminant are particulate
-Dkey_species add this key to grouping species of a contaminant

+ BIOLOGY
-Dkey_substance -Dkey_biolo
-Dkey_messat add this key to read sea surface temperature mesurement

52
Retour au Sommaire Return new configuration list Mars : Version 9.06

Before compilation – choice of CPP keys (5/5)

In Makefile.caparmor : add CPP_keys in

CPPFLAGS =

MPI
-Dkey_MPI_2D

MPI + OMP
-Dkey_MPI_2D -Dkey_MPIOMP

TRAJECTORIES
lfremer

-Dkey_trajec3d trajectories for 3D simulations (called alone without -Dkey_ibm)

Individual Based Model

-Dkey_ibm Biological processes for individual particules
(called alone without -Dkey_trajec3d)

TEST CASES
-Dkey_tssub_casadvsmolartime test tracers schemes for Smolarkievicz configuration

OFFLINE (no estimate of dynamics, tracers advection only)

-Dkey_offline_outuvflux save time averaged 3d velocities and sea level
-Dkey_offline read time averaged dynamics variables and
run the code without the dynamics part
53
 Mars : Version 9.06

After compilation – Before running

Parametrization (1)

 cd $RDIR/CONF/CONF-CASE
 Choice parameters and options in namelists files :
 paramain.txt : general for the configuration (shared with all ranks)
 paracom.txt : shared with all ranks
 paraspec.txt : different for each rank
 parasubs.txt : need if transport substances (key_substance)
 parasedim.txt : need if sediment using (key_sedim)
 parabiolo.txt : need if using modules for biology (key_biolo)
lfremer

 paraconta.txt : need if using modules for contaminant (key_contaminant)

 Choice of options and parameters definig outputs files in output.dat

 Build a file mpi.txt (for MPI purpose if you want to divide the domain along X AND Y)

54
Retour au Sommaire Mars : Version 9.06

After compilation – Before running

Parametrization (2)

 cd $RDIR/CONF/inputs
 modify head.CONF (keep the same format !!!)
 add the input files (like bathymetry, initial condition, open boundary conditions,
meteo …)
 modify all default .dat files read by the simulation before any use and
add the good path in namelists (ex river.dat, trajec2d*.dat, outflow.dat,
variables.dat)
lfremer

 cd $RDIR/CONF/CONF-CASE
 Run with mars_exe or in batch with batch_seq, batch_omp, runmpi_connect
(which uses batch_mpi and batch_connect(bydate) ), or runmpt_connect…
depending of your choice (click here for details)

55
Return to parametrization Mars : Version 9.06

Parametrization (3) : File head.CONF

 A default file head.CONF is automatically copied onto inputs directory
 DO NOT MODIFY ITS FORMAT (no more space or less space !)
 The first 4 letters must be equal to the variable nrac in routine parameters.F90
 The following number is the number of the rank
 The 4 following reals define the extension of the domain
 Northern latitude (at jmax)
 Southern latitude (at jmin+1)
 Eastern longitude (at imax)
 Western longitude (at imin+1)
 The 2 following reals set the step of latitude and longitude respectively (in degrees)
 The 2 following reals set the averaged step of latitude and longitude respectively (in meter)
lfremer

– indicative values, not used by the model

 1st integer = imax
 2nd integer = jmax
 The last 4 integers are the location of the open boundaries of the following rank (child grid)
in the grid of the current rank : jobcmax,jobcmin,iobcmax,iobcmin.
 The last 4 letters is the CONF name of the child grid (following rank or rank of another
configuration as a unique rank is able to save 2d open bounary files for several
configurations at a same time)
PS : to save several 2d obc files, duplicate the line and modify the last 4 integers and the last 4
letters.

56
Return to parametrization Mars : Version 9.06

Parametrization (4a) : file output.dat

setting up of output files

Located under $RDIR/CONF/CONF-CASE(/rank*)

THREE REQUIREMENTS :
1. THE MAIN OUPUT MUST BE DEFINED AT FIRST
2. The user can choose l_demer=.true. or l_out_precdate=.true. (both at the same time is not allowed)
3. The choice l_demer=.true. or l_out_precdate=.true. is possible just once inside the file output.dat.

Definition of parameters are given at the beginning of the output.dat file

See below the list of parameters
lfremer

57
Return to parametrization Mars : Version 9.06

Parametrization (4b) : file output.dat

 ***************************************************************************
 Champs ,| the name of the output file will be champs_suffix.nc, suffix being defining in paracom.txt
 06/03/2005 00:00:00 ,| date_startout (date of the beginning output)
 09/03/2005 00:00:00 ,| date_endout (date to the end output)

 1.0d0 ,| pasor (time step of saving)

 .true. ,| l_posit (.true. If horizontal extent given with indexes i & j ;
 .false. if given with longitude/latitudes)
 0 421 0 501 ,| iextmin,iextmax,jextmin,jextmax (horizontal extent)
 1 30 ,| kextmin,kextmax (between 1 and kmax) (vertical extent)
 111 ,| i-step,j-step,k-step (space step for i,j, k)
 .false. ,| l_filebydate (a file for each date (if.true) or all records in the same (oif false)
 .false. ,| l_out_off (saving of 3D velocities in conservative flux form (for offline purpose)
lfremer

 .false. ,| l_out_precdate (saving at precise dates)

 .false. ,| l_demer (filtering fields using Demerliac filter)
 .true. ,| l_out_ssh (saving of sea surface height)
 .true. ,| l_out_ubt (saving of barotropic (2D) velocities)
 .false. ,| l_out_u3d (saving of total (3D) velocities)
 .true. ,| l_out_sal (saving of salinities)
 .false. ,| l_out_temp (saving of temperature)
 .false. ,| l_out_wind (saving of wind stress)
 .false. ,| l_out_visc (saving of viscosity coefficients)
 .false. ,| l_out_turb (saving of turbulence coefficients)
 .false. ,| l_out_buoy (saving of buoyancy)
 .false. ,| L_out_ atmsflx (saving of cumulative atmospheric fluxes)
 .false. , ,| l_out_wave (saving of waves parameters)
 ***************************************************************************
58
 Mars : Version 9.06

MPI job : domain decomposition (1/2)

PREPARATION : Create a mpi.txt file if you want to divide your

domain along X and Y
 Use decoupe program under $HOMEMARS/../TOOLS/MPI2D_DOMAIN (see
Readme.txt)
 cp bathy_CONF.txt $RDIR/CONF/CONF-CASE[/rankX]/mpi.txt

MPI2D management :
lfremer

 none mpi.txt, imax > jmax :

domain decomposition through vertical stripes

 none mpi.txt, jmax > imax :

domain decomposition through vertical stripes

 mpi.txt exist (issued from decoupe program) :

optimized 2D domain decomposition
59
Retour au Sommaire Return to parametrization Mars : Version 9.06

MPI job : domain decomposition : mpi.txt 2/2)

42 total
7 largeur
6 hauteur
42 demandé
32 nb actifs
Niproc Njproc limin limax ljmin ljmax actif effectif
0 0 0 195 0 62 T 11251
1 0 196 350 0 62 T 7966
2 0 351 487 0 62 T 8631
3 0 488 604 0 62 T 5945
0 1 0 195 63 134 T 11209
lfremer

1 1 196 350 63 134 T 11160

2 1 351 487 63 134 T 9864 For simplification
5 1 791 942 63 134 T 11225 purpose, this file is not
6 1 943 1100 63 134 T 4644 complete
0 3 0 195 207 288 F 0
1 3 196 350 207 288 T 10977
2 3 351 487 207 288 T 11234
3 3 488 604 207 288 T 8792
4 3 605 790 207 288 T 10516
5 3 791 942 207 288 T 1878
6 3 943 1100 207 288 F 0
0 4 0 195 289 384 F 0
Retour au Sommaire Mars : Version 9.06

MARS USE
How to run a job

cd $RDIR/CONF/CONF-CASE/(rank*)
PATHS , MODULES and batchs are specific of each way

batch_seq, batch_omp, Runmpi[t]_connect and batch_ompmpi[t]* etc….

are created at the time of the creation of the configuration (mkconfdir).
They are copied automatically under $RDIR/CONF/CONF-CASE/(rank*)
lfremer

 Sequential job
 OpenMP job
 MPI job
 Hybrid job
(MPI+OPENMP)
Return to run job Mars : Version 9.06

Sequential job
RUN
 If interactive job over 10’ maximum : mars_exe
 If long run (> 10 min under caparmor calculator) : qsub batch_seq

This sequential job is possible if you are using :

PATH & MODULES (caparmor)

 source /usr/share/modules/init/csh
 module load intel-comp/11.1.073 (path for intel fortran and c++)
lfremer

COMPILATION
 cd $UDIR/CONF/CONF-CASE
 vi Makefile.caparmor[_rankX]
 FC = ifort ; LINKER = ifort
 DIRNC = /home1/caparmor/rramel/netcdf4.1.2
 CPPFLAGS = everything except key_MPI_2D
 OMP = (empty)
 Compile

62
Return to run job Mars : Version 9.06

OpenMP job
RUN
 qsub ./batch_omp (queue parallel8)
(click here to see an example
 select=1 to use with 1 unique node of batch_omp file)
 ncpus=8 to use 8 CPUs by node

This OpenMP job is possible if you are using :

PATH & MODULES (caparmor)

 source /usr/share/modules/init/csh
lfremer

 module load intel-comp/11.1.073 (path for intel fortran and c++)

COMPILATION
 cd $UDIR/CONF/CONF-CASE
 vi Makefile.caparmor[_rankX]
 FC = ifort ; LINKER = ifort
 DIRNC = /home1/caparmor/rramel/netcdf4.1.2
 CPPFLAGS = everything except key_MPI_2D
 OMP = -openmp
 Compile 63
 Mars : Version 9.06

MPI job (1/3)

RUN
MPI linux
 runmpi_connect name_run name_outputfile nb_cpu Z [$HOME]
MPT = MPI for SGI but more efficient than MPI under Caparmor
 runmpt_connect name_run name_outputfile nb_cpu Z [$HOME]

 name_run : will get name_run0.oXXX and name_run1.oXXX files at the end of the
job, XXX being the job number under caparmor
 name_outputfile = “string_suffix”, “string” is choosen at the first line in output.dat,
suffix is defined in paracom.txt (ex : champs_V9.06)
 nb_cpu : from 2 to XXX
lfremer

 Z=0 if l_out_nc4par=.true. ; Z=1 if l_out_nc4par=.false. And l_filebydate=.false. ;

Z=2 if l_out_nc4par=.false. and l_filebydate=.true.
 The fifth argument [home] allows a faster creation of the concatened
champs_VX.nc.all file under Caparmor calculator
 This script submits the mpi run and the gathering of output files
(click here to see an example
(1st use of MPI : of batch_mpi file which is
cd $HOME (off Caparmor) ; launched by runmpi_connect)
mkdir pbs ; touch .mpd.conf ; chmod 600 .mpd.conf)

64
 Mars : Version 9.06

MPI job (2/3)

This MPI job is possible if you are using :

PATH & MODULES (caparmor)

 MPI linux
 source /usr/share/modules/init/csh
 module load intel-comp/11.1.073 (path for intel fortran and c++)
 module load intel-mpi/4.0.0.028 (path for library MPI)
lfremer

 MPT (=MPI for SGI, more efficient under Caparmor)

 source /usr/share/modules/init/csh
 module load intel-comp/11.1.073 (path for intel fortran and c++)
 module load mpt/2.01 (path for library MPT) (better to use the command
module switch intel-mpi mpt/2.01)

65
Return to run job Mars : Version 9.06

MPI job (3/3)

This MPI job is possible if you are using for :

COMPILATION
 cd $UDIR/CONF/CONF-CASE
 vi Makefile.caparmor[_rankX]
 FC = mpiifort ; LINKER = mpiifort
 DIRNC = /home1/caparmor/rramel/netcdf4.1.2_par (for MPI use)
 DIRNC = /home1/caparmor/rramel/netcdf4.1.2_mpt (for MPT use)
lfremer

 CPPFLAGS = add -Dkey_MPI_2D

 OMP = (empty)
 Compile

66
 Mars : Version 9.06

Hybrid job (openMP+MPI) (1/3)

RUN
MPI linux
 runmpiomp_connect name_run name_outputfile nb_cpu
nb_omp Z [$HOME]
MPT = MPI for SGI
 runmptomp_connect name_run name_outputfile nb_cpu
nb_omp Z [$HOME]
lfremer

 nb_cpu is the total cpus used for the run

 nb_omp is the number of processes involved in the OMP parallelization. It
must be smaller or equal to 8 (on Caparmor, since there are 8 cpus a node)
 The number of MPI cpus is nb_cpu / nb_cpu. nb_cpu must be a multiple of
nb_cpu !
 For the other arguments, same signification as for runmpi_connect

67
 Mars : Version 9.06

Hybrid job (openMP+MPI) (2/3)

This openMP+MPI job is possible if you are using :

PATH & MODULES (caparmor)

 MPI linux
 source /usr/share/modules/init/csh
 module load intel-comp/11.1.073 (path for intel fortran and c++)
 module load intel-mpi/4.0.0.028 (path for library MPI)
 MPT (=MPI for SGI)
lfremer

 source /usr/share/modules/init/csh
 module load intel-comp/11.1.073 (path for intel fortran and c++)
 module mpt/2.01 (path for library MPT) (better to use the command
module switch intel-mpi mpt/2.01)

68
Return to run job Mars : Version 9.06

Hybrid job (openMP+MPI) (3/3)

This openMP+MPI job is possible if you are using :

COMPILATION
 cd $UDIR/CONF/CONF-CASE
 vi Makefile.caparmor[_rankX]
 FC = mpiifort ; LINKER = mpiifort
 DIRNC = /home1/caparmor/rramel/netcdf4.1.2_par (for MPI use)
 DIRNC = /home1/caparmor/rramel/netcdf4.1.2_mpt (for MPT use)
lfremer

 CPPFLAGS = add -Dkey_MPI_2D –Dkey_MPIOMP

 OMP = -openmp
 Compile

69
Retour au Sommaire Return to run job Mars : Version 9.06

Run a job under caparmor : example of a batch file (OMP)

Batch_omp
 #!/bin/csh ! Script in cshell
 #PBS -q parallel8 ! Choice of the queue
 #PBS -l select=1:ncpus=4 ! 1 node, 4 cpus

 # get the path for library MKL

 source /usr/share/modules/init/csh
 module load cmkl/recent
 setenv MKL_SERIAL YES

 #default OMP_SCHEDULE is "static", instead use dynamic for mars

lfremer

 setenv OMP_SCHEDULE "dynamic,1"

 # cd to the directory you submitted your job

 cd $PBS_O_WORKDIR
 pwd

 date
 ./mars_exe > ./mars_out_omp
 date
70
Retour au Sommaire Return to run job Mars : Version 9.06

Run a job under caparmor : example of a batch file (MPI)

 Defined from runmpi_connect that launches batch_mpi

 #PBS -q parallel8 ! Choice of the queue
 #PBS -l select=1:ncpus=4:mpiprocs=4 ! 1 node, 4 cpus and 4 procs a node (one is
certain not to share the node with another job)

Batch_mpi
 #!/bin/csh ! Script in cshell
 # get the path for mpirun
 source /usr/share/modules/init/csh
 module load intel-mpi/recent
 # get the path for library MKL
lfremer

 module load cmkl/recent

 setenv MKL_SERIAL YES

 # cd to the directory you submitted your job

 cd $PBS_O_WORKDIR
 pwd

 date
 mpirun -np $number_cpu time ./mars_exe > ./mars_out_mpi
 date
71
 Mars : Version 9.06

Run a job under caparmor (Summary 1/2)

batch_seq, batch_omp, are created at the time of the creation of the
configuration (mkconfdir). They are copied automatically under
$RDIR/CONF/CONF-CASE/(rank*)

Path and modules (caparmor)

 source /usr/share/modules/init/csh
 module load intel-comp/11.1.073

Sequential job
 Makefile.caparmor :
lfremer

 FC = ifort ; LINKER = ifort

 DIRNC = /home1/caparmor/rramel/netcdf4.1.2
 qsub batch_seq

OpenMP job
 Makefile.caparmor :
 FC = ifort ; LINKER = ifort
 DIRNC = /home1/caparmor/rramel/netcdf4.1.2
 OMP = -openmp
 qsub batch_omp
72
Retour au Sommaire Mars : Version 9.06

Run a job under caparmor (Summary 2/2)

Runmpi[t]_connect and batch_ompmpi[t]* are created at the time of the creation of the
configuration (mkconfdir). They are copied automatically under $RDIR/CONF/CONF-
CASE/(rank*)
Path and modules (caparmor)
 source /usr/share/modules/init/csh
 module load intel-comp/11.1.073
 module load intel-mpi/4.0.0.028 or module mpt/2.01
MPI job
 Makefile.caparmor :
 FC = mpiifort ; LINKER = mpiifort
 DIRNC = /home1/caparmor/rramel/netcdf4.1.2_par (for MPI use)
 DIRNC = /home1/caparmor/rramel/netcdf4.1.2_mpt (for MPT use)
lfremer

 CPPFLAGS = add -Dkey_MPI_2D

 runmpi[t]_connect
[t] name_run name_outputfile nb_cpu 1/2 [$HOME]

MPI-OpenMP job
 Makefile.caparmor :
 FC = mpiifort ; LINKER = mpiifort
 DIRNC = /home1/caparmor/rramel/netcdf4.1.2_par (for MPI use)
 DIRNC = /home1/caparmor/rramel/netcdf4.1.2_mpt (for MPT use)
 CPPFLAGS = add -Dkey_MPI_2D -Dkey_MPIOMP
 OMP = -openmp
 runmpi[t]omp_connect
[t] name_run name_outputfile nb_cpu nb_omp 1/2 [$HOME]
 nb_cpu must be a multiple of nb_omp
73
Retour au Sommaire Mars : Version 9.06

How to use a test case ?

 ssh -X -l username caparmor

 mkconfdir grav V9.06 (CONF=grav, CASE=V9.06)
 cd … (follow the screen directive)
 vi makefile
 TESTCASE = 'use'
 TEST = casgravadj (here choice of « casgravadj » test case)
lfremer

 Modify cpp key in ‘Makefile.caparmor’ as indicated above the variable

TESTCASE in ‘makefile’
 gmake install
 gmake or gmake clean; gmake
 cd $RDIR/grav/grav-V9.06
 mars_exe

74
Retour au Sommaire Mars : Version 9.06

How to set a new test case ?

 ssh -X -l username caparmor
 mkconfdir ktest V9.06
 cd … (follow screen directive)
 gmake install
 getfile WORK/parameters.F90_casseamount
 mv INC/parameters.F90_casseamount INC/parameters.F90_casktest
 modifications of parameters.F90_casktest (num_testcase,l_testcase,imax,jmax,kmax),
 getfile WORK/seamount.F90
 ls –lt
 mv PHYS/KTEST/seamount.F90 PHYS/KTEST/casktest.F90
lfremer

 modifications of casktest.F90
 Introduction of routine casktest.F90 inside Makefile.caparmor
 getfile WORK/casinit.F90
 Introduction of routine casktest.F90 inside casinit.F90, choose a number of test case
(num_testcase) and save the same num_testcase number in parameters.F90_casktest
 vi makefile :
 TESTCASE = 'use'
 TEST = casktest
 vi Makefile.caparmor (choice of CPP keys, compilation option)
 gmake or gmake clean; gmake

75
Retour au Sommaire Mars : Version 9.06

How to modify MARS code

 Rules of use and coding inside MARS code

lfremer

 Add a new routine

76
 Mars : Version 9.06

Rules of use and coding inside MARS code

1. MARS environment
2. Follow evolutions of the code (MAJ users)
3. Report any evolutions or bug
4. Code
 LANGUE : English
 LANGAGE : Fortran 90
1. none SAVE
2. none declaration integer :: i=0
3. No dynamics allocation of state variables
4. Logique l_
lfremer

 MODULES : gather routines inside a module

1. Module name : without _
2. routines : module name + _init...
3. ex module output content : routines output_init, output_mng,output_save
4. example of module : $HOMEMARS/../EXAMPLES/module_type.F90
5. Parallelization : results in sequential = results in parallel (all digits)
6. Update of a configuration : old version = new one (all digits)
7. Ask Franck or Valérie for any new development
77
Retour au Sommaire Mars : Version 9.06

Add a new routine

 cd $UDIR/CONF/CONF-CASE
 save routine under
 PHYS/KTEST if it is a test case
 PHYS/M2D if concerns 2D code
 PHYS/MAIN if it is a main routine
 PHYS/COL if it is a routine called during computation along the column
 PHYS/ROW if it is a routine called during computation along the row
 PHYS/INIT if routine of initialization
 PHYS/TRA if concerns tracers temp, sal or passives tracers
 PHYS/SFLX if concerns atmospherical fluxes
lfremer

 BIOLO if it is a routine relative to the biological dynamics

 TRAJ if concerns trajectories
 IBM if concerns Individual Based Model
 SEDIM if concerns sedimentology
 CONTA if concerns contaminants
 vi Makefile.caparmor[_rankX] or Makefile.linux[_rankX]
 add routine into list of routines
 gmake
78