2.6 Simulation Setup

Sub-sections

In the directory STD_TEST/modipsl/modeles/IPSLCM4_v1/EXP00 can be found the reference script for the IPSL Coupled model :
AA_job as well as the just created launching script (Job_LJ31).

Figure 2.7 : Arborescence de IPSLCM4_v1

In the first part of this script, between the following headers, are parameters that the user can modify :

Figure 2.8 : Start of the zone reserved for user modifications

Figure 2.9 : End of the zone reserved for user modifications

In the above mentionned block, it is possbile to set the simulation length by initilising the starting date :

JOUR_DEBUT_EXP, (starting day)
MOIS_DEBUT_EXP, (starting month)
AN_DEBUT_EXP (starting year)

and the stopping date :

JOUR_FINAL_EXP, (stopping day)
MOIS_FINAL_EXP, (stopping month)
AN_FINAL_EXP(stopping year)

of the simulation or experience.

A simulation is sub-divided in a whole number of simulated periods for which it is mandatory to set the duration.

JOURS, (days)
MOIS, (months)
AN (years)

Note that it can not exceed the overall simulation duration.

If it is clearer !!! (For a 360 days calendar)
The last day of simulation (JOUR_DERNIER_EXP) - The first day of the simulation (JOUR_PREMIER_EXP) + 1 =
an integer multiple of period duration ( n x (JOUR_DERNIER_JOB -JOUR_PREMIER_JOB + 1))

with
##-- Job start and stop days computation
(( JOUR_PREMIER_JOB = ( ${AN_DEB_JOB} - 1 ) * 360 + ( ${MOIS_DEB_JOB} - 1) * 30 + ( ${JOUR_DEB_JOB} ) ))
(( JOUR_DERNIER_JOB = ( ${AN_FIN_JOB} - 1 ) * 360 + ( ${MOIS_FIN_JOB} - 1) * 30 + ( ${JOUR_FIN_JOB} ) ))
##-- Simulation start and stop days computation
(( JOUR_PREMIER_EXP = ( ${AN_DEBUT_EXP} - 1 ) * 360 + ( ${MOIS_DEBUT_EXP} - 1 ) * 30 + ( ${JOUR_DEBUT_EXP} ) ))
(( JOUR_DERNIER_EXP = ( ${AN_FINAL_EXP} - 1 ) * 360 + ( ${MOIS_FINAL_EXP} - 1 ) * 30 + ( ${JOUR_FINAL_EXP} ) ))

For information : n is the maximal value taken by NITER_MAX

The following figure illustrates the way in which dates are stored in the AA_job script.

Figure 2.10 : Simulation duration definition

2.6.1 CPU limits and memory

It is also important, considering simulation duration, to define CPU time limits

#-Q- sxnec #@$-lT,
#-Q- sxnec #@$-lt,

as well as the memory limits

#-Q- sxnec#@$-lM.

Which, in the AA_job script are represented as follows :

Figure 2.11 : CPU and memory limits

To appriopriately set the above parameters, the following table can be of use.
It gives the multi-processor classes of the IDRIS center.
If the parameter NQS "-q multi" on Uqbar (see GENERAL REMARKS bellow),
the code will be placed in a parallel class according the following table
and depending on the time, memory and number of processors requested :

Figure 2.12 : multi-processors table for a node of Uqbar

GENERAL REMARKS :

The umxy classes are mono-processor ones;
The number x, increases with the CPU time and the number y with the memory.
The upxy are 4 processors max. classes, while the uppxy are classes of 5 to 8 processors.

Directing the simulation to a multi-processors class is done via the option NQS "-q".
It can be coded with an NQD directive in the submission script at the level of the qsub command :

From Uqbar :

: #@$-q multi (or qsub -q multi job_parallel)
: #@$-q multinode
: (or qsub -q multinode job_multinodes)

For more information on the Uqbar Classes, please see the IDRIS center web site :

Uqbar : code execution

The multi-processors classes must not be used for mono-processor type of jobs, and a sufficient speedup will be required.
The parameter NQS -lM enables the fix the memory of the job.
Waiting time depends on the precision of this value.
The first column of the table gives the maximum limit in terms of CPU time that can be specified for the parameters NQS -lT and/or -lt.

ATTENTION
For the moment, these limits are identical.
Waiting time depends on the precision of this parameter.

The CPU time specified by the parameter NQS -lT controls the global maximum time consumed by all the job processes
("Per-request CPU Time Limit").
The CPU time specified by the parameter NQS -lt represents the maximum CPU time associated to a process
("Per-process CPU Time Limit").

RECOMMANDATIONS :

Estimate, as precisely as possible, the memory, the CPU time necessary and the number of needed processors in order to reduce waiting time.

In case of an error : In case the requested parameters do not satisfy any class the job goes in class "z_error".

The following table, given for a 30 days period, provides useful information for the above parameterisation.

	Real Time (min)	User Time (min)	Memory Size (Gb)
In medium resolution	48	40	3
In low resolution	38	28	3

Moreover, it is possible, in the same job, to chain several times the same simulated period.
This can be done by appropriately setting the following variable in AA_job :

NITER_MAX

Figure 2.13 : Variable enabling chaining in a simulation

and, in this case, memory and time limits of the job must be adapted.

2.6.2 Variables responsible for output frequency

It is possible to modify the default output frequencies;
simulationFor a more than 30 days long simulation, the output frequency of each component is monthly, otherwise it is a daily frequency.
For the variables to be considered, they must be un-commented.

For the Ocean and the Sea Ice, the lines with the following variable must be un-commented :

ORCA_NWRITE : Writing frequency for the output files of the Ocean and Sea Ice components.

This variable accepts the following values in AA_job :

Figure 2.14 : Variable enabling the change of output frequencies for ORCA and LIM

It is also possible to have, in a same and given simulation, files with a monthly output frequency and others with a daily output frequency.

In addition, for the Atmosphere and Orchidee desired output frequencies can be set by using the following variables :

OK_journe : Daily output frequency
OK_mensuel : Monthly output frequency
OK_instan : Immediate output frequency
ecritphy : Output frequency (in days) for the "histphy" file.
iperiod : Period for the Matsuno step (in steps)
iphysiq : Period fot eh Physic (in steps)

These variables accept the following values in AA_job :

Figure 2.14 : Variables enabling the change of output frequency for ATM and SRF

2.6.3 Post treatment flags

Thanks to post treatment flags it is possible to request different types of analyses
on the gross output files of each component of the Coupled model of the IPSL.
By default these flags are unset (positioned to ''n'') and can be set by positioning them to ''y''.

Figure 2.15 : Flags enabling the activation or not of post treatment

Two types of flags exist :

The flags enabling the generation of new files. They all have OK_POST as prefix.
The flag enabling to put the generated files, on a per analysis type basis, on the DODS server of the IDRIS center. They all have OK_DODS as prefix.

Follows a summary of the different types of analyses available :

OK_POST_ATLAS_SE	`Atlas generations`
OK_POST_monitoring	`Monitoring set-up`
OK_POST_DA2ST	`Progressive generation of temporal series for daily frequencies`
OK_POST_MO2ST	`Progressive generation of temporal series for monthly frequencies`
OK_POST_SE	`Progressive generation of the meduim seasonal cycle (by default, every 10 years).` `Default value can be changed by assigning an integer (different to 10) value to freq_SE variable.` For example : freq_SE=5
OK_POST_MO2SN	`Progressive generation of seasonal averages for monthly output frequencies files`
OK_POST_DA2MO	`Monthly average for thr daily output frequencies files`
OK_POST_DA4MO	`Monthly concatenation for daily output frequencies files`
OK_POST_DA2YE	`Yearly average for daily output frequencies files`
OK_POST_DA4YE	`Yearly concatenation for daily output frequencies files`
OK_POST_MO2YE	`Yearly average for monthly output frequencies files`
OK_POST_MO4YE	`Yearly concatenation for monthly output frequencies files`
OK_POST_txt2tar	`Archive files generation (.tar files) + text file clean-up (by default every 10 years).` `The default value can also be changed by assigning an integer (different to 10) value to freq_txt2tar.`

2.6.4 Initialising a simulation

The simulation can be started following different modes :

Either by creating an initial state for a simulation that has never run before, and in this case there is nothing more to do.
Or by starting from Restart files of a simulation having the same name and that has already run on your account and on the same machine (OK_RESTART_SS),
Or by starting from Restart files of a simulation that has run on another account and which does not necessarily have the same name than the one to be placed in machine (OK_RESTART_OS)

2.6.5 Launch a simulation from a RESTART file of an identical simulation

To start a simulation from Restart files of a simulation that has already run on your account the flag OK_RESTART_SS must be set to ''y''.
In AA_job it is represented as follows :

Figure 2.15 : Restarting from a RESTART of a simulation that has already run on your account

2.6.6 Launch a simulation from a RESTART file of another simulation

To start a simulation from a Restart file of another simulation that has run on another account the flag OK_RESTART_OS must be set to ''y''.
Moreover the following variables must be defined :

AN_RESTART : The end year of the Restart file,
MOIS_RESTART : The end month of the Restart file,
JOUR_RESTART : The end day of the Restart file,
CEXPER_OS : The name of the simulation that has generated the Restart file
GROUP : The group to which belongs the person that has launched the simulation from which will be extracted the Restart files,
LOGIN : The login of the person that has launched the simulation from which will be extracted the Restart files

These variables can be found in the launching script of the Coupled model :

Figure 2.16 : Restarting from a RESTART of a simulation that has run on another account

CNRS - Institut Pierre Simon Laplace

Université Pierre et Marie Curie
4, place Jussieu
casier 101
75252 PARIS Cedex 5
FRANCE