{{{ #!html

Simulation setup

}}} ---- [[NoteBox(note,This chapter describes how to setup your simulation once you have compiled your configuration at a chosen resolution., 600px)]] [[PageOutline(1-2,Table of contents)]] ---- In this chapter, we suppose that you have followed the previous steps ([wiki:Doc/Install installation] and [wiki:Doc/Compile compilation]). After the compilation, you should have the following tree view: [[Image(compilation.png, 400px)]] # Create submission directory and main job # ## Create submission directory ## The configuration directory (`modipsl/config/`) contains tools to compile (''Makefile'' and ''AA_make'' files) and tools to run a simulation (two directories (EXPERIMENTS and GENERAL) that allow you to create submission directories for your model configuration). [[BR]] If one or several submission directories (e.g. EXP00, LMDZOR01, historical, OOL_SEC, etc...) have already been created, you can directly go to the [#Contentsofthesubmissiondirectory next step]. * '''EXPERIMENTS''' : In this directory you will find several sub-directories for each experiment you can produce with the same executable. For example : * for IPSLCM6 you can choose experiments between LMDZ, LMDZOR, IPSLCM, and IPSLESM; * for LMDZOR_v6 you can choose experiments between LMDZOR et LMDZ; * for LMDZORINCA_v6 you can choose experiments between LMDZ, LMDZOR and LMDZORINCA. * '''GENERAL''' : In this directory you will find scripts and parameters files independent of the experiment (divided into 3 directories POST, PARAM and DRIVER). Each of the sub-directories in EXPERIMENTS will contain a reference experiment (e.g. ''clim'', ''amip''... for LMDZOR, ''NMHC_AER'', ''AER'' and ''GES''... for LMDZORINCA, ''piControl'', ''historical''... for IPSLCM) and the file '''config.card''' which will be your simulation's initial setup. To prepare your working directory you must know what kind of simulation you want to perform (i.e. choose a predefined experiment). Then, you must copy its own '''`config.card`''' file at the same level as the main ''Makefile''. [[BR]] For example, to perform a '''clim_360d''' experiment with '''LMDZOR_v6''' configuration: {{{ cd modipsl/config/LMDZOR_v6 cp EXPERIMENTS/LMDZOR/clim_360d/config.card . ls AA_Make Makefile EXPERIMENTS GENERAL config.card }}} [[Image(creation_exp_v6.png, 800px)]] When you copied the ''config.card'', you must change at least '''!JobName''' field (your simulation's name) and check the '''Parallelization''' option (will depend of your resolution). In the following example, a simulation called ''!MyJobTest'' is created: {{{ #D-- UserChoices - [UserChoices] #============================ JobName=MyJobTest #D- For each component, ATM= (gcm.e, gcm.e, 71MPI, 8OMP) SRF= ("", "") SBG= ("", "") IOS= (xios_server.exe, xios.x, 1MPI) }}} [[NoteBox(note, The standard parallelization options are: \\ - '''144x142x79''' => 71 MPI x 8 OMP + 1 MPI \\ - '''96x95x39''' => 31 MPI x 4 OMP + 1 MPI, 350px)]] Then run the '''`ins_job`''' script to create the submission directory. This directory will have the same name as ''!JobName'' and the ''config.card'' file is moved in. {{{ ../../libIGCM/ins_job ls AA_Make Makefile EXPERIMENTS GENERAL MyJobTest }}} When you launch ''ins_job'' command, it will ask you some questions : - '''TGCC''' (irene) : (2 questions) {{{ Hit Enter or give project ID (default is gencmip6), possible projects are gen2201 gen7719 gencmip6 : }}} => ''indicate on which project you will work'' [[BR]] {{{ ProjectID is gen2201 and ProjectNode for PostProcessing is standard Hit Enter or give NUMBER OF CORES required for post-processing (default is "4"), possible numbers of cores are "1" to "48" : }}} => ''choose the default except if you have some problem of time and memory'' - '''IDRIS''' (jean-zay) : [[BR]] ''to come'' When you launched `../../libIGCM/ins_job`, you will have the following new files and directories: [[Image(ins_job_v6.png, 800px)]] For more details about the `ins_job` script, you can have a look at the [#Thescriptins_job following sub-section.] ---- '''To summarize to create a submission directory:''' \\ \\ {{{ cd modipsl/config/LMDZOR_v6 cp EXPERIMENTS/LMDZOR/clim_360d/config.card . vi config.card ### Modify at least JobName=MYEXP ../../libIGCM/ins_job ### Answer the questions }}} ---- ## The script ins_job ## `ins_job` is a script with 4 purposes : * create a submission directory. This will only be done if `ins_job` is launched from a directory where EXPERIMENTS and GENERAL sub-directories are found; * create a main job corresponding to a simulation. The main job is based on libIGCM/AA_job; * create post-processing jobs in libIGCM using all other libIGCM/AA_* files; * prepare the ensembles (optional, see section 6 bellow). The script `ins_job` should be run directly from where the config.card is found (in the configuration directory or in the submission directory). Note that `ins_job` will never overwrite an existing job or directory. [[BR]] The !JobName and options about parallelization (number of cores per executable) are used to create the main job header (cf [#Mainjobofthesimulation Main job of the simulation] section bellow). ---- # Contents of the submission directory # The contents of the new directory are described below. {{{ cd MyJobTest ls config.card COMP/ PARAM/ POST/ DRIVER/ }}} [[BR]] ## config.card ## The ''config.card'' file contains the settings of your simulation configuration. The file contains several sections with the simulation settings (e.g. name, duration, processors' number, post processing, initial state). [[BR]]Below is a list of the file sections: ### The [!UserChoices] section ### * !JobName --> simulation name * !ExperimentName --> experiment name (following the CMIP6 nomenclature for the IPCC simulations) * !SpaceName --> variable indicating the type of a simulation. Choose between PROD, DEVT and TEST. !SpaceName=TEST is a special case deactivating pack and storage. * !LongName --> description of your simulation * !TagName --> '''do not change this field'''; describes to which configuration family your experiment belongs * !ExpType --> '''do not change this field'''; allows you to find the EXPERIMENTS directory in which you are working * !DateBegin --> simulation start date (yyyy-mm-dd) * !DateEnd --> simulation end date. It must be the last day "included" in your simulation * !PeriodLength --> frequency of the executable run. This parameter can be 1M, 1Y or 10Y (1D or 5D for a test) * !JobNumProcTot --> number of processors required by your simulation. * ARCHIVE --> optional: path to base directory for output files. By default this is set in libIGCM depending on the machine. This variable is suggested to be used at obelix to change the default output directory which is /home/scratch01/login. * !DataProject --> optional: This variable can be added currently only for use at irene if you want to store output on another project space than the one used in the job headers. The main job and all post-processing jobs will store at this project even if the computing hours are taken from another project. Note that by default the storage space is the same as the computing project. For example !DataProject=gen6328. The parameters ''!ExperimentName'' and ''!SpaceName'' are optional. They impact on the path to the storage directory for the simulation output. `SpaceName=TEST` is a specific case which deactivate pack and storage at archive directory which means that the output will be stored only at SCRATCHDIR (at TGCC) or WORKDIR (at IDRIS). [[BR]] [[BR]] ''Example 1'': With a !ExperimentName and !SpaceName=TEST {{{ #!sh JobName=MyJobTest ExperimentName=DIADEME SpaceName=TEST TagName=LMDZOR }}} The output directory will be `IGCM_OUT/LMDZOR/TEST/DIADEME/MyJobTest`.[[BR]] ''Example 2'': without !ExperimentName and !SpaceName {{{ #!sh JobName=MyJobTest TagName=LMDZOR }}} The output directory will be `IGCM_OUT/LMDZOR/MyJobTest`. [[NoteBox(warn,The character "_" is not allowed in the variables !JobName\, !ExperimentName and !SpaceName, 600px)]] [[NoteBox(note,!PeriodLength allows you to determine the integration length of an execution for your configuration (restart files creation frequency), 600px)]] [[Image(simulation_periodLength.png, 600px)]] A simulation is a succession of periods, !PeriodLength parameter allow to determine the size of these periods. At the end of each period the simulation will create outputs and restart files. Restart files will be use to initial the next period of simulation. [[NoteBox(note,If !SpaceName=TEST all output will be store on $SCRATCHDIR (at TGCC) or $WORKDIR (at IDRIS), 600px)]] ### The section [Executable] ### This section contains one line for each model component giving the executable's name in the `bin/` directory, the executable's name copied to the working directory and resource specifications. You should only change this section if your executable is running in parallel using MPI and OpenMP or if you have changed the executable's name. Note : (",") indicates that this component has no executable. It is defined in a library linked to another executable (e.g. Orchidee in LMDZOR or Inca in LMDZINCA). '''Example for an MPMD MPI execution with NEMO and XIOS''' : Ocean on 127 MPI processes and IO Server on 1 MPI processes. {{{ #!sh [Executable] #D- For each component, Real name of executable, Name of executable in RUN_DIR directory, Number of MPI processes, Number of OpenMP threads OCE= (opa, opa.xx, 127MPI) ICE= ("" ,"" ) MBG= ("" ,"" ) IOS= (xios_server.exe, xios.x, 1MPI) }}} '''Example for an MPMD hybrid MPI/OpenMP execution with IPSLCM coupled configuration''' : Atmosphere on 27 MPI processes and 4 OMP threads per processes, Ocean on 19 MPI processes, IO Server on 1 MPI processes. {{{ #!sh [Executable] #D- For each component, Real name of executable, Name of executable in RUN_DIR directory, Number of MPI processes, Number of OpenMP threads ATM= (gcm.e, lmdz.x, 27MPI, 4OMP) SRF= ("" ,"" ) SBG= ("" ,"" ) OCE= (opa, opa.xx, 19MPI) ICE= ("" ,"" ) MBG= ("" ,"" ) CPL= ("", "" ) IOS= (xios_server.exe, xios.x, 1MPI) }}} '''Another example for an MPMD hybrid MPI/OpenMP execution with LMDZ and XIOS''' : Atmosphere on 47 MPI processes, 8 OMP threads per processes and and IO server on 1 MPI processes. {{{ #!sh [Executable] #D- For each component, Real name of executable, Name of executable in RUN_DIR directory, Number of MPI processes, Number of OpenMP threads ATM= (gcm.e, lmdz.x, 47MPI, 8OMP) SRF= ("" ,"" ) SBG= ("" ,"" ) IOS= (xios_server.exe, xios.x, 1MPI) }}} '''Example for an SPMD hybrid MPI/OpenMP simulation with LMDZ''' : Atmosphere on 32 MPI processes and 4 OMP threads per processes. {{{ #!sh [Executable] #D- For each component, Real name of executable, Name of executable in RUN_DIR directory, Number of MPI processes, Number of OpenMP threads ATM= (gcm.e, lmdz.x, 32MPI, 4OMP) SRF= ("" ,"" ) SBG= ("" ,"" ) }}} ### The [Restarts] section ### The Restarts section allows to start a simulation from another existing simulation. This simulation can be found at the archive machine or at local scratch- or workdir. Activate by setting '''!OverRule=y'''. All components (e.g. ATM, SRF, etc) will then use the same simulation as restart state. {{{ [Restarts] OverRule=y RestartDate=2319-12-31 # Last day of the experience used as restart for all components RestartJobName=CM607-LR-pdCtrl-01 # Define restart simulation name for all components RestartPath=${R_IN}/RESTART/IPSLCM6/DEVT/pdControl # Path Server Group Login }}} The root path for the '''!RestartPath''' depends of the computing center (details about file systems at [wiki:Doc/ComputingCenters/TGCC##Aboutfilesystems TGCC] and [wiki:Doc/ComputingCenters/IDRIS#Thingstoknowaboutfilesystems IDRIS]). They can be: {{{ ${ARCHIVE} # The storage machine of the computing center. # This space can contain tar of restarts or # usual restarts files. # TGCC : ${STOREDIR} # IDRIS : ${STORE} ${SCRATCHDIR} # The large TGCC workspace (no backup). # This kind of space can contain usual # restarts files. ${SCRATCH} # The large IDRIS workspace (no backup). # This kind of space can contain usual # restarts files. }}} [[NoteBox(note, On both computing centers (TGCC and IDRIS)\, if you have multiple projects you will have one unique login connected to all your projects. Thus you will have have a home\, a storedir\, a workdir and a scratchdir by project ,600px)]] [[NoteBox(note, libIGCM manages the difference in treatment between a path pointing to restart files that are directly accessible (without pack) and a path pointing to restart files that are in tar format (after pack). The management is made according to the path you provided. ,600px)]] ### The [ATM], ..., sections of the model components ### This section for each of the model components allows you to: * '''define the output frequency'''; * '''define whether this component is installed''' which will only be considered if you specified '''!OverRule=n''' in the [[#TheRestartssection Restarts]] section. The possible settings for the '''!RestartPath''' options are the same as for the [Restarts] section. The possible settings for the '''!WriteFrequency''' options are: * 1M (monthly) * 5D (5-day) * 1D (daily) * HF (6-hour high frequency) * HF3h (real-time 3-hour frequency - specific to LMDZ) * HF3hm (3-hour averaged high frequency - specific to LMDZ) * STN (instantaneous output only for the CFMIP stations - specific to LMDZ). {{{ #!sh [ATM] WriteFrequency="1M 1D" # Activate the writing frequency of this component Restart=y # If config_Restarts_OverRule == 'n' next 4 params are read RestartDate=2319-12-31 # Last day of the experience used as restart for this component if Restart=y RestartJobName=CM607-LR-pdCtrl-01 # Define restart simulation name for this component RestartPath=${R_IN}/RESTART/IPSLCM6/DEVT/pdControl # Path Server Group Login }}} '''!WriteFrequency specific to the model components'''[[BR]] * '''LMDZ''' : ([ATM]) Each of the frequencies settings 1M, 1D, HF, HF3h, HF3hm, and STN correspond to a given output file. For example, if you specify ''1M'', a histmth.nc file will be created. If you want to change the output frequency in the histmth file you must change the corresponding lmdz parameter file. See [wiki:Doc/Models/LMDZ here]. * '''ORCHIDEE''' * '''[SRF]''' : The first frequency corresponds to the output frequency for the sechiba_history.nc file. The available frequencies are: xY, xM, 5D, 1D and xs, where x is an integer and s means seconds. This file is required. If you add HF, a second sechiba_out_2.nc file will be written with the 3H frequency. * '''[SBG]''' : Only one frequency (xY, xM, 5D, 1D or xs) can be specified. The same frequency is applied to both the stomate_history.nc and stomate_ipcc_history.nc files. Exception in v5 configurations, the stomate_ipcc_history.nc file is always containing daily output. * '''INCA''' : the section !WriteFrequency does not work. Click [wiki:Doc/Models/INCA here] to learn more about how to change the writing frequency. ### The [Post] section ### The options of the [Post] section will allow you to set or disable the frequencies for submitting [wiki:Doc/Running#Simulation-Postprocessing post processing jobs] by changing the 5 following options (see the diagram below). If you do not wish to run post processing jobs, you must specify '''NONE''' for both '''!TimeSeriesFrequency''' and '''!SeasonalFrequency'''. '''!RebuildFrequency''' and '''!PackFrequency''' should not be disabled except in the case of running in expert mode. [[Image(wiki:Doc/Running:chaine.jpg, 50%)]] {{{ #!sh RebuildFrequency=1Y # Frequency of rebuild submission (use NONE for DRYRUN=3) PackFrequency=1Y # If absent default to RebuildFrequency. TimeSeriesFrequency=1Y # Frequency of post-processing submission (NONE if you don't want) SeasonalFrequency=2Y # Seasonal average period (NONE if you don't want, # 2Y at least, 10Y by default) SeasonalFrequencyOffset=0 # Offset for seasonal average first start dates ; # same unit as SeasonalFrequency }}} ## COMP directory ## [[Image(COMP dataflow.jpg, 360px)]] This directory contains the architecture (or map) of each model component. Each map specifies inputs and outputs required by a component. Input files of each component are organized into different '''sections'''. * '''[!UserChoices]''' contains specific options. --> used by the component's drivers (e.g.: lmdz.driver) * '''[!InitialStateFiles]''' '''Initial conditions''' files such as vegetation maps, topography,... --> retrieved by the IGCM_comp_GetInputInitialStateFiles function * '''[!BoundaryFiles]''' '''Boundary conditions''' files such as forcings or a LAI --> retrieved by the IGCM_comp_GetInputBoundaryFiles function * '''[!SmoothFiles]''' '''Time-varying boundary conditions''' files such as aerosols --> retrieved by the IGCM_comp_GetInputSmoothFiles function * '''[!ParametersFiles]''' '''Parameters''' files such as namelist or the run.def file --> retrieved by the IGCM_comp_GetInputParametersFiles function * '''[!RestartFiles]''' '''Restart''' files --> retrieved by the IGCM_comp_GetInputRestartFiles function ### The [!UserChoices] section ### Contains several options which change the simulation setup by ''drivers'' files of the components (lmdz.driver, opa9.driver, ...). For example : {{{ #!sh [UserChoices] # Physics package to use : # LMDZ_Physics=AP for standard/old physics(default), can be used with LMDZ4_AR5 or LMDZ5/trunk sources # LMDZ_Physics=NPv3.1 for new physics, to be used with LMDZ5/trunk revision 1554 or later LMDZ_Physics=AP }}} See the description for LMDZ [wiki:Doc/Models/LMDZ#lmdz.card here]. ### The [!InitialStateFiles] section ### Files needed to create initial files. This section is not activated if you chose to start or continue from an existing simulation (Section [Restart] in ''config.card''). The files in this list will be only copied at the startup of your simulation. {{{ #!sh # ------------------------------------------------------------------ #D- Get initial state (Etat0, carteveg,relief...) #D- READ AND USE BY GCM FOR ONLY FOR THE FIRST EXECUTION. # ------------------------------------------------------------------ [InitialStateFiles] # IGCM_comp_GetInputInitialStateFiles from main Job List= (SOURCE, DESTINATION) }}} ### The [!BoundaryFiles] section ### The files containing the boundary conditions are copied to the working directory. The files in the '''List''' list will be copied at each integration period (one 1-month integration per period in general). A job can consist of several periods (!PeriodNb). The files in the '''!ListNonDel''' list will only be copied for the first period of each job. These files will be accessible but will not change during the simulation. {{{ #!sh # ------------------------------------------------------------------ #D- Get Boundaries Conditions (SST, WIND[X,Y,Z], LAI ...) #D- READ AND USE BY GCM AT EACH EXECUTION. # ------------------------------------------------------------------ [BoundaryFiles] # IGCM_comp_GetInputBoundaryFiles List= (SOURCE, DESTINATION) ListNonDel= (SOURCE, DESTINATION) }}} [[NoteBox(warn, Be very careful : if there is any space at the end of a line\, libIGCM will not take in account the next line in the list, 600px)]] ### The [!SmoothFiles] section ### These are also files containing boundary conditions but their retrieval is only done at specific time integrals and it is not systematic. '''1:12:''' means that the file will be copied to the working directory at the first integration step and then every 12 iterations until the simulation is finished. {{{ #!sh # ------------------------------------------------------------------ #D- Get SmoothFiles Conditions (SST, WIND[X,Y,Z], LAI ...) #D- READ AND USE BY GCM AT EACH EXECUTION but varying in time # ------------------------------------------------------------------ [SmoothFiles] # IGCM_comp_GetInputSmoothFiles List= (SOURCE, DESTINATION, FREQUENCE DE COPIE) }}} ### The [!ParametersFiles] section ### The parameter files of the component (namelist, run.def,...) {{{ #!sh # ------------------------------------------------------------------ #D- Get parameters text files updated by job (.def, namelist ...) #D- READ AND USE BY GCM AT EACH EXECUTION. # ------------------------------------------------------------------ [ParametersFiles] # IGCM_comp_GetInputParametersFiles List= (SOURCE, DESTINATION) }}} ### The [!RestartFiles] section ### The files providing the restart data. '''You must not change this section''' it is needed to link the jobs. {{{ #!sh # ------------------------------------------------------------------ #D- Get restart files (restartphy.nc, orca_restart.nc ...) #D- READ AND USE BY GCM AT EACH EXECUTION. # ------------------------------------------------------------------ [RestartFiles] # IGCM_comp_GetInputRestartFiles List= (MODEL OUTPUT NAME, ARCHIVED NAME, MODEL INPUT NAME) }}} ### The [!OutputText] section ### This section contains text files which will be produced during the simulation and model input parameter files. You might want to save these files. {{{ #!sh [OutputText] List= (NAME OF TEXT1 FILE, NAME OF TEXT2 FILE ....) }}} These files will be saved in tar stored in the output directory * TGCC : $CCCSTOREDIR/IGCM_OUT/!TagName/[!SpaceName]/[!ExperimentName]/!JobName/DEBUG * IDRIS : $STORE/IGCM_OUT/!TagName/[!SpaceName]/[!ExperimentName]/!JobName/DEBUG ### The [!OutputFiles] section ### The netcdf files produced by the simulation are listed in this paragraph. This paragraph is associated with the [Post_***] sections. {{{ #!sh [OutputFiles] List = (OUTPUT_FILE_NAME, SAVE_PATH, POSSIBLE ASSOCIATED POST PROCESSING) }}} Refer to this [wiki:Doc/Running#TimeSeries chapter] to learn everything about this section. ## DRIVER directory ## This directory contains the different ''drivers'' (predefined libIGCM functions for the component) of the different configuration's components. These drivers modify the parameter files of each component (*.def, namelist, ...) setting the integration times, the outputs, and the forcing files. '''Note''' : If this directory does not exist the ''driver'' files are located in the COMP directory. ## PARAM directory ## This directory contains input text files for the configuration's components. ## POST directory ## This directory contains configuration files for additional diagnostic output. Click [wiki:Doc/Running#Addingavariabletothemonitoring here] for more details. ---- # Set up initial state for the simulation # [[NoteBox(note,When you setup a simulation make sure that the list of input files in each card file of the model components and the selected options correspond to your experiment., 600px)]] There are three different ways to define your simulation's initial conditions: * [#TheRestartssection Start using restart files from an existing simulation] by setting ''!OverRule=y'' in the [Restart] section of the config.card file * [#TheATM...sectionsofthemodelcomponents Start using different restart files from different simulations for each model component] by setting ''Restart=y'' in each associated part of the ''config.card'' file. ''!OverRule=n'' must be set in ''config.card''. * [#COMPdirectory Use the default section InitialStateFiles in the comp.card file of the model components] in ''config.card'' you must have ''!OverRule=n'' and and ''Restart=n'' for this case. ## Example for different restart ## ### Example with !OverRule=y ### If you wish to use the start state of a given simulation, set in ''config.card'': {{{ #======================================================================== #D-- Restarts - [Restarts] #D- If you want a GENERAL RULE FOR ALL COMPONENTS RESTARTS, put this flag to 'y' OverRule=y #D- Last day of the experience used as restart RestartDate=1869-12-30 #D- Define restart simulation name RestartJobName=CD1 #D- Path Server Group Login RestartPath=${ARCHIVE}/IGCM_OUT/IPSLCM5A/DEVT/pdControl }}} [[BR]] For the same case but if the simulation was performed by someone else, you must give the complete path of the directory, for example: {{{ #!sh RestartPath=/u/rech/lab/plabxxx/IGCM_OUT/IPSLCM5A/DEVT/pdControl }}} [[BR]] ### Example with !OverRule=n and [COMP]/Restart=y ### You can also distinguish the setup parameters for each model components. Set `OverRule=n` and use the `Restart`, `RestartDate`, `RestartJobName` and `RestartPath` parameters for each model component (section). For example, use restart files for the atmosphere but not for the surface component. For the surface component the !InitialStateFiles will then be used : {{{ #D-- ATM - [ATM] # WriteFrequency="1M 1D HF" # If config_Restarts_OverRule == 'n' all params are read Restart= y # Last day of the experience used as restart for this component RestartDate=1999-12-30 # Define restart simulation name RestartJobName=2L18 RestartPath=${ARCHIVE}/IGCM_OUT/IPSLCM5A/DEVT/pdControl # #D-- SRF - [SRF] # WriteFrequency="1M" # If config_Restarts_OverRule == 'n' all params are read Restart= n # Last day of the experience used as restart for this component RestartDate=1999-12-30 # Define restart simulation name RestartJobName=2L18 RestartPath=${ARCHIVE}/IGCM_OUT/IPSLCM5A/DEVT/pdControl }}} ## Note for LMDZ using v5 configurations ## [[NoteBox(note,To obtain exactly the same outputs in different simulations\, you must choose the same LMDZ `Bands` files. This is explained in `COMP/lmdz.card` with the `LMDZ_NbPeriod_adjust` and `LMDZ_Bands_file_name` parameters. In the v6 configurations this is no problem as adjust is never activated and the Bands file is not needed.,600px)]] {{{ #!sh LMDZ_NbPeriod_adjust=0 # To force the use of this Bands file, set LMDZ_NbPeriod_adjust=0 and replace XXXXXXX by Restart Job Name LMDZ_Bands_file_name=${ARCHIVE}/IGCM_OUT/IPSLCM5/CEPRO0/ATM/Debug/CEPRO0_Bands_96x95x39_3prc.dat_3 }}} Click [wiki:Doc/Models/LMDZ#ParallelismandtheBandsfile here] for more details. ---- # Main job of the simulation # The main job contains scripts that will be executed by the system. With libIGCM, this job is unique (with the name '''`Job_MYJOBNAME`''') for all type of configurations. It contains all scripts to initialize a simulation, to summarize the chosen model configuration and to run identical experiments for all model components. It resubmits itself in order to continue the simulation if needed. The job header depends of the machine type. It contains the job name and the parameters. At TGCC you must specify the project number: [wiki:Doc/ComputingCenters/TGCC#Projectandcomputingneeds #MSUB -A MY_PROJECT]. You should change the `!PeriodNb` parameter in the job to change the number of runs in one job (see the example of computation in the next section) : {{{ #D- Number of execution in one job PeriodNb=10 }}} A temporary run directory will be created for the execution of the job. This directory is always removed after successful run but when the job run fails it depends on the system if the directory is kept or not. Therefore you can change the default location by setting RUN_DIR_PATH variable as you like. This is very useful for debugging. {{{ #D- Define running directory #D- Default=${TMPDIR} ie temporary batch directory #RUN_DIR_PATH=/workdir/or/scratchdir/of/this/machine }}} Here is the diagram of the steps in the template script (''AA_job'') that will be use to create the job file: [[Image(AA_job.jpg, 50%)]] ## Choosing !PeriodNb ## To avoid starting a lot of short jobs which might be queued, the production job starts n integrations ''(!PeriodNb)'', whose length are [#PeriodLength PeriodLength.] These are calculated as followed: [[BR]] Time limit = !PeriodNb * max(Real time of a !PeriodLength) where ''Time limit'' is the requested time in the job header. [[BR]] At the end of a simulation, the run.card file returns the used CPU time for each simulation step. This will allow you to perform this computation. It is therefore important, for each simulation with a new configuration, to perform a 1-3 month test to estimate beforehand the CPU time. ---- # Prepare a new experiment # There are two ways to prepare a new working directory for your model configuration: 1. Start again from the first step described above by copying the desired ''config.card'' file to your configuration directory using a new !JobName. 2. Copy an existing submission directory, delete the files created by the [wiki:Doc/Running#Endofthesimulation simulation], and change !JobName in ''config.card''. [[BR]]For example: {{{ #!sh cd modipsl/config/LMDZOR_v6 cp -r MyJobTest CHOUCROUTE cd CHOUCROUTE rm -f Job_MyJobTest run.card Script_Output_MyJobTest.000001 vi config.card JobName=CHOUCROUTE ../../../libIGCM/ins_job # Check and complete job's header }}} [[NoteBox(note, The ins_job script allows you to create a submission directory from a ''config.card'' file or if the directory already exists it allows you to only create the job corresponding to ''config.card''. '''`ins_job`''' will not overwrite a directory or an existing job.,600px)]] ## Post-processing jobs ## Jobs headers for post-processing have to be carefully checked, especially elapsed time limits. They are in libIGCM directory (xxx.job) and are adapted for IPSLCM5A with 1Y for !RebuildFrequency and !PackFrequency. Change time limits if you use larger frequencies. ---- # Prepare ensembles with ins_job -e # For details about how to generate/use ensemble simulation, please go to its [wiki:Doc/Setup/Ensemble specific page].