source: trunk/libIGCM/AA_job @ 252

Last change on this file since 252 was 252, checked in by sdipsl, 14 years ago
  • Bugfix in AA_monitoring ; when using cfg files from POST directory the filename is monitoring01_${compname}.cfg
  • Add back an echo in AA_job to keep track of the model execution in the Script_Output
  • Property licence set to
    The following licence information concerns ONLY the libIGCM tools
    ==================================================================

    Copyright © Centre National de la Recherche Scientifique CNRS
    Commissariat à l'Énergie Atomique CEA

    libIGCM : Library for Portable Models Computation of IGCM Group.

    IGCM Group is the french IPSL Global Climate Model Group.

    This library is a set of shell scripts and functions whose purpose is
    the management of the initialization, the launch, the transfer of
    output files, the post-processing and the monitoring of datas produce
    by any numerical program on any plateforme.

    This software is governed by the CeCILL license under French law and
    abiding by the rules of distribution of free software. You can use,
    modify and/ or redistribute the software under the terms of the CeCILL
    license as circulated by CEA, CNRS and INRIA at the following URL
    "http://www.cecill.info".

    As a counterpart to the access to the source code and rights to copy,
    modify and redistribute granted by the license, users are provided only
    with a limited warranty and the software's author, the holder of the
    economic rights, and the successive licensors have only limited
    liability.

    In this respect, the user's attention is drawn to the risks associated
    with loading, using, modifying and/or developing or reproducing the
    software by the user in light of its specific status of free software,
    that may mean that it is complicated to manipulate, and that also
    therefore means that it is reserved for developers and experienced
    professionals having in-depth computer knowledge. Users are therefore
    encouraged to load and test the software's suitability as regards their
    requirements in conditions enabling the security of their systems and/or
    data to be ensured and, more generally, to use and operate it in the
    same conditions as regards security.

    The fact that you are presently reading this means that you have had
    knowledge of the CeCILL license and that you accept its terms.
  • Property svn:keywords set to Date Author Revision
File size: 19.4 KB
Line 
1#-Q- platine #!/usr/bin/ksh
2#-Q- platine ##################
3#-Q- platine ## PLATINE   CEA ##
4#-Q- platine ##################
5#-Q- platine #BSUB -J ::Jobname::       # nom de la requete
6#-Q- platine #BSUB -N                   # message a la fin du job
7#-Q- platine #BSUB -o Script_Output_::Jobname::.000001    # nom du fichier de sortie
8#-Q- platine #BSUB -n ::JobNumProcTot::             # reservation des processeurs pour le job
9#-Q- platine #BSUB -W 1:00           # Limite temps
10#-Q- platine BATCH_NUM_PROC_TOT=$SLURM_NPROCS
11#-Q- titane #!/usr/bin/ksh
12#-Q- titane ##################
13#-Q- titane ## TITANE   CEA ##
14#-Q- titane ##################
15#-Q- titane #MSUB -r ::Jobname::       # nom de la requete
16#-Q- titane #MSUB -o Script_Output_::Jobname::.000001    # nom du fichier de sortie
17#-Q- titane #MSUB -n ::JobNumProcTot::             # reservation des processeurs pour le job
18#-Q- titane #MSUB -T 86400           # Limite temps (en secondes)
19#-Q- titane #MSUB -p gen2211
20#-Q- titane BATCH_NUM_PROC_TOT=$BRIDGE_MSUB_NPROC
21#-Q- sx8brodie #######################
22#-Q- sx8brodie ## SX8BRODIE   IDRIS ##
23#-Q- sx8brodie #######################
24#-Q- sx8brodie #PBS -N ::Jobname::         # nom de la requete
25#-Q- sx8brodie #PBS -m a                   # message a la fin du job
26#-Q- sx8brodie #PBS -j o                   # regroupement des sorties du job : standard error et output
27#-Q- sx8brodie #PBS -o Script_Output_::Jobname::.000001  # nom du fichier de sortie
28#-Q- sx8brodie #PBS -S /usr/bin/ksh        # shell de soumission
29#-Q- sx8brodie #PBS -v BATCH_NUM_PROC_TOT=::JobNumProcTot::
30#-Q- sx8brodie #PBS -q ::JobClass::                # classe multi pour BATCH_NUM_PROC_TOT superieur a 2, mono sinon
31#-Q- sx8brodie #PBS -l cpunum_job=${BATCH_NUM_PROC_TOT}  # reservation des processeurs pour le job
32#-Q- sx8brodie #PBS -l memsz_job=15.0gb     # Limite memoire
33#-Q- sx8brodie #PBS -l cputim_job=1:00:00  # Limite temps
34#-Q- aix6 #!/usr/bin/ksh
35#-Q- aix6 # ######################
36#-Q- aix6 # ##   VARGAS IDRIS   ##
37#-Q- aix6 # ######################
38#-Q- aix6 # Nom de la requete
39#-Q- aix6 # @ job_name = ::Jobname::
40#-Q- aix6 # Type de travail
41#-Q- aix6 # @ job_type = parallel
42#-Q- aix6 # Fichier de sortie standard
43#-Q- aix6 # @ output = $(job_name).$(jobid)
44#-Q- aix6 # Fichier de sortie erreur (le meme)
45#-Q- aix6 # @ error =  $(job_name).$(jobid)
46#-Q- aix6 # Nombre de processus demandes
47#-Q- aix6 # @ total_tasks = ::JobNumProcTot::
48#-Q- aix6 # @ environment = "BATCH_NUM_PROC_TOT=::JobNumProcTot::"
49#-Q- aix6 # Temps CPU max. par processus MPI hh:mm:ss
50#-Q- aix6 # @ wall_clock_limit = 1:00:00
51#-Q- aix6 # Memoire max. utilisee par processus
52#-Q- aix6 # @ data_limit = 3.2gb
53#-Q- aix6 # Memoire stack demandee
54#-Q- aix6 # @ stack_limit = 0.3gb,0.3gb
55#-Q- aix6 # Pas d OpenMP
56#-Q- aix6 # @ resources = ConsumableCpus(1)
57#-Q- aix6 # Fin de l entete
58#-Q- aix6 # @ queue
59#-Q- sx8mercure #!/bin/ksh
60#-Q- sx8mercure ######################
61#-Q- sx8mercure ## SX8MERCURE   CEA ##
62#-Q- sx8mercure ######################
63#-Q- sx8mercure #PBS -N ::Jobname::           # nom de la requete
64#-Q- sx8mercure #PBS -m a                     # message a la fin du job
65#-Q- sx8mercure #PBS -j o                     # regroupement des sorties du job : standard error et output
66#-Q- sx8mercure ####PBS -A gen6178
67#-Q- sx8mercure #PBS -o Script_Output_::Jobname::.000001   # nom du fichier de sortie
68#-Q- sx8mercure #PBS -S /usr/bin/ksh          # shell de soumission
69#-Q- sx8mercure #PBS -v BATCH_NUM_PROC_TOT=::JobNumProcTot::
70#-Q- sx8mercure #PBS -l cpunum_job=${BATCH_NUM_PROC_TOT}          # reservation des processeurs pour le job
71#-Q- sx8mercure #PBS -l memsz_job=15.0gb       # limite memoire
72#-Q- sx8mercure #PBS -l elapstim_req=01:00:00 # limite en temps elapsed
73#-Q- sx9mercure #!/bin/ksh
74#-Q- sx9mercure ######################
75#-Q- sx9mercure ## SX9MERCURE   CEA ##
76#-Q- sx9mercure ######################
77#-Q- sx9mercure #PBS -N ::Jobname::           # nom de la requete
78#-Q- sx9mercure #PBS -m a                     # message a la fin du job
79#-Q- sx9mercure #PBS -j o                     # regroupement des sorties du job : standard error et output
80#-Q- sx9mercure #PBS -o Script_Output_::Jobname::.000001   # nom du fichier de sortie
81#-Q- sx9mercure #PBS -S /usr/bin/ksh          # shell de soumission
82#-Q- sx9mercure #PBS -v BATCH_NUM_PROC_TOT=::JobNumProcTot::
83#-Q- sx9mercure #PBS -l cpunum_job=${BATCH_NUM_PROC_TOT}          # reservation des processeurs pour le job
84#-Q- sx9mercure #PBS -l memsz_job=16.0gb       # limite memoire
85#-Q- sx9mercure #PBS -l elapstim_req=24:00:00 # limite en temps elapsed
86#-Q- sx9mercure #PBS -A gen6178               # groupe obligatoire sur SX9
87#-Q- default #!/bin/ksh
88#-Q- default ##################
89#-Q- default ## DEFAULT HOST ##
90#-Q- default ##################
91#-Q- default #For MPI use, uncomment next line :
92#-Q- default #BATCH_NUM_PROC_TOT=::JobNumProcTot::
93
94# $Date$
95# $Author$
96# $Revision$
97# IPSL (2006)
98#  This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
99
100#set -eu
101#set -vx
102
103date
104echo
105echo "#######################################"
106echo "#       ANOTHER GREAT SIMULATION      #"
107echo "#######################################"
108echo
109
110#D--------------------------------------------------------------------==
111#D--------------------------------------------------------------------==
112#D-
113#D-                      Job to launch IGCM models
114#D-
115#D--------------------------------------------------------------------==
116#D--------------------------------------------------------------------==
117#D-
118
119MODIPSL=::modipsl::
120libIGCM=${MODIPSL}/libIGCM
121
122#D--------------------------------------------------------------------==
123#D-                   -1. User modification place
124#D-                      - Job Verbosity
125#D-                      - PeriodNb
126#D-                      - Experience type : DEB(ug), DEV(elopment), RUN
127#D--------------------------------------------------------------------==
128
129#D- Increased verbosity (1, 2, 3)
130Verbosity=3
131#D- Number of execution in one job
132PeriodNb=1
133#-Q- sx9mercure PeriodNb=60
134
135#D-
136# --------------------------------------------------------------------
137#D- ! Optionnal features /!\ Use with care !
138# --------------------------------------------------------------------
139
140#D- Low level debug : to bypass lib test checks and stack construction
141#D- Default=true
142DEBUG_debug=false
143
144#D- Set DEBUG_sys to false to disable output calls of function
145#D- Default=true
146#DEBUG_sys=false
147
148#D- Experience type : DEB(ug), DEV(elopment), RUN (default)
149JobType=RUN
150
151#D- Define running directory
152#D- Default=${TMPDIR} ie temporary batch directory
153#RUN_DIR_PATH=/workdir/or/scratchdir/of/this/machine
154
155#D- Define submit directory
156#D- Default= where you launch qsub (variable from scheduler)
157#SUBMIT_DIR=$( pwd )
158
159#D- Define BIG_DIR directory
160#D- Default=(strongly machine dependant/see libIGCM_sys for details)
161#BIG_DIR=/somewhere/with/lot/of/space/scratchdir/may/be
162
163#D- Turn in dry run mode ? (0,1,2,3)
164#D- Default=0
165#DRYRUN=3
166# YOU MUST COMPILE YOUR EXE FILES FOR DRYRUN MODE !
167# -------------------------------------------------------------------------------------
168# | DRYRUN=  |  Date computations, | sys_Get    |  Exe    | sys_Put_Out; sys_Put_Rest |
169# |          |  Cp/Exe param files |            |  Chmod  |                           |
170# |          |      Qsub           |            |         |                           |
171# -------------------------------------------------------------------------------------
172# |    0     |       yes           |    yes     |  yes    |      yes                  |
173# -------------------------------------------------------------------------------------
174# |    1     |       yes           |    yes     |  yes    |      no                   |
175# -------------------------------------------------------------------------------------
176# |    2     |       yes           |    yes     |  no     |      no                   |
177# -------------------------------------------------------------------------------------
178# |    3     |       yes           |    no      |  no     |      no                   |
179# -------------------------------------------------------------------------------------
180
181#D- Define input file root directory
182#D- Default=/IGCMG/common/account/of/this/machine
183#R_IN=/u/rech/por/rpor111/DATA
184
185#D-
186#D--------------------------------------------------------------------==
187#D-                   0. System Environment
188#D-                      - Define variables for Number of processors
189#D-                      - Define MPI variables
190#D-                      - Define batch scheduler variables
191#D-                      - Source IGCM Library
192#D-                      - Get RESOLution in .resol file (temporary)
193#D--------------------------------------------------------------------==
194
195#D--------------------------------------------------------------------==
196
197. ${libIGCM}/libIGCM_debug/libIGCM_debug.ksh
198       ( ${DEBUG_debug} ) && IGCM_debug_Check
199. ${libIGCM}/libIGCM_card/libIGCM_card.ksh
200       ( ${DEBUG_debug} ) && IGCM_card_Check
201. ${libIGCM}/libIGCM_date/libIGCM_date.ksh
202       ( ${DEBUG_debug} ) && IGCM_date_Check
203#-------
204. ${libIGCM}/libIGCM_sys/libIGCM_sys.ksh
205. ${libIGCM}/libIGCM_config/libIGCM_config.ksh
206. ${libIGCM}/libIGCM_comp/libIGCM_comp.ksh
207. ${libIGCM}/libIGCM_post/libIGCM_post.ksh
208
209#D--------------------------------------------------------------------==
210
211#-- Resolution FLAG for oasis input
212[ -f ${SUBMIT_DIR}/../.resol ] && RESOL=$(head -1 ${SUBMIT_DIR}/../.resol)
213#RESOL="ORCA2xLMD9671"
214
215#D-
216#D--------------------------------------------------------------------==
217#D-           1. INITIALIZE CONFIGURATION AND THEN COMPONENTS
218#D--------------------------------------------------------------------==
219
220# --------------------------------------------------------------------==
221#D- Initialize configuration :
222#D-              - Simulation configuration
223#D-              - Simulation parameters
224#D-              - Execution parameter
225#D-              - Define input files directory
226#D-              - Read or initialize CumulPeriod
227#D-              - run.card
228#D-
229# --------------------------------------------------------------------==
230IGCM_config_Initialize
231
232# --------------------------------------------------------------------==
233#D- Define, create and cd RUN_DIR
234#D- 
235# --------------------------------------------------------------------==
236RUN_DIR=${RUN_DIR_PATH}/${config_UserChoices_TagName}/${config_UserChoices_JobName}.${$}
237IGCM_sys_MkdirWork ${RUN_DIR}
238IGCM_sys_Cd ${RUN_DIR}
239
240# --------------------------------------------------------------------==
241#D- Initialize each components taking part
242#D- into simulation configuration :
243#D-         - Define component executable
244#D-         - Define Write Frequency by component "1M" "1D" ...
245#D-         - Source ${comp}.driver
246#D-         - Define Storage Places by component
247#D-         - ${comp}_Initialize :
248#D-         - Patterns to sed
249#D-         - Variables from namelist
250#D-
251# --------------------------------------------------------------------==
252IGCM_comp_Initialize
253
254# --------------------------------------------------------------------==
255#D- Verify compatibility of period length, write frequencies, ...
256#D- Verify compatibility of rebuild choices and post-process choices
257#D- Will stop here if something is wrong
258# --------------------------------------------------------------------==
259#IGCM_config_Check
260
261#D--------------------------------------------------------------------==
262#D-                  2. ENTER THE DEEP EXECUTION LOOP
263#D-                  !!!! FASTEN YOUR SEAT BELTS !!!!
264#D-                  !!!!!      PLEASE.          !!!!
265#D-                  !! WE WILL CROSS TURBULENCES. !!
266#D--------------------------------------------------------------------==
267
268Period=1
269
270PeriodContinue=false
271if [ ${Period} -le ${PeriodNb} ]; then
272    PeriodContinue=true
273fi
274while ( ${PeriodContinue} ) ; do
275
276    echo
277    echo "Starting iteration ${Period} / ${PeriodNb}"
278
279    #D-
280    # ------------------------------------------------------------------
281    #D- COMPUTE AND DEFINE DATE STUFF.
282    # ------------------------------------------------------------------
283    IGCM_config_PeriodStart
284    #D- => ${PeriodDateBegin}
285    #D- => ${PeriodDateEnd}
286    #D- => ${CumulPeriod}
287    #D- => ${DatesPeriod}=${PeriodDateBegin}_${PeriodDateEnd}
288    #D- => ${PeriodLength} => JOURS=31 , 28, 29, 30 ; MOIS=0 ; ANS=0
289    #D- => update run.card value
290   
291    #D-
292    # ------------------------------------------------------------------
293    #D- Get parameters text files updated by job (.def, namelist ...)
294    #D- READ AND USE BY GCM AT EACH EXECUTION.
295    # ------------------------------------------------------------------
296    IGCM_comp_GetInputParametersFiles
297
298    #D-
299    # ------------------------------------------------------------------
300    #D- Get initial state (Etat0, carteveg,relief...)
301    #D- NECESSARY ONLY IF CumulPeriod= (=> NUMERO) == 1
302    #D- AND ???_Restart=NO
303    #D- READ AND USE BY GCM FOR ONLY ONE EXECUTION.
304    # ------------------------------------------------------------------
305    IGCM_comp_GetInputInitialStateFiles
306
307    #D-
308    # ------------------------------------------------------------------
309    #D- Get Boundaries Conditions (SST, WIND[X,Y,Z], LAI ...)
310    #D- READ AND USE BY GCM AT EACH EXECUTION.
311    # ------------------------------------------------------------------
312    IGCM_comp_GetInputBoundaryFiles
313
314    #D-
315    # ------------------------------------------------------------------
316    #D- Get restart files (restartphy.nc, orca_restart.nc ...)
317    #D- READ AND USE BY GCM AT EACH EXECUTION.
318    #D-
319    #D- IF A COMPONENT DO NOT RESTART FROM PREVIOULSY COMPUTED RESTART
320    #D- ONLY IF CumulPeriod= (=> NUMERO) == 1
321    #D- MUST EXECUTE CREATE ETAT0_LIMIT (TYPICALLY LMDZ AND ./create_etat0_limit.e)
322    #D-
323    #D- IF CumulPeriod == 1
324    #D- AND A COMPONENT RESTART FROM PREVIOULSY COMPUTED RESTART
325    #D- MUST BRING THIS RESTART IN ${RUN_DIR_PATH}
326    #D-
327    #D- ELSE BRING RESTART FROM ${JobName} IN ${RUN_DIR_PATH}
328    # ------------------------------------------------------------------
329    IGCM_comp_GetInputRestartFiles
330   
331    #D-
332    # ------------------------------------------------------------------
333    #D- Test if all was right before Update
334    # ------------------------------------------------------------------
335    IGCM_debug_Verif_Exit
336
337    # ------------------------------------------------------------------
338    #D- Activate running environnement variables
339    # ------------------------------------------------------------------
340    IGCM_sys_activ_variables
341
342    #D-
343    # ------------------------------------------------------------------
344    #D- UPDATE ParametersFiles (.def, namelist ...) with
345    #D- current value of (ORCA_NIT00, ORCA_LRSTAR, RAZ_DATE, ...)
346    # ------------------------------------------------------------------
347    IGCM_comp_Update
348
349    #D-
350    # ------------------------------------------------------------------
351    #D- Test if all was right before execution
352    # ------------------------------------------------------------------
353    IGCM_debug_Verif_Exit
354
355    #D-
356    #D----------------------------------------------------------------==
357    #D-                          3. Execution
358    #D----------------------------------------------------------------==
359
360    echo
361    echo "#######################################"
362    echo "#      DIR BEFORE RUN EXECUTION       #"
363    echo "#######################################"
364    echo
365    ls -lrt
366
367    echo "========================================================================"
368    if ( [ X${BATCH_NUM_PROC_TOT} != X ] && [ "${BATCH_NUM_PROC_TOT}" -gt 1 ] ) ; then
369        MPIRUN_COMMAND=${HOST_MPIRUN_COMMAND}
370        eval MPIRUN_OPTIONS=${config_UserChoices_JobRunOptions}
371    else
372        MPIRUN_COMMAND=${MPIRUN_COMMAND:="time "}
373        MPIRUN_OPTIONS=${MPIRUN_OPTIONS:=""}
374    fi
375
376    if [ ${DRYRUN} -le 1 ] ; then
377        REAL_DATE_INIT=$( date )
378        echo "EXECUTION of : ${MPIRUN_COMMAND} ${MPIRUN_OPTIONS} ./${config_Executable_Name}   > ${Exe_Output} 2>&1"
379        echo                                                                                   > ${Exe_Output}
380        echo "#######################################"                                        >> ${Exe_Output}
381        echo "EXECUTION of : ${MPIRUN_COMMAND} ${MPIRUN_OPTIONS} ./${config_Executable_Name}" >> ${Exe_Output}
382        echo                                                                                  >> ${Exe_Output}
383        typeset RET
384        RUN_DATE_BEGIN=$( date '+%Y-%m-%dT%H:%M:%S' )
385        ${MPIRUN_COMMAND} ${MPIRUN_OPTIONS} ./${config_Executable_Name} >> ${Exe_Output} 2>&1
386        RET=$?
387        RUN_DATE_END=$( date '+%Y-%m-%dT%H:%M:%S' )
388        if [ ${RET} -gt 0 ] ; then
389            echo "Return code of executable :" ${RET}
390            IGCM_debug_Exit "EXECUTABLE"
391            IGCM_sys_Cp ${Exe_Output} ${SUBMIT_DIR}/${PREFIX}_${Exe_Output}_error
392        fi
393        echo                                            >> ${Exe_Output}
394        echo "#######################################"  >> ${Exe_Output}
395        echo "libIGCM RunDateBegin=${RUN_DATE_BEGIN}"   >> ${Exe_Output}
396        echo "libIGCM RunDateEnd=${RUN_DATE_END}"       >> ${Exe_Output}
397        echo                                            >> ${Exe_Output}
398    else
399        echo "EXECUTION of : ${MPIRUN_COMMAND} ${MPIRUN_OPTIONS} ./${config_Executable_Name} simulated for DRYRUN = " $DRYRUN
400        if ( $DEBUG_debug ) ; then
401            echo "FOR EXECUTION DRYRUN mode = " $DRYRUN >> stack
402        fi
403    fi
404    echo "========================================================================"
405
406    echo
407    echo "#######################################"
408    echo "#       DIR AFTER RUN EXECUTION       #"
409    echo "#######################################"
410    echo
411    ls -lrt
412
413    #D-
414    # ------------------------------------------------------------------
415    #D- Desactivate running environnement variables
416    # ------------------------------------------------------------------
417    IGCM_sys_desactiv_variables
418
419    #D-
420    #D----------------------------------------------------------------==
421    #D-                    4. Post execution operations
422    #D----------------------------------------------------------------==
423
424    #D-
425    # ------------------------------------------------------------------
426    #D- Save on archive machine netcdf outputs, restarts
427    #D- and text files of models
428    # ------------------------------------------------------------------
429    IGCM_comp_Finalize
430
431    #D-
432    # ------------------------------------------------------------------
433    #D- Test if all was right after Finalize
434    # ------------------------------------------------------------------
435    IGCM_debug_Verif_Exit
436
437    #D-
438    # ------------------------------------------------------------------
439    #D- Launch remote rebuild or/and post-treatment process if necessary
440    # ------------------------------------------------------------------
441    IGCM_post_Configure
442
443    #D-
444    # ------------------------------------------------------------------
445    #D- Save job output
446    #D- Manage executable size to save only different binary
447    #D- Write in run.card user, system and elapse time
448    #D- Check that everything went well ! No ? then we stop.
449    #D- Determine next computed period
450    # ------------------------------------------------------------------
451    IGCM_config_PeriodEnd
452
453    echo "Ending iteration ${Period}"
454    (( Period = Period + 1 ))
455
456    # End loop if date end is reached
457    if [ ${SimulationLengthInDays} -ge ${ExperienceLengthInDays} ] ; then
458        break ;
459    fi
460
461    PeriodContinue=false
462    if [ ${Period} -le ${PeriodNb} ]; then
463        PeriodContinue=true
464    fi
465done
466
467#D-
468#D--------------------------------------------------------------------==
469#D-     5. Submit next job or send mail if simulation over.
470#D--------------------------------------------------------------------==
471IGCM_config_Finalize
472
473date
Note: See TracBrowser for help on using the repository browser.