source: trunk/libIGCM/AA_job @ 804

Last change on this file since 804 was 804, checked in by sdipsl, 11 years ago
  • Files waiting for rebuild are stored within the relevant simulation tree. see #87

--> /IGCM_OUT/TagName/SpaceName/ExperiementName/JobName/REBUILD

  • Using only 1 proc and having RebuildFrequency? != NONE won't be an issue. see #88
  • Homogenize RUN_DIR PATH during computing and post-processing. see #92
  • Save text files before restarts and outputs. (more chance to have a clue of what went wrong) see #103
  • Remove useless variable BIG_DIR
  • Cosmectics
  • Property licence set to
    The following licence information concerns ONLY the libIGCM tools
    ==================================================================

    Copyright © Centre National de la Recherche Scientifique CNRS
    Commissariat à l'Énergie Atomique CEA

    libIGCM : Library for Portable Models Computation of IGCM Group.

    IGCM Group is the french IPSL Global Climate Model Group.

    This library is a set of shell scripts and functions whose purpose is
    the management of the initialization, the launch, the transfer of
    output files, the post-processing and the monitoring of datas produce
    by any numerical program on any plateforme.

    This software is governed by the CeCILL license under French law and
    abiding by the rules of distribution of free software. You can use,
    modify and/ or redistribute the software under the terms of the CeCILL
    license as circulated by CEA, CNRS and INRIA at the following URL
    "http://www.cecill.info".

    As a counterpart to the access to the source code and rights to copy,
    modify and redistribute granted by the license, users are provided only
    with a limited warranty and the software's author, the holder of the
    economic rights, and the successive licensors have only limited
    liability.

    In this respect, the user's attention is drawn to the risks associated
    with loading, using, modifying and/or developing or reproducing the
    software by the user in light of its specific status of free software,
    that may mean that it is complicated to manipulate, and that also
    therefore means that it is reserved for developers and experienced
    professionals having in-depth computer knowledge. Users are therefore
    encouraged to load and test the software's suitability as regards their
    requirements in conditions enabling the security of their systems and/or
    data to be ensured and, more generally, to use and operate it in the
    same conditions as regards security.

    The fact that you are presently reading this means that you have had
    knowledge of the CeCILL license and that you accept its terms.
  • Property svn:keywords set to Revision Author Date
File size: 20.3 KB
Line 
1#-Q- curie #!/bin/ksh
2#-Q- curie ######################
3#-Q- curie ## CURIE   TGCC/CEA ##
4#-Q- curie ######################
5#-Q- curie #MSUB -r ::Jobname::       # nom de la requete
6#-Q- curie #MSUB -o Script_Output_::Jobname::.000001    # nom du fichier de sortie
7#-Q- curie #MSUB -e Script_Output_::Jobname::.000001    # nom du fichier de sortie
8#-Q- curie #MSUB -eo
9#-Q- curie #MSUB -n ::JobNumProcTot:: # reservation des processeurs pour le job
10#-Q- curie #MSUB -T 86400             # Limite temps (en secondes)
11#-Q- curie #MSUB -q standard
12#-Q- curie ##MSUB -A gen****
13#-Q- curie BATCH_NUM_PROC_TOT=$BRIDGE_MSUB_NPROC
14#-Q- curie set +x
15#-Q- titane #!/usr/bin/ksh
16#-Q- titane ##################
17#-Q- titane ## TITANE   CEA ##
18#-Q- titane ##################
19#-Q- titane #MSUB -r ::Jobname::       # nom de la requete
20#-Q- titane #MSUB -o Script_Output_::Jobname::.000001    # nom du fichier de sortie
21#-Q- titane #MSUB -e Script_Output_::Jobname::.000001    # nom du fichier de sortie
22#-Q- titane #MSUB -eo
23#-Q- titane #MSUB -n ::JobNumProcTot:: # reservation des processeurs pour le job
24#-Q- titane #MSUB -T 86400             # Limite temps (en secondes)
25#-Q- titane #MSUB -p gen2211
26#-Q- titane BATCH_NUM_PROC_TOT=$BRIDGE_MSUB_NPROC
27#-Q- ada #!/bin/ksh
28#-Q- ada # ######################
29#-Q- ada # ##   ADA IDRIS   ##
30#-Q- ada # ######################
31#-Q- ada # Nom de la requete
32#-Q- ada # @ job_name = ::Jobname::
33#-Q- ada # Type de travail
34#-Q- ada # @ job_type = parallel
35#-Q- ada # Fichier de sortie standard
36#-Q- ada # @ output = Script_Output_::Jobname::.000001
37#-Q- ada # Fichier de sortie erreur (le meme)
38#-Q- ada # @ error = Script_Output_::Jobname::.000001
39#-Q- ada # Nombre de processus demandes
40#-Q- ada # @ total_tasks = ::JobNumProcTot::
41#-Q- ada # @ environment = "BATCH_NUM_PROC_TOT=::JobNumProcTot::"
42#-Q- ada # Temps CPU max. par processus MPI hh:mm:ss
43#-Q- ada # @ wall_clock_limit = 1:00:00
44#-Q- ada # Nombre de taches OpenMP/pthreads par processus MPI
45#-Q- ada ### @ parallel_threads = 4
46#-Q- ada # Fin de l entete
47#-Q- ada # @ queue
48#-Q- sx9mercure #!/bin/ksh
49#-Q- sx9mercure ######################
50#-Q- sx9mercure ## SX9MERCURE   CEA ##
51#-Q- sx9mercure ######################
52#-Q- sx9mercure #PBS -N ::Jobname::           # nom de la requete
53#-Q- sx9mercure #PBS -m a                     # message a la fin du job
54#-Q- sx9mercure #PBS -j o                     # regroupement des sorties du job : standard error et output
55#-Q- sx9mercure #PBS -o Script_Output_::Jobname::.000001   # nom du fichier de sortie
56#-Q- sx9mercure #PBS -S /usr/bin/ksh          # shell de soumission
57#-Q- sx9mercure #PBS -v BATCH_NUM_PROC_TOT=::JobNumProcTot::
58#-Q- sx9mercure #PBS -l cpunum_job=${BATCH_NUM_PROC_TOT}          # reservation des processeurs pour le job
59#-Q- sx9mercure #PBS -l memsz_job=16.0gb      # limite memoire
60#-Q- sx9mercure #PBS -l elapstim_req=24:00:00 # limite en temps elapsed
61#-Q- sx9mercure #PBS -A gen6178               # groupe obligatoire sur SX9
62#-Q- sx9mercure #PBS -q para-sx9              # para-sx9 or prio-sx9 (for CMIP5 top priorities), bckg-sx9 (run with free slot) and test-sx9 (if elapsed<1h)
63#-Q- lxiv8 ######################
64#-Q- lxiv8 ## OBELIX      LSCE ##
65#-Q- lxiv8 ######################
66#-Q- lxiv8 #PBS -N ::Jobname::
67#-Q- lxiv8 #PBS -m a
68#-Q- lxiv8 #PBS -j oe
69#-Q- lxiv8 #PBS -q medium
70#-Q- lxiv8 #PBS -o Script_Output_::Jobname::.000001
71#-Q- lxiv8 #PBS -S /bin/ksh
72#-Q- lxiv8 #PBS -v BATCH_NUM_PROC_TOT=::JobNumProcTot::
73#-Q- lxiv8 #PBS -l nodes=1:ppn=::JobNumProcTot::
74#-Q- default #!/bin/ksh
75#-Q- default ##################
76#-Q- default ## DEFAULT HOST ##
77#-Q- default ##################
78#-Q- default #For MPI use, uncomment next line :
79#-Q- default #BATCH_NUM_PROC_TOT=::JobNumProcTot::
80
81#**************************************************************
82# Author: Sebastien Denvil
83# Contact: Sebastien.Denvil__at__ipsl.jussieu.fr
84# $Revision::                                          $ Revision of last commit
85# $Author::                                            $ Author of last commit
86# $Date::                                              $ Date of last commit
87# IPSL (2006)
88#  This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
89#
90#**************************************************************
91
92#set -eu
93#set -vx
94
95date
96echo
97echo "#######################################"
98echo "#       ANOTHER GREAT SIMULATION      #"
99echo "#######################################"
100echo
101
102#D--------------------------------------------------------------------==
103#D--------------------------------------------------------------------==
104#D-
105#D-                      Job to launch IGCM models
106#D-
107#D--------------------------------------------------------------------==
108#D--------------------------------------------------------------------==
109#D-
110
111MODIPSL=::modipsl::
112libIGCM=${MODIPSL}/libIGCM
113
114#D- Task type (computing or post-processing)
115TaskType=computing
116
117#D--------------------------------------------------------------------==
118#D-                   -1. User modification place
119#D-                      - Job Verbosity
120#D-                      - PeriodNb
121#D-                      - Experience type : DEB(ug), DEV(elopment), RUN
122#D--------------------------------------------------------------------==
123
124#D- Increased verbosity (1, 2, 3)
125Verbosity=3
126#D- Experience type : DEB(ug), DEV(elopment), RUN (default)
127JobType=RUN
128#D- Number of execution in one job
129PeriodNb=1
130#-Q- sx9mercure PeriodNb=60
131
132#D-
133# --------------------------------------------------------------------------------------------------#
134#D-                          ! Optionnal features /!\ Use with care !                               #
135# --------------------------------------------------------------------------------------------------#
136
137
138#D- Low level debug : to bypass lib test checks and stack construction
139#D- Default=true
140DEBUG_debug=false
141
142#D- Set DEBUG_sys to false to disable output calls of function
143#D- Default=true
144#DEBUG_sys=false
145
146#D- Define running directory
147#D- Default=${TMPDIR} ie temporary batch directory
148#RUN_DIR_PATH=/workdir/or/scratchdir/of/this/machine
149
150#D- Define submit directory
151#D- Default= where you launch qsub (variable from scheduler)
152#SUBMIT_DIR=$( pwd )
153
154#D- Define input file root directory
155#D- Default=/IGCMG/common/account/of/this/machine
156#R_IN=/u/rech/por/rpor111/DATA
157
158#D- Turn in dry run mode ? (0,1,2,3)
159#D- Default=0
160#DRYRUN=3
161# YOU HAVE TO COMPILE YOUR EXE FILES to USE DRYRUN MODE !
162# -------------------------------------------------------------------------------------
163# | DRYRUN=  |  Date computations, | sys_Get    |  Exe    | sys_Put_Out; sys_Put_Rest |
164# |          |  Cp/Exe param files |            |  Chmod  |                           |
165# -------------------------------------------------------------------------------------
166# |    0     |       yes           |    yes     |  yes    |      yes                  |
167# -------------------------------------------------------------------------------------
168# |    1     |       yes           |    yes     |  yes    |      no                   |
169# -------------------------------------------------------------------------------------
170# |    2     |       yes           |    yes     |  no     |      no                   |
171# -------------------------------------------------------------------------------------
172# |    3     |       yes           |    no      |  no     |      no                   |
173# -------------------------------------------------------------------------------------
174
175#D-
176# --------------------------------------------------------------------------------------------------#
177#D- ! End of optionnal features /!\ Do not modify anything below unless you know what you are doing #
178# --------------------------------------------------------------------------------------------------#
179
180#D-
181#D--------------------------------------------------------------------==
182#D-                   0. System Environment
183#D-                      - Define variables for Number of processors
184#D-                      - Define MPI variables
185#D-                      - Define batch scheduler variables
186#D-                      - Source IGCM Library
187#D-                      - Get RESOLution in .resol file (temporary)
188#D--------------------------------------------------------------------==
189
190#D--------------------------------------------------------------------==
191
192. ${libIGCM}/libIGCM_debug/libIGCM_debug.ksh
193       ( ${DEBUG_debug} ) && IGCM_debug_Check
194. ${libIGCM}/libIGCM_card/libIGCM_card.ksh
195       ( ${DEBUG_debug} ) && IGCM_card_Check
196. ${libIGCM}/libIGCM_date/libIGCM_date.ksh
197       ( ${DEBUG_debug} ) && IGCM_date_Check
198#-------
199. ${libIGCM}/libIGCM_sys/libIGCM_sys.ksh
200. ${libIGCM}/libIGCM_config/libIGCM_config.ksh
201. ${libIGCM}/libIGCM_comp/libIGCM_comp.ksh
202. ${libIGCM}/libIGCM_post/libIGCM_post.ksh
203
204#D--------------------------------------------------------------------==
205
206#-- Resolution FLAG for oasis input
207#RESOL="ORCA2xLMD9671"
208[ -f ${SUBMIT_DIR}/../.resol ] && RESOL=$(head -1 ${SUBMIT_DIR}/../.resol)
209#-- Chemistrie FLAG to ease chemistries configurations management
210#CHEM="AER"
211[ -f ${SUBMIT_DIR}/../.chimie ] && eval $(grep CHEM ${SUBMIT_DIR}/../.chimie) 
212
213#D-
214#D--------------------------------------------------------------------==
215#D-           1. INITIALIZE CONFIGURATION AND THEN COMPONENTS
216#D--------------------------------------------------------------------==
217
218# --------------------------------------------------------------------==
219#D- Initialize configuration :
220#D-              - Simulation configuration
221#D-              - Simulation parameters
222#D-              - Execution parameter
223#D-              - Define input files directory
224#D-              - Read or initialize CumulPeriod
225#D-              - run.card
226#D-
227# --------------------------------------------------------------------==
228IGCM_config_Initialize
229
230# --------------------------------------------------------------------==
231#D- Define, create and cd RUN_DIR
232#D- 
233# --------------------------------------------------------------------==
234RUN_DIR=${RUN_DIR_PATH}/${config_UserChoices_JobName}.${$}
235IGCM_sys_MkdirWork ${RUN_DIR}
236IGCM_sys_Cd ${RUN_DIR}
237
238# ------------------------------------------------------------------
239# Test if all was right before proceeding further
240# ------------------------------------------------------------------
241IGCM_debug_Verif_Exit
242
243# --------------------------------------------------------------------==
244#D- Initialize each components taking part
245#D- into simulation configuration :
246#D-         - Define component executable
247#D-         - Define Write Frequency by component "1M" "1D" ...
248#D-         - Source ${comp}.driver
249#D-         - Define Storage Places by component
250#D-         - ${comp}_Initialize :
251#D-         - Patterns to sed
252#D-         - Variables from namelist
253#D-
254# --------------------------------------------------------------------==
255IGCM_comp_Initialize
256
257# --------------------------------------------------------------------==
258#D- Verify compatibility of period length, write frequencies, ...
259#D- Verify compatibility of rebuild choices and post-process choices
260#D- Will stop here if something is wrong
261# --------------------------------------------------------------------==
262IGCM_config_Check
263
264# ------------------------------------------------------------------
265#D- Test if all was right before entering the period loop
266# ------------------------------------------------------------------
267IGCM_debug_Verif_Exit
268
269#D--------------------------------------------------------------------==
270#D-                  2. ENTER THE DEEP EXECUTION LOOP
271#D-                  !!!! FASTEN YOUR SEAT BELTS !!!!
272#D-                  !!!!!      PLEASE.          !!!!
273#D-                  !! WE WILL CROSS TURBULENCES. !!
274#D--------------------------------------------------------------------==
275
276Period=1
277
278PeriodContinue=false
279if [ ${Period} -le ${PeriodNb} ]; then
280    PeriodContinue=true
281fi
282while ( ${PeriodContinue} ) ; do
283
284  echo
285  echo "Starting iteration ${Period} / ${PeriodNb}"
286
287  #D-
288  # ------------------------------------------------------------------
289  #D- COMPUTE AND DEFINE DATE STUFF.
290  # ------------------------------------------------------------------
291  IGCM_config_PeriodStart
292  #D- => ${PeriodDateBegin}
293  #D- => ${PeriodDateEnd}
294  #D- => ${CumulPeriod}
295  #D- => ${DatesPeriod}=${PeriodDateBegin}_${PeriodDateEnd}
296  #D- => ${PeriodLength} => JOURS=31 , 28, 29, 30 ; MOIS=0 ; ANS=0
297  #D- => update run.card value
298
299  #D-
300  # ------------------------------------------------------------------
301  #D- BEGIN COMPUTATION ONLY IF NEEDED
302  # ------------------------------------------------------------------
303  if [ ${SimulationLengthInDays} -gt ${ExperienceLengthInDays} ] ; then
304    IGCM_debug_Print 1 "break because '${SimulationLengthInDays} -gt ${ExperienceLengthInDays}' : ${SimulationLengthInDays} -gt ${ExperienceLengthInDays}"
305    break ;
306  fi
307
308  #D-
309  # ------------------------------------------------------------------
310  #D- Optionnal function in driver
311  #D- to set special variables used in forward lists (Param, Init or Bc).
312  # ------------------------------------------------------------------
313  IGCM_comp_PeriodStart
314
315  #D-
316  # ------------------------------------------------------------------
317  #D- Get parameters text files updated by job (.def, namelist ...)
318  #D- READ AND USE BY GCM AT EACH EXECUTION.
319  # ------------------------------------------------------------------
320  IGCM_comp_GetInputParametersFiles
321
322  #D-
323  # ------------------------------------------------------------------
324  #D- Get initial state (Etat0, carteveg,relief...)
325  #D- NECESSARY ONLY IF CumulPeriod= (=> NUMERO) == 1
326  #D- AND ???_Restart=NO
327  #D- READ AND USE BY GCM FOR ONLY ONE EXECUTION.
328  # ------------------------------------------------------------------
329  IGCM_comp_GetInputInitialStateFiles
330
331  #D-
332  # ------------------------------------------------------------------
333  #D- Get Boundaries Conditions (SST, WIND[X,Y,Z], LAI ...)
334  #D- READ AND USE BY GCM AT EACH EXECUTION.
335  # ------------------------------------------------------------------
336  IGCM_comp_GetInputBoundaryFiles
337
338  #D-
339  # ------------------------------------------------------------------
340  #D- Get SmoothFiles Conditions (SST, WIND[X,Y,Z], LAI ...)
341  #D- READ AND USE BY GCM AT EACH EXECUTION but varying in time
342  # ------------------------------------------------------------------
343  IGCM_comp_GetInputSmoothFiles
344
345  #D-
346  # ------------------------------------------------------------------
347  #D- Get restart files (restartphy.nc, orca_restart.nc ...)
348  #D- READ AND USE BY GCM AT EACH EXECUTION.
349  #D-
350  #D- IF A COMPONENT DO NOT RESTART FROM PREVIOULSY COMPUTED RESTART
351  #D- ONLY IF CumulPeriod= (=> NUMERO) == 1
352  #D- MUST EXECUTE CREATE ETAT0_LIMIT (TYPICALLY LMDZ AND ./create_etat0_limit.e)
353  #D-
354  #D- IF CumulPeriod == 1
355  #D- AND A COMPONENT RESTART FROM PREVIOULSY COMPUTED RESTART
356  #D- MUST BRING THIS RESTART IN ${RUN_DIR_PATH}
357  #D-
358  #D- ELSE BRING RESTART FROM ${JobName} IN ${RUN_DIR_PATH}
359  # ------------------------------------------------------------------
360  IGCM_comp_GetInputRestartFiles
361
362  #D-
363  # ------------------------------------------------------------------
364  #D- Test if all was right before Update
365  # ------------------------------------------------------------------
366  IGCM_debug_Verif_Exit
367
368  # ------------------------------------------------------------------
369  #D- Activate running environnement variables
370  # ------------------------------------------------------------------
371  IGCM_sys_activ_variables
372
373  #D-
374  # ------------------------------------------------------------------
375  #D- UPDATE ParametersFiles (.def, namelist ...) with
376  #D- current value of (ORCA_NIT00, ORCA_LRSTAR, RAZ_DATE, ...)
377  # ------------------------------------------------------------------
378  IGCM_comp_Update
379
380  #D-
381  # ------------------------------------------------------------------
382  #D- Test if all was right before execution
383  # ------------------------------------------------------------------
384  IGCM_debug_Verif_Exit
385
386  #D-
387  #D----------------------------------------------------------------==
388  #D-                          3. Execution
389  #D----------------------------------------------------------------==
390
391  echo
392  echo "#######################################"
393  echo "#      DIR BEFORE RUN EXECUTION       #"
394  echo "#######################################"
395  echo
396  ls -lrt
397
398  echo "========================================================================"
399  if [ ${DRYRUN} -le 1 ] ; then
400    REAL_DATE_INIT=$( date )
401    echo                                                                                   > ${Exe_Output}
402    echo "#######################################"                                        >> ${Exe_Output}
403    echo "EXECUTION of : ${EXECUTION}"
404    echo "EXECUTION of : ${EXECUTION}"                                                    >> ${Exe_Output}
405    echo                                                                                  >> ${Exe_Output}
406    typeset RET
407    RUN_DATE_BEGIN=$( date '+%Y-%m-%dT%H:%M:%S' )
408    ${EXECUTION}  >> ${Exe_Output} 2>&1
409    RET=$?
410    RUN_DATE_END=$( date '+%Y-%m-%dT%H:%M:%S' )
411    if [ ${RET} -gt 0 ] ; then
412      echo "Return code of executable :" ${RET}
413      IGCM_debug_Exit "EXECUTABLE"
414      IGCM_sys_Mkdir ${SUBMIT_DIR}/Debug
415      IGCM_sys_Cp ${Exe_Output} ${SUBMIT_DIR}/Debug/${PREFIX}_${Exe_Output}_error
416      ExecutionFail=true
417    fi
418    echo                                            >> ${Exe_Output}
419    echo "#######################################"  >> ${Exe_Output}
420    echo "libIGCM RunDateBegin=${RUN_DATE_BEGIN}"   >> ${Exe_Output}
421    echo "libIGCM RunDateEnd=${RUN_DATE_END}"       >> ${Exe_Output}
422    echo                                            >> ${Exe_Output}
423  else
424    echo "EXECUTION of : ${EXECUTION} simulated for DRYRUN = " $DRYRUN
425    echo "EXECUTION of : ${EXECUTION} simulated for DRYRUN = " $DRYRUN > ${Exe_Output}
426    if ( $DEBUG_debug ) ; then
427      echo "FOR EXECUTION DRYRUN mode = " $DRYRUN >> stack
428    fi
429  fi
430  echo "========================================================================"
431
432  echo
433  echo "#######################################"
434  echo "#       DIR AFTER RUN EXECUTION       #"
435  echo "#######################################"
436  echo
437  ls -lrt
438
439  #D-
440  # ------------------------------------------------------------------
441  #D- Desactivate running environnement variables
442  # ------------------------------------------------------------------
443  IGCM_sys_desactiv_variables
444
445  #D-
446  #D----------------------------------------------------------------==
447  #D-                    4. Post execution operations
448  #D----------------------------------------------------------------==
449
450  #D-
451  # ------------------------------------------------------------------
452  #D- Save on archive machine netcdf outputs, restarts
453  #D- and text files of models
454  # ------------------------------------------------------------------
455  IGCM_comp_Finalize
456
457  #D-
458  # ------------------------------------------------------------------
459  #D- Test if all was right after Finalize
460  # ------------------------------------------------------------------
461  IGCM_debug_Verif_Exit
462
463  #D-
464  # ------------------------------------------------------------------
465  #D- Configure asynchronous post-treatment (Level 0) when necessary (rebuild)
466  #D- Configure classic post-treatment (Level 1) when necessary (TimeSeries/Seasonal)
467  # ------------------------------------------------------------------
468  IGCM_post_Configure
469
470  #D-
471  # ------------------------------------------------------------------
472  #D- Launch remote rebuild or/and post-treatment process if necessary
473  # ------------------------------------------------------------------
474  IGCM_post_Submit
475
476  #D-
477  # ------------------------------------------------------------------
478  #D- Save job output
479  #D- Manage executable size to save only different binary
480  #D- Write in run.card user, system and elapse time
481  #D- Check that everything went well ! No ? then we stop.
482  #D- Determine next computed period
483  # ------------------------------------------------------------------
484  IGCM_config_PeriodEnd
485
486  echo "Ending iteration ${Period}"
487  (( Period = Period + 1 ))
488
489  # End loop if date end is reached
490  if [ ${SimulationLengthInDays} -ge ${ExperienceLengthInDays} ] ; then
491    break ;
492  fi
493
494  PeriodContinue=false
495  if [ ${Period} -le ${PeriodNb} ]; then
496    PeriodContinue=true
497  fi
498done
499
500#D-
501#D--------------------------------------------------------------------==
502#D-     5. Submit next job or send mail if simulation over.
503#D--------------------------------------------------------------------==
504IGCM_config_Finalize
505
506date
Note: See TracBrowser for help on using the repository browser.