source: trunk/libIGCM/AA_job @ 1095

Last change on this file since 1095 was 1095, checked in by sdipsl, 10 years ago
  • Property licence set to
    The following licence information concerns ONLY the libIGCM tools
    ==================================================================

    Copyright © Centre National de la Recherche Scientifique CNRS
    Commissariat à l'Énergie Atomique CEA

    libIGCM : Library for Portable Models Computation of IGCM Group.

    IGCM Group is the french IPSL Global Climate Model Group.

    This library is a set of shell scripts and functions whose purpose is
    the management of the initialization, the launch, the transfer of
    output files, the post-processing and the monitoring of datas produce
    by any numerical program on any plateforme.

    This software is governed by the CeCILL license under French law and
    abiding by the rules of distribution of free software. You can use,
    modify and/ or redistribute the software under the terms of the CeCILL
    license as circulated by CEA, CNRS and INRIA at the following URL
    "http://www.cecill.info".

    As a counterpart to the access to the source code and rights to copy,
    modify and redistribute granted by the license, users are provided only
    with a limited warranty and the software's author, the holder of the
    economic rights, and the successive licensors have only limited
    liability.

    In this respect, the user's attention is drawn to the risks associated
    with loading, using, modifying and/or developing or reproducing the
    software by the user in light of its specific status of free software,
    that may mean that it is complicated to manipulate, and that also
    therefore means that it is reserved for developers and experienced
    professionals having in-depth computer knowledge. Users are therefore
    encouraged to load and test the software's suitability as regards their
    requirements in conditions enabling the security of their systems and/or
    data to be ensured and, more generally, to use and operate it in the
    same conditions as regards security.

    The fact that you are presently reading this means that you have had
    knowledge of the CeCILL license and that you accept its terms.
  • Property svn:keywords set to Revision Author Date
File size: 20.1 KB
Line 
1#-Q- curie #!/bin/ksh
2#-Q- curie ######################
3#-Q- curie ## CURIE   TGCC/CEA ##
4#-Q- curie ######################
5#-Q- curie #MSUB -r ::Jobname::        # Job Name
6#-Q- curie #MSUB -o Script_Output_::Jobname::.000001    # standard output
7#-Q- curie #MSUB -e Script_Output_::Jobname::.000001    # error output
8#-Q- curie #MSUB -eo
9#-Q- curie #MSUB -n ::JobNumProcTot::  # reservation des processeurs pour le job
10#-Q- curie ##MSUB -N 8                 # Number of nodes (16 cores per node)
11#-Q- curie ##MSUB -x                    # exclusive node
12#-Q- curie ##MSUB -E '--cpu_bind=none'
13#-Q- curie #MSUB -T 86400              # Wall clock limit (seconds)
14#-Q- curie #MSUB -q standard           # thin nodes
15#-Q- curie #MSUB -A ::default_project::
16#-Q- curie ###MSUB -q ivybridge
17#-Q- curie ###MSUB -A dsm
18#-Q- curie BATCH_NUM_PROC_TOT=$BRIDGE_MSUB_NPROC
19#-Q- curie set +x
20#-Q- ada #!/bin/ksh
21#-Q- ada # ######################
22#-Q- ada # ##  ADA       IDRIS ##
23#-Q- ada # ######################
24#-Q- ada # Nom de la requete
25#-Q- ada # @ job_name = ::Jobname::
26#-Q- ada # Type de travail
27#-Q- ada # @ job_type = parallel
28#-Q- ada # Fichier de sortie standard
29#-Q- ada # @ output = Script_Output_::Jobname::.000001
30#-Q- ada # Fichier de sortie erreur (le meme)
31#-Q- ada # @ error = Script_Output_::Jobname::.000001
32#-Q- ada # Nombre de processus demandes
33#-Q- ada # @ total_tasks = ::JobNumProcTot::
34#-Q- ada # @ environment = "BATCH_NUM_PROC_TOT=::JobNumProcTot::"
35#-Q- ada # Temps CPU max. par processus MPI hh:mm:ss
36#-Q- ada # @ wall_clock_limit = 1:00:00
37#-Q- ada # Nombre de taches OpenMP/pthreads par processus MPI
38#-Q- ada ### @ parallel_threads = 4
39#-Q- ada # Fin de l entete
40#-Q- ada # @ queue
41#-Q- lxiv8 ######################
42#-Q- lxiv8 ## OBELIX      LSCE ##
43#-Q- lxiv8 ######################
44#-Q- lxiv8 #PBS -N ::Jobname::
45#-Q- lxiv8 #PBS -m a
46#-Q- lxiv8 #PBS -j oe
47#-Q- lxiv8 #PBS -q medium
48#-Q- lxiv8 #PBS -o Script_Output_::Jobname::.000001
49#-Q- lxiv8 #PBS -S /bin/ksh
50#-Q- lxiv8 #PBS -v BATCH_NUM_PROC_TOT=::JobNumProcTot::
51#-Q- lxiv8 #PBS -l nodes=1:ppn=::JobNumProcTot::
52#-Q- default #!/bin/ksh
53#-Q- default ##################
54#-Q- default ## DEFAULT HOST ##
55#-Q- default ##################
56#-Q- default #For MPI use, uncomment next line :
57#-Q- default #BATCH_NUM_PROC_TOT=::JobNumProcTot::
58
59#**************************************************************
60# Author: Sebastien Denvil
61# Contact: Sebastien.Denvil__at__ipsl.jussieu.fr
62# $Revision::                                          $ Revision of last commit
63# $Author::                                            $ Author of last commit
64# $Date::                                              $ Date of last commit
65# IPSL (2006)
66#  This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
67#
68#**************************************************************
69
70#set -eu
71#set -vx
72
73date
74echo
75echo "#######################################"
76echo "#       ANOTHER GREAT SIMULATION      #"
77echo "#######################################"
78echo
79
80#D--------------------------------------------------------------------==
81#D-
82#D-                      Job to launch IGCM models
83#D-
84#D--------------------------------------------------------------------==
85#D-
86
87MODIPSL=::modipsl::
88libIGCM=${MODIPSL}/libIGCM
89
90#D--------------------------------------------------------------------==
91#D- -1. PLACE FOR USER MODIFICATION
92#D-     - Job Verbosity
93#D-     - PeriodNb
94#D-     - Experience type : DEB(ug), DEV(elopment), RUN
95#D--------------------------------------------------------------------==
96
97#D- Task type (computing or post-processing)
98TaskType=computing
99#D- Increased verbosity (1, 2, 3)
100Verbosity=3
101#D- Experience type : DEB(ug), DEV(elopment), RUN (default)
102JobType=RUN
103#D- Number of execution in one job
104PeriodNb=1
105#-Q- sx9mercure PeriodNb=60
106
107#D-
108#D- --------------------------------------------------------------------------------------------------#
109#D-                          ! OPTIONNAL FEATURES /!\ USE WITH CARE !                                 #
110#D- --------------------------------------------------------------------------------------------------#
111
112
113#D- Low level debug : to bypass lib test checks and stack construction
114#D- Default=true
115#D-
116DEBUG_debug=true
117
118#D- Messaging : all activities and call stacks will be sent to ipsl servers
119#D- If true will imply DEBUG_debug=true
120#D- Default=false
121#D-
122BigBrother=true
123
124#D- Set DEBUG_sys to false to disable output calls of function
125#D- true when JobType=DEB # means DEB(ug)
126#D-
127#DEBUG_sys=false
128
129#D- Define running directory
130#D- Default=${TMPDIR} ie temporary batch directory
131#D-
132#RUN_DIR_PATH=/workdir/or/scratchdir/of/this/machine
133
134#D- Define submit directory
135#D- Default= where you launch qsub (variable from scheduler)
136#D-
137#SUBMIT_DIR=$( pwd )
138
139#D- Define input file root directory
140#D- Default=/IGCMG/common/account/of/this/machine
141#D-
142#R_IN=/u/rech/por/rpor111/DATA
143
144#D- Turn in dry run mode ? (0,1,2,3)
145#D- Default=0
146#D-
147#DRYRUN=3
148#D-# YOU HAVE TO COMPILE YOUR EXE FILES to USE DRYRUN MODE !
149#D-# -------------------------------------------------------------------------------------
150#D-# | DRYRUN=  |  Date computations, | sys_Get    |  Exe    | sys_Put_Out; sys_Put_Rest |
151#D-# |          |  Cp/Exe/param/files |            |         |                           |
152#D-# |          |  Chmod Qsub         |            |         |                           |
153#D-# -------------------------------------------------------------------------------------
154#D-# |    0     |       yes           |    yes     |  yes    |      yes                  |
155#D-# -------------------------------------------------------------------------------------
156#D-# |    1     |       yes           |    yes     |  yes    |      no                   |
157#D-# -------------------------------------------------------------------------------------
158#D-# |    2     |       yes           |    yes     |  no     |      no                   |
159#D-# -------------------------------------------------------------------------------------
160#D-# |    3     |       yes           |    no      |  no     |      no                   |
161#D-# -------------------------------------------------------------------------------------
162
163#D-
164# --------------------------------------------------------------------------------------------------#
165#D- ! END OF OPTIONNAL FEATURES /!\ DO NOT MODIFY ANYTHING BELOW UNLESS YOU KNOW WHAT YOU ARE DOING #
166# --------------------------------------------------------------------------------------------------#
167
168#D-
169#D--------------------------------------------------------------------==
170#D- 0.0 SYSTEM ENVIRONMENT
171#D-     - Define variables for Number of processors
172#D-     - Define MPI variables
173#D-     - Define batch scheduler variables
174#D-     - Source IGCM Library
175#D-     - Get RESOLution in .resol file (temporary)
176#D--------------------------------------------------------------------==
177
178#---------------------------------------------------------------------==
179
180. ${libIGCM}/libIGCM_debug/libIGCM_debug.ksh
181. ${libIGCM}/libIGCM_card/libIGCM_card.ksh
182. ${libIGCM}/libIGCM_date/libIGCM_date.ksh
183#-------
184. ${libIGCM}/libIGCM_sys/libIGCM_sys.ksh
185. ${libIGCM}/libIGCM_config/libIGCM_config.ksh
186. ${libIGCM}/libIGCM_comp/libIGCM_comp.ksh
187. ${libIGCM}/libIGCM_post/libIGCM_post.ksh
188#-------
189( ${DEBUG_debug} ) && IGCM_debug_Check
190( ${DEBUG_debug} ) && IGCM_card_Check
191( ${DEBUG_debug} ) && IGCM_date_Check
192
193#--------------------------------------------------------------------==
194
195#-- Resolution FLAG for oasis input
196#RESOL="ORCA2xLMD9671"
197[ -f ${SUBMIT_DIR}/../.resol ] && RESOL=$(head -1 ${SUBMIT_DIR}/../.resol)
198#-- Chemistrie FLAG to ease chemistries configurations management
199#CHEM="AER"
200[ -f ${SUBMIT_DIR}/../.chimie ] && eval $(grep CHEM ${SUBMIT_DIR}/../.chimie)
201
202#D-
203#D--------------------------------------------------------------------==
204#D- 0.1 COMMON ENVIRONMENT
205#D-     - Read libIGCM compatibility version in config.card
206#D-     - Read UserChoices section
207#D-     - Read Ensemble section
208#D-     - Read Post section
209#D-     - Define all netcdf output directories
210#D--------------------------------------------------------------------==
211IGCM_config_CommonConfiguration ${SUBMIT_DIR}/config.card
212
213if [ ! -r ${SUBMIT_DIR}/run.card ] ; then
214  #================================================#
215  #         The file run.card doesn't exist        #
216  #================================================#
217  FirstInitialize=true
218  #copy initial run.card
219  IGCM_sys_Cp ${SUBMIT_DIR}/run.card.init ${SUBMIT_DIR}/run.card
220  IGCM_debug_Print 2 "run.card copied from run.card.init"
221else
222  FirstInitialize=false
223  IGCM_debug_Print 2 "run.card exists"
224fi
225
226# ------------------------------------------------------------------
227# Activate BigBrother so as to supervise this simulation
228# ------------------------------------------------------------------
229IGCM_debug_ActivateBigBro
230
231#D-
232#D--------------------------------------------------------------------==
233#D- 1. INITIALIZE CONFIGURATION
234#D-    - Simulation configuration
235#D-    - Simulation parameters
236#D-    - Execution parameter
237#D-    - Define input files directory
238#D-    - Read or initialize CumulPeriod
239#D-    - run.card
240#D--------------------------------------------------------------------==
241IGCM_config_Initialize
242
243# --------------------------------------------------------------------==
244# Define, create and cd RUN_DIR
245# --------------------------------------------------------------------==
246RUN_DIR=${RUN_DIR_PATH}/${config_UserChoices_JobName}.${$}
247IGCM_sys_MkdirWork ${RUN_DIR}
248IGCM_sys_Cd ${RUN_DIR}
249
250# ------------------------------------------------------------------
251# Test if all was right before proceeding further
252# ------------------------------------------------------------------
253IGCM_debug_Verif_Exit
254
255#D-
256#D---------------------------------------------------------------------==
257#D- 2. INITIALIZE ALL COMPONENT OF THE CONFIGURATION
258#D-    - Define component executable
259#D-    - Define Write Frequency by component "1M" "1D" ...
260#D-    - Source ${comp}.driver
261#D-    - Define Storage Places by component
262#D-    - ${comp}_Initialize :
263#D-    - Patterns to sed
264#D-    - Variables from namelist
265#D---------------------------------------------------------------------==
266IGCM_comp_Initialize
267
268#D-
269#D---------------------------------------------------------------------==
270#D- 3. PERFORM GENERAL VERIFICATION
271#D-    - Verify compatibility of period length, write frequencies, ...
272#D-    - Verify compatibility of rebuild choices and post-process choices
273#D-    - Will stop here if something is wrong
274#D---------------------------------------------------------------------==
275IGCM_config_Check
276
277# ------------------------------------------------------------------
278# Test if all was right before entering the period loop
279# ------------------------------------------------------------------
280IGCM_debug_Verif_Exit
281
282#D-
283#D---------------------------------------------------------------------==
284#D- 4. ENTER THE DEEP EXECUTION LOOP
285#D-    ! FASTEN YOUR SEAT BELTS PLEASE. !
286#D-    !   WE WILL CROSS TURBULENCES.   !
287#D---------------------------------------------------------------------==
288
289Period=1
290
291PeriodContinue=false
292if [ ${Period} -le ${PeriodNb} ]; then
293  PeriodContinue=true
294fi
295while ( ${PeriodContinue} ) ; do
296
297  echo
298  echo "Starting iteration ${Period} / ${PeriodNb}"
299
300  #D-
301  # ------------------------------------------------------------------
302  #D-   4.1 COMPUTE AND DEFINE DATE INFORMATION.
303  #D-       - ${PeriodDateBegin}
304  #D-       - ${PeriodDateEnd}
305  #D-       - ${CumulPeriod}
306  #D-       - ${DatesPeriod}=${PeriodDateBegin}_${PeriodDateEnd}
307  #D-       - ${PeriodLength} => JOURS=31 , 28, 29, 30 ; MOIS=0 ; ANS=0
308  #D-       - update run.card value
309  # ------------------------------------------------------------------
310  IGCM_config_PeriodStart
311
312  #D-
313  # ------------------------------------------------------------------
314  #D-   4.2 BEGIN COMPUTATION ONLY IF NEEDED
315  # ------------------------------------------------------------------
316  if [ ${SimulationLengthInDays} -gt ${ExperienceLengthInDays} ] ; then
317    IGCM_debug_Print 1 "break because '${SimulationLengthInDays} -gt ${ExperienceLengthInDays}' : ${SimulationLengthInDays} -gt ${ExperienceLengthInDays}"
318    break ;
319  fi
320
321  #D-
322  # ------------------------------------------------------------------
323  #D-   4.3 CALL OPTIONNAL FUNCTION WITHIN DRIVER
324  #D-       - to set special variables used in lists (Param, Init or Bc).
325  # ------------------------------------------------------------------
326  IGCM_comp_PeriodStart
327
328  #D-
329  # ------------------------------------------------------------------
330  #D-   4.4 GET PARAMETERS TEXT FILES UPDATED BY JOB
331  #D-       - (.def, namelist ...)
332  #D-       - READ AND USE BY GCM AT EACH EXECUTION.
333  # ------------------------------------------------------------------
334  IGCM_comp_GetInputParametersFiles
335
336  #D-
337  # ------------------------------------------------------------------
338  #D-   4.5 GET INITIAL STATE
339  #D-       - (Etat0, carteveg,relief...)
340  #D-       - NECESSARY ONLY IF CumulPeriod= (=> NUMERO) == 1
341  #D-       - AND ???_Restart=NO
342  #D-       - READ AND USE BY GCM FOR ONLY ONE EXECUTION.
343  # ------------------------------------------------------------------
344  IGCM_comp_GetInputInitialStateFiles
345
346  #D-
347  # ------------------------------------------------------------------
348  #D-   4.6 GET BOUNDARIES CONDITIONS
349  #D-       - (SST, WIND[X,Y,Z], LAI ...)
350  #D-       - READ AND USE BY GCM AT EACH EXECUTION.
351  # ------------------------------------------------------------------
352  IGCM_comp_GetInputBoundaryFiles
353
354  #D-
355  # ------------------------------------------------------------------
356  #D-   4.7 GET SmoothFiles CONDITIONS
357  #D-       - (SST, WIND[X,Y,Z], LAI ...)
358  #D-       - READ AND USE BY GCM AT EACH EXECUTION
359  #D-       - Do not change at each ${Period}
360  # ------------------------------------------------------------------
361  IGCM_comp_GetInputSmoothFiles
362
363  #D-
364  # ------------------------------------------------------------------
365  #D-   4.8 GET RESTART FILES
366  #D-       - (restartphy.nc, orca_restart.nc ...)
367  #D-       - READ AND USE BY GCM AT EACH EXECUTION.
368  #D-
369  #D-       - IF A COMPONENT DO NOT RESTART FROM PREVIOULSY COMPUTED RESTART
370  #D-       - ONLY IF CumulPeriod= (=> NUMERO) == 1
371  #D-       - MUST EXECUTE CREATE ETAT0_LIMIT (TYPICALLY LMDZ AND ./create_etat0_limit.e)
372  #D-
373  #D-       - IF CumulPeriod == 1
374  #D-       - AND A COMPONENT RESTART FROM PREVIOULSY COMPUTED RESTART
375  #D-       - MUST BRING THIS RESTART IN ${RUN_DIR_PATH}
376  #D-
377  #D-       - ELSE BRING RESTART FROM ${JobName} IN ${RUN_DIR_PATH}
378  # ------------------------------------------------------------------
379  IGCM_comp_GetInputRestartFiles
380
381  # ------------------------------------------------------------------
382  # Test if all was right before Update
383  # ------------------------------------------------------------------
384  IGCM_debug_Verif_Exit
385
386  #D-
387  # ------------------------------------------------------------------
388  #D-   4.9 ACTIVATE RUNNING ENVIRONNEMENT VARIABLES
389  # ------------------------------------------------------------------
390  IGCM_sys_activ_variables
391
392  #D-
393  # ------------------------------------------------------------------
394  #D-   4.10 UPDATE ParametersFiles
395  #D-        - (.def, namelist ...)
396  #D-        - with current value of (ORCA_NIT00, ORCA_LRSTAR, RAZ_DATE, ...)
397  # ------------------------------------------------------------------
398  IGCM_comp_Update
399
400  # ------------------------------------------------------------------
401  # Test if all was right before execution
402  # ------------------------------------------------------------------
403  IGCM_debug_Verif_Exit
404
405  #D-
406  #D---------------------------------------------------------------------==
407  #D- 5. EXECUTION
408  #D---------------------------------------------------------------------==
409
410  echo
411  echo "#######################################"
412  echo "#      DIR BEFORE RUN EXECUTION       #"
413  echo "#######################################"
414  echo
415  ls -lrt
416
417  echo "========================================================================"
418  if [ ${DRYRUN} -le 1 ] ; then
419    REAL_DATE_INIT=$( date )
420    echo                                                                                   > ${Exe_Output}
421    echo "#######################################"                                        >> ${Exe_Output}
422    echo "EXECUTION of : ${EXECUTION}"
423    echo "EXECUTION of : ${EXECUTION}"                                                    >> ${Exe_Output}
424    echo                                                                                  >> ${Exe_Output}
425    typeset RET
426    RUN_DATE_BEGIN=$( date '+%Y-%m-%dT%H:%M:%S' )
427    ${EXECUTION}  >> ${Exe_Output} 2>&1
428    RET=$?
429    RUN_DATE_END=$( date '+%Y-%m-%dT%H:%M:%S' )
430    if [ ${RET} -gt 0 ] ; then
431      echo "Return code of executable :" ${RET}
432      IGCM_debug_Exit "EXECUTABLE"
433      IGCM_sys_Mkdir ${SUBMIT_DIR}/Debug
434      IGCM_sys_Cp ${Exe_Output} ${SUBMIT_DIR}/Debug/${PREFIX}_${Exe_Output}_error
435      ExecutionFail=true
436    fi
437    echo                                            >> ${Exe_Output}
438    echo "#######################################"  >> ${Exe_Output}
439    echo "libIGCM RunDateBegin=${RUN_DATE_BEGIN}"   >> ${Exe_Output}
440    echo "libIGCM RunDateEnd=${RUN_DATE_END}"       >> ${Exe_Output}
441    echo                                            >> ${Exe_Output}
442  else
443    echo "EXECUTION of : ${EXECUTION} simulated for DRYRUN = " $DRYRUN
444    echo "EXECUTION of : ${EXECUTION} simulated for DRYRUN = " $DRYRUN > ${Exe_Output}
445    if ( $DEBUG_debug ) ; then
446      echo "FOR EXECUTION DRYRUN mode = " $DRYRUN >> stack
447    fi
448  fi
449  echo "========================================================================"
450
451  echo
452  echo "#######################################"
453  echo "#       DIR AFTER RUN EXECUTION       #"
454  echo "#######################################"
455  echo
456  ls -lrt
457
458  #D-
459  #D---------------------------------------------------------------------==
460  #D- 6. POST EXECUTION OPERATIONS
461  #D---------------------------------------------------------------------==
462
463  #D-
464  # ------------------------------------------------------------------
465  #D-   6.1. DESACTIVATE RUNNING ENVIRONNEMENT VARIABLES
466  # ------------------------------------------------------------------
467  IGCM_sys_desactiv_variables
468
469  #D-
470  # ------------------------------------------------------------------
471  #D-   6.2. SAVE OUTPUTS
472  #D-        - On archive/storage machine
473  #D-        - netcdf outputs, restarts and text files of models
474  # ------------------------------------------------------------------
475  IGCM_comp_Finalize
476
477  # ------------------------------------------------------------------
478  # Test if all was right after Finalize
479  # ------------------------------------------------------------------
480  IGCM_debug_Verif_Exit
481
482  #D-
483  # ------------------------------------------------------------------
484  #D-   6.3. CONFIGURE POST-PROCESSING
485  #D-        - asynchronous post-treatment (Level 0) when necessary (rebuild/pack)
486  #D-        - classic post-treatment      (Level 1) when necessary (TimeSeries/Seasonal)
487  # ------------------------------------------------------------------
488  IGCM_post_Configure
489
490  #D-
491  # ------------------------------------------------------------------
492  #D-   6.4. SUBMIT POST-PROCESSING
493  #D-        - remote rebuild or/and post-treatment process if necessary
494  # ------------------------------------------------------------------
495  IGCM_post_Submit
496
497  #D-
498  # ------------------------------------------------------------------
499  #D-   6.5. FINALYZE CURRENT LOOP AND PREPARE NEXT ONE
500  #D-        - Manage executable size to save only different binary
501  #D-        - Write in run.card user, system and elapse time
502  #D-        - Check that everything went well ! No ? then we stop.
503  #D-        - Determine next computed period
504  # ------------------------------------------------------------------
505  IGCM_config_PeriodEnd
506
507  echo "Ending iteration ${Period}"
508  (( Period = Period + 1 ))
509
510  # End loop if date end is reached
511  if [ ${SimulationLengthInDays} -ge ${ExperienceLengthInDays} ] ; then
512    break ;
513  fi
514
515  PeriodContinue=false
516  if [ ${Period} -le ${PeriodNb} ]; then
517    PeriodContinue=true
518  fi
519done
520
521#D-
522#D---------------------------------------------------------------------==
523#D- 7. SUBMIT NEXT JOB OR SEND MAIL IF SIMULATION IS OVER.
524#D---------------------------------------------------------------------==
525IGCM_config_Finalize
526
527date
Note: See TracBrowser for help on using the repository browser.