source: trunk/libIGCM/AA_job @ 1030

Last change on this file since 1030 was 1030, checked in by sdipsl, 10 years ago

See #199: Add mixte mpi/omp curie examples
  • Property licence set to
    The following licence information concerns ONLY the libIGCM tools
    ==================================================================

    Copyright © Centre National de la Recherche Scientifique CNRS
    Commissariat à l'Énergie Atomique CEA

    libIGCM : Library for Portable Models Computation of IGCM Group.

    IGCM Group is the french IPSL Global Climate Model Group.

    This library is a set of shell scripts and functions whose purpose is
    the management of the initialization, the launch, the transfer of
    output files, the post-processing and the monitoring of datas produce
    by any numerical program on any plateforme.

    This software is governed by the CeCILL license under French law and
    abiding by the rules of distribution of free software. You can use,
    modify and/ or redistribute the software under the terms of the CeCILL
    license as circulated by CEA, CNRS and INRIA at the following URL
    "http://www.cecill.info".

    As a counterpart to the access to the source code and rights to copy,
    modify and redistribute granted by the license, users are provided only
    with a limited warranty and the software's author, the holder of the
    economic rights, and the successive licensors have only limited
    liability.

    In this respect, the user's attention is drawn to the risks associated
    with loading, using, modifying and/or developing or reproducing the
    software by the user in light of its specific status of free software,
    that may mean that it is complicated to manipulate, and that also
    therefore means that it is reserved for developers and experienced
    professionals having in-depth computer knowledge. Users are therefore
    encouraged to load and test the software's suitability as regards their
    requirements in conditions enabling the security of their systems and/or
    data to be ensured and, more generally, to use and operate it in the
    same conditions as regards security.

    The fact that you are presently reading this means that you have had
    knowledge of the CeCILL license and that you accept its terms.
  • Property svn:keywords set to Revision Author Date
File size: 19.2 KB
Line 
1#-Q- curie #!/bin/ksh
2#-Q- curie ######################
3#-Q- curie ## CURIE   TGCC/CEA ##
4#-Q- curie ######################
5#-Q- curie #MSUB -r ::Jobname::        # Job Name
6#-Q- curie #MSUB -o Script_Output_::Jobname::.000001    # standard output
7#-Q- curie #MSUB -e Script_Output_::Jobname::.000001    # error output
8#-Q- curie #MSUB -eo
9#-Q- curie #MSUB -n ::JobNumProcTot::  # reservation des processeurs pour le job
10#-Q- curie ##MSUB -N 8                 # Number of nodes (16 cores per node)
11#-Q- curie ##MSUB -x                    # exclusive node
12#-Q- curie ##MSUB -E '--cpu_bind=none'  # no bindings
13#-Q- curie #MSUB -T 86400              # Wall clock limit (seconds)
14#-Q- curie #MSUB -q standard           # thin nodes
15#-Q- curie #MSUB -A ::default_project::
16#-Q- curie BATCH_NUM_PROC_TOT=$BRIDGE_MSUB_NPROC
17#-Q- curie set +x
18#-Q- ada #!/bin/ksh
19#-Q- ada # ######################
20#-Q- ada # ##  ADA       IDRIS ##
21#-Q- ada # ######################
22#-Q- ada # Nom de la requete
23#-Q- ada # @ job_name = ::Jobname::
24#-Q- ada # Type de travail
25#-Q- ada # @ job_type = parallel
26#-Q- ada # Fichier de sortie standard
27#-Q- ada # @ output = Script_Output_::Jobname::.000001
28#-Q- ada # Fichier de sortie erreur (le meme)
29#-Q- ada # @ error = Script_Output_::Jobname::.000001
30#-Q- ada # Nombre de processus demandes
31#-Q- ada # @ total_tasks = ::JobNumProcTot::
32#-Q- ada # @ environment = "BATCH_NUM_PROC_TOT=::JobNumProcTot::"
33#-Q- ada # Temps CPU max. par processus MPI hh:mm:ss
34#-Q- ada # @ wall_clock_limit = 1:00:00
35#-Q- ada # Nombre de taches OpenMP/pthreads par processus MPI
36#-Q- ada ### @ parallel_threads = 4
37#-Q- ada # Fin de l entete
38#-Q- ada # @ queue
39#-Q- lxiv8 ######################
40#-Q- lxiv8 ## OBELIX      LSCE ##
41#-Q- lxiv8 ######################
42#-Q- lxiv8 #PBS -N ::Jobname::
43#-Q- lxiv8 #PBS -m a
44#-Q- lxiv8 #PBS -j oe
45#-Q- lxiv8 #PBS -q medium
46#-Q- lxiv8 #PBS -o Script_Output_::Jobname::.000001
47#-Q- lxiv8 #PBS -S /bin/ksh
48#-Q- lxiv8 #PBS -v BATCH_NUM_PROC_TOT=::JobNumProcTot::
49#-Q- lxiv8 #PBS -l nodes=1:ppn=::JobNumProcTot::
50#-Q- default #!/bin/ksh
51#-Q- default ##################
52#-Q- default ## DEFAULT HOST ##
53#-Q- default ##################
54#-Q- default #For MPI use, uncomment next line :
55#-Q- default #BATCH_NUM_PROC_TOT=::JobNumProcTot::
56
57#**************************************************************
58# Author: Sebastien Denvil
59# Contact: Sebastien.Denvil__at__ipsl.jussieu.fr
60# $Revision::                                          $ Revision of last commit
61# $Author::                                            $ Author of last commit
62# $Date::                                              $ Date of last commit
63# IPSL (2006)
64#  This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
65#
66#**************************************************************
67
68#set -eu
69#set -vx
70
71date
72echo
73echo "#######################################"
74echo "#       ANOTHER GREAT SIMULATION      #"
75echo "#######################################"
76echo
77
78#D--------------------------------------------------------------------==
79#D-
80#D-                      Job to launch IGCM models
81#D-
82#D--------------------------------------------------------------------==
83#D-
84
85MODIPSL=::modipsl::
86libIGCM=${MODIPSL}/libIGCM
87
88#D--------------------------------------------------------------------==
89#D- -1. PLACE FOR USER MODIFICATION
90#D-     - Job Verbosity
91#D-     - PeriodNb
92#D-     - Experience type : DEB(ug), DEV(elopment), RUN
93#D--------------------------------------------------------------------==
94
95#D- Task type (computing or post-processing)
96TaskType=computing
97#D- Increased verbosity (1, 2, 3)
98Verbosity=3
99#D- Experience type : DEB(ug), DEV(elopment), RUN (default)
100JobType=RUN
101#D- Number of execution in one job
102PeriodNb=1
103#-Q- sx9mercure PeriodNb=60
104
105#D-
106#D- --------------------------------------------------------------------------------------------------#
107#D-                          ! OPTIONNAL FEATURES /!\ USE WITH CARE !                                 #
108#D- --------------------------------------------------------------------------------------------------#
109
110
111#D- Low level debug : to bypass lib test checks and stack construction
112#D- Default=true
113#D-
114DEBUG_debug=false
115
116#D- Messaging : all activities and call stacks will be sent to ipsl servers
117#D- If true will imply DEBUG_debug=true
118#D- Default=false
119#D-
120BigBrother=false
121
122#D- Set DEBUG_sys to false to disable output calls of function
123#D- true when JobType=DEB # means DEB(ug)
124#D-
125#DEBUG_sys=false
126
127#D- Define running directory
128#D- Default=${TMPDIR} ie temporary batch directory
129#D-
130#RUN_DIR_PATH=/workdir/or/scratchdir/of/this/machine
131
132#D- Define submit directory
133#D- Default= where you launch qsub (variable from scheduler)
134#D-
135#SUBMIT_DIR=$( pwd )
136
137#D- Define input file root directory
138#D- Default=/IGCMG/common/account/of/this/machine
139#D-
140#R_IN=/u/rech/por/rpor111/DATA
141
142#D- Turn in dry run mode ? (0,1,2,3)
143#D- Default=0
144#D-
145#DRYRUN=3
146#D-# YOU HAVE TO COMPILE YOUR EXE FILES to USE DRYRUN MODE !
147#D-# -------------------------------------------------------------------------------------
148#D-# | DRYRUN=  |  Date computations, | sys_Get    |  Exe    | sys_Put_Out; sys_Put_Rest |
149#D-# |          |  Cp/Exe/param/files |            |         |                           |
150#D-# |          |  Chmod Qsub         |            |         |                           |
151#D-# -------------------------------------------------------------------------------------
152#D-# |    0     |       yes           |    yes     |  yes    |      yes                  |
153#D-# -------------------------------------------------------------------------------------
154#D-# |    1     |       yes           |    yes     |  yes    |      no                   |
155#D-# -------------------------------------------------------------------------------------
156#D-# |    2     |       yes           |    yes     |  no     |      no                   |
157#D-# -------------------------------------------------------------------------------------
158#D-# |    3     |       yes           |    no      |  no     |      no                   |
159#D-# -------------------------------------------------------------------------------------
160
161#D-
162# --------------------------------------------------------------------------------------------------#
163#D- ! END OF OPTIONNAL FEATURES /!\ DO NOT MODIFY ANYTHING BELOW UNLESS YOU KNOW WHAT YOU ARE DOING #
164# --------------------------------------------------------------------------------------------------#
165
166#D-
167#D--------------------------------------------------------------------==
168#D- 0. SYSTEM ENVIRONMENT
169#D-    - Define variables for Number of processors
170#D-    - Define MPI variables
171#D-    - Define batch scheduler variables
172#D-    - Source IGCM Library
173#D-    - Get RESOLution in .resol file (temporary)
174#D--------------------------------------------------------------------==
175
176#---------------------------------------------------------------------==
177
178. ${libIGCM}/libIGCM_debug/libIGCM_debug.ksh
179. ${libIGCM}/libIGCM_card/libIGCM_card.ksh
180. ${libIGCM}/libIGCM_date/libIGCM_date.ksh
181#-------
182. ${libIGCM}/libIGCM_sys/libIGCM_sys.ksh
183. ${libIGCM}/libIGCM_config/libIGCM_config.ksh
184. ${libIGCM}/libIGCM_comp/libIGCM_comp.ksh
185. ${libIGCM}/libIGCM_post/libIGCM_post.ksh
186#-------
187( ${DEBUG_debug} ) && IGCM_debug_Check
188( ${DEBUG_debug} ) && IGCM_card_Check
189( ${DEBUG_debug} ) && IGCM_date_Check
190
191#--------------------------------------------------------------------==
192
193#-- Resolution FLAG for oasis input
194#RESOL="ORCA2xLMD9671"
195[ -f ${SUBMIT_DIR}/../.resol ] && RESOL=$(head -1 ${SUBMIT_DIR}/../.resol)
196#-- Chemistrie FLAG to ease chemistries configurations management
197#CHEM="AER"
198[ -f ${SUBMIT_DIR}/../.chimie ] && eval $(grep CHEM ${SUBMIT_DIR}/../.chimie)
199
200#D-
201#D--------------------------------------------------------------------==
202#D- 1. INITIALIZE CONFIGURATION
203#D-    - Simulation configuration
204#D-    - Simulation parameters
205#D-    - Execution parameter
206#D-    - Define input files directory
207#D-    - Read or initialize CumulPeriod
208#D-    - run.card
209#D---------------------------------------------------------------------==
210IGCM_config_Initialize
211
212# --------------------------------------------------------------------==
213# Define, create and cd RUN_DIR
214# --------------------------------------------------------------------==
215RUN_DIR=${RUN_DIR_PATH}/${config_UserChoices_JobName}.${$}
216IGCM_sys_MkdirWork ${RUN_DIR}
217IGCM_sys_Cd ${RUN_DIR}
218
219# ------------------------------------------------------------------
220# Test if all was right before proceeding further
221# ------------------------------------------------------------------
222IGCM_debug_Verif_Exit
223
224#D-
225#D---------------------------------------------------------------------==
226#D- 2. INITIALIZE ALL COMPONENT OF THE CONFIGURATION
227#D-    - Define component executable
228#D-    - Define Write Frequency by component "1M" "1D" ...
229#D-    - Source ${comp}.driver
230#D-    - Define Storage Places by component
231#D-    - ${comp}_Initialize :
232#D-    - Patterns to sed
233#D-    - Variables from namelist
234#D---------------------------------------------------------------------==
235IGCM_comp_Initialize
236
237#D-
238#D---------------------------------------------------------------------==
239#D- 3. PERFORM GENERAL VERIFICATION
240#D-    - Verify compatibility of period length, write frequencies, ...
241#D-    - Verify compatibility of rebuild choices and post-process choices
242#D-    - Will stop here if something is wrong
243#D---------------------------------------------------------------------==
244IGCM_config_Check
245
246# ------------------------------------------------------------------
247# Test if all was right before entering the period loop
248# ------------------------------------------------------------------
249IGCM_debug_Verif_Exit
250
251# ------------------------------------------------------------------
252# Activate BigBrother so as to supervise this simulation
253# ------------------------------------------------------------------
254IGCM_debug_ActivateBigBro
255
256#D-
257#D---------------------------------------------------------------------==
258#D- 4. ENTER THE DEEP EXECUTION LOOP
259#D-    ! FASTEN YOUR SEAT BELTS PLEASE. !
260#D-    !   WE WILL CROSS TURBULENCES.   !
261#D---------------------------------------------------------------------==
262
263Period=1
264
265PeriodContinue=false
266if [ ${Period} -le ${PeriodNb} ]; then
267  PeriodContinue=true
268fi
269while ( ${PeriodContinue} ) ; do
270
271  echo
272  echo "Starting iteration ${Period} / ${PeriodNb}"
273
274  #D-
275  # ------------------------------------------------------------------
276  #D-   4.1 COMPUTE AND DEFINE DATE INFORMATION.
277  #D-       - ${PeriodDateBegin}
278  #D-       - ${PeriodDateEnd}
279  #D-       - ${CumulPeriod}
280  #D-       - ${DatesPeriod}=${PeriodDateBegin}_${PeriodDateEnd}
281  #D-       - ${PeriodLength} => JOURS=31 , 28, 29, 30 ; MOIS=0 ; ANS=0
282  #D-       - update run.card value
283  # ------------------------------------------------------------------
284  IGCM_config_PeriodStart
285
286  #D-
287  # ------------------------------------------------------------------
288  #D-   4.2 BEGIN COMPUTATION ONLY IF NEEDED
289  # ------------------------------------------------------------------
290  if [ ${SimulationLengthInDays} -gt ${ExperienceLengthInDays} ] ; then
291    IGCM_debug_Print 1 "break because '${SimulationLengthInDays} -gt ${ExperienceLengthInDays}' : ${SimulationLengthInDays} -gt ${ExperienceLengthInDays}"
292    break ;
293  fi
294
295  #D-
296  # ------------------------------------------------------------------
297  #D-   4.3 CALL OPTIONNAL FUNCTION WITHIN DRIVER
298  #D-       - to set special variables used in lists (Param, Init or Bc).
299  # ------------------------------------------------------------------
300  IGCM_comp_PeriodStart
301
302  #D-
303  # ------------------------------------------------------------------
304  #D-   4.4 GET PARAMETERS TEXT FILES UPDATED BY JOB
305  #D-       - (.def, namelist ...)
306  #D-       - READ AND USE BY GCM AT EACH EXECUTION.
307  # ------------------------------------------------------------------
308  IGCM_comp_GetInputParametersFiles
309
310  #D-
311  # ------------------------------------------------------------------
312  #D-   4.5 GET INITIAL STATE
313  #D-       - (Etat0, carteveg,relief...)
314  #D-       - NECESSARY ONLY IF CumulPeriod= (=> NUMERO) == 1
315  #D-       - AND ???_Restart=NO
316  #D-       - READ AND USE BY GCM FOR ONLY ONE EXECUTION.
317  # ------------------------------------------------------------------
318  IGCM_comp_GetInputInitialStateFiles
319
320  #D-
321  # ------------------------------------------------------------------
322  #D-   4.6 GET BOUNDARIES CONDITIONS
323  #D-       - (SST, WIND[X,Y,Z], LAI ...)
324  #D-       - READ AND USE BY GCM AT EACH EXECUTION.
325  # ------------------------------------------------------------------
326  IGCM_comp_GetInputBoundaryFiles
327
328  #D-
329  # ------------------------------------------------------------------
330  #D-   4.7 GET SmoothFiles CONDITIONS
331  #D-       - (SST, WIND[X,Y,Z], LAI ...)
332  #D-       - READ AND USE BY GCM AT EACH EXECUTION
333  #D-       - Do not change at each ${Period}
334  # ------------------------------------------------------------------
335  IGCM_comp_GetInputSmoothFiles
336
337  #D-
338  # ------------------------------------------------------------------
339  #D-   4.8 GET RESTART FILES
340  #D-       - (restartphy.nc, orca_restart.nc ...)
341  #D-       - READ AND USE BY GCM AT EACH EXECUTION.
342  #D-
343  #D-       - IF A COMPONENT DO NOT RESTART FROM PREVIOULSY COMPUTED RESTART
344  #D-       - ONLY IF CumulPeriod= (=> NUMERO) == 1
345  #D-       - MUST EXECUTE CREATE ETAT0_LIMIT (TYPICALLY LMDZ AND ./create_etat0_limit.e)
346  #D-
347  #D-       - IF CumulPeriod == 1
348  #D-       - AND A COMPONENT RESTART FROM PREVIOULSY COMPUTED RESTART
349  #D-       - MUST BRING THIS RESTART IN ${RUN_DIR_PATH}
350  #D-
351  #D-       - ELSE BRING RESTART FROM ${JobName} IN ${RUN_DIR_PATH}
352  # ------------------------------------------------------------------
353  IGCM_comp_GetInputRestartFiles
354
355  # ------------------------------------------------------------------
356  # Test if all was right before Update
357  # ------------------------------------------------------------------
358  IGCM_debug_Verif_Exit
359
360  #D-
361  # ------------------------------------------------------------------
362  #D-   4.9 ACTIVATE RUNNING ENVIRONNEMENT VARIABLES
363  # ------------------------------------------------------------------
364  IGCM_sys_activ_variables
365
366  #D-
367  # ------------------------------------------------------------------
368  #D-   4.10 UPDATE ParametersFiles
369  #D-        - (.def, namelist ...)
370  #D-        - with current value of (ORCA_NIT00, ORCA_LRSTAR, RAZ_DATE, ...)
371  # ------------------------------------------------------------------
372  IGCM_comp_Update
373
374  # ------------------------------------------------------------------
375  # Test if all was right before execution
376  # ------------------------------------------------------------------
377  IGCM_debug_Verif_Exit
378
379  #D-
380  #D---------------------------------------------------------------------==
381  #D- 5. EXECUTION
382  #D---------------------------------------------------------------------==
383
384  echo
385  echo "#######################################"
386  echo "#      DIR BEFORE RUN EXECUTION       #"
387  echo "#######################################"
388  echo
389  ls -lrt
390
391  echo "========================================================================"
392  if [ ${DRYRUN} -le 1 ] ; then
393    REAL_DATE_INIT=$( date )
394    echo                                                                                   > ${Exe_Output}
395    echo "#######################################"                                        >> ${Exe_Output}
396    echo "EXECUTION of : ${EXECUTION}"
397    echo "EXECUTION of : ${EXECUTION}"                                                    >> ${Exe_Output}
398    echo                                                                                  >> ${Exe_Output}
399    typeset RET
400    RUN_DATE_BEGIN=$( date '+%Y-%m-%dT%H:%M:%S' )
401    ${EXECUTION}  >> ${Exe_Output} 2>&1
402    RET=$?
403    RUN_DATE_END=$( date '+%Y-%m-%dT%H:%M:%S' )
404    if [ ${RET} -gt 0 ] ; then
405      echo "Return code of executable :" ${RET}
406      IGCM_debug_Exit "EXECUTABLE"
407      IGCM_sys_Mkdir ${SUBMIT_DIR}/Debug
408      IGCM_sys_Cp ${Exe_Output} ${SUBMIT_DIR}/Debug/${PREFIX}_${Exe_Output}_error
409      ExecutionFail=true
410    fi
411    echo                                            >> ${Exe_Output}
412    echo "#######################################"  >> ${Exe_Output}
413    echo "libIGCM RunDateBegin=${RUN_DATE_BEGIN}"   >> ${Exe_Output}
414    echo "libIGCM RunDateEnd=${RUN_DATE_END}"       >> ${Exe_Output}
415    echo                                            >> ${Exe_Output}
416  else
417    echo "EXECUTION of : ${EXECUTION} simulated for DRYRUN = " $DRYRUN
418    echo "EXECUTION of : ${EXECUTION} simulated for DRYRUN = " $DRYRUN > ${Exe_Output}
419    if ( $DEBUG_debug ) ; then
420      echo "FOR EXECUTION DRYRUN mode = " $DRYRUN >> stack
421    fi
422  fi
423  echo "========================================================================"
424
425  echo
426  echo "#######################################"
427  echo "#       DIR AFTER RUN EXECUTION       #"
428  echo "#######################################"
429  echo
430  ls -lrt
431
432  #D-
433  #D---------------------------------------------------------------------==
434  #D- 6. POST EXECUTION OPERATIONS
435  #D---------------------------------------------------------------------==
436
437  #D-
438  # ------------------------------------------------------------------
439  #D-   6.1. DESACTIVATE RUNNING ENVIRONNEMENT VARIABLES
440  # ------------------------------------------------------------------
441  IGCM_sys_desactiv_variables
442
443  #D-
444  # ------------------------------------------------------------------
445  #D-   6.2. SAVE OUTPUTS
446  #D-        - On archive/storage machine
447  #D-        - netcdf outputs, restarts and text files of models
448  # ------------------------------------------------------------------
449  IGCM_comp_Finalize
450
451  # ------------------------------------------------------------------
452  # Test if all was right after Finalize
453  # ------------------------------------------------------------------
454  IGCM_debug_Verif_Exit
455
456  #D-
457  # ------------------------------------------------------------------
458  #D-   6.3. CONFIGURE POST-PROCESSING
459  #D-        - asynchronous post-treatment (Level 0) when necessary (rebuild/pack)
460  #D-        - classic post-treatment      (Level 1) when necessary (TimeSeries/Seasonal)
461  # ------------------------------------------------------------------
462  IGCM_post_Configure
463
464  #D-
465  # ------------------------------------------------------------------
466  #D-   6.4. SUBMIT POST-PROCESSING
467  #D-        - remote rebuild or/and post-treatment process if necessary
468  # ------------------------------------------------------------------
469  IGCM_post_Submit
470
471  #D-
472  # ------------------------------------------------------------------
473  #D-   6.5. FINALYZE CURRENT LOOP AND PREPARE NEXT ONE
474  #D-        - Manage executable size to save only different binary
475  #D-        - Write in run.card user, system and elapse time
476  #D-        - Check that everything went well ! No ? then we stop.
477  #D-        - Determine next computed period
478  # ------------------------------------------------------------------
479  IGCM_config_PeriodEnd
480
481  echo "Ending iteration ${Period}"
482  (( Period = Period + 1 ))
483
484  # End loop if date end is reached
485  if [ ${SimulationLengthInDays} -ge ${ExperienceLengthInDays} ] ; then
486    break ;
487  fi
488
489  PeriodContinue=false
490  if [ ${Period} -le ${PeriodNb} ]; then
491    PeriodContinue=true
492  fi
493done
494
495#D-
496#D---------------------------------------------------------------------==
497#D- 7. SUBMIT NEXT JOB OR SEND MAIL IF SIMULATION IS OVER.
498#D---------------------------------------------------------------------==
499IGCM_config_Finalize
500
501date
Note: See TracBrowser for help on using the repository browser.