source: trunk/libIGCM/AA_monitoring @ 717

Last change on this file since 717 was 712, checked in by aclsce, 12 years ago
  • Added TaskType? variable in jobs to specify the type of the job (computing or post-processing).
  • Added the use of TaskType? variable in Curie libsys to source the appropriate .atlas_env* configuration file.
  • Property licence set to
    The following licence information concerns ONLY the libIGCM tools
    ==================================================================

    Copyright © Centre National de la Recherche Scientifique CNRS
    Commissariat à l'Énergie Atomique CEA

    libIGCM : Library for Portable Models Computation of IGCM Group.

    IGCM Group is the french IPSL Global Climate Model Group.

    This library is a set of shell scripts and functions whose purpose is
    the management of the initialization, the launch, the transfer of
    output files, the post-processing and the monitoring of datas produce
    by any numerical program on any plateforme.

    This software is governed by the CeCILL license under French law and
    abiding by the rules of distribution of free software. You can use,
    modify and/ or redistribute the software under the terms of the CeCILL
    license as circulated by CEA, CNRS and INRIA at the following URL
    "http://www.cecill.info".

    As a counterpart to the access to the source code and rights to copy,
    modify and redistribute granted by the license, users are provided only
    with a limited warranty and the software's author, the holder of the
    economic rights, and the successive licensors have only limited
    liability.

    In this respect, the user's attention is drawn to the risks associated
    with loading, using, modifying and/or developing or reproducing the
    software by the user in light of its specific status of free software,
    that may mean that it is complicated to manipulate, and that also
    therefore means that it is reserved for developers and experienced
    professionals having in-depth computer knowledge. Users are therefore
    encouraged to load and test the software's suitability as regards their
    requirements in conditions enabling the security of their systems and/or
    data to be ensured and, more generally, to use and operate it in the
    same conditions as regards security.

    The fact that you are presently reading this means that you have had
    knowledge of the CeCILL license and that you accept its terms.
  • Property svn:keywords set to Revision Author Date
File size: 15.3 KB
Line 
1#-Q- cesium #!/bin/ksh
2#-Q- cesium ######################
3#-Q- cesium ## CESIUM   CEA ##
4#-Q- cesium ######################
5#-Q- cesium #MSUB -r MONITORING     # Nom du job               
6#-Q- cesium #MSUB -N 1              # Reservation du noeud
7#-Q- cesium #MSUB -n 1              # Reservation du processus
8#-Q- cesium #MSUB -T 86400          # Limite de temps elapsed du job
9#-Q- cesium #MSUB -E "-j o"
10#-Q- cesium #MSUB -E "-S /bin/ksh"
11#-Q- curie ######################
12#-Q- curie ## CURIE   TGCC/CEA ##
13#-Q- curie ######################
14#-Q- curie #MSUB -r MONITORING     # Nom du job               
15#-Q- curie #MSUB -eo
16#-Q- curie #MSUB -n 1              # Reservation du processus
17#-Q- curie #MSUB -T 86400          # Limite de temps elapsed du job
18#-Q- curie #MSUB -E '--no-requeue'
19#-Q- curie #MSUB -q large
20#-Q- curie #MSUB -Q normal
21#-Q- curie set +x
22#-Q- platine #!/usr/bin/ksh
23#-Q- platine ##################
24#-Q- platine ## PLATINE   CEA ##
25#-Q- platine ##################
26#-Q- platine #BSUB -J MONITORING             # Nom du job
27#-Q- platine #BSUB -N                        # message a la fin du job
28#-Q- platine #BSUB -n 1                      # reservation des processeurs pour le job
29#-Q- platine #BSUB -W 1:00                   # Limite temps
30#-Q- platine #BSUB -q post              # Passage en queue post
31#-Q- platine #BSUB -L /bin/ksh
32#-Q- sx8brodie #!/bin/ksh
33#-Q- sx8brodie #######################
34#-Q- sx8brodie ## SX8BRODIE   IDRIS ##
35#-Q- sx8brodie #######################
36#-Q- sx8brodie # Temps Elapsed max. d'une requete hh:mm:ss
37#-Q- sx8brodie # @ wall_clock_limit = 1:00:00
38#-Q- sx8brodie # Nom du travail LoadLeveler
39#-Q- sx8brodie # @ job_name   = MONITORING
40#-Q- sx8brodie # Fichier de sortie standard du travail       
41#-Q- sx8brodie # @ output     = $(job_name).$(jobid)
42#-Q- sx8brodie # Fichier de sortie d'erreur du travail
43#-Q- sx8brodie # @ error      =  $(job_name).$(jobid)
44#-Q- sx8brodie # pour recevoir un mail en cas de depassement du temps Elapsed (ou autre pb.)
45#-Q- sx8brodie # @ notification = error
46#-Q- sx8brodie # @ environment  = $POST_DIR ; $SUBMIT_DIR ; $libIGCM ;  $libIGCM_SX ; $R_INIT ; $R_BC ; $StandAlone ; $RESOL_ATM ; $RESOL_OCE ; $RESOL_ICE ; $RESOL_MBG ; $RESOL_SRF ; $R_SAVE ; $config_UserChoices_JobName ; $config_UserChoices_TagName ; $YEARS ; $MASTER
47#-Q- sx8brodie # @ queue
48#-Q- aix6 #!/bin/ksh
49#-Q- aix6 #######################
50#-Q- aix6 ##   VARGAS   IDRIS  ##
51#-Q- aix6 #######################
52#-Q- aix6 # Temps Elapsed max. d'une requete hh:mm:ss
53#-Q- aix6 # @ wall_clock_limit = 1:00:00
54#-Q- aix6 # Nom du travail LoadLeveler
55#-Q- aix6 # @ job_name   = MONITORING
56#-Q- aix6 # Fichier de sortie standard du travail       
57#-Q- aix6 # @ output     = $(job_name).$(jobid)
58#-Q- aix6 # Fichier de sortie d'erreur du travail
59#-Q- aix6 # @ error      =  $(job_name).$(jobid)
60#-Q- aix6 # pour recevoir un mail en cas de depassement du temps Elapsed (ou autre pb.)
61#-Q- aix6 # @ notification = error
62#-Q- aix6 # @ environment  = $POST_DIR ; $SUBMIT_DIR ; $libIGCM ; $libIGCM_SX ; $R_INIT ; $R_BC ; $StandAlone ; $RESOL_ATM ; $RESOL_OCE ; $RESOL_ICE ; $RESOL_MBG ; $RESOL_SRF ; $R_SAVE ; $config_UserChoices_JobName ; $config_UserChoices_TagName ; $YEARS ; $MASTER
63#-Q- aix6 # @ queue
64#-Q- ulam #!/bin/ksh
65#-Q- ulam #######################
66#-Q- ulam ##   ULAM   IDRIS  ##
67#-Q- ulam #######################
68#-Q- ulam # Temps Elapsed max. d'une requete hh:mm:ss
69#-Q- ulam # @ wall_clock_limit = 1:00:00
70#-Q- ulam # Nom du travail LoadLeveler
71#-Q- ulam # @ job_name   = MONITORING
72#-Q- ulam # Fichier de sortie standard du travail       
73#-Q- ulam # @ output     = $(job_name).$(jobid)
74#-Q- ulam # Fichier de sortie d'erreur du travail
75#-Q- ulam # @ error      =  $(job_name).$(jobid)
76#-Q- ulam # pour recevoir un mail en cas de depassement du temps Elapsed (ou autre pb.)
77#-Q- ulam # @ notification = error
78#-Q- ulam # @ environment  = $POST_DIR ; $SUBMIT_DIR ; $libIGCM ; $libIGCM_SX ; $R_INIT ; $R_BC ; $StandAlone ; $RESOL_ATM ; $RESOL_OCE ; $RESOL_ICE ; $RESOL_MBG ; $RESOL_SRF ; $R_SAVE ; $config_UserChoices_JobName ; $config_UserChoices_TagName ; $YEARS ; $MASTER
79#-Q- ulam # @ queue
80#-Q- sx8mercure #!/bin/ksh
81#-Q- sx8mercure ######################
82#-Q- sx8mercure ## SX8MERCURE   CEA ##
83#-Q- sx8mercure ######################
84#-Q- sx8mercure #PBS -N  MONITORING          # Nom du job
85#-Q- sx8mercure #PBS -j o                    # regroupement des stdout et stderr
86#-Q- sx8mercure #PBS -S /usr/bin/ksh         # shell de soumission
87#-Q- sx8mercure #PBS -l memsz_job=1gb        # Limite memoire a 1 Go
88#-Q- sx8mercure #PBS -l cputim_job=1:00:00   # Limite temps a 2 heures
89#-Q- sx8mercure #PBS -q scalaire
90#-Q- sx8mercure #PBS -r n
91#-Q- sx9mercure #!/bin/ksh
92#-Q- sx9mercure ######################
93#-Q- sx9mercure ## SX9MERCURE  CCRT ##
94#-Q- sx9mercure ######################
95#-Q- sx9mercure #PBS -N  MONITORING          # Nom du job
96#-Q- sx9mercure #PBS -j o                    # regroupement des stdout et stderr
97#-Q- sx9mercure #PBS -S /usr/bin/ksh         # shell de soumission
98#-Q- sx9mercure #PBS -l memsz_job=1gb        # Limite memoire a 1 Go
99#-Q- sx9mercure #PBS -l elapstim_req=1:00:00   # Limite temps a 2 heures
100#-Q- sx9mercure #PBS -q scalaire
101#-Q- sx9mercure #PBS -r n
102#-Q- titane #!/bin/ksh
103#-Q- titane ######################
104#-Q- titane ## TITANE   CEA ##
105#-Q- titane ######################
106#-Q- titane #MSUB -r MONITORING     # Nom du job               
107#-Q- titane #MSUB -eo
108#-Q- titane #MSUB -n 1              # Reservation du processus
109#-Q- titane #MSUB -T 86400          # Limite de temps elapsed du job
110#-Q- titane #MSUB -q mono
111#-Q- titane #MSUB -E '-rn'
112#-Q- lxiv8 ######################
113#-Q- lxiv8 ## OBELIX      LSCE ##
114#-Q- lxiv8 ######################
115#-Q- lxiv8 #PBS -N MONITORING
116#-Q- lxiv8 #PBS -m a
117#-Q- lxiv8 #PBS -j oe
118#-Q- lxiv8 #PBS -q medium
119#-Q- lxiv8 #PBS -o MONITORING.$$
120#-Q- lxiv8 #PBS -S /bin/ksh
121#-Q- default #!/bin/ksh
122#-Q- default ##################
123#-Q- default ## DEFAULT HOST ##
124#-Q- default ##################
125
126#**************************************************************
127# Author: Sebastien Denvil
128# Contact: Sebastien.Denvil__at__ipsl.jussieu.fr
129# $Revision::                                          $ Revision of last commit
130# $Author::                                            $ Author of last commit
131# $Date::                                              $ Date of last commit
132# IPSL (2006)
133#  This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
134#
135#**************************************************************
136
137#set -eu
138#set -vx
139
140date
141
142#-Q- sx8brodie export OMP_NUM_THREADS=1
143#-Q- aix6 export OMP_NUM_THREADS=1
144#-Q- ulam export OMP_NUM_THREADS=1
145
146#D- Task type (computing or post-processing)
147TaskType=post-processing
148
149########################################################################
150
151#D- Flag to determine if this job in a standalone mode
152#D- Default : value from AA_job if any
153StandAlone=${StandAlone:=true}
154
155#D- Path to libIGCM
156#D- Default : value from AA_job if any
157# WARNING For StandAlone use : To run this script on some machine (ulam and cesium)
158# WARNING you must check MirrorlibIGCM variable in sys library.
159# WARNING If this variable is true, you must use libIGCM_POST path instead
160# WARNING of your running libIGCM directory.
161libIGCM=${libIGCM:=::modipsl::/libIGCM}
162
163#D- Increased verbosity (1, 2, 3)
164#D- Default : value from AA_job if any
165Verbosity=${Verbosity:=3}
166
167#D- Low level debug : to bypass lib test checks and stack construction
168#D- Default : value from AA_job if any
169DEBUG_debug=${DEBUG_debug:=false}
170
171#D- TEMPORARY Flag to determine atmospheric resolution
172#D- Default : value from atmospheric driver if any
173RESOL_ATM=${RESOL_ATM:=ALL}
174
175#D- Flag to determine surface resolution
176#D- Default : value from surface driver if any
177RESOL_SRF=${RESOL_SRF:=ALL}
178
179#D- Flag to determine surface resolution
180#D- Default : value from surface driver if any
181RESOL_SBG=${RESOL_SBG:=ALL}
182
183#D- TEMPORARY Flag to determine ocean resolution
184#D- Default : value from ocean driver if any
185RESOL_OCE=${RESOL_OCE:=ORCA2}
186
187#D- TEMPORARY Flag to determine ice resolution
188#D- Default : value from ice driver if any
189RESOL_ICE=${RESOL_ICE:=ORCA2}
190
191#D- TEMPORARY Flag to determine marine biogeochemistry resolution
192#D- Default : value from ice driver if any
193RESOL_MBG=${RESOL_MBG:=ORCA2}
194
195######################################################################
196
197. ${libIGCM}/libIGCM_debug/libIGCM_debug.ksh
198      ( ${DEBUG_debug} ) && IGCM_debug_Check
199. ${libIGCM}/libIGCM_card/libIGCM_card.ksh
200     ( ${DEBUG_debug} ) && IGCM_card_Check
201. ${libIGCM}/libIGCM_date/libIGCM_date.ksh
202     ( ${DEBUG_debug} ) && IGCM_date_Check
203#-------
204. ${libIGCM}/libIGCM_sys/libIGCM_sys.ksh
205
206######################################################################
207
208#set -vx
209
210#===========================================
211RUN_DIR=${RUN_DIR_PATH}
212IGCM_sys_MkdirWork ${RUN_DIR}
213IGCM_sys_Cd ${RUN_DIR}
214
215# ------------------------------------------------------------------
216# Test if all was right before proceeding further
217# ------------------------------------------------------------------
218IGCM_debug_Verif_Exit_Post
219
220if [ ${StandAlone} = true ] ; then
221    CARD_DIR=${SUBMIT_DIR}
222else
223    CARD_DIR=${RUN_DIR}
224    IGCM_sys_Get_Master ${SUBMIT_DIR}/config.card ${RUN_DIR_PATH}
225    IGCM_sys_Get_Master ${SUBMIT_DIR}/run.card    ${RUN_DIR_PATH}
226    IGCM_sys_Get_Master ${SUBMIT_DIR}/COMP        ${RUN_DIR_PATH}
227    IGCM_sys_Get_Master ${SUBMIT_DIR}/POST        ${RUN_DIR_PATH}
228fi
229
230#
231# First of all
232#
233IGCM_card_DefineArrayFromSection       ${CARD_DIR}/config.card UserChoices
234typeset option
235for option in ${config_UserChoices[*]} ; do
236    IGCM_card_DefineVariableFromOption ${CARD_DIR}/config.card UserChoices ${option}
237done
238
239IGCM_card_DefineArrayFromSection       ${CARD_DIR}/config.card Ensemble
240typeset option
241for option in ${config_Ensemble[*]} ; do
242    IGCM_card_DefineVariableFromOption ${CARD_DIR}/config.card Ensemble ${option}
243done
244IGCM_card_DefineArrayFromSection       ${CARD_DIR}/config.card ListOfComponents
245
246#==================================
247#R_SAVE : Job output directory
248#R_BUFR : Job output buffered directory
249if ( [ ! X${config_UserChoices_SpaceName} = X ] && [ ! X${config_UserChoices_ExperimentName} = X ] ) ; then
250  FreeName=$( echo ${config_UserChoices_JobName} | sed 's/.*_//' )
251  if ( [ ! X${config_Ensemble_EnsembleName} = X ] && [ ! X${config_Ensemble_EnsembleDate} = X ] ) ; then
252    R_SAVE=${R_OUT}/${config_UserChoices_TagName}/${config_UserChoices_SpaceName}/${config_UserChoices_ExperimentName}/${config_Ensemble_EnsembleName}/${config_Ensemble_EnsembleDate}/${FreeName}
253    R_FIGR=${R_FIG}/${config_UserChoices_TagName}/${config_UserChoices_SpaceName}/${config_UserChoices_ExperimentName}/${config_Ensemble_EnsembleName}/${config_Ensemble_EnsembleDate}/${FreeName}
254    R_BUFR=${R_BUF}/${config_UserChoices_TagName}/${config_UserChoices_SpaceName}/${config_UserChoices_ExperimentName}/${config_Ensemble_EnsembleName}/${config_Ensemble_EnsembleDate}/${FreeName}
255    R_DODS=${config_UserChoices_TagName}/${config_UserChoices_SpaceName}/${config_UserChoices_ExperimentName}/${config_Ensemble_EnsembleName}/${config_Ensemble_EnsembleDate}/${FreeName}
256  else
257    R_SAVE=${R_OUT}/${config_UserChoices_TagName}/${config_UserChoices_SpaceName}/${config_UserChoices_ExperimentName}/${FreeName}
258    R_FIGR=${R_FIG}/${config_UserChoices_TagName}/${config_UserChoices_SpaceName}/${config_UserChoices_ExperimentName}/${FreeName}
259    R_BUFR=${R_BUF}/${config_UserChoices_TagName}/${config_UserChoices_SpaceName}/${config_UserChoices_ExperimentName}/${FreeName}
260    R_DODS=${config_UserChoices_TagName}/${config_UserChoices_SpaceName}/${config_UserChoices_ExperimentName}/${FreeName}
261  fi
262else
263  if ( [ ! X${config_Ensemble_EnsembleName} = X ] && [ ! X${config_Ensemble_EnsembleDate} = X ] ) ; then
264    R_SAVE=${R_OUT}/${config_UserChoices_TagName}/${config_Ensemble_EnsembleName}/${config_Ensemble_EnsembleDate}/${config_UserChoices_JobName}
265    R_FIGR=${R_FIG}/${config_UserChoices_TagName}/${config_Ensemble_EnsembleName}/${config_Ensemble_EnsembleDate}/${config_UserChoices_JobName}
266    R_BUFR=${R_BUF}/${config_UserChoices_TagName}/${config_Ensemble_EnsembleName}/${config_Ensemble_EnsembleDate}/${config_UserChoices_JobName}
267    R_DODS=${config_UserChoices_TagName}/${config_Ensemble_EnsembleName}/${config_Ensemble_EnsembleDate}/${config_UserChoices_JobName}
268  else
269    R_SAVE=${R_OUT}/${config_UserChoices_TagName}/${config_UserChoices_JobName}
270    R_FIGR=${R_FIG}/${config_UserChoices_TagName}/${config_UserChoices_JobName}
271    R_BUFR=${R_BUF}/${config_UserChoices_TagName}/${config_UserChoices_JobName}
272    R_DODS=${config_UserChoices_TagName}/${config_UserChoices_JobName}
273  fi
274fi
275
276IGCM_sys_MkdirArchive ${R_FIGR}
277#
278IGCM_sys_TestDirArchive ${R_FIGR}/MONITORING
279if [ $? = 0 ] ; then
280    IGCM_debug_Print 1 "Get MONITORING directory from archive"
281    IGCM_sys_Get_Dir ${R_FIGR}/MONITORING ${RUN_DIR}
282else
283    IGCM_debug_Print 1 "MONITORING first pass. Nothing has been done before"
284fi
285# --------------------------------------------
286# Insert your commands between III...III lines
287# and precise produced directories to save
288# IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII
289
290for comp in ${config_ListOfComponents[*]} ; do
291    IGCM_debug_Print 1 "################## Component: ${comp} ######################"
292    liste_file_monitoring=""
293    IGCM_card_DefineArrayFromOption ${CARD_DIR}/config.card ListOfComponents ${comp}
294    eval compname=\${config_ListOfComponents_${comp}[0]}                > /dev/null 2>&1
295
296    PATH_monitoring_file=""
297    eval monitoring_file=monitoring01_${compname}_\${RESOL_${comp}}.cfg > /dev/null 2>&1
298    if [[ -d ${CARD_DIR}/POST && -f ${CARD_DIR}/POST/monitoring01_${compname}.cfg ]] ; then
299        PATH_monitoring_file=${CARD_DIR}/POST/monitoring01_${compname}.cfg
300    elif [ -f ${FER_ATLAS}/${monitoring_file} ] ; then
301        PATH_monitoring_file=${FER_ATLAS}/${monitoring_file}
302    else
303        IGCM_debug_Print 1 "No monitoring file found for this component. Was expecting ${monitoring_file}"
304        IGCM_debug_Print 1 "Step to next component"
305        continue
306    fi
307    #
308    if [ X${PATH_monitoring_file} != X"" ] ; then
309        IGCM_debug_Print 1 "Monitoring file used : ${PATH_monitoring_file}"
310        IGCM_debug_Print 1 "Determine which files we need."
311        . monitoring01 -l2 --listcommand 'IGCM_sys_RshArchive ls' ${PATH_monitoring_file} ${R_SAVE}/${comp}/Analyse/TS_MO
312        liste_file_monitoring=$( . monitoring01 -q -l2 --listcommand 'IGCM_sys_RshArchive ls' ${PATH_monitoring_file} ${R_SAVE}/${comp}/Analyse/TS_MO )
313    fi
314    #
315    if [ ! "X${liste_file_monitoring}" = X ] ; then
316        IGCM_sys_Get /l liste_file_monitoring ${RUN_DIR}
317        IGCM_debug_Print 1 "monitoring01 -c ${CARD_DIR} -p ${comp} --time -t \"${config_UserChoices_JobName} monitoring\" -o ${RUN_DIR}/MONITORING ${PATH_monitoring_file} ."
318        IGCM_debug_Print 1 "monitoring01 starts ................................................."
319        monitoring01 -c ${CARD_DIR} -p ${comp} --time -t "${config_UserChoices_JobName} monitoring" -o ${RUN_DIR}/MONITORING ${PATH_monitoring_file} .
320    else
321        IGCM_debug_Print 1 "No time series detected by this command :"
322        IGCM_debug_Print 1 "monitoring01 -l2 --listcommand 'IGCM_sys_RshArchive ls' ${PATH_monitoring_file} ${R_SAVE}/${comp}/Analyse/TS_MO"
323        . monitoring01 -l2 --listcommand 'IGCM_sys_RshArchive ls' ${PATH_monitoring_file} ${R_SAVE}/${comp}/Analyse/TS_MO
324    fi
325    #
326done
327
328# IIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII
329
330# Save files
331IGCM_sys_Put_Dir MONITORING ${R_FIGR}
332
333# Dods copy
334IGCM_sys_Put_Dods MONITORING
335
336# Clean RUN_DIR_PATH (necessary for cesium and titane only)
337IGCM_sys_RmRunDir -Rf ${RUN_DIR_PATH}
Note: See TracBrowser for help on using the repository browser.