source: trunk/libIGCM/AA_create_se @ 1603

Last change on this file since 1603 was 1603, checked in by aclsce, 8 months ago
  • Changed libIGCM behaviour in case of pb in pack_output.job : now, we exit from the job if there is any problem, whatever we are in PROD, DEVT, TEST mode.
  • Specification of pack frequency per file :

In order to reduce the number of inodes, it is possible to specify by file the frequency of packing. The syntax to do that is in the 4th column of OutputFiles? section of the component.card, for example as follows in lmdz.card :
[OutputFiles?]
List= (histmth.nc, ${R_OUT_ATM_O_M}/${PREFIX}_1M_histmth.nc, Post_1M_histmth, 100Y), \

(histday.nc, ${R_OUT_ATM_O_D}/${PREFIX}_1D_histday.nc, Post_1D_histday, 10Y), \

...

In this example, histmth files will be packed every 100 years and histday files will be packed every 10 years.
The pack frequency you defined in config.card is the frequency of pack by default, that means if a specific frequency of pack is specified for a file in a component.card, this file will be packed at the specific frequency whereas all other files will be packed at global pack frequency (specified in config.card) and in this case, the frequency pack (from the config.card) is the frequency the pack_output job will be launched at.
There is a constraint to use this fonctionality : the Packfrequency you defined in config.card must be greater or equal to the pack frequencies you specified for each type of file in component.card, otherwise the computing job will be stopped (with an explicit error message).
Surpack mode :
A surpack mode functionality is available through the use of pack_output.job. To enable this functionality, you have to put "surpack_mode=y" (default value is n). The way to use is similar to restart post-processing pack_output jobs, as indicated here : ​http://forge.ipsl.jussieu.fr/igcmg_doc/wiki/Doc/CheckDebug#RestartPack_output. You can either use a global pack frequency in config.card or specific pack frequency per file, as explained above.

  • Property licence set to
    The following licence information concerns ONLY the libIGCM tools
    ==================================================================

    Copyright © Centre National de la Recherche Scientifique CNRS
    Commissariat à l'Énergie Atomique CEA

    libIGCM : Library for Portable Models Computation of IGCM Group.

    IGCM Group is the french IPSL Global Climate Model Group.

    This library is a set of shell scripts and functions whose purpose is
    the management of the initialization, the launch, the transfer of
    output files, the post-processing and the monitoring of datas produce
    by any numerical program on any plateforme.

    This software is governed by the CeCILL license under French law and
    abiding by the rules of distribution of free software. You can use,
    modify and/ or redistribute the software under the terms of the CeCILL
    license as circulated by CEA, CNRS and INRIA at the following URL
    "http://www.cecill.info".

    As a counterpart to the access to the source code and rights to copy,
    modify and redistribute granted by the license, users are provided only
    with a limited warranty and the software's author, the holder of the
    economic rights, and the successive licensors have only limited
    liability.

    In this respect, the user's attention is drawn to the risks associated
    with loading, using, modifying and/or developing or reproducing the
    software by the user in light of its specific status of free software,
    that may mean that it is complicated to manipulate, and that also
    therefore means that it is reserved for developers and experienced
    professionals having in-depth computer knowledge. Users are therefore
    encouraged to load and test the software's suitability as regards their
    requirements in conditions enabling the security of their systems and/or
    data to be ensured and, more generally, to use and operate it in the
    same conditions as regards security.

    The fact that you are presently reading this means that you have had
    knowledge of the CeCILL license and that you accept its terms.
  • Property svn:keywords set to Revision Author Date
File size: 31.0 KB
RevLine 
[622]1#-Q- curie ######################
2#-Q- curie ## CURIE   TGCC/CEA ##
3#-Q- curie ######################
[904]4#-Q- curie #MSUB -r SE             # Nom du job
[1468]5#-Q- curie #MSUB -o SE.out_%I
6#-Q- curie #MSUB -e SE.out_%I
[622]7#-Q- curie #MSUB -n 1              # Reservation du processus
[880]8#-Q- curie #MSUB -T 36000          # Limite de temps elapsed du job
[1154]9#-Q- curie #MSUB -q ::default_node::
[1274]10#-Q- curie #MSUB -c ::default_core::
[704]11#-Q- curie #MSUB -Q normal
[837]12#-Q- curie #MSUB -A ::default_project::
[681]13#-Q- curie set +x
[1437]14#-Q- irene ######################
15#-Q- irene ## IRENE   TGCC/CEA ##
16#-Q- irene ######################
17#-Q- irene #MSUB -r SE            # Job name
[1468]18#-Q- irene #MSUB -o SE.out_%I
19#-Q- irene #MSUB -e SE.out_%I
[1437]20#-Q- irene #MSUB -n 1
21#-Q- irene #MSUB -T 36000         # Maximum elapsed time
[1468]22#-Q- irene #MSUB -q ::default_node::
23#-Q- irene #MSUB -c ::default_core::
[1437]24#-Q- irene #MSUB -Q normal
[1468]25#-Q- irene #MSUB -A ::default_post_project::
[1460]26#-Q- irene #MSUB -m store,work,scratch
[1437]27#-Q- irene set +x
[1525]28#-Q- irene-amd ######################
29#-Q- irene-amd ## IRENE   TGCC/CEA ##
30#-Q- irene-amd ######################
31#-Q- irene-amd #MSUB -r SE     # Job name
32#-Q- irene-amd #MSUB -o SE.out_%I
33#-Q- irene-amd #MSUB -e SE.out_%I
34#-Q- irene-amd #MSUB -n 1              # Number of cores
35#-Q- irene-amd #MSUB -T 36000          # Maximum elapsed time
36#-Q- irene-amd #MSUB -q ::default_node::
37#-Q- irene-amd #MSUB -c ::default_core::
38#-Q- irene-amd #MSUB -Q normal
39#-Q- irene-amd #MSUB -A ::default_post_project::
40#-Q- irene-amd #MSUB -m store,work,scratch
41#-Q- irene-amd set +x
[1491]42#-Q- jeanzay #!/bin/ksh
43#-Q- jeanzay ######################
44#-Q- jeanzay ## JEANZAY    IDRIS ##
45#-Q- jeanzay ######################
46#-Q- jeanzay #SBATCH --job-name=SE         # Job Name
47#-Q- jeanzay #SBATCH --output=SE.out_%I    # standard output
48#-Q- jeanzay #SBATCH --error=SE.out_%I     # error output
[1494]49#-Q- jeanzay #SBATCH -N  1                        # Number of core
50#-Q- jeanzay #SBATCH --partition=prepost          # Post-processing partition
[1491]51#-Q- jeanzay #SBATCH --time=10:00:00       # Wall clock limit (seconds)
52#-Q- jeanzay #SBATCH --account ::default_project::@cpu
53#-Q- jeanzay set +x
[770]54#-Q- ada #!/bin/ksh
55#-Q- ada #######################
[929]56#-Q- ada ## ADA         IDRIS ##
[770]57#-Q- ada #######################
[1409]58#-Q- ada # @ job_type = mpich
[848]59#-Q- ada # @ requirements = (Feature == "prepost")
[770]60#-Q- ada # Temps Elapsed max. d'une requete hh:mm:ss
61#-Q- ada # @ wall_clock_limit = 10:00:00
[1336]62#-Q- ada # Memory required for ncrcat
63#-Q- ada # @ as_limit = 30Gb
[770]64#-Q- ada # Nom du travail LoadLeveler
65#-Q- ada # @ job_name   = SE
[904]66#-Q- ada # Fichier de sortie standard du travail
[770]67#-Q- ada # @ output     = $(job_name).$(jobid)
68#-Q- ada # Fichier de sortie d'erreur du travail
69#-Q- ada # @ error      =  $(job_name).$(jobid)
70#-Q- ada # pour recevoir un mail en cas de depassement du temps Elapsed (ou autre pb.)
71#-Q- ada # @ notification = error
[1290]72#-Q- ada # @ environment  = $DEBUG_debug ; $BigBrother ; $postProcessingStopLevel ; $MODIPSL ; $libIGCM ; $libIGCM_SX ; $SUBMIT_DIR ; $POST_DIR ; $Script_Post_Output ; $MASTER ; $DateBegin ; $PeriodDateEnd ; $StandAlone ; $RESOL_ATM ; $RESOL_OCE ; $RESOL_ICE ; $RESOL_MBG ; $RESOL_SRF ; $RESOL_SBG ; wall_clock_limit=$(wall_clock_limit)
[770]73#-Q- ada # @ queue
[1568]74#-Q- obelix ######################
75#-Q- obelix ## OBELIX      LSCE ##
76#-Q- obelix ######################
77#-Q- obelix #PBS -N SE
78#-Q- obelix #PBS -m a
79#-Q- obelix #PBS -j oe
80#-Q- obelix #PBS -q medium
81#-Q- obelix #PBS -o SE.$$
82#-Q- obelix #PBS -S /bin/ksh
[1558]83#-Q- mesoipsl #!/bin/ksh
84#-Q- mesoipsl ######################
85#-Q- mesoipsl ## MESO ESPRI IPSL  ##
86#-Q- mesoipsl ######################
87#-Q- mesoipsl #SBATCH --job-name=TS         # Job Name
88#-Q- mesoipsl #SBATCH --output=TS.out_%I    # standard output
89#-Q- mesoipsl #SBATCH --error=TS.out_%I     # error output
90#-Q- mesoipsl #SBATCH -N  1                        # Number of core
91#-Q- mesoipsl #SBATCH --time=10:00:00       # Wall clock limit (seconds)
[1184]92#-Q- ifort_CICLAD ######################
93#-Q- ifort_CICLAD ##   CICLAD    IPSL ##
94#-Q- ifort_CICLAD ######################
95#-Q- ifort_CICLAD #PBS -N SE
96#-Q- ifort_CICLAD #PBS -m a
97#-Q- ifort_CICLAD #PBS -j oe
[1314]98#-Q- ifort_CICLAD ###PBS -q h12       # Queue for 12 hours at ciclad only
[1184]99#-Q- ifort_CICLAD #PBS -S /bin/ksh
[2]100#-Q- default #!/bin/ksh
101#-Q- default ##################
102#-Q- default ## DEFAULT HOST ##
103#-Q- default ##################
104
[373]105#**************************************************************
106# Author: Sebastien Denvil
107# Contact: Sebastien.Denvil__at__ipsl.jussieu.fr
108# $Revision::                                          $ Revision of last commit
109# $Author::                                            $ Author of last commit
110# $Date::                                              $ Date of last commit
[108]111# IPSL (2006)
112#  This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
[373]113#
114#**************************************************************
[108]115
116#set -eu
[2]117#set -vx
118
119date
120
[770]121#-Q- ada export OMP_NUM_THREADS=1
[2]122
[1356]123#D- Task type DO NOT CHANGE (computing, post-processing or checking)
[712]124TaskType=post-processing
125
[2]126########################################################################
127
128#D- Flag to determine if this job in a standalone mode
129#D- Default : value from AA_job if any
130StandAlone=${StandAlone:=true}
131
[554]132#D- Path to libIGCM
[206]133#D- Default : value from AA_job if any
[229]134libIGCM=${libIGCM:=::modipsl::/libIGCM}
[206]135
[213]136#D- Flag to determine if rebuild process has been performed asynchronously
[904]137#D- Possible value true or false.
[213]138#D- If true raw files has allready been patched by rebuild job
139#D- If false create_ts will patch the raw files
140#D- Default : Value from AA_job if any. Usefull in StandAlone case.
141RebuildFrequency=${RebuildFrequency:=true}
142
[2]143#D- Flag to determine atlas job's output directory
144#D- Default : value from libIGCM_post.ksh if any
[370]145POST_DIR=${POST_DIR:=${PBS_O_WORKDIR:=$(pwd)}}
[2]146
147#D- Increased verbosity (1, 2, 3)
148#D- Default : value from AA_job if any
149Verbosity=${Verbosity:=3}
150
151#D- Low level debug : to bypass lib test checks and stack construction
152#D- Default : value from AA_job if any
153DEBUG_debug=${DEBUG_debug:=false}
154
155#D- TEMPORARY Flag to determine atmospheric resolution
156#D- Default : value from atmospheric driver if any
[135]157RESOL_ATM=${RESOL_ATM:=ALL}
[2]158
[257]159#D- Flag to determine surface resolution
160#D- Default : value from surface driver if any
161RESOL_SRF=ALL
162
163#D- Flag to determine surface resolution
164#D- Default : value from surface driver if any
165RESOL_SBG=ALL
166
[2]167#D- TEMPORARY Flag to determine ocean resolution
168#D- Default : value from ocean driver if any
169RESOL_OCE=${RESOL_OCE:=ORCA2}
170
171#D- TEMPORARY Flag to determine ice resolution
172#D- Default : value from ice driver if any
[1326]173RESOL_ICE=${RESOL_OCE:=ORCA2}
[2]174
[178]175#D- TEMPORARY Flag to determine marine biogeochemistry resolution
176#D- Default : value from ice driver if any
[1326]177RESOL_MBG=${RESOL_OCE:=ORCA2}
[178]178
[2]179########################################################################
180
[108]181. ${libIGCM}/libIGCM_debug/libIGCM_debug.ksh
182. ${libIGCM}/libIGCM_card/libIGCM_card.ksh
183. ${libIGCM}/libIGCM_date/libIGCM_date.ksh
184#-------
185. ${libIGCM}/libIGCM_sys/libIGCM_sys.ksh
[731]186. ${libIGCM}/libIGCM_config/libIGCM_config.ksh
[832]187#-------
[1192]188IGCM_sys_MkdirWork ${RUN_DIR_PATH}
189IGCM_sys_Cd ${RUN_DIR_PATH}
190#-------
[832]191( ${DEBUG_debug} ) && IGCM_debug_Check
192( ${DEBUG_debug} ) && IGCM_card_Check
193( ${DEBUG_debug} ) && IGCM_date_Check
[2]194
195########################################################################
196
197#set -vx
198
[436]199# ------------------------------------------------------------------
200# Test if all was right before proceeding further
201# ------------------------------------------------------------------
[1206]202IGCM_debug_Verif_Exit
[436]203
[2]204if [ ${StandAlone} = true ] ; then
205    CARD_DIR=${SUBMIT_DIR}
206else
[647]207    CARD_DIR=${RUN_DIR_PATH}
[640]208    IGCM_sys_Get_Master ${SUBMIT_DIR}/config.card ${RUN_DIR_PATH}
209    IGCM_sys_Get_Master ${SUBMIT_DIR}/run.card    ${RUN_DIR_PATH}
210    IGCM_sys_Get_Master ${SUBMIT_DIR}/COMP        ${RUN_DIR_PATH}
211    IGCM_sys_Get_Master ${SUBMIT_DIR}/POST        ${RUN_DIR_PATH}
[2]212fi
213
[616]214#==================================
[2]215# First of all
216#
[727]217# Read libIGCM compatibility version in config.card
218# Read UserChoices section
219# Read Ensemble section
220# Read Post section
221# Define all netcdf output directories
[616]222#==================================
[727]223IGCM_config_CommonConfiguration ${CARD_DIR}/config.card
[616]224
[1198]225# ------------------------------------------------------------------
226# Activate BigBrother so as to supervise this job
227# ------------------------------------------------------------------
228IGCM_debug_BigBro_Initialize
229
[616]230#==================================
[633]231# Define default value to keep compatibility with previous card: means before changes due to TGCC
232if [ X${PackDefault} = Xtrue ] ; then
233  [ X${config_Post_PackFrequency} = X ] && config_Post_PackFrequency=${config_Post_RebuildFrequency}
234else
235  config_Post_PackFrequency=NONE
236fi
[616]237
238#==================================
239# If pack is active then PackFrequency overule the config_UserChoices_PeriodLength
240if ( [ ! X${config_Post_PackFrequency} = X${NULL_STR} ] && [ ! X${config_Post_PackFrequency} = XNONE ] ) ; then
241  config_UserChoices_PeriodLength=${config_Post_PackFrequency}
242fi
243
244#==================================
[727]245# Read ListOfComponents section:
[2]246IGCM_card_DefineArrayFromSection   ${CARD_DIR}/config.card ListOfComponents
[727]247
248#==================================
249# Read SeasonalFrequency:
[2]250IGCM_card_DefineVariableFromOption ${CARD_DIR}/config.card Post SeasonalFrequency
[727]251
[2]252#
253# Determine period for seasonnal average. Default : value from AA_job if any
254#
255DateBegin=${DateBegin:=$( IGCM_date_ConvertFormatToGregorian ${config_UserChoices_DateBegin} )}
256#
257PeriodDateEnd=${PeriodDateEnd:=$( IGCM_date_ConvertFormatToGregorian ${config_UserChoices_DateEnd} )}
258#
259freq=$( echo ${config_Post_SeasonalFrequency} |  sed -e "s/[yY]//" )
260#
[149]261an_fin=$(( $( IGCM_date_ConvertGregorianDateToJulian ${PeriodDateEnd} ) / 1000 ))
[2]262an_deb=$( IGCM_date_YearDigit $(( an_fin - freq + 1 )) )
263an_fin=$( IGCM_date_YearDigit ${an_fin} )
264#
265# For each component selected determine which files need post-processing
266#
267for comp in ${config_ListOfComponents[*]} ; do
[785]268  #
269  IGCM_card_DefineArrayFromOption ${CARD_DIR}/config.card ListOfComponents ${comp}
270  eval compname=\${config_ListOfComponents_${comp}[0]} > /dev/null 2>&1
271  #
272  card=${CARD_DIR}/COMP/${compname}.card
273  IGCM_card_DefineArrayFromOption ${card} OutputFiles List
274  #
275  ListFilesName=${compname}_OutputFiles_List
276  eval FileName0=\${${ListFilesName}[0]} > /dev/null 2>&1
277  #
278  if [ X${FileName0} != X${NULL_STR} ] ; then
[2]279    #
[785]280    # INITIALISATION
[2]281    #
[785]282    eval NbFiles=\${#${ListFilesName}[@]} > /dev/null 2>&1
283    typeset i=2
[2]284    #
[785]285    until [ $i -gt $NbFiles ]; do
286      #
287      eval flag_post=\${${ListFilesName}[$i]} > /dev/null 2>&1
288      #
289      if [ X${flag_post} != XNONE ] ; then
290        #
291        # Check if seasonal is required for this file
292        #
293        IGCM_card_DefineArrayFromSection ${card} ${flag_post}
294        #
295        # Seasonal case : If option Seasonal is not found (old cards) put SEASONAL ON by default
296        #
297        # variable option allready typeset above
298        for option in $( eval echo \${${compname}_${flag_post}[*]} ) ; do
299          if [ ${option} = Seasonal ] ; then
300            FoundSeasonal=true
301            IGCM_card_DefineVariableFromOption ${card} ${flag_post} Seasonal
302          fi
303        done
304        #
305        if [ ! X${FoundSeasonal} = Xtrue ] ; then
306            eval ${compname}_${flag_post}_Seasonal=ON
307        fi
308        #
309        if [ ! X$( eval echo \${${compname}_${flag_post}_Seasonal} ) = XON ] ; then
[1603]310            (( i_ = i+1 ))
311            eval Testvar_=\${${ListFilesName}[$i_]} > /dev/null 2>&1
312            eval Testvar=${Testvar_}
313            if [[ ${Testvar} =~ [0-9][DMY]$ ]] ; then
314                (( i = i+4 ))
315            else
[785]316          (( i=i+3 ))
[1603]317            fi
[785]318          continue
319        fi
320        #
321        # CREATE LIST
322        #
323        FILE=$( echo ${flag_post} | awk "-FPost_" '{print $2}' )
324        #
325        DIRECTORY=${R_SAVE}/${comp}/Analyse/SE
326        #
327        TestedFile=${config_UserChoices_JobName}_SE_${an_deb}_${an_fin}_${FILE}.nc
328        #
329        IGCM_sys_TestFileArchive ${DIRECTORY}/${TestedFile}
330        if [ ! $? = 0 ] ; then
331          eval set +A LISTE_FILE_${comp} \$\{LISTE_FILE_${comp}[*]} ${FILE}
[1603]332          (( i_ = i+1 ))
333          eval Testvar_=\${${ListFilesName}[$i_]} > /dev/null 2>&1
334          eval Testvar=${Testvar_}
335          if [[ ${Testvar} =~ [0-9][DMY]$ ]] ; then
336              eval PeriodPack_FileType=${Testvar}
337          else
338              PeriodPack_FileType=${config_Post_PackFrequency}
339          fi
340          eval set +A LISTE_PACKFREQUENCY_${comp} \${LISTE_PACKFREQUENCY_${comp}[*]} ${PeriodPack_FileType}
[785]341          if [ ! ${RebuildFrequency} = true ] ; then
342            IGCM_card_DefineArrayFromOption ${card} Post_${FILE} Patches
343            eval set +A LISTE_PATCHES_${FILE} \$\{${compname}_Post_${FILE}_Patches[*]}
344            if [ X$( eval echo \${LISTE_PATCHES_${FILE}[0]} ) !=  X${NULL_STR} ]; then
345              for Patch in $( eval echo \${LISTE_PATCHES_${FILE}[*]} ); do
346                if [ Xload${Patch} != X ] ; then
347                  . ${libIGCM}/libIGCM_post/IGCM_${Patch}.ksh
348                  eval load${Patch}=loaded
349                fi
350              done
351            fi
352          fi
353        fi
354      fi
[1603]355      (( i_ = i+1 ))
356      eval Testvar_=\${${ListFilesName}[$i_]} > /dev/null 2>&1
357      eval Testvar=${Testvar_}
358      if [[ ${Testvar} =~ [0-9][DMY]$ ]] ; then
359          (( i = i+4 ))
360      else
[785]361      (( i=i+3 ))
[1603]362      fi
[785]363    done # until [ $i -eq $NbFiles ]
364  fi
[150]365done         # for comp
[2]366
367for comp in ${config_ListOfComponents[*]} ; do
[785]368  #
369  eval R_OUT_${comp}=${R_SAVE}/${comp}
370  #
371  i=0
372  for file in $( eval echo \${LISTE_FILE_${comp}[*]} ); do
[1603]373  config_UserChoices_PeriodLength=$( eval echo \${LISTE_PACKFREQUENCY_${comp}[$i]} ) 
[2]374    #
[785]375    # Determine in which output can we find file
376    # IE : Output/MO or Output/DA or ...
[2]377    #
[785]378    FlagDir=$( echo ${file} | awk -F "_" '{print $1}' )
379    case ${FlagDir} in
380      1Y)    FreqDir=YE  ;;
381      1M)    FreqDir=MO  ;;
382      5D|1D) FreqDir=DA  ;;
383      HF)    FreqDir=HF  ;;
384      3H)    FreqDir=HF  ;;
385      INS)   FreqDir=INS ;;
386    esac
387    #
388    RESULT_SE=${config_UserChoices_JobName}_SE_${an_deb}_${an_fin}_${file}.nc
389    # Do not redoo an existing SE
390    IGCM_sys_TestFileArchive ${R_SAVE}/${comp}/Analyse/SE/${RESULT_SE}
391    if [ $? = 0 ] ; then
392      continue
393    fi
394    #
395    eval file_path=\${R_OUT_${comp}}/Output/${FreqDir}/
396    #
397    case ${config_UserChoices_PeriodLength} in
398      # For Yearly simulation :
[904]399      *Y|*y)
400        PeriodLengthInYears=$( echo ${config_UserChoices_PeriodLength} | sed -e 's/[yY]//' )
[785]401        (( an = an_deb ))
402        # Initialize array
403        unset liste_file_se
404        unset liste_file_se_tmp
405        while [ ${an} -le ${an_fin} ] ; do
406          #
407          an4=$(printf "%04i\n" ${an} )
408          #
[905]409          diy=$( IGCM_date_DaysInCurrentPeriod ${an4}0101 ${config_UserChoices_PeriodLength} )
410          afin=$( IGCM_date_AddDaysToGregorianDate ${an4}0101 $(( diy - 1 )) )
[904]411          #
[785]412          date=${an4}0101_${afin}
[905]413          #
[785]414          liste_file_se[${#liste_file_se[*]}]=${file_path}${config_UserChoices_JobName}_${date}_${file}.nc
415          liste_file_se_tmp[${#liste_file_se_tmp[*]}]=${config_UserChoices_JobName}_${date}_${file}.nc
[904]416          (( an = an + ${PeriodLengthInYears} ))
[785]417        done
418        #
419        IGCM_sys_Get /l liste_file_se[*] ${RUN_DIR_PATH}
[1206]420        IGCM_debug_Verif_Exit
[785]421
422        # Apply IGCM_Patch if needed
423        if [ ! ${RebuildFrequency} = true ] ; then
424          if [ X$( eval echo \${LISTE_PATCHES_${file}[0]} ) !=  X${NULL_STR} ]; then
425            for file_t in $( eval echo ${liste_file_se_tmp[*]} ); do
426              for Patch in $( eval echo \${LISTE_PATCHES_${file}[*]} ); do
427                IGCM_${Patch} ${file_t}
428              done
429            done
430          fi
431        fi
432
433        # Search of common list of variables to be treated
434        nbfile=0
435        for file_common_var in ${liste_file_se_tmp[*]} ; do
436          ncdump -h ${file_common_var} | grep -E 'float|double' | cut -f 1 -d '(' | cut -f 2 -d ' ' >> tmpfile_ncra
437          (( nbfile = nbfile + 1 ))
438        done
439        list_var_final_ncra=`cat tmpfile_ncra | sort | uniq -c | awk -v nbfile=$nbfile '{if ($1 == nbfile) {print $2}}' | paste -s -d ','`
440        IGCM_sys_Rm tmpfile_ncra
441
442        # detect time counter : time or time_counter ie unlimited variable
[951]443        var_unlim=$(ncdump -h ${liste_file_se_tmp[0]}|grep UNLIMITED|awk '{print $1}')
[785]444
445        for mois in  01 02 03 04 05 06 07 08 09 10 11 12 ; do
446          #
447          (( nb = ${mois} - 1 ))
448          #
449          OUT_SE[${nb}]=${config_UserChoices_JobName}_SE${mois}_${an_deb}_${an_fin}_${file}.nc
450          IGCM_sys_ncra -O -v ${list_var_final_ncra} -d ${var_unlim},${nb},,12 ${liste_file_se_tmp[*]} ${OUT_SE[${nb}]}
451        done
452        IGCM_sys_Rm ${liste_file_se_tmp[*]}
453        ;;
454      # For Monthly simulation :
455      1M|1m)
456        for mois in 01 02 03 04 05 06 07 08 09 10 11 12 ; do
457          #
458          (( nb = ${mois} - 1 ))
459          #
460          OUT_SE[${nb}]=${config_UserChoices_JobName}_SE${mois}_${an_deb}_${an_fin}_${file}.nc
461          #
462          (( an = an_deb ))
[2]463          # Initialize array
[785]464          unset liste_file_se
465          unset liste_file_se_tmp
466          while [ ${an} -le ${an_fin} ] ; do
467            moisL=$( IGCM_date_DaysInMonth ${an} ${mois} )
468            #
469            an4=$(printf "%04i\n" ${an} )
470            #
471            date=${an4}${mois}01_${an4}${mois}${moisL}
[905]472            #
[785]473            liste_file_se[${#liste_file_se[*]}]=${file_path}${config_UserChoices_JobName}_${date}_${file}.nc
474            liste_file_se_tmp[${#liste_file_se_tmp[*]}]=${config_UserChoices_JobName}_${date}_${file}.nc
[2]475
[785]476            (( an = an + 1 ))
477          done
478          #
479          IGCM_sys_Get /l liste_file_se[*] ${RUN_DIR_PATH}
[1206]480          IGCM_debug_Verif_Exit
[370]481
[785]482          # Apply IGCM_Patch if needed
483          if [ ! ${RebuildFrequency} = true ] ; then
484            if [ X$( eval echo \${LISTE_PATCHES_${file}[0]} ) !=  X${NULL_STR} ]; then
485              for file_t in $( eval echo ${liste_file_se_tmp[*]} ); do
486                for Patch in $( eval echo \${LISTE_PATCHES_${file}[*]} ); do
487                  IGCM_${Patch} ${file_t}
488                done
489              done
490            fi
491          fi
[66]492
[785]493          # Search of common list of variables to be treated
494          nbfile=0
495          for file_common_var in ${liste_file_se_tmp[*]} ; do
496            ncdump -h ${file_common_var} | grep -E 'float|double' | cut -f 1 -d '(' | cut -f 2 -d ' ' >> tmpfile_ncra
497            (( nbfile = nbfile + 1 ))
498          done
499            list_var_final_ncra=`cat tmpfile_ncra | sort | uniq -c | awk -v nbfile=$nbfile '{if ($1 == nbfile) {print $2}}' | paste -s -d ','`
500          IGCM_sys_Rm tmpfile_ncra
[370]501
[785]502          IGCM_sys_ncra -O -v ${list_var_final_ncra} ${liste_file_se_tmp[*]} ${OUT_SE[${nb}]}
[231]503
[785]504          IGCM_sys_Rm ${liste_file_se_tmp[*]}
505        done
506        ;;
507      *D|*d)
508        for mois in 01 02 03 04 05 06 07 08 09 10 11 12 ; do
509          #
510          (( nb = ${mois} - 1 ))
511          #
512          OUT_SE[${nb}]=${config_UserChoices_JobName}_SE${mois}_${an_deb}_${an_fin}_${file}.nc
513          #
514          (( an = an_deb ))
515          # Initialize array
516          unset liste_file_se
517          unset liste_file_se_tmp
518          while [ ${an} -le ${an_fin} ] ; do
519            moisL=$( IGCM_date_DaysInMonth ${an} ${mois} )
520            #
521            an4=$(printf "%04i\n" ${an} )
522              #
523            length_d=$(( ${config_UserChoices_PeriodLength} - 1 ))
524            # for one day
525            if [ ${length_d} -eq 1 ] ; then
526              (( length_d = 0 ))
527            fi
528            (( day  = 1 ))
529            while [ ${day} -le ${moisL} ] ; do
530              day2=$(printf "%02i\n" ${day} )
531              day_beg=${an4}${mois}${day2}
532              day_end=$( IGCM_date_AddDaysToGregorianDate ${day_beg} ${length_d} )
533              date=${an4}${mois}${day_beg}_${an4}${mois}${day_end}
[905]534              #
[785]535              liste_file_se[${#liste_file_se[*]}]=${file_path}${config_UserChoices_JobName}_${date}_${file}.nc
536              liste_file_se_tmp[${#liste_file_se_tmp[*]}]=${config_UserChoices_JobName}_${date}_${file}.nc
537              (( day = day + 1 ))
538            done
539            (( an = an + 1 ))
540          done
541          #
542          IGCM_sys_Get /l liste_file_se[*] ${RUN_DIR_PATH}
[1206]543          IGCM_debug_Verif_Exit
[2]544
[785]545          # Apply IGCM_Patch if needed
546          if [ ! ${RebuildFrequency} = true ] ; then
547            if [ X$( eval echo \${LISTE_PATCHES_${file}[0]} ) !=  X${NULL_STR} ]; then
548              for file_t in $( eval echo ${liste_file_se_tmp[*]} ); do
549                for Patch in $( eval echo \${LISTE_PATCHES_${file}[*]} ); do
550                  IGCM_${Patch} ${file_t}
551                done
552              done
553            fi
554          fi
[231]555
[785]556          # Search of common list of variables to be treated
557          nbfile=0
558          for file_common_var in ${liste_file_se_tmp[*]} ; do
559            ncdump -h ${file_common_var} | grep -E 'float|double' | cut -f 1 -d '(' | cut -f 2 -d ' ' >> tmpfile_ncra
560            (( nbfile = nbfile + 1 ))
561          done
562            list_var_final_ncra=`cat tmpfile_ncra | sort | uniq -c | awk -v nbfile=$nbfile '{if ($1 == nbfile) {print $2}}' | paste -s -d ','`
563          IGCM_sys_Rm tmpfile_ncra
[231]564
[785]565          IGCM_sys_ncra -O -v ${list_var_final_ncra} ${liste_file_se_tmp[*]} ${OUT_SE[${nb}]}
566          IGCM_sys_Rm ${liste_file_se_tmp[*]}
[231]567        done
[785]568        ;;
569      esac
570      #
[231]571
[785]572      # Search of common list of variables to be treated
573      nbfile=0
574      for file_common_var in ${OUT_SE[*]} ; do
575        ncdump -h ${file_common_var} | grep -E 'float|double' | cut -f 1 -d '(' | cut -f 2 -d ' ' >> tmpfile_ncrcat
576        (( nbfile = nbfile + 1 ))
577      done
[1219]578      list_var_final_ncrcat=`cat tmpfile_ncrcat | sort | uniq -c | grep -v ^time_ | awk -v nbfile=$nbfile '{if ($1 == nbfile) {print $2}}' | paste -s -d ','`
[785]579      IGCM_sys_Rm tmpfile_ncrcat
[370]580
[785]581    IGCM_sys_ncrcat -O -v ${list_var_final_ncrcat} ${OUT_SE[*]} ${RESULT_SE}
582    #
583    IGCM_sys_Rm ${OUT_SE[*]}
584    #
585    # detect time counter : time or time_counter ie unlimited variable
[1304]586    var_unlim=$( ncdump -h ${RESULT_SE} | grep UNLIMITED |  cut -d ' ' -f 1 | sed -e 's/^[[:space:]]*//' )
[785]587
[904]588    # Put correct climatological axis following calendar
[785]589    case ${config_UserChoices_CalendarType} in
590      360d|360_day)
591      # 360 days
592        ncap2cmd="'defdim(\"tbnds\",2) ; ${var_unlim}[${var_unlim}]={15, 45, 75, 105, 135, 165, 195, 225, 255, 285, 315, 345.} ; ${var_unlim}_bnds[${var_unlim},tbnds]={0, 30, 30, 60, 60, 90, 90, 120, 120, 150, 150, 180, 180, 210, 210, 240, 240, 270, 270, 300, 300, 330, 330, 360.} ; ${var_unlim}@units=\"days since 0000-01-01 00:00:00\" ; ${var_unlim}@time_origin=\"01-JAN-0000 00:00:00\" ; ${var_unlim}@calendar=\"360_day\" ; ${var_unlim}@bounds=\"${var_unlim}_bnds\"'"
593        ;;
594      noleap|365_day)
595      # 365 days
596        ncap2cmd="'defdim(\"tbnds\",2) ; ${var_unlim}[${var_unlim}]={15.5, 45, 74.5, 105, 135.5, 166, 196.5, 227.5, 258, 288.5, 319, 349.5} ; ${var_unlim}_bnds[${var_unlim},tbnds]={0, 31, 31, 59, 59, 90, 90, 120, 120, 151, 151, 181, 181, 212, 212, 243, 243, 273, 273, 304, 304, 334, 334, 365.} ; ${var_unlim}@units=\"days since 0000-01-01 00:00:00\" ; ${var_unlim}@time_origin=\"01-JAN-0000 00:00:00\" ; ${var_unlim}@calendar=\"365_day\" ; ${var_unlim}@bounds=\"${var_unlim}_bnds\"'"
597        ;;
598      all_leap|366_day)
599      # 366 days
600        ncap2cmd="'defdim(\"tbnds\",2) ; ${var_unlim}[${var_unlim}]={15.5, 45.5, 75.5, 106, 136.5, 167, 197.5, 228.5, 259, 289.5, 320, 350.5} ; ${var_unlim}_bnds[${var_unlim},tbnds]={0, 31, 31, 60, 60, 91, 91, 121, 121, 152, 152, 182, 182, 213, 213, 244, 244, 274, 274, 305, 305, 335, 335, 366.} ; ${var_unlim}@units=\"days since 0000-01-01 00:00:00\" ; ${var_unlim}@time_origin=\"01-JAN-0000 00:00:00\" ; ${var_unlim}@calendar=\"366_day\" ; ${var_unlim}@bounds=\"${var_unlim}_bnds\"'"
601        ;;
602      leap|gregorian|standard)
603      # 365.2425 days
604        ncap2cmd="'defdim(\"tbnds\",2) ; ${var_unlim}[${var_unlim}]={15.5, 45.12125, 74.7425, 105.2425, 135.7425, 166.2425, 196.7425, 227.7425, 258.2425, 288.7425, 319.2425, 349.7425} ; ${var_unlim}_bnds[${var_unlim},tbnds]={0, 31, 31, 59.2425, 59.2425, 90.2425, 90.2425, 120.2425, 120.2425, 151.2425, 151.2425, 181.2425, 181.2425, 212.2425, 212.2425, 243.2425, 243.2425, 273.2425, 273.2425, 304.2425, 304.2425, 334.2425, 334.2425, 365.2425} ; ${var_unlim}@units=\"days since 0000-01-01 00:00:00\" ; ${var_unlim}@time_origin=\"01-JAN-0000 00:00:00\" ; ${var_unlim}@calendar=\"gregorian\" ; ${var_unlim}@bounds=\"${var_unlim}_bnds\"'"
605        ;;
606    esac
607    eval IGCM_sys_ncap2 -Oh -s $ncap2cmd ${RESULT_SE} tmp.nc
608    IGCM_sys_Rm ${RESULT_SE}
609    IGCM_sys_Mv tmp.nc ${RESULT_SE}
610    #
611    eval IGCM_sys_Put_Out ${RESULT_SE} \${R_OUT_${comp}}/Analyse/SE/${RESULT_SE}
[1040]612    [ -f ${RESULT_SE} ] && IGCM_sys_Rm ${RESULT_SE}
[785]613    #
[1603]614    (( i = i + 1 ))
[785]615  done
[2]616done
617
618# DODS copy
[903]619if [ X${config_Post_DodsCopy} = XTRUE ]  ; then
620  for comp in ${config_ListOfComponents[*]} ; do
621    IGCM_sys_TestDirArchive ${R_SAVE}/${comp}/Analyse/SE
622    [ $? = 0 ] && IGCM_sys_Put_Dods ${comp}/Analyse/SE
623  done
624fi
[2]625
[903]626# Temporal coverage id for atlas
[2]627YEARS=${an_deb}_${an_fin}
[370]628DIR_OUT=SE
[2]629
[1112]630export DEBUG_debug; export BigBrother ; export libIGCM; export SUBMIT_DIR; export POST_DIR; export Script_Output ;
631export StandAlone ; export RESOL_ATM; export RESOL_OCE ; export RESOL_ICE ; export RESOL_MBG ; export RESOL_SRF ;
632export RESOL_SBG ; export YEARS; export DIR_OUT
633listVarEnv="DEBUG_debug,BigBrother,libIGCM,SUBMIT_DIR,POST_DIR,Script_Post_Output,StandAlone,RESOL_ATM,RESOL_OCE,RESOL_ICE,RESOL_MBG,RESOL_SRF,RESOL_SBG,YEARS,DIR_OUT"
[2]634export listVarEnv;
635
636#D-
637# --------------------------------------------------------------------
638#D- Test if all was right before ATLAS
639# --------------------------------------------------------------------
[1206]640IGCM_debug_Verif_Exit
[2]641
642if [ ${config_UserChoices_TagName} = "IPSLCM4_v1_OASIS3" ] || \
[15]643   [ ${config_UserChoices_TagName} = "IPSLCM4_v2" ]        || \
[2]644   [ ${config_UserChoices_TagName} = "IPSL_ESM_V1" ] ; then
[1366]645  # ATLAS
646  if [ X${config_Post_AtlasIPSL} = XTRUE ]  ; then
647    export Script_Post_Output=atlas_ORCHIDEE.${PeriodDateEnd} ;
648    IGCM_sys_QsubPost atlas_ORCHIDEE                          ;
649    export Script_Post_Output=atlas_LMDZ.${PeriodDateEnd}     ;
650    IGCM_sys_QsubPost atlas_LMDZ                              ;
651    export Script_Post_Output=atlas_ORCA_LIM.${PeriodDateEnd} ;
652    IGCM_sys_QsubPost atlas_ORCA_LIM                          ;
653    export Script_Post_Output=additionnal.${PeriodDateEnd}    ;
654    IGCM_sys_QsubPost additionnal                             ;
655  fi
[785]656  #
[1112]657elif [ ${config_UserChoices_TagName} = "IPSLCM5" ]         || \
658     [ ${config_UserChoices_TagName} = "IPSLCM5A" ]        || \
659     [ ${config_UserChoices_TagName} = "IPSLCM5B" ]        || \
660     [ ${config_UserChoices_TagName} = "IPSLCM5A-MR" ]     || \
[1377]661     [ ${config_UserChoices_TagName} = "IPSLCM5A2" ]       || \
[957]662     [ ${config_UserChoices_TagName} = "IPSLCM6" ] ; then
[1225]663  # METRICS
[1335]664  if [ X${config_Post_MetricsPCMDI} = XTRUE ]  ; then
665    export Script_Post_Output=metrics_LMDZ.${PeriodDateEnd}   ;
666    IGCM_sys_QsubPost metrics_LMDZ                            ;
667    export Script_Post_Output=metrics_NEMO.${PeriodDateEnd}   ;
668    IGCM_sys_QsubPost metrics_NEMO                            ;
669  fi
[1225]670  # ATLAS
[1366]671  if [ X${config_Post_AtlasIPSL} = XTRUE ]  ; then
672    export Script_Post_Output=atlas_ORCHIDEE.${PeriodDateEnd} ;
673    IGCM_sys_QsubPost atlas_ORCHIDEE                          ;
674    export Script_Post_Output=atlas_LMDZ.${PeriodDateEnd}     ;
675    IGCM_sys_QsubPost atlas_LMDZ                              ;
676    export Script_Post_Output=atlas_ORCA_LIM.${PeriodDateEnd} ;
677    IGCM_sys_QsubPost atlas_ORCA_LIM                          ;
678    export Script_Post_Output=atlas_PISCES.${PeriodDateEnd}   ;
679    IGCM_sys_QsubPost atlas_PISCES                            ;
680  fi
[1112]681elif [ ${config_UserChoices_TagName} = "LMDZ4OR" ]         || \
682     [ ${config_UserChoices_TagName} = "LMDZOR" ]          || \
[25]683     [ ${config_UserChoices_TagName} = "LMDZ4OR_v2" ] ; then
[1225]684  # METRICS
[1335]685  if [ X${config_Post_MetricsPCMDI} = XTRUE ]  ; then
686    export Script_Post_Output=metrics_LMDZ.${PeriodDateEnd}   ;
687    IGCM_sys_QsubPost metrics_LMDZ                            ;
688  fi
[1225]689  # ATLAS
[1366]690  if [ X${config_Post_AtlasIPSL} = XTRUE ]  ; then
691    export Script_Post_Output=atlas_LMDZ.${PeriodDateEnd}     ;
692    IGCM_sys_QsubPost atlas_LMDZ                              ;
693    export Script_Post_Output=atlas_ORCHIDEE.${PeriodDateEnd} ;
694    IGCM_sys_QsubPost atlas_ORCHIDEE                          ;
695  fi
[45]696elif [ ${config_UserChoices_TagName} = "LMDZ" ] ; then
[1225]697  # METRICS
[1335]698  if [ X${config_Post_MetricsPCMDI} = XTRUE ]  ; then
699    export Script_Post_Output=metrics_LMDZ.${PeriodDateEnd}   ;
700    IGCM_sys_QsubPost metrics_LMDZ                            ;
701  fi
[1225]702  # ATLAS
[1366]703  if [ X${config_Post_AtlasIPSL} = XTRUE ]  ; then
704    export Script_Post_Output=atlas_LMDZ.${PeriodDateEnd}     ;
705    IGCM_sys_QsubPost atlas_LMDZ                              ;
706  fi
[1112]707elif [ ${config_UserChoices_TagName} = "OL" ]              || \
[2]708     [ ${config_UserChoices_TagName} = "OL2" ] ; then
[1225]709  # METRICS
710  # ATLAS
[1366]711  if [ X${config_Post_AtlasIPSL} = XTRUE ]  ; then
712    export Script_Post_Output=atlas_ORCHIDEE.${PeriodDateEnd} ;
713    IGCM_sys_QsubPost atlas_ORCHIDEE                          ;
714  fi
[1112]715elif [ ${config_UserChoices_TagName} = "ORCA2_LIM" ]       || \
716     [ ${config_UserChoices_TagName} = "ORCA2_LIM2" ]      || \
[1309]717     [ ${config_UserChoices_TagName} = "ORCA2_LIM3" ]      || \
718     [ ${config_UserChoices_TagName} = "ORCA1_LIM2" ]      || \
719     [ ${config_UserChoices_TagName} = "ORCA1_LIM3" ] ; then
[1225]720  # METRICS
[1335]721  if [ X${config_Post_MetricsPCMDI} = XTRUE ]  ; then
722    export Script_Post_Output=metrics_NEMO.${PeriodDateEnd}   ;
723    IGCM_sys_QsubPost metrics_NEMO                            ;
724  fi
[1225]725  # ATLAS
[1366]726  if [ X${config_Post_AtlasIPSL} = XTRUE ]  ; then
727    export Script_Post_Output=atlas_ORCA_LIM.${PeriodDateEnd} ;
728    IGCM_sys_QsubPost atlas_ORCA_LIM                          ;
729  fi
[957]730elif [ ${config_UserChoices_TagName} = "ORCA2_LIM2_PISCES" ] || \
[1309]731     [ ${config_UserChoices_TagName} = "ORCA2_LIM3_PISCES" ] || \
732     [ ${config_UserChoices_TagName} = "ORCA1_LIM2_PISCES" ] || \
733     [ ${config_UserChoices_TagName} = "ORCA1_LIM3_PISCES" ] ; then
[1225]734  # METRICS
[1335]735  if [ X${config_Post_MetricsPCMDI} = XTRUE ]  ; then
736    export Script_Post_Output=metrics_NEMO.${PeriodDateEnd}   ;
737    IGCM_sys_QsubPost metrics_NEMO                            ;
738  fi
[1225]739  # ATLAS
[1366]740  if [ X${config_Post_AtlasIPSL} = XTRUE ]  ; then
741    export Script_Post_Output=atlas_ORCA_LIM.${PeriodDateEnd} ;
742    IGCM_sys_QsubPost atlas_ORCA_LIM                          ;
743    export Script_Post_Output=atlas_PISCES.${PeriodDateEnd}   ;
744    IGCM_sys_QsubPost atlas_PISCES                            ;
745  fi
[1112]746elif [ ${config_UserChoices_TagName} = "ORCA2_OFF_PISCES" ] ; then
[1225]747  # METRICS
748  # ATLAS
[1366]749  if [ X${config_Post_AtlasIPSL} = XTRUE ]  ; then
[1112]750  export Script_Post_Output=atlas_PISCES.${PeriodDateEnd}   ;
751  IGCM_sys_QsubPost atlas_PISCES                            ;
[1366]752  fi
[2]753fi
[203]754
[1598]755# Execute C-ESM-EP driver (which launches jobs)
756if [ $PostCesmep = SE ] ; then
757    ${SUBMIT_DIR}/cesmep_lite/libIGCM_post.sh $DateBegin $PeriodDateEnd
758fi
759
[590]760# Clean RUN_DIR_PATH (necessary for cesium and titane only)
[203]761IGCM_sys_RmRunDir -Rf ${RUN_DIR_PATH}
[1198]762
763# ------------------------------------------------------------------
764# Finalize BigBrother to inform that the jobs end
765# ------------------------------------------------------------------
766IGCM_debug_BigBro_Finalize
Note: See TracBrowser for help on using the repository browser.