source: trunk/libIGCM/libIGCM_config/libIGCM_config.ksh @ 1189

Last change on this file since 1189 was 1189, checked in by sdipsl, 9 years ago
  • Extend Big Brother scope to also watch post-processing jobs. Only the metrics LMDZ job for now. See #252
  • Some refactoring
  • Property licence set to
    The following licence information concerns ONLY the libIGCM tools
    ==================================================================

    Copyright © Centre National de la Recherche Scientifique CNRS
    Commissariat à l'Énergie Atomique CEA

    libIGCM : Library for Portable Models Computation of IGCM Group.

    IGCM Group is the french IPSL Global Climate Model Group.

    This library is a set of shell scripts and functions whose purpose is
    the management of the initialization, the launch, the transfer of
    output files, the post-processing and the monitoring of datas produce
    by any numerical program on any plateforme.

    This software is governed by the CeCILL license under French law and
    abiding by the rules of distribution of free software. You can use,
    modify and/ or redistribute the software under the terms of the CeCILL
    license as circulated by CEA, CNRS and INRIA at the following URL
    "http://www.cecill.info".

    As a counterpart to the access to the source code and rights to copy,
    modify and redistribute granted by the license, users are provided only
    with a limited warranty and the software's author, the holder of the
    economic rights, and the successive licensors have only limited
    liability.

    In this respect, the user's attention is drawn to the risks associated
    with loading, using, modifying and/or developing or reproducing the
    software by the user in light of its specific status of free software,
    that may mean that it is complicated to manipulate, and that also
    therefore means that it is reserved for developers and experienced
    professionals having in-depth computer knowledge. Users are therefore
    encouraged to load and test the software's suitability as regards their
    requirements in conditions enabling the security of their systems and/or
    data to be ensured and, more generally, to use and operate it in the
    same conditions as regards security.

    The fact that you are presently reading this means that you have had
    knowledge of the CeCILL license and that you accept its terms.
  • Property svn:keywords set to Revision Author Date
File size: 51.7 KB
RevLine 
[2]1#!/bin/ksh
2
3#**************************************************************
4# Author: Sebastien Denvil, Martial Mancip
[373]5# Contact: Sebastien.Denvil__at__ipsl.jussieu.fr Martial.Mancip__at__ipsl.jussieu.fr
6# $Revision::                                          $ Revision of last commit
7# $Author::                                            $ Author of last commit
8# $Date::                                              $ Date of last commit
[2]9# IPSL (2006)
10#  This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
11#
12#**************************************************************
13
14#===================================
[726]15function IGCM_config_CommonConfiguration
16{
17  IGCM_debug_PushStack "IGCM_config_CommonConfiguration" $@
18
19  # Debug Print :
[769]20  [ ${Verbosity} -gt 0 ] && echo
[726]21  IGCM_debug_Print 1 "IGCM_config_CommonConfiguration" $@
22
23  # config.card path
24  configPath=$1
25
26  #==================================
27  typeset option auxprint CompatibilityTag
28
29  #==================================
30  # Read libIGCM compatibility version in config.card
31  IGCM_card_DefineVariableFromOption ${configPath} Compatibility libIGCM
[939]32  CompatibilityTag=${config_Compatibility_libIGCM}
[726]33
34  if [ ! "${CompatibilityTag}" = "${libIGCM_CurrentTag}" ] ; then
35    IGCM_debug_Exit "config.card is not compatible with libIGCM version ${libIGCM_CurrentTag} see libIGCM FAQ http://wiki.ipsl.jussieu.fr/wiki_ipsl/IGCMG/libIGCM/DocUtilisateur/FAQ ."
36  fi
37
38  #==================================
39  # Read UserChoices section:
[769]40  [ ${Verbosity} -gt 0 ] && echo
[726]41  IGCM_debug_Print 1 "DefineArrayFromOption  : config_UserChoices"
42
43  IGCM_card_DefineArrayFromSection ${configPath} UserChoices
44  for option in ${config_UserChoices[*]} ; do
45    IGCM_card_DefineVariableFromOption ${configPath} UserChoices ${option}
46    eval auxprint=\${config_UserChoices_${option}}
47    IGCM_debug_Print 3 "${option} : ${auxprint}"
48  done
49
50  #==================================
51  # Read Ensemble section:
[769]52  [ ${Verbosity} -gt 0 ] && echo
[726]53  IGCM_debug_Print 1 "DefineArrayFromOption  : config_Ensemble"
54
55  IGCM_card_DefineArrayFromSection ${configPath} Ensemble
56  for option in ${config_Ensemble[*]} ; do
57    IGCM_card_DefineVariableFromOption ${configPath} Ensemble ${option}
58    eval auxprint=\${config_Ensemble_${option}}
59    IGCM_debug_Print 3 "${option} : ${auxprint}"
60  done
61
62  #==================================
63  # Read Post section:
[769]64  [ ${Verbosity} -gt 0 ] && echo
[726]65  IGCM_debug_Print 1 "DefineArrayFromOption : config_Post"
66
67  IGCM_card_DefineArrayFromSection ${configPath} Post
68  for option in ${config_Post[*]} ; do
69    IGCM_card_DefineVariableFromOption ${configPath} Post ${option}
70    eval auxprint=\${config_Post_${option}}
71    IGCM_debug_Print 3 "${option} : ${auxprint}"
72  done
[769]73  [ ${Verbosity} -gt 0 ] && echo
[726]74
[728]75  #==================================
76  # Define default value to keep compatibility with previous card: means before changes due to TGCC
77  # Apply some overrules to ensure proper usage of computing centres resources
[804]78  #
[728]79  if [ X${PackDefault} = Xtrue ] ; then
80    if [ X${config_UserChoices_SpaceName} = XTEST ]; then
81      # TEST simulations will not be packed and will stay on SCRATCHDIR filesystem
82      IGCM_debug_Print 1 "SpaceName=TEST. OVERRULE PackFrequency to NONE"
83      config_Post_PackFrequency=NONE
84    else
85      # Default to RebuildFrequency if nothing has been set up related to PackFrequency
86      [ X${config_Post_PackFrequency} = X ] && config_Post_PackFrequency=${config_Post_RebuildFrequency}
87    fi
88  else
89    # If we DO NOT apply pack in this computing center
90    config_Post_PackFrequency=NONE
91  fi
92
[726]93  #====================================================
[1057]94  # Define ARCHIVE : Dedicated to large files
95  # Define STORAGE : Dedicated to small/medium files
96  # Define R_OUT   : Output tree located on ARCHIVE
[1150]97  # Define R_BUF   : Output tree located on STORAGE (files waiting treatment, or file lcoation when SpaceName=!PROD)
[1057]98  # Define R_FIG   : Output tree located on STORAGE hosting figures (monitoring and atlas, and/or small files)
[1150]99  # Define R_TMP   : A temporary space used by IGCM_debug_send_AMQP_msg__MAILTUNNEL. Must be persistent in between jobs
[1057]100  IGCM_sys_defineArchives
101
102  #====================================================
[726]103  # R_SAVE : Job output directory
104  # R_BUFR : Job output buffered directory
105
106  if ( [ ! X${config_UserChoices_SpaceName} = X ] && [ ! X${config_UserChoices_ExperimentName} = X ] ) ; then
107    FreeName=$( echo ${config_UserChoices_JobName} | sed 's/.*_//' )
108    if ( [ ! X${config_Ensemble_EnsembleName} = X ] && [ ! X${config_Ensemble_EnsembleDate} = X ] ) ; then
109      R_SAVE=${R_OUT}/${config_UserChoices_TagName}/${config_UserChoices_SpaceName}/${config_UserChoices_ExperimentName}/${config_Ensemble_EnsembleName}/${config_Ensemble_EnsembleDate}/${FreeName}
110      R_FIGR=${R_FIG}/${config_UserChoices_TagName}/${config_UserChoices_SpaceName}/${config_UserChoices_ExperimentName}/${config_Ensemble_EnsembleName}/${config_Ensemble_EnsembleDate}/${FreeName}
111      R_BUFR=${R_BUF}/${config_UserChoices_TagName}/${config_UserChoices_SpaceName}/${config_UserChoices_ExperimentName}/${config_Ensemble_EnsembleName}/${config_Ensemble_EnsembleDate}/${FreeName}
112      R_DODS=${config_UserChoices_TagName}/${config_UserChoices_SpaceName}/${config_UserChoices_ExperimentName}/${config_Ensemble_EnsembleName}/${config_Ensemble_EnsembleDate}/${FreeName}
113    else
114      R_SAVE=${R_OUT}/${config_UserChoices_TagName}/${config_UserChoices_SpaceName}/${config_UserChoices_ExperimentName}/${FreeName}
115      R_FIGR=${R_FIG}/${config_UserChoices_TagName}/${config_UserChoices_SpaceName}/${config_UserChoices_ExperimentName}/${FreeName}
116      R_BUFR=${R_BUF}/${config_UserChoices_TagName}/${config_UserChoices_SpaceName}/${config_UserChoices_ExperimentName}/${FreeName}
117      R_DODS=${config_UserChoices_TagName}/${config_UserChoices_SpaceName}/${config_UserChoices_ExperimentName}/${FreeName}
118    fi
119  else
120    if ( [ ! X${config_Ensemble_EnsembleName} = X ] && [ ! X${config_Ensemble_EnsembleDate} = X ] ) ; then
121      R_SAVE=${R_OUT}/${config_UserChoices_TagName}/${config_Ensemble_EnsembleName}/${config_Ensemble_EnsembleDate}/${config_UserChoices_JobName}
122      R_FIGR=${R_FIG}/${config_UserChoices_TagName}/${config_Ensemble_EnsembleName}/${config_Ensemble_EnsembleDate}/${config_UserChoices_JobName}
123      R_BUFR=${R_BUF}/${config_UserChoices_TagName}/${config_Ensemble_EnsembleName}/${config_Ensemble_EnsembleDate}/${config_UserChoices_JobName}
124      R_DODS=${config_UserChoices_TagName}/${config_Ensemble_EnsembleName}/${config_Ensemble_EnsembleDate}/${config_UserChoices_JobName}
125    else
126      R_SAVE=${R_OUT}/${config_UserChoices_TagName}/${config_UserChoices_JobName}
127      R_FIGR=${R_FIG}/${config_UserChoices_TagName}/${config_UserChoices_JobName}
128      R_BUFR=${R_BUF}/${config_UserChoices_TagName}/${config_UserChoices_JobName}
129      R_DODS=${config_UserChoices_TagName}/${config_UserChoices_JobName}
130    fi
131  fi
132
[782]133  #====================================================
[1110]134  # Define R_OUT_KSH : Storage place for job output
135  # Define R_OUT_EXE : Storage place for binary used during simulation
136  R_OUT_KSH=${R_SAVE}/Out
137  R_OUT_EXE=${R_SAVE}/Exe
138
139  #====================================================
140  # Define R_BUF_KSH : Buffer place for job output
141  # Define R_BUF_EXE : Buffer place for binary used during simulation
142  R_BUF_KSH=${R_BUFR}/Out
143  R_BUF_EXE=${R_BUFR}/Exe
144
145  #====================================================
[1077]146  # Define REBUILD_DIR : where we store files needing rebuild process
[1031]147  REBUILD_DIR=${R_BUFR}/REBUILD
148  if [ ! X${TaskType} = Xchecking ] ; then
149    IGCM_sys_MkdirWork ${REBUILD_DIR}
[782]150  fi
151
[903]152  #====================================================
153  # DodsCopy : apply default value if not defined
154  if ( [ X${config_Post_DodsCopy} = X${NULL_STR} ] || [ X${config_Post_DodsCopy} = X ] ) ; then
155    config_Post_DodsCopy=TRUE
156  fi
157
[1115]158  #====================================================
159  # Define StackFileLocation : directory where we store stack files
160  # Define StackFileName : stack file containing call tree and instrumentation
161  # Stack file containing call tree will be stored there.
162  if ( $DEBUG_debug ) ; then
163    StackFileLocation=${StackFileLocation:=${R_BUF_KSH}}
164    [ ! -d ${StackFileLocation} ] && mkdir -p ${StackFileLocation}
165    if [ X${TaskType} = Xcomputing ]; then
166      StackFileName=computing.stack.$$
167    elif [ X${TaskType} = Xpost-processing ]; then
168      StackFileName=${Script_Post_Output}.stack.$$
169    elif [ X${TaskType} = Xchecking ]; then
170      StackFileName=checking.stack.$$
171    else
172      IGCM_debug_Exit "IGCM_config_CommonConfiguration unknown TaskType : ${TaskType}"
173      IGCM_debug_Verif_Exit
174    fi
175
176    # This boolean will trigger the filling of the stack
177    # Only now we know where things should be ...
[1127]178    # We don't fill the stack when we perform checking task
179    if [ ! X${TaskType} = Xchecking ] ; then
180      ActivateStackFilling=true
181    fi
[1115]182  fi
183
[726]184  IGCM_debug_PopStack "IGCM_config_CommonConfiguration"
185}
186
187#===================================
[2]188function IGCM_config_Initialize
189{
[544]190  IGCM_debug_PushStack "IGCM_config_Initialize"
[2]191
[544]192  # Debug Print :
193  echo
194  IGCM_debug_Print 1 "IGCM_config_Initialize"
[2]195
[544]196  # Test modipsl tree existence.
197  IGCM_sys_TestDir ${MODIPSL}
198  [ $? != 0 ] && IGCM_debug_Exit "IGCM_sys_TestDir"
199  IGCM_sys_TestDir ${libIGCM}
200  [ $? != 0 ] && IGCM_debug_Exit "IGCM_sys_TestDir"
201  IGCM_sys_TestDir ${R_EXE}
202  [ $? != 0 ] && IGCM_debug_Exit "IGCM_sys_TestDir"
203  IGCM_sys_TestDir ${SUBMIT_DIR}
204  [ $? != 0 ] && IGCM_debug_Exit "IGCM_sys_TestDir"
[2]205
[544]206  if ( $DEBUG_debug ) ; then
207    echo "Keep trace of inital SUBMIT_DIR : "
208    ls -lta ${SUBMIT_DIR}
209  fi
[144]210
[544]211  #==================================
[619]212  # Read ListOfComponents section:
[726]213  echo
214  IGCM_debug_Print 1 "DefineArrayFromSection : ListOfComponents"
215
[544]216  IGCM_card_DefineArrayFromSection ${SUBMIT_DIR}/config.card ListOfComponents
217  for comp in ${config_ListOfComponents[*]} ; do
218    IGCM_card_DefineArrayFromOption ${SUBMIT_DIR}/config.card ListOfComponents ${comp}
219  done
220  IGCM_debug_Print 3 ${config_ListOfComponents[*]}
[2]221
[544]222  #==================================
[726]223  # Read Executable section:
[544]224  IGCM_card_DefineArrayFromSection   ${SUBMIT_DIR}/config.card Executable
[619]225
[544]226  #==================================
[726]227  # Read Restarts section:
228  # Restarts : Gerneral rule or local for each component.
229  echo
230  IGCM_debug_Print 1 "DefineArrayFromOption : config_Restarts"
231
232  IGCM_card_DefineArrayFromSection ${SUBMIT_DIR}/config.card Restarts
233  for option in ${config_Restarts[*]} ; do
234    IGCM_card_DefineVariableFromOption ${SUBMIT_DIR}/config.card Restarts ${option}
235    eval auxprint=\${config_Restarts_${option}}
236    IGCM_debug_Print 3 "${option} : ${auxprint}"
237  done
238
239  #==================================
240  # Define Job Outputs Name
241  echo
[544]242  IGCM_debug_Print 2 "Define Script_Output_Prefix and Exe_Output"
[939]243  Script_Output_Prefix=${config_UserChoices_Script_Output_Prefix:='Script_Output'}
[544]244  IGCM_debug_Print 3 "Script_Output_Prefix = ${Script_Output_Prefix}"
[939]245  Exe_Output=out_execution
[544]246  IGCM_debug_Print 3 "Exe_Output           = ${Exe_Output}"
[2]247
[544]248  #===================================================================#
249  # Prepare variables available for ${COMP}.card and ${COMP}.driver   #
250  #             But available to any son functions                    #
251  #===================================================================#
[2]252
[544]253  # Convert yyyy-mm-dd date to gregorian yyyymmdd
254  DateBegin=$( IGCM_date_ConvertFormatToGregorian ${config_UserChoices_DateBegin} )
[619]255  DateEnd=$(   IGCM_date_ConvertFormatToGregorian ${config_UserChoices_DateEnd}   )
[2]256
[544]257  # Period Length In Days between DateBegin and DateEnd
258  (( ExperienceLengthInDays=$( IGCM_date_DaysBetweenGregorianDate ${DateEnd} ${DateBegin} )  + 1 ))
259  if [ ${ExperienceLengthInDays} -lt 0 ] ; then
260    IGCM_debug_Print 1 "Problem with dates in config.card : ${DateEnd} < ${DateBegin} ! You must check that."
261    IGCM_debug_Exit "IGCM_config_Initialize" " Wrong Dates."
[619]262    IGCM_debug_Verif_Exit
[544]263  fi
[2]264
[544]265  # Day and Year of Initial State (Given in julian format)
266  InitDay=$(( $( IGCM_date_ConvertGregorianDateToJulian $DateBegin ) % 1000 ))
267  InitYear=$(( $( IGCM_date_ConvertGregorianDateToJulian $DateBegin ) / 1000 ))
[2]268
[544]269  #================================================================#
270  #                  Test and Prepare directories                  #
271  #================================================================#
[2]272
[544]273  # ==> 4 kinds of input files :
274  #     1) R_INIT  : Initial State Files   (Etat0, carteveg)
275  #     2) R_BC    : Boundary Conditions   (Forcages, lai)
276  #     3) Parameters files (allready define through ${SUBMIT_DIR})
277  #     4) Restarts files   (allready define in IGCM_config_Initialize)
[2]278
[544]279  # Here we offer the possibility to redefine R_INIT, R_BC
280  # and PeriodNb through config.card
281  R_INIT=${config_UserChoices_R_INIT:=${R_IN}/INIT}
[731]282  echo
[544]283  IGCM_debug_Print 2 "(Re)Define R_INIT, R_BC and PeriodNb"
284  IGCM_debug_Print 3 "R_INIT=${R_INIT}"
285  R_BC=${config_UserChoices_R_BC:=${R_IN}/BC}
286  IGCM_debug_Print 3  "R_BC=${R_BC}"
287  PeriodNb=${config_UserChoices_PeriodNb:=${PeriodNb}}
288  IGCM_debug_Print 3  "Loop in main Job with ${PeriodNb} period(s)"
[2]289
[771]290  # SD ADA SPECIFIC #
291  #      TO FIX     #
292  #IGCM_sys_TestDirArchive ${R_IN}
293  #[ $? != 0 ] && IGCM_debug_Exit "IGCM_sys_TestDirArchive"
294
[619]295  if ( ${FirstInitialize} ) ; then
[726]296    IGCM_sys_MkdirArchive   ${R_SAVE}
[805]297    [ ! ${config_Post_PackFrequency} = NONE ] && IGCM_sys_Mkdir ${R_BUFR}
[544]298  else
299    IGCM_sys_TestDirArchive ${R_SAVE}
300    [ $? != 0 ] && IGCM_debug_Exit "IGCM_sys_TestDirArchive ${R_SAVE}"
[2]301
[805]302    if [ ! ${config_Post_PackFrequency} = NONE ] ; then
303      IGCM_sys_TestDir ${R_BUFR}
304      [ $? != 0 ] && IGCM_debug_Exit "IGCM_sys_TestDir ${R_BUFR}"
305    fi
[545]306
[544]307    #Test state of run in run.card
308    IGCM_card_DefineVariableFromOption ${SUBMIT_DIR}/run.card Configuration PeriodState
[619]309
[544]310    if ( [ ${run_Configuration_PeriodState} != "Start" ] && [ ${run_Configuration_PeriodState} != "Running" ] && [ ${run_Configuration_PeriodState} != "OnQueue" ] && [ ${run_Configuration_PeriodState} != "Continue" ] ) ; then
[619]311      echo
[544]312      IGCM_debug_Print 1 "!! Error in run.card with PeriodState : " ${run_Configuration_PeriodState} "!!"
[985]313      IGCM_debug_Print 1 "Check post-processing jobs carefully by running ${libIGCM}/RunChecker.job -p ${SUBMIT_DIR}"
314      IGCM_debug_Print 1 "Then try running ${libIGCM}/clean_month.job to rerun one period"
315      IGCM_debug_Print 1 "Then try running ${libIGCM}/clean_year.job to rerun more."
[544]316      IGCM_debug_Exit
317      IGCM_debug_Verif_Exit
[2]318    fi
[544]319  fi
[2]320
[544]321  #====================================================
[619]322  # Experience type : DEB(ug), DEV(elopment), RUN
[544]323  if [ X${JobType} != XRUN ] ; then
324    echo
325    echo "===================================================="
326    echo "libIGCM JOB is NOT in RUN type mode."
327    echo "!! OUTPUT files will NOT be PROTECTED !!"
328    echo "Be carefull : you can ERASE the result of this job !"
[128]329
[544]330    case ${JobType} in
331    DEB)
332      echo "DEBUG mode : activation of 'set -vx' mode."
333      echo "DEBUG mode : no protection for output files."
334      echo "DEBUG mode : if active force asynchronous rebuild frequency to PeriodLength frequency."
335      ;;
336    DEV)
337      echo "DEVelopment mode : no protection for output files."
338      echo "DEVelopment mode : if active force asynchronous rebuild frequency to PeriodLength frequency."
339      ;;
340    esac
[166]341
[544]342    if ( [ X${config_Post_RebuildFrequency} != XNONE ] && [ ${DRYRUN} -eq 0 ] ) ; then
343      if [ X${config_Post_RebuildFrequency} != X${config_UserChoices_PeriodLength} ] ; then
[619]344        echo "------------"
345        echo "WARNING : Job is NOT in RUN mode then we will force REBUILD Frequency"
[786]346        echo "          to PeriodLength : ${config_UserChoices_PeriodLength}"
[619]347        echo "------------"
348        config_Post_RebuildFrequency=${config_UserChoices_PeriodLength}
[544]349      fi
[128]350    fi
[544]351    echo "===================================================="
352    echo
353  fi
[128]354
[544]355  IGCM_debug_PopStack "IGCM_config_Initialize"
[2]356}
357
358#===================================
[890]359function IGCM_config_DaysInPeriodLength
[118]360{
[890]361  IGCM_debug_PushStack "IGCM_config_DaysInPeriodLength"
[118]362
[544]363  typeset i
[124]364
[544]365  # Determine number of day(s) in PeriodLength :
366  case ${config_UserChoices_PeriodLength} in
367  *Y|*y)
[619]368    PeriodLengthInYears=$( echo ${config_UserChoices_PeriodLength} | sed -e 's/[yY]//' )
[731]369    echo
[544]370    IGCM_debug_Print 2 "Number of years for PeriodLength : ${PeriodLengthInYears}"
371    PeriodLengthInDays=0
372    i=0
[619]373    until [ $i -ge $PeriodLengthInYears ] ; do
374      (( PeriodLengthInDays = PeriodLengthInDays + $( IGCM_date_DaysInYear $(( year + i )) ) ))
375      (( i=i+1 ))
376    done
[544]377    ;;
378  *M|*m)
[619]379    PeriodLengthInMonths=$( echo ${config_UserChoices_PeriodLength} | sed -e 's/[mM]//' )
[731]380    echo
[544]381    IGCM_debug_Print 2 "Number of months for PeriodLength : ${PeriodLengthInMonths}"
382    PeriodLengthInDays=0
383    i=0
[619]384    until [ $i -ge $PeriodLengthInMonths ] ; do
[829]385      if [ $(( 10#${month} + ${i} )) -lt 13 ] ; then
386        (( PeriodLengthInDays  = PeriodLengthInDays + $( IGCM_date_DaysInMonth $year $(( 10#${month} + ${i} )) ) ))
[619]387      else
[829]388        (( PeriodLengthInDays  = PeriodLengthInDays + $( IGCM_date_DaysInMonth $year $(( 10#${month} + ${i} - 12 )) ) ))
[619]389      fi
390      (( i=i+1 ))
391    done
[544]392    ;;
[619]393  *D|*d)
[544]394    PeriodLengthInMonths=0
[731]395    PeriodLengthInDays=$( echo ${config_UserChoices_PeriodLength} | sed -e 's/[dD]//' )
396    echo
397    IGCM_debug_Print 2 "Number of days for PeriodLength : ${PeriodLengthInDays}";;
[619]398  *)
[890]399    IGCM_debug_Exit "IGCM_config_DaysInPeriodLength " ${config_UserChoices_PeriodLength} " invalid period length : choose in *Y, *M, *D."
[544]400    IGCM_debug_Verif_Exit ;;
401  esac
[118]402
[890]403  IGCM_debug_PopStack "IGCM_config_DaysInPeriodLength"
[380]404}
[118]405
[380]406#===================================
[890]407function IGCM_config_DateCoherency
[891]408{
[890]409  IGCM_debug_PushStack "IGCM_config_DateCoherency"
410
411  echo
412  IGCM_debug_Print 1 "IGCM_config_DateCoherency"
413  echo
414
415  typeset Length VerifiedPeriodDateBegin VerifiedPeriodDateEnd
416
417  # check coherency between (PeriodDateBegin, PeriodDateEnd) and (DateBegin, CumulPeriod, PeriodLength)
418  # DateBegin + CumulPeriod*PeriodLength = PeriodDateBegin
419  echo
[922]420
[890]421  case ${config_UserChoices_PeriodLength} in
422  *Y|*y)
423    Length=$( IGCM_date_DaysInCurrentPeriod ${DateBegin} $(( ${CumulPeriod} * ${PeriodLengthInYears} ))Y )
424    ;;
425  *M|*m)
426    Length=$( IGCM_date_DaysInCurrentPeriod ${DateBegin} $(( ${CumulPeriod} * ${PeriodLengthInMonths} ))M )
427    ;;
428  *D|*d)
429    Length=$( IGCM_date_DaysInCurrentPeriod ${DateBegin} $(( ${CumulPeriod} * ${PeriodLengthInDays} ))D )
430    ;;
431  esac
[892]432  VerifiedPeriodDateEnd=$( IGCM_date_AddDaysToGregorianDate ${DateBegin} ${Length}-1 )
[890]433
[892]434  if [ ${VerifiedPeriodDateEnd} != ${PeriodDateEnd} ] ; then
435    IGCM_debug_Print 1 "From run.card PeriodDateEnd is not consistent with DateBegin and CumulPeriod."
436    IGCM_debug_Print 1 "We have DateBegin = ${DateBegin}"
437    IGCM_debug_Print 1 "We have CumulPeriod = ${CumulPeriod}"
438    IGCM_debug_Print 1 "We have PeriodDateEnd = ${PeriodDateEnd}"
439    IGCM_debug_Print 1 "We have VerifiedPeriodDateEnd = ${VerifiedPeriodDateEnd}"
440    IGCM_debug_Print 1 "You must have change run.card in an inconsistent way."
441
442    IGCM_debug_Exit "STOP here to avoid further issues."
443  fi
444
[890]445  # PeriodDateBegin + PeriodLength = PeriodDateEnd
446  VerifiedPeriodDateBegin=$( IGCM_date_AddDaysToGregorianDate ${VerifiedPeriodDateEnd} $(( ${PeriodLengthInDays} * -1 )) )
447
448  IGCM_debug_PopStack "IGCM_config_DateCoherency"
449}
450
451
452#===================================
[380]453function IGCM_config_Check
454{
[544]455  IGCM_debug_PushStack "IGCM_config_Check"
[380]456
[544]457  # If one of the following modulo is not zero :
458  # we will issue an error then explain and exit in
459  # AA_job IGCM_debug_Verif_Exit call before binary submission
[380]460
[544]461  echo
462  IGCM_debug_Print 1 "IGCM_config_Check"
463  echo
464
465  typeset i
466
[554]467  # Check RebuildFrequency against key frequencies : PeriodLength ; PackFrequency ; TimeSeriesFrequency ; SeasonalFrequency
[545]468  if ( [ ! X${config_Post_RebuildFrequency} = X${NULL_STR} ] && [ ! X${config_Post_RebuildFrequency} = XNONE ] ) ; then
[544]469    AsynchronousRebuild=true
470    IGCM_debug_Print 1 "Asynchronous rebuild has been activated."
[380]471    echo
[544]472    # modulo (RebuildFrequency and PeriodLength/TimeSeriesFrequency/SeasonalFrequency) must be zero
473    IGCM_debug_Print 1 "Check coherence between RebuildFrequency and PeriodLength"
474    IGCM_post_CheckModuloFrequency config_Post_RebuildFrequency config_UserChoices_PeriodLength
[554]475    IGCM_debug_Print 1 "Check coherence between PackFrequency and RebuildFrequency"
476    IGCM_post_CheckModuloFrequency config_Post_PackFrequency config_Post_RebuildFrequency
[544]477    IGCM_debug_Print 1 "Check coherence between TimeSeriesFrequency and RebuildFrequency"
478    IGCM_post_CheckModuloFrequency config_Post_TimeSeriesFrequency config_Post_RebuildFrequency
479    IGCM_debug_Print 1 "Check coherence between SeasonalFrequency and RebuildFrequency"
480    IGCM_post_CheckModuloFrequency config_Post_SeasonalFrequency config_Post_RebuildFrequency
481  else
482    AsynchronousRebuild=false
483    IGCM_debug_Print 1 "Asynchronous rebuild has not been activated"
484    IGCM_debug_Print 1 "Proceed with standard post-treatment pathway"
[380]485    echo
[554]486    #modulo (PeriodLength and TimeSeriesFrequency/SeasonalFrequency) must be zero
[544]487    IGCM_debug_Print 1 "Check coherence between TimeSeriesFrequency and PeriodLength"
488    IGCM_post_CheckModuloFrequency config_Post_TimeSeriesFrequency config_UserChoices_PeriodLength
489    IGCM_debug_Print 1 "Check coherence between SeasonalFrequency and PeriodLength"
490    IGCM_post_CheckModuloFrequency config_Post_SeasonalFrequency   config_UserChoices_PeriodLength
491  fi
[380]492
[619]493  # Check PackFrequency against other key frequencies
[554]494  # Modulo (PackFrequency and TimeSeriesFrequency/SeasonalFrequency and PeriodLenght) must be zero
495  if ( [ ! X${config_Post_PackFrequency} = X${NULL_STR} ] && [ ! X${config_Post_PackFrequency} = XNONE ] ) ; then
496    Pack=true
497    #
498    IGCM_debug_Print 1 "Check coherence between PackFrequency and PeriodLength"
499    IGCM_post_CheckModuloFrequency config_Post_PackFrequency config_UserChoices_PeriodLength
500    IGCM_debug_Print 1 "Check coherence between TimeSeriesFrequency and PackFrequency"
501    IGCM_post_CheckModuloFrequency config_Post_TimeSeriesFrequency config_Post_PackFrequency
502    IGCM_debug_Print 1 "Check coherence between SeasonalFrequency and PackFrequency"
503    IGCM_post_CheckModuloFrequency config_Post_SeasonalFrequency config_Post_PackFrequency
504  else
505    Pack=false
506  fi
507
[890]508  # modulo (TimeSeriesFrequency and all Chunck2D) must be zero
[544]509  NbJob=${#CHUNCK2D_SIZE[@]}
510  i=0
511  until [ $i -ge $NbJob ]; do
512    value=${CHUNCK2D_SIZE[${i}]}
513    IGCM_debug_Print 1 "Check coherence between All Chunck2D frequency and TimeSeriesFrequency"
514    IGCM_post_CheckModuloFrequency value config_Post_TimeSeriesFrequency
[619]515    case ${value} in
516    *Y|*y) ;;
[786]517    *)
518      IGCM_debug_Print 1 "All ChunckJob2D frequency must be expressed in year *Y|*y in comp.card"
[544]519      IGCM_debug_Exit "This will stop the job" ;;
520    esac
521    (( i=i+1 ))
522  done
[387]523
[890]524  # modulo (TimeSeriesFrequency and all Chunck3D) must be zero
[544]525  NbJob=${#CHUNCK3D_SIZE[@]}
526  i=0
527  until [ $i -ge $NbJob ]; do
528    value=${CHUNCK3D_SIZE[${i}]}
529    IGCM_debug_Print 1 "Check coherence between All Chunck3D frequency and TimeSeriesFrequency"
530    IGCM_post_CheckModuloFrequency value config_Post_TimeSeriesFrequency
[619]531    case ${value} in
532    *Y|*y) ;;
[786]533    *)
534      IGCM_debug_Print 1 "All ChunckJob3D frequency must be expressed in year *Y|*y in comp.card"
[544]535      IGCM_debug_Exit "This will stop the job" ;;
536    esac
537    (( i=i+1 ))
538  done
[380]539
[890]540  # check to be sure there is enough space on temporary filesystems to run
[731]541  echo
[651]542  IGCM_debug_Print 1 "Check if there is enough space on temporary filesystem"
543  IGCM_sys_check_quota
544
[933]545  # check to be sure that RUN_DIR_PATH, that will be removed is not pointing to an important directory
546  echo
547  IGCM_debug_Print 1 "Check where RUN_DIR_PATH variable is pointing to"
548  IGCM_sys_check_path
549
550
[544]551  IGCM_debug_PopStack "IGCM_config_Check"
[118]552}
553
554#===================================
[2]555function IGCM_config_PeriodStart
556{
[544]557  IGCM_debug_PushStack "IGCM_config_PeriodStart"
[2]558
[544]559  echo
560  IGCM_debug_Print 1 "IGCM_config_PeriodStart"
561  echo
[2]562
[544]563  if ( ${FirstInitialize} ) ; then
564    #================================================#
565    #         Initialize date/period information     #
566    #================================================#
[2]567
[890]568    IGCM_date_GetYearMonthDay ${DateBegin} year month day
569    IGCM_config_DaysInPeriodLength
[2]570
[939]571    PeriodDateBegin=${DateBegin}
[544]572    PeriodDateEnd=$( IGCM_date_AddDaysToGregorianDate ${DateBegin} $(( ${PeriodLengthInDays} - 1 )) )
[939]573    CumulPeriod=1
[2]574
[544]575    #=================================================#
576    #              Write updated run.card             #
577    #=================================================#
[2]578
[544]579    #Correct run.card Configuration for this period
580    IGCM_card_WriteOption ${SUBMIT_DIR}/run.card Configuration PeriodDateBegin ${PeriodDateBegin}
581    IGCM_card_WriteOption ${SUBMIT_DIR}/run.card Configuration PeriodDateEnd ${PeriodDateEnd}
582    IGCM_card_WriteOption ${SUBMIT_DIR}/run.card Configuration CumulPeriod ${CumulPeriod}
583    if [ X$( grep "SubmitPath" ${SUBMIT_DIR}/run.card ) != X ] ; then
584      IGCM_card_WriteOption ${SUBMIT_DIR}/run.card Configuration SubmitPath ${SUBMIT_DIR}
585    fi
[427]586
[544]587    IGCM_card_WriteOption ${SUBMIT_DIR}/run.card Configuration PeriodState "Running"
[619]588
[544]589  else
590    #================================================#
591    #         The file run.card allready exist       #
592    #================================================#
[2]593
[544]594    #Test state of run in run.card
595    IGCM_card_DefineVariableFromOption ${SUBMIT_DIR}/run.card Configuration PeriodState
596    if ( [ ${run_Configuration_PeriodState} != "Running" ] && [ ${run_Configuration_PeriodState} != "OnQueue" ] && [ ${run_Configuration_PeriodState} != "Continue" ] ) ; then
[619]597      echo
[544]598      IGCM_debug_Print 1 "!! Error in run.card with PeriodState : " ${run_Configuration_PeriodState} "!!"
599      IGCM_debug_Print 1 "Try running ${libIGCM}/clean_month.job to fix this"
600      echo
601      IGCM_debug_Exit
602      IGCM_debug_Verif_Exit
603    fi
[2]604
[544]605    #===================================#
[786]606    #        Read updated run.card      #
[544]607    #===================================#
[2]608
[544]609    IGCM_card_DefineVariableFromOption ${SUBMIT_DIR}/run.card Configuration PeriodDateBegin
610    IGCM_card_DefineVariableFromOption ${SUBMIT_DIR}/run.card Configuration PeriodDateEnd
611    IGCM_card_DefineVariableFromOption ${SUBMIT_DIR}/run.card Configuration CumulPeriod
[2]612
[544]613    PeriodDateBegin=$( IGCM_date_ConvertFormatToGregorian ${run_Configuration_PeriodDateBegin} )
614    PeriodDateEnd=$( IGCM_date_ConvertFormatToGregorian ${run_Configuration_PeriodDateEnd} )
[939]615    CumulPeriod=${run_Configuration_CumulPeriod}
[2]616
[1028]617    LastPeriodDateEnd=$( IGCM_date_AddDaysToGregorianDate $( IGCM_date_ConvertFormatToGregorian ${PeriodDateBegin} ) -1 )
[471]618
[544]619    if [ ${Period} = 1 ]; then
[910]620      # save last Job output and current run.card
621      typeset Potential
622      IGCM_sys_Cd ${SUBMIT_DIR}
623      #
624      IGCM_debug_Print 2 "Save previous ksh job output"
625      for Potential in $( ls ${Script_Output_Prefix}_${config_UserChoices_JobName}.[0-9][0-9][0-9][0-9][0-9][0-9] ) ; do
626        if [ X${Pack} = Xtrue ] ; then
[1103]627          ( IGCM_sys_TestFileBuffer  ${R_BUF_KSH}/${Potential} ) || IGCM_sys_Cp ${Potential} ${R_BUF_KSH}/${Potential}.$$
[910]628        else
[1103]629          ( IGCM_sys_TestFileArchive ${R_OUT_KSH}/${Potential} ) || IGCM_sys_Cp ${Potential} ${R_OUT_KSH}/${Potential}.$$
[910]630        fi
631      done
632      #
633      IGCM_debug_Print 2 "Save current run.card"
[544]634      IGCM_card_CheckConflict run.card
[634]635      if [ X${Pack} = Xtrue ] ; then
[1103]636        IGCM_sys_Cp ${SUBMIT_DIR}/run.card ${R_BUF_KSH}/run.card
[634]637      else
[1103]638        IGCM_sys_Cp ${SUBMIT_DIR}/run.card ${R_OUT_KSH}/run.card
[634]639      fi
[910]640      #
[544]641      IGCM_sys_Cd ${RUN_DIR}
642    else
643      unset FileToBeDeleted
644    fi
[2]645
[892]646    # Determine number of day(s) in PeriodLength
[544]647    IGCM_date_GetYearMonthDay $PeriodDateBegin year month day
[890]648    IGCM_config_DaysInPeriodLength
[2]649
[892]650    # Check coherency between (PeriodDateBegin, PeriodDateEnd) and (DateBegin, CumulPeriod, PeriodLength)
651    IGCM_config_DateCoherency
652    # And EXIT if not OK
653    IGCM_debug_Verif_Exit
654
[544]655    IGCM_card_WriteOption ${SUBMIT_DIR}/run.card Configuration PeriodState "Running"
[890]656  fi
[2]657
[544]658  # BEGIN: SHOULD GO IN A FUNCTION FROM libIGCM_date.ksh
659  # Compute year_m1 and year_p1 (year minus 1Y and year plus 1Y)
660  year_m1=$(( year - 1 ))
[619]661  year_p1=$(( year + 1 ))
[544]662  # Compute month_m1 (month minus 1M)
663  # Compute yyyymm_m1 (yyyymm minus 1M)
[829]664  month_m1=$(( 10#${month} - 1 ))
[544]665  if [ ${month_m1} = 0 ]; then
666    month_m1=12
667    yyyymm_m1=${year_m1}12
668  elif [ ${month_m1} -le 9 ]; then
669    month_m1=0${month_m1}
670    yyyymm_m1=${year}${month_m1}
671  else
672    yyyymm_m1=${year}${month_m1}
673  fi
674  # Compute month_p1 (month plus 1M)
675  # Compute yyyymm_p1 (yyyymm plus 1M)
[829]676  month_p1=$(( 10#${month} + 1 ))
[544]677  if [ ${month_p1} = 13 ]; then
678    month_p1=01
679    yyyymm_p1=${year_p1}01
680  elif [ ${month_p1} -le 9 ]; then
681    month_p1=0${month_p1}
682    yyyymm_p1=${year}${month_p1}
683  else
684    yyyymm_p1=${year}${month_p1}
685  fi
686  #IGCM_debug_Print 1 "jg 1 month_m1 = ${month_m1} month_p1 = ${month_p1} "
687  #IGCM_debug_Print 1 "jg 1 calculate yyyymm_m1 = ${yyyymm_m1} "
688  #IGCM_debug_Print 1 "jg 1 calculate yyyymm_p1 = ${yyyymm_p1} "
[522]689
[799]690  #===================================================================#
691  # Calculate CyclicYear to be used for looping over a given forcing  #
692  # period. Add CyclicBegin and CyclicEnd in config.card UserChoices. #
693  #===================================================================#
694
695  # To use the variable CyclicYear, one must add in config.card CyclicBegin and CyclicEnd.
696  # CyclicBegin is the first year in the cycle. CyclicEnd is the last year included in the cycle.
697  if ( [ ! X${config_UserChoices_CyclicBegin} = X ] && [ ! X${config_UserChoices_CyclicEnd} = X ] ) ; then
698    CycleNb=$(( ${config_UserChoices_CyclicEnd} - ${config_UserChoices_CyclicBegin} + 1 ))
[819]699    CyclicYear_p1=NOTDEFINED
700
701    # For current year
702    yeartmp=$year
703    diffy=$(( $yeartmp - ${config_UserChoices_CyclicBegin} ))
704    while [ $diffy -lt 0 ] ; do
705      yeartmp=$(( ${yeartmp} + ${CycleNb} ))
706      diffy=$(( $yeartmp - ${config_UserChoices_CyclicBegin} ))
707    done
708    CyclicYear=$(( ( ${diffy} % ${CycleNb} ) + ${config_UserChoices_CyclicBegin} ))
709
710    # For next coming year
711    yeartmp=$(( $year + 1 ))
712    diffy=$(( $yeartmp - ${config_UserChoices_CyclicBegin} ))
713    while [ $diffy -lt 0 ] ; do
714      yeartmp=$(( ${yeartmp} + ${CycleNb} ))
715      diffy=$(( $yeartmp - ${config_UserChoices_CyclicBegin} ))
716    done
717    CyclicYear_p1=$(( ( ${diffy} % ${CycleNb} ) + ${config_UserChoices_CyclicBegin} ))
718
[812]719    IGCM_debug_Print 1 "CyclicYear   = ${CyclicYear}, CyclicYear_p1 = ${CyclicYear_p1}, current year=$year"
[799]720  else
721    CyclicYear="ERROR_CyclicYear_Variable_Not_Defined"
[812]722    CyclicYear_p1="ERROR_CyclicYear_p1_Variable_Not_Defined"
[819]723    IGCM_debug_Print 1 "CyclicYear wont be use without adding CyclicBegin and CyclicEnd in config.card"
[799]724  fi
725
[544]726  # END: SHOULD GO IN A FUNCTION FROM libIGCM_date.ksh
[522]727
[544]728  #===================================================================#
729  # Prepare variables available for ${COMP}.card and ${COMP}.driver   #
[786]730  #             But available to any son functions                    #
[544]731  #===================================================================#
[2]732
[544]733  # Period Length In Days between DateBegin and DateCurrent (at end of period == PeriodDateEnd !)
734  (( SimulationLengthInDays = $( IGCM_date_DaysBetweenGregorianDate ${PeriodDateEnd} ${DateBegin} ) + 1 ))
[2]735
[544]736  # Debug Print :
[731]737  echo
[544]738  IGCM_debug_Print 1 "IGCM_config_PeriodStart : Before Execution"
739  IGCM_debug_Print 1 "Year of simulation      : ${year}"
740  IGCM_debug_Print 1 "Month of simulation     : ${month}"
741  IGCM_debug_Print 1 "PeriodLengthInDays      : ${PeriodLengthInDays}"
742  IGCM_debug_Print 1 "PeriodDateBegin         : ${PeriodDateBegin}"
743  IGCM_debug_Print 1 "PeriodDateEnd           : ${PeriodDateEnd}"
744  IGCM_debug_Print 1 "SimulationLengthInDays  : ${SimulationLengthInDays}"
745  IGCM_debug_Print 1 "ExperienceLengthInDays  : ${ExperienceLengthInDays}"
[2]746
[544]747  #================================================================#
748  #         Prepare variables available for comp_finalyze          #
749  #================================================================#
[2]750
[544]751  # Period for save files
[939]752  DatesPeriod=${PeriodDateBegin}_${PeriodDateEnd}
[2]753
[544]754  # Prefix for save files of this period
[939]755  PREFIX=${config_UserChoices_JobName}_${DatesPeriod}
[2]756
[544]757  # List of files that will be deleted in RUN_DIR after run
758  [ -f stack ] && FileToBeDeleted[0]="stack"
[2]759
[544]760  # Test if the same run as already been saved :
761  if [ X${JobType} = XRUN ] ; then
762    if [ ${DRYRUN} -le 0 ] ; then
763      if ( IGCM_sys_TestFileBuffer ${R_BUF_KSH}/${PREFIX}_${Exe_Output} ) ; then
[619]764        IGCM_debug_Exit "IGCM_config_PeriodStart" "You are currently RErunning an old job."
765        IGCM_debug_Print 1 "Because of readonly permissions, you can't RErun a job when saved files"
766        IGCM_debug_Print 1 " are still in the ARCHIVE directory. You must deleted those files, or "
767        IGCM_debug_Print 1 " the whole ${R_SAVE} tree. See clean_month.job in ${libIGCM} directory."
768        IGCM_debug_Print 1 " This exit has been initiated because at least ${R_BUF_KSH}/${PREFIX}_${Exe_Output} exists."
769        IGCM_debug_Verif_Exit
[544]770      fi
[2]771    fi
[544]772  else
773    if ( IGCM_sys_TestFileBuffer ${R_BUF_KSH}/${PREFIX}_${Exe_Output} ) ; then
774      IGCM_debug_Print 1 "IGCM_config_PeriodStart" "RErun an old job. Allowed in DEBUG or DEV mode."
775    fi
776  fi
[128]777
[939]778  #================================================================#
779  #       Prepare variables available for binary execution         #
[1009]780  #       Call function for creation of run script                 #
781  #       Only done once per job                                   #
[939]782  #================================================================#
[616]783
[1009]784  if [ ${Period} -eq 1 ]; then
785    typeset ExeNameIn ExeNameFirst CompNameFirst comp i j
786    typeset tempvar tempvarMPI tempvarNOD NbElts NbExec
[616]787
[1009]788    PROCESSUS_NUMBER=0
789    NbExec=0
790    i=0
[939]791
[1009]792    OK_PARA_MPI=false
793    OK_PARA_OMP=false
794    OK_PARA_NOD=false
795    OK_PARA_MPMD=false
[616]796
[1009]797    for comp in ${config_ListOfComponents[*]} ; do
[616]798
[1009]799      IGCM_debug_Print 1 ${comp}
[616]800
[1009]801      eval ExeNameIn=\${config_Executable_${comp}[0]}
[616]802
[1009]803      # NO order in config.card for parallelized values !
804      # just use suffix : MPI , OMP and NOD (for number of NODes.)
[616]805
[1009]806      # NOD is the number of NODes allocated
807      eval ${comp}_PROC_NOD=0
[616]808
[1009]809      # MPI is the number of MPI processus per nodes
810      eval ${comp}_PROC_MPI=0
[616]811
[1009]812      # OMP is the number of OpenMP threads per MPI processus
813      eval ${comp}_PROC_OMP=0
[619]814
[1009]815      # Only if we really have an executable for the component :
816      if ( [ "X${ExeNameIn}" != X\"\" ] && [ "X${ExeNameIn}" != "Xinca.dat" ] ) ; then
[616]817
[1009]818        # Keep the first executable found and the first CompName
819        ExeNameFirst=${ExeNameIn}
820        CompNameFirst=${comp}
[616]821
[1009]822        # Are we a second executable?
823        (( NbExec = NbExec + 1 ))
[616]824
[1009]825        # set 1 MPI task, 1 OpenMP thread and 1 node as default
826        eval ${comp}_PROC_MPI=1
827        eval ${comp}_PROC_OMP=1
828        eval ${comp}_PROC_NOD=1
[616]829
[1009]830        eval NbElts=\${#config_Executable_${comp}[@]}
[616]831
[1009]832        if [ ${NbElts} -ge 2 ] ; then
833          (( j = 2 ))
834          while [ $j -lt ${NbElts} ] ; do
835            eval tempvar=\${config_Executable_${comp}[${j}]}
836            IGCM_debug_Print 2 ${tempvar}
[616]837
[1009]838            if [ X${tempvar} = X ] ; then
839              IGCM_debug_Print 2 "Error reading MPI/OMP parameters !!!"
840              IGCM_debug_Exit "Check your config.card. Exit now"
841              IGCM_debug_Verif_Exit
842            fi
[616]843
[1009]844            case ${tempvar} in
845            *[mM][pP][iI]*)
846              # Read MPI parameter for composante
847              eval ${comp}_PROC_MPI=$( echo ${tempvar} | tr '[a-z]' '[A-Z]' | sed -e "s/MPI//" )
848              OK_PARA_MPI=true;;
849            *[oO][mM][pP]*)
850              # Read OMP parameter for composante
851              eval ${comp}_PROC_OMP=$( echo ${tempvar} | tr '[a-z]' '[A-Z]' | sed -e "s/OMP//" )
[1173]852              ;;
[1009]853            *[nN][oO][dD]*)
854              # Read NOD (NumBer of Nodes) parameter for composante
855              eval ${comp}_PROC_NOD=$( echo ${tempvar} | tr '[a-z]' '[A-Z]' | sed -e "s/NOD//" )
856              OK_PARA_NOD=true
857              OK_PARA_MPI=true;;
858            esac
859            (( j = j + 1 ))
860          done
861        fi
862        eval tempvarMPI=\${${comp}_PROC_MPI}
863        eval tempvarNOD=\${${comp}_PROC_NOD}
864        eval tempvarOMP=\${${comp}_PROC_OMP}
865
[1173]866        # set OMP mode if more than 1 OMP thread.
867        [ ${tempvarOMP} -ge 2 ] && OK_PARA_OMP=true
868
[1009]869        (( PROCESSUS_NUMBER = PROCESSUS_NUMBER + tempvarMPI * tempvarNOD * tempvarOMP ))
[616]870      fi
[1009]871      (( i=i+1 ))
872    done
[616]873
[1009]874    # set MPMD mode if more than 2 executable names.
875    [ ${NbExec} -ge 2 ] && OK_PARA_MPMD=true
[616]876
[1009]877    # Verification of BATCH_NUM_PROC_TOT total number of processors set in job header.
878    if [ X${BATCH_NUM_PROC_TOT} != X ] ; then
879      # BATCH_NUM_PROC_TOT is set
880      if ( ${OK_PARA_MPI} ) ; then
881        IGCM_debug_Print 1 "MPI/OMP/NOD found into config.card and BATCH_NUM_PROC_TOT = ${BATCH_NUM_PROC_TOT} "
882      else
883        # with previous method.
884        if [ ${BATCH_NUM_PROC_TOT} -gt 1 ] ; then
885          # with more than 1 proc
886          if ( ${OK_PARA_MPMD} ) ; then
887            # with MPMD ie CPL/oasis method
888            IGCM_debug_Print 2 "Use default number of MPI tasks for this machine : "
889            IGCM_debug_Print 2 "${DEFAULT_NUM_PROC_OCE} for OCE"
890            IGCM_debug_Print 2 "${DEFAULT_NUM_PROC_CPL} for CPL"
891            IGCM_debug_Print 2 "${DEFAULT_NUM_PROC_ATM} for ATM"
892            OK_PARA_MPI=true
893            CPL_PROC_MPI=${DEFAULT_NUM_PROC_CPL}
894            OCE_PROC_MPI=${DEFAULT_NUM_PROC_OCE}
895            ATM_PROC_MPI=${DEFAULT_NUM_PROC_ATM}
896            PROCESSUS_NUMBER=${DEFAULT_NUM_PROC_TOTAL}
897          else
898            # with have only one executable
899            IGCM_debug_Print 2 "Use ${BATCH_NUM_PROC_TOT} MPI tasks for ${CompNameFirst} : ${ExeNameFirst} "
900            OK_PARA_MPI=true
901            eval ${CompNameFirst}_PROC_MPI=${BATCH_NUM_PROC_TOT}
902            PROCESSUS_NUMBER=${BATCH_NUM_PROC_TOT}
903          fi
[616]904        else
[1009]905          PROCESSUS_NUMBER=1
[616]906        fi
[1009]907      fi
908      # Verification with PBS parameter
909      if [ ${BATCH_NUM_PROC_TOT} -ne ${PROCESSUS_NUMBER} ] ; then
910        IGCM_debug_Exit "ERROR with parallelization parameters !"
911        IGCM_debug_Print 2 "Job header variable BATCH_NUM_PROC_TOT = ${BATCH_NUM_PROC_TOT} "
912        IGCM_debug_Print 2 "is the total number of _processors_ reserved."
913        IGCM_debug_Print 2 "It is not equal to the sum of _processus_  = ${PROCESSUS_NUMBER}."
914        IGCM_debug_Verif_Exit
915      fi
916      NUM_PROC_CPL=${CPL_PROC_MPI} # for backward compatibility
917      NUM_PROC_OCE=${OCE_PROC_MPI} # for backward compatibility
918      NUM_PROC_ATM=${ATM_PROC_MPI} # for backward compatibility
919    else # BATCH_NUM_PROC_TOT=""
920      if ( ${OK_PARA_MPI} ) ; then
921        IGCM_debug_Exit "ERROR : missing value for ${BATCH_NUM_PROC_TOT} processors,"
922        IGCM_debug_Print 2 "You have parallel parameters in config->Executable->list."
923        IGCM_debug_Print 2 "Please add BATCH_NUM_PROC_TOT variable in job header as well."
924        IGCM_debug_Exit "Exit now."
925        IGCM_debug_Verif_Exit
[619]926      else
[1009]927        # sequential case !
928        if [ ${PROCESSUS_NUMBER} -eq 0 ] ; then
929          (( PROCESSUS_NUMBER = 1 ))
930          IGCM_debug_Print 2 "PROCESSUS_NUMBER is all 0 (sequential use of old definition in config->Executable->list)."
931          IGCM_debug_Print 2 "We set it to 1."
932        fi
[616]933      fi
934    fi
935
[1009]936    IGCM_debug_Print 1 "MPI/OMP treatment PROCESSUS_NUMBER = ${PROCESSUS_NUMBER}"
[616]937
[1009]938    IGCM_sys_build_execution_scripts
939  fi
[616]940
[544]941  ExecutionFail=false
[473]942
[544]943  IGCM_debug_PopStack "IGCM_config_PeriodStart"
[2]944}
945
946#===================================
[523]947function IGCM_config_SaveSourceModifications
948{
[544]949  IGCM_debug_PushStack "IGCM_config_SaveSourceModifications"
[523]950
[544]951  typeset ExeOutDateMax listVarEnv
952  ExeOutDateMax=$1
[523]953
[544]954  listVarEnv="ExeOutDateMax,R_OUT_EXE,PREFIX,SUBMIT_DIR"
955  IGCM_sys_RshMaster "\
[545]956    . ${libIGCM}/libIGCM_sys/libIGCM_sys.ksh; \
[544]957    export ExeOutDateMax=${ExeOutDateMax};\
[545]958    export R_OUT_EXE=${R_OUT_EXE};\
959    export PREFIX=${PREFIX};\
960    export SUBMIT_DIR=${SUBMIT_DIR};\
961    export listVarEnv=${listVarEnv};\
962    Script_Output=out_SaveSourceModifications;\
963    IGCM_sys_Qsub ${libIGCM}/SaveSourceModifications.job ${ExeOutDateMax} ${R_OUT_EXE} ${PREFIX} ${SUBMIT_DIR}"
[523]964
[544]965  IGCM_debug_PopStack "IGCM_config_SaveSourceModifications"
[523]966}
967
968#===================================
[2]969function IGCM_config_PeriodEnd
970{
[544]971  IGCM_debug_PushStack "IGCM_config_PeriodEnd"
[2]972
[544]973  echo
974  IGCM_debug_Print 1 "IGCM_config_PeriodEnd"
975  echo
[2]976
[622]977  if [ ${DRYRUN} -le 1 ] ; then
[457]978
[622]979    IGCM_debug_Print 1 "Check components binary : size and creation date"
[2]980
[622]981    typeset LS_comp LS_bin ExeDate ExeCpuLog NextExeSize LastCompExeSize
982    typeset comp i
983    typeset ExeNameIn ExeNameOut UpdateExe ExeSecDateMax
[2]984
[622]985    #==================================#
986    #        Get last Exe Size         #
987    #==================================#
[2]988
[622]989    (( i=0 ))
990    if ( ${FirstInitialize} ) ; then
991      run_Log_LastExeSize=""
992      for comp in ${config_ListOfComponents[*]} ; do
993        run_Log_LastExeSize[$i]=0
994        (( i=i+1 ))
995      done
996    else
997      IGCM_card_DefineArrayFromOption ${SUBMIT_DIR}/run.card Log LastExeSize
998    fi
999    #==================================#
1000    #         And Build ExeDate        #
1001    #==================================#
[2]1002
[622]1003    # ExeDate = ATM_Jun_12_09:34-SRF_Jun_12_09:34-OCE_Jun_12_09:34-ICE_Jun_12_09:34-CPL_Jun_12_09:33
1004    # Would be nice to have next line but no way to format ls output (need to ls -l --time-style "+%Y-%m-%dT%H:%M")
1005    # ExeDate = ATM_2009-06-12T09:34+SRF_2009-06-12T09:34+OCE_2009-06-12T09:34+ICE_2009-06-12T09:34+CPL_2009-06-12T09:34
1006    ExeDate=""
1007    NextExeSize="( "
1008    (( i=0 ))
1009    UpdateExe=false
1010    (( ExeSecDateMax = 0 ))
1011    for comp in ${config_ListOfComponents[*]} ; do
[2]1012
[622]1013      IGCM_debug_Print 3 ${comp}
[2]1014
[622]1015      eval ExeNameIn=\${config_Executable_${comp}[0]}
1016      eval ExeNameOut=\${config_Executable_${comp}[1]}
1017      # Only if we really have an executable for the component :
1018      if [ X${ExeNameIn} = X\"\" ] ; then
1019        # If there is no exe file for this component
1020        (( ExeSize=0 ))
[619]1021      else
[939]1022        LS_bin=${R_EXE}/${ExeNameIn}
[622]1023        IGCM_sys_FileSize ${LS_bin} ExeSize
1024
1025        set +A LS_comp -- $( LC_TIME=en_US ls -l ${LS_bin} )
1026        if [ X${ExeDate} = X ] ; then
1027          # First component exe date
1028          ExeDate=${comp}_${LS_comp[5]}_${LS_comp[6]}
1029        else
1030          ExeDate=${ExeDate}-${comp}_${LS_comp[5]}_${LS_comp[6]}
1031        fi
1032        ExeDate=${ExeDate}_${LS_comp[7]}
[619]1033      fi
[2]1034
[622]1035      if [ ${i} -eq 0 ] ; then
1036        # First component
1037        NextExeSize="( "${ExeSize}
[619]1038      else
[622]1039        NextExeSize=${NextExeSize}", "${ExeSize}
[619]1040      fi
[939]1041      LastCompExeSize=${run_Log_LastExeSize[$i]}
[622]1042      (( i=i+1 ))
[523]1043
[622]1044      if [ ${ExeSize} -ne ${LastCompExeSize} ] ; then
1045        if ( ${FirstInitialize} ) ; then
1046          IGCM_debug_Print 1 "Save first ${ExeNameIn} in ${R_OUT_EXE} !"
1047        else
1048          IGCM_debug_Print 1 "${ExeNameIn} has changed in ${R_EXE} !"
1049          IGCM_debug_Print 1 "Save latest ${ExeNameIn} in ${R_OUT_EXE} !"
[939]1050          FileToBeDeleted[${#FileToBeDeleted[@]}]=${ExeNameOut}
[622]1051        fi
[939]1052        IGCM_sys_Put_Out ${ExeNameOut} ${R_OUT_EXE}/${PREFIX}_${ExeNameIn} rw
[622]1053        UpdateExe=true
1054
[691]1055        # SD : switch off for now
1056        #IGCM_sys_GetDate_FichWork ${LS_bin} ExeSecDate
1057        #if [ $ExeSecDateMax -lt $ExeSecDate ] ; then
1058        #  ExeSecDateMax=$ExeSecDate
1059        #fi
[619]1060      fi
[622]1061    done
[2]1062
[622]1063    # SD : switch off for now
1064    #if ( ${UpdateExe} ) ; then
1065    #  echo "Launch SaveSourceModifications."
1066    #  IGCM_config_SaveSourceModifications ${ExeSecDateMax}
1067    #fi
[523]1068
[622]1069    NextExeSize=${NextExeSize}" )"
1070    IGCM_card_WriteOption ${SUBMIT_DIR}/run.card Log LastExeSize "${NextExeSize}"
[457]1071
[622]1072    if [ ${DRYRUN} -le 1 ] ; then
1073      tail -1500 ${Exe_Output} > ${Exe_Output}_tail.txt
1074      ExeCpuLog=$( gawk -f ${libIGCM}/libIGCM_sys/IGCM_add_out.awk ${Exe_Output}_tail.txt )
1075      RET=$?
1076      if [ $RET -eq 0 ] ; then
1077        # ExeCpuLog variable contents 5 fields
1078        echo "${CumulPeriod} ${PeriodDateBegin} ${PeriodDateEnd} ${ExeCpuLog} ${ExeDate}" |   \
1079          gawk '{printf("# %11d | %15s | %15s | %19s | %19s | %15.5f | %15.5f | %15.5f | %s\n", \
[544]1080            $1,$2,$3,$4,$5,$6,$7,$8,$9)}' >> ${SUBMIT_DIR}/run.card
[622]1081      fi
[939]1082      FileToBeDeleted[${#FileToBeDeleted[@]}]=${Exe_Output}_tail.txt
[619]1083    fi
[622]1084
[619]1085  fi
[2]1086
[1041]1087  #==================================#
1088  #         Save Job output          #
1089  #==================================#
1090  if [ X${Pack} = Xtrue ] ; then
1091    IGCM_sys_PutBuffer_Out ${Exe_Output} ${R_BUF_KSH}/${PREFIX}_${Exe_Output}
1092  else
1093    IGCM_sys_Put_Out ${Exe_Output} ${R_OUT_KSH}/${PREFIX}_${Exe_Output}
1094  fi
1095  FileToBeDeleted[${#FileToBeDeleted[@]}]=${Exe_Output}
1096
[622]1097  # All was right ? no ? then we stop.
[619]1098  IGCM_debug_Verif_Exit
[2]1099
[622]1100  # If all was OK, we can delete all files not necessary for next Job
[619]1101  echo
1102  IGCM_debug_Print 1 "Files that will be deleted before next period-run : "
[2]1103
[619]1104  if [ ${DRYRUN} -le 2 ] ; then
1105    for f in ${FileToBeDeleted[@]} ; do [ -f ${f} ] && ls -la $f ; [ -f ${f} ] && rm -f $f ; done
1106  else
1107    echo ${FileToBeDeleted[@]}
1108  fi
[2]1109
[922]1110  # Send some accounting element to the user if CumulPeriod=3
1111  if [ ${CumulPeriod} -eq 3 ] ; then
1112    echo
1113    IGCM_debug_Print 1 "Send email containing some accounting information : "
1114
1115    RealCpuTime=$( echo ${ExeCpuLog} | gawk '{print $3}' )
1116
1117    consumeHoursPerPeriod=$( echo "scale=6;${RealCpuTime}*${PROCESSUS_NUMBER}/3600" | bc )
1118
1119    consumeHoursPerWholeSimulation=$( echo "scale=6;${consumeHoursPerPeriod}/${PeriodLengthInDays}*${ExperienceLengthInDays}" | bc )
1120
1121    recommendedPeriodNb=$( echo "scale=6;24/${consumeHoursPerPeriod}*${PROCESSUS_NUMBER}" | bc )
1122
1123    IGCM_sys_SendMail Accounting
1124  fi
1125
[622]1126  #=================================================#
1127  #         Modification of libIGCM behaviour       #
1128  #=================================================#
[119]1129
[622]1130  # To use this function, one must copy libIGCM.card from ${libIGCM} directory
1131  # and put it in ${SUBMIT_DIR} directory. After modifications of ${SUBMIT_DIR}/libIGCM.card,
1132  # variables define inside [UserChanges] will be modified for next Period of libIGCM main loop.
[619]1133  if [ -f ${SUBMIT_DIR}/libIGCM.card ] ; then
1134    echo
1135    echo "########################################################################"
1136    echo "!!!                 Modification of libIGCM behaviour                !!!"
1137    echo
[119]1138
[619]1139    IGCM_debug_Print 1 "DefineArrayFromOption  : libIGCM_UserChanges in libIGCM.card"
1140    IGCM_card_DefineArrayFromSection ${SUBMIT_DIR}/libIGCM.card UserChanges
1141    IGCM_debug_Print 2 "libIGCM_UserChanges" ${libIGCM_UserChanges[*]}
[119]1142
[622]1143    # Special treatments for libIGCM internals
[619]1144    for option in ${libIGCM_UserChanges[*]} ; do
1145      IGCM_card_DefineVariableFromOption ${SUBMIT_DIR}/libIGCM.card UserChanges ${option}
[119]1146
[619]1147      echo "We will change : ${option}."
1148      eval echo "Previous value : " \${${option}}
1149      eval echo "Change to : " \${libIGCM_UserChanges_${option}}
[119]1150
[619]1151      eval ${option}=\${libIGCM_UserChanges_${option}}
[119]1152
[619]1153      case ${option} in
1154      config_UserChoices_DateEnd)
1155        IGCM_debug_PrintVariables 1 config_UserChoices_DateEnd
1156        DateEnd=$( IGCM_date_ConvertFormatToGregorian ${config_UserChoices_DateEnd} )
1157
[622]1158        # Period Length In Days between DateBegin and DateEnd
[619]1159        (( ExperienceLengthInDays=$( IGCM_date_DaysBetweenGregorianDate ${DateEnd} ${DateBegin} )  + 1 ))
1160        if [ ${ExperienceLengthInDays} -lt 0 ] ; then
1161          IGCM_debug_Print 1 "Problem with dates in libIGCM.card : ${DateEnd} < ${DateBegin} ! You must check that."
[939]1162          IGCM_debug_Exit "IGCM_PeriodEnd have wrong dates."
[619]1163          IGCM_debug_Verif_Exit
1164        fi
1165        ;;
1166      config_UserChoices_PeriodLength)
1167        IGCM_debug_Print 1  "Change config_UserChoices_PeriodLength=${config_UserChoices_PeriodLength}"
1168        ;;
1169      PeriodNb)
1170        IGCM_debug_Print 1  "Loop in main Job with ${PeriodNb} period(s)"
1171        ;;
1172      config_Post_RebuildFrequency)
1173        IGCM_debug_Print 1  "Change config_Post_RebuildFrequency=${config_Post_RebuildFrequency} : IGCM_post_Configure"
1174        IGCM_post_Configure
1175        ;;
1176      config_Post_TimeSeriesFrequency)
[653]1177        IGCM_debug_Print 1  "Change config_Post_TimeSeriesFrequency = ${config_Post_TimeSeriesFrequency} : IGCM_post_Configure"
[619]1178        IGCM_post_Configure
1179        ;;
1180      config_Post_SeasonalFrequency)
[653]1181        IGCM_debug_Print 1  "Change config_Post_SeasonalFrequency = ${config_Post_SeasonalFrequency} : IGCM_post_Configure"
[619]1182        IGCM_post_Configure
1183        ;;
1184      esac
1185    done
[119]1186
[619]1187    echo
1188    echo "########################################################################"
1189    echo
[544]1190  fi
[119]1191
[544]1192  #=================================================#
1193  #         Determine next computed period          #
1194  #=================================================#
[2]1195
[544]1196  PeriodDateBegin=$( IGCM_date_AddDaysToGregorianDate ${PeriodDateEnd} 1 )
1197  IGCM_date_GetYearMonthDay $PeriodDateBegin year month day
1198  year_m1=$(( year - 1 ))
1199  year_p1=$(( year + 1 ))
[890]1200  IGCM_config_DaysInPeriodLength
[544]1201  PeriodDateEnd=$( IGCM_date_AddDaysToGregorianDate ${PeriodDateBegin} $(( ${PeriodLengthInDays} - 1 )) )
[2]1202
[544]1203  # Debug Print :
1204  echo
1205  IGCM_debug_Print 1 "IGCM_config_PeriodEnd : Preparing Next Execution"
1206  IGCM_debug_Print 1 "PeriodDateBegin       : ${PeriodDateBegin}"
1207  IGCM_debug_Print 1 "PeriodDateEnd         : ${PeriodDateEnd}"
1208  IGCM_debug_Print 1 "PeriodLengthInDays    : ${PeriodLengthInDays}"
[2]1209
[544]1210  PeriodDateBegin=$( IGCM_date_ConvertFormatToHuman ${PeriodDateBegin} )
1211  PeriodDateEnd=$( IGCM_date_ConvertFormatToHuman ${PeriodDateEnd} )
[2]1212
[544]1213  (( CumulPeriod = CumulPeriod + 1 ))
[2]1214
[544]1215  # Debug Print :
1216  echo
1217  IGCM_debug_Print 3 "PeriodDateBegin Human : ${PeriodDateBegin}"
1218  IGCM_debug_Print 3 "PeriodDateEnd Human   : ${PeriodDateEnd}"
1219  IGCM_debug_Print 3 "CumulPeriod           : ${CumulPeriod}"
[2]1220
[544]1221  #=================================================#
[786]1222  #             Write updated run.card              #
[544]1223  #=================================================#
[2]1224
[544]1225  IGCM_card_WriteOption ${SUBMIT_DIR}/run.card Configuration PeriodDateBegin ${PeriodDateBegin}
1226  IGCM_card_WriteOption ${SUBMIT_DIR}/run.card Configuration PeriodDateEnd ${PeriodDateEnd}
1227  IGCM_card_WriteOption ${SUBMIT_DIR}/run.card Configuration CumulPeriod ${CumulPeriod}
[2]1228
[544]1229  if ( ${FirstInitialize} ) ; then
1230    # It's no more the first time
1231    FirstInitialize=false
1232  fi
[446]1233
[544]1234  IGCM_debug_PopStack "IGCM_config_PeriodEnd"
[2]1235}
1236
1237#===================================
1238function IGCM_config_Finalize
1239{
[544]1240  IGCM_debug_PushStack "IGCM_config_Finalize"
[2]1241
[544]1242  echo
1243  IGCM_debug_Print 1 "IGCM_config_Finalize"
1244  echo
[2]1245
[544]1246  if [ ${SimulationLengthInDays} -ge ${ExperienceLengthInDays} ] ; then
1247    #==========================#
1248    # End of entire simulation #
1249    #==========================#
[1189]1250    simulationIsOver=true
[2]1251
[544]1252    # Mail notification
1253    IGCM_sys_SendMail
1254    #
1255    IGCM_card_WriteOption ${SUBMIT_DIR}/run.card Configuration PeriodState "Completed"
[920]1256    #
[544]1257    IGCM_debug_Print 1 "Normal End of computation."
[873]1258
[544]1259  else
1260    #=================#
1261    # Submit next job #
1262    #=================#
[1189]1263    simulationIsOver=false
[2]1264
[544]1265    IGCM_card_WriteOption ${SUBMIT_DIR}/run.card Configuration PeriodState "OnQueue"
[2]1266
[544]1267    # Name of next Ksh Script output :
[939]1268    Script_Output=${Script_Output_Prefix}_${config_UserChoices_JobName}.$( printf "%06d" ${CumulPeriod} )
[2]1269
[544]1270    IGCM_debug_Print 1 "Submit next job"
[922]1271    # SUBMIT NEXT JOB from SUBMIT_DIR and come back in RUN_DIR
1272    IGCM_sys_Cd ${SUBMIT_DIR}
[544]1273    # Keep only the 5 latest ${Script_Output_Prefix}_${config_UserChoices_JobName}
1274    ScriptTot=$( ls ${Script_Output_Prefix}_${config_UserChoices_JobName}.?????? 2>/dev/null | wc -l )
1275    [ ${ScriptTot} -gt 5 ] && rm -f $( ls ${Script_Output_Prefix}_${config_UserChoices_JobName}.?????? | head -$(( ${ScriptTot} - 5 )) )
[861]1276    # Submit next job and come back
[544]1277    IGCM_sys_Qsub ${SUBMIT_DIR}/Job_${config_UserChoices_JobName}
[861]1278    IGCM_sys_Cd -
[544]1279  fi
[154]1280
[804]1281  # Clean ${RUN_DIR}=${RUN_DIR_PATH}/${config_UserChoices_JobName}.${$}
[544]1282  # Only for production run (No clean up in DEV or DEB mode)
1283  # and command sent from .. directory.
1284  IGCM_sys_Cd ..
[943]1285  [ X${JobType} = XRUN ] && IGCM_sys_RmRunDir -rf ${RUN_DIR_PATH}
[544]1286
[1189]1287  # Inform the rabbitMQ queue
1288  IGCM_debug_BigBro_Finalize
[1076]1289
[544]1290  IGCM_debug_PopStack "IGCM_config_Finalize"
[2]1291}
1292
1293#===================================
Note: See TracBrowser for help on using the repository browser.