Context Navigation

source: trunk/libIGCM/AA_create_ts @ 254

Last change on this file since 254 was 254, checked in by sdipsl, 14 years ago
Use ALL for RESOL_ATM in create_ts.job and thus in monitoring.job. So that we use this file atlas/monitoring01_lmdz_ALL.cfg
Property licence set to The following licence information concerns ONLY the libIGCM tools ================================================================== Copyright © Centre National de la Recherche Scientifique CNRS Commissariat à l'Énergie Atomique CEA libIGCM : Library for Portable Models Computation of IGCM Group. IGCM Group is the french IPSL Global Climate Model Group. This library is a set of shell scripts and functions whose purpose is the management of the initialization, the launch, the transfer of output files, the post-processing and the monitoring of datas produce by any numerical program on any plateforme. This software is governed by the CeCILL license under French law and abiding by the rules of distribution of free software. You can use, modify and/ or redistribute the software under the terms of the CeCILL license as circulated by CEA, CNRS and INRIA at the following URL "http://www.cecill.info". As a counterpart to the access to the source code and rights to copy, modify and redistribute granted by the license, users are provided only with a limited warranty and the software's author, the holder of the economic rights, and the successive licensors have only limited liability. In this respect, the user's attention is drawn to the risks associated with loading, using, modifying and/or developing or reproducing the software by the user in light of its specific status of free software, that may mean that it is complicated to manipulate, and that also therefore means that it is reserved for developers and experienced professionals having in-depth computer knowledge. Users are therefore encouraged to load and test the software's suitability as regards their requirements in conditions enabling the security of their systems and/or data to be ensured and, more generally, to use and operate it in the same conditions as regards security. The fact that you are presently reading this means that you have had knowledge of the CeCILL license and that you accept its terms. Property svn:keywords set to `Date Author Revision`
File size: 31.2 KB

Line
1	#-Q- platine #!/usr/bin/ksh
2	#-Q- platine ###################
3	#-Q- platine ## PLATINE CEA ##
4	#-Q- platine ###################
5	#-Q- platine #BSUB -J TS # Nom du job
6	#-Q- platine #BSUB -N # message a la fin du job
7	#-Q- platine #BSUB -n 1 # reservation des processeurs pour le job
8	#-Q- platine #BSUB -W 1:00 # Limite temps
9	#-Q- platine #BSUB -q post # Passage en queue post
10	#-Q- sx8brodie #!/bin/ksh
11	#-Q- sx8brodie #######################
12	#-Q- sx8brodie ## SX8BRODIE IDRIS ##
13	#-Q- sx8brodie #######################
14	#-Q- sx8brodie # Temps Elapsed max. d'une requete hh:mm:ss
15	#-Q- sx8brodie # @ wall_clock_limit = 20:00:00
16	#-Q- sx8brodie # Nom du travail LoadLeveler
17	#-Q- sx8brodie # @ job_name = TS
18	#-Q- sx8brodie # Fichier de sortie standard du travail
19	#-Q- sx8brodie # @ output = $(job_name).$(jobid)
20	#-Q- sx8brodie # Fichier de sortie d'erreur du travail
21	#-Q- sx8brodie # @ error = $(job_name).$(jobid)
22	#-Q- sx8brodie # pour recevoir un mail en cas de depassement du temps Elapsed (ou autre pb.)
23	#-Q- sx8brodie # @ notification = error
24	#-Q- sx8brodie # @ environment = $DEBUG_debug ; $MODIPSL ; $libIGCM ; $libIGCM_SX ; $SUBMIT_DIR ; $REBUILD_DIR ; $RebuildFromArchive ; $POST_DIR ; $MASTER ; $RebuildFrequency ; $PeriodDateBegin ; $PeriodDateEnd ; $NbRebuildDir ; $StandAlone ; $CompletedFlag ; $TsTask ; $CompToRead ; $FlagToRead ; $RESOL_ATM ; $RESOL_OCE ; $RESOL_ICE ; $RESOL_MBG ; $RESOL_SRF ; $RESOL_SBG ; $MASTER
25	#-Q- aix6 #!/bin/ksh
26	#-Q- aix6 #######################
27	#-Q- aix6 ## VARGAS IDRIS ##
28	#-Q- aix6 #######################
29	#-Q- aix6 # Temps Elapsed max. d'une requete hh:mm:ss
30	#-Q- aix6 # @ wall_clock_limit = 20:00:00
31	#-Q- aix6 # Nom du travail LoadLeveler
32	#-Q- aix6 # @ job_name = TS
33	#-Q- aix6 # Fichier de sortie standard du travail
34	#-Q- aix6 # @ output = $(job_name).$(jobid)
35	#-Q- aix6 # Fichier de sortie d'erreur du travail
36	#-Q- aix6 # @ error = $(job_name).$(jobid)
37	#-Q- aix6 # pour recevoir un mail en cas de depassement du temps Elapsed (ou autre pb.)
38	#-Q- aix6 # @ notification = error
39	#-Q- aix6 # @ environment = $DEBUG_debug ; $MODIPSL ; $libIGCM ; $libIGCM_SX ; $SUBMIT_DIR ; $REBUILD_DIR ; $RebuildFromArchive ; $POST_DIR ; $MASTER ; $RebuildFrequency ; $DateBegin ; $PeriodDateBegin ; $PeriodDateEnd ; $NbRebuildDir ; $StandAlone ; $CompletedFlag ; $TsTask ; $CompToRead ; $FlagToRead ; $RESOL_ATM ; $RESOL_OCE ; $RESOL_ICE ; $RESOL_MBG ; $RESOL_SRF ; $RESOL_SBG ; $MASTER
40	#-Q- aix6 # @ queue
41	#-Q- sx8brodie # @ queue
42	#-Q- sx8mercure #!/bin/ksh
43	#-Q- sx8mercure ######################
44	#-Q- sx8mercure ## SX8MERCURE CEA ##
45	#-Q- sx8mercure ######################
46	#-Q- sx8mercure #PBS -N TS # Nom du job
47	#-Q- sx8mercure #PBS -j o # regroupement des stdout et stderr
48	#-Q- sx8mercure #PBS -S /usr/bin/ksh # shell de soumission
49	#-Q- sx8mercure #PBS -l memsz_job=1gb # Limite memoire a 1 Go
50	#-Q- sx8mercure #PBS -l cputim_job=24:00:00 # Limite temps a 2 heures
51	#-Q- sx8mercure #PBS -q scalaire
52	#-Q- sx9mercure #!/bin/ksh
53	#-Q- sx9mercure ######################
54	#-Q- sx9mercure ## SX9MERCURE CEA ##
55	#-Q- sx9mercure ######################
56	#-Q- sx9mercure #MSUB -r TS # Nom du job
57	#-Q- sx9mercure #MSUB -N 1 # Reservation du noeud
58	#-Q- sx9mercure #MSUB -n 1 # Reservation du processus
59	#-Q- sx9mercure #MSUB -T 86400 # Limite de temps elapsed du job
60	#-Q- sx9mercure #MSUB -E "-j o"
61	#-Q- sx9mercure #MSUB -E "-S /bin/ksh"
62	#-Q- sx9mercure ##MSUB -e nco.out # Sortie standard
63	#-Q- sx9mercure ##MSUB -o nco.out # Sortie standard
64	#-Q- titane #!/bin/ksh
65	#-Q- titane ######################
66	#-Q- titane ## TITANE CEA ##
67	#-Q- titane ######################
68	#-Q- titane #MSUB -r TS # Nom du job
69	#-Q- titane #MSUB -N 1 # Reservation du noeud
70	#-Q- titane #MSUB -n 1 # Reservation du processus
71	#-Q- titane #MSUB -T 86400 # Limite de temps elapsed du job
72	#-Q- titane #MSUB -E "-j o"
73	#-Q- titane #MSUB -E "-S /bin/ksh"
74	#-Q- titane ##MSUB -e nco.out # Sortie standard
75	#-Q- titane ##MSUB -o nco.out # Sortie standard
76	#-Q- default #!/bin/ksh
77	#-Q- default ##################
78	#-Q- default ## DEFAULT HOST ##
79	#-Q- default ##################
80
81	# $Date$
82	# $Author$
83	# $Revision$
84	# IPSL (2006)
85	# This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
86
87	#set -eu
88	#set -vx
89
90	date
91
92	#-Q- sx8brodie export OMP_NUM_THREADS=1
93	#-Q- aix6 export OMP_NUM_THREADS=1
94
95	########################################################################
96
97	#D- Flag to determine if this job in a standalone mode
98	#D- Default : value from AA_job if any
99	StandAlone=${StandAlone:=true}
100
101	#D- Low level debug : to bypass lib test checks and stack construction
102	#D- Default : value from AA_job if any
103	libIGCM=${libIGCM:=::modipsl::/libIGCM}
104	# WARNING for StandAlone used : To run this script on some machine,
105	# you must check MirrorlibIGCM variable in sys library.
106	# If this variable is true, you must use libIGCM_POST path instead
107	# of your running libIGCM directory.
108
109	#D- Flag to determine allready produced time series. Empty if you start from the beginning
110	#D- Default : Value from AA_job if any. Usefull in StandAlone case if you want to continue a time series
111	CompletedFlag=${CompletedFlag:=}
112
113	#D- Flag to determine task type this script will perform.
114	#D- Possible Value : 2D, 3D, Chunck2D, Chunck3D
115	#D- Default : Value from AA_job if any. Usefull in StandAlone case.
116	TsTask=${TsTask:=}
117
118	#D- Flag to determine if rebuild process has been performed asynchronously
119	#D- Possible value true or false.
120	#D- If true raw files has allready been patched by rebuild job
121	#D- If false create_ts will patch the raw files
122	#D- Default : Value from AA_job if any. Usefull in StandAlone case.
123	RebuildFrequency=${RebuildFrequency:=true}
124
125	#D- Flag to determine job's output directory
126	#D- Default : value from libIGCM_post.ksh if any
127	POST_DIR=${POST_DIR:=${PBS_O_WORKDIR}}
128
129	#D- Increased verbosity (1, 2, 3)
130	#D- Default : value from AA_job if any
131	Verbosity=${Verbosity:=3}
132
133	#D- Low level debug : to bypass lib test checks and stack construction
134	#D- Default : value from AA_job if any
135	DEBUG_debug=${DEBUG_debug:=false}
136
137	#D- TEMPORARY Flag to determine atmospheric resolution
138	#D- Default : value from atmospheric driver if any
139	RESOL_ATM=ALL
140
141	#D- TEMPORARY Flag to determine ocean resolution
142	#D- Default : value from ocean driver if any
143	RESOL_OCE=${RESOL_OCE:=ORCA2}
144
145	#D- TEMPORARY Flag to determine ice resolution
146	#D- Default : value from ice driver if any
147	RESOL_ICE=${RESOL_ICE:=ORCA2}
148
149	#D- TEMPORARY Flag to determine marine biogeochemistry resolution
150	#D- Default : value from ice driver if any
151	RESOL_MBG=${RESOL_MBG:=ORCA2}
152
153	#D- Flag to determine surface resolution
154	#D- Default : value from surface driver if any
155	RESOL_SRF=ALL
156
157	#D- Flag to determine surface resolution
158	#D- Default : value from surface driver if any
159	RESOL_SBG=ALL
160
161	########################################################################
162
163	. ${libIGCM}/libIGCM_debug/libIGCM_debug.ksh
164	( ${DEBUG_debug} ) && IGCM_debug_Check
165	. ${libIGCM}/libIGCM_card/libIGCM_card.ksh
166	( ${DEBUG_debug} ) && IGCM_card_Check
167	. ${libIGCM}/libIGCM_date/libIGCM_date.ksh
168	( ${DEBUG_debug} ) && IGCM_date_Check
169	#-------
170	. ${libIGCM}/libIGCM_sys/libIGCM_sys.ksh
171
172	########################################################################
173
174	#set -vx
175
176	# Useful cleaning function
177	MENAGE ()
178	{
179	[ -f $3 ] && ( IGCM_sys_RshArchive rm -f $1/$2 ;)
180	}
181
182	########################################################################
183
184	IGCM_sys_MkdirWork ${RUN_DIR_PATH}
185	IGCM_sys_Cd ${RUN_DIR_PATH}
186
187	if [ ${StandAlone} = true ] ; then
188	CARD_DIR=${SUBMIT_DIR}
189	else
190	CARD_DIR=${RUN_DIR_PATH}/$( basename ${SUBMIT_DIR} )
191	IGCM_sys_Get_Master ${SUBMIT_DIR} ${RUN_DIR_PATH}
192	fi
193
194	#
195	# First of all
196	#
197	IGCM_card_DefineArrayFromSection ${CARD_DIR}/config.card UserChoices
198	typeset option
199	for option in ${config_UserChoices[*]} ; do
200	IGCM_card_DefineVariableFromOption ${CARD_DIR}/config.card UserChoices ${option}
201	done
202	#
203	echo
204	IGCM_debug_Print 1 "DefineArrayFromOption : config_UserChoices"
205	IGCM_debug_PrintVariables 3 config_UserChoices_JobName
206	IGCM_debug_PrintVariables 3 config_UserChoices_LongName
207	IGCM_debug_PrintVariables 3 config_UserChoices_TagName
208	IGCM_debug_PrintVariables 3 config_UserChoices_CalendarType
209	IGCM_debug_PrintVariables 3 config_UserChoices_DateBegin
210	IGCM_debug_PrintVariables 3 config_UserChoices_DateEnd
211	IGCM_debug_PrintVariables 3 config_UserChoices_PeriodLength
212	echo
213
214	#==================================
215	R_SAVE=${R_OUT}/${config_UserChoices_TagName}/${config_UserChoices_JobName}
216	#
217	# Determine component to take care of depending on kind of task create_ts will perform
218	#
219	( [ X${TsTask} = XChunck2D ] \|\| [ X${TsTask} = XChunck3D ] ) && set -A config_ListOfComponents ${CompToRead} \|\| IGCM_card_DefineArrayFromSection ${CARD_DIR}/config.card ListOfComponents
220	#
221	# Determine period for time series. Default : value from AA_job or from launch_TS if any
222	#
223	DateBegin=${DateBegin:=$( IGCM_date_ConvertFormatToGregorian ${config_UserChoices_DateBegin} )}
224	#
225	PeriodDateEnd=${PeriodDateEnd:=$( IGCM_date_ConvertFormatToGregorian ${config_UserChoices_DateEnd} )}
226	#
227	# Determine Dimension of the produced time series : 2D or 3D
228	if ( [ X${TsTask} = XChunck2D ] \|\| [ X${TsTask} = X2D ] ) ; then
229	Dimension=2D
230	elif ( [ X${TsTask} = XChunck3D ] \|\| [ X${TsTask} = X3D ] ) ; then
231	Dimension=3D
232	else
233	Dimension=""
234	fi
235
236
237	########################################################################
238	#
239	# Depending on requested time series fill following variables :
240	#
241	# - LISTE_FILE_${comp}[*]
242	# - LISTE_VARS_${file}[*]
243	# - LISTE_AXIS_${file}[*]
244	# - LISTE_PATCH_${file}[*]
245	########################################################################
246
247	#
248	# For each selected component determine which files need post-processing
249	#
250	DoJob=false
251	for comp in ${config_ListOfComponents[*]} ; do
252	#
253	ActiveComp=false
254	#
255	IGCM_card_DefineArrayFromOption ${CARD_DIR}/config.card ListOfComponents ${comp}
256	eval compname=\${config_ListOfComponents_${comp}[0]} > /dev/null 2>&1
257	#
258	card=${CARD_DIR}/COMP/${compname}.card
259
260	IGCM_card_DefineArrayFromOption ${card} OutputFiles List
261	#
262	ListFilesName=${compname}_OutputFiles_List
263	eval FileName0=\${${ListFilesName}[0]} > /dev/null 2>&1
264	#
265	if [ X${FileName0} != X${NULL_STR} ] ; then
266	#
267	#IGCM_debug_Print 1 "Component : ${compname}"
268	#
269	# INITIALISATION
270	#
271	typeset i
272	if ( [ X${TsTask} = XChunck2D ] \|\| [ X${TsTask} = XChunck3D ] ) ; then
273	((NbFiles=${FlagToRead}))
274	((i=${NbFiles}))
275	else
276	eval NbFiles=\${#${ListFilesName}[@]} > /dev/null 2>&1
277	i=2
278	fi
279	#
280	until [ $i -gt $NbFiles ]; do
281	#
282	eval flag_post=\${${ListFilesName}[$i]} > /dev/null 2>&1
283	#
284	if [ X${flag_post} != XNONE ] ; then
285	#
286	# For each comp and each file who needs post-processing
287	# determine which variables need time series.
288	#
289	FILE=$( echo ${flag_post} \| awk "-FPost_" '{print $2}' )
290	IGCM_card_DefineArrayFromOption ${card} Post_${FILE} TimeSeriesVars${Dimension}
291	IGCM_card_DefineVariableFromOption ${card} Post_${FILE} ChunckJob${Dimension}
292	#
293	# If TimeSeriesVars list is empty we skip
294	#
295	if [ $( eval echo \${${compname}_Post_${FILE}_TimeSeriesVars${Dimension}} ) = ${NULL_STR} ] ; then
296	#
297	IGCM_debug_Print 3 "Nothing to do for ${FILE} according to ${card}"
298	IGCM_debug_Print 3 "TimeSeriesVars${Dimension} for ${FILE} is empty"
299	(( i=i+3 ))
300	continue
301	#
302	fi
303	#
304	# If files/variables has been switch off by ChunckJob${Dimension}=0 we skip
305	#
306	if [ X$( eval echo \${${compname}_Post_${FILE}_ChunckJob${Dimension}} ) = XOFF ] ; then
307	IGCM_debug_Print 3 "Nothing to do for ${FILE} according to ${card}"
308	IGCM_debug_Print 3 "ChunckJob${Dimension}=OFF for ${FILE} so it has been switched off"
309	(( i=i+3 ))
310	continue
311	fi
312	#
313	# If we are not a Chunck type task, we skip files/variables handled by chunck task
314	#
315	if ( [ ! X${TsTask} = XChunck${Dimension} ] && [ ! X${TsTask} = X ] && [ ! XChunck${Dimension} = XChunck ] ) ; then
316	if [ ! $( eval echo \${${compname}_Post_${FILE}_ChunckJob${Dimension}} ) = NONE ] ; then
317	IGCM_debug_Print 3 "Nothing to do for ${FILE} TimeSeriesVars${Dimension} according to ${card}"
318	IGCM_debug_Print 3 "Let another chunck task do the job $( eval echo \${${compname}_Post_${FILE}_ChunckJob${Dimension}} )"
319	(( i=i+3 ))
320	continue
321	fi
322	fi
323	#
324	# Now on we know we have things to do
325	#
326
327	#
328	# We need coordinates axis to properly fill produced time series
329	#
330	IGCM_card_DefineArrayFromOption ${card} Post_${FILE} GatherWithInternal
331	#
332	if [ $( eval echo \${${compname}_Post_${FILE}_GatherWithInternal} ) = ${NULL_STR} ] ; then
333	#
334	IGCM_debug_Print 1 "ERROR !! We miss coordinate axis for ${FILE} according to ${card}"
335	IGCM_debug_Print 1 "So no time series will be produced for ${FILE} !!"
336	(( i=i+3 ))
337	continue
338	#
339	fi
340	#
341	# We need LIST of variables not allready produced (useful for standalone mode)
342	#
343	FlagDir=$( echo ${FILE} \| awk -F "_" '{print $1}' )
344	case ${FlagDir} in
345	*Y) TS_Dir=TS_YE ;;
346	*M) TS_Dir=TS_MO ;;
347	*D) TS_Dir=TS_DA ;;
348	HF) TS_Dir=TS_HF ;;
349	INS) TS_Dir=TS_INS ;;
350	esac
351	#
352	count=0
353	for var in $( eval echo \${${compname}_Post_${FILE}_TimeSeriesVars${Dimension}[*]} ) ; do
354	#
355	DIRECTORY=${R_SAVE}/${comp}/Analyse/${TS_Dir}
356	TestedFile=${config_UserChoices_JobName}_${DateBegin}_${PeriodDateEnd}_${FlagDir}_${var}.nc
357	#
358	IGCM_sys_TestFileArchive ${DIRECTORY}/${TestedFile}
359	if [ ! $? = 0 ] ; then
360	eval set +A LISTE_VARS_${FILE} \${LISTE_VARS_${FILE}[*]} ${var}
361	ActiveComp=true
362	(( count = count + 1 ))
363	fi
364	done
365	#
366	if [ ${count} -gt 0 ] ; then
367	#
368	eval set +A LISTE_FILE_${comp} \${LISTE_FILE_${comp}[*]} ${FILE}
369	eval set +A LISTE_AXIS_${FILE} \${${compname}_Post_${FILE}_GatherWithInternal[*]}
370	#
371	if [ ! X${RebuildFrequency} = Xtrue ] ; then
372	IGCM_card_DefineArrayFromOption ${card} Post_${FILE} Patches
373	eval set +A LISTE_PATCHES_${FILE} \${${compname}_Post_${FILE}_Patches[*]}
374	if [ X$( eval echo \${LISTE_PATCHES_${FILE}[0]} ) != X${NULL_STR} ]; then
375	for Patch in $( eval echo \${LISTE_PATCHES_${FILE}[*]} ); do
376	if [ Xload${Patch} != X ] ; then
377	. ${libIGCM}/libIGCM_post/IGCM_${Patch}.ksh
378	eval load${Patch}=loaded
379	fi
380	done
381	fi
382	fi
383	fi
384	fi
385	(( i=i+3 ))
386	done
387	fi
388	#
389	if [ ${ActiveComp} = true ] ; then
390	set +A ActiveListOfComponents ${ActiveListOfComponents[*]} ${comp}
391	fi
392	done
393
394	set -A config_ListOfComponents ${ActiveListOfComponents[*]}
395	[ ! -z "${ActiveListOfComponents}" ] && DoJob=true
396
397	########################################################################
398	#
399	# DEFINE DATE FUNCTIONS
400	#
401	########################################################################
402
403	# Give the end of the next $1 period from $2 date
404	function create_ts_next_date {
405	IGCM_debug_PushStack "create_ts_next_date" $@
406	typeset Length
407	case ${config_UserChoices_PeriodLength} in
408	Y\|y)
409	PeriodLengthInYears=$( echo ${config_UserChoices_PeriodLength} \| awk -F '[yY]' '{print $1}' )
410	IGCM_date_GetYearMonth ${1} year month
411	(( Length=0 ))
412	(( i=0 ))
413	until [ $i -ge $PeriodLengthInYears ] ; do
414	(( Length = Length + $( IGCM_date_DaysInYear $(( year + i + 1 )) ) ))
415	(( i=i+1 ))
416	done
417	;;
418	M\|m)
419	PeriodLengthInMonths=$( echo ${config_UserChoices_PeriodLength} \| awk -F '[mM]' '{print $1}' )
420	IGCM_date_GetYearMonth ${1} year month
421	(( year0 = year ))
422	if [ $(( month + 1 )) -lt 13 ] ; then
423	month0=$(( month + 1 ))
424	else
425	month0=$(( month + 1 - 12 ))
426	(( year = year0 + 1 ))
427	fi
428	(( Length=0 ))
429	(( i=0 ))
430	until [ $i -ge $PeriodLengthInMonths ] ; do
431	if [ $(( month0 + i )) -lt 13 ] ; then
432	(( Length = Length + $( IGCM_date_DaysInMonth $year $(( month0 + i )) ) ))
433	else
434	(( year = year0 + 1 ))
435	(( Length = Length + $( IGCM_date_DaysInMonth $year $(( month0 + i - 12 )) ) ))
436	fi
437	(( i=i+1 ))
438	done
439	;;
440	D\|d)
441	Length=$( echo ${config_UserChoices_PeriodLength} \| sed -e "s/[dD]//" ) ;;
442	*)
443	IGCM_debug_Exit "create_ts " ${config_UserChoices_PeriodLength} " invalid PeriodLength : choose in 1Y, M, D."
444	IGCM_debug_Verif_Exit_Post ;;
445	esac
446	echo ${Length}
447
448	IGCM_debug_PopStack "create_ts_next_date"
449	}
450
451	# Give the end of the actual $1 period from $2 date
452	function create_ts_begin_date {
453	IGCM_debug_PushStack "create_ts_begin_date" $@
454	typeset Length
455	case ${config_UserChoices_PeriodLength} in
456	Y\|y)
457	PeriodLengthInYears=$( echo ${config_UserChoices_PeriodLength} \| awk -F '[yY]' '{print $1}' )
458	IGCM_date_GetYearMonth ${1} year month
459	if [ X${2} = Xend ] ; then
460	(( year = year - PeriodLengthInYears + 1))
461	fi
462	(( Length=0 ))
463	(( i=0 ))
464	until [ $i -ge $PeriodLengthInYears ] ; do
465	(( Length = Length + $( IGCM_date_DaysInYear $(( year + i )) ) ))
466	(( i=i+1 ))
467	done
468	;;
469	M\|m)
470	PeriodLengthInMonths=$( echo ${config_UserChoices_PeriodLength} \| awk -F '[mM]' '{print $1}' )
471	IGCM_date_GetYearMonth ${1} year month
472	if [ X${2} = Xend ] ; then
473	(( month = month - PeriodLengthInMonths + 1 ))
474	fi
475	(( year0 = year ))
476	if [ $month -le 0 ] ; then
477	(( month = month + 12 ))
478	year=$( printf "%04i\n" $(( year - 1 )) )
479	fi
480	month=$( printf "%02i\n" ${month} )
481	(( Length=0 ))
482	(( i=0 ))
483	until [ $i -ge $PeriodLengthInMonths ] ; do
484	if [ $(( month + i )) -lt 13 ] ; then
485	(( Length = Length + $( IGCM_date_DaysInMonth $year $(( month + i )) ) ))
486	else
487	(( year = year0 + 1 ))
488	(( Length = Length + $( IGCM_date_DaysInMonth $year $(( month + i - 12 )) ) ))
489	fi
490	(( i=i+1 ))
491	done
492	;;
493	D\|d)
494	Length=$( echo ${config_UserChoices_PeriodLength} \| sed -e "s/[dD]//" ) ;;
495	*)
496	IGCM_debug_Exit "create_ts " ${config_UserChoices_PeriodLength} " invalid PeriodLength : choose in 1Y, M, D."
497	IGCM_debug_Verif_Exit_Post ;;
498	esac
499	echo ${Length}
500
501	IGCM_debug_PopStack "create_ts_begin_date"
502	}
503
504	########################################################################
505	#
506	# IS THERE SOME ALLREADY PRODUCED TIME SERIES ?
507	# IF SO BRING THEM IN THE WORKING DIRECTORY
508	#
509	# PS : Keep in mind that IGCM_sys_Get here is a weak link :
510	# - especially for IDRiS
511	# - no special protection against "command too long" !
512	# - libIGCM_sys should handle that or modify following code part
513	#
514	########################################################################
515
516	if [ ${StandAlone} != true ] ; then
517	Running_Flag=$( IGCM_sys_RshMaster "IGCM_card_DefineVariableFromOption \
518	${SUBMIT_DIR}/run.card PostProcessing TimeSeriesRunning ; \
519	echo \${run_PostProcessing_TimeSeriesRunning}" )
520
521	#if [ X${Running_Flag} = Xy ] ; then
522	# # Time Series Job allready running
523	# IGCM_debug_Print 1 "Time Series Job allready running exit"
524	# exit
525	#fi
526	# Now run.card know we are running
527
528
529
530
531	IGCM_sys_RshMaster IGCM_card_WriteOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesRunning "y"
532
533
534
535
536
537	# Get information from last execution
538	#CompletedFlag=$( IGCM_sys_RshMaster \
539	# "IGCM_card_DefineVariableFromOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesCompleted ;\
540	# echo \${run_PostProcessing_TimeSeriesCompleted}" )
541	fi
542
543	# Is it the first submission or not ?
544
545	if ( [ X${CompletedFlag} = X ] \|\| [ X${CompletedFlag} = X$( IGCM_date_AddDaysToGregorianDate ${DateBegin} -1 ) ] ) ; then
546	# First Time Series Submission
547	FIRST_PASS=TRUE
548	Length=$( create_ts_begin_date ${DateBegin} begin)
549	DATE_FIN_JOB_B=$( IGCM_date_AddDaysToGregorianDate ${DateBegin} $(( Length - 1 )) )
550	else
551	# Last Completed Time Series
552	FIRST_PASS=FALSE
553	DATE_FIN_JOB_B=${CompletedFlag}
554	date=${DateBegin}_${DATE_FIN_JOB_B}
555	for comp in ${config_ListOfComponents[*]} ; do
556	#
557	IGCM_sys_Mkdir ${RUN_DIR_PATH}/${comp}
558	#
559	IGCM_sys_Cd ${RUN_DIR_PATH}/${comp}
560	#
561	#set -A liste_file
562	unset liste_file
563	#
564	file_path=${R_SAVE}/${comp}/Analyse
565	#
566	i=0
567	file=${NULL_STR}
568	#
569	for file in $( eval echo \${LISTE_FILE_${comp}[*]} ); do
570	#
571	# Determine in which output can we find file
572	# IE : Analyse/TS or Analyse/DA2TS or ...
573	#
574	FlagDir=$( echo ${file} \| awk -F "_" '{print $1}' )
575	case ${FlagDir} in
576	*Y) TS_Dir=TS_YE ;;
577	*M) TS_Dir=TS_MO ;;
578	*D) TS_Dir=TS_DA ;;
579	HF) TS_Dir=TS_HF ;;
580	INS) TS_Dir=TS_INS ;;
581	esac
582	for var in $( eval echo \${LISTE_VARS_${file}[*]} ) ; do
583	#
584	#set +A liste_file ${liste_file[*]} ${file_path}/${TS_Dir}/${config_UserChoices_JobName}_${date}_${FlagDir}_${var}.nc
585	liste_file[${#liste_file[*]}]=${file_path}/${TS_Dir}/${config_UserChoices_JobName}_${date}_${FlagDir}_${var}.nc
586	#
587	done
588	done
589	if [ X${file} != X${NULL_STR} ] ; then
590	IGCM_sys_Get /l liste_file[*] ${RUN_DIR_PATH}/${comp}
591	ExitFlag=false
592	IGCM_debug_Verif_Exit_Post
593	fi
594	done
595	#
596	LEVEL=0
597	#
598	fi
599
600	########################################################################
601	#
602	# DEFINE LOOP PARAMETERS
603	#
604	########################################################################
605
606	DATE_COUNT=${DATE_FIN_JOB_B}
607	DATE_FIN_JOB_B_LOOP=${DATE_FIN_JOB_B}
608
609	( [ -n "${LEVEL}" ] && [ ${LEVEL} -eq 0 ] ) && NBRE_FILE_TOT=0 \|\| NBRE_FILE_TOT=1
610
611	if [ ${DoJob} = true ] ; then
612	while [ ${DATE_COUNT} -lt ${PeriodDateEnd} ] ; do
613	(( NBRE_FILE_TOT = NBRE_FILE_TOT + 1 ))
614	Length=$( create_ts_next_date ${DATE_COUNT} )
615	DATE_COUNT=$( IGCM_date_AddDaysToGregorianDate ${DATE_COUNT} ${Length} )
616	done
617	else
618	NBRE_FILE_TOT=0
619	fi
620
621	if [ ${NBRE_FILE_TOT} -le ${UNIX_MAX_LIMIT} ] ; then
622	NBRE_TOUR=1
623	elif [ ${NBRE_FILE_TOT}%${UNIX_MAX_LIMIT} -eq 0 ] ; then
624	let NBRE_TOUR=${NBRE_FILE_TOT}/${UNIX_MAX_LIMIT}
625	else
626	let NBRE_TOUR=${NBRE_FILE_TOT}/${UNIX_MAX_LIMIT}+1
627	fi
628
629	CURRENT_LOOP=1
630	# THIS LOOP GIVES PARAMETERS FOR THE DATE'S LOOP ie : DATE_COURANTE, DATE_FIN.
631	# WE HAVE 3 LEVELS
632	# - LEVEL 1 : JUST ONE LOOP TO COMPLETE
633	# - LEVEL 2 : WE ARE IN THE "DEEP" LOOP
634	# - LEVEL 3 : WE ARE IN THE LAST LOOP
635
636	while [ ${CURRENT_LOOP} -le ${NBRE_TOUR} ] ; do
637
638	if [ ${NBRE_FILE_TOT} -le ${UNIX_MAX_LIMIT} ] ; then
639	# - LEVEL 1 : JUST ONE LOOP TO COMPLETE
640	NBRE_FILE_LOOP=${NBRE_FILE_TOT}
641
642	if [ ${FIRST_PASS} = TRUE ] ; then
643	DATE_COURANTE=${DATE_FIN_JOB_B_LOOP}
644	else
645	Length=$( create_ts_next_date ${DATE_FIN_JOB_B_LOOP} )
646	DATE_COURANTE=$( IGCM_date_AddDaysToGregorianDate ${DATE_FIN_JOB_B_LOOP} ${Length} )
647	fi
648
649	DATE_FIN=${PeriodDateEnd}
650	DATE_FIN_JOB_B_LOOP_PREC=${DATE_FIN_JOB_B_LOOP}
651
652	[ -n "${DATE_LOOP}" ] && DATE_FIN_JOB_B_LOOP=${DATE_LOOP}
653	[ -n "${LEVEL}" ] && FLAG_B="TRUE" \|\| FLAG_B="FALSE"
654
655	LEVEL=1
656	DEPOT="TRUE"
657	elif [ ${CURRENT_LOOP} -ne ${NBRE_TOUR} ] ; then
658	# - LEVEL 2 : WE ARE IN THE "DEEP" LOOP
659	NBRE_FILE_LOOP=${UNIX_MAX_LIMIT}
660
661	if [ ! ${FIRST_PASS} = TRUE ] && [ ${CURRENT_LOOP} -eq 1 ] ; then
662	Length=$( create_ts_next_date ${DATE_FIN_JOB_B_LOOP} )
663	DATE_COURANTE=$( IGCM_date_AddDaysToGregorianDate ${DATE_FIN_JOB_B_LOOP} ${Length} )
664	elif [ -z "${DATE_COURANTE}" ] ; then
665	DATE_COURANTE=${DATE_FIN_JOB_B}
666	else
667	Length=$( create_ts_next_date ${DATE_FIN} )
668	DATE_COURANTE=$( IGCM_date_AddDaysToGregorianDate ${DATE_FIN} ${Length} )
669	fi
670
671	(( TotLength = 0 ))
672	COMPTEUR=1
673	DATE_LOOP=${DATE_COURANTE}
674
675	while [ ${COMPTEUR} -lt ${NBRE_FILE_LOOP} ] ; do
676	#
677	Length=$( create_ts_next_date ${DATE_LOOP} )
678	DATE_LOOP=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} ${Length} )
679	(( TotLength = TotLength + Length ))
680	(( COMPTEUR = COMPTEUR + 1 ))
681	#
682	done # while [ ${COMPTEUR} -le ${NBRE_FILE_LOOP} ] done
683	DATE_FIN=$( IGCM_date_AddDaysToGregorianDate ${DATE_COURANTE} ${TotLength} )
684
685	[ -n "${DATE_LOOP}" ] && DATE_FIN_JOB_B_LOOP_PREC=${DATE_FIN_JOB_B_LOOP} ; DATE_FIN_JOB_B_LOOP=${DATE_LOOP}
686	[ -n "${LEVEL}" ] && [ ${LEVEL} -eq 0 ] && FLAG_B="TRUE" \|\| [ ${CURRENT_LOOP} -gt 1 ] && FLAG_B="TRUE" \|\| FLAG_B="FALSE"
687	DEPOT="FALSE"
688	LEVEL=2
689	else
690	# - LEVEL 3 : WE ARE IN THE LAST LOOP
691	NBRE_FILE_LOOP=$(( ${NBRE_FILE_TOT} % ( ${UNIX_MAX_LIMIT} * ( ${CURRENT_LOOP} - 1 ) ) ))
692	[ ${NBRE_FILE_LOOP} -eq 0 ] && NBRE_FILE_LOOP=${UNIX_MAX_LIMIT}
693
694	Length=$( create_ts_next_date ${DATE_FIN} )
695	DATE_COURANTE=$(IGCM_date_AddDaysToGregorianDate ${DATE_FIN} ${Length} )
696	DATE_FIN=${PeriodDateEnd}
697	[ -n "${DATE_LOOP}" ] && DATE_FIN_JOB_B_LOOP_PREC=${DATE_FIN_JOB_B_LOOP} ; DATE_FIN_JOB_B_LOOP=${DATE_LOOP}
698
699	LEVEL=3
700	DEPOT="TRUE"
701	FLAG_B="TRUE"
702	fi
703
704	# THEN FOR EACH FILE TYPE WE LOOP BETWEEN DATE_COURANTE AND DATE_FIN.
705	for comp in ${config_ListOfComponents[*]} ; do
706	#
707	IGCM_sys_Mkdir ${RUN_DIR_PATH}/${comp}
708	#
709	IGCM_sys_Cd ${RUN_DIR_PATH}/${comp}
710	#
711	eval R_OUT_${comp}=${R_SAVE}/${comp}
712	#
713	i=0
714	#
715	for file in $( eval echo \${LISTE_FILE_${comp}[*]} ); do
716	#
717	# Determine in which output can we find file
718	# IE : Output/MO or Output/DA or ...
719	#
720	FlagDir=$( echo ${file} \| awk -F "_" '{print $1}' )
721	case ${FlagDir} in
722	*Y) TS_Dir=TS_YE ; FreqDir=YE ;;
723	*M) TS_Dir=TS_MO ; FreqDir=MO ;;
724	*D) TS_Dir=TS_DA ; FreqDir=DA ;;
725	HF) TS_Dir=TS_HF ; FreqDir=HF ;;
726	INS) TS_Dir=TS_INS ; FreqDir=INS ;;
727	*)
728	IGCM_debug_Print 1 "Error in FlagDir=${FlagDir} for file ${file}."
729	IGCM_sys_Exit "Job create_ts" ;;
730	esac
731
732	# Initialize array
733	#set -A liste_file
734	#set -A liste_file_tmp
735	unset liste_file
736	unset liste_file_tmp
737	eval file_path=\${R_OUT_${comp}}/Output/${FreqDir}/
738
739	COMPTEUR=1
740	DATE_LOOP=${DATE_COURANTE}
741
742	while [ ${COMPTEUR} -le ${NBRE_FILE_LOOP} ] ; do
743	#
744	Length1=$( create_ts_begin_date ${DATE_LOOP} end)
745	DATE_TAB=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} $(( 1 - Length1 )) )_${DATE_LOOP}
746
747	Length2=$( create_ts_next_date ${DATE_LOOP} end)
748	DATE_LOOP=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} ${Length2} )
749
750	#set +A liste_file ${liste_file[*]} ${file_path}${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc
751	#set +A liste_file_tmp ${liste_file_tmp[*]} ${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc
752	liste_file[${#liste_file[*]}]=${file_path}${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc
753	liste_file_tmp[${#liste_file_tmp[*]}]=${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc
754	(( COMPTEUR = COMPTEUR + 1 ))
755	#
756	done # while [ ${COMPTEUR} -le ${NBRE_FILE_LOOP} ] done
757	DATE_LOOP=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} -${Length2} )
758
759	# Get selected files
760	IGCM_sys_Get /l liste_file[*] ${RUN_DIR_PATH}/${comp}
761	ExitFlag=false
762	IGCM_debug_Verif_Exit_Post
763
764	# Apply IGCM_Patch if needed
765	if [ ! X${RebuildFrequency} = Xtrue ] ; then
766	if [ X$( eval echo \${LISTE_PATCHES_${file}[0]} ) != X${NULL_STR} ]; then
767	for file_t in $( eval echo ${liste_file_tmp[*]} ); do
768	for Patch in $( eval echo \${LISTE_PATCHES_${file}[*]} ); do
769	IGCM_${Patch} ${file_t}
770	done
771	done
772	fi
773	fi
774
775	# WE CAN CONCATENATE FILES AT THIS POINT
776	[ "${FLAG_B}" = "TRUE" ] && DATE_BUILD_B=${DateBegin}_${DATE_FIN_JOB_B_LOOP_PREC}
777	Length=$( create_ts_begin_date ${DATE_COURANTE} end)
778	DATE_BUILD1=$( IGCM_date_AddDaysToGregorianDate ${DATE_COURANTE} -$(( Length - 1 )) )
779	DATE_BUILD=${DATE_BUILD1}_${DATE_FIN}
780	DATE_BUILD_END=${DateBegin}_${DATE_FIN}
781
782	liste_coord=" "
783	for axis in $( eval echo \${LISTE_AXIS_${file}[*]} ); do
784	liste_coord=${liste_coord}${axis}","
785	done
786
787	# Time axis must be the last dimension
788	time_axis=${axis}
789
790	# SWITCH BETWEEN MODEL TO FIT PARTICULAR CASE
791	for var in $( eval echo \${LISTE_VARS_${file}[*]} ) ; do
792	#
793	IGCM_sys_ncrcat --hst -v ${liste_coord}${var} ${liste_file_tmp[*]} ${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
794	#
795	# # Change time axis name to be CF compliant
796	# if [ ${time_axis} != time_counter ]; then
797	# IGCM_sys_ncrename -O -v ${time_axis},time_counter ${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
798	# fi
799	#
800	if [ "${FLAG_B}" = "TRUE" ] ; then
801	# WE CONCATENATE WITH EXISTING FILES
802	file1=${config_UserChoices_JobName}_${DATE_BUILD_B}_${FlagDir}_${var}.nc
803	file2=${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
804	file_out=${config_UserChoices_JobName}_${DATE_BUILD_END}_${FlagDir}_${var}.nc
805	IGCM_sys_ncrcat --hst ${file1} ${file2} ${file_out}
806	#
807	IGCM_sys_Rm ${file1} ${file2}
808	fi
809	#
810	if [ "${DEPOT}" = "TRUE" ] ; then
811	# WE PUT FINAL FILE ON FILE SYSTEM
812	[ "${FLAG_B}" = "FALSE" ] && file_out=${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
813	#${ncatted} -a modulo,time_counter,c,c," " ${file_out}
814	file_before=${config_UserChoices_JobName}_${DateBegin}_${DATE_FIN_JOB_B}_${FlagDir}_${var}.nc
815	eval IGCM_sys_Put_Out ${file_out} \${R_OUT_${comp}}/Analyse/${TS_Dir}/${file_out}
816	[ "${FLAG_B}" = "TRUE" ] && eval MENAGE \${R_OUT_${comp}}/Analyse/${TS_Dir} ${file_before} ${file_out}
817	IGCM_sys_Rm ${file_out}
818	#
819	fi
820	#
821	ExitFlag=false
822	IGCM_debug_Verif_Exit_Post
823	done # for var in ${liste_var} ; do
824	IGCM_sys_Rm ${liste_file_tmp[*]}
825	done # for file in ${liste_file_brut} do
826	done # for comp in ${config_ListOfComponents[*]} ; do
827	ExitFlag=false
828	IGCM_debug_Verif_Exit_Post
829	# GO AHEAD IN DEEPEST LOOP
830	(( CURRENT_LOOP = CURRENT_LOOP + 1 ))
831	done # while [ ${CURRENT_LOOP} -le ${NBRE_TOUR} ] do
832	# FINISH
833
834	# DODS copy
835	for comp in ${config_ListOfComponents[*]} ; do
836	for TS_Dir in TS_YE TS_MO TS_DA TS_HF TS_INS ; do
837	IGCM_sys_TestDirArchive ${R_SAVE}/${comp}/Analyse/${TS_Dir}
838	[ $? = 0 ] && IGCM_sys_Put_Dods ${comp}/Analyse/${TS_Dir}
839	done
840	done
841
842	export POST_DIR; export SUBMIT_DIR; export libIGCM; export R_INIT; export R_BC; export StandAlone
843	export RESOL_ATM; export RESOL_OCE ; export RESOL_ICE ; export RESOL_MBG ; export RESOL_SRF ; export RESOL_SBG
844
845	listVarEnv="POST_DIR,SUBMIT_DIR,libIGCM,R_INIT,R_BC,StandAlone,RESOL_ATM,RESOL_OCE,RESOL_ICE,RESOL_MBG,RESOL_SRF,RESOL_SBG"
846	export listVarEnv
847
848	#D-
849	# --------------------------------------------------------------------
850	#D- Test if all was right
851	# --------------------------------------------------------------------
852	IGCM_debug_Verif_Exit_Post
853	# --------------------------------------------------------------------
854	#D- Submit MONITORING
855	# --------------------------------------------------------------------
856	( [ X${TsTask} = X2D ] \|\| [ X${TsTask} = X ] ) && IGCM_sys_QsubPost monitoring
857
858	#D-
859	# --------------------------------------------------------------------
860	#D- Update the run.card
861	# --------------------------------------------------------------------
862	if [ ${StandAlone} != true ] ; then
863	# Put in run.card end period of time series
864	IGCM_sys_RshMaster IGCM_card_WriteOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesCompleted "${DATE_FIN}"
865	# We have finish the job !
866	IGCM_sys_RshMaster IGCM_card_WriteOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesRunning "n"
867	fi
868
869	# Clean RUN_DIR_PATH (necessary for cesium)
870	IGCM_sys_RmRunDir -Rf ${RUN_DIR_PATH}
871
872	date

Note: See TracBrowser for help on using the repository browser.

Download in other formats: