Context Navigation

source: trunk/libIGCM/AA_create_ts @ 257

Last change on this file since 257 was 257, checked in by sdipsl, 14 years ago
Add mercure-new : libIGCM_sys_mercure.ksh SX9 use mercure-new for post-processing Few feature missing on mercure-new dmget not yet install for example Properly reset all svn:keywords feature Bugfix in libIGCM_post.ksh when RebuildFromArchive? was false Bugfix in IGCM_Patch_20090317_histcomFillvalue.ksh when renaming _Fillvalue we now just add missing_value to be retro-compatible Some more comments
Property licence set to The following licence information concerns ONLY the libIGCM tools ================================================================== Copyright © Centre National de la Recherche Scientifique CNRS Commissariat à l'Énergie Atomique CEA libIGCM : Library for Portable Models Computation of IGCM Group. IGCM Group is the french IPSL Global Climate Model Group. This library is a set of shell scripts and functions whose purpose is the management of the initialization, the launch, the transfer of output files, the post-processing and the monitoring of datas produce by any numerical program on any plateforme. This software is governed by the CeCILL license under French law and abiding by the rules of distribution of free software. You can use, modify and/ or redistribute the software under the terms of the CeCILL license as circulated by CEA, CNRS and INRIA at the following URL "http://www.cecill.info". As a counterpart to the access to the source code and rights to copy, modify and redistribute granted by the license, users are provided only with a limited warranty and the software's author, the holder of the economic rights, and the successive licensors have only limited liability. In this respect, the user's attention is drawn to the risks associated with loading, using, modifying and/or developing or reproducing the software by the user in light of its specific status of free software, that may mean that it is complicated to manipulate, and that also therefore means that it is reserved for developers and experienced professionals having in-depth computer knowledge. Users are therefore encouraged to load and test the software's suitability as regards their requirements in conditions enabling the security of their systems and/or data to be ensured and, more generally, to use and operate it in the same conditions as regards security. The fact that you are presently reading this means that you have had knowledge of the CeCILL license and that you accept its terms. Property svn:keywords set to `Date Author Revision`
File size: 31.1 KB

Line
1	#-Q- platine #!/usr/bin/ksh
2	#-Q- platine ###################
3	#-Q- platine ## PLATINE CEA ##
4	#-Q- platine ###################
5	#-Q- platine #BSUB -J TS # Nom du job
6	#-Q- platine #BSUB -N # message a la fin du job
7	#-Q- platine #BSUB -n 1 # reservation des processeurs pour le job
8	#-Q- platine #BSUB -W 1:00 # Limite temps
9	#-Q- platine #BSUB -q post # Passage en queue post
10	#-Q- sx8brodie #!/bin/ksh
11	#-Q- sx8brodie #######################
12	#-Q- sx8brodie ## SX8BRODIE IDRIS ##
13	#-Q- sx8brodie #######################
14	#-Q- sx8brodie # Temps Elapsed max. d'une requete hh:mm:ss
15	#-Q- sx8brodie # @ wall_clock_limit = 20:00:00
16	#-Q- sx8brodie # Nom du travail LoadLeveler
17	#-Q- sx8brodie # @ job_name = TS
18	#-Q- sx8brodie # Fichier de sortie standard du travail
19	#-Q- sx8brodie # @ output = $(job_name).$(jobid)
20	#-Q- sx8brodie # Fichier de sortie d'erreur du travail
21	#-Q- sx8brodie # @ error = $(job_name).$(jobid)
22	#-Q- sx8brodie # pour recevoir un mail en cas de depassement du temps Elapsed (ou autre pb.)
23	#-Q- sx8brodie # @ notification = error
24	#-Q- sx8brodie # @ environment = $DEBUG_debug ; $MODIPSL ; $libIGCM ; $libIGCM_SX ; $SUBMIT_DIR ; $REBUILD_DIR ; $RebuildFromArchive ; $POST_DIR ; $MASTER ; $RebuildFrequency ; $PeriodDateBegin ; $PeriodDateEnd ; $NbRebuildDir ; $StandAlone ; $CompletedFlag ; $TsTask ; $CompToRead ; $FlagToRead ; $RESOL_ATM ; $RESOL_OCE ; $RESOL_ICE ; $RESOL_MBG ; $RESOL_SRF ; $RESOL_SBG ; $MASTER
25	#-Q- aix6 #!/bin/ksh
26	#-Q- aix6 #######################
27	#-Q- aix6 ## VARGAS IDRIS ##
28	#-Q- aix6 #######################
29	#-Q- aix6 # Temps Elapsed max. d'une requete hh:mm:ss
30	#-Q- aix6 # @ wall_clock_limit = 20:00:00
31	#-Q- aix6 # Nom du travail LoadLeveler
32	#-Q- aix6 # @ job_name = TS
33	#-Q- aix6 # Fichier de sortie standard du travail
34	#-Q- aix6 # @ output = $(job_name).$(jobid)
35	#-Q- aix6 # Fichier de sortie d'erreur du travail
36	#-Q- aix6 # @ error = $(job_name).$(jobid)
37	#-Q- aix6 # pour recevoir un mail en cas de depassement du temps Elapsed (ou autre pb.)
38	#-Q- aix6 # @ notification = error
39	#-Q- aix6 # @ environment = $DEBUG_debug ; $MODIPSL ; $libIGCM ; $libIGCM_SX ; $SUBMIT_DIR ; $REBUILD_DIR ; $RebuildFromArchive ; $POST_DIR ; $MASTER ; $RebuildFrequency ; $DateBegin ; $PeriodDateBegin ; $PeriodDateEnd ; $NbRebuildDir ; $StandAlone ; $CompletedFlag ; $TsTask ; $CompToRead ; $FlagToRead ; $RESOL_ATM ; $RESOL_OCE ; $RESOL_ICE ; $RESOL_MBG ; $RESOL_SRF ; $RESOL_SBG ; $MASTER
40	#-Q- aix6 # @ queue
41	#-Q- sx8brodie # @ queue
42	#-Q- sx8mercure #!/bin/ksh
43	#-Q- sx8mercure ######################
44	#-Q- sx8mercure ## SX8MERCURE CEA ##
45	#-Q- sx8mercure ######################
46	#-Q- sx8mercure #PBS -N TS # Nom du job
47	#-Q- sx8mercure #PBS -j o # regroupement des stdout et stderr
48	#-Q- sx8mercure #PBS -S /usr/bin/ksh # shell de soumission
49	#-Q- sx8mercure #PBS -l memsz_job=1gb # Limite memoire a 1 Go
50	#-Q- sx8mercure #PBS -l cputim_job=24:00:00 # Limite temps a 2 heures
51	#-Q- sx8mercure #PBS -q scalaire
52	#-Q- sx9mercure #!/bin/ksh
53	#-Q- sx9mercure ######################
54	#-Q- sx9mercure ## SX9MERCURE CEA ##
55	#-Q- sx9mercure ######################
56	#-Q- sx9mercure #PBS -N TS # Nom du job
57	#-Q- sx9mercure #PBS -j o # regroupement des stdout et stderr
58	#-Q- sx9mercure #PBS -S /usr/bin/ksh # shell de soumission
59	#-Q- sx9mercure #PBS -l memsz_job=1gb # Limite memoire a 1 Go
60	#-Q- sx9mercure #PBS -l cputim_job=24:00:00 # Limite temps a 2 heures
61	#-Q- sx9mercure #PBS -q scal-new
62	#-Q- titane #!/bin/ksh
63	#-Q- titane ######################
64	#-Q- titane ## TITANE CEA ##
65	#-Q- titane ######################
66	#-Q- titane #MSUB -r TS # Nom du job
67	#-Q- titane #MSUB -N 1 # Reservation du noeud
68	#-Q- titane #MSUB -n 1 # Reservation du processus
69	#-Q- titane #MSUB -T 86400 # Limite de temps elapsed du job
70	#-Q- titane #MSUB -E "-j o"
71	#-Q- titane #MSUB -E "-S /bin/ksh"
72	#-Q- titane ##MSUB -e nco.out # Sortie standard
73	#-Q- titane ##MSUB -o nco.out # Sortie standard
74	#-Q- default #!/bin/ksh
75	#-Q- default ##################
76	#-Q- default ## DEFAULT HOST ##
77	#-Q- default ##################
78
79	# $Date$
80	# $Author$
81	# $Revision$
82	# IPSL (2006)
83	# This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
84
85	#set -eu
86	#set -vx
87
88	date
89
90	#-Q- sx8brodie export OMP_NUM_THREADS=1
91	#-Q- aix6 export OMP_NUM_THREADS=1
92
93	########################################################################
94
95	#D- Flag to determine if this job in a standalone mode
96	#D- Default : value from AA_job if any
97	StandAlone=${StandAlone:=true}
98
99	#D- Low level debug : to bypass lib test checks and stack construction
100	#D- Default : value from AA_job if any
101	libIGCM=${libIGCM:=::modipsl::/libIGCM}
102	# WARNING for StandAlone used : To run this script on some machine,
103	# you must check MirrorlibIGCM variable in sys library.
104	# If this variable is true, you must use libIGCM_POST path instead
105	# of your running libIGCM directory.
106
107	#D- Flag to determine allready produced time series. Empty if you start from the beginning
108	#D- Default : Value from AA_job if any. Usefull in StandAlone case if you want to continue a time series
109	CompletedFlag=${CompletedFlag:=}
110
111	#D- Flag to determine task type this script will perform.
112	#D- Possible Value : 2D, 3D, Chunck2D, Chunck3D
113	#D- Default : Value from AA_job if any. Usefull in StandAlone case.
114	TsTask=${TsTask:=2D}
115
116	#D- Flag to determine if rebuild process has been performed asynchronously
117	#D- Possible value true or false.
118	#D- If true raw files has allready been patched by rebuild job
119	#D- If false create_ts will patch the raw files
120	#D- Default : Value from AA_job if any. Usefull in StandAlone case.
121	RebuildFrequency=${RebuildFrequency:=true}
122
123	#D- Flag to determine job's output directory
124	#D- Default : value from libIGCM_post.ksh if any
125	POST_DIR=${POST_DIR:=${PBS_O_WORKDIR}}
126
127	#D- Increased verbosity (1, 2, 3)
128	#D- Default : value from AA_job if any
129	Verbosity=${Verbosity:=3}
130
131	#D- Low level debug : to bypass lib test checks and stack construction
132	#D- Default : value from AA_job if any
133	DEBUG_debug=${DEBUG_debug:=false}
134
135	#D- TEMPORARY Flag to determine atmospheric resolution
136	#D- Default : value from atmospheric driver if any
137	RESOL_ATM=ALL
138
139	#D- Flag to determine surface resolution
140	#D- Default : value from surface driver if any
141	RESOL_SRF=ALL
142
143	#D- Flag to determine surface resolution
144	#D- Default : value from surface driver if any
145	RESOL_SBG=ALL
146
147	#D- TEMPORARY Flag to determine ocean resolution
148	#D- Default : value from ocean driver if any
149	RESOL_OCE=${RESOL_OCE:=ORCA2}
150
151	#D- TEMPORARY Flag to determine ice resolution
152	#D- Default : value from ice driver if any
153	RESOL_ICE=${RESOL_ICE:=ORCA2}
154
155	#D- TEMPORARY Flag to determine marine biogeochemistry resolution
156	#D- Default : value from ice driver if any
157	RESOL_MBG=${RESOL_MBG:=ORCA2}
158
159	########################################################################
160
161	. ${libIGCM}/libIGCM_debug/libIGCM_debug.ksh
162	( ${DEBUG_debug} ) && IGCM_debug_Check
163	. ${libIGCM}/libIGCM_card/libIGCM_card.ksh
164	( ${DEBUG_debug} ) && IGCM_card_Check
165	. ${libIGCM}/libIGCM_date/libIGCM_date.ksh
166	( ${DEBUG_debug} ) && IGCM_date_Check
167	#-------
168	. ${libIGCM}/libIGCM_sys/libIGCM_sys.ksh
169
170	########################################################################
171
172	#set -vx
173
174	# Useful cleaning function
175	MENAGE ()
176	{
177	[ -f $3 ] && ( IGCM_sys_RshArchive rm -f $1/$2 ;)
178	}
179
180	########################################################################
181
182	IGCM_sys_MkdirWork ${RUN_DIR_PATH}
183	IGCM_sys_Cd ${RUN_DIR_PATH}
184
185	if [ ${StandAlone} = true ] ; then
186	CARD_DIR=${SUBMIT_DIR}
187	else
188	CARD_DIR=${RUN_DIR_PATH}/$( basename ${SUBMIT_DIR} )
189	IGCM_sys_Get_Master ${SUBMIT_DIR} ${RUN_DIR_PATH}
190	fi
191
192	#
193	# First of all
194	#
195	IGCM_card_DefineArrayFromSection ${CARD_DIR}/config.card UserChoices
196	typeset option
197	for option in ${config_UserChoices[*]} ; do
198	IGCM_card_DefineVariableFromOption ${CARD_DIR}/config.card UserChoices ${option}
199	done
200	#
201	echo
202	IGCM_debug_Print 1 "DefineArrayFromOption : config_UserChoices"
203	IGCM_debug_PrintVariables 3 config_UserChoices_JobName
204	IGCM_debug_PrintVariables 3 config_UserChoices_LongName
205	IGCM_debug_PrintVariables 3 config_UserChoices_TagName
206	IGCM_debug_PrintVariables 3 config_UserChoices_CalendarType
207	IGCM_debug_PrintVariables 3 config_UserChoices_DateBegin
208	IGCM_debug_PrintVariables 3 config_UserChoices_DateEnd
209	IGCM_debug_PrintVariables 3 config_UserChoices_PeriodLength
210	echo
211
212	#==================================
213	R_SAVE=${R_OUT}/${config_UserChoices_TagName}/${config_UserChoices_JobName}
214	#
215	# Determine component to take care of depending on kind of task create_ts will perform
216	#
217	( [ X${TsTask} = XChunck2D ] \|\| [ X${TsTask} = XChunck3D ] ) && set -A config_ListOfComponents ${CompToRead} \|\| IGCM_card_DefineArrayFromSection ${CARD_DIR}/config.card ListOfComponents
218	#
219	# Determine period for time series. Default : value from AA_job or from launch_TS if any
220	#
221	DateBegin=${DateBegin:=$( IGCM_date_ConvertFormatToGregorian ${config_UserChoices_DateBegin} )}
222	#
223	PeriodDateEnd=${PeriodDateEnd:=$( IGCM_date_ConvertFormatToGregorian ${config_UserChoices_DateEnd} )}
224	#
225	# Determine Dimension of the produced time series : 2D or 3D
226	if ( [ X${TsTask} = XChunck2D ] \|\| [ X${TsTask} = X2D ] ) ; then
227	Dimension=2D
228	elif ( [ X${TsTask} = XChunck3D ] \|\| [ X${TsTask} = X3D ] ) ; then
229	Dimension=3D
230	else
231	Dimension=""
232	fi
233
234
235	########################################################################
236	#
237	# Depending on requested time series fill following variables :
238	#
239	# - LISTE_FILE_${comp}[*]
240	# - LISTE_VARS_${file}[*]
241	# - LISTE_AXIS_${file}[*]
242	# - LISTE_PATCH_${file}[*]
243	########################################################################
244
245	#
246	# For each selected component determine which files need post-processing
247	#
248	DoJob=false
249	for comp in ${config_ListOfComponents[*]} ; do
250	#
251	ActiveComp=false
252	#
253	IGCM_card_DefineArrayFromOption ${CARD_DIR}/config.card ListOfComponents ${comp}
254	eval compname=\${config_ListOfComponents_${comp}[0]} > /dev/null 2>&1
255	#
256	card=${CARD_DIR}/COMP/${compname}.card
257
258	IGCM_card_DefineArrayFromOption ${card} OutputFiles List
259	#
260	ListFilesName=${compname}_OutputFiles_List
261	eval FileName0=\${${ListFilesName}[0]} > /dev/null 2>&1
262	#
263	if [ X${FileName0} != X${NULL_STR} ] ; then
264	#
265	#IGCM_debug_Print 1 "Component : ${compname}"
266	#
267	# INITIALISATION
268	#
269	typeset i
270	if ( [ X${TsTask} = XChunck2D ] \|\| [ X${TsTask} = XChunck3D ] ) ; then
271	((NbFiles=${FlagToRead}))
272	((i=${NbFiles}))
273	else
274	eval NbFiles=\${#${ListFilesName}[@]} > /dev/null 2>&1
275	i=2
276	fi
277	#
278	until [ $i -gt $NbFiles ]; do
279	#
280	eval flag_post=\${${ListFilesName}[$i]} > /dev/null 2>&1
281	#
282	if [ X${flag_post} != XNONE ] ; then
283	#
284	# For each comp and each file who needs post-processing
285	# determine which variables need time series.
286	#
287	FILE=$( echo ${flag_post} \| awk "-FPost_" '{print $2}' )
288	IGCM_card_DefineArrayFromOption ${card} Post_${FILE} TimeSeriesVars${Dimension}
289	IGCM_card_DefineVariableFromOption ${card} Post_${FILE} ChunckJob${Dimension}
290	#
291	# If TimeSeriesVars list is empty we skip
292	#
293	if [ $( eval echo \${${compname}_Post_${FILE}_TimeSeriesVars${Dimension}} ) = ${NULL_STR} ] ; then
294	#
295	IGCM_debug_Print 3 "Nothing to do for ${FILE} according to ${card}"
296	IGCM_debug_Print 3 "TimeSeriesVars${Dimension} for ${FILE} is empty"
297	(( i=i+3 ))
298	continue
299	#
300	fi
301	#
302	# If files/variables has been switch off by ChunckJob${Dimension}=0 we skip
303	#
304	if [ X$( eval echo \${${compname}_Post_${FILE}_ChunckJob${Dimension}} ) = XOFF ] ; then
305	IGCM_debug_Print 3 "Nothing to do for ${FILE} according to ${card}"
306	IGCM_debug_Print 3 "ChunckJob${Dimension}=OFF for ${FILE} so it has been switched off"
307	(( i=i+3 ))
308	continue
309	fi
310	#
311	# If we are not a Chunck type task, we skip files/variables handled by chunck task
312	#
313	if ( [ ! X${TsTask} = XChunck${Dimension} ] && [ ! X${TsTask} = X ] && [ ! XChunck${Dimension} = XChunck ] ) ; then
314	if [ ! $( eval echo \${${compname}_Post_${FILE}_ChunckJob${Dimension}} ) = NONE ] ; then
315	IGCM_debug_Print 3 "Nothing to do for ${FILE} TimeSeriesVars${Dimension} according to ${card}"
316	IGCM_debug_Print 3 "Let another chunck task do the job $( eval echo \${${compname}_Post_${FILE}_ChunckJob${Dimension}} )"
317	(( i=i+3 ))
318	continue
319	fi
320	fi
321	#
322	# Now on we know we have things to do
323	#
324
325	#
326	# We need coordinates axis to properly fill produced time series
327	#
328	IGCM_card_DefineArrayFromOption ${card} Post_${FILE} GatherWithInternal
329	#
330	if [ $( eval echo \${${compname}_Post_${FILE}_GatherWithInternal} ) = ${NULL_STR} ] ; then
331	#
332	IGCM_debug_Print 1 "ERROR !! We miss coordinate axis for ${FILE} according to ${card}"
333	IGCM_debug_Print 1 "So no time series will be produced for ${FILE} !!"
334	(( i=i+3 ))
335	continue
336	#
337	fi
338	#
339	# We need LIST of variables not allready produced (useful for standalone mode)
340	#
341	FlagDir=$( echo ${FILE} \| awk -F "_" '{print $1}' )
342	case ${FlagDir} in
343	*Y) TS_Dir=TS_YE ;;
344	*M) TS_Dir=TS_MO ;;
345	*D) TS_Dir=TS_DA ;;
346	HF) TS_Dir=TS_HF ;;
347	INS) TS_Dir=TS_INS ;;
348	esac
349	#
350	count=0
351	for var in $( eval echo \${${compname}_Post_${FILE}_TimeSeriesVars${Dimension}[*]} ) ; do
352	#
353	DIRECTORY=${R_SAVE}/${comp}/Analyse/${TS_Dir}
354	TestedFile=${config_UserChoices_JobName}_${DateBegin}_${PeriodDateEnd}_${FlagDir}_${var}.nc
355	#
356	IGCM_sys_TestFileArchive ${DIRECTORY}/${TestedFile}
357	if [ ! $? = 0 ] ; then
358	eval set +A LISTE_VARS_${FILE} \${LISTE_VARS_${FILE}[*]} ${var}
359	ActiveComp=true
360	(( count = count + 1 ))
361	fi
362	done
363	#
364	if [ ${count} -gt 0 ] ; then
365	#
366	eval set +A LISTE_FILE_${comp} \${LISTE_FILE_${comp}[*]} ${FILE}
367	eval set +A LISTE_AXIS_${FILE} \${${compname}_Post_${FILE}_GatherWithInternal[*]}
368	#
369	if [ ! X${RebuildFrequency} = Xtrue ] ; then
370	IGCM_card_DefineArrayFromOption ${card} Post_${FILE} Patches
371	eval set +A LISTE_PATCHES_${FILE} \${${compname}_Post_${FILE}_Patches[*]}
372	if [ X$( eval echo \${LISTE_PATCHES_${FILE}[0]} ) != X${NULL_STR} ]; then
373	for Patch in $( eval echo \${LISTE_PATCHES_${FILE}[*]} ); do
374	if [ Xload${Patch} != X ] ; then
375	. ${libIGCM}/libIGCM_post/IGCM_${Patch}.ksh
376	eval load${Patch}=loaded
377	fi
378	done
379	fi
380	fi
381	fi
382	fi
383	(( i=i+3 ))
384	done
385	fi
386	#
387	if [ ${ActiveComp} = true ] ; then
388	set +A ActiveListOfComponents ${ActiveListOfComponents[*]} ${comp}
389	fi
390	done
391
392	set -A config_ListOfComponents ${ActiveListOfComponents[*]}
393	[ ! -z "${ActiveListOfComponents}" ] && DoJob=true
394
395	########################################################################
396	#
397	# DEFINE DATE FUNCTIONS
398	#
399	########################################################################
400
401	# Give the end of the next $1 period from $2 date
402	function create_ts_next_date {
403	IGCM_debug_PushStack "create_ts_next_date" $@
404	typeset Length
405	case ${config_UserChoices_PeriodLength} in
406	Y\|y)
407	PeriodLengthInYears=$( echo ${config_UserChoices_PeriodLength} \| awk -F '[yY]' '{print $1}' )
408	IGCM_date_GetYearMonth ${1} year month
409	(( Length=0 ))
410	(( i=0 ))
411	until [ $i -ge $PeriodLengthInYears ] ; do
412	(( Length = Length + $( IGCM_date_DaysInYear $(( year + i + 1 )) ) ))
413	(( i=i+1 ))
414	done
415	;;
416	M\|m)
417	PeriodLengthInMonths=$( echo ${config_UserChoices_PeriodLength} \| awk -F '[mM]' '{print $1}' )
418	IGCM_date_GetYearMonth ${1} year month
419	(( year0 = year ))
420	if [ $(( month + 1 )) -lt 13 ] ; then
421	month0=$(( month + 1 ))
422	else
423	month0=$(( month + 1 - 12 ))
424	(( year = year0 + 1 ))
425	fi
426	(( Length=0 ))
427	(( i=0 ))
428	until [ $i -ge $PeriodLengthInMonths ] ; do
429	if [ $(( month0 + i )) -lt 13 ] ; then
430	(( Length = Length + $( IGCM_date_DaysInMonth $year $(( month0 + i )) ) ))
431	else
432	(( year = year0 + 1 ))
433	(( Length = Length + $( IGCM_date_DaysInMonth $year $(( month0 + i - 12 )) ) ))
434	fi
435	(( i=i+1 ))
436	done
437	;;
438	D\|d)
439	Length=$( echo ${config_UserChoices_PeriodLength} \| sed -e "s/[dD]//" ) ;;
440	*)
441	IGCM_debug_Exit "create_ts " ${config_UserChoices_PeriodLength} " invalid PeriodLength : choose in 1Y, M, D."
442	IGCM_debug_Verif_Exit_Post ;;
443	esac
444	echo ${Length}
445
446	IGCM_debug_PopStack "create_ts_next_date"
447	}
448
449	# Give the end of the actual $1 period from $2 date
450	function create_ts_begin_date {
451	IGCM_debug_PushStack "create_ts_begin_date" $@
452	typeset Length
453	case ${config_UserChoices_PeriodLength} in
454	Y\|y)
455	PeriodLengthInYears=$( echo ${config_UserChoices_PeriodLength} \| awk -F '[yY]' '{print $1}' )
456	IGCM_date_GetYearMonth ${1} year month
457	if [ X${2} = Xend ] ; then
458	(( year = year - PeriodLengthInYears + 1))
459	fi
460	(( Length=0 ))
461	(( i=0 ))
462	until [ $i -ge $PeriodLengthInYears ] ; do
463	(( Length = Length + $( IGCM_date_DaysInYear $(( year + i )) ) ))
464	(( i=i+1 ))
465	done
466	;;
467	M\|m)
468	PeriodLengthInMonths=$( echo ${config_UserChoices_PeriodLength} \| awk -F '[mM]' '{print $1}' )
469	IGCM_date_GetYearMonth ${1} year month
470	if [ X${2} = Xend ] ; then
471	(( month = month - PeriodLengthInMonths + 1 ))
472	fi
473	(( year0 = year ))
474	if [ $month -le 0 ] ; then
475	(( month = month + 12 ))
476	year=$( printf "%04i\n" $(( year - 1 )) )
477	fi
478	month=$( printf "%02i\n" ${month} )
479	(( Length=0 ))
480	(( i=0 ))
481	until [ $i -ge $PeriodLengthInMonths ] ; do
482	if [ $(( month + i )) -lt 13 ] ; then
483	(( Length = Length + $( IGCM_date_DaysInMonth $year $(( month + i )) ) ))
484	else
485	(( year = year0 + 1 ))
486	(( Length = Length + $( IGCM_date_DaysInMonth $year $(( month + i - 12 )) ) ))
487	fi
488	(( i=i+1 ))
489	done
490	;;
491	D\|d)
492	Length=$( echo ${config_UserChoices_PeriodLength} \| sed -e "s/[dD]//" ) ;;
493	*)
494	IGCM_debug_Exit "create_ts " ${config_UserChoices_PeriodLength} " invalid PeriodLength : choose in 1Y, M, D."
495	IGCM_debug_Verif_Exit_Post ;;
496	esac
497	echo ${Length}
498
499	IGCM_debug_PopStack "create_ts_begin_date"
500	}
501
502	########################################################################
503	#
504	# IS THERE SOME ALLREADY PRODUCED TIME SERIES ?
505	# IF SO BRING THEM IN THE WORKING DIRECTORY
506	#
507	# PS : Keep in mind that IGCM_sys_Get here is a weak link :
508	# - especially for IDRiS
509	# - no special protection against "command too long" !
510	# - libIGCM_sys should handle that or modify following code part
511	#
512	########################################################################
513
514	if [ ${StandAlone} != true ] ; then
515	Running_Flag=$( IGCM_sys_RshMaster "IGCM_card_DefineVariableFromOption \
516	${SUBMIT_DIR}/run.card PostProcessing TimeSeriesRunning ; \
517	echo \${run_PostProcessing_TimeSeriesRunning}" )
518
519	#if [ X${Running_Flag} = Xy ] ; then
520	# # Time Series Job allready running
521	# IGCM_debug_Print 1 "Time Series Job allready running exit"
522	# exit
523	#fi
524	# Now run.card know we are running
525
526
527
528
529	IGCM_sys_RshMaster IGCM_card_WriteOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesRunning "y"
530
531
532
533
534
535	# Get information from last execution
536	#CompletedFlag=$( IGCM_sys_RshMaster \
537	# "IGCM_card_DefineVariableFromOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesCompleted ;\
538	# echo \${run_PostProcessing_TimeSeriesCompleted}" )
539	fi
540
541	# Is it the first submission or not ?
542
543	if ( [ X${CompletedFlag} = X ] \|\| [ X${CompletedFlag} = X$( IGCM_date_AddDaysToGregorianDate ${DateBegin} -1 ) ] ) ; then
544	# First Time Series Submission
545	FIRST_PASS=TRUE
546	Length=$( create_ts_begin_date ${DateBegin} begin)
547	DATE_FIN_JOB_B=$( IGCM_date_AddDaysToGregorianDate ${DateBegin} $(( Length - 1 )) )
548	else
549	# Last Completed Time Series
550	FIRST_PASS=FALSE
551	DATE_FIN_JOB_B=${CompletedFlag}
552	date=${DateBegin}_${DATE_FIN_JOB_B}
553	for comp in ${config_ListOfComponents[*]} ; do
554	#
555	IGCM_sys_Mkdir ${RUN_DIR_PATH}/${comp}
556	#
557	IGCM_sys_Cd ${RUN_DIR_PATH}/${comp}
558	#
559	#set -A liste_file
560	unset liste_file
561	#
562	file_path=${R_SAVE}/${comp}/Analyse
563	#
564	i=0
565	file=${NULL_STR}
566	#
567	for file in $( eval echo \${LISTE_FILE_${comp}[*]} ); do
568	#
569	# Determine in which output can we find file
570	# IE : Analyse/TS or Analyse/DA2TS or ...
571	#
572	FlagDir=$( echo ${file} \| awk -F "_" '{print $1}' )
573	case ${FlagDir} in
574	*Y) TS_Dir=TS_YE ;;
575	*M) TS_Dir=TS_MO ;;
576	*D) TS_Dir=TS_DA ;;
577	HF) TS_Dir=TS_HF ;;
578	INS) TS_Dir=TS_INS ;;
579	esac
580	for var in $( eval echo \${LISTE_VARS_${file}[*]} ) ; do
581	#
582	#set +A liste_file ${liste_file[*]} ${file_path}/${TS_Dir}/${config_UserChoices_JobName}_${date}_${FlagDir}_${var}.nc
583	liste_file[${#liste_file[*]}]=${file_path}/${TS_Dir}/${config_UserChoices_JobName}_${date}_${FlagDir}_${var}.nc
584	#
585	done
586	done
587	if [ X${file} != X${NULL_STR} ] ; then
588	IGCM_sys_Get /l liste_file[*] ${RUN_DIR_PATH}/${comp}
589	ExitFlag=false
590	IGCM_debug_Verif_Exit_Post
591	fi
592	done
593	#
594	LEVEL=0
595	#
596	fi
597
598	########################################################################
599	#
600	# DEFINE LOOP PARAMETERS
601	#
602	########################################################################
603
604	DATE_COUNT=${DATE_FIN_JOB_B}
605	DATE_FIN_JOB_B_LOOP=${DATE_FIN_JOB_B}
606
607	( [ -n "${LEVEL}" ] && [ ${LEVEL} -eq 0 ] ) && NBRE_FILE_TOT=0 \|\| NBRE_FILE_TOT=1
608
609	if [ ${DoJob} = true ] ; then
610	while [ ${DATE_COUNT} -lt ${PeriodDateEnd} ] ; do
611	(( NBRE_FILE_TOT = NBRE_FILE_TOT + 1 ))
612	Length=$( create_ts_next_date ${DATE_COUNT} )
613	DATE_COUNT=$( IGCM_date_AddDaysToGregorianDate ${DATE_COUNT} ${Length} )
614	done
615	else
616	NBRE_FILE_TOT=0
617	fi
618
619	if [ ${NBRE_FILE_TOT} -le ${UNIX_MAX_LIMIT} ] ; then
620	NBRE_TOUR=1
621	elif [ ${NBRE_FILE_TOT}%${UNIX_MAX_LIMIT} -eq 0 ] ; then
622	let NBRE_TOUR=${NBRE_FILE_TOT}/${UNIX_MAX_LIMIT}
623	else
624	let NBRE_TOUR=${NBRE_FILE_TOT}/${UNIX_MAX_LIMIT}+1
625	fi
626
627	CURRENT_LOOP=1
628	# THIS LOOP GIVES PARAMETERS FOR THE DATE'S LOOP ie : DATE_COURANTE, DATE_FIN.
629	# WE HAVE 3 LEVELS
630	# - LEVEL 1 : JUST ONE LOOP TO COMPLETE
631	# - LEVEL 2 : WE ARE IN THE "DEEP" LOOP
632	# - LEVEL 3 : WE ARE IN THE LAST LOOP
633
634	while [ ${CURRENT_LOOP} -le ${NBRE_TOUR} ] ; do
635
636	if [ ${NBRE_FILE_TOT} -le ${UNIX_MAX_LIMIT} ] ; then
637	# - LEVEL 1 : JUST ONE LOOP TO COMPLETE
638	NBRE_FILE_LOOP=${NBRE_FILE_TOT}
639
640	if [ ${FIRST_PASS} = TRUE ] ; then
641	DATE_COURANTE=${DATE_FIN_JOB_B_LOOP}
642	else
643	Length=$( create_ts_next_date ${DATE_FIN_JOB_B_LOOP} )
644	DATE_COURANTE=$( IGCM_date_AddDaysToGregorianDate ${DATE_FIN_JOB_B_LOOP} ${Length} )
645	fi
646
647	DATE_FIN=${PeriodDateEnd}
648	DATE_FIN_JOB_B_LOOP_PREC=${DATE_FIN_JOB_B_LOOP}
649
650	[ -n "${DATE_LOOP}" ] && DATE_FIN_JOB_B_LOOP=${DATE_LOOP}
651	[ -n "${LEVEL}" ] && FLAG_B="TRUE" \|\| FLAG_B="FALSE"
652
653	LEVEL=1
654	DEPOT="TRUE"
655	elif [ ${CURRENT_LOOP} -ne ${NBRE_TOUR} ] ; then
656	# - LEVEL 2 : WE ARE IN THE "DEEP" LOOP
657	NBRE_FILE_LOOP=${UNIX_MAX_LIMIT}
658
659	if [ ! ${FIRST_PASS} = TRUE ] && [ ${CURRENT_LOOP} -eq 1 ] ; then
660	Length=$( create_ts_next_date ${DATE_FIN_JOB_B_LOOP} )
661	DATE_COURANTE=$( IGCM_date_AddDaysToGregorianDate ${DATE_FIN_JOB_B_LOOP} ${Length} )
662	elif [ -z "${DATE_COURANTE}" ] ; then
663	DATE_COURANTE=${DATE_FIN_JOB_B}
664	else
665	Length=$( create_ts_next_date ${DATE_FIN} )
666	DATE_COURANTE=$( IGCM_date_AddDaysToGregorianDate ${DATE_FIN} ${Length} )
667	fi
668
669	(( TotLength = 0 ))
670	COMPTEUR=1
671	DATE_LOOP=${DATE_COURANTE}
672
673	while [ ${COMPTEUR} -lt ${NBRE_FILE_LOOP} ] ; do
674	#
675	Length=$( create_ts_next_date ${DATE_LOOP} )
676	DATE_LOOP=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} ${Length} )
677	(( TotLength = TotLength + Length ))
678	(( COMPTEUR = COMPTEUR + 1 ))
679	#
680	done # while [ ${COMPTEUR} -le ${NBRE_FILE_LOOP} ] done
681	DATE_FIN=$( IGCM_date_AddDaysToGregorianDate ${DATE_COURANTE} ${TotLength} )
682
683	[ -n "${DATE_LOOP}" ] && DATE_FIN_JOB_B_LOOP_PREC=${DATE_FIN_JOB_B_LOOP} ; DATE_FIN_JOB_B_LOOP=${DATE_LOOP}
684	[ -n "${LEVEL}" ] && [ ${LEVEL} -eq 0 ] && FLAG_B="TRUE" \|\| [ ${CURRENT_LOOP} -gt 1 ] && FLAG_B="TRUE" \|\| FLAG_B="FALSE"
685	DEPOT="FALSE"
686	LEVEL=2
687	else
688	# - LEVEL 3 : WE ARE IN THE LAST LOOP
689	NBRE_FILE_LOOP=$(( ${NBRE_FILE_TOT} % ( ${UNIX_MAX_LIMIT} * ( ${CURRENT_LOOP} - 1 ) ) ))
690	[ ${NBRE_FILE_LOOP} -eq 0 ] && NBRE_FILE_LOOP=${UNIX_MAX_LIMIT}
691
692	Length=$( create_ts_next_date ${DATE_FIN} )
693	DATE_COURANTE=$(IGCM_date_AddDaysToGregorianDate ${DATE_FIN} ${Length} )
694	DATE_FIN=${PeriodDateEnd}
695	[ -n "${DATE_LOOP}" ] && DATE_FIN_JOB_B_LOOP_PREC=${DATE_FIN_JOB_B_LOOP} ; DATE_FIN_JOB_B_LOOP=${DATE_LOOP}
696
697	LEVEL=3
698	DEPOT="TRUE"
699	FLAG_B="TRUE"
700	fi
701
702	# THEN FOR EACH FILE TYPE WE LOOP BETWEEN DATE_COURANTE AND DATE_FIN.
703	for comp in ${config_ListOfComponents[*]} ; do
704	#
705	IGCM_sys_Mkdir ${RUN_DIR_PATH}/${comp}
706	#
707	IGCM_sys_Cd ${RUN_DIR_PATH}/${comp}
708	#
709	eval R_OUT_${comp}=${R_SAVE}/${comp}
710	#
711	i=0
712	#
713	for file in $( eval echo \${LISTE_FILE_${comp}[*]} ); do
714	#
715	# Determine in which output can we find file
716	# IE : Output/MO or Output/DA or ...
717	#
718	FlagDir=$( echo ${file} \| awk -F "_" '{print $1}' )
719	case ${FlagDir} in
720	*Y) TS_Dir=TS_YE ; FreqDir=YE ;;
721	*M) TS_Dir=TS_MO ; FreqDir=MO ;;
722	*D) TS_Dir=TS_DA ; FreqDir=DA ;;
723	HF) TS_Dir=TS_HF ; FreqDir=HF ;;
724	INS) TS_Dir=TS_INS ; FreqDir=INS ;;
725	*)
726	IGCM_debug_Print 1 "Error in FlagDir=${FlagDir} for file ${file}."
727	IGCM_sys_Exit "Job create_ts" ;;
728	esac
729
730	# Initialize array
731	#set -A liste_file
732	#set -A liste_file_tmp
733	unset liste_file
734	unset liste_file_tmp
735	eval file_path=\${R_OUT_${comp}}/Output/${FreqDir}/
736
737	COMPTEUR=1
738	DATE_LOOP=${DATE_COURANTE}
739
740	while [ ${COMPTEUR} -le ${NBRE_FILE_LOOP} ] ; do
741	#
742	Length1=$( create_ts_begin_date ${DATE_LOOP} end)
743	DATE_TAB=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} $(( 1 - Length1 )) )_${DATE_LOOP}
744
745	Length2=$( create_ts_next_date ${DATE_LOOP} end)
746	DATE_LOOP=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} ${Length2} )
747
748	#set +A liste_file ${liste_file[*]} ${file_path}${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc
749	#set +A liste_file_tmp ${liste_file_tmp[*]} ${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc
750	liste_file[${#liste_file[*]}]=${file_path}${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc
751	liste_file_tmp[${#liste_file_tmp[*]}]=${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc
752	(( COMPTEUR = COMPTEUR + 1 ))
753	#
754	done # while [ ${COMPTEUR} -le ${NBRE_FILE_LOOP} ] done
755	DATE_LOOP=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} -${Length2} )
756
757	# Get selected files
758	IGCM_sys_Get /l liste_file[*] ${RUN_DIR_PATH}/${comp}
759	ExitFlag=false
760	IGCM_debug_Verif_Exit_Post
761
762	# Apply IGCM_Patch if needed
763	if [ ! X${RebuildFrequency} = Xtrue ] ; then
764	if [ X$( eval echo \${LISTE_PATCHES_${file}[0]} ) != X${NULL_STR} ]; then
765	for file_t in $( eval echo ${liste_file_tmp[*]} ); do
766	for Patch in $( eval echo \${LISTE_PATCHES_${file}[*]} ); do
767	IGCM_${Patch} ${file_t}
768	done
769	done
770	fi
771	fi
772
773	# WE CAN CONCATENATE FILES AT THIS POINT
774	[ "${FLAG_B}" = "TRUE" ] && DATE_BUILD_B=${DateBegin}_${DATE_FIN_JOB_B_LOOP_PREC}
775	Length=$( create_ts_begin_date ${DATE_COURANTE} end)
776	DATE_BUILD1=$( IGCM_date_AddDaysToGregorianDate ${DATE_COURANTE} -$(( Length - 1 )) )
777	DATE_BUILD=${DATE_BUILD1}_${DATE_FIN}
778	DATE_BUILD_END=${DateBegin}_${DATE_FIN}
779
780	liste_coord=" "
781	for axis in $( eval echo \${LISTE_AXIS_${file}[*]} ); do
782	liste_coord=${liste_coord}${axis}","
783	done
784
785	# Time axis must be the last dimension
786	time_axis=${axis}
787
788	# SWITCH BETWEEN MODEL TO FIT PARTICULAR CASE
789	for var in $( eval echo \${LISTE_VARS_${file}[*]} ) ; do
790	#
791	IGCM_sys_ncrcat --hst -v ${liste_coord}${var} ${liste_file_tmp[*]} ${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
792	#
793	# # Change time axis name to be CF compliant
794	# if [ ${time_axis} != time_counter ]; then
795	# IGCM_sys_ncrename -O -v ${time_axis},time_counter ${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
796	# fi
797	#
798	if [ "${FLAG_B}" = "TRUE" ] ; then
799	# WE CONCATENATE WITH EXISTING FILES
800	file1=${config_UserChoices_JobName}_${DATE_BUILD_B}_${FlagDir}_${var}.nc
801	file2=${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
802	file_out=${config_UserChoices_JobName}_${DATE_BUILD_END}_${FlagDir}_${var}.nc
803	IGCM_sys_ncrcat --hst ${file1} ${file2} ${file_out}
804	#
805	IGCM_sys_Rm ${file1} ${file2}
806	fi
807	#
808	if [ "${DEPOT}" = "TRUE" ] ; then
809	# WE PUT FINAL FILE ON FILE SYSTEM
810	[ "${FLAG_B}" = "FALSE" ] && file_out=${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
811	#${ncatted} -a modulo,time_counter,c,c," " ${file_out}
812	file_before=${config_UserChoices_JobName}_${DateBegin}_${DATE_FIN_JOB_B}_${FlagDir}_${var}.nc
813	eval IGCM_sys_Put_Out ${file_out} \${R_OUT_${comp}}/Analyse/${TS_Dir}/${file_out}
814	[ "${FLAG_B}" = "TRUE" ] && eval MENAGE \${R_OUT_${comp}}/Analyse/${TS_Dir} ${file_before} ${file_out}
815	IGCM_sys_Rm ${file_out}
816	#
817	fi
818	#
819	ExitFlag=false
820	IGCM_debug_Verif_Exit_Post
821	done # for var in ${liste_var} ; do
822	IGCM_sys_Rm ${liste_file_tmp[*]}
823	done # for file in ${liste_file_brut} do
824	done # for comp in ${config_ListOfComponents[*]} ; do
825	ExitFlag=false
826	IGCM_debug_Verif_Exit_Post
827	# GO AHEAD IN DEEPEST LOOP
828	(( CURRENT_LOOP = CURRENT_LOOP + 1 ))
829	done # while [ ${CURRENT_LOOP} -le ${NBRE_TOUR} ] do
830	# FINISH
831
832	# DODS copy
833	for comp in ${config_ListOfComponents[*]} ; do
834	for TS_Dir in TS_YE TS_MO TS_DA TS_HF TS_INS ; do
835	IGCM_sys_TestDirArchive ${R_SAVE}/${comp}/Analyse/${TS_Dir}
836	[ $? = 0 ] && IGCM_sys_Put_Dods ${comp}/Analyse/${TS_Dir}
837	done
838	done
839
840	export POST_DIR; export SUBMIT_DIR; export libIGCM; export R_INIT; export R_BC; export StandAlone
841	export RESOL_ATM; export RESOL_OCE ; export RESOL_ICE ; export RESOL_MBG ; export RESOL_SRF ; export RESOL_SBG
842
843	listVarEnv="POST_DIR,SUBMIT_DIR,libIGCM,R_INIT,R_BC,StandAlone,RESOL_ATM,RESOL_OCE,RESOL_ICE,RESOL_MBG,RESOL_SRF,RESOL_SBG"
844	export listVarEnv
845
846	#D-
847	# --------------------------------------------------------------------
848	#D- Test if all was right
849	# --------------------------------------------------------------------
850	IGCM_debug_Verif_Exit_Post
851	# --------------------------------------------------------------------
852	#D- Submit MONITORING
853	# --------------------------------------------------------------------
854	( [ X${TsTask} = X2D ] \|\| [ X${TsTask} = X ] ) && IGCM_sys_QsubPost monitoring
855
856	#D-
857	# --------------------------------------------------------------------
858	#D- Update the run.card
859	# --------------------------------------------------------------------
860	if [ ${StandAlone} != true ] ; then
861	# Put in run.card end period of time series
862	IGCM_sys_RshMaster IGCM_card_WriteOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesCompleted "${DATE_FIN}"
863	# We have finish the job !
864	IGCM_sys_RshMaster IGCM_card_WriteOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesRunning "n"
865	fi
866
867	# Clean RUN_DIR_PATH (necessary for cesium)
868	IGCM_sys_RmRunDir -Rf ${RUN_DIR_PATH}
869
870	date

Note: See TracBrowser for help on using the repository browser.

Download in other formats: