Context Navigation

AA_create_ts @ 218

Last change on this file since 218 was 213, checked in by sdipsl, 14 years ago

In AA_create_ts and AA_create_se no need to try to patch raw files when asynchronous rebuild has been used

Some enhancements to ease the use in StandAlone? mode

Property licence set to
The following licence information concerns ONLY the libIGCM tools
==================================================================

Copyright © Centre National de la Recherche Scientifique CNRS
Commissariat à l'Énergie Atomique CEA

libIGCM : Library for Portable Models Computation of IGCM Group.

IGCM Group is the french IPSL Global Climate Model Group.

This library is a set of shell scripts and functions whose purpose is
the management of the initialization, the launch, the transfer of
output files, the post-processing and the monitoring of datas produce
by any numerical program on any plateforme.

This software is governed by the CeCILL license under French law and
abiding by the rules of distribution of free software. You can use,
modify and/ or redistribute the software under the terms of the CeCILL
license as circulated by CEA, CNRS and INRIA at the following URL
"http://www.cecill.info".

As a counterpart to the access to the source code and rights to copy,
modify and redistribute granted by the license, users are provided only
with a limited warranty and the software's author, the holder of the
economic rights, and the successive licensors have only limited
liability.

In this respect, the user's attention is drawn to the risks associated
with loading, using, modifying and/or developing or reproducing the
software by the user in light of its specific status of free software,
that may mean that it is complicated to manipulate, and that also
therefore means that it is reserved for developers and experienced
professionals having in-depth computer knowledge. Users are therefore
encouraged to load and test the software's suitability as regards their
requirements in conditions enabling the security of their systems and/or
data to be ensured and, more generally, to use and operate it in the
same conditions as regards security.

The fact that you are presently reading this means that you have had
knowledge of the CeCILL license and that you accept its terms.
Property svn:keywords set to Date Author Revision

File size: 30.0 KB

Line
1	#-Q- platine #!/usr/bin/ksh
2	#-Q- platine ###################
3	#-Q- platine ## PLATINE CEA ##
4	#-Q- platine ###################
5	#-Q- platine #BSUB -J TS # Nom du job
6	#-Q- platine #BSUB -N # message a la fin du job
7	#-Q- platine #BSUB -n 1 # reservation des processeurs pour le job
8	#-Q- platine #BSUB -W 1:00 # Limite temps
9	#-Q- platine #BSUB -q post # Passage en queue post
10	#-Q- sx8brodie #!/bin/ksh
11	#-Q- sx8brodie #######################
12	#-Q- sx8brodie ## SX8BRODIE IDRIS ##
13	#-Q- sx8brodie #######################
14	#-Q- sx8brodie # Temps Elapsed max. d'une requete hh:mm:ss
15	#-Q- sx8brodie # @ wall_clock_limit = 10:00:00
16	#-Q- sx8brodie # Nom du travail LoadLeveler
17	#-Q- sx8brodie # @ job_name = TS
18	#-Q- sx8brodie # Fichier de sortie standard du travail
19	#-Q- sx8brodie # @ output = $(job_name).$(jobid)
20	#-Q- sx8brodie # Fichier de sortie d'erreur du travail
21	#-Q- sx8brodie # @ error = $(job_name).$(jobid)
22	#-Q- sx8brodie # pour recevoir un mail en cas de depassement du temps Elapsed (ou autre pb.)
23	#-Q- sx8brodie # @ notification = error
24	#-Q- sx8brodie # @ environment = $DEBUG_debug ; $MODIPSL ; $libIGCM ; $libIGCM_SX ; $SUBMIT_DIR ; $REBUILD_DIR ; $RebuildFromArchive ; $POST_DIR ; $MASTER ; $RebuildFrequency ; $PeriodDateBegin ; $PeriodDateEnd ; $NbRebuildDir ; $StandAlone ; $CompletedFlag ; $TsTask ; $CompToRead ; $FlagToRead ; $RESOL_ATM ; $RESOL_OCE ; $RESOL_ICE ; $RESOL_MBG ; $RESOL_SRF ; $RESOL_SBG ; $MASTER
25	#-Q- aix6 #!/bin/ksh
26	#-Q- aix6 #######################
27	#-Q- aix6 ## VARGAS IDRIS ##
28	#-Q- aix6 #######################
29	#-Q- aix6 # Temps Elapsed max. d'une requete hh:mm:ss
30	#-Q- aix6 # @ wall_clock_limit = 10:00:00
31	#-Q- aix6 # Nom du travail LoadLeveler
32	#-Q- aix6 # @ job_name = TS
33	#-Q- aix6 # Fichier de sortie standard du travail
34	#-Q- aix6 # @ output = $(job_name).$(jobid)
35	#-Q- aix6 # Fichier de sortie d'erreur du travail
36	#-Q- aix6 # @ error = $(job_name).$(jobid)
37	#-Q- aix6 # pour recevoir un mail en cas de depassement du temps Elapsed (ou autre pb.)
38	#-Q- aix6 # @ notification = error
39	#-Q- aix6 # @ environment = $DEBUG_debug ; $MODIPSL ; $libIGCM ; $libIGCM_SX ; $SUBMIT_DIR ; $REBUILD_DIR ; $RebuildFromArchive ; $POST_DIR ; $MASTER ; $RebuildFrequency ; $DateBegin ; $PeriodDateBegin ; $PeriodDateEnd ; $NbRebuildDir ; $StandAlone ; $CompletedFlag ; $TsTask ; $CompToRead ; $FlagToRead ; $RESOL_ATM ; $RESOL_OCE ; $RESOL_ICE ; $RESOL_MBG ; $RESOL_SRF ; $RESOL_SBG ; $MASTER
40	#-Q- aix6 # @ queue
41	#-Q- sx8brodie # @ queue
42	#-Q- sx8mercure #!/bin/ksh
43	#-Q- sx8mercure ######################
44	#-Q- sx8mercure ## SX8MERCURE CEA ##
45	#-Q- sx8mercure ######################
46	#-Q- sx8mercure #PBS -N TS # Nom du job
47	#-Q- sx8mercure #PBS -j o # regroupement des stdout et stderr
48	#-Q- sx8mercure #PBS -S /usr/bin/ksh # shell de soumission
49	#-Q- sx8mercure #PBS -l memsz_job=1gb # Limite memoire a 1 Go
50	#-Q- sx8mercure #PBS -l cputim_job=24:00:00 # Limite temps a 2 heures
51	#-Q- sx8mercure #PBS -q scalaire
52	#-Q- sx9mercure #!/bin/ksh
53	#-Q- sx9mercure ######################
54	#-Q- sx9mercure ## SX9MERCURE CEA ##
55	#-Q- sx9mercure ######################
56	#-Q- sx9mercure #MSUB -r TS # Nom du job
57	#-Q- sx9mercure #MSUB -N 1 # Reservation du noeud
58	#-Q- sx9mercure #MSUB -n 1 # Reservation du processus
59	#-Q- sx9mercure #MSUB -T 86400 # Limite de temps elapsed du job
60	#-Q- sx9mercure #MSUB -E "-j o"
61	#-Q- sx9mercure #MSUB -E "-S /bin/ksh"
62	#-Q- sx9mercure ##MSUB -e nco.out # Sortie standard
63	#-Q- sx9mercure ##MSUB -o nco.out # Sortie standard
64	#-Q- default #!/bin/ksh
65	#-Q- default ##################
66	#-Q- default ## DEFAULT HOST ##
67	#-Q- default ##################
68
69	# $Date$
70	# $Author$
71	# $Revision$
72	# IPSL (2006)
73	# This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
74
75	#set -eu
76	#set -vx
77
78	date
79
80	#-Q- sx8brodie export OMP_NUM_THREADS=1
81	#-Q- aix6 export OMP_NUM_THREADS=1
82
83	########################################################################
84
85	#D- Flag to determine if this job in a standalone mode
86	#D- Default : value from AA_job if any
87	StandAlone=${StandAlone:=true}
88
89	#D- Low level debug : to bypass lib test checks and stack construction
90	#D- Default : value from AA_job if any
91	libIGCM=${libIGCM:=/path/to/your/libIGCM}
92
93	#D- Flag to determine allready produced time series. Empty if you start from the beginning
94	#D- Default : Value from AA_job if any. Usefull in StandAlone case if you want to continue a time series
95	CompletedFlag=${CompletedFlag:=}
96
97	#D- Flag to determine task type this script will perform.
98	#D- Possible Value : 2D, 3D, Chunck2D, Chunck3D
99	#D- Default : Value from AA_job if any. Usefull in StandAlone case.
100	TsTask=${TsTask:=2D}
101
102	#D- Flag to determine if rebuild process has been performed asynchronously
103	#D- Possible value true or false.
104	#D- If true raw files has allready been patched by rebuild job
105	#D- If false create_ts will patch the raw files
106	#D- Default : Value from AA_job if any. Usefull in StandAlone case.
107	RebuildFrequency=${RebuildFrequency:=true}
108
109	#D- Flag to determine job's output directory
110	#D- Default : value from libIGCM_post.ksh if any
111	POST_DIR=${POST_DIR:=${PBS_O_WORKDIR}}
112
113	#D- Increased verbosity (1, 2, 3)
114	#D- Default : value from AA_job if any
115	Verbosity=${Verbosity:=3}
116
117	#D- Low level debug : to bypass lib test checks and stack construction
118	#D- Default : value from AA_job if any
119	DEBUG_debug=${DEBUG_debug:=false}
120
121	#D- TEMPORARY Flag to determine atmospheric resolution
122	#D- Default : value from atmospheric driver if any
123	RESOL_ATM=${RESOL_ATM:=ALL}
124
125	#D- TEMPORARY Flag to determine ocean resolution
126	#D- Default : value from ocean driver if any
127	RESOL_OCE=${RESOL_OCE:=ORCA2}
128
129	#D- TEMPORARY Flag to determine ice resolution
130	#D- Default : value from ice driver if any
131	RESOL_ICE=${RESOL_ICE:=ORCA2}
132
133	#D- TEMPORARY Flag to determine marine biogeochemistry resolution
134	#D- Default : value from ice driver if any
135	RESOL_MBG=${RESOL_MBG:=ORCA2}
136
137	#D- Flag to determine surface resolution
138	#D- Default : value from surface driver if any
139	RESOL_SRF=ALL
140
141	#D- Flag to determine surface resolution
142	#D- Default : value from surface driver if any
143	RESOL_SBG=ALL
144
145	########################################################################
146
147	. ${libIGCM}/libIGCM_debug/libIGCM_debug.ksh
148	( ${DEBUG_debug} ) && IGCM_debug_Check
149	. ${libIGCM}/libIGCM_card/libIGCM_card.ksh
150	( ${DEBUG_debug} ) && IGCM_card_Check
151	. ${libIGCM}/libIGCM_date/libIGCM_date.ksh
152	( ${DEBUG_debug} ) && IGCM_date_Check
153	#-------
154	. ${libIGCM}/libIGCM_sys/libIGCM_sys.ksh
155
156	########################################################################
157
158	#set -vx
159
160	# Useful cleaning function
161	MENAGE ()
162	{
163	[ -f $3 ] && ( IGCM_sys_RshArchive rm -f $1/$2 ;)
164	}
165
166	########################################################################
167
168	IGCM_sys_MkdirWork ${RUN_DIR_PATH}
169	IGCM_sys_Cd ${RUN_DIR_PATH}
170
171	if [ ${StandAlone} = true ] ; then
172	CARD_DIR=${SUBMIT_DIR}
173	else
174	CARD_DIR=${RUN_DIR_PATH}/$( basename ${SUBMIT_DIR} )
175	IGCM_sys_Get_Master ${SUBMIT_DIR} ${RUN_DIR_PATH}
176	fi
177
178	#
179	# First of all
180	#
181	IGCM_card_DefineArrayFromSection ${CARD_DIR}/config.card UserChoices
182	typeset option
183	for option in ${config_UserChoices[*]} ; do
184	IGCM_card_DefineVariableFromOption ${CARD_DIR}/config.card UserChoices ${option}
185	done
186	#
187	echo
188	IGCM_debug_Print 1 "DefineArrayFromOption : config_UserChoices"
189	IGCM_debug_PrintVariables 3 config_UserChoices_JobName
190	IGCM_debug_PrintVariables 3 config_UserChoices_LongName
191	IGCM_debug_PrintVariables 3 config_UserChoices_TagName
192	IGCM_debug_PrintVariables 3 config_UserChoices_CalendarType
193	IGCM_debug_PrintVariables 3 config_UserChoices_DateBegin
194	IGCM_debug_PrintVariables 3 config_UserChoices_DateEnd
195	IGCM_debug_PrintVariables 3 config_UserChoices_PeriodLength
196	echo
197
198	#==================================
199	R_SAVE=${R_OUT}/${config_UserChoices_TagName}/${config_UserChoices_JobName}
200	#
201	# Determine component to take care of depending on kind of task create_ts will perform
202	#
203	( [ X${TsTask} = XChunck2D ] \|\| [ X${TsTask} = XChunck3D ] ) && set -A config_ListOfComponents ${CompToRead} \|\| IGCM_card_DefineArrayFromSection ${CARD_DIR}/config.card ListOfComponents
204	#
205	# Determine period for time series. Default : value from AA_job or from launch_TS if any
206	#
207	DateBegin=${DateBegin:=$( IGCM_date_ConvertFormatToGregorian ${config_UserChoices_DateBegin} )}
208	#
209	PeriodDateEnd=${PeriodDateEnd:=$( IGCM_date_ConvertFormatToGregorian ${config_UserChoices_DateEnd} )}
210	#
211	# Determine Dimension of the produced time series : 2D or 3D
212	if ( [ X${TsTask} = XChunck2D ] \|\| [ X${TsTask} = X2D ] ) ; then
213	Dimension=2D
214	elif ( [ X${TsTask} = XChunck3D ] \|\| [ X${TsTask} = X3D ] ) ; then
215	Dimension=3D
216	else
217	Dimension=""
218	fi
219
220
221	########################################################################
222	#
223	# Depending on requested time series fill following variables :
224	#
225	# - LISTE_FILE_${comp}[*]
226	# - LISTE_VARS_${file}[*]
227	# - LISTE_AXIS_${file}[*]
228	# - LISTE_PATCH_${file}[*]
229	########################################################################
230
231	#
232	# For each selected component determine which files need post-processing
233	#
234	DoJob=false
235	for comp in ${config_ListOfComponents[*]} ; do
236	#
237	ActiveComp=false
238	#
239	IGCM_card_DefineArrayFromOption ${CARD_DIR}/config.card ListOfComponents ${comp}
240	eval compname=\${config_ListOfComponents_${comp}[0]} > /dev/null 2>&1
241	#
242	card=${CARD_DIR}/COMP/${compname}.card
243
244	IGCM_card_DefineArrayFromOption ${card} OutputFiles List
245	#
246	ListFilesName=${compname}_OutputFiles_List
247	eval FileName0=\${${ListFilesName}[0]} > /dev/null 2>&1
248	#
249	if [ X${FileName0} != X${NULL_STR} ] ; then
250	#
251	#IGCM_debug_Print 1 "Component : ${compname}"
252	#
253	# INITIALISATION
254	#
255	typeset i
256	if ( [ X${TsTask} = XChunck2D ] \|\| [ X${TsTask} = XChunck3D ] ) ; then
257	((NbFiles=${FlagToRead}))
258	((i=${NbFiles}-2))
259	else
260	eval NbFiles=\${#${ListFilesName}[@]} > /dev/null 2>&1
261	i=0
262	fi
263	#
264	until [ $i -ge $NbFiles ]; do
265	#
266	(( i_ = i+2 ))
267	eval flag_post_=\${${ListFilesName}[$i_]} > /dev/null 2>&1
268	eval flag_post=${flag_post_}
269	#
270	if [ X${flag_post} != XNONE ] ; then
271	#
272	# For each comp and each file who needs post-processing
273	# determine which variables need time series.
274	#
275	FILE=$( echo ${flag_post} \| awk "-FPost_" '{print $2}' )
276	IGCM_card_DefineArrayFromOption ${card} Post_${FILE} TimeSeriesVars${Dimension}
277	#
278	if [ $( eval echo \${${compname}_Post_${FILE}_TimeSeriesVars${Dimension}} ) = ${NULL_STR} ] ; then
279	#
280	IGCM_debug_Print 3 "Nothing to do for ${FILE} according to ${card}"
281	(( i=i+3 ))
282	continue
283	#
284	fi
285	#
286	# If we are not a Chunck type task, skip files/variables handled by chunck task
287	#
288	if ( [ ! X${TsTask} = XChunck${Dimension} ] && [ ! X${TsTask} = X ] && [ ! XChunck${Dimension} = XChunck ] ) ; then
289	IGCM_card_DefineVariableFromOption ${card} Post_${FILE} ChunckJob${Dimension}
290	if [ ! $( eval echo \${${compname}_Post_${FILE}_ChunckJob${Dimension}} ) = NONE ] ; then
291	IGCM_debug_Print 3 "Nothing to do for ${FILE} TimeSeriesVars${Dimension} according to ${card}"
292	IGCM_debug_Print 3 "Let another chunck task do the job $( eval echo \${${compname}_Post_${FILE}_ChunckJob${Dimension}} )"
293	(( i=i+3 ))
294	continue
295	fi
296	fi
297	#
298	# We need coordinates axis to properly fill produced time series
299	#
300	IGCM_card_DefineArrayFromOption ${card} Post_${FILE} GatherWithInternal
301	#
302	if [ $( eval echo \${${compname}_Post_${FILE}_GatherWithInternal} ) = ${NULL_STR} ] ; then
303	#
304	IGCM_debug_Print 1 "ERROR !! We miss coordinate axis for ${FILE} according to ${card}"
305	IGCM_debug_Print 1 "So no time series will be produced for ${FILE} !!"
306	(( i=i+3 ))
307	continue
308	#
309	fi
310	#
311	# We need LIST of variables not allready produced (useful for standalone mode)
312	#
313	FlagDir=$( echo ${FILE} \| awk -F "_" '{print $1}' )
314	case ${FlagDir} in
315	*Y) TS_Dir=TS_YE ;;
316	*M) TS_Dir=TS_MO ;;
317	*D) TS_Dir=TS_DA ;;
318	HF) TS_Dir=TS_HF ;;
319	INS) TS_Dir=TS_INS ;;
320	esac
321	#
322	count=0
323	for var in $( eval echo \${${compname}_Post_${FILE}_TimeSeriesVars${Dimension}[*]} ) ; do
324	#
325	DIRECTORY=${R_SAVE}/${comp}/Analyse/${TS_Dir}
326	TestedFile=${config_UserChoices_JobName}_${DateBegin}_${PeriodDateEnd}_${FlagDir}_${var}.nc
327	#
328	IGCM_sys_TestFileArchive ${DIRECTORY}/${TestedFile}
329	if [ ! $? = 0 ] ; then
330	eval set +A LISTE_VARS_${FILE} \${LISTE_VARS_${FILE}[*]} ${var}
331	ActiveComp=true
332	(( count = count + 1 ))
333	fi
334	done
335	#
336	if [ ${count} -gt 0 ] ; then
337	#
338	eval set +A LISTE_FILE_${comp} \${LISTE_FILE_${comp}[*]} ${FILE}
339	eval set +A LISTE_AXIS_${FILE} \${${compname}_Post_${FILE}_GatherWithInternal[*]}
340	#
341	if [ ! X${RebuildFrequency} = Xtrue ] ; then
342	IGCM_card_DefineArrayFromOption ${card} Post_${FILE} Patches
343	eval set +A LISTE_PATCHES_${FILE} \${${compname}_Post_${FILE}_Patches[*]}
344	if [ X$( eval echo \${LISTE_PATCHES_${FILE}[0]} ) != X${NULL_STR} ]; then
345	for Patch in $( eval echo \${LISTE_PATCHES_${FILE}[*]} ); do
346	if [ Xload${Patch} != X ] ; then
347	. ${libIGCM}/libIGCM_post/IGCM_${Patch}.ksh
348	eval load${Patch}=loaded
349	fi
350	done
351	fi
352	fi
353	fi
354	fi
355	(( i=i+3 ))
356	done
357	fi
358	#
359	if [ ${ActiveComp} = true ] ; then
360	set +A ActiveListOfComponents ${ActiveListOfComponents[*]} ${comp}
361	fi
362	done
363
364	set -A config_ListOfComponents ${ActiveListOfComponents[*]}
365	[ ! -z "${ActiveListOfComponents}" ] && DoJob=true
366
367	########################################################################
368	#
369	# DEFINE DATE FUNCTIONS
370	#
371	########################################################################
372
373	# Give the end of the next $1 period from $2 date
374	function create_ts_next_date {
375	IGCM_debug_PushStack "create_ts_next_date" $@
376	typeset Length
377	case ${config_UserChoices_PeriodLength} in
378	Y\|y)
379	PeriodLengthInYears=$( echo ${config_UserChoices_PeriodLength} \| awk -F '[yY]' '{print $1}' )
380	IGCM_date_GetYearMonth ${1} year month
381	(( Length=0 ))
382	(( i=0 ))
383	until [ $i -ge $PeriodLengthInYears ] ; do
384	(( Length = Length + $( IGCM_date_DaysInYear $(( year + i + 1 )) ) ))
385	(( i=i+1 ))
386	done
387	;;
388	M\|m)
389	PeriodLengthInMonths=$( echo ${config_UserChoices_PeriodLength} \| awk -F '[mM]' '{print $1}' )
390	IGCM_date_GetYearMonth ${1} year month
391	(( year0 = year ))
392	if [ $(( month + 1 )) -lt 13 ] ; then
393	month0=$(( month + 1 ))
394	else
395	month0=$(( month + 1 - 12 ))
396	(( year = year0 + 1 ))
397	fi
398	(( Length=0 ))
399	(( i=0 ))
400	until [ $i -ge $PeriodLengthInMonths ] ; do
401	if [ $(( month0 + i )) -lt 13 ] ; then
402	(( Length = Length + $( IGCM_date_DaysInMonth $year $(( month0 + i )) ) ))
403	else
404	(( year = year0 + 1 ))
405	(( Length = Length + $( IGCM_date_DaysInMonth $year $(( month0 + i - 12 )) ) ))
406	fi
407	(( i=i+1 ))
408	done
409	;;
410	D\|d)
411	Length=$( echo ${config_UserChoices_PeriodLength} \| sed -e "s/[dD]//" ) ;;
412	*)
413	IGCM_debug_Exit "create_ts " ${config_UserChoices_PeriodLength} " invalid PeriodLength : choose in 1Y, M, D."
414	IGCM_debug_Verif_Exit_Post ;;
415	esac
416	echo ${Length}
417
418	IGCM_debug_PopStack "create_ts_next_date"
419	}
420
421	# Give the end of the actual $1 period from $2 date
422	function create_ts_begin_date {
423	IGCM_debug_PushStack "create_ts_begin_date" $@
424	typeset Length
425	case ${config_UserChoices_PeriodLength} in
426	Y\|y)
427	PeriodLengthInYears=$( echo ${config_UserChoices_PeriodLength} \| awk -F '[yY]' '{print $1}' )
428	IGCM_date_GetYearMonth ${1} year month
429	if [ X${2} = Xend ] ; then
430	(( year = year - PeriodLengthInYears + 1))
431	fi
432	(( Length=0 ))
433	(( i=0 ))
434	until [ $i -ge $PeriodLengthInYears ] ; do
435	(( Length = Length + $( IGCM_date_DaysInYear $(( year + i )) ) ))
436	(( i=i+1 ))
437	done
438	;;
439	M\|m)
440	PeriodLengthInMonths=$( echo ${config_UserChoices_PeriodLength} \| awk -F '[mM]' '{print $1}' )
441	IGCM_date_GetYearMonth ${1} year month
442	if [ X${2} = Xend ] ; then
443	(( month = month - PeriodLengthInMonths + 1 ))
444	fi
445	(( year0 = year ))
446	if [ $month -le 0 ] ; then
447	(( month = month + 12 ))
448	year=$( printf "%04i\n" $(( year - 1 )) )
449	fi
450	month=$( printf "%02i\n" ${month} )
451	(( Length=0 ))
452	(( i=0 ))
453	until [ $i -ge $PeriodLengthInMonths ] ; do
454	if [ $(( month + i )) -lt 13 ] ; then
455	(( Length = Length + $( IGCM_date_DaysInMonth $year $(( month + i )) ) ))
456	else
457	(( year = year0 + 1 ))
458	(( Length = Length + $( IGCM_date_DaysInMonth $year $(( month + i - 12 )) ) ))
459	fi
460	(( i=i+1 ))
461	done
462	;;
463	D\|d)
464	Length=$( echo ${config_UserChoices_PeriodLength} \| sed -e "s/[dD]//" ) ;;
465	*)
466	IGCM_debug_Exit "create_ts " ${config_UserChoices_PeriodLength} " invalid PeriodLength : choose in 1Y, M, D."
467	IGCM_debug_Verif_Exit_Post ;;
468	esac
469	echo ${Length}
470
471	IGCM_debug_PopStack "create_ts_begin_date"
472	}
473
474	########################################################################
475	#
476	# IS THERE SOME ALLREADY PRODUCED TIME SERIES ?
477	# IF SO BRING THEM IN THE WORKING DIRECTORY
478	#
479	# PS : Keep in mind that IGCM_sys_Get here is a weak link :
480	# - especially for IDRiS
481	# - no special protection against "command too long" !
482	# - libIGCM_sys should handle that or modify following code part
483	#
484	########################################################################
485
486	if [ ${StandAlone} != true ] ; then
487	Running_Flag=$( IGCM_sys_RshMaster "IGCM_card_DefineVariableFromOption \
488	${SUBMIT_DIR}/run.card PostProcessing TimeSeriesRunning ; \
489	echo \${run_PostProcessing_TimeSeriesRunning}" )
490
491	#if [ X${Running_Flag} = Xy ] ; then
492	# # Time Series Job allready running
493	# IGCM_debug_Print 1 "Time Series Job allready running exit"
494	# exit
495	#fi
496	# Now run.card know we are running
497
498
499
500
501	IGCM_sys_RshMaster IGCM_card_WriteOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesRunning "y"
502
503
504
505
506
507	# Get information from last execution
508	#CompletedFlag=$( IGCM_sys_RshMaster \
509	# "IGCM_card_DefineVariableFromOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesCompleted ;\
510	# echo \${run_PostProcessing_TimeSeriesCompleted}" )
511	fi
512
513	# Is it the first submission or not ?
514
515	if ( [ X${CompletedFlag} = X ] \|\| [ X${CompletedFlag} = X$( IGCM_date_AddDaysToGregorianDate ${DateBegin} -1 ) ] ) ; then
516	# First Time Series Submission
517	FIRST_PASS=TRUE
518	Length=$( create_ts_begin_date ${DateBegin} begin)
519	DATE_FIN_JOB_B=$( IGCM_date_AddDaysToGregorianDate ${DateBegin} $(( Length - 1 )) )
520	else
521	# Last Completed Time Series
522	FIRST_PASS=FALSE
523	DATE_FIN_JOB_B=${CompletedFlag}
524	date=${DateBegin}_${DATE_FIN_JOB_B}
525	for comp in ${config_ListOfComponents[*]} ; do
526	#
527	IGCM_sys_Mkdir ${RUN_DIR_PATH}/${comp}
528	#
529	IGCM_sys_Cd ${RUN_DIR_PATH}/${comp}
530	#
531	#set -A liste_file
532	unset liste_file
533	#
534	file_path=${R_SAVE}/${comp}/Analyse
535	#
536	i=0
537	file=${NULL_STR}
538	#
539	for file in $( eval echo \${LISTE_FILE_${comp}[*]} ); do
540	#
541	# Determine in which output can we find file
542	# IE : Analyse/TS or Analyse/DA2TS or ...
543	#
544	FlagDir=$( echo ${file} \| awk -F "_" '{print $1}' )
545	case ${FlagDir} in
546	*Y) TS_Dir=TS_YE ;;
547	*M) TS_Dir=TS_MO ;;
548	*D) TS_Dir=TS_DA ;;
549	HF) TS_Dir=TS_HF ;;
550	INS) TS_Dir=TS_INS ;;
551	esac
552	for var in $( eval echo \${LISTE_VARS_${file}[*]} ) ; do
553	#
554	#set +A liste_file ${liste_file[*]} ${file_path}/${TS_Dir}/${config_UserChoices_JobName}_${date}_${FlagDir}_${var}.nc
555	liste_file[${#liste_file[*]}]=${file_path}/${TS_Dir}/${config_UserChoices_JobName}_${date}_${FlagDir}_${var}.nc
556	#
557	done
558	done
559	if [ X${file} != X${NULL_STR} ] ; then
560	IGCM_sys_Get /l liste_file[*] ${RUN_DIR_PATH}/${comp}
561	ExitFlag=false
562	IGCM_debug_Verif_Exit_Post
563	fi
564	done
565	#
566	LEVEL=0
567	#
568	fi
569
570	########################################################################
571	#
572	# DEFINE LOOP PARAMETERS
573	#
574	########################################################################
575
576	DATE_COUNT=${DATE_FIN_JOB_B}
577	DATE_FIN_JOB_B_LOOP=${DATE_FIN_JOB_B}
578
579	( [ -n "${LEVEL}" ] && [ ${LEVEL} -eq 0 ] ) && NBRE_FILE_TOT=0 \|\| NBRE_FILE_TOT=1
580
581	if [ ${DoJob} = true ] ; then
582	while [ ${DATE_COUNT} -lt ${PeriodDateEnd} ] ; do
583	(( NBRE_FILE_TOT = NBRE_FILE_TOT + 1 ))
584	Length=$( create_ts_next_date ${DATE_COUNT} )
585	DATE_COUNT=$( IGCM_date_AddDaysToGregorianDate ${DATE_COUNT} ${Length} )
586	done
587	else
588	NBRE_FILE_TOT=0
589	fi
590
591	if [ ${NBRE_FILE_TOT} -le ${UNIX_MAX_LIMIT} ] ; then
592	NBRE_TOUR=1
593	elif [ ${NBRE_FILE_TOT}%${UNIX_MAX_LIMIT} -eq 0 ] ; then
594	let NBRE_TOUR=${NBRE_FILE_TOT}/${UNIX_MAX_LIMIT}
595	else
596	let NBRE_TOUR=${NBRE_FILE_TOT}/${UNIX_MAX_LIMIT}+1
597	fi
598
599	CURRENT_LOOP=1
600	# THIS LOOP GIVES PARAMETERS FOR THE DATE'S LOOP ie : DATE_COURANTE, DATE_FIN.
601	# WE HAVE 3 LEVELS
602	# - LEVEL 1 : JUST ONE LOOP TO COMPLETE
603	# - LEVEL 2 : WE ARE IN THE "DEEP" LOOP
604	# - LEVEL 3 : WE ARE IN THE LAST LOOP
605
606	while [ ${CURRENT_LOOP} -le ${NBRE_TOUR} ] ; do
607
608	if [ ${NBRE_FILE_TOT} -le ${UNIX_MAX_LIMIT} ] ; then
609	# - LEVEL 1 : JUST ONE LOOP TO COMPLETE
610	NBRE_FILE_LOOP=${NBRE_FILE_TOT}
611
612	if [ ${FIRST_PASS} = TRUE ] ; then
613	DATE_COURANTE=${DATE_FIN_JOB_B_LOOP}
614	else
615	Length=$( create_ts_next_date ${DATE_FIN_JOB_B_LOOP} )
616	DATE_COURANTE=$( IGCM_date_AddDaysToGregorianDate ${DATE_FIN_JOB_B_LOOP} ${Length} )
617	fi
618
619	DATE_FIN=${PeriodDateEnd}
620	DATE_FIN_JOB_B_LOOP_PREC=${DATE_FIN_JOB_B_LOOP}
621
622	[ -n "${DATE_LOOP}" ] && DATE_FIN_JOB_B_LOOP=${DATE_LOOP}
623	[ -n "${LEVEL}" ] && FLAG_B="TRUE" \|\| FLAG_B="FALSE"
624
625	LEVEL=1
626	DEPOT="TRUE"
627	elif [ ${CURRENT_LOOP} -ne ${NBRE_TOUR} ] ; then
628	# - LEVEL 2 : WE ARE IN THE "DEEP" LOOP
629	NBRE_FILE_LOOP=${UNIX_MAX_LIMIT}
630
631	if [ ! ${FIRST_PASS} = TRUE ] && [ ${CURRENT_LOOP} -eq 1 ] ; then
632	Length=$( create_ts_next_date ${DATE_FIN_JOB_B_LOOP} )
633	DATE_COURANTE=$( IGCM_date_AddDaysToGregorianDate ${DATE_FIN_JOB_B_LOOP} ${Length} )
634	elif [ -z "${DATE_COURANTE}" ] ; then
635	DATE_COURANTE=${DATE_FIN_JOB_B}
636	else
637	Length=$( create_ts_next_date ${DATE_FIN} )
638	DATE_COURANTE=$( IGCM_date_AddDaysToGregorianDate ${DATE_FIN} ${Length} )
639	fi
640
641	(( TotLength = 0 ))
642	COMPTEUR=1
643	DATE_LOOP=${DATE_COURANTE}
644
645	while [ ${COMPTEUR} -lt ${NBRE_FILE_LOOP} ] ; do
646	#
647	Length=$( create_ts_next_date ${DATE_LOOP} )
648	DATE_LOOP=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} ${Length} )
649	(( TotLength = TotLength + Length ))
650	(( COMPTEUR = COMPTEUR + 1 ))
651	#
652	done # while [ ${COMPTEUR} -le ${NBRE_FILE_LOOP} ] done
653	DATE_FIN=$( IGCM_date_AddDaysToGregorianDate ${DATE_COURANTE} ${TotLength} )
654
655	[ -n "${DATE_LOOP}" ] && DATE_FIN_JOB_B_LOOP_PREC=${DATE_FIN_JOB_B_LOOP} ; DATE_FIN_JOB_B_LOOP=${DATE_LOOP}
656	[ -n "${LEVEL}" ] && [ ${LEVEL} -eq 0 ] && FLAG_B="TRUE" \|\| [ ${CURRENT_LOOP} -gt 1 ] && FLAG_B="TRUE" \|\| FLAG_B="FALSE"
657	DEPOT="FALSE"
658	LEVEL=2
659	else
660	# - LEVEL 3 : WE ARE IN THE LAST LOOP
661	NBRE_FILE_LOOP=$(( ${NBRE_FILE_TOT} % ( ${UNIX_MAX_LIMIT} * ( ${CURRENT_LOOP} - 1 ) ) ))
662	[ ${NBRE_FILE_LOOP} -eq 0 ] && NBRE_FILE_LOOP=${UNIX_MAX_LIMIT}
663
664	Length=$( create_ts_next_date ${DATE_FIN} )
665	DATE_COURANTE=$(IGCM_date_AddDaysToGregorianDate ${DATE_FIN} ${Length} )
666	DATE_FIN=${PeriodDateEnd}
667	[ -n "${DATE_LOOP}" ] && DATE_FIN_JOB_B_LOOP_PREC=${DATE_FIN_JOB_B_LOOP} ; DATE_FIN_JOB_B_LOOP=${DATE_LOOP}
668
669	LEVEL=3
670	DEPOT="TRUE"
671	FLAG_B="TRUE"
672	fi
673
674	# THEN FOR EACH FILE TYPE WE LOOP BETWEEN DATE_COURANTE AND DATE_FIN.
675	for comp in ${config_ListOfComponents[*]} ; do
676	#
677	IGCM_sys_Mkdir ${RUN_DIR_PATH}/${comp}
678	#
679	IGCM_sys_Cd ${RUN_DIR_PATH}/${comp}
680	#
681	eval R_OUT_${comp}=${R_SAVE}/${comp}
682	#
683	i=0
684	#
685	for file in $( eval echo \${LISTE_FILE_${comp}[*]} ); do
686	#
687	# Determine in which output can we find file
688	# IE : Output/MO or Output/DA or ...
689	#
690	FlagDir=$( echo ${file} \| awk -F "_" '{print $1}' )
691	case ${FlagDir} in
692	*Y) TS_Dir=TS_YE ; FreqDir=YE ;;
693	*M) TS_Dir=TS_MO ; FreqDir=MO ;;
694	*D) TS_Dir=TS_DA ; FreqDir=DA ;;
695	HF) TS_Dir=TS_HF ; FreqDir=HF ;;
696	INS) TS_Dir=TS_INS ; FreqDir=INS ;;
697	*)
698	IGCM_debug_Print 1 "Error in FlagDir=${FlagDir} for file ${file}."
699	IGCM_sys_Exit "Job create_ts" ;;
700	esac
701
702	# Initialize array
703	#set -A liste_file
704	#set -A liste_file_tmp
705	unset liste_file
706	unset liste_file_tmp
707	eval file_path=\${R_OUT_${comp}}/Output/${FreqDir}/
708
709	COMPTEUR=1
710	DATE_LOOP=${DATE_COURANTE}
711
712	while [ ${COMPTEUR} -le ${NBRE_FILE_LOOP} ] ; do
713	#
714	Length1=$( create_ts_begin_date ${DATE_LOOP} end)
715	DATE_TAB=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} $(( 1 - Length1 )) )_${DATE_LOOP}
716
717	Length2=$( create_ts_next_date ${DATE_LOOP} end)
718	DATE_LOOP=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} ${Length2} )
719
720	#set +A liste_file ${liste_file[*]} ${file_path}${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc
721	#set +A liste_file_tmp ${liste_file_tmp[*]} ${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc
722	liste_file[${#liste_file[*]}]=${file_path}${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc
723	liste_file_tmp[${#liste_file_tmp[*]}]=${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc
724	(( COMPTEUR = COMPTEUR + 1 ))
725	#
726	done # while [ ${COMPTEUR} -le ${NBRE_FILE_LOOP} ] done
727	DATE_LOOP=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} -${Length2} )
728
729	# Get selected files
730	IGCM_sys_Get /l liste_file[*] ${RUN_DIR_PATH}/${comp}
731	ExitFlag=false
732	IGCM_debug_Verif_Exit_Post
733
734	# Apply IGCM_Patch if needed
735	if [ ! X${RebuildFrequency} = Xtrue ] ; then
736	if [ X$( eval echo \${LISTE_PATCHES_${file}[0]} ) != X${NULL_STR} ]; then
737	for file_t in $( eval echo ${liste_file_tmp[*]} ); do
738	for Patch in $( eval echo \${LISTE_PATCHES_${file}[*]} ); do
739	IGCM_${Patch} ${file_t}
740	done
741	done
742	fi
743	fi
744
745	# WE CAN CONCATENATE FILES AT THIS POINT
746	[ "${FLAG_B}" = "TRUE" ] && DATE_BUILD_B=${DateBegin}_${DATE_FIN_JOB_B_LOOP_PREC}
747	Length=$( create_ts_begin_date ${DATE_COURANTE} end)
748	DATE_BUILD1=$( IGCM_date_AddDaysToGregorianDate ${DATE_COURANTE} -$(( Length - 1 )) )
749	DATE_BUILD=${DATE_BUILD1}_${DATE_FIN}
750	DATE_BUILD_END=${DateBegin}_${DATE_FIN}
751
752	liste_coord=" "
753	for axis in $( eval echo \${LISTE_AXIS_${file}[*]} ); do
754	liste_coord=${liste_coord}${axis}","
755	done
756
757	# Time axis must be the last dimension
758	time_axis=${axis}
759
760	# SWITCH BETWEEN MODEL TO FIT PARTICULAR CASE
761	for var in $( eval echo \${LISTE_VARS_${file}[*]} ) ; do
762	#
763	IGCM_sys_ncrcat --hst -v ${liste_coord}${var} ${liste_file_tmp[*]} ${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
764	#
765	# # Change time axis name to be CF compliant
766	# if [ ${time_axis} != time_counter ]; then
767	# IGCM_sys_ncrename -O -v ${time_axis},time_counter ${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
768	# fi
769	#
770	if [ "${FLAG_B}" = "TRUE" ] ; then
771	# WE CONCATENATE WITH EXISTING FILES
772	file1=${config_UserChoices_JobName}_${DATE_BUILD_B}_${FlagDir}_${var}.nc
773	file2=${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
774	file_out=${config_UserChoices_JobName}_${DATE_BUILD_END}_${FlagDir}_${var}.nc
775	IGCM_sys_ncrcat --hst ${file1} ${file2} ${file_out}
776	#
777	IGCM_sys_Rm ${file1} ${file2}
778	fi
779	#
780	if [ "${DEPOT}" = "TRUE" ] ; then
781	# WE PUT FINAL FILE ON FILE SYSTEM
782	[ "${FLAG_B}" = "FALSE" ] && file_out=${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
783	#${ncatted} -a modulo,time_counter,c,c," " ${file_out}
784	file_before=${config_UserChoices_JobName}_${DateBegin}_${DATE_FIN_JOB_B}_${FlagDir}_${var}.nc
785	eval IGCM_sys_Put_Out ${file_out} \${R_OUT_${comp}}/Analyse/${TS_Dir}/${file_out}
786	[ "${FLAG_B}" = "TRUE" ] && eval MENAGE \${R_OUT_${comp}}/Analyse/${TS_Dir} ${file_before} ${file_out}
787	IGCM_sys_Rm ${file_out}
788	#
789	fi
790	#
791	ExitFlag=false
792	IGCM_debug_Verif_Exit_Post
793	done # for var in ${liste_var} ; do
794	IGCM_sys_Rm ${liste_file_tmp[*]}
795	done # for file in ${liste_file_brut} do
796	done # for comp in ${config_ListOfComponents[*]} ; do
797	ExitFlag=false
798	IGCM_debug_Verif_Exit_Post
799	# GO AHEAD IN DEEPEST LOOP
800	(( CURRENT_LOOP = CURRENT_LOOP + 1 ))
801	done # while [ ${CURRENT_LOOP} -le ${NBRE_TOUR} ] do
802	# FINISH
803
804	# DODS copy
805	for comp in ${config_ListOfComponents[*]} ; do
806	for TS_Dir in TS_YE TS_MO TS_DA TS_HF TS_INS ; do
807	IGCM_sys_TestDirArchive ${R_SAVE}/${comp}/Analyse/${TS_Dir}
808	[ $? = 0 ] && IGCM_sys_Put_Dods ${comp}/Analyse/${TS_Dir}
809	done
810	done
811
812	export POST_DIR; export SUBMIT_DIR; export libIGCM; export R_INIT; export R_BC; export StandAlone
813	export RESOL_ATM; export RESOL_OCE ; export RESOL_ICE ; export RESOL_MBG ; export RESOL_SRF ; export RESOL_SBG
814
815	listVarEnv="POST_DIR,SUBMIT_DIR,libIGCM,R_INIT,R_BC,StandAlone,RESOL_ATM,RESOL_OCE,RESOL_ICE,RESOL_MBG,RESOL_SRF,RESOL_SBG"
816	export listVarEnv
817
818	#D-
819	# --------------------------------------------------------------------
820	#D- Test if all was right
821	# --------------------------------------------------------------------
822	IGCM_debug_Verif_Exit_Post
823	# --------------------------------------------------------------------
824	#D- Submit MONITORING
825	# --------------------------------------------------------------------
826	( [ X${TsTask} = X2D ] \|\| [ X${TsTask} = X ] ) && IGCM_sys_QsubPost monitoring
827
828	#D-
829	# --------------------------------------------------------------------
830	#D- Update the run.card
831	# --------------------------------------------------------------------
832	if [ ${StandAlone} != true ] ; then
833	# Put in run.card end period of time series
834	IGCM_sys_RshMaster IGCM_card_WriteOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesCompleted "${DATE_FIN}"
835	# We have finish the job !
836	IGCM_sys_RshMaster IGCM_card_WriteOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesRunning "n"
837	fi
838
839	# Clean RUN_DIR_PATH (necessary for cesium)
840	IGCM_sys_RmRunDir -Rf ${RUN_DIR_PATH}
841
842	date

Note: See TracBrowser for help on using the repository browser.

Context Navigation

source: trunk/libIGCM/AA_create_ts @ 218

Download in other formats: