Context Navigation

source: trunk/libIGCM/AA_create_ts @ 196

Last change on this file since 196 was 196, checked in by sdipsl, 15 years ago
Some more step towards post processing from cesium Wait from CCRT actions (cesium node can not ssh mercure), necessary because cesium do not see mercure workdir and thus do not see card Do not use this version on sx9, post-process will fail
Property licence set to The following licence information concerns ONLY the libIGCM tools ================================================================== Copyright © Centre National de la Recherche Scientifique CNRS Commissariat à l'Énergie Atomique CEA libIGCM : Library for Portable Models Computation of IGCM Group. IGCM Group is the french IPSL Global Climate Model Group. This library is a set of shell scripts and functions whose purpose is the management of the initialization, the launch, the transfer of output files, the post-processing and the monitoring of datas produce by any numerical program on any plateforme. This software is governed by the CeCILL license under French law and abiding by the rules of distribution of free software. You can use, modify and/ or redistribute the software under the terms of the CeCILL license as circulated by CEA, CNRS and INRIA at the following URL "http://www.cecill.info". As a counterpart to the access to the source code and rights to copy, modify and redistribute granted by the license, users are provided only with a limited warranty and the software's author, the holder of the economic rights, and the successive licensors have only limited liability. In this respect, the user's attention is drawn to the risks associated with loading, using, modifying and/or developing or reproducing the software by the user in light of its specific status of free software, that may mean that it is complicated to manipulate, and that also therefore means that it is reserved for developers and experienced professionals having in-depth computer knowledge. Users are therefore encouraged to load and test the software's suitability as regards their requirements in conditions enabling the security of their systems and/or data to be ensured and, more generally, to use and operate it in the same conditions as regards security. The fact that you are presently reading this means that you have had knowledge of the CeCILL license and that you accept its terms. Property svn:keywords set to `Date Author Revision`
File size: 29.4 KB

Line
1	#-Q- platine #!/usr/bin/ksh
2	#-Q- platine ###################
3	#-Q- platine ## PLATINE CEA ##
4	#-Q- platine ###################
5	#-Q- platine #BSUB -J TS # Nom du job
6	#-Q- platine #BSUB -N # message a la fin du job
7	#-Q- platine #BSUB -n 1 # reservation des processeurs pour le job
8	#-Q- platine #BSUB -W 1:00 # Limite temps
9	#-Q- platine #BSUB -q post # Passage en queue post
10	#-Q- sx8brodie #!/bin/ksh
11	#-Q- sx8brodie #######################
12	#-Q- sx8brodie ## SX8BRODIE IDRIS ##
13	#-Q- sx8brodie #######################
14	#-Q- sx8brodie # Temps Elapsed max. d'une requete hh:mm:ss
15	#-Q- sx8brodie # @ wall_clock_limit = 10:00:00
16	#-Q- sx8brodie # Nom du travail LoadLeveler
17	#-Q- sx8brodie # @ job_name = TS
18	#-Q- sx8brodie # Fichier de sortie standard du travail
19	#-Q- sx8brodie # @ output = $(job_name).$(jobid)
20	#-Q- sx8brodie # Fichier de sortie d'erreur du travail
21	#-Q- sx8brodie # @ error = $(job_name).$(jobid)
22	#-Q- sx8brodie # pour recevoir un mail en cas de depassement du temps Elapsed (ou autre pb.)
23	#-Q- sx8brodie # @ notification = error
24	#-Q- sx8brodie # @ environment = $DEBUG_debug ; $MODIPSL ; $libIGCM ; $libIGCM_SX ; $SUBMIT_DIR ; $REBUILD_DIR ; $RebuildFromArchive ; $POST_DIR ; $MASTER ; $RebuildFrequency ; $PeriodDateBegin ; $PeriodDateEnd ; $NbRebuildDir ; $StandAlone ; $CompletedFlag ; $TsTask ; $CompToRead ; $FlagToRead ; $RESOL_ATM ; $RESOL_OCE ; $RESOL_ICE ; $RESOL_MBG ; $RESOL_SRF ; $RESOL_BIOS ; $MASTER
25	#-Q- aix6 #!/bin/ksh
26	#-Q- aix6 #######################
27	#-Q- aix6 ## VARGAS IDRIS ##
28	#-Q- aix6 #######################
29	#-Q- aix6 # Temps Elapsed max. d'une requete hh:mm:ss
30	#-Q- aix6 # @ wall_clock_limit = 10:00:00
31	#-Q- aix6 # Nom du travail LoadLeveler
32	#-Q- aix6 # @ job_name = TS
33	#-Q- aix6 # Fichier de sortie standard du travail
34	#-Q- aix6 # @ output = $(job_name).$(jobid)
35	#-Q- aix6 # Fichier de sortie d'erreur du travail
36	#-Q- aix6 # @ error = $(job_name).$(jobid)
37	#-Q- aix6 # pour recevoir un mail en cas de depassement du temps Elapsed (ou autre pb.)
38	#-Q- aix6 # @ notification = error
39	#-Q- aix6 # @ environment = $DEBUG_debug ; $MODIPSL ; $libIGCM ; $libIGCM_SX ; $SUBMIT_DIR ; $REBUILD_DIR ; $RebuildFromArchive ; $POST_DIR ; $MASTER ; $RebuildFrequency ; $DateBegin ; $PeriodDateBegin ; $PeriodDateEnd ; $NbRebuildDir ; $StandAlone ; $CompletedFlag ; $TsTask ; $CompToRead ; $FlagToRead ; $RESOL_ATM ; $RESOL_OCE ; $RESOL_ICE ; $RESOL_MBG ; $RESOL_SRF ; $RESOL_BIOS ; $MASTER
40	#-Q- aix6 # @ queue
41	#-Q- sx8brodie # @ queue
42	#-Q- sx8mercure #!/bin/ksh
43	#-Q- sx8mercure ######################
44	#-Q- sx8mercure ## SX8MERCURE CEA ##
45	#-Q- sx8mercure ######################
46	#-Q- sx8mercure #PBS -N TS # Nom du job
47	#-Q- sx8mercure #PBS -j o # regroupement des stdout et stderr
48	#-Q- sx8mercure #PBS -S /usr/bin/ksh # shell de soumission
49	#-Q- sx8mercure #PBS -l memsz_job=1gb # Limite memoire a 1 Go
50	#-Q- sx8mercure #PBS -l cputim_job=24:00:00 # Limite temps a 2 heures
51	#-Q- sx8mercure #PBS -q scalaire
52	#-Q- sx9mercure #!/bin/ksh
53	#-Q- sx9mercure ######################
54	#-Q- sx9mercure ## SX9MERCURE CEA ##
55	#-Q- sx9mercure ######################
56	#-Q- sx9mercure #MSUB -r TS # Nom du job
57	#-Q- sx9mercure #MSUB -N 1 # Reservation du noeud
58	#-Q- sx9mercure #MSUB -n 1 # Reservation du processus
59	#-Q- sx9mercure #MSUB -T 86400 # Limite de temps elapsed du job
60	#-Q- sx9mercure #MSUB -E "-j o"
61	#-Q- sx9mercure #MSUB -E "-S /bin/ksh"
62	#-Q- sx9mercure ##MSUB -e nco.out # Sortie standard
63	#-Q- sx9mercure ##MSUB -o nco.out # Sortie standard
64	#-Q- default #!/bin/ksh
65	#-Q- default ##################
66	#-Q- default ## DEFAULT HOST ##
67	#-Q- default ##################
68
69	# $Date$
70	# $Author$
71	# $Revision$
72	# IPSL (2006)
73	# This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
74
75	#set -eu
76	#set -vx
77
78	date
79
80	#-Q- sx8brodie export OMP_NUM_THREADS=1
81	#-Q- aix6 export OMP_NUM_THREADS=1
82
83	########################################################################
84
85	#D- Flag to determine if this job in a standalone mode
86	#D- Default : value from AA_job if any
87	StandAlone=${StandAlone:=true}
88
89	#D- Flag to determine allready produced time series.
90	#D- Default : Commented. Uncomment for StandAlone case if you want to continue a time series
91	#CompletedFlag=
92
93	#D- Flag to determine job's output directory
94	#D- Default : value from libIGCM_post.ksh if any
95	POST_DIR=${POST_DIR:=${PBS_O_WORKDIR}}
96
97	#D- Increased verbosity (1, 2, 3)
98	#D- Default : value from AA_job if any
99	Verbosity=${Verbosity:=3}
100
101	#D- Low level debug : to bypass lib test checks and stack construction
102	#D- Default : value from AA_job if any
103	DEBUG_debug=${DEBUG_debug:=false}
104
105	#D- Low level debug : to bypass lib test checks and stack construction
106	#D- Default : value from AA_job if any
107	libIGCM=${libIGCM:=/path/to/your/libIGCM}
108
109	#D- TEMPORARY Flag to determine atmospheric resolution
110	#D- Default : value from atmospheric driver if any
111	RESOL_ATM=ALL
112
113	#D- TEMPORARY Flag to determine ocean resolution
114	#D- Default : value from ocean driver if any
115	RESOL_OCE=${RESOL_OCE:=ORCA2}
116
117	#D- TEMPORARY Flag to determine ice resolution
118	#D- Default : value from ice driver if any
119	RESOL_ICE=${RESOL_ICE:=ORCA2}
120
121	#D- TEMPORARY Flag to determine marine biogeochemistry resolution
122	#D- Default : value from ice driver if any
123	RESOL_MBG=${RESOL_MBG:=ORCA2}
124
125	#D- Flag to determine surface resolution
126	#D- Default : value from surface driver if any
127	RESOL_SRF=ALL
128
129	#D- Flag to determine surface resolution
130	#D- Default : value from surface driver if any
131	RESOL_BIOS=ALL
132
133	########################################################################
134
135	. ${libIGCM}/libIGCM_debug/libIGCM_debug.ksh
136	( ${DEBUG_debug} ) && IGCM_debug_Check
137	. ${libIGCM}/libIGCM_card/libIGCM_card.ksh
138	( ${DEBUG_debug} ) && IGCM_card_Check
139	. ${libIGCM}/libIGCM_date/libIGCM_date.ksh
140	( ${DEBUG_debug} ) && IGCM_date_Check
141	#-------
142	. ${libIGCM}/libIGCM_sys/libIGCM_sys.ksh
143
144	########################################################################
145
146	#set -vx
147
148	# Useful cleaning function
149	MENAGE ()
150	{
151	[ -f $3 ] && ( IGCM_sys_RshArchive rm -f $1/$2 ;)
152	}
153
154	########################################################################
155
156	if [ ${DEBUG_debug} = true ] ; then
157	IGCM_sys_MkdirWork ${RUN_DIR_PATH}
158	echo "RUN_DIR_PATH ${RUN_DIR_PATH} ok."
159	fi
160	IGCM_sys_Cd ${RUN_DIR_PATH}
161
162	if [ ${StandAlone} = true ] ; then
163	CARD_DIR=${SUBMIT_DIR}
164	else
165	CARD_DIR=${RUN_DIR_PATH}/$( basename ${SUBMIT_DIR} )
166	IGCM_sys_Get_Master ${SUBMIT_DIR} ${RUN_DIR_PATH}
167	fi
168
169	#
170	# First of all
171	#
172	IGCM_card_DefineArrayFromSection ${CARD_DIR}/config.card UserChoices
173	typeset option
174	for option in ${config_UserChoices[*]} ; do
175	IGCM_card_DefineVariableFromOption ${CARD_DIR}/config.card UserChoices ${option}
176	done
177	#
178	echo
179	IGCM_debug_Print 1 "DefineArrayFromOption : config_UserChoices"
180	IGCM_debug_PrintVariables 3 config_UserChoices_JobName
181	IGCM_debug_PrintVariables 3 config_UserChoices_LongName
182	IGCM_debug_PrintVariables 3 config_UserChoices_TagName
183	IGCM_debug_PrintVariables 3 config_UserChoices_CalendarType
184	IGCM_debug_PrintVariables 3 config_UserChoices_DateBegin
185	IGCM_debug_PrintVariables 3 config_UserChoices_DateEnd
186	IGCM_debug_PrintVariables 3 config_UserChoices_PeriodLength
187	echo
188
189	#==================================
190	R_SAVE=${R_OUT}/${config_UserChoices_TagName}/${config_UserChoices_JobName}
191	#
192	# Determine component to take care of depending on kind of task create_ts will perform
193	#
194	( [ X${TsTask} = XChunck2D ] \|\| [ X${TsTask} = XChunck3D ] ) && set -A config_ListOfComponents ${CompToRead} \|\| IGCM_card_DefineArrayFromSection ${CARD_DIR}/config.card ListOfComponents
195	#
196	# Determine period for time series. Default : value from AA_job or from launch_TS if any
197	#
198	DateBegin=${DateBegin:=$( IGCM_date_ConvertFormatToGregorian ${config_UserChoices_DateBegin} )}
199	#
200	PeriodDateEnd=${PeriodDateEnd:=$( IGCM_date_ConvertFormatToGregorian ${config_UserChoices_DateEnd} )}
201	#
202	# Determine Dimension of the produced time series : 2D or 3D
203	if ( [ X${TsTask} = XChunck2D ] \|\| [ X${TsTask} = X2D ] ) ; then
204	Dimension=2D
205	elif ( [ X${TsTask} = XChunck3D ] \|\| [ X${TsTask} = X3D ] ) ; then
206	Dimension=3D
207	else
208	Dimension=""
209	fi
210
211
212	########################################################################
213	#
214	# Depending on requested time series fill following variables :
215	#
216	# - LISTE_FILE_${comp}[*]
217	# - LISTE_VARS_${file}[*]
218	# - LISTE_AXIS_${file}[*]
219	# - LISTE_PATCH_${file}[*]
220	########################################################################
221
222	#
223	# For each selected component determine which files need post-processing
224	#
225	DoJob=false
226	for comp in ${config_ListOfComponents[*]} ; do
227	#
228	ActiveComp=false
229	#
230	IGCM_card_DefineArrayFromOption ${CARD_DIR}/config.card ListOfComponents ${comp}
231	eval compname=\${config_ListOfComponents_${comp}[0]} > /dev/null 2>&1
232	#
233	card=${CARD_DIR}/COMP/${compname}.card
234
235	IGCM_card_DefineArrayFromOption ${card} OutputFiles List
236	#
237	ListFilesName=${compname}_OutputFiles_List
238	eval FileName0=\${${ListFilesName}[0]} > /dev/null 2>&1
239	#
240	if [ X${FileName0} != X${NULL_STR} ] ; then
241	#
242	#IGCM_debug_Print 1 "Component : ${compname}"
243	#
244	# INITIALISATION
245	#
246	typeset i
247	if ( [ X${TsTask} = XChunck2D ] \|\| [ X${TsTask} = XChunck3D ] ) ; then
248	((NbFiles=${FlagToRead}))
249	((i=${NbFiles}-2))
250	else
251	eval NbFiles=\${#${ListFilesName}[@]} > /dev/null 2>&1
252	i=0
253	fi
254	#
255	until [ $i -ge $NbFiles ]; do
256	#
257	(( i_ = i+2 ))
258	eval flag_post_=\${${ListFilesName}[$i_]} > /dev/null 2>&1
259	eval flag_post=${flag_post_}
260	#
261	if [ X${flag_post} != XNONE ] ; then
262	#
263	# For each comp and each file who needs post-processing
264	# determine which variables need time series.
265	#
266	FILE=$( echo ${flag_post} \| awk "-FPost_" '{print $2}' )
267	IGCM_card_DefineArrayFromOption ${card} Post_${FILE} TimeSeriesVars${Dimension}
268	#
269	if [ $( eval echo \${${compname}_Post_${FILE}_TimeSeriesVars${Dimension}} ) = ${NULL_STR} ] ; then
270	#
271	IGCM_debug_Print 3 "Nothing to do for ${FILE} according to ${card}"
272	(( i=i+3 ))
273	continue
274	#
275	fi
276	#
277	# If we are not a Chunck type task, skip files/variables handled by chunck task
278	#
279	if ( [ ! X${TsTask} = XChunck${Dimension} ] && [ ! X${TsTask} = X ] && [ ! XChunck${Dimension} = XChunck ] ) ; then
280	IGCM_card_DefineVariableFromOption ${card} Post_${FILE} ChunckJob${Dimension}
281	if [ ! $( eval echo \${${compname}_Post_${FILE}_ChunckJob${Dimension}} ) = NONE ] ; then
282	IGCM_debug_Print 3 "Nothing to do for ${FILE} TimeSeriesVars${Dimension} according to ${card}"
283	IGCM_debug_Print 3 "Let another chunck task do the job $( eval echo \${${compname}_Post_${FILE}_ChunckJob${Dimension}} )"
284	(( i=i+3 ))
285	continue
286	fi
287	fi
288	#
289	# We need coordinates axis to properly fill produced time series
290	#
291	IGCM_card_DefineArrayFromOption ${card} Post_${FILE} GatherWithInternal
292	#
293	if [ $( eval echo \${${compname}_Post_${FILE}_GatherWithInternal} ) = ${NULL_STR} ] ; then
294	#
295	IGCM_debug_Print 1 "ERROR !! We miss coordinate axis for ${FILE} according to ${card}"
296	IGCM_debug_Print 1 "So no time series will be produced for ${FILE} !!"
297	(( i=i+3 ))
298	continue
299	#
300	fi
301	#
302	# We need LIST of variables not allready produced (useful for standalone mode)
303	#
304	FlagDir=$( echo ${FILE} \| awk -F "_" '{print $1}' )
305	case ${FlagDir} in
306	*Y) TS_Dir=TS_YE ;;
307	*M) TS_Dir=TS_MO ;;
308	*D) TS_Dir=TS_DA ;;
309	HF) TS_Dir=TS_HF ;;
310	INS) TS_Dir=TS_INS ;;
311	esac
312	#
313	count=0
314	for var in $( eval echo \${${compname}_Post_${FILE}_TimeSeriesVars${Dimension}[*]} ) ; do
315	#
316	DIRECTORY=${R_SAVE}/${comp}/Analyse/${TS_Dir}
317	TestedFile=${config_UserChoices_JobName}_${DateBegin}_${PeriodDateEnd}_${FlagDir}_${var}.nc
318	#
319	IGCM_sys_TestFileArchive ${DIRECTORY}/${TestedFile}
320	if [ ! $? = 0 ] ; then
321	eval set +A LISTE_VARS_${FILE} \${LISTE_VARS_${FILE}[*]} ${var}
322	ActiveComp=true
323	(( count = count + 1 ))
324	fi
325	done
326	#
327	if [ ${count} -gt 0 ] ; then
328	#
329	eval set +A LISTE_FILE_${comp} \${LISTE_FILE_${comp}[*]} ${FILE}
330	eval set +A LISTE_AXIS_${FILE} \${${compname}_Post_${FILE}_GatherWithInternal[*]}
331	#
332	if [ ! X${RebuildFrequency} = Xtrue ] ; then
333	IGCM_card_DefineArrayFromOption ${card} Post_${FILE} Patches
334	eval set +A LISTE_PATCHES_${FILE} \${${compname}_Post_${FILE}_Patches[*]}
335	if [ X$( eval echo \${LISTE_PATCHES_${FILE}[0]} ) != X${NULL_STR} ]; then
336	for Patch in $( eval echo \${LISTE_PATCHES_${FILE}[*]} ); do
337	if [ Xload${Patch} != X ] ; then
338	. ${libIGCM}/libIGCM_post/IGCM_${Patch}.ksh
339	eval load${Patch}=loaded
340	fi
341	done
342	fi
343	fi
344	fi
345	fi
346	(( i=i+3 ))
347	done
348	fi
349	#
350	if [ ${ActiveComp} = true ] ; then
351	set +A ActiveListOfComponents ${ActiveListOfComponents[*]} ${comp}
352	fi
353	done
354
355	set -A config_ListOfComponents ${ActiveListOfComponents[*]}
356	[ ! -z "${ActiveListOfComponents}" ] && DoJob=true
357
358	########################################################################
359	#
360	# DEFINE DATE FUNCTIONS
361	#
362	########################################################################
363
364	# Give the end of the next $1 period from $2 date
365	function create_ts_next_date {
366	IGCM_debug_PushStack "create_ts_next_date" $@
367	typeset Length
368	case ${config_UserChoices_PeriodLength} in
369	Y\|y)
370	PeriodLengthInYears=$( echo ${config_UserChoices_PeriodLength} \| awk -F '[yY]' '{print $1}' )
371	IGCM_date_GetYearMonth ${1} year month
372	(( Length=0 ))
373	(( i=0 ))
374	until [ $i -ge $PeriodLengthInYears ] ; do
375	(( Length = Length + $( IGCM_date_DaysInYear $(( year + i + 1 )) ) ))
376	(( i=i+1 ))
377	done
378	;;
379	M\|m)
380	PeriodLengthInMonths=$( echo ${config_UserChoices_PeriodLength} \| awk -F '[mM]' '{print $1}' )
381	IGCM_date_GetYearMonth ${1} year month
382	(( year0 = year ))
383	if [ $(( month + 1 )) -lt 13 ] ; then
384	month0=$(( month + 1 ))
385	else
386	month0=$(( month + 1 - 12 ))
387	(( year = year0 + 1 ))
388	fi
389	(( Length=0 ))
390	(( i=0 ))
391	until [ $i -ge $PeriodLengthInMonths ] ; do
392	if [ $(( month0 + i )) -lt 13 ] ; then
393	(( Length = Length + $( IGCM_date_DaysInMonth $year $(( month0 + i )) ) ))
394	else
395	(( year = year0 + 1 ))
396	(( Length = Length + $( IGCM_date_DaysInMonth $year $(( month0 + i - 12 )) ) ))
397	fi
398	(( i=i+1 ))
399	done
400	;;
401	D\|d)
402	Length=$( echo ${config_UserChoices_PeriodLength} \| sed -e "s/[dD]//" ) ;;
403	*)
404	IGCM_debug_Exit "create_ts " ${config_UserChoices_PeriodLength} " invalid PeriodLength : choose in 1Y, M, D."
405	IGCM_debug_Verif_Exit_Post ;;
406	esac
407	echo ${Length}
408
409	IGCM_debug_PopStack "create_ts_next_date"
410	}
411
412	# Give the end of the actual $1 period from $2 date
413	function create_ts_begin_date {
414	IGCM_debug_PushStack "create_ts_begin_date" $@
415	typeset Length
416	case ${config_UserChoices_PeriodLength} in
417	Y\|y)
418	PeriodLengthInYears=$( echo ${config_UserChoices_PeriodLength} \| awk -F '[yY]' '{print $1}' )
419	IGCM_date_GetYearMonth ${1} year month
420	if [ X${2} = Xend ] ; then
421	(( year = year - PeriodLengthInYears + 1))
422	fi
423	(( Length=0 ))
424	(( i=0 ))
425	until [ $i -ge $PeriodLengthInYears ] ; do
426	(( Length = Length + $( IGCM_date_DaysInYear $(( year + i )) ) ))
427	(( i=i+1 ))
428	done
429	;;
430	M\|m)
431	PeriodLengthInMonths=$( echo ${config_UserChoices_PeriodLength} \| awk -F '[mM]' '{print $1}' )
432	IGCM_date_GetYearMonth ${1} year month
433	if [ X${2} = Xend ] ; then
434	(( month = month - PeriodLengthInMonths + 1 ))
435	fi
436	(( year0 = year ))
437	if [ $month -le 0 ] ; then
438	(( month = month + 12 ))
439	year=$( printf "%04i\n" $(( year - 1 )) )
440	fi
441	month=$( printf "%02i\n" ${month} )
442	(( Length=0 ))
443	(( i=0 ))
444	until [ $i -ge $PeriodLengthInMonths ] ; do
445	if [ $(( month + i )) -lt 13 ] ; then
446	(( Length = Length + $( IGCM_date_DaysInMonth $year $(( month + i )) ) ))
447	else
448	(( year = year0 + 1 ))
449	(( Length = Length + $( IGCM_date_DaysInMonth $year $(( month + i - 12 )) ) ))
450	fi
451	(( i=i+1 ))
452	done
453	;;
454	D\|d)
455	Length=$( echo ${config_UserChoices_PeriodLength} \| sed -e "s/[dD]//" ) ;;
456	*)
457	IGCM_debug_Exit "create_ts " ${config_UserChoices_PeriodLength} " invalid PeriodLength : choose in 1Y, M, D."
458	IGCM_debug_Verif_Exit_Post ;;
459	esac
460	echo ${Length}
461
462	IGCM_debug_PopStack "create_ts_begin_date"
463	}
464
465	########################################################################
466	#
467	# IS THERE SOME ALLREADY PRODUCED TIME SERIES ?
468	# IF SO BRING THEM IN THE WORKING DIRECTORY
469	#
470	# PS : Keep in mind that IGCM_sys_Get here is a weak link :
471	# - especially for IDRiS
472	# - no special protection against "command too long" !
473	# - libIGCM_sys should handle that or modify following code part
474	#
475	########################################################################
476
477	if [ ${StandAlone} != true ] ; then
478	Running_Flag=$( IGCM_sys_RshMaster "IGCM_card_DefineVariableFromOption \
479	${SUBMIT_DIR}/run.card PostProcessing TimeSeriesRunning ; \
480	echo \${run_PostProcessing_TimeSeriesRunning}" )
481
482	#if [ X${Running_Flag} = Xy ] ; then
483	# # Time Series Job allready running
484	# IGCM_debug_Print 1 "Time Series Job allready running exit"
485	# exit
486	#fi
487	# Now run.card know we are running
488
489
490
491
492	IGCM_sys_RshMaster IGCM_card_WriteOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesRunning "y"
493
494
495
496
497
498	# Get information from last execution
499	#CompletedFlag=$( IGCM_sys_RshMaster \
500	# "IGCM_card_DefineVariableFromOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesCompleted ;\
501	# echo \${run_PostProcessing_TimeSeriesCompleted}" )
502	fi
503
504	# Is it the first submission or not ?
505
506	if ( [ X${CompletedFlag} = X ] \|\| [ X${CompletedFlag} = X$( IGCM_date_AddDaysToGregorianDate ${DateBegin} -1 ) ] ) ; then
507	# First Time Series Submission
508	FIRST_PASS=TRUE
509	Length=$( create_ts_begin_date ${DateBegin} begin)
510	DATE_FIN_JOB_B=$( IGCM_date_AddDaysToGregorianDate ${DateBegin} $(( Length - 1 )) )
511	else
512	# Last Completed Time Series
513	FIRST_PASS=FALSE
514	DATE_FIN_JOB_B=${CompletedFlag}
515	date=${DateBegin}_${DATE_FIN_JOB_B}
516	for comp in ${config_ListOfComponents[*]} ; do
517	#
518	IGCM_sys_Mkdir ${RUN_DIR_PATH}/${comp}
519	#
520	IGCM_sys_Cd ${RUN_DIR_PATH}/${comp}
521	#
522	#set -A liste_file
523	unset liste_file
524	#
525	file_path=${R_SAVE}/${comp}/Analyse
526	#
527	i=0
528	file=${NULL_STR}
529	#
530	for file in $( eval echo \${LISTE_FILE_${comp}[*]} ); do
531	#
532	# Determine in which output can we find file
533	# IE : Analyse/TS or Analyse/DA2TS or ...
534	#
535	FlagDir=$( echo ${file} \| awk -F "_" '{print $1}' )
536	case ${FlagDir} in
537	*Y) TS_Dir=TS_YE ;;
538	*M) TS_Dir=TS_MO ;;
539	*D) TS_Dir=TS_DA ;;
540	HF) TS_Dir=TS_HF ;;
541	INS) TS_Dir=TS_INS ;;
542	esac
543	for var in $( eval echo \${LISTE_VARS_${file}[*]} ) ; do
544	#
545	#set +A liste_file ${liste_file[*]} ${file_path}/${TS_Dir}/${config_UserChoices_JobName}_${date}_${FlagDir}_${var}.nc
546	liste_file[${#liste_file[*]}]=${file_path}/${TS_Dir}/${config_UserChoices_JobName}_${date}_${FlagDir}_${var}.nc
547	#
548	done
549	done
550	if [ X${file} != X${NULL_STR} ] ; then
551	IGCM_sys_Get /l liste_file[*] ${RUN_DIR_PATH}/${comp}
552	ExitFlag=false
553	IGCM_debug_Verif_Exit_Post
554	fi
555	done
556	#
557	LEVEL=0
558	#
559	fi
560
561	########################################################################
562	#
563	# DEFINE LOOP PARAMETERS
564	#
565	########################################################################
566
567	DATE_COUNT=${DATE_FIN_JOB_B}
568	DATE_FIN_JOB_B_LOOP=${DATE_FIN_JOB_B}
569
570	( [ -n "${LEVEL}" ] && [ ${LEVEL} -eq 0 ] ) && NBRE_FILE_TOT=0 \|\| NBRE_FILE_TOT=1
571
572	if [ ${DoJob} = true ] ; then
573	while [ ${DATE_COUNT} -lt ${PeriodDateEnd} ] ; do
574	(( NBRE_FILE_TOT = NBRE_FILE_TOT + 1 ))
575	Length=$( create_ts_next_date ${DATE_COUNT} )
576	DATE_COUNT=$( IGCM_date_AddDaysToGregorianDate ${DATE_COUNT} ${Length} )
577	done
578	else
579	NBRE_FILE_TOT=0
580	fi
581
582	if [ ${NBRE_FILE_TOT} -le ${UNIX_MAX_LIMIT} ] ; then
583	NBRE_TOUR=1
584	elif [ ${NBRE_FILE_TOT}%${UNIX_MAX_LIMIT} -eq 0 ] ; then
585	let NBRE_TOUR=${NBRE_FILE_TOT}/${UNIX_MAX_LIMIT}
586	else
587	let NBRE_TOUR=${NBRE_FILE_TOT}/${UNIX_MAX_LIMIT}+1
588	fi
589
590	CURRENT_LOOP=1
591	# THIS LOOP GIVES PARAMETERS FOR THE DATE'S LOOP ie : DATE_COURANTE, DATE_FIN.
592	# WE HAVE 3 LEVELS
593	# - LEVEL 1 : JUST ONE LOOP TO COMPLETE
594	# - LEVEL 2 : WE ARE IN THE "DEEP" LOOP
595	# - LEVEL 3 : WE ARE IN THE LAST LOOP
596
597	while [ ${CURRENT_LOOP} -le ${NBRE_TOUR} ] ; do
598
599	if [ ${NBRE_FILE_TOT} -le ${UNIX_MAX_LIMIT} ] ; then
600	# - LEVEL 1 : JUST ONE LOOP TO COMPLETE
601	NBRE_FILE_LOOP=${NBRE_FILE_TOT}
602
603	if [ ${FIRST_PASS} = TRUE ] ; then
604	DATE_COURANTE=${DATE_FIN_JOB_B_LOOP}
605	else
606	Length=$( create_ts_next_date ${DATE_FIN_JOB_B_LOOP} )
607	DATE_COURANTE=$( IGCM_date_AddDaysToGregorianDate ${DATE_FIN_JOB_B_LOOP} ${Length} )
608	fi
609
610	DATE_FIN=${PeriodDateEnd}
611	DATE_FIN_JOB_B_LOOP_PREC=${DATE_FIN_JOB_B_LOOP}
612
613	[ -n "${DATE_LOOP}" ] && DATE_FIN_JOB_B_LOOP=${DATE_LOOP}
614	[ -n "${LEVEL}" ] && FLAG_B="TRUE" \|\| FLAG_B="FALSE"
615
616	LEVEL=1
617	DEPOT="TRUE"
618	elif [ ${CURRENT_LOOP} -ne ${NBRE_TOUR} ] ; then
619	# - LEVEL 2 : WE ARE IN THE "DEEP" LOOP
620	NBRE_FILE_LOOP=${UNIX_MAX_LIMIT}
621
622	if [ ! ${FIRST_PASS} = TRUE ] && [ ${CURRENT_LOOP} -eq 1 ] ; then
623	Length=$( create_ts_next_date ${DATE_FIN_JOB_B_LOOP} )
624	DATE_COURANTE=$( IGCM_date_AddDaysToGregorianDate ${DATE_FIN_JOB_B_LOOP} ${Length} )
625	elif [ -z "${DATE_COURANTE}" ] ; then
626	DATE_COURANTE=${DATE_FIN_JOB_B}
627	else
628	Length=$( create_ts_next_date ${DATE_FIN} )
629	DATE_COURANTE=$( IGCM_date_AddDaysToGregorianDate ${DATE_FIN} ${Length} )
630	fi
631
632	(( TotLength = 0 ))
633	COMPTEUR=1
634	DATE_LOOP=${DATE_COURANTE}
635
636	while [ ${COMPTEUR} -lt ${NBRE_FILE_LOOP} ] ; do
637	#
638	Length=$( create_ts_next_date ${DATE_LOOP} )
639	DATE_LOOP=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} ${Length} )
640	(( TotLength = TotLength + Length ))
641	(( COMPTEUR = COMPTEUR + 1 ))
642	#
643	done # while [ ${COMPTEUR} -le ${NBRE_FILE_LOOP} ] done
644	DATE_FIN=$( IGCM_date_AddDaysToGregorianDate ${DATE_COURANTE} ${TotLength} )
645
646	[ -n "${DATE_LOOP}" ] && DATE_FIN_JOB_B_LOOP_PREC=${DATE_FIN_JOB_B_LOOP} ; DATE_FIN_JOB_B_LOOP=${DATE_LOOP}
647	[ -n "${LEVEL}" ] && [ ${LEVEL} -eq 0 ] && FLAG_B="TRUE" \|\| [ ${CURRENT_LOOP} -gt 1 ] && FLAG_B="TRUE" \|\| FLAG_B="FALSE"
648	DEPOT="FALSE"
649	LEVEL=2
650	else
651	# - LEVEL 3 : WE ARE IN THE LAST LOOP
652	NBRE_FILE_LOOP=$(( ${NBRE_FILE_TOT} % ( ${UNIX_MAX_LIMIT} * ( ${CURRENT_LOOP} - 1 ) ) ))
653	[ ${NBRE_FILE_LOOP} -eq 0 ] && NBRE_FILE_LOOP=${UNIX_MAX_LIMIT}
654
655	Length=$( create_ts_next_date ${DATE_FIN} )
656	DATE_COURANTE=$(IGCM_date_AddDaysToGregorianDate ${DATE_FIN} ${Length} )
657	DATE_FIN=${PeriodDateEnd}
658	[ -n "${DATE_LOOP}" ] && DATE_FIN_JOB_B_LOOP_PREC=${DATE_FIN_JOB_B_LOOP} ; DATE_FIN_JOB_B_LOOP=${DATE_LOOP}
659
660	LEVEL=3
661	DEPOT="TRUE"
662	FLAG_B="TRUE"
663	fi
664
665	# THEN FOR EACH FILE TYPE WE LOOP BETWEEN DATE_COURANTE AND DATE_FIN.
666	for comp in ${config_ListOfComponents[*]} ; do
667	#
668	IGCM_sys_Mkdir ${RUN_DIR_PATH}/${comp}
669	#
670	IGCM_sys_Cd ${RUN_DIR_PATH}/${comp}
671	#
672	eval R_OUT_${comp}=${R_SAVE}/${comp}
673	#
674	i=0
675	#
676	for file in $( eval echo \${LISTE_FILE_${comp}[*]} ); do
677	#
678	# Determine in which output can we find file
679	# IE : Output/MO or Output/DA or ...
680	#
681	FlagDir=$( echo ${file} \| awk -F "_" '{print $1}' )
682	case ${FlagDir} in
683	*Y) TS_Dir=TS_YE ; FreqDir=YE ;;
684	*M) TS_Dir=TS_MO ; FreqDir=MO ;;
685	*D) TS_Dir=TS_DA ; FreqDir=DA ;;
686	HF) TS_Dir=TS_HF ; FreqDir=HF ;;
687	INS) TS_Dir=TS_INS ; FreqDir=INS ;;
688	*)
689	IGCM_debug_Print 1 "Error in FlagDir=${FlagDir} for file ${file}."
690	IGCM_sys_Exit "Job create_ts" ;;
691	esac
692
693	# Initialize array
694	#set -A liste_file
695	#set -A liste_file_tmp
696	unset liste_file
697	unset liste_file_tmp
698	eval file_path=\${R_OUT_${comp}}/Output/${FreqDir}/
699
700	COMPTEUR=1
701	DATE_LOOP=${DATE_COURANTE}
702
703	while [ ${COMPTEUR} -le ${NBRE_FILE_LOOP} ] ; do
704	#
705	Length1=$( create_ts_begin_date ${DATE_LOOP} end)
706	DATE_TAB=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} $(( 1 - Length1 )) )_${DATE_LOOP}
707
708	Length2=$( create_ts_next_date ${DATE_LOOP} end)
709	DATE_LOOP=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} ${Length2} )
710
711	#set +A liste_file ${liste_file[*]} ${file_path}${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc
712	#set +A liste_file_tmp ${liste_file_tmp[*]} ${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc
713	liste_file[${#liste_file[*]}]=${file_path}${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc
714	liste_file_tmp[${#liste_file_tmp[*]}]=${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc
715	(( COMPTEUR = COMPTEUR + 1 ))
716	#
717	done # while [ ${COMPTEUR} -le ${NBRE_FILE_LOOP} ] done
718	DATE_LOOP=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} -${Length2} )
719
720	# Get selected files
721	IGCM_sys_Get /l liste_file[*] ${RUN_DIR_PATH}/${comp}
722	ExitFlag=false
723	IGCM_debug_Verif_Exit_Post
724
725	# Apply IGCM_Patch if needed
726	if [ ! X${RebuildFrequency} = Xtrue ] ; then
727	if [ X$( eval echo \${LISTE_PATCHES_${file}[0]} ) != X${NULL_STR} ]; then
728	for file_t in $( eval echo ${liste_file_tmp[*]} ); do
729	for Patch in $( eval echo \${LISTE_PATCHES_${file}[*]} ); do
730	IGCM_${Patch} ${file_t}
731	done
732	done
733	fi
734	fi
735
736	# WE CAN CONCATENATE FILES AT THIS POINT
737	[ "${FLAG_B}" = "TRUE" ] && DATE_BUILD_B=${DateBegin}_${DATE_FIN_JOB_B_LOOP_PREC}
738	Length=$( create_ts_begin_date ${DATE_COURANTE} end)
739	DATE_BUILD1=$( IGCM_date_AddDaysToGregorianDate ${DATE_COURANTE} -$(( Length - 1 )) )
740	DATE_BUILD=${DATE_BUILD1}_${DATE_FIN}
741	DATE_BUILD_END=${DateBegin}_${DATE_FIN}
742
743	liste_coord=" "
744	for axis in $( eval echo \${LISTE_AXIS_${file}[*]} ); do
745	liste_coord=${liste_coord}${axis}","
746	done
747
748	# Time axis must be the last dimension
749	time_axis=${axis}
750
751	# SWITCH BETWEEN MODEL TO FIT PARTICULAR CASE
752	for var in $( eval echo \${LISTE_VARS_${file}[*]} ) ; do
753	#
754	IGCM_sys_ncrcat --hst -v ${liste_coord}${var} ${liste_file_tmp[*]} ${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
755	#
756	# # Change time axis name to be CF compliant
757	# if [ ${time_axis} != time_counter ]; then
758	# IGCM_sys_ncrename -O -v ${time_axis},time_counter ${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
759	# fi
760	#
761	if [ "${FLAG_B}" = "TRUE" ] ; then
762	# WE CONCATENATE WITH EXISTING FILES
763	file1=${config_UserChoices_JobName}_${DATE_BUILD_B}_${FlagDir}_${var}.nc
764	file2=${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
765	file_out=${config_UserChoices_JobName}_${DATE_BUILD_END}_${FlagDir}_${var}.nc
766	IGCM_sys_ncrcat --hst ${file1} ${file2} ${file_out}
767	#
768	IGCM_sys_Rm ${file1} ${file2}
769	fi
770	#
771	if [ "${DEPOT}" = "TRUE" ] ; then
772	# WE PUT FINAL FILE ON FILE SYSTEM
773	[ "${FLAG_B}" = "FALSE" ] && file_out=${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
774	#${ncatted} -a modulo,time_counter,c,c," " ${file_out}
775	file_before=${config_UserChoices_JobName}_${DateBegin}_${DATE_FIN_JOB_B}_${FlagDir}_${var}.nc
776	eval IGCM_sys_Put_Out ${file_out} \${R_OUT_${comp}}/Analyse/${TS_Dir}/${file_out}
777	[ "${FLAG_B}" = "TRUE" ] && eval MENAGE \${R_OUT_${comp}}/Analyse/${TS_Dir} ${file_before} ${file_out}
778	IGCM_sys_Rm ${file_out}
779	#
780	fi
781	#
782	ExitFlag=false
783	IGCM_debug_Verif_Exit_Post
784	done # for var in ${liste_var} ; do
785	IGCM_sys_Rm ${liste_file_tmp[*]}
786	done # for file in ${liste_file_brut} do
787	done # for comp in ${config_ListOfComponents[*]} ; do
788	ExitFlag=false
789	IGCM_debug_Verif_Exit_Post
790	# GO AHEAD IN DEEPEST LOOP
791	(( CURRENT_LOOP = CURRENT_LOOP + 1 ))
792	done # while [ ${CURRENT_LOOP} -le ${NBRE_TOUR} ] do
793	# FINISH
794
795	# DODS copy
796	for comp in ${config_ListOfComponents[*]} ; do
797	for TS_Dir in TS_YE TS_MO TS_DA TS_HF TS_INS ; do
798	IGCM_sys_TestDirArchive ${R_SAVE}/${comp}/Analyse/${TS_Dir}
799	[ $? = 0 ] && IGCM_sys_Put_Dods ${comp}/Analyse/${TS_Dir}
800	done
801	done
802
803	export POST_DIR; export SUBMIT_DIR; export libIGCM; export R_INIT; export R_BC; export StandAlone
804	export RESOL_ATM; export RESOL_OCE ; export RESOL_ICE ; export RESOL_MBG ; export RESOL_SRF ; export RESOL_BIOS
805
806	listVarEnv="POST_DIR,SUBMIT_DIR,libIGCM,R_INIT,R_BC,StandAlone,RESOL_ATM,RESOL_OCE,RESOL_ICE,RESOL_MBG,RESOL_SRF,RESOL_BIOS"
807	export listVarEnv
808
809	#D-
810	# --------------------------------------------------------------------
811	#D- Test if all was right
812	# --------------------------------------------------------------------
813	IGCM_debug_Verif_Exit_Post
814	# --------------------------------------------------------------------
815	#D- Submit MONITORING
816	# --------------------------------------------------------------------
817	( [ X${TsTask} = X2D ] \|\| [ X${TsTask} = X ] ) && IGCM_sys_QsubPost monitoring
818
819	#D-
820	# --------------------------------------------------------------------
821	#D- Update the run.card
822	# --------------------------------------------------------------------
823	if [ ${StandAlone} != true ] ; then
824	# Put in run.card end period of time series
825	IGCM_sys_RshMaster IGCM_card_WriteOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesCompleted "${DATE_FIN}"
826	# We have finish the job !
827	IGCM_sys_RshMaster IGCM_card_WriteOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesRunning "n"
828	fi
829	date

Note: See TracBrowser for help on using the repository browser.

Download in other formats: