Context Navigation

AA_create_ts @ 838

Last change on this file since 838 was 837, checked in by labetoulle, 11 years ago

On Curie, ins_job sets up project ID. See #98.

Add

#-Q- curie #MSUB -A ::default_project::

to job headers, to be replaced by proper project ID by ins_job.

Property licence set to
The following licence information concerns ONLY the libIGCM tools
==================================================================

Copyright © Centre National de la Recherche Scientifique CNRS
Commissariat à l'Énergie Atomique CEA

libIGCM : Library for Portable Models Computation of IGCM Group.

IGCM Group is the french IPSL Global Climate Model Group.

This library is a set of shell scripts and functions whose purpose is
the management of the initialization, the launch, the transfer of
output files, the post-processing and the monitoring of datas produce
by any numerical program on any plateforme.

This software is governed by the CeCILL license under French law and
abiding by the rules of distribution of free software. You can use,
modify and/ or redistribute the software under the terms of the CeCILL
license as circulated by CEA, CNRS and INRIA at the following URL
"http://www.cecill.info".

As a counterpart to the access to the source code and rights to copy,
modify and redistribute granted by the license, users are provided only
with a limited warranty and the software's author, the holder of the
economic rights, and the successive licensors have only limited
liability.

In this respect, the user's attention is drawn to the risks associated
with loading, using, modifying and/or developing or reproducing the
software by the user in light of its specific status of free software,
that may mean that it is complicated to manipulate, and that also
therefore means that it is reserved for developers and experienced
professionals having in-depth computer knowledge. Users are therefore
encouraged to load and test the software's suitability as regards their
requirements in conditions enabling the security of their systems and/or
data to be ensured and, more generally, to use and operate it in the
same conditions as regards security.

The fact that you are presently reading this means that you have had
knowledge of the CeCILL license and that you accept its terms.
Property svn:keywords set to Revision Author Date

File size: 31.2 KB

Line
1	#-Q- curie ######################
2	#-Q- curie ## CURIE TGCC/CEA ##
3	#-Q- curie ######################
4	#-Q- curie #MSUB -r TS # Nom du job
5	#-Q- curie #MSUB -eo
6	#-Q- curie #MSUB -n 1 # Reservation du processus
7	#-Q- curie #MSUB -T 86400 # Limite de temps elapsed du job
8	#-Q- curie #MSUB -q xlarge
9	#-Q- curie #MSUB -Q normal
10	#-Q- curie #MSUB -A ::default_project::
11	#-Q- curie set +x
12	#-Q- ada #!/bin/ksh
13	#-Q- ada #######################
14	#-Q- ada ## ADA IDRIS ##
15	#-Q- ada #######################
16	#-Q- ada # Temps Elapsed max. d'une requete hh:mm:ss
17	#-Q- ada # @ wall_clock_limit = 20:00:00
18	#-Q- ada # Nom du travail LoadLeveler
19	#-Q- ada # @ job_name = TS
20	#-Q- ada # Fichier de sortie standard du travail
21	#-Q- ada # @ output = $(job_name).$(jobid)
22	#-Q- ada # Fichier de sortie d'erreur du travail
23	#-Q- ada # @ error = $(job_name).$(jobid)
24	#-Q- ada # pour recevoir un mail en cas de depassement du temps Elapsed (ou autre pb.)
25	#-Q- ada # @ notification = error
26	#-Q- ada # @ environment = $DEBUG_debug ; $MODIPSL ; $libIGCM ; $libIGCM_SX ; $SUBMIT_DIR ; $REBUILD_DIR ; $RebuildFromArchive ; $POST_DIR ; $MASTER ; $RebuildFrequency ; $DateBegin ; $PeriodDateBegin ; $PeriodDateEnd ; $NbRebuildDir ; $StandAlone ; $CompletedFlag ; $TsTask ; $CompToRead ; $FlagToRead ; $RESOL_ATM ; $RESOL_OCE ; $RESOL_ICE ; $RESOL_MBG ; $RESOL_SRF ; $RESOL_SBG ; $MASTER
27	#-Q- ada # @ queue
28	#-Q- sx9mercure #!/bin/ksh
29	#-Q- sx9mercure ######################
30	#-Q- sx9mercure ## SX9MERCURE CCRT ##
31	#-Q- sx9mercure ######################
32	#-Q- sx9mercure #PBS -N TS # Nom du job
33	#-Q- sx9mercure #PBS -j o # regroupement des stdout et stderr
34	#-Q- sx9mercure #PBS -S /usr/bin/ksh # shell de soumission
35	#-Q- sx9mercure #PBS -l memsz_job=1gb # Limite memoire a 1 Go
36	#-Q- sx9mercure #PBS -l elapstim_req=24:00:00 # Limite temps a 2 heures
37	#-Q- sx9mercure #PBS -q scalaire
38	#-Q- sx9mercure #PBS -r n
39	#-Q- titane #!/bin/ksh
40	#-Q- titane ######################
41	#-Q- titane ## TITANE CEA ##
42	#-Q- titane ######################
43	#-Q- titane #MSUB -r TS # Nom du job
44	#-Q- titane #MSUB -eo
45	#-Q- titane #MSUB -n 1 # Reservation du processus
46	#-Q- titane #MSUB -T 86400 # Limite de temps elapsed du job
47	#-Q- titane #MSUB -q mono
48	#-Q- titane #MSUB -E '-rn'
49	#-Q- lxiv8 ######################
50	#-Q- lxiv8 ## OBELIX LSCE ##
51	#-Q- lxiv8 ######################
52	#-Q- lxiv8 #PBS -N TS
53	#-Q- lxiv8 #PBS -m a
54	#-Q- lxiv8 #PBS -j oe
55	#-Q- lxiv8 #PBS -q medium
56	#-Q- lxiv8 #PBS -o TS.$$
57	#-Q- lxiv8 #PBS -S /bin/ksh
58	#-Q- default #!/bin/ksh
59	#-Q- default ##################
60	#-Q- default ## DEFAULT HOST ##
61	#-Q- default ##################
62
63	#**************************************************************
64	# Author: Sebastien Denvil
65	# Contact: Sebastien.Denvil__at__ipsl.jussieu.fr
66	# $Revision:: $ Revision of last commit
67	# $Author:: $ Author of last commit
68	# $Date:: $ Date of last commit
69	# IPSL (2006)
70	# This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
71	#
72	#**************************************************************
73
74	#set -eu
75	#set -vx
76
77	date
78
79	#-Q- ada export OMP_NUM_THREADS=1
80
81	#D- Task type (computing or post-processing)
82	TaskType=post-processing
83
84	########################################################################
85
86	#D- Flag to determine if this job in a standalone mode
87	#D- Default : value from AA_job if any
88	StandAlone=${StandAlone:=true}
89
90	#D- Path to libIGCM
91	#D- Default : value from AA_job if any
92	libIGCM=${libIGCM:=::modipsl::/libIGCM}
93
94	#D- Flag to determine allready produced time series. Empty if you start from the beginning
95	#D- Default : Value from AA_job if any. Usefull in StandAlone case if you want to continue a time series
96	CompletedFlag=${CompletedFlag:=}
97
98	#D- Flag to determine task type this script will perform.
99	#D- Possible Value : 2D, 3D, Chunck2D, Chunck3D
100	#D- Default : Value from AA_job if any. Usefull in StandAlone case.
101	TsTask=${TsTask:=2D}
102	if [ X${TsTask} = Xempty ] ; then
103	TsTask=""
104	fi
105
106	#D- Flag to determine if rebuild process has been performed asynchronously
107	#D- Possible value true or false.
108	#D- If true raw files has allready been patched by rebuild job
109	#D- If false create_ts will patch the raw files
110	#D- Default : Value from AA_job if any. Usefull in StandAlone case.
111	RebuildFrequency=${RebuildFrequency:=true}
112
113	#D- Flag to determine job's output directory
114	#D- Default : value from libIGCM_post.ksh if any
115	POST_DIR=${POST_DIR:=${PBS_O_WORKDIR:=$(pwd)}}
116
117	#D- Increased verbosity (1, 2, 3)
118	#D- Default : value from AA_job if any
119	Verbosity=${Verbosity:=3}
120
121	#D- Low level debug : to bypass lib test checks and stack construction
122	#D- Default : value from AA_job if any
123	DEBUG_debug=${DEBUG_debug:=false}
124
125	#D- TEMPORARY Flag to determine atmospheric resolution
126	#D- Default : value from atmospheric driver if any
127	RESOL_ATM=ALL
128
129	#D- Flag to determine surface resolution
130	#D- Default : value from surface driver if any
131	RESOL_SRF=ALL
132
133	#D- Flag to determine surface resolution
134	#D- Default : value from surface driver if any
135	RESOL_SBG=ALL
136
137	#D- TEMPORARY Flag to determine ocean resolution
138	#D- Default : value from ocean driver if any
139	RESOL_OCE=${RESOL_OCE:=ORCA2}
140
141	#D- TEMPORARY Flag to determine ice resolution
142	#D- Default : value from ice driver if any
143	RESOL_ICE=${RESOL_ICE:=ORCA2}
144
145	#D- TEMPORARY Flag to determine marine biogeochemistry resolution
146	#D- Default : value from ice driver if any
147	RESOL_MBG=${RESOL_MBG:=ORCA2}
148
149	########################################################################
150
151	. ${libIGCM}/libIGCM_debug/libIGCM_debug.ksh
152	. ${libIGCM}/libIGCM_card/libIGCM_card.ksh
153	. ${libIGCM}/libIGCM_date/libIGCM_date.ksh
154	#-------
155	. ${libIGCM}/libIGCM_sys/libIGCM_sys.ksh
156	. ${libIGCM}/libIGCM_config/libIGCM_config.ksh
157	#-------
158	( ${DEBUG_debug} ) && IGCM_debug_Check
159	( ${DEBUG_debug} ) && IGCM_card_Check
160	( ${DEBUG_debug} ) && IGCM_date_Check
161
162	########################################################################
163
164	#set -vx
165
166	# Useful cleaning function
167	MENAGE ()
168	{
169	[ -f $3 ] && ( IGCM_sys_RshArchive rm -f $1/$2 ;)
170	}
171
172	########################################################################
173
174	IGCM_sys_MkdirWork ${RUN_DIR_PATH}
175	IGCM_sys_Cd ${RUN_DIR_PATH}
176
177	# ------------------------------------------------------------------
178	# Test if all was right before proceeding further
179	# ------------------------------------------------------------------
180	IGCM_debug_Verif_Exit_Post
181
182	if [ ${StandAlone} = true ] ; then
183	CARD_DIR=${SUBMIT_DIR}
184	else
185	CARD_DIR=${RUN_DIR_PATH}
186	IGCM_sys_Get_Master ${SUBMIT_DIR}/config.card ${RUN_DIR_PATH}
187	IGCM_sys_Get_Master ${SUBMIT_DIR}/run.card ${RUN_DIR_PATH}
188	IGCM_sys_Get_Master ${SUBMIT_DIR}/COMP ${RUN_DIR_PATH}
189	IGCM_sys_Get_Master ${SUBMIT_DIR}/POST ${RUN_DIR_PATH}
190	fi
191
192	#==================================
193	# First of all
194	#
195	# Read libIGCM compatibility version in config.card
196	# Read UserChoices section
197	# Read Ensemble section
198	# Read Post section
199	# Define all netcdf output directories
200	#==================================
201	IGCM_config_CommonConfiguration ${CARD_DIR}/config.card
202
203	#==================================
204	# Define default value to keep compatibility with previous card: means before changes due to TGCC
205	if [ X${PackDefault} = Xtrue ] ; then
206	[ X${config_Post_PackFrequency} = X ] && config_Post_PackFrequency=${config_Post_RebuildFrequency}
207	else
208	config_Post_PackFrequency=NONE
209	fi
210
211	#==================================
212	# If pack is active then PackFrequency overule the config_UserChoices_PeriodLength
213	if ( [ ! X${config_Post_PackFrequency} = X${NULL_STR} ] && [ ! X${config_Post_PackFrequency} = XNONE ] ) ; then
214	config_UserChoices_PeriodLength=${config_Post_PackFrequency}
215	fi
216
217	#
218	# Determine component to take care of depending on kind of task create_ts will perform
219	#
220	[ ${CompToRead} ] && set -A config_ListOfComponents ${CompToRead} \|\| IGCM_card_DefineArrayFromSection ${CARD_DIR}/config.card ListOfComponents
221	#
222	# Determine period for time series. Default : value from AA_job or from launch_TS if any
223	#
224	[ X${CompletedFlag} = X ] \|\| CompletedFlag=$( IGCM_date_ConvertFormatToGregorian ${CompletedFlag} )
225
226	DateBegin=${DateBegin:=$( IGCM_date_ConvertFormatToGregorian ${config_UserChoices_DateBegin} )}
227	#
228	PeriodDateEnd=${PeriodDateEnd:=$( IGCM_date_ConvertFormatToGregorian ${config_UserChoices_DateEnd} )}
229	#
230	# Determine Dimension of the produced time series : 2D or 3D
231	if ( [ X${TsTask} = XChunck2D ] \|\| [ X${TsTask} = X2D ] ) ; then
232	Dimension=2D
233	elif ( [ X${TsTask} = XChunck3D ] \|\| [ X${TsTask} = X3D ] ) ; then
234	Dimension=3D
235	else
236	Dimension=""
237	fi
238
239	# ------------------------------------------------------------------
240	#D- Test if all was right before entering the loop
241	# ------------------------------------------------------------------
242	IGCM_debug_Verif_Exit_Post
243
244	########################################################################
245	#
246	# Depending on requested time series fill following variables :
247	#
248	# - LISTE_FILE_${comp}[*]
249	# - LISTE_VARS_${file}[*]
250	# - LISTE_AXIS_${file}[*]
251	# - LISTE_PATCH_${file}[*]
252	########################################################################
253
254	#
255	# For each selected component determine which files need post-processing
256	#
257	DoJob=false
258	for comp in ${config_ListOfComponents[*]} ; do
259	#
260	ActiveComp=false
261	#
262	IGCM_card_DefineArrayFromOption ${CARD_DIR}/config.card ListOfComponents ${comp}
263	eval compname=\${config_ListOfComponents_${comp}[0]} > /dev/null 2>&1
264	#
265	card=${CARD_DIR}/COMP/${compname}.card
266
267	IGCM_card_DefineArrayFromOption ${card} OutputFiles List
268	#
269	ListFilesName=${compname}_OutputFiles_List
270	eval FileName0=\${${ListFilesName}[0]} > /dev/null 2>&1
271	#
272	if [ X${FileName0} != X${NULL_STR} ] ; then
273	#
274	#IGCM_debug_Print 1 "Component : ${compname}"
275	#
276	# INITIALISATION
277	#
278	typeset i
279	if [ ${FlagToRead} ] ; then
280	((NbFiles=${FlagToRead}))
281	((i=${NbFiles}))
282	else
283	eval NbFiles=\${#${ListFilesName}[@]} > /dev/null 2>&1
284	i=2
285	fi
286	#
287	until [ $i -gt $NbFiles ]; do
288	#
289	eval flag_post=\${${ListFilesName}[$i]} > /dev/null 2>&1
290	#
291	if [ X${flag_post} != XNONE ] ; then
292	#
293	# For each comp and each file who needs post-processing
294	# determine which variables need time series.
295	#
296	FILE=$( echo ${flag_post} \| awk "-FPost_" '{print $2}' )
297	IGCM_card_DefineArrayFromOption ${card} Post_${FILE} TimeSeriesVars${Dimension}
298	IGCM_card_DefineVariableFromOption ${card} Post_${FILE} ChunckJob${Dimension}
299	#
300	# If TimeSeriesVars list is empty we skip
301	#
302	if [ $( eval echo \${${compname}_Post_${FILE}_TimeSeriesVars${Dimension}} ) = ${NULL_STR} ] ; then
303	#
304	IGCM_debug_Print 3 "Nothing to do for ${FILE} according to ${card}"
305	IGCM_debug_Print 3 "TimeSeriesVars${Dimension} for ${FILE} is empty"
306	(( i=i+3 ))
307	continue
308	#
309	fi
310	#
311	# If files/variables has been switch off by ChunckJob${Dimension}=0 we skip
312	#
313	if [ X$( eval echo \${${compname}_Post_${FILE}_ChunckJob${Dimension}} ) = XOFF ] ; then
314	IGCM_debug_Print 3 "Nothing to do for ${FILE} according to ${card}"
315	IGCM_debug_Print 3 "ChunckJob${Dimension}=OFF for ${FILE} so it has been switched off"
316	(( i=i+3 ))
317	continue
318	fi
319	#
320	# If we are not a Chunck type task, we skip files/variables handled by chunck task
321	#
322	if ( [ ! X${TsTask} = XChunck${Dimension} ] && [ ! X${TsTask} = X ] && [ ! XChunck${Dimension} = XChunck ] ) ; then
323	if [ ! $( eval echo \${${compname}_Post_${FILE}_ChunckJob${Dimension}} ) = NONE ] ; then
324	IGCM_debug_Print 3 "Nothing to do for ${FILE} TimeSeriesVars${Dimension} according to ${card}"
325	IGCM_debug_Print 3 "Let another chunck task do the job $( eval echo \${${compname}_Post_${FILE}_ChunckJob${Dimension}} )"
326	(( i=i+3 ))
327	continue
328	fi
329	fi
330	#
331	# Now on we know we have things to do
332	#
333
334	#
335	# We need coordinates axis to properly fill produced time series
336	#
337	IGCM_card_DefineArrayFromOption ${card} Post_${FILE} GatherWithInternal
338	#
339	if [ $( eval echo \${${compname}_Post_${FILE}_GatherWithInternal} ) = ${NULL_STR} ] ; then
340	#
341	IGCM_debug_Print 1 "ERROR !! We miss coordinate axis for ${FILE} according to ${card}"
342	IGCM_debug_Print 1 "So no time series will be produced for ${FILE} !!"
343	(( i=i+3 ))
344	continue
345	#
346	fi
347	#
348	# We need LIST of variables not allready produced (useful for standalone mode)
349	#
350	FlagDir=$( echo ${FILE} \| awk -F "_" '{print $1}' )
351	case ${FlagDir} in
352	*Y) TS_Dir=TS_YE ;;
353	*M) TS_Dir=TS_MO ;;
354	*D) TS_Dir=TS_DA ;;
355	HF) TS_Dir=TS_HF ;;
356	3H) TS_Dir=TS_HF ;;
357	INS) TS_Dir=TS_INS ;;
358	esac
359	#
360	count=0
361	for var in $( eval echo \${${compname}_Post_${FILE}_TimeSeriesVars${Dimension}[*]} ) ; do
362	#
363	DIRECTORY=${R_SAVE}/${comp}/Analyse/${TS_Dir}
364	TestedFile=${config_UserChoices_JobName}_${DateBegin}_${PeriodDateEnd}_${FlagDir}_${var}.nc
365	#
366	IGCM_sys_TestFileArchive ${DIRECTORY}/${TestedFile}
367	if [ ! $? = 0 ] ; then
368	eval set +A LISTE_VARS_${FILE} \${LISTE_VARS_${FILE}[*]} ${var}
369	ActiveComp=true
370	(( count = count + 1 ))
371	fi
372	done
373	#
374	if [ ${count} -gt 0 ] ; then
375	#
376	eval set +A LISTE_FILE_${comp} \${LISTE_FILE_${comp}[*]} ${FILE}
377	eval set +A LISTE_AXIS_${FILE} \${${compname}_Post_${FILE}_GatherWithInternal[*]}
378	#
379	if [ ! X${RebuildFrequency} = Xtrue ] ; then
380	IGCM_card_DefineArrayFromOption ${card} Post_${FILE} Patches
381	eval set +A LISTE_PATCHES_${FILE} \${${compname}_Post_${FILE}_Patches[*]}
382	if [ X$( eval echo \${LISTE_PATCHES_${FILE}[0]} ) != X${NULL_STR} ]; then
383	for Patch in $( eval echo \${LISTE_PATCHES_${FILE}[*]} ); do
384	if [ Xload${Patch} != X ] ; then
385	. ${libIGCM}/libIGCM_post/IGCM_${Patch}.ksh
386	eval load${Patch}=loaded
387	fi
388	done
389	fi
390	fi
391	fi
392	fi
393	(( i=i+3 ))
394	done
395	fi
396	#
397	if [ ${ActiveComp} = true ] ; then
398	set +A ActiveListOfComponents ${ActiveListOfComponents[*]} ${comp}
399	fi
400	done
401
402	set -A config_ListOfComponents ${ActiveListOfComponents[*]}
403	[ ! -z "${ActiveListOfComponents}" ] && DoJob=true
404
405	########################################################################
406	#
407	# IS THERE SOME ALLREADY PRODUCED TIME SERIES ?
408	# IF SO BRING THEM IN THE WORKING DIRECTORY
409	#
410	# PS : Keep in mind that IGCM_sys_Get here is a weak link :
411	# - especially for IDRiS
412	# - no special protection against "command too long" !
413	# - libIGCM_sys should handle that or modify following code part
414	#
415	########################################################################
416
417	if [ ${StandAlone} != true ] ; then
418	Running_Flag=$( IGCM_sys_RshMaster "IGCM_card_DefineVariableFromOption \
419	${SUBMIT_DIR}/run.card PostProcessing TimeSeriesRunning ; \
420	echo \${run_PostProcessing_TimeSeriesRunning}" )
421
422	#if [ X${Running_Flag} = Xy ] ; then
423	# # Time Series Job allready running
424	# IGCM_debug_Print 1 "Time Series Job allready running exit"
425	# exit
426	#fi
427	# Now run.card know we are running
428
429
430
431
432	IGCM_sys_RshMaster IGCM_card_WriteOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesRunning "y"
433
434
435
436
437
438	# Get information from last execution
439	#CompletedFlag=$( IGCM_sys_RshMaster \
440	# "IGCM_card_DefineVariableFromOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesCompleted ;\
441	# echo \${run_PostProcessing_TimeSeriesCompleted}" )
442	fi
443
444	# Is it the first submission or not ?
445
446	if ( [ X${CompletedFlag} = X ] \|\| [ X${CompletedFlag} = X$( IGCM_date_AddDaysToGregorianDate ${DateBegin} -1 ) ] ) ; then
447	# First Time Series Submission
448	FIRST_PASS=TRUE
449	Length=$( IGCM_date_DaysInPreviousPeriod ${DateBegin} ${config_UserChoices_PeriodLength} begin)
450	DATE_FIN_JOB_B=$( IGCM_date_AddDaysToGregorianDate ${DateBegin} $(( Length - 1 )) )
451	else
452	# Last Completed Time Series
453	FIRST_PASS=FALSE
454	DATE_FIN_JOB_B=${CompletedFlag}
455	date=${DateBegin}_${DATE_FIN_JOB_B}
456	for comp in ${config_ListOfComponents[*]} ; do
457	#
458	IGCM_sys_Mkdir ${RUN_DIR_PATH}/${comp}
459	#
460	IGCM_sys_Cd ${RUN_DIR_PATH}/${comp}
461	#
462	#set -A liste_file
463	unset liste_file
464	#
465	file_path=${R_SAVE}/${comp}/Analyse
466	#
467	i=0
468	file=${NULL_STR}
469	#
470	for file in $( eval echo \${LISTE_FILE_${comp}[*]} ); do
471	#
472	# Determine in which output can we find file
473	# IE : Analyse/TS or Analyse/DA2TS or ...
474	#
475	FlagDir=$( echo ${file} \| awk -F "_" '{print $1}' )
476	case ${FlagDir} in
477	*Y) TS_Dir=TS_YE ;;
478	*M) TS_Dir=TS_MO ;;
479	*D) TS_Dir=TS_DA ;;
480	HF) TS_Dir=TS_HF ;;
481	3H) TS_Dir=TS_HF ;;
482	INS) TS_Dir=TS_INS ;;
483	esac
484	for var in $( eval echo \${LISTE_VARS_${file}[*]} ) ; do
485	#
486	#set +A liste_file ${liste_file[*]} ${file_path}/${TS_Dir}/${config_UserChoices_JobName}_${date}_${FlagDir}_${var}.nc
487	liste_file[${#liste_file[*]}]=${file_path}/${TS_Dir}/${config_UserChoices_JobName}_${date}_${FlagDir}_${var}.nc
488	#
489	done
490	done
491	if [ X${file} != X${NULL_STR} ] ; then
492	IGCM_sys_Get /l liste_file[*] ${RUN_DIR_PATH}/${comp}
493	# Even if some files are missing we tried to continue
494	# Either ListVar in compname.card are asked but not present in original history file
495	# Either a previous TS task failed
496	ExitFlag=false
497	IGCM_debug_Verif_Exit_Post
498	fi
499	done
500	#
501	LEVEL=0
502	#
503	fi
504
505	########################################################################
506	#
507	# DEFINE LOOP PARAMETERS
508	#
509	########################################################################
510
511	DATE_COUNT=${DATE_FIN_JOB_B}
512	DATE_FIN_JOB_B_LOOP=${DATE_FIN_JOB_B}
513
514	( [ -n "${LEVEL}" ] && [ ${LEVEL} -eq 0 ] ) && NBRE_FILE_TOT=0 \|\| NBRE_FILE_TOT=1
515
516	if [ ${DoJob} = true ] ; then
517	while [ ${DATE_COUNT} -lt ${PeriodDateEnd} ] ; do
518	(( NBRE_FILE_TOT = NBRE_FILE_TOT + 1 ))
519	Length=$( IGCM_date_DaysInNextPeriod ${DATE_COUNT} ${config_UserChoices_PeriodLength} )
520	DATE_COUNT=$( IGCM_date_AddDaysToGregorianDate ${DATE_COUNT} ${Length} )
521	done
522	else
523	NBRE_FILE_TOT=0
524	fi
525
526	if [ ${NBRE_FILE_TOT} -le ${UNIX_MAX_LIMIT} ] ; then
527	NBRE_TOUR=1
528	elif [ ${NBRE_FILE_TOT}%${UNIX_MAX_LIMIT} -eq 0 ] ; then
529	let NBRE_TOUR=${NBRE_FILE_TOT}/${UNIX_MAX_LIMIT}
530	else
531	let NBRE_TOUR=${NBRE_FILE_TOT}/${UNIX_MAX_LIMIT}+1
532	fi
533
534	CURRENT_LOOP=1
535	# THIS LOOP GIVES PARAMETERS FOR THE DATE'S LOOP ie : DATE_COURANTE, DATE_FIN.
536	# WE HAVE 3 LEVELS
537	# - LEVEL 1 : JUST ONE LOOP TO COMPLETE
538	# - LEVEL 2 : WE ARE IN THE "DEEP" LOOP
539	# - LEVEL 3 : WE ARE IN THE LAST LOOP
540
541	while [ ${CURRENT_LOOP} -le ${NBRE_TOUR} ] ; do
542
543	if [ ${NBRE_FILE_TOT} -le ${UNIX_MAX_LIMIT} ] ; then
544	# - LEVEL 1 : JUST ONE LOOP TO COMPLETE
545	NBRE_FILE_LOOP=${NBRE_FILE_TOT}
546
547	if [ ${FIRST_PASS} = TRUE ] ; then
548	DATE_COURANTE=${DATE_FIN_JOB_B_LOOP}
549	else
550	Length=$( IGCM_date_DaysInNextPeriod ${DATE_FIN_JOB_B_LOOP} ${config_UserChoices_PeriodLength} )
551	DATE_COURANTE=$( IGCM_date_AddDaysToGregorianDate ${DATE_FIN_JOB_B_LOOP} ${Length} )
552	fi
553
554	DATE_FIN=${PeriodDateEnd}
555	DATE_FIN_JOB_B_LOOP_PREC=${DATE_FIN_JOB_B_LOOP}
556
557	[ -n "${DATE_LOOP}" ] && DATE_FIN_JOB_B_LOOP=${DATE_LOOP}
558	[ -n "${LEVEL}" ] && FLAG_B="TRUE" \|\| FLAG_B="FALSE"
559
560	LEVEL=1
561	DEPOT="TRUE"
562	elif [ ${CURRENT_LOOP} -ne ${NBRE_TOUR} ] ; then
563	# - LEVEL 2 : WE ARE IN THE "DEEP" LOOP
564	NBRE_FILE_LOOP=${UNIX_MAX_LIMIT}
565
566	if [ ! ${FIRST_PASS} = TRUE ] && [ ${CURRENT_LOOP} -eq 1 ] ; then
567	Length=$( IGCM_date_DaysInNextPeriod ${DATE_FIN_JOB_B_LOOP} ${config_UserChoices_PeriodLength} )
568	DATE_COURANTE=$( IGCM_date_AddDaysToGregorianDate ${DATE_FIN_JOB_B_LOOP} ${Length} )
569	elif [ -z "${DATE_COURANTE}" ] ; then
570	DATE_COURANTE=${DATE_FIN_JOB_B}
571	else
572	Length=$( IGCM_date_DaysInNextPeriod ${DATE_FIN} ${config_UserChoices_PeriodLength} )
573	DATE_COURANTE=$( IGCM_date_AddDaysToGregorianDate ${DATE_FIN} ${Length} )
574	fi
575
576	(( TotLength = 0 ))
577	COMPTEUR=1
578	DATE_LOOP=${DATE_COURANTE}
579
580	while [ ${COMPTEUR} -lt ${NBRE_FILE_LOOP} ] ; do
581	#
582	Length=$( IGCM_date_DaysInNextPeriod ${DATE_LOOP} ${config_UserChoices_PeriodLength} )
583	DATE_LOOP=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} ${Length} )
584	(( TotLength = TotLength + Length ))
585	(( COMPTEUR = COMPTEUR + 1 ))
586	#
587	done # while [ ${COMPTEUR} -le ${NBRE_FILE_LOOP} ] done
588	DATE_FIN=$( IGCM_date_AddDaysToGregorianDate ${DATE_COURANTE} ${TotLength} )
589
590	[ -n "${DATE_LOOP}" ] && DATE_FIN_JOB_B_LOOP_PREC=${DATE_FIN_JOB_B_LOOP} ; DATE_FIN_JOB_B_LOOP=${DATE_LOOP}
591	[ -n "${LEVEL}" ] && [ ${LEVEL} -eq 0 ] && FLAG_B="TRUE" \|\| [ ${CURRENT_LOOP} -gt 1 ] && FLAG_B="TRUE" \|\| FLAG_B="FALSE"
592	DEPOT="FALSE"
593	LEVEL=2
594	else
595	# - LEVEL 3 : WE ARE IN THE LAST LOOP
596	NBRE_FILE_LOOP=$(( ${NBRE_FILE_TOT} % ( ${UNIX_MAX_LIMIT} * ( ${CURRENT_LOOP} - 1 ) ) ))
597	[ ${NBRE_FILE_LOOP} -eq 0 ] && NBRE_FILE_LOOP=${UNIX_MAX_LIMIT}
598
599	Length=$( IGCM_date_DaysInNextPeriod ${DATE_FIN} ${config_UserChoices_PeriodLength} )
600	DATE_COURANTE=$(IGCM_date_AddDaysToGregorianDate ${DATE_FIN} ${Length} )
601	DATE_FIN=${PeriodDateEnd}
602	[ -n "${DATE_LOOP}" ] && DATE_FIN_JOB_B_LOOP_PREC=${DATE_FIN_JOB_B_LOOP} ; DATE_FIN_JOB_B_LOOP=${DATE_LOOP}
603
604	LEVEL=3
605	DEPOT="TRUE"
606	FLAG_B="TRUE"
607	fi
608
609	# THEN FOR EACH FILE TYPE WE LOOP BETWEEN DATE_COURANTE AND DATE_FIN.
610	for comp in ${config_ListOfComponents[*]} ; do
611	#
612	IGCM_sys_Mkdir ${RUN_DIR_PATH}/${comp}
613	#
614	IGCM_sys_Cd ${RUN_DIR_PATH}/${comp}
615	#
616	eval R_OUT_${comp}=${R_SAVE}/${comp}
617	#
618	i=0
619	#
620	for file in $( eval echo \${LISTE_FILE_${comp}[*]} ); do
621	#
622	# Determine in which output we can find file
623	# IE : Output/MO or Output/DA or ...
624	#
625	FlagDir=$( echo ${file} \| awk -F "_" '{print $1}' )
626	case ${FlagDir} in
627	*Y) TS_Dir=TS_YE ; FreqDir=YE ;;
628	*M) TS_Dir=TS_MO ; FreqDir=MO ;;
629	*D) TS_Dir=TS_DA ; FreqDir=DA ;;
630	HF) TS_Dir=TS_HF ; FreqDir=HF ;;
631	3H) TS_Dir=TS_HF ; FreqDir=HF ;;
632	INS) TS_Dir=TS_INS ; FreqDir=INS ;;
633	*) IGCM_debug_Print 1 "Error in FlagDir=${FlagDir} for file ${file}."
634	IGCM_sys_Exit "Job create_ts" ;;
635	esac
636
637	# Initialize array
638	#set -A liste_file
639	#set -A liste_file_tmp
640	unset liste_file
641	unset liste_file_tmp
642	eval file_path=\${R_OUT_${comp}}/Output/${FreqDir}/
643
644	COMPTEUR=1
645	MissingFile=FALSE
646	DATE_LOOP=${DATE_COURANTE}
647
648	while [ ${COMPTEUR} -le ${NBRE_FILE_LOOP} ] ; do
649	#
650	Length1=$( IGCM_date_DaysInPreviousPeriod ${DATE_LOOP} ${config_UserChoices_PeriodLength} end)
651	DATE_TAB=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} $(( 1 - Length1 )) )_${DATE_LOOP}
652
653	Length2=$( IGCM_date_DaysInNextPeriod ${DATE_LOOP} ${config_UserChoices_PeriodLength} )
654	DATE_LOOP=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} ${Length2} )
655
656	#set +A liste_file ${liste_file[*]} ${file_path}${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc
657	#set +A liste_file_tmp ${liste_file_tmp[*]} ${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc
658
659	IGCM_sys_TestFileArchive ${file_path}${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc
660	if [ $? = 0 ] ; then
661	liste_file[${#liste_file[*]}]=${file_path}${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc
662	liste_file_tmp[${#liste_file_tmp[*]}]=${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc
663	else
664	IGCM_debug_Print 1 "WARNING ${file_path}${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc dont exist"
665	MissingFile=TRUE
666	fi
667	(( COMPTEUR = COMPTEUR + 1 ))
668	#
669	done # while [ ${COMPTEUR} -le ${NBRE_FILE_LOOP} ] done
670
671	# We skip this file type if one of them is missing.
672	if [ "${MissingFile}" = "TRUE" ] ; then
673	IGCM_debug_Print 1 "WARNING We skip ${file} type processing"
674	continue
675	fi
676
677	DATE_LOOP=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} -${Length2} )
678
679	# Get selected files
680	IGCM_sys_Get /l liste_file[*] ${RUN_DIR_PATH}/${comp}
681	IGCM_debug_Verif_Exit_Post
682
683	# Apply IGCM_Patch if needed
684	if [ ! X${RebuildFrequency} = Xtrue ] ; then
685	if [ X$( eval echo \${LISTE_PATCHES_${file}[0]} ) != X${NULL_STR} ]; then
686	for file_t in $( eval echo ${liste_file_tmp[*]} ); do
687	for Patch in $( eval echo \${LISTE_PATCHES_${file}[*]} ); do
688	IGCM_${Patch} ${file_t}
689	IGCM_debug_Verif_Exit_Post
690	done
691	done
692	fi
693	fi
694
695	# WE CAN CONCATENATE FILES AT THIS POINT
696	[ "${FLAG_B}" = "TRUE" ] && DATE_BUILD_B=${DateBegin}_${DATE_FIN_JOB_B_LOOP_PREC}
697	Length=$( IGCM_date_DaysInPreviousPeriod ${DATE_COURANTE} ${config_UserChoices_PeriodLength} end)
698	DATE_BUILD1=$( IGCM_date_AddDaysToGregorianDate ${DATE_COURANTE} -$(( Length - 1 )) )
699	DATE_BUILD=${DATE_BUILD1}_${DATE_FIN}
700	DATE_BUILD_END=${DateBegin}_${DATE_FIN}
701
702	liste_coord=" "
703	for axis in $( eval echo \${LISTE_AXIS_${file}[*]} ); do
704	liste_coord=${liste_coord}${axis}","
705	done
706
707	# Time axis must be the last dimension
708	time_axis=${axis}
709
710	# SWITCH BETWEEN MODEL TO FIT PARTICULAR CASE
711	for var in $( eval echo \${LISTE_VARS_${file}[*]} ) ; do
712	#
713	if [ "${FLAG_B}" = "TRUE" ] ; then
714	# WE CONCATENATE WITH EXISTING FILES
715	file1=${config_UserChoices_JobName}_${DATE_BUILD_B}_${FlagDir}_${var}.nc
716	file_out=${config_UserChoices_JobName}_${DATE_BUILD_END}_${FlagDir}_${var}.nc
717	IGCM_sys_ncrcat --hst -v ${liste_coord}${var} ${file1} ${liste_file_tmp[*]} ${file_out}
718	#
719	IGCM_sys_Rm ${file1}
720	if [ ! "${DEPOT}" = "TRUE" ] ; then
721	eval IGCM_sys_Put_Out ${file_out} \${R_OUT_${comp}}/Analyse/${TS_Dir}/${file_out}
722	# "${file1} = file_before ?
723	[ ! "${file1}" = "${config_UserChoices_JobName}_${DateBegin}_${DATE_FIN_JOB_B}_${FlagDir}_${var}.nc" ] && eval MENAGE \${R_OUT_${comp}}/Analyse/${TS_Dir} ${file1} ${file_out}
724	fi
725
726	ExitFlag=false
727	else
728	# OR NOT
729	IGCM_sys_ncrcat --hst -v ${liste_coord}${var} ${liste_file_tmp[*]} ${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
730	fi
731
732	#
733	if [ "${DEPOT}" = "TRUE" ] ; then
734	# WE PUT FINAL FILE ON FILE SYSTEM
735	[ "${FLAG_B}" = "FALSE" ] && file_out=${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
736	file_before=${config_UserChoices_JobName}_${DateBegin}_${DATE_FIN_JOB_B}_${FlagDir}_${var}.nc
737	#
738	eval IGCM_sys_Put_Out ${file_out} \${R_OUT_${comp}}/Analyse/${TS_Dir}/${file_out}
739	[ "${FLAG_B}" = "TRUE" ] && eval MENAGE \${R_OUT_${comp}}/Analyse/${TS_Dir} ${file_before} ${file_out}
740	if [ ! "X${file1}" = "X" ] ; then
741	[ ! "${file1}" = "${config_UserChoices_JobName}_${DateBegin}_${DATE_FIN_JOB_B}_${FlagDir}_${var}.nc" ] && eval MENAGE \${R_OUT_${comp}}/Analyse/${TS_Dir} ${file1} ${file_out}
742	fi
743	#
744	if ( [ ${FlagDir} = "1M" ] && [ -f ${file_out} ] ) ; then
745	file_before_YE=${config_UserChoices_JobName}_${DateBegin}_${DATE_FIN_JOB_B}_1Y_${var}.nc
746	file_out_YE=${config_UserChoices_JobName}_${DATE_BUILD_END}_1Y_${var}.nc
747	#
748	# Compute yearly average considering calendar
749	# and set date of the mid year to 01/07 at 00:00
750	IGCM_sys_cdo -settime,00:00 -setday,1 -setmon,7 -settunits,days -divdpy -yearsum -muldpm -selvar,${var} ${file_out} ${file_out_YE}
751	# Extract all ancillary variables except $var, they will be appended after the cdo command
752	# Needed because cdo applies calculs on all variables
753	IGCM_sys_ncks -Oh -x -v ${var} ${file_out} ${file_out_YE%%.nc}_tmp1.nc
754	# This will remove the time axis because cdo rewrites severely with only considered useful variables
755	IGCM_sys_cdo -mulc,1 ${file_out_YE%%.nc}_tmp1.nc ${file_out_YE%%.nc}_tmp2.nc 2>/dev/null
756	if [ $? -eq 0 ] ; then
757	# Now append all ancillary variables if previous cdo operator ok
758	IGCM_sys_ncks -h -A ${file_out_YE%%.nc}_tmp2.nc ${file_out_YE}
759	fi
760
761	# Put global history attribut
762	IGCM_sys_ncatted -Oh -a history,global,o,c,"build by libIGCM/create_ts" ${file_out_YE}
763	#
764	eval IGCM_sys_Put_Out ${file_out_YE} \${R_OUT_${comp}}/Analyse/TS_MO_YE/${file_out_YE}
765	[ "${FLAG_B}" = "TRUE" ] && eval MENAGE \${R_OUT_${comp}}/Analyse/TS_MO_YE/ ${file_before_YE} ${file_out_YE}
766	IGCM_sys_Rm ${file_out_YE%%.nc}_tmp*.nc
767	fi
768	#
769	IGCM_sys_Rm ${file_out}
770	ExitFlag=false
771	#
772	fi
773	#
774	done # for var in ${liste_var} ; do
775	IGCM_sys_Rm ${liste_file_tmp[*]}
776	done # for file in ${liste_file_brut} do
777	done # for comp in ${config_ListOfComponents[*]} ; do
778	# GO AHEAD IN DEEPEST LOOP
779	(( CURRENT_LOOP = CURRENT_LOOP + 1 ))
780	done # while [ ${CURRENT_LOOP} -le ${NBRE_TOUR} ] do
781	# FINISH
782
783	# DODS copy
784	for comp in ${config_ListOfComponents[*]} ; do
785	for TS_Dir in TS_YE TS_MO TS_MO_YE TS_DA TS_HF TS_INS ; do
786	IGCM_sys_TestDirArchive ${R_SAVE}/${comp}/Analyse/${TS_Dir}
787	[ $? = 0 ] && IGCM_sys_Put_Dods ${comp}/Analyse/${TS_Dir}
788	done
789	done
790
791	export POST_DIR; export SUBMIT_DIR; export libIGCM; export R_INIT; export R_BC; export StandAlone
792	export RESOL_ATM; export RESOL_OCE ; export RESOL_ICE ; export RESOL_MBG ; export RESOL_SRF ; export RESOL_SBG
793
794	listVarEnv="POST_DIR,SUBMIT_DIR,libIGCM,R_INIT,R_BC,StandAlone,RESOL_ATM,RESOL_OCE,RESOL_ICE,RESOL_MBG,RESOL_SRF,RESOL_SBG"
795	export listVarEnv
796
797	#D-
798	# --------------------------------------------------------------------
799	#D- Test if all was right
800	# --------------------------------------------------------------------
801	IGCM_debug_Verif_Exit_Post
802	# --------------------------------------------------------------------
803	#D- Submit MONITORING
804	# --------------------------------------------------------------------
805	if ( [ X${TsTask} = X2D ] \|\| [ X${TsTask} = X3D ] \|\| [ X${TsTask} = X ] ) ; then
806	Script_Post_Output=monitoring.${PeriodDateEnd}
807	IGCM_sys_QsubPost monitoring
808	fi
809
810	#D-
811	# --------------------------------------------------------------------
812	#D- Update the run.card
813	# --------------------------------------------------------------------
814	if [ ${StandAlone} != true ] ; then
815	# Put in run.card end period of time series
816	[ X"${DATE_FIN}" = X ] \|\| IGCM_sys_RshMaster IGCM_card_WriteOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesCompleted "${DATE_FIN}"
817	# We have finish the job !
818	IGCM_sys_RshMaster IGCM_card_WriteOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesRunning "n"
819	fi
820
821	# Clean RUN_DIR_PATH (necessary for cesium and titane only)
822	IGCM_sys_RmRunDir -Rf ${RUN_DIR_PATH}
823
824	date

Note: See TracBrowser for help on using the repository browser.

Context Navigation

source: trunk/libIGCM/AA_create_ts @ 838

Download in other formats: