Context Navigation

AA_create_ts @ 1488

Last change on this file since 1488 was 1468, checked in by mafoipsl, 6 years ago

On curie : set output files for all jobs.

On irene :

set output files for all jobs like curie
add the possibility to use xlarge nodes (for free) for post-processing or skylake nodes
add a question to set project used for post-processing and delete -A option in libIGCM_sys_irene
display only available projects for computing or post-processing on irene

Property licence set to
The following licence information concerns ONLY the libIGCM tools
==================================================================

Copyright © Centre National de la Recherche Scientifique CNRS
Commissariat à l'Énergie Atomique CEA

libIGCM : Library for Portable Models Computation of IGCM Group.

IGCM Group is the french IPSL Global Climate Model Group.

This library is a set of shell scripts and functions whose purpose is
the management of the initialization, the launch, the transfer of
output files, the post-processing and the monitoring of datas produce
by any numerical program on any plateforme.

This software is governed by the CeCILL license under French law and
abiding by the rules of distribution of free software. You can use,
modify and/ or redistribute the software under the terms of the CeCILL
license as circulated by CEA, CNRS and INRIA at the following URL
"http://www.cecill.info".

As a counterpart to the access to the source code and rights to copy,
modify and redistribute granted by the license, users are provided only
with a limited warranty and the software's author, the holder of the
economic rights, and the successive licensors have only limited
liability.

In this respect, the user's attention is drawn to the risks associated
with loading, using, modifying and/or developing or reproducing the
software by the user in light of its specific status of free software,
that may mean that it is complicated to manipulate, and that also
therefore means that it is reserved for developers and experienced
professionals having in-depth computer knowledge. Users are therefore
encouraged to load and test the software's suitability as regards their
requirements in conditions enabling the security of their systems and/or
data to be ensured and, more generally, to use and operate it in the
same conditions as regards security.

The fact that you are presently reading this means that you have had
knowledge of the CeCILL license and that you accept its terms.
Property svn:keywords set to Revision Author Date

File size: 30.6 KB

Rev	Line
[622]	1	#-Q- curie ######################
	2	#-Q- curie ## CURIE TGCC/CEA ##
	3	#-Q- curie ######################
[906]	4	#-Q- curie #MSUB -r TS # Nom du job
[1468]	5	#-Q- curie #MSUB -o TS.out_%I
	6	#-Q- curie #MSUB -e TS.out_%I
[622]	7	#-Q- curie #MSUB -n 1 # Reservation du processus
[880]	8	#-Q- curie #MSUB -T 80000 # Limite de temps elapsed du job
[1154]	9	#-Q- curie #MSUB -q ::default_node::
[1274]	10	#-Q- curie #MSUB -c ::default_core::
[704]	11	#-Q- curie #MSUB -Q normal
[837]	12	#-Q- curie #MSUB -A ::default_project::
[681]	13	#-Q- curie set +x
[1433]	14	#-Q- irene ######################
	15	#-Q- irene ## IRENE TGCC/CEA ##
	16	#-Q- irene ######################
	17	#-Q- irene #MSUB -r TS # Job name
[1468]	18	#-Q- irene #MSUB -o TS.out_%I
	19	#-Q- irene #MSUB -e TS.out_%I
[1433]	20	#-Q- irene #MSUB -n 1 # Number of cores
	21	#-Q- irene #MSUB -T 80000 # Maximum elapsed time
[1468]	22	#-Q- irene #MSUB -q ::default_node::
	23	#-Q- irene #MSUB -c ::default_core::
[1433]	24	#-Q- irene #MSUB -Q normal
[1468]	25	#-Q- irene #MSUB -A ::default_post_project::
[1460]	26	#-Q- irene #MSUB -m store,work,scratch
[1433]	27	#-Q- irene set +x
[770]	28	#-Q- ada #!/bin/ksh
	29	#-Q- ada #######################
[929]	30	#-Q- ada ## ADA IDRIS ##
[770]	31	#-Q- ada #######################
[1409]	32	#-Q- ada # @ job_type = mpich
[848]	33	#-Q- ada # @ requirements = (Feature == "prepost")
[770]	34	#-Q- ada # Temps Elapsed max. d'une requete hh:mm:ss
	35	#-Q- ada # @ wall_clock_limit = 20:00:00
	36	#-Q- ada # Nom du travail LoadLeveler
	37	#-Q- ada # @ job_name = TS
	38	#-Q- ada # Fichier de sortie standard du travail
	39	#-Q- ada # @ output = $(job_name).$(jobid)
	40	#-Q- ada # Fichier de sortie d'erreur du travail
	41	#-Q- ada # @ error = $(job_name).$(jobid)
	42	#-Q- ada # pour recevoir un mail en cas de depassement du temps Elapsed (ou autre pb.)
	43	#-Q- ada # @ notification = error
[1290]	44	#-Q- ada # @ environment = $DEBUG_debug ; $BigBrother ; $postProcessingStopLevel ; $MODIPSL ; $libIGCM ; $libIGCM_SX ; $SUBMIT_DIR ; $POST_DIR ; $Script_Post_Output ; $MASTER ; $RebuildFrequency ; $DateBegin ; $PeriodDateEnd ; $StandAlone ; $CompletedFlag ; $TsTask ; $CompToRead ; $FlagToRead ; $RESOL_ATM ; $RESOL_OCE ; $RESOL_ICE ; $RESOL_MBG ; $RESOL_SRF ; $RESOL_SBG ; wall_clock_limit=$(wall_clock_limit)
[770]	45	#-Q- ada # @ queue
[287]	46	#-Q- lxiv8 ######################
	47	#-Q- lxiv8 ## OBELIX LSCE ##
	48	#-Q- lxiv8 ######################
	49	#-Q- lxiv8 #PBS -N TS
	50	#-Q- lxiv8 #PBS -m a
	51	#-Q- lxiv8 #PBS -j oe
	52	#-Q- lxiv8 #PBS -q medium
	53	#-Q- lxiv8 #PBS -o TS.$$
	54	#-Q- lxiv8 #PBS -S /bin/ksh
[1184]	55	#-Q- ifort_CICLAD ######################
	56	#-Q- ifort_CICLAD ## CICLAD IPSL ##
	57	#-Q- ifort_CICLAD ######################
	58	#-Q- ifort_CICLAD #PBS -N TS
	59	#-Q- ifort_CICLAD #PBS -m a
	60	#-Q- ifort_CICLAD #PBS -j oe
[1314]	61	#-Q- ifort_CICLAD ###PBS -q h12 # Queue for 12 hours at ciclad only
[1184]	62	#-Q- ifort_CICLAD #PBS -S /bin/ksh
[2]	63	#-Q- default #!/bin/ksh
	64	#-Q- default ##################
	65	#-Q- default ## DEFAULT HOST ##
	66	#-Q- default ##################
	67
[373]	68	#**************************************************************
	69	# Author: Sebastien Denvil
	70	# Contact: Sebastien.Denvil__at__ipsl.jussieu.fr
	71	# $Revision:: $ Revision of last commit
	72	# $Author:: $ Author of last commit
	73	# $Date:: $ Date of last commit
[108]	74	# IPSL (2006)
	75	# This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
[373]	76	#
	77	#**************************************************************
[108]	78
	79	#set -eu
[2]	80	#set -vx
	81
	82	date
	83
[770]	84	#-Q- ada export OMP_NUM_THREADS=1
[2]	85
[1356]	86	#D- Task type DO NOT CHANGE (computing, post-processing or checking)
[712]	87	TaskType=post-processing
	88
[2]	89	########################################################################
	90
	91	#D- Flag to determine if this job in a standalone mode
	92	#D- Default : value from AA_job if any
	93	StandAlone=${StandAlone:=true}
	94
[554]	95	#D- Path to libIGCM
[206]	96	#D- Default : value from AA_job if any
[229]	97	libIGCM=${libIGCM:=::modipsl::/libIGCM}
[206]	98
	99	#D- Flag to determine allready produced time series. Empty if you start from the beginning
[213]	100	#D- Default : Value from AA_job if any. Usefull in StandAlone case if you want to continue a time series
[209]	101	CompletedFlag=${CompletedFlag:=}
[135]	102
[213]	103	#D- Flag to determine task type this script will perform.
	104	#D- Possible Value : 2D, 3D, Chunck2D, Chunck3D
	105	#D- Default : Value from AA_job if any. Usefull in StandAlone case.
[257]	106	TsTask=${TsTask:=2D}
[267]	107	if [ X${TsTask} = Xempty ] ; then
[785]	108	TsTask=""
[267]	109	fi
[213]	110
	111	#D- Flag to determine if rebuild process has been performed asynchronously
[906]	112	#D- Possible value true or false.
[213]	113	#D- If true raw files has allready been patched by rebuild job
	114	#D- If false create_ts will patch the raw files
	115	#D- Default : Value from AA_job if any. Usefull in StandAlone case.
	116	RebuildFrequency=${RebuildFrequency:=true}
	117
[2]	118	#D- Flag to determine job's output directory
	119	#D- Default : value from libIGCM_post.ksh if any
[370]	120	POST_DIR=${POST_DIR:=${PBS_O_WORKDIR:=$(pwd)}}
[2]	121
	122	#D- Increased verbosity (1, 2, 3)
	123	#D- Default : value from AA_job if any
	124	Verbosity=${Verbosity:=3}
	125
	126	#D- Low level debug : to bypass lib test checks and stack construction
	127	#D- Default : value from AA_job if any
	128	DEBUG_debug=${DEBUG_debug:=false}
	129
	130	#D- TEMPORARY Flag to determine atmospheric resolution
	131	#D- Default : value from atmospheric driver if any
[254]	132	RESOL_ATM=ALL
[2]	133
[257]	134	#D- Flag to determine surface resolution
	135	#D- Default : value from surface driver if any
	136	RESOL_SRF=ALL
	137
	138	#D- Flag to determine surface resolution
	139	#D- Default : value from surface driver if any
	140	RESOL_SBG=ALL
	141
[2]	142	#D- TEMPORARY Flag to determine ocean resolution
	143	#D- Default : value from ocean driver if any
	144	RESOL_OCE=${RESOL_OCE:=ORCA2}
	145
	146	#D- TEMPORARY Flag to determine ice resolution
	147	#D- Default : value from ice driver if any
[1326]	148	RESOL_ICE=${RESOL_OCE:=ORCA2}
[2]	149
[178]	150	#D- TEMPORARY Flag to determine marine biogeochemistry resolution
	151	#D- Default : value from ice driver if any
[1326]	152	RESOL_MBG=${RESOL_OCE:=ORCA2}
[178]	153
[2]	154	########################################################################
	155
[108]	156	. ${libIGCM}/libIGCM_debug/libIGCM_debug.ksh
	157	. ${libIGCM}/libIGCM_card/libIGCM_card.ksh
	158	. ${libIGCM}/libIGCM_date/libIGCM_date.ksh
	159	#-------
	160	. ${libIGCM}/libIGCM_sys/libIGCM_sys.ksh
[731]	161	. ${libIGCM}/libIGCM_config/libIGCM_config.ksh
[832]	162	#-------
[1192]	163	IGCM_sys_MkdirWork ${RUN_DIR_PATH}
	164	IGCM_sys_Cd ${RUN_DIR_PATH}
	165	#-------
[832]	166	( ${DEBUG_debug} ) && IGCM_debug_Check
	167	( ${DEBUG_debug} ) && IGCM_card_Check
	168	( ${DEBUG_debug} ) && IGCM_date_Check
[2]	169
	170	########################################################################
	171
	172	#set -vx
	173
	174	# Useful cleaning function
[906]	175	MENAGE ()
[81]	176	{
[1306]	177	typeset ExistFlag
	178	ExistFlag=$( IGCM_sys_RshArchive "[ -f $1/$3 ] && echo 0 \|\| echo 1" )
	179	[ ${ExistFlag} -eq 0 ] && ( IGCM_sys_RshArchive rm -f $1/$2 ;)
[2]	180	}
	181
	182	########################################################################
	183
[436]	184	# ------------------------------------------------------------------
	185	# Test if all was right before proceeding further
	186	# ------------------------------------------------------------------
[1206]	187	IGCM_debug_Verif_Exit
[436]	188
[2]	189	if [ ${StandAlone} = true ] ; then
[785]	190	CARD_DIR=${SUBMIT_DIR}
[2]	191	else
[785]	192	CARD_DIR=${RUN_DIR_PATH}
[903]	193
[785]	194	IGCM_sys_Get_Master ${SUBMIT_DIR}/config.card ${RUN_DIR_PATH}
[903]	195
[785]	196	IGCM_sys_Get_Master ${SUBMIT_DIR}/run.card ${RUN_DIR_PATH}
[903]	197
[785]	198	IGCM_sys_Get_Master ${SUBMIT_DIR}/COMP ${RUN_DIR_PATH}
[903]	199
[785]	200	IGCM_sys_Get_Master ${SUBMIT_DIR}/POST ${RUN_DIR_PATH}
[2]	201	fi
	202
[591]	203	#==================================
[2]	204	# First of all
	205	#
[727]	206	# Read libIGCM compatibility version in config.card
	207	# Read UserChoices section
	208	# Read Ensemble section
	209	# Read Post section
	210	# Define all netcdf output directories
[591]	211	#==================================
[727]	212	IGCM_config_CommonConfiguration ${CARD_DIR}/config.card
[591]	213
[1198]	214	# ------------------------------------------------------------------
	215	# Activate BigBrother so as to supervise this job
	216	# ------------------------------------------------------------------
	217	IGCM_debug_BigBro_Initialize
	218
[591]	219	#==================================
[633]	220	# Define default value to keep compatibility with previous card: means before changes due to TGCC
	221	if [ X${PackDefault} = Xtrue ] ; then
	222	[ X${config_Post_PackFrequency} = X ] && config_Post_PackFrequency=${config_Post_RebuildFrequency}
	223	else
	224	config_Post_PackFrequency=NONE
	225	fi
[591]	226
	227	#==================================
	228	# If pack is active then PackFrequency overule the config_UserChoices_PeriodLength
	229	if ( [ ! X${config_Post_PackFrequency} = X${NULL_STR} ] && [ ! X${config_Post_PackFrequency} = XNONE ] ) ; then
	230	config_UserChoices_PeriodLength=${config_Post_PackFrequency}
	231	fi
	232
[2]	233	#
[149]	234	# Determine component to take care of depending on kind of task create_ts will perform
[2]	235	#
[392]	236	[ ${CompToRead} ] && set -A config_ListOfComponents ${CompToRead} \|\| IGCM_card_DefineArrayFromSection ${CARD_DIR}/config.card ListOfComponents
[2]	237	#
[149]	238	# Determine period for time series. Default : value from AA_job or from launch_TS if any
	239	#
[309]	240	[ X${CompletedFlag} = X ] \|\| CompletedFlag=$( IGCM_date_ConvertFormatToGregorian ${CompletedFlag} )
[301]	241
[2]	242	DateBegin=${DateBegin:=$( IGCM_date_ConvertFormatToGregorian ${config_UserChoices_DateBegin} )}
	243	#
	244	PeriodDateEnd=${PeriodDateEnd:=$( IGCM_date_ConvertFormatToGregorian ${config_UserChoices_DateEnd} )}
[149]	245	#
	246	# Determine Dimension of the produced time series : 2D or 3D
	247	if ( [ X${TsTask} = XChunck2D ] \|\| [ X${TsTask} = X2D ] ) ; then
[785]	248	Dimension=2D
[149]	249	elif ( [ X${TsTask} = XChunck3D ] \|\| [ X${TsTask} = X3D ] ) ; then
[785]	250	Dimension=3D
[149]	251	else
[785]	252	Dimension=""
[149]	253	fi
[2]	254
[441]	255	# ------------------------------------------------------------------
	256	#D- Test if all was right before entering the loop
	257	# ------------------------------------------------------------------
[1206]	258	IGCM_debug_Verif_Exit
[149]	259
[2]	260	########################################################################
	261	# Depending on requested time series fill following variables :
	262	#
	263	# - LISTE_FILE_${comp}[*]
	264	# - LISTE_VARS_${file}[*]
	265	# - LISTE_AXIS_${file}[*]
	266	# - LISTE_PATCH_${file}[*]
	267	########################################################################
	268
	269	#
	270	# For each selected component determine which files need post-processing
	271	#
	272	DoJob=false
	273	for comp in ${config_ListOfComponents[*]} ; do
[785]	274	#
	275	ActiveComp=false
	276	#
	277	IGCM_card_DefineArrayFromOption ${CARD_DIR}/config.card ListOfComponents ${comp}
	278	eval compname=\${config_ListOfComponents_${comp}[0]} > /dev/null 2>&1
	279	#
	280	card=${CARD_DIR}/COMP/${compname}.card
	281
	282	IGCM_card_DefineArrayFromOption ${card} OutputFiles List
	283	#
	284	ListFilesName=${compname}_OutputFiles_List
	285	eval FileName0=\${${ListFilesName}[0]} > /dev/null 2>&1
	286	#
	287	if [ X${FileName0} != X${NULL_STR} ] ; then
[2]	288	#
[785]	289	#IGCM_debug_Print 1 "Component : ${compname}"
[2]	290	#
[785]	291	# INITIALISATION
[2]	292	#
[785]	293	typeset i
	294	if [ ${FlagToRead} ] ; then
	295	((NbFiles=${FlagToRead}))
	296	((i=${NbFiles}))
	297	else
	298	eval NbFiles=\${#${ListFilesName}[@]} > /dev/null 2>&1
	299	i=2
	300	fi
[2]	301	#
[785]	302	until [ $i -gt $NbFiles ]; do
	303	#
	304	eval flag_post=\${${ListFilesName}[$i]} > /dev/null 2>&1
	305	#
	306	if [ X${flag_post} != XNONE ] ; then
	307	#
	308	# For each comp and each file who needs post-processing
	309	# determine which variables need time series.
	310	#
	311	FILE=$( echo ${flag_post} \| awk "-FPost_" '{print $2}' )
	312	IGCM_card_DefineArrayFromOption ${card} Post_${FILE} TimeSeriesVars${Dimension}
	313	IGCM_card_DefineVariableFromOption ${card} Post_${FILE} ChunckJob${Dimension}
	314	#
	315	# If TimeSeriesVars list is empty we skip
	316	#
	317	if [ $( eval echo \${${compname}_Post_${FILE}_TimeSeriesVars${Dimension}} ) = ${NULL_STR} ] ; then
	318	#
	319	IGCM_debug_Print 3 "Nothing to do for ${FILE} according to ${card}"
	320	IGCM_debug_Print 3 "TimeSeriesVars${Dimension} for ${FILE} is empty"
	321	(( i=i+3 ))
	322	continue
	323	#
	324	fi
	325	#
	326	# If files/variables has been switch off by ChunckJob${Dimension}=0 we skip
	327	#
	328	if [ X$( eval echo \${${compname}_Post_${FILE}_ChunckJob${Dimension}} ) = XOFF ] ; then
	329	IGCM_debug_Print 3 "Nothing to do for ${FILE} according to ${card}"
	330	IGCM_debug_Print 3 "ChunckJob${Dimension}=OFF for ${FILE} so it has been switched off"
	331	(( i=i+3 ))
	332	continue
	333	fi
	334	#
	335	# If we are not a Chunck type task, we skip files/variables handled by chunck task
	336	#
	337	if ( [ ! X${TsTask} = XChunck${Dimension} ] && [ ! X${TsTask} = X ] && [ ! XChunck${Dimension} = XChunck ] ) ; then
	338	if [ ! $( eval echo \${${compname}_Post_${FILE}_ChunckJob${Dimension}} ) = NONE ] ; then
	339	IGCM_debug_Print 3 "Nothing to do for ${FILE} TimeSeriesVars${Dimension} according to ${card}"
	340	IGCM_debug_Print 3 "Let another chunck task do the job $( eval echo \${${compname}_Post_${FILE}_ChunckJob${Dimension}} )"
	341	(( i=i+3 ))
	342	continue
	343	fi
	344	fi
	345	#
	346	# Now on we know we have things to do
	347	#
[236]	348
[785]	349	#
	350	# We need coordinates axis to properly fill produced time series
	351	#
	352	IGCM_card_DefineArrayFromOption ${card} Post_${FILE} GatherWithInternal
	353	#
	354	if [ $( eval echo \${${compname}_Post_${FILE}_GatherWithInternal} ) = ${NULL_STR} ] ; then
	355	#
	356	IGCM_debug_Print 1 "ERROR !! We miss coordinate axis for ${FILE} according to ${card}"
	357	IGCM_debug_Print 1 "So no time series will be produced for ${FILE} !!"
	358	(( i=i+3 ))
	359	continue
	360	#
	361	fi
	362	#
	363	# We need LIST of variables not allready produced (useful for standalone mode)
	364	#
	365	FlagDir=$( echo ${FILE} \| awk -F "_" '{print $1}' )
	366	case ${FlagDir} in
	367	*Y) TS_Dir=TS_YE ;;
	368	*M) TS_Dir=TS_MO ;;
	369	*D) TS_Dir=TS_DA ;;
	370	HF) TS_Dir=TS_HF ;;
	371	3H) TS_Dir=TS_HF ;;
	372	INS) TS_Dir=TS_INS ;;
	373	esac
	374	#
	375	count=0
	376	for var in $( eval echo \${${compname}_Post_${FILE}_TimeSeriesVars${Dimension}[*]} ) ; do
	377	#
	378	DIRECTORY=${R_SAVE}/${comp}/Analyse/${TS_Dir}
	379	TestedFile=${config_UserChoices_JobName}_${DateBegin}_${PeriodDateEnd}_${FlagDir}_${var}.nc
	380	#
	381	IGCM_sys_TestFileArchive ${DIRECTORY}/${TestedFile}
	382	if [ ! $? = 0 ] ; then
	383	eval set +A LISTE_VARS_${FILE} \${LISTE_VARS_${FILE}[*]} ${var}
	384	ActiveComp=true
	385	(( count = count + 1 ))
	386	fi
	387	done
	388	#
	389	if [ ${count} -gt 0 ] ; then
	390	#
	391	eval set +A LISTE_FILE_${comp} \${LISTE_FILE_${comp}[*]} ${FILE}
	392	eval set +A LISTE_AXIS_${FILE} \${${compname}_Post_${FILE}_GatherWithInternal[*]}
	393	#
	394	if [ ! X${RebuildFrequency} = Xtrue ] ; then
	395	IGCM_card_DefineArrayFromOption ${card} Post_${FILE} Patches
	396	eval set +A LISTE_PATCHES_${FILE} \${${compname}_Post_${FILE}_Patches[*]}
	397	if [ X$( eval echo \${LISTE_PATCHES_${FILE}[0]} ) != X${NULL_STR} ]; then
	398	for Patch in $( eval echo \${LISTE_PATCHES_${FILE}[*]} ); do
	399	if [ Xload${Patch} != X ] ; then
	400	. ${libIGCM}/libIGCM_post/IGCM_${Patch}.ksh
	401	eval load${Patch}=loaded
	402	fi
	403	done
	404	fi
	405	fi
	406	fi
	407	fi
	408	(( i=i+3 ))
	409	done
	410	fi
	411	#
	412	if [ ${ActiveComp} = true ] ; then
	413	set +A ActiveListOfComponents ${ActiveListOfComponents[*]} ${comp}
	414	fi
[2]	415	done
	416
	417	set -A config_ListOfComponents ${ActiveListOfComponents[*]}
	418	[ ! -z "${ActiveListOfComponents}" ] && DoJob=true
	419
	420	########################################################################
	421	#
	422	# IS THERE SOME ALLREADY PRODUCED TIME SERIES ?
	423	# IF SO BRING THEM IN THE WORKING DIRECTORY
	424	#
	425	# PS : Keep in mind that IGCM_sys_Get here is a weak link :
	426	# - especially for IDRiS
	427	# - no special protection against "command too long" !
	428	# - libIGCM_sys should handle that or modify following code part
	429	#
	430	########################################################################
	431
	432	if [ ${StandAlone} != true ] ; then
[785]	433	Running_Flag=$( IGCM_sys_RshMaster "IGCM_card_DefineVariableFromOption \
	434	${SUBMIT_DIR}/run.card PostProcessing TimeSeriesRunning ; \
	435	echo \${run_PostProcessing_TimeSeriesRunning}" )
[2]	436
[785]	437	#if [ X${Running_Flag} = Xy ] ; then
	438	# # Time Series Job allready running
	439	# IGCM_debug_Print 1 "Time Series Job allready running exit"
	440	# exit
	441	#fi
	442	# Now run.card know we are running
[149]	443
	444
	445
	446
[785]	447	IGCM_sys_RshMaster IGCM_card_WriteOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesRunning "y"
[149]	448
	449
	450
	451
	452
[785]	453	# Get information from last execution
	454	#CompletedFlag=$( IGCM_sys_RshMaster \
	455	# "IGCM_card_DefineVariableFromOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesCompleted ;\
	456	# echo \${run_PostProcessing_TimeSeriesCompleted}" )
[2]	457	fi
	458
	459	# Is it the first submission or not ?
[149]	460
	461	if ( [ X${CompletedFlag} = X ] \|\| [ X${CompletedFlag} = X$( IGCM_date_AddDaysToGregorianDate ${DateBegin} -1 ) ] ) ; then
[785]	462	# First Time Series Submission
	463	FIRST_PASS=TRUE
	464	Length=$( IGCM_date_DaysInPreviousPeriod ${DateBegin} ${config_UserChoices_PeriodLength} begin)
	465	DATE_FIN_JOB_B=$( IGCM_date_AddDaysToGregorianDate ${DateBegin} $(( Length - 1 )) )
[2]	466	else
[785]	467	# Last Completed Time Series
	468	FIRST_PASS=FALSE
	469	DATE_FIN_JOB_B=${CompletedFlag}
	470	date=${DateBegin}_${DATE_FIN_JOB_B}
[1038]	471	#
[785]	472	for comp in ${config_ListOfComponents[*]} ; do
	473	#
	474	IGCM_sys_Mkdir ${RUN_DIR_PATH}/${comp}
	475	#
	476	IGCM_sys_Cd ${RUN_DIR_PATH}/${comp}
	477	#
	478	unset liste_file
	479	#
	480	file_path=${R_SAVE}/${comp}/Analyse
	481	#
	482	i=0
	483	file=${NULL_STR}
	484	#
	485	for file in $( eval echo \${LISTE_FILE_${comp}[*]} ); do
	486	#
	487	# Determine in which output can we find file
	488	# IE : Analyse/TS or Analyse/DA2TS or ...
	489	#
	490	FlagDir=$( echo ${file} \| awk -F "_" '{print $1}' )
	491	case ${FlagDir} in
	492	*Y) TS_Dir=TS_YE ;;
	493	*M) TS_Dir=TS_MO ;;
	494	*D) TS_Dir=TS_DA ;;
	495	HF) TS_Dir=TS_HF ;;
	496	3H) TS_Dir=TS_HF ;;
	497	INS) TS_Dir=TS_INS ;;
	498	esac
	499	for var in $( eval echo \${LISTE_VARS_${file}[*]} ) ; do
[2]	500	#
[1033]	501	IGCM_sys_TestFileArchive ${file_path}/${TS_Dir}/${config_UserChoices_JobName}_${date}_${FlagDir}_${var}.nc
	502	if [ $? = 0 ] ; then
	503	liste_file[${#liste_file[*]}]=${file_path}/${TS_Dir}/${config_UserChoices_JobName}_${date}_${FlagDir}_${var}.nc
	504	else
	505	IGCM_debug_Print 1 "WARNING ${file_path}/${TS_Dir}/${config_UserChoices_JobName}_${date}_${FlagDir}_${var}.nc dont exist"
	506	fi
[785]	507	#
	508	done
[2]	509	done
[785]	510	if [ X${file} != X${NULL_STR} ] ; then
	511	IGCM_sys_Get /l liste_file[*] ${RUN_DIR_PATH}/${comp}
[1206]	512	IGCM_debug_Verif_Exit
[785]	513	fi
	514	done
	515	#
	516	LEVEL=0
	517	#
[2]	518	fi
	519
	520	########################################################################
	521	#
	522	# DEFINE LOOP PARAMETERS
	523	#
	524	########################################################################
	525
	526	DATE_COUNT=${DATE_FIN_JOB_B}
	527	DATE_FIN_JOB_B_LOOP=${DATE_FIN_JOB_B}
	528
	529	( [ -n "${LEVEL}" ] && [ ${LEVEL} -eq 0 ] ) && NBRE_FILE_TOT=0 \|\| NBRE_FILE_TOT=1
	530
	531	if [ ${DoJob} = true ] ; then
[785]	532	while [ ${DATE_COUNT} -lt ${PeriodDateEnd} ] ; do
	533	(( NBRE_FILE_TOT = NBRE_FILE_TOT + 1 ))
	534	Length=$( IGCM_date_DaysInNextPeriod ${DATE_COUNT} ${config_UserChoices_PeriodLength} )
	535	DATE_COUNT=$( IGCM_date_AddDaysToGregorianDate ${DATE_COUNT} ${Length} )
	536	done
[2]	537	else
[785]	538	NBRE_FILE_TOT=0
[2]	539	fi
	540
	541	if [ ${NBRE_FILE_TOT} -le ${UNIX_MAX_LIMIT} ] ; then
[906]	542	NBRE_TOUR=1
[2]	543	elif [ ${NBRE_FILE_TOT}%${UNIX_MAX_LIMIT} -eq 0 ] ; then
[906]	544	let NBRE_TOUR=${NBRE_FILE_TOT}/${UNIX_MAX_LIMIT}
[2]	545	else
[785]	546	let NBRE_TOUR=${NBRE_FILE_TOT}/${UNIX_MAX_LIMIT}+1
[2]	547	fi
	548
	549	CURRENT_LOOP=1
	550	# THIS LOOP GIVES PARAMETERS FOR THE DATE'S LOOP ie : DATE_COURANTE, DATE_FIN.
[906]	551	# WE HAVE 3 LEVELS
[2]	552	# - LEVEL 1 : JUST ONE LOOP TO COMPLETE
	553	# - LEVEL 2 : WE ARE IN THE "DEEP" LOOP
	554	# - LEVEL 3 : WE ARE IN THE LAST LOOP
	555
	556	while [ ${CURRENT_LOOP} -le ${NBRE_TOUR} ] ; do
	557
[906]	558	if [ ${NBRE_FILE_TOT} -le ${UNIX_MAX_LIMIT} ] ; then
[785]	559	# - LEVEL 1 : JUST ONE LOOP TO COMPLETE
	560	NBRE_FILE_LOOP=${NBRE_FILE_TOT}
[2]	561
[785]	562	if [ ${FIRST_PASS} = TRUE ] ; then
	563	DATE_COURANTE=${DATE_FIN_JOB_B_LOOP}
	564	else
	565	Length=$( IGCM_date_DaysInNextPeriod ${DATE_FIN_JOB_B_LOOP} ${config_UserChoices_PeriodLength} )
	566	DATE_COURANTE=$( IGCM_date_AddDaysToGregorianDate ${DATE_FIN_JOB_B_LOOP} ${Length} )
	567	fi
[2]	568
[785]	569	DATE_FIN=${PeriodDateEnd}
	570	DATE_FIN_JOB_B_LOOP_PREC=${DATE_FIN_JOB_B_LOOP}
[2]	571
[785]	572	[ -n "${DATE_LOOP}" ] && DATE_FIN_JOB_B_LOOP=${DATE_LOOP}
	573	[ -n "${LEVEL}" ] && FLAG_B="TRUE" \|\| FLAG_B="FALSE"
[2]	574
[785]	575	LEVEL=1
[906]	576	DEPOT="TRUE"
[785]	577	elif [ ${CURRENT_LOOP} -ne ${NBRE_TOUR} ] ; then
	578	# - LEVEL 2 : WE ARE IN THE "DEEP" LOOP
	579	NBRE_FILE_LOOP=${UNIX_MAX_LIMIT}
[2]	580
[785]	581	if [ ! ${FIRST_PASS} = TRUE ] && [ ${CURRENT_LOOP} -eq 1 ] ; then
	582	Length=$( IGCM_date_DaysInNextPeriod ${DATE_FIN_JOB_B_LOOP} ${config_UserChoices_PeriodLength} )
	583	DATE_COURANTE=$( IGCM_date_AddDaysToGregorianDate ${DATE_FIN_JOB_B_LOOP} ${Length} )
	584	elif [ -z "${DATE_COURANTE}" ] ; then
	585	DATE_COURANTE=${DATE_FIN_JOB_B}
[2]	586	else
[785]	587	Length=$( IGCM_date_DaysInNextPeriod ${DATE_FIN} ${config_UserChoices_PeriodLength} )
	588	DATE_COURANTE=$( IGCM_date_AddDaysToGregorianDate ${DATE_FIN} ${Length} )
	589	fi
[2]	590
[785]	591	(( TotLength = 0 ))
	592	COMPTEUR=1
	593	DATE_LOOP=${DATE_COURANTE}
[2]	594
[785]	595	while [ ${COMPTEUR} -lt ${NBRE_FILE_LOOP} ] ; do
	596	#
	597	Length=$( IGCM_date_DaysInNextPeriod ${DATE_LOOP} ${config_UserChoices_PeriodLength} )
	598	DATE_LOOP=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} ${Length} )
	599	(( TotLength = TotLength + Length ))
	600	(( COMPTEUR = COMPTEUR + 1 ))
	601	#
	602	done # while [ ${COMPTEUR} -le ${NBRE_FILE_LOOP} ] done
	603	DATE_FIN=$( IGCM_date_AddDaysToGregorianDate ${DATE_COURANTE} ${TotLength} )
[2]	604
[785]	605	[ -n "${DATE_LOOP}" ] && DATE_FIN_JOB_B_LOOP_PREC=${DATE_FIN_JOB_B_LOOP} ; DATE_FIN_JOB_B_LOOP=${DATE_LOOP}
	606	[ -n "${LEVEL}" ] && [ ${LEVEL} -eq 0 ] && FLAG_B="TRUE" \|\| [ ${CURRENT_LOOP} -gt 1 ] && FLAG_B="TRUE" \|\| FLAG_B="FALSE"
	607	DEPOT="FALSE"
	608	LEVEL=2
	609	else
	610	# - LEVEL 3 : WE ARE IN THE LAST LOOP
	611	NBRE_FILE_LOOP=$(( ${NBRE_FILE_TOT} % ( ${UNIX_MAX_LIMIT} * ( ${CURRENT_LOOP} - 1 ) ) ))
	612	[ ${NBRE_FILE_LOOP} -eq 0 ] && NBRE_FILE_LOOP=${UNIX_MAX_LIMIT}
[2]	613
[785]	614	Length=$( IGCM_date_DaysInNextPeriod ${DATE_FIN} ${config_UserChoices_PeriodLength} )
	615	DATE_COURANTE=$(IGCM_date_AddDaysToGregorianDate ${DATE_FIN} ${Length} )
	616	DATE_FIN=${PeriodDateEnd}
	617	[ -n "${DATE_LOOP}" ] && DATE_FIN_JOB_B_LOOP_PREC=${DATE_FIN_JOB_B_LOOP} ; DATE_FIN_JOB_B_LOOP=${DATE_LOOP}
[2]	618
[785]	619	LEVEL=3
	620	DEPOT="TRUE"
	621	FLAG_B="TRUE"
	622	fi
[2]	623
[785]	624	# THEN FOR EACH FILE TYPE WE LOOP BETWEEN DATE_COURANTE AND DATE_FIN.
	625	for comp in ${config_ListOfComponents[*]} ; do
	626	#
	627	IGCM_sys_Mkdir ${RUN_DIR_PATH}/${comp}
	628	#
	629	IGCM_sys_Cd ${RUN_DIR_PATH}/${comp}
	630	#
	631	eval R_OUT_${comp}=${R_SAVE}/${comp}
	632	#
	633	i=0
	634	#
	635	for file in $( eval echo \${LISTE_FILE_${comp}[*]} ); do
	636	#
	637	# Determine in which output we can find file
	638	# IE : Output/MO or Output/DA or ...
	639	#
	640	FlagDir=$( echo ${file} \| awk -F "_" '{print $1}' )
	641	case ${FlagDir} in
	642	*Y) TS_Dir=TS_YE ; FreqDir=YE ;;
	643	*M) TS_Dir=TS_MO ; FreqDir=MO ;;
	644	*D) TS_Dir=TS_DA ; FreqDir=DA ;;
	645	HF) TS_Dir=TS_HF ; FreqDir=HF ;;
	646	3H) TS_Dir=TS_HF ; FreqDir=HF ;;
	647	INS) TS_Dir=TS_INS ; FreqDir=INS ;;
	648	*) IGCM_debug_Print 1 "Error in FlagDir=${FlagDir} for file ${file}."
[1205]	649	IGCM_debug_Exit "Job create_ts" ;;
[785]	650	esac
[2]	651
[785]	652	# Initialize array
	653	unset liste_file
	654	unset liste_file_tmp
	655	eval file_path=\${R_OUT_${comp}}/Output/${FreqDir}/
[2]	656
[785]	657	COMPTEUR=1
[817]	658	MissingFile=FALSE
[785]	659	DATE_LOOP=${DATE_COURANTE}
[2]	660
[785]	661	while [ ${COMPTEUR} -le ${NBRE_FILE_LOOP} ] ; do
	662	#
	663	Length1=$( IGCM_date_DaysInPreviousPeriod ${DATE_LOOP} ${config_UserChoices_PeriodLength} end)
	664	DATE_TAB=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} $(( 1 - Length1 )) )_${DATE_LOOP}
[2]	665
[785]	666	Length2=$( IGCM_date_DaysInNextPeriod ${DATE_LOOP} ${config_UserChoices_PeriodLength} )
	667	DATE_LOOP=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} ${Length2} )
[2]	668
[817]	669	IGCM_sys_TestFileArchive ${file_path}${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc
	670	if [ $? = 0 ] ; then
	671	liste_file[${#liste_file[*]}]=${file_path}${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc
	672	liste_file_tmp[${#liste_file_tmp[*]}]=${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc
	673	else
	674	IGCM_debug_Print 1 "WARNING ${file_path}${config_UserChoices_JobName}_${DATE_TAB}_${file}.nc dont exist"
	675	MissingFile=TRUE
	676	fi
[785]	677	(( COMPTEUR = COMPTEUR + 1 ))
	678	#
	679	done # while [ ${COMPTEUR} -le ${NBRE_FILE_LOOP} ] done
[817]	680
	681	# We skip this file type if one of them is missing.
	682	if [ "${MissingFile}" = "TRUE" ] ; then
	683	IGCM_debug_Print 1 "WARNING We skip ${file} type processing"
	684	continue
	685	fi
	686
[785]	687	DATE_LOOP=$( IGCM_date_AddDaysToGregorianDate ${DATE_LOOP} -${Length2} )
[2]	688
[785]	689	# Get selected files
	690	IGCM_sys_Get /l liste_file[*] ${RUN_DIR_PATH}/${comp}
[1206]	691	IGCM_debug_Verif_Exit
[2]	692
[785]	693	# Apply IGCM_Patch if needed
	694	if [ ! X${RebuildFrequency} = Xtrue ] ; then
	695	if [ X$( eval echo \${LISTE_PATCHES_${file}[0]} ) != X${NULL_STR} ]; then
[906]	696	for file_t in $( eval echo ${liste_file_tmp[*]} ); do
[785]	697	for Patch in $( eval echo \${LISTE_PATCHES_${file}[*]} ); do
	698	IGCM_${Patch} ${file_t}
[1206]	699	IGCM_debug_Verif_Exit
[785]	700	done
	701	done
	702	fi
	703	fi
[2]	704
[785]	705	# WE CAN CONCATENATE FILES AT THIS POINT
	706	[ "${FLAG_B}" = "TRUE" ] && DATE_BUILD_B=${DateBegin}_${DATE_FIN_JOB_B_LOOP_PREC}
	707	Length=$( IGCM_date_DaysInPreviousPeriod ${DATE_COURANTE} ${config_UserChoices_PeriodLength} end)
	708	DATE_BUILD1=$( IGCM_date_AddDaysToGregorianDate ${DATE_COURANTE} -$(( Length - 1 )) )
	709	DATE_BUILD=${DATE_BUILD1}_${DATE_FIN}
	710	DATE_BUILD_END=${DateBegin}_${DATE_FIN}
[514]	711
[785]	712	liste_coord=" "
	713	for axis in $( eval echo \${LISTE_AXIS_${file}[*]} ); do
	714	liste_coord=${liste_coord}${axis}","
	715	done
[287]	716
[785]	717	# Time axis must be the last dimension
	718	time_axis=${axis}
[388]	719
[785]	720	# SWITCH BETWEEN MODEL TO FIT PARTICULAR CASE
	721	for var in $( eval echo \${LISTE_VARS_${file}[*]} ) ; do
	722	#
[1033]	723	# We skip variables not in the first file of the list.
	724	# Seems overkill to test thel all... but will do if needed
	725	ncdump -hv ${var} ${liste_file_tmp[0]} > /dev/null 2>&1
	726	if [ ! $? = 0 ] ; then
	727	IGCM_debug_Print 1 "WARNING We skip ${var}, not present within ${liste_file_tmp[0]}"
	728	continue
	729	fi
	730	#
[785]	731	if [ "${FLAG_B}" = "TRUE" ] ; then
	732	# WE CONCATENATE WITH EXISTING FILES
	733	file1=${config_UserChoices_JobName}_${DATE_BUILD_B}_${FlagDir}_${var}.nc
	734	file_out=${config_UserChoices_JobName}_${DATE_BUILD_END}_${FlagDir}_${var}.nc
	735	IGCM_sys_ncrcat --hst -v ${liste_coord}${var} ${file1} ${liste_file_tmp[*]} ${file_out}
[1206]	736	IGCM_debug_Verif_Exit
[785]	737	#
	738	IGCM_sys_Rm ${file1}
	739	if [ ! "${DEPOT}" = "TRUE" ] ; then
	740	eval IGCM_sys_Put_Out ${file_out} \${R_OUT_${comp}}/Analyse/${TS_Dir}/${file_out}
	741	# "${file1} = file_before ?
	742	[ ! "${file1}" = "${config_UserChoices_JobName}_${DateBegin}_${DATE_FIN_JOB_B}_${FlagDir}_${var}.nc" ] && eval MENAGE \${R_OUT_${comp}}/Analyse/${TS_Dir} ${file1} ${file_out}
	743	fi
	744	else
	745	# OR NOT
[1047]	746	file_out=${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
	747	IGCM_sys_ncrcat --hst -v ${liste_coord}${var} ${liste_file_tmp[*]} ${file_out}
[1206]	748	IGCM_debug_Verif_Exit
[785]	749	fi
	750
	751	#
	752	if [ "${DEPOT}" = "TRUE" ] ; then
[1042]	753	# WE PUT FINAL FILE ON FILE SYSTEM
	754	[ "${FLAG_B}" = "FALSE" ] && file_out=${config_UserChoices_JobName}_${DATE_BUILD}_${FlagDir}_${var}.nc
	755	file_before=${config_UserChoices_JobName}_${DateBegin}_${DATE_FIN_JOB_B}_${FlagDir}_${var}.nc
[785]	756	#
[1042]	757	eval IGCM_sys_Put_Out ${file_out} \${R_OUT_${comp}}/Analyse/${TS_Dir}/${file_out}
	758	[ "${FLAG_B}" = "TRUE" ] && eval MENAGE \${R_OUT_${comp}}/Analyse/${TS_Dir} ${file_before} ${file_out}
	759	if [ ! "X${file1}" = "X" ] ; then
	760	[ ! "${file1}" = "${config_UserChoices_JobName}_${DateBegin}_${DATE_FIN_JOB_B}_${FlagDir}_${var}.nc" ] && eval MENAGE \${R_OUT_${comp}}/Analyse/${TS_Dir} ${file1} ${file_out}
	761	fi
	762	#
[1039]	763	[ -f ${file_out} ] && IGCM_sys_Rm ${file_out}
[785]	764	#
	765	fi
	766	#
[1038]	767	done # for var in ${liste_var} ; do
[785]	768	IGCM_sys_Rm ${liste_file_tmp[*]}
[1038]	769	done # for file in ${liste_file_brut} do
	770	done # for comp in ${config_ListOfComponents[*]} ; do
[785]	771	# GO AHEAD IN DEEPEST LOOP
	772	(( CURRENT_LOOP = CURRENT_LOOP + 1 ))
[1038]	773	done # while [ ${CURRENT_LOOP} -le ${NBRE_TOUR} ] do
[2]	774	# FINISH
	775
	776	# DODS copy
	777	for comp in ${config_ListOfComponents[*]} ; do
[1393]	778	for TS_Dir in TS_YE TS_MO TS_DA TS_HF TS_INS ; do
[785]	779	IGCM_sys_TestDirArchive ${R_SAVE}/${comp}/Analyse/${TS_Dir}
	780	[ $? = 0 ] && IGCM_sys_Put_Dods ${comp}/Analyse/${TS_Dir}
	781	done
[2]	782	done
	783
[1112]	784	export DEBUG_debug; export BigBrother ; export libIGCM; export SUBMIT_DIR; export POST_DIR; export Script_Output ;
	785	export StandAlone ; export RESOL_ATM; export RESOL_OCE ; export RESOL_ICE ; export RESOL_MBG ; export RESOL_SRF ;
	786	export RESOL_SBG
	787	listVarEnv="DEBUG_debug,BigBrother,libIGCM,SUBMIT_DIR,POST_DIR,Script_Post_Output,StandAlone,RESOL_ATM,RESOL_OCE,RESOL_ICE,RESOL_MBG,RESOL_SRF,RESOL_SBG"
[2]	788	export listVarEnv
	789
	790	#D-
	791	# --------------------------------------------------------------------
	792	#D- Test if all was right
	793	# --------------------------------------------------------------------
[1206]	794	IGCM_debug_Verif_Exit
[2]	795	# --------------------------------------------------------------------
	796	#D- Submit MONITORING
	797	# --------------------------------------------------------------------
[414]	798	if ( [ X${TsTask} = X2D ] \|\| [ X${TsTask} = X3D ] \|\| [ X${TsTask} = X ] ) ; then
[1439]	799	# From 2D and 3D it is the last one to finish that will submit the monitoring.
	800	if [ ! "X$( grep "IGCM_sys_RmRunDir" ${POST_DIR}/create_ts.${PeriodDateEnd}.[23]D.out 2> /dev/null )" = "X" ] ; then
	801	Script_Post_Output=monitoring.${PeriodDateEnd}
	802	IGCM_sys_QsubPost monitoring
	803	fi
[387]	804	fi
[2]	805
	806	#D-
	807	# --------------------------------------------------------------------
	808	#D- Update the run.card
	809	# --------------------------------------------------------------------
	810	if [ ${StandAlone} != true ] ; then
[785]	811	# Put in run.card end period of time series
	812	[ X"${DATE_FIN}" = X ] \|\| IGCM_sys_RshMaster IGCM_card_WriteOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesCompleted "${DATE_FIN}"
	813	# We have finish the job !
	814	IGCM_sys_RshMaster IGCM_card_WriteOption ${SUBMIT_DIR}/run.card PostProcessing TimeSeriesRunning "n"
[2]	815	fi
[203]	816
[590]	817	# Clean RUN_DIR_PATH (necessary for cesium and titane only)
[203]	818	IGCM_sys_RmRunDir -Rf ${RUN_DIR_PATH}
	819
[1198]	820	# ------------------------------------------------------------------
	821	# Finalize BigBrother to inform that the jobs end
	822	# ------------------------------------------------------------------
	823	IGCM_debug_BigBro_Finalize
	824
[2]	825	date

Note: See TracBrowser for help on using the repository browser.

Download in other formats:

Original Format