Context Navigation

libIGCM_sys_curie.ksh @ 1173

Last change on this file since 1173 was 1173, checked in by aclsce, 9 years ago

MPMD + MPI-OpenMP mode

Modified to fit with new mpirun assignment method (method changed beacuse of activation of hyperthreading on Curie nodes).
Use of classic method in case of 1 OMP thread.

Property licence set to
The following licence information concerns ONLY the libIGCM tools
==================================================================

Copyright © Centre National de la Recherche Scientifique CNRS
Commissariat à l'Énergie Atomique CEA

libIGCM : Library for Portable Models Computation of IGCM Group.

IGCM Group is the french IPSL Global Climate Model Group.

This library is a set of shell scripts and functions whose purpose is
the management of the initialization, the launch, the transfer of
output files, the post-processing and the monitoring of datas produce
by any numerical program on any plateforme.

This software is governed by the CeCILL license under French law and
abiding by the rules of distribution of free software. You can use,
modify and/ or redistribute the software under the terms of the CeCILL
license as circulated by CEA, CNRS and INRIA at the following URL
"http://www.cecill.info".

As a counterpart to the access to the source code and rights to copy,
modify and redistribute granted by the license, users are provided only
with a limited warranty and the software's author, the holder of the
economic rights, and the successive licensors have only limited
liability.

In this respect, the user's attention is drawn to the risks associated
with loading, using, modifying and/or developing or reproducing the
software by the user in light of its specific status of free software,
that may mean that it is complicated to manipulate, and that also
therefore means that it is reserved for developers and experienced
professionals having in-depth computer knowledge. Users are therefore
encouraged to load and test the software's suitability as regards their
requirements in conditions enabling the security of their systems and/or
data to be ensured and, more generally, to use and operate it in the
same conditions as regards security.

The fact that you are presently reading this means that you have had
knowledge of the CeCILL license and that you accept its terms.
Property svn:keywords set to Revision Author Date

File size: 93.7 KB

Rev	Line
[622]	1	#!/bin/ksh
	2
	3	#**************************************************************
	4	# Author: Sebastien Denvil, Martial Mancip, Arnaud Caubel
	5	# Contact: Arnaud.Caubel__at__lsce.ipsl.fr
	6	# $Revision:: $ Revision of last commit
	7	# $Author:: $ Author of last commit
	8	# $Date:: $ Date of last commit
	9	# IPSL (2006)
	10	# This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
	11	#
	12	#**************************************************************
	13
	14	#=========================================================
	15	# The documentation of this file can be automatically generated
[623]	16	# if you use the prefix #D- for comments to be extracted.
[622]	17	# Extract with command: cat lib* \| grep "^#D-" \| cut -c "4-"
	18	#=========================================================
	19
	20	#D-#==================================================
[623]	21	#D-LibIGCM_sys for Curie
[622]	22	#D-#==================================================
	23	#D-
[623]	24	#D- This ksh library if a layer under some usefull
[622]	25	#D-environment variables and shell commands.
	26	#D-All those definitions depend on host particularities.
	27	#D-It manages a stack mechanism and test validity of operations.
	28	#D-All function described bellow must be prefixed by IGCM_sys.
	29
	30	#====================================================
	31	# libIGCM_sys PARAMETERS
	32	#====================================================
	33
	34	#====================================================
	35	# set DEBUG_sys to true to output calls of function
	36	typeset -r DEBUG_sys=${DEBUG_sys:=true}
	37
	38	#====================================================
	39	# Turn in dry run mode ? (sys_Put_Rest, sys_Put_Out, sys_Get)
	40	typeset -r DRYRUN=${DRYRUN:=0}
	41
	42	# YOU MUST COMPILE YOUR EXE FILES FOR DRYRUN MODE !
	43	# -------------------------------------------------------------------------------------
	44	# \| DRYRUN= \| Date computations, \| sys_Get \| Exe \| sys_Put_Out; sys_Put_Rest \|
[809]	45	# \| \| Cp/Exe/param/files \| \| \| \|
	46	# \| \| Chmod Qsub \| \| \| \|
[622]	47	# -------------------------------------------------------------------------------------
	48	# \| 0 \| yes \| yes \| yes \| yes \|
	49	# -------------------------------------------------------------------------------------
	50	# \| 1 \| yes \| yes \| yes \| no \|
	51	# -------------------------------------------------------------------------------------
	52	# \| 2 \| yes \| yes \| no \| no \|
	53	# -------------------------------------------------------------------------------------
	54	# \| 3 \| yes \| no \| no \| no \|
	55	# -------------------------------------------------------------------------------------
	56
	57	#=====================================================
	58	# Global Variables :
	59	#=====================================================
	60	# Language : "fr" or "en"
	61	typeset -r MYLANG="fr"
	62
	63	#=====================================================
	64	# Host and user names
	65	# $hostname ou hostname
[1158]	66	typeset HOST=${HOST:=$( hostname )}
[622]	67	# $username ou whoami
[1158]	68	typeset LOGIN=${LOGIN:=$( whoami )}
[622]	69	# $hostname of the MASTER job
	70	typeset MASTER=curie
[880]	71	# add default project on curie
[900]	72	typeset DEFAULT_PROJECT=gen0826
[622]	73
	74	#D-
	75	#D-#==================================================
	76	#D-Program used in libIGCM
	77	#D-#==================================================
	78
[742]	79	# Submit command
[747]	80	typeset SUBMIT=${SUBMIT:=ccc_msub}
[622]	81	# rsync with path
	82	typeset -r RSYNC=/usr/bin/rsync
	83	# RSYNC_opt args to rsync
	84	typeset -r RSYNC_opt="-va"
	85	# ie storage filesystem
[623]	86	typeset -r STOREHOST=${MASTER}
[622]	87
	88	#====================================================
	89	# Source default environment
	90	#====================================================
	91	##. /etc/profile
	92
	93	#====================================================
	94	# Set environment tools (ferret, nco, cdo)
	95	#====================================================
[996]	96	if [ X${TaskType} = Xcomputing ] ; then
	97	. /ccc/cont003/home/dsm/p86ipsl/.atlas_env_netcdf4_curie_ksh > /dev/null 2>&1
	98	# to run with netcdf 3.6.3 ie compilation done before 17/2/2014
	99	# uncomment 2 lines :
	100	# module unload netcdf
	101	# module load netcdf/3.6.3
	102	export PATH=${PATH}:/ccc/cont003/home/dsm/p86ipsl/AddNoise/src_X64_CURIE/bin
[1026]	103	export PATH=${PATH}:/ccc/cont003/home/dsm/p86ipsl/AddPerturbation/src_X64_CURIE/bin
[996]	104	else
	105	. /ccc/cont003/home/dsm/p86ipsl/.atlas_env_netcdf4_curie_ksh > /dev/null 2>&1
[1166]	106	PCMDI_MP=/ccc/work/cont003/dsm/p86ipsl/PCMDI-MP
[996]	107	fi
[622]	108
	109	#====================================================
	110	# Specific for ocean additionnal diagnostic
	111	export FER_GO="$FER_GO /home/cont003/p86denv/IGCM_POST_UTIL/JNL /home/cont003/p86denv/GRAF /home/cont003/p86denv/GRAF/GO"
	112	export FER_PALETTE="$FER_PALETTE /home/cont003/p86denv/GRAF/PALET"
	113
	114	#====================================================
	115	# Host specific DIRECTORIES
	116	#====================================================
	117
[623]	118	# ============ CESIUM START ============ #
	119
[622]	120	#====================================================
[623]	121	#- Mirror libIGCM from titane to cesium if needed
	122	#ROOTSYS=$( echo ${libIGCM} \| gawk -F"/" '{print $3}' )
	123	#if [ ! ${ROOTSYS} = "home" ] ; then
	124	# typeset -r MirrorlibIGCM=${MirrorlibIGCM:=true}
	125	#else
	126	# typeset -r MirrorlibIGCM=${MirrorlibIGCM:=false}
	127	#fi
	128
	129	#====================================================
	130	#- libIGCM_POST
	131	#if ( ${MirrorlibIGCM} ) ; then
	132	# PATHlibIGCM=$( echo ${libIGCM} \| gawk -F"${LOGIN}/" '{print $2}' \| sed -e "s&/libIGCM&&" )
	133	# typeset -r libIGCM_POST=${HOME}/MIRROR/${PATHlibIGCM}/libIGCM
	134	#else
	135	# typeset -r libIGCM_POST=${libIGCM}
	136	#fi
	137
	138	# ============ CESIUM END ============ #
	139
	140	#====================================================
	141	#- MirrorlibIGCM for frontend
	142	typeset -r MirrorlibIGCM=${MirrorlibIGCM:=false}
	143
	144	#====================================================
	145	#- libIGCM_POST for frontend
	146	typeset -r libIGCM_POST=${libIGCM}
	147
	148	#====================================================
[622]	149	#- R_EXE (==> BIN_DIR = ${MODIPSL}/bin )
	150	typeset -r R_EXE="${MODIPSL}/bin"
	151
	152	#====================================================
	153	#- SUBMIT_DIR : submission dir
[1010]	154	typeset -x SUBMIT_DIR=${SUBMIT_DIR:=${BRIDGE_MSUB_PWD}}
[622]	155
	156	#====================================================
	157	#- IN
	158	typeset -r R_IN=${R_IN:=/ccc/work/cont003/dsm/p86ipsl/IGCM}
	159	typeset -r R_IN_ECMWF=${R_IN_ECMWF:=/ccc/work/cont003/dsm/p24data}
	160
	161	#====================================================
	162	#- RUN_DIR_PATH : Temporary working directory (=> TMP)
[850]	163	typeset -r RUN_DIR_PATH=${RUN_DIR_PATH:=${SCRATCHDIR}/RUN_DIR/${BRIDGE_MSUB_JOBID}_${$}}
[622]	164
	165	#====================================================
	166	#- HOST_MPIRUN_COMMAND
[808]	167	typeset -r HOST_MPIRUN_COMMAND=${HOST_MPIRUN_COMMAND:="/usr/bin/time ccc_mprun -E-K1"}
[622]	168
	169	#====================================================
	170	#- Max number of arguments passed to nco operator or demigration command
	171	UNIX_MAX_LIMIT=360
	172
[623]	173	#====================================================
	174	#- set PackDefault true on curie
	175	PackDefault=true
	176
[664]	177	#====================================================
	178	#- Number of core per node (max number of OpenMP task)
[622]	179	NUM_COREPERNODE=8
	180
[623]	181	#====================================================
	182	#- Default number of MPI task for IPSL coupled model
	183	#- required for backward compatibility
	184	#-
	185	DEFAULT_NUM_PROC_OCE=5
	186	DEFAULT_NUM_PROC_CPL=1
[637]	187	(( DEFAULT_NUM_PROC_ATM = BATCH_NUM_PROC_TOT - DEFAULT_NUM_PROC_OCE - DEFAULT_NUM_PROC_CPL ))
	188	DEFAULT_NUM_PROC_TOTAL=${BATCH_NUM_PROC_TOT}
[623]	189
[725]	190	#D-#==================================================
[1057]	191	#D-function IGCM_sys_defineArchives
	192	#D-* Purpose:
	193	#D-* Define ARCHIVE : Dedicated to large files
	194	#D-* Define STORAGE : Dedicated to small/medium files
	195	#D-* Define R_OUT : Output tree located on ARCHIVE
	196	#D-* Define R_FIG : Output tree located on STORAGE hosting figures (monitoring and atlas, and/or small files)
	197	#D-* Define R_BUF : Output tree located on SCRATCHDIR hosting files waiting for rebuild or pack processes
	198	#D-* if SpaceName=TEST everything is stored on SCRATCHDIR
[725]	199	#D-* Examples:
	200	#D-
[1057]	201	function IGCM_sys_defineArchives {
	202	IGCM_debug_PushStack "IGCM_sys_defineArchives"
[676]	203
[1060]	204	if [ ! X${config_UserChoices_ARCHIVE} = X ]; then
[1057]	205	#====================================================
	206	#- ARCHIVE (dedicated to large files)
[1063]	207	ARCHIVE=${config_UserChoices_ARCHIVE}
[1057]	208	else
	209	#====================================================
	210	#- ARCHIVE (dedicated to large files)
[1063]	211	ARCHIVE=${CCCSTOREDIR}
[1057]	212	fi
[725]	213
[1060]	214	if [ ! X${config_UserChoices_STORAGE} = X ]; then
[1057]	215	#====================================================
	216	#- STORAGE (dedicated to small/medium files)
[1063]	217	STORAGE=${config_UserChoices_STORAGE}
[1057]	218	else
	219	#====================================================
	220	#- STORAGE (dedicated to small/medium files)
[1063]	221	STORAGE=${CCCWORKDIR}
[1057]	222	fi
[725]	223
[1057]	224	if [ X${config_UserChoices_SpaceName} = XTEST ]; then
	225	#====================================================
	226	#- R_OUT
[1063]	227	R_OUT=${SCRATCHDIR}/IGCM_OUT
[1057]	228
	229	#====================================================
	230	#- R_FIG (hosting figures : monitoring and atlas, and/or small files)
[1063]	231	R_FIG=${SCRATCHDIR}/IGCM_OUT
[1057]	232
	233	IGCM_debug_Print 1 "SpaceName=TEST ==> OVERRULE destination path directories"
	234
	235	else
	236	#====================================================
	237	#- R_OUT
[1063]	238	R_OUT=${ARCHIVE}/IGCM_OUT
[1057]	239
	240	#====================================================
	241	#- R_FIG (hosting figures : monitoring and atlas, and/or small files)
[1063]	242	R_FIG=${STORAGE}/IGCM_OUT
[1057]	243	fi
	244
	245	#====================================================
	246	#- R_BUF (ONLY FOR double copy an scratch)
[1063]	247	R_BUF=${SCRATCHDIR}/IGCM_OUT
[1057]	248
	249	IGCM_debug_Print 1 "R_OUT has been defined = ${R_OUT}"
	250	IGCM_debug_Print 1 "R_BUF has been defined = ${R_BUF}"
	251	IGCM_debug_Print 1 "R_FIG has been defined = ${R_FIG}"
	252
	253	IGCM_debug_PopStack "IGCM_sys_defineArchives"
[725]	254	}
	255
[622]	256	#D-#==================================================
	257	#D-function IGCM_sys_RshMaster
[623]	258	#D-* Purpose: Connection to frontend machine.
[622]	259	#D-* Examples:
	260	#D-
	261	function IGCM_sys_RshMaster {
[623]	262	IGCM_debug_PushStack "IGCM_sys_RshMaster" $@
	263	/bin/ksh <<-EOF
[622]	264	export libIGCM=${libIGCM}
	265	export DEBUG_debug=${DEBUG_debug}
	266	. ${libIGCM}/libIGCM_debug/libIGCM_debug.ksh
	267	. ${libIGCM}/libIGCM_card/libIGCM_card.ksh
	268	${@}
	269	EOF
[623]	270	if [ $? -gt 0 ] ; then
	271	echo "IGCM_sys_RshMaster : erreur."
	272	IGCM_debug_Exit "IGCM_sys_RshMaster"
	273	fi
	274	IGCM_debug_PopStack "IGCM_sys_RshMaster"
[622]	275	}
	276
	277	#D-#==================================================
	278	#D-function IGCM_sys_RshArchive
	279	#D-* Purpose: Archive rsh command
	280	#D-* Examples:
	281	#D-
	282	function IGCM_sys_RshArchive {
[623]	283	IGCM_debug_PushStack "IGCM_sys_RshArchive" $@
	284	/bin/ksh <<-EOF
[622]	285	${@}
	286	EOF
[811]	287	status=$?
	288	if [ ${status} -gt 0 ] ; then
	289	IGCM_debug_Print 2 "IGCM_sys_RshArchive : command failed error code ${status}"
[623]	290	IGCM_debug_Exit "IGCM_sys_RshArchive"
	291	fi
	292	IGCM_debug_PopStack "IGCM_sys_RshArchive"
[622]	293	}
	294
	295	#D-#==================================================
[845]	296	#D-function IGCM_sys_RshArchive_NoError
	297	#D-* Purpose: Archive rsh command, without error
	298	#D-* used only in monitoring.job
	299	#D-* Examples:
	300	#D-
	301	function IGCM_sys_RshArchive_NoError {
	302	IGCM_debug_PushStack "IGCM_sys_RshArchive_NoError" $@
	303	/bin/ksh <<-EOF
	304	${@} 2> \dev\null
	305	EOF
	306	IGCM_debug_PopStack "IGCM_sys_RshArchive_NoError"
	307	}
	308
	309	#D-#==================================================
[622]	310	#D-function IGCM_sys_RshPost
	311	#D-* Purpose: Post-process rsh command
	312	#D-* Examples:
	313	#D-
	314	function IGCM_sys_RshPost {
[623]	315	IGCM_debug_PushStack "IGCM_sys_RshPost" $@
	316	if ( $DEBUG_sys ) ; then
	317	echo "IGCM_sys_RshPost :" $@
	318	fi
	319	# keep standard input (stdin) for the loop onto temporary file
[811]	320	cat >/tmp/tmp_IGCM_sys_RshPost_$$_${LOGNAME}
[623]	321
[811]	322	/bin/ksh </tmp/tmp_IGCM_sys_RshPost_$$_${LOGNAME}
[623]	323	if [ $? -gt 0 ] ; then
	324	echo "IGCM_sys_RshPost : erreur."
	325	IGCM_debug_Exit "IGCM_sys_RshPost"
	326	fi
	327	# delete temporary file
[811]	328	\rm /tmp/tmp_IGCM_sys_RshPost_$$_${LOGNAME}
[623]	329
	330	# ============ FRONTEND END ============ #
	331
	332	# ============ CESIUM START ============ #
	333	# typeset NB_ESSAI DELAI status i
	334	# if [ "X$( grep rebuild_from tmp_IGCM_sys_RshPost_$$ )" != "X" ] ; then
	335	# #little hack so that rebuild submission is done on titane not an cesium
	336	#
	337	# libIGCM_POST_sed=$( echo $libIGCM_POST \| sed 's/\//\\\//g' )
	338	# POST_DIR_sed=$( echo ${POST_DIR} \| sed 's/\//\\\//g' )
	339	# sed "s/IGCM_sys_QsubPost/IGCM_sys_Qsub/g" tmp_IGCM_sys_RshPost_$$ > tmp.txt
	340	# sed "s/ rebuild_fromWorkdir/ ${libIGCM_POST_sed}\/rebuild_fromWorkdir.job/g" tmp.txt > tmp_IGCM_sys_RshPost_$$
	341	# sed "s/ rebuild_fromArchive/ ${libIGCM_POST_sed}\/rebuild_fromArchive.job/g" tmp_IGCM_sys_RshPost_$$ > tmp.txt
	342	# sed "s/Script_Post_Output=/Script_Output=${POST_DIR_sed}\//g" tmp.txt > tmp_IGCM_sys_RshPost_$$
	343	# \mv tmp.txt tmp_IGCM_sys_RshPost_$$
	344	#
	345	# echo cat tmp_IGCM_sys_RshPost_$$ AFTER
	346	# cat tmp_IGCM_sys_RshPost_$$
	347	#
	348	# /bin/ksh <tmp_IGCM_sys_RshPost_$$
	349	# if [ $? -gt 0 ] ; then
	350	# echo "IGCM_sys_RshPost : erreur."
	351	# IGCM_debug_Exit "IGCM_sys_RshPost"
	352	# fi
	353	# # delete temporary file
	354	# \rm tmp_IGCM_sys_RshPost_$$
	355	#
	356	# else
	357	# # number of tentative
	358	# NB_ESSAI=10
	359	# # time delay between tentative
	360	# DELAI=10
	361	# i=0
	362	# while [ $i -ne $NB_ESSAI ] ; do
	363	# ssh -t titane996 ssh cesium /bin/ksh <tmp_IGCM_sys_RshPost_$$
	364	# status=$?
	365	# if [ ${status} -ne 0 ];
	366	# then
[785]	367	# sleep $DELAI
[623]	368	# else
[785]	369	# break
[623]	370	# fi
	371	# let i=$i+1
	372	# done
	373	# # delete temporary file
	374	# \rm tmp_IGCM_sys_RshPost_$$
	375	#
	376	# if [ ${status} -gt 0 ] ; then
	377	# echo "IGCM_sys_RshPost : erreur."
	378	# IGCM_debug_Exit "IGCM_sys_RshPost"
	379	# fi
	380	# fi
	381
	382	# ============ CESIUM END ============ #
	383
	384	IGCM_debug_PopStack "IGCM_sys_RshPost"
[622]	385	}
	386
	387	#D-#==================================================
	388	#D-function IGCM_sys_SendMail
	389	#D-* Purpose: Send mail when simulation is over
	390	#D-* Examples:
	391	#D-
	392	function IGCM_sys_SendMail {
[623]	393	IGCM_debug_PushStack "IGCM_sys_SendMail" $@
[922]	394
[623]	395	if ( $DEBUG_sys ) ; then
	396	echo "IGCM_sys_SendMail :" $@
	397	fi
[622]	398
[922]	399	if [ X${1} = XAccounting ] ; then
	400	status=Accounting
	401	mailText=jobAccounting.mail
	402	elif ( ${ExitFlag} ) ; then
[623]	403	status=failed
[922]	404	mailText=jobEnd.mail
[623]	405	else
	406	status=completed
[922]	407	mailText=jobEnd.mail
[623]	408	fi
[622]	409
[922]	410	# Update selected mail template
	411	while read -r line; do
	412	eval echo $line >> mail.txt ;
	413	done < ${libIGCM}/libIGCM_sys/${mailText}
[622]	414
[806]	415	if [ ! -z ${config_UserChoices_MailName} ] ; then
[922]	416	mail -s "${config_UserChoices_JobName} ${status}" ${config_UserChoices_MailName} < mail.txt
[806]	417	elif [ -f ~/.forward ] ; then
[922]	418	mail -s "${config_UserChoices_JobName} ${status}" $( cat ~/.forward ) < mail.txt
[806]	419	fi
[811]	420
[806]	421	sleep 10
[922]	422	rm -f mail.txt
[811]	423
[623]	424	IGCM_debug_PopStack "IGCM_sys_SendMail"
[622]	425	}
	426
	427	#D-#==================================================
	428	#D-function IGCM_sys_Mkdir
	429	#D-* Purpose: Master locale mkdir command
	430	#D-* Examples:
	431	#D-
	432	function IGCM_sys_Mkdir {
[623]	433	IGCM_debug_PushStack "IGCM_sys_Mkdir" $@
	434	if ( $DEBUG_sys ) ; then
	435	echo "IGCM_sys_Mkdir :" $@
	436	fi
	437	if [ ! -d ${1} ]; then
	438	\mkdir -p $1
	439	if [ $? -gt 0 ] ; then
	440	echo "IGCM_sys_Mkdir : erreur."
	441	IGCM_debug_Exit "IGCM_sys_Mkdir"
[622]	442	fi
[623]	443	fi
	444	# vérification :
	445	if [ ! -d ${1} ] ; then
	446	echo "IGCM_sys_Mkdir : erreur."
	447	IGCM_debug_Exit "IGCM_sys_Mkdir"
	448	fi
	449	IGCM_debug_PopStack "IGCM_sys_Mkdir"
[622]	450	}
	451
	452	#D-#==================================================
	453	#D-function IGCM_sys_MkdirArchive
	454	#D-* Purpose: Mkdir on Archive
	455	#D-* Examples:
	456	#D-
	457	function IGCM_sys_MkdirArchive {
[623]	458	IGCM_debug_PushStack "IGCM_sys_MkdirArchive" $@
	459	if ( $DEBUG_sys ) ; then
	460	echo "IGCM_sys_MkdirArchive :" $@
	461	fi
	462	#- creation de repertoire sur le serveur fichier
	463	if [ ! -d ${1} ]; then
	464	\mkdir -p $1
[811]	465	status=$?
	466
	467	if [ ${status} -gt 0 ] ; then
	468	IGCM_debug_Print 2 "IGCM_sys_MkdirArchive : mkdir failed error code ${status}"
[623]	469	IGCM_debug_Exit "IGCM_sys_MkdirArchive"
[622]	470	fi
[623]	471	fi
	472	IGCM_debug_PopStack "IGCM_sys_MkdirArchive"
[622]	473	}
	474
	475	#D-#==================================================
	476	#D-function IGCM_sys_MkdirWork
	477	#D-* Purpose: Mkdir on Work
	478	#D-* Examples:
	479	#D-
	480	function IGCM_sys_MkdirWork {
[623]	481	IGCM_debug_PushStack "IGCM_sys_MkdirWork" $@
	482	if ( $DEBUG_sys ) ; then
	483	echo "IGCM_sys_MkdirWork :" $@
	484	fi
	485	#- creation de repertoire sur le serveur fichier
	486	if [ ! -d ${1} ]; then
	487	\mkdir -p $1
	488	if [ $? -gt 0 ] ; then
	489	echo "IGCM_sys_MkdirWork : erreur."
	490	IGCM_debug_Exit "IGCM_sys_MkdirWork"
[622]	491	fi
[623]	492	fi
	493	IGCM_debug_PopStack "IGCM_sys_MkdirWork"
[622]	494	}
	495
	496	#D-#==================================================
	497	#D-function IGCM_sys_Cd
	498	#D-* Purpose: master cd command
	499	#D-* Examples:
	500	#D-
	501	function IGCM_sys_Cd {
[623]	502	IGCM_debug_PushStack "IGCM_sys_Cd" $@
	503	if ( $DEBUG_sys ) ; then
	504	echo "IGCM_sys_Cd :" $@
	505	fi
	506	\cd $1
	507	if [ $? -gt 0 ] ; then
	508	echo "IGCM_sys_Cd : erreur."
	509	IGCM_debug_Exit "IGCM_sys_Cd"
	510	fi
	511	IGCM_debug_PopStack "IGCM_sys_Cd"
[622]	512	}
	513
	514	#D-#==================================================
	515	#D-function IGCM_sys_Chmod
	516	#D-* Purpose: Chmod
	517	#D-* Examples:
	518	#D-
	519	function IGCM_sys_Chmod {
[1057]	520	IGCM_debug_PushStack "IGCM_sys_Chmod" $@
[623]	521	if ( $DEBUG_sys ) ; then
	522	echo "IGCM_sys_Chmod :" $@
	523	fi
[809]	524	\chmod $@
	525	if [ $? -gt 0 ] ; then
	526	echo "IGCM_sys_Chmod : erreur."
	527	IGCM_debug_Exit "IGCM_sys_Chmod"
[623]	528	fi
	529	IGCM_debug_PopStack "IGCM_sys_Chmod"
[622]	530	}
	531
	532	#D-#==================================================
	533	#D-function IGCM_sys_FileSize
	534	#D-* Purpose: Filesize
	535	#D-* Examples:
	536	#D-
	537	function IGCM_sys_FileSize {
[623]	538	IGCM_debug_PushStack "IGCM_sys_FileSize" $@
[622]	539
[623]	540	typeset sizeF
	541	set +A sizeF -- $( ls -la ${1} )
	542	if [ $? -gt 0 ] ; then
	543	IGCM_debug_Exit "IGCM_sys_FileSize"
	544	fi
	545	eval ${2}=${sizeF[4]}
[622]	546
[623]	547	IGCM_debug_PopStack "IGCM_sys_FileSize"
[622]	548	}
	549
	550	#D-#==================================================
	551	#D-function IGCM_sys_TestDir
	552	#D-* Purpose: Test Directory that must exists
	553	#D-* Examples:
	554	#D-
	555	function IGCM_sys_TestDir {
[623]	556	IGCM_debug_PushStack "IGCM_sys_TestDir" $@
	557	if ( $DEBUG_sys ) ; then
	558	echo "IGCM_sys_TestDir :" $@
	559	fi
	560	typeset ExistFlag
	561	ExistFlag=$( [ -d $1 ] && echo 0 \|\| echo 1 )
	562	IGCM_debug_PopStack "IGCM_sys_TestDir"
[622]	563
[623]	564	return ${ExistFlag}
[622]	565	}
	566
	567	#D-#==================================================
	568	#D-function IGCM_sys_TestDirArchive
	569	#D-* Purpose: Test Directory that must exists on Archive
	570	#D-* Examples:
	571	#D-
	572	function IGCM_sys_TestDirArchive {
[623]	573	IGCM_debug_PushStack "IGCM_sys_TestDirArchive" $@
	574	if ( $DEBUG_sys ) ; then
	575	echo "IGCM_sys_TestDirArchive :" $@
	576	fi
	577	typeset ExistFlag
	578	ExistFlag=$( [ -d $1 ] && echo 0 \|\| echo 1 )
	579	IGCM_debug_PopStack "IGCM_sys_TestDirArchive"
	580	return ${ExistFlag}
[622]	581	}
	582
	583	#D-#==================================================
[623]	584	#D-function IGCM_sys_IsFileArchived
	585	#D-* Purpose: Test file that must NOT EXISTS on Archive
	586	#D-* Examples:
	587	#D-
	588	function IGCM_sys_IsFileArchived {
	589	IGCM_debug_PushStack "IGCM_sys_IsFileArchived" $@
	590	if ( $DEBUG_sys ) ; then
	591	echo "IGCM_sys_IsFileArchived :" $@
	592	fi
	593	typeset IsArchivedFlag
[818]	594	IsArchivedFlag=$( [ X$( echo $@ \| grep ^\/ccc\/store ) != X ] && echo 0 \|\| echo 1 )
[623]	595	IGCM_debug_PopStack "IGCM_sys_IsFileArchived"
	596
	597	return ${IsArchivedFlag}
	598	}
	599
	600	#D-#==================================================
[622]	601	#D-function IGCM_sys_TestFileArchive
	602	#D-* Purpose: Test file that must NOT EXISTS on Archive
	603	#D-* Examples:
	604	#D-
	605	function IGCM_sys_TestFileArchive {
[623]	606	IGCM_debug_PushStack "IGCM_sys_TestFileArchive" $@
	607	typeset ExistFlag
	608	ExistFlag=$( [ -f $1 ] && echo 0 \|\| echo 1 )
	609	IGCM_debug_PopStack "IGCM_sys_TestFileArchive"
[622]	610
[623]	611	return ${ExistFlag}
[622]	612	}
	613
	614	#D-#==================================================
[623]	615	#D-function IGCM_sys_TestFileBuffer
	616	#D-* Purpose: Test file that must NOT EXISTS on Buffer
	617	#D-* Examples:
	618	#D-
	619	function IGCM_sys_TestFileBuffer {
	620	IGCM_debug_PushStack "IGCM_sys_TestFileBuffer" $@
	621	typeset ExistFlag
	622	ExistFlag=$( [ -f $1 ] && echo 0 \|\| echo 1 )
	623	IGCM_debug_PopStack "IGCM_sys_TestFileBuffer"
	624
	625	return ${ExistFlag}
	626	}
	627
	628	#D-#==================================================
[622]	629	#D-function IGCM_sys_CountFileArchive
	630	#D-* Purpose: Count files on Archive filesystem
	631	#D-* Examples:
	632	#D-
	633	function IGCM_sys_CountFileArchive {
[623]	634	IGCM_debug_PushStack "IGCM_sys_CountFileArchive" $@
	635	ls ${@} 2>/dev/null \| wc -l
	636	if [ $? -gt 0 ] ; then
	637	echo "IGCM_sys_CountFileArchive : erreur."
	638	fi
	639	IGCM_debug_PopStack "IGCM_sys_CountFileArchive"
[622]	640	}
	641
	642	#D-#==================================================
[623]	643	#D-function IGCM_sys_CountFileBuffer
	644	#D-* Purpose: Count files on Scratch filesystem
	645	#D-* Examples:
	646	#D-
	647	function IGCM_sys_CountFileBuffer {
	648	IGCM_debug_PushStack "IGCM_sys_CountFileBuffer" $@
	649	ls ${@} 2>/dev/null \| wc -l
	650	if [ $? -gt 0 ] ; then
	651	echo "IGCM_sys_CountFileBuffer : erreur."
	652	fi
	653	IGCM_debug_PopStack "IGCM_sys_CountFileBuffer"
	654	}
	655
	656	#D-#==================================================
[622]	657	#D-function IGCM_sys_Tree
	658	#D-* Purpose: Tree directories with files on ${ARCHIVE}
	659	#D-* Examples: IGCM_sys_Tree ${R_IN} ${R_OUT}
	660	#D-
	661	function IGCM_sys_Tree {
[623]	662	IGCM_debug_PushStack "IGCM_sys_Tree" $@
	663	if ( $DEBUG_sys ) ; then
	664	echo "IGCM_sys_Tree :" $@
	665	fi
[622]	666
[623]	667	\ls -lR ${@}
[622]	668
[623]	669	IGCM_debug_PopStack "IGCM_sys_Tree"
[622]	670	}
	671
	672	#D-#==================================================
	673	#D-function IGCM_sys_Tar
[623]	674	#D-* Purpose: master tar command
[622]	675	#D-* Examples:
	676	#D-
	677	function IGCM_sys_Tar {
[623]	678	IGCM_debug_PushStack "IGCM_sys_Tar" $@
	679	if ( $DEBUG_sys ) ; then
	680	echo "IGCM_sys_Tar :" $@
	681	fi
	682	\tar cf $@
	683	if [ $? -gt 0 ] ; then
	684	echo "IGCM_sys_Tar : erreur."
	685	IGCM_debug_Exit "IGCM_sys_Tar"
	686	fi
	687	IGCM_debug_PopStack "IGCM_sys_Tar"
[622]	688	}
	689
	690	#D-#==================================================
	691	#D-function IGCM_sys_UnTar
	692	#D-* Purpose: master un-tar command
	693	#D-* Examples:
	694	#D-
	695	function IGCM_sys_UnTar {
[623]	696	IGCM_debug_PushStack "IGCM_sys_UnTar" $@
	697	if ( $DEBUG_sys ) ; then
	698	echo "IGCM_sys_UnTar :" $@
	699	fi
	700	\tar xvf $1
	701	if [ $? -gt 0 ] ; then
	702	echo "IGCM_sys_UnTar : erreur."
	703	IGCM_debug_Exit "IGCM_sys_UnTar"
	704	fi
	705	IGCM_debug_PopStack "IGCM_sys_UnTar"
[622]	706	}
	707
	708	#D-#==================================================
	709	#D-function IGCM_sys_Qsub
	710	#D-* Purpose: Qsub new job
	711	#D-* Examples:
	712	#D-
	713	function IGCM_sys_Qsub {
[623]	714	IGCM_debug_PushStack "IGCM_sys_Qsub" $@
	715	if ( $DEBUG_sys ) ; then
	716	echo "IGCM_sys_Qsub :" $@
	717	fi
[811]	718	typeset options status
[923]	719	options="-o ${SUBMIT_DIR}/${Script_Output} -e ${SUBMIT_DIR}/${Script_Output}"
[622]	720
[948]	721	/usr/bin/ccc_msub ${options} $1 > /tmp/out_command.$$ 2>&1
[811]	722	status=$?
[948]	723
	724	cat /tmp/out_command.$$
[811]	725	if [ ${status} -gt 0 ] ; then
	726	IGCM_debug_Print 2 "IGCM_sys_Qsub ${options} $1 : error code ${status}"
[623]	727	IGCM_debug_Exit "IGCM_sys_Qsub"
[948]	728	else
	729	JobID=$( gawk {'print $4'} /tmp/out_command.$$ )
[623]	730	fi
	731	IGCM_debug_PopStack "IGCM_sys_Qsub"
[622]	732	}
	733
	734	#D-#==================================================
	735	#D-function IGCM_sys_QsubPost
	736	#D-* Purpose: Qsub new job on scalaire
	737	#D-* Examples:
	738	#D-
	739	function IGCM_sys_QsubPost {
[623]	740	IGCM_debug_PushStack "IGCM_sys_QsubPost" $@
	741	if ( $DEBUG_sys ) ; then
	742	echo "IGCM_sys_QsubPost :" $@
	743	fi
[923]	744	typeset options status
[880]	745	MY_PROJECT=$(echo ${BRIDGE_MSUB_PROJECT:=${DEFAULT_PROJECT}} \| cut -d@ -f1 )
[923]	746
[1062]	747	# EASIER TO DO THIS INSTEAD OF DUPLICATING libIGCM_sys_curie.ksh
	748	case $( hostname -s ) in
	749	curie*)
	750	options="-Q normal -A ${MY_PROJECT} -o ${POST_DIR}/${Script_Post_Output}.out -e ${POST_DIR}/${Script_Post_Output}.out";;
	751	airain*)
	752	options="-q ivybridge -A dsm -o ${POST_DIR}/${Script_Post_Output}.out -e ${POST_DIR}/${Script_Post_Output}.out"
	753	esac
	754
[948]	755	/usr/bin/ccc_msub ${options} ${libIGCM_POST}/$1.job > /tmp/out_command.$$ 2>&1
[811]	756	status=$?
[948]	757
	758	cat /tmp/out_command.$$
[811]	759	if [ ${status} -gt 0 ] ; then
[923]	760	IGCM_debug_Print 2 "IGCM_sys_QsubPost ${options} ${libIGCM_POST}/$1.job : error code ${status}"
[623]	761	IGCM_debug_Exit "IGCM_sys_QsubPost"
[948]	762	else
	763	JobID=$( gawk {'print $4'} /tmp/out_command.$$ )
[623]	764	fi
	765	IGCM_debug_PopStack "IGCM_sys_QsubPost"
[622]	766	}
	767
	768	#D-*************************
[623]	769	#D- File transfer functions
[622]	770	#D-*************************
	771	#D-
	772
	773	#D-#==================================================
	774	#D-function IGCM_sys_Rsync_out
	775	#D-* Purpose: treat return val of rsync
	776	#D-* Examples: IGCM_sys_Rsync_out out_RET_rsync
	777	#D- Error values and explanations can depend on your system version.
	778	function IGCM_sys_Rsync_out {
[719]	779	status=$1
	780	if [ ! $status ] ; then
[623]	781	echo "rsync error !"
	782	fi
[622]	783
[623]	784	if [ $MYLANG = "fr" ]; then
[719]	785	case $status in
[623]	786	0) return ;;
	787	1) echo "Erreur de rsync ; RERR_SYNTAX : "
	788	echo "Erreur de syntaxe ou d'utilisation."
	789	return;;
	790	2) echo "Erreur de rsync ; RERR_PROTOCOL : "
	791	echo "Incompatibilité de protocole."
	792	return;;
	793	3) echo "Erreur de rsync ; RERR_FILESELECT 3"
	794	echo "Erreurs lors de la sélection des fichiers d'entrée sortie et"
	795	echo "répertoires"
	796	return;;
	797	4) echo "Erreur de rsync ; RERR_UNSUPPORTED"
	798	echo "Action demandée non supportée : une tentative de manipulation de"
	799	echo "fichiers 64-bits sur une plate-forme qui ne les supporte pas a"
	800	echo "été faite ; ou une option qui est supportée par le client mais"
	801	echo "pas par le serveur a été spécifiée."
	802	return;;
	803	10) echo "Erreur de rsync ; RERR_SOCKETIO"
	804	echo "Erreur dans le socket d'entrée sortie"
	805	return;;
	806	11) echo "Erreur de rsync ; RERR_FILEIO"
	807	echo "Erreur d'entrée sortie fichier"
	808	return;;
	809	12) echo "Erreur de rsync ; RERR_STREAMIO"
	810	echo "Erreur dans flux de donnée du protocole rsync"
	811	return;;
	812	13) echo "Erreur de rsync ; RERR_MESSAGEIO"
	813	echo "Erreur avec les diagnostics du programme"
	814	return;;
	815	14) echo "Erreur de rsync ; RERR_IPC"
	816	echo "Erreur dans le code IPC"
	817	return;;
	818	20) echo "Erreur de rsync ; RERR_SIGNAL"
	819	echo "SIGUSR1 ou SIGINT reçu"
	820	return;;
	821	21) echo "Erreur de rsync ; RERR_WAITCHILD"
	822	echo "Une erreur retournée par waitpid()"
	823	return;;
	824	22) echo "Erreur de rsync ; RERR_MALLOC"
	825	echo "Erreur lors de l'allocation des tampons de mémoire de coeur"
	826	return;;
	827	23) echo ""
	828	echo "Erreur fichier inexistant"
	829	return;;
	830	30) echo "Erreur de rsync ; RERR_TIMEOUT"
	831	echo "Temps d'attente écoulé dans l'envoi/réception de données"
	832	return;;
[719]	833	*) echo "Erreur de rsync : code de retour de rsync inconnu :" $status
[623]	834	return;;
	835	esac
	836	elif [ $MYLANG = "en" ] ; then
[719]	837	case $status in
[623]	838	0) return;;
	839	1) echo "rsync error : Syntax or usage error "
	840	return;;
	841	2) echo "rsync error : Protocol incompatibility "
	842	return;;
	843	3) echo "rsync error : Errors selecting input/output files, dirs"
	844	return;;
	845	4) echo "rsync error : Requested action not supported: an attempt"
	846	echo "was made to manipulate 64-bit files on a platform that cannot support"
	847	echo "them; or an option was specified that is supported by the client and"
	848	echo "not by the server."
	849	return;;
	850	5) echo "rsync error : Error starting client-server protocol"
	851	return;;
	852	10) echo "rsync error : Error in socket I/O "
	853	return;;
	854	11) echo "rsync error : Error in file I/O "
	855	return;;
	856	12) echo "rsync error : Error in rsync protocol data stream "
	857	return;;
	858	13) echo "rsync error : Errors with program diagnostics "
	859	return;;
	860	14) echo "rsync error : Error in IPC code "
	861	return;;
	862	20) echo "rsync error : Received SIGUSR1 or SIGINT "
	863	return;;
	864	21) echo "rsync error : Some error returned by waitpid() "
	865	return;;
	866	22) echo "rsync error : Error allocating core memory buffers "
	867	return;;
	868	23) echo "rsync error : Partial transfer due to error"
	869	return;;
	870	24) echo "rsync error : Partial transfer due to vanished source files"
	871	return;;
	872	30) echo "rsync error : Timeout in data send/receive "
	873	return;;
[719]	874	*) echo "rsync error : return code of rsync unknown :" $status
[623]	875	return;;
	876	esac
	877	else
	878	echo "unknown language $MYLANG."
	879	return
	880	fi
[622]	881	}
	882
	883	#D-#==================================================
	884	#D-function IGCM_sys_Miror_libIGCM
[811]	885	#D-* Purpose: Mirror libIGCM PATH and lib to frontend
[622]	886	#D-* Examples:
	887	#D-
	888	function IGCM_sys_Mirror_libIGCM {
[623]	889	IGCM_debug_PushStack "IGCM_sys_Mirror_libIGCM"
	890	if ( $DEBUG_sys ) ; then
	891	echo "IGCM_sys_Mirror_libIGCM"
	892	fi
[622]	893
[773]	894	typeset status
[622]	895
[623]	896	mkdir -p ${HOME}/MIRROR/${PATHlibIGCM}
[622]	897
[938]	898	echo ${RSYNC} ${RSYNC_opt} ${libIGCM} ${HOME}/MIRROR/${PATHlibIGCM} > /tmp/out_command.$$ 2>&1
	899	${RSYNC} ${RSYNC_opt} ${libIGCM} ${HOME}/MIRROR/${PATHlibIGCM} >> /tmp/out_command.$$ 2>&1
[715]	900	status=$?
[622]	901
[715]	902	if [ ${status} -gt 0 ] ; then
[811]	903	echo "IGCM_sys_Mirror_libIGCM Warning : no libIGCM on frontend."
[938]	904	cat /tmp/out_command.$$
[623]	905	fi
	906	IGCM_debug_PopStack "IGCM_sys_Mirror_libIGCM"
[622]	907	}
	908
	909	#D-#==================================================
	910	#D-function IGCM_sys_Cp
	911	#D-* Purpose: generic cp
	912	#D-* Examples:
	913	#D-
	914	function IGCM_sys_Cp {
[623]	915	IGCM_debug_PushStack "IGCM_sys_Cp" $@
	916	if ( $DEBUG_sys ) ; then
	917	echo "IGCM_sys_Cp :" $@
	918	fi
[622]	919
[715]	920	typeset status
[622]	921
[938]	922	echo cp $@ > /tmp/out_command.$$ 2>&1
	923	\cp $@ >> /tmp/out_command.$$ 2>&1
[715]	924	status=$?
[622]	925
[715]	926	if [ ${status} -gt 0 ] ; then
	927	echo "IGCM_sys_Cp : error code ${status}"
[938]	928	cat /tmp/out_command.$$
[623]	929	IGCM_debug_Exit "IGCM_sys_Cp"
	930	else
[938]	931	\rm /tmp/out_command.$$
[623]	932	fi
	933	IGCM_debug_PopStack "IGCM_sys_Cp"
[622]	934	}
	935
	936	#D-#==================================================
	937	#D-function IGCM_sys_Rm
	938	#D-* Purpose: generic rm
	939	#D-* Examples:
	940	#D-
	941	function IGCM_sys_Rm {
[1057]	942	IGCM_debug_PushStack "IGCM_sys_Rm" $@
[623]	943	if ( $DEBUG_sys ) ; then
	944	echo "IGCM_sys_Rm :" $@
	945	fi
[622]	946
[715]	947	typeset status
[622]	948
[938]	949	echo rm $@ > /tmp/out_command.$$ 2>&1
	950	\rm $@ >> /tmp/out_command.$$ 2>&1
[715]	951	status=$?
[622]	952
[715]	953	if [ ${status} -gt 0 ] ; then
	954	echo "IGCM_sys_Rm : error code ${status}"
[938]	955	cat /tmp/out_command.$$
[623]	956	IGCM_debug_Exit "IGCM_sys_Rm"
	957	else
[938]	958	\rm /tmp/out_command.$$
[623]	959	fi
	960	IGCM_debug_PopStack "IGCM_sys_Rm"
[622]	961	}
	962
	963	#D-#==================================================
	964	#D-function IGCM_sys_RmRunDir
	965	#D-* Purpose: rm tmpdir (dummy function most of the time batch
	966	#D- scheduler will do the job)
	967	#D-* Examples:
	968	#D-
	969	function IGCM_sys_RmRunDir {
[1057]	970	IGCM_debug_PushStack "IGCM_sys_RmRunDir" $@
[623]	971	if ( $DEBUG_sys ) ; then
	972	echo "IGCM_sys_RmRunDir :" $@
	973	fi
[622]	974
[715]	975	typeset status
[622]	976
[938]	977	echo rm $@ > /tmp/out_command.$$ 2>&1
	978	\rm $@ >> /tmp/out_command.$$ 2>&1
[715]	979	status=$?
[622]	980
[715]	981	if [ ${status} -gt 0 ] ; then
[811]	982	echo "IGCM_sys_RmRunDir : rm error code is ${status}."
[938]	983	cat /tmp/out_command.$$
[623]	984	IGCM_debug_Exit "IGCM_sys_RmRunDir"
	985	else
[938]	986	\rm /tmp/out_command.$$
[623]	987	fi
	988	IGCM_debug_PopStack "IGCM_sys_RmRunDir"
[622]	989	}
	990
	991	#D-#==================================================
	992	#D-function IGCM_sys_Mv
	993	#D-* Purpose: generic move
	994	#D-* Examples:
	995	#D-
	996	function IGCM_sys_Mv {
[623]	997	IGCM_debug_PushStack "IGCM_sys_Mv" $@
	998	if ( $DEBUG_sys ) ; then
	999	echo "IGCM_sys_Mv :" $@
	1000	fi
[622]	1001
[623]	1002	if [ $DRYRUN = 0 ]; then
[622]	1003
[715]	1004	typeset status
[623]	1005
[938]	1006	echo mv $@ > /tmp/out_command.$$ 2>&1
	1007	\mv $@ >> /tmp/out_command.$$ 2>&1
[715]	1008	status=$?
[623]	1009
[715]	1010	if [ ${status} -gt 0 ] ; then
	1011	echo "IGCM_sys_Mv : error code ${status}"
[938]	1012	cat /tmp/out_command.$$
[623]	1013	IGCM_debug_Exit "IGCM_sys_Mv"
[622]	1014	else
[938]	1015	\rm /tmp/out_command.$$
[622]	1016	fi
[623]	1017	fi
[622]	1018
[623]	1019	IGCM_debug_PopStack "IGCM_sys_Mv"
[622]	1020	}
	1021
	1022	#D-#==================================================
	1023	#D-function IGCM_sys_Put_Dir
	1024	#D-* Purpose: Copy a complete directory on $(ARCHIVE)
	1025	#D-* Examples:
	1026	#D-
	1027	function IGCM_sys_Put_Dir {
[623]	1028	IGCM_debug_PushStack "IGCM_sys_Put_Dir" $@
	1029	if ( $DEBUG_sys ) ; then
	1030	echo "IGCM_sys_Put_Dir :" $@
	1031	fi
	1032	if [ $DRYRUN = 0 ]; then
	1033	if [ ! -d ${1} ] ; then
	1034	echo "WARNING : IGCM_sys_Put_Dir ${1} DOES NOT EXIST ."
	1035	IGCM_debug_PopStack "IGCM_sys_Put_Dir"
	1036	return
[622]	1037	fi
	1038
[715]	1039	typeset status
[622]	1040
[623]	1041	# Only if we use rsync
	1042	#IGCM_sys_TestDirArchive $( dirname $2 )
	1043	#
	1044	#USUAL WAY
[938]	1045	\cp -r $1 $2 > /tmp/out_command.$$ 2>&1
[715]	1046	status=$?
[622]	1047
[715]	1048	if [ ${status} -gt 0 ] ; then
[811]	1049	IGCM_debug_Print 2 "IGCM_sys_Put_Dir : cp failed error code ${status}"
[938]	1050	cat /tmp/out_command.$$
[623]	1051	IGCM_debug_Exit "IGCM_sys_Put_Dir"
[622]	1052	else
[938]	1053	\rm /tmp/out_command.$$
[622]	1054	fi
[623]	1055	fi
	1056	IGCM_debug_PopStack "IGCM_sys_Put_Dir"
[622]	1057	}
	1058
	1059	#D-#==================================================
	1060	#D-function IGCM_sys_Get_Dir
[623]	1061	#D-* Purpose: Copy a complete directory from ${ARCHIVE}
[622]	1062	#D-* Examples:
	1063	#D-
	1064	function IGCM_sys_Get_Dir {
[623]	1065	IGCM_debug_PushStack "IGCM_sys_Get_Dir" $@
	1066	if ( $DEBUG_sys ) ; then
	1067	echo "IGCM_sys_Get_Dir :" $@
	1068	fi
	1069	if [ $DRYRUN = 0 ]; then
	1070	# if [ ! -d ${1} ] ; then
	1071	# echo "WARNING : IGCM_sys_Get_Dir ${1} DOES NOT EXIST ."
	1072	# IGCM_debug_PopStack "IGCM_sys_Get_Dir"
	1073	# return
	1074	# fi
[622]	1075
[708]	1076	typeset NB_ESSAI DELAI status i
	1077	# number of tentative
	1078	NB_ESSAI=3
	1079	# time delay between tentative
	1080	DELAI=2
[622]	1081
[676]	1082	# Only if we use rsync
	1083	#IGCM_sys_TestDirArchive $( dirname $2 )
	1084	#
[708]	1085	# USUAL WAY
	1086	# add 'ccc_hsm get' (to demigrate all offline files) to reduce time of this command :
[713]	1087	ccc_hsm get -r $1
[622]	1088
[709]	1089	i=0
[708]	1090	while [ $i -lt $NB_ESSAI ] ; do
[938]	1091	\cp -ur $1 $2 >> /tmp/out_command.$$ 2>&1
[708]	1092	status=$?
[811]	1093	if [ ${status} -gt 0 ] ; then
[713]	1094	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : cp failed error code ${status} ${i}/${NB_ESSAI}"
[708]	1095	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : sleep ${DELAI} seconds and try again."
	1096	sleep $DELAI
	1097	else
	1098	break
	1099	fi
	1100	(( i = i + 1 ))
	1101	done
	1102
	1103	if [ ${status} -gt 0 ] ; then
[623]	1104	echo "IGCM_sys_Get_Dir : error."
[938]	1105	cat /tmp/out_command.$$
[623]	1106	IGCM_debug_Exit "IGCM_sys_Get_Dir"
[622]	1107	else
[938]	1108	\rm /tmp/out_command.$$
[622]	1109	fi
[623]	1110	fi
	1111	IGCM_debug_PopStack "IGCM_sys_Get_Dir"
[622]	1112	}
	1113
	1114	#D-#==================================================
	1115	#D-function IGCM_sys_Get_Master
	1116	#D-* Purpose: Copy a complete directory from MASTER filesystem
	1117	#D-* Examples:
	1118	#D-
	1119	function IGCM_sys_Get_Master {
[623]	1120	IGCM_debug_PushStack "IGCM_sys_Get_Master" $@
	1121	if ( $DEBUG_sys ) ; then
	1122	echo "IGCM_sys_Get_Master :" $@
	1123	fi
	1124	if [ $DRYRUN = 0 ]; then
[640]	1125	if ( [ ! -d ${1} ] && [ ! -f ${1} ] ) ; then
[623]	1126	echo "WARNING : IGCM_sys_Get_Master ${1} DOES NOT EXIST ."
	1127	IGCM_debug_PopStack "IGCM_sys_Get_Master"
	1128	return
[622]	1129	fi
	1130
[708]	1131	typeset NB_ESSAI DELAI status i
	1132	# number of tentative
	1133	NB_ESSAI=3
	1134	# time delay between tentative
	1135	DELAI=2
[622]	1136
[709]	1137	i=0
[708]	1138	while [ $i -lt $NB_ESSAI ] ; do
[938]	1139	\cp -urL $1 $2 > /tmp/out_command.$$ 2>&1
[708]	1140	status=$?
	1141	if [ ${status} -gt 0 ]; then
[713]	1142	IGCM_debug_Print 2 "IGCM_sys_Get_Master : cp failed error code ${status} ${i}/${NB_ESSAI}"
[708]	1143	IGCM_debug_Print 2 "IGCM_sys_Get_Master : sleep ${DELAI} seconds and try again."
	1144	sleep $DELAI
	1145	else
	1146	break
	1147	fi
	1148	(( i = i + 1 ))
	1149	done
[622]	1150
[708]	1151	if [ ${status} -gt 0 ] ; then
[623]	1152	echo "IGCM_sys_Get_Master : error."
[938]	1153	cat /tmp/out_command.$$
[623]	1154	IGCM_debug_Exit "IGCM_sys_Get_Master"
[622]	1155	else
[938]	1156	\rm /tmp/out_command.$$
[622]	1157	fi
[623]	1158	fi
	1159	IGCM_debug_PopStack "IGCM_sys_Get_Master"
[622]	1160	}
	1161
[689]	1162	#====================================================
	1163	#- Call IGCM_sys_Mirror_libIGCM now !
	1164	if ( $MirrorlibIGCM ) ; then
	1165	IGCM_sys_Mirror_libIGCM
	1166	fi
	1167
[622]	1168	#D-#==================================================
	1169	#D-function IGCM_sys_Put_Rest
[623]	1170	#D-* Purpose: Put computied restarts on ${ARCHIVE}.
[622]	1171	#D- File and target directory must exist.
	1172	#D-* Examples:
	1173	#D-
	1174	function IGCM_sys_Put_Rest {
[623]	1175	IGCM_debug_PushStack "IGCM_sys_Put_Rest" $@
	1176	if ( $DEBUG_sys ) ; then
	1177	echo "IGCM_sys_Put_Rest :" $@
	1178	fi
	1179	if [ $DRYRUN = 0 ]; then
	1180	if [ ! -f ${1} ] ; then
	1181	echo "ERROR : IGCM_sys_Put_Rest ${1} DOES NOT EXIST ."
	1182	IGCM_debug_Exit "IGCM_sys_Put_Rest"
[622]	1183	fi
	1184
[715]	1185	typeset status
[623]	1186	#
	1187	# USUAL WAY
[938]	1188	\cp $1 $2 > /tmp/out_command.$$ 2>&1
[715]	1189	status=$?
[622]	1190
[623]	1191	# #RSYNC WITH NETWORK SSH CALL
[938]	1192	# echo ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} > /tmp/out_command.$$ 2>&1
	1193	# ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} >> /tmp/out_command.$$ 2>&1
[622]	1194
[623]	1195	# #RSYNC WITH NFS USE
[938]	1196	# echo ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} > /tmp/out_command.$$ 2>&1
	1197	# ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} >> /tmp/out_command.$$ 2>&1
[622]	1198
[715]	1199	# status=$?
	1200	# IGCM_sys_Rsync_out $status
[622]	1201
[938]	1202	# ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk /tmp/out_command.$$
[715]	1203	# (( status=status+$? ))
[623]	1204
[715]	1205	if [ ${status} -gt 0 ] ; then
	1206	echo "IGCM_sys_Put_Rest : cp failed error code ${status}"
[765]	1207	[ -f ${1} ] && ls -l ${1}
[759]	1208	[ -f ${2} ] && ls -l ${2}
	1209	[ -f ${2}/${1} ] && ls -l ${2}/${1}
[938]	1210	cat /tmp/out_command.$$
[623]	1211	IGCM_debug_Exit "IGCM_sys_Put_Rest"
[622]	1212	else
[759]	1213
	1214	if [ X${JobType} = XRUN ] ; then
	1215	[ -f ${2} ] && IGCM_sys_Chmod 444 ${2}
	1216	[ -f ${2}/${1} ] && IGCM_sys_Chmod 444 ${2}/${1}
	1217	fi
	1218
[938]	1219	\rm /tmp/out_command.$$
[622]	1220	fi
[623]	1221	fi
	1222	IGCM_debug_PopStack "IGCM_sys_Put_Rest"
[622]	1223	}
	1224
	1225	#D-#==================================================
[623]	1226	#D-function IGCM_sys_PutBuffer_Rest
	1227	#D-* Purpose: Put computied restarts on ${SCRATCHDIR}.
	1228	#D- File and target directory must exist.
	1229	#D-* Examples:
	1230	#D-
	1231	function IGCM_sys_PutBuffer_Rest {
	1232	IGCM_debug_PushStack "IGCM_sys_PutBuffer_Rest" $@
	1233	if ( $DEBUG_sys ) ; then
	1234	echo "IGCM_sys_PutBuffer_Rest :" $@
	1235	fi
	1236	if [ $DRYRUN = 0 ]; then
	1237	if [ ! -f ${1} ] ; then
	1238	echo "ERROR : IGCM_sys_PutBuffer_Rest ${1} DOES NOT EXIST ."
	1239	IGCM_debug_Exit "IGCM_sys_PutBuffer_Rest"
	1240	fi
	1241
[715]	1242	typeset status
[623]	1243	#
	1244	# USUAL WAY
[938]	1245	\cp $1 $2 > /tmp/out_command.$$ 2>&1
[715]	1246	status=$?
[623]	1247
[715]	1248	if [ ${status} -gt 0 ] ; then
	1249	echo "IGCM_sys_PutBuffer_Rest : error code ${status}"
[759]	1250	[ -f ${2} ] && ls -l ${2}
	1251	[ -f ${2}/${1} ] && ls -l ${2}/${1}
[938]	1252	cat /tmp/out_command.$$
[623]	1253	IGCM_debug_Exit "IGCM_sys_PutBuffer_Rest"
	1254	else
[759]	1255
	1256	if [ X${JobType} = XRUN ] ; then
	1257	[ -f ${2} ] && IGCM_sys_Chmod 444 ${2}
	1258	[ -f ${2}/${1} ] && IGCM_sys_Chmod 444 ${2}/${1}
	1259	fi
	1260
[938]	1261	\rm /tmp/out_command.$$
[623]	1262	fi
	1263	fi
	1264	IGCM_debug_PopStack "IGCM_sys_PutBuffer_Rest"
	1265	}
	1266
	1267	#D-#==================================================
[801]	1268	#D-function IGCM_sys_PrepareTaredRestart
	1269	#D-* Purpose: Prepare tared restart to be access by computing job. Identity here.
	1270	#D-* Examples:
	1271	#D-
	1272	function IGCM_sys_PrepareTaredRestart {
	1273	IGCM_debug_PushStack "IGCM_sys_PrepareTaredRestart" $@
[802]	1274	if [ $DRYRUN = 0 ]; then
[918]	1275	[ ! -f $( basename $1 ) ] && IGCM_sys_Get $1 .
[802]	1276	fi
[801]	1277	IGCM_debug_PopStack "IGCM_sys_PrepareTaredRestart"
	1278	}
	1279
	1280	#D-#==================================================
[622]	1281	#D-function IGCM_sys_Put_Out
[623]	1282	#D-* Purpose: Copy a file on ${ARCHIVE} after having chmod it in readonly
[622]	1283	#D-* Examples:
	1284	#D-
	1285	function IGCM_sys_Put_Out {
[623]	1286	IGCM_debug_PushStack "IGCM_sys_Put_Out" $@
	1287	if ( $DEBUG_sys ) ; then
	1288	echo "IGCM_sys_Put_Out :" $@
	1289	fi
[667]	1290
[711]	1291	typeset NB_ESSAI DELAI status i exist skip
[667]	1292
	1293	# number of tentative
[668]	1294	NB_ESSAI=3
[667]	1295	# time delay between tentative
[668]	1296	DELAI=2
[667]	1297
[623]	1298	if [ $DRYRUN = 0 ]; then
	1299	if [ ! -f ${1} ] ; then
	1300	echo "WARNING : IGCM_sys_Put_Out ${1} DOES NOT EXIST ."
	1301	IGCM_debug_PopStack "IGCM_sys_Put_Out"
	1302	return 1
[622]	1303	fi
[623]	1304	#
	1305	IGCM_sys_MkdirArchive $( dirname $2 )
	1306	#
[622]	1307
[623]	1308	#=====================================================
	1309	# COMMENT OUT DOUBLE COPY ON SCRATCHDIR
	1310	#=====================================================
[622]	1311
[623]	1312	#echo ${2} \| grep "${R_OUT}" > /dev/null 2>&1
	1313	#if [ $? -eq 0 ] ; then
	1314	# typeset WORKPATH FILEPATH
	1315	# WORKPATH=$( dirname $2 \| sed -e "s\|${R_OUT}\|${R_BUF}\|" )
	1316	# IGCM_sys_MkdirWork ${WORKPATH}
	1317	# FILEPATH=${WORKPATH}/$( basename $2 )
	1318	# #
	1319	# IGCM_sys_Cp ${1} ${FILEPATH}
	1320	#fi
[622]	1321
[711]	1322	exist=false
	1323	skip=false
	1324	if [ -f $2 ] ; then
	1325	IGCM_debug_Print 1 "$2 already exist"
	1326	ccc_hsm get $2
	1327	exist=true
	1328	if [ "X$( diff $1 $2 )" = X ] ; then
	1329	IGCM_debug_Print 2 "$1 and $2 are the same file, we skip the copy"
	1330	skip=true
[667]	1331	else
[711]	1332	IGCM_debug_Print 2 "$1 and $2 are not the same file, we force the copy"
	1333	skip=false
[667]	1334	fi
[711]	1335	fi
	1336	#
	1337	if ( [ X${exist} = Xtrue ] && [ X${skip} = Xfalse ] ) ; then
	1338	IGCM_sys_Chmod u+w $2
	1339	fi
[622]	1340
[711]	1341	if [ X${skip} = Xfalse ] ; then
[719]	1342	i=0
[711]	1343	while [ $i -lt $NB_ESSAI ] ; do
[1037]	1344	if [ $( stat -c %d $1 ) -ne $( stat -c %d $( dirname $2 ) ) ] ; then
	1345	# USUAL WAY
	1346	\cp $1 $2 > /tmp/out_command.$$ 2>&1
	1347	status=$?
	1348	else
	1349	# NOT SO USUAL WAY
	1350	\mv $1 $2 > /tmp/out_command.$$ 2>&1
	1351	status=$?
	1352	fi
[711]	1353	if [ ${status} -gt 0 ]; then
[713]	1354	IGCM_debug_Print 2 "IGCM_sys_Put_Out : cp failed error code ${status} ${i}/${NB_ESSAI}"
[711]	1355	IGCM_debug_Print 2 "IGCM_sys_Put_Out : sleep ${DELAI} seconds and try again."
[765]	1356	[ -f ${1} ] && ls -l ${1}
[759]	1357	[ -f ${2} ] && ls -l ${2}
	1358	[ -f ${2}/${1} ] && ls -l ${2}/${1}
[711]	1359	sleep $DELAI
	1360	else
	1361	break
	1362	fi
	1363	(( i = i + 1 ))
	1364	done
	1365	fi
	1366
[623]	1367	# #RSYNC WITH NETWORK SSH CALL
[938]	1368	# echo ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} > /tmp/out_command.$$ 2>&1
	1369	# ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} >> /tmp/out_command.$$ 2>&1
[622]	1370
[623]	1371	# #RSYNC WITH NFS USE
[938]	1372	# echo ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} > /tmp/out_command.$$ 2>&1
	1373	# ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} >> /tmp/out_command.$$ 2>&1
[622]	1374
[711]	1375	# status=$?
	1376	# IGCM_sys_Rsync_out $status
[622]	1377
[938]	1378	# ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk /tmp/out_command.$$
[711]	1379	# (( status=status+$? ))
[622]	1380
[711]	1381	if [ ${status} -gt 0 ] ; then
[623]	1382	echo "IGCM_sys_Put_Out : error."
[765]	1383	[ -f ${1} ] && ls -l ${1}
[759]	1384	[ -f ${2} ] && ls -l ${2}
	1385	[ -f ${2}/${1} ] && ls -l ${2}/${1}
[938]	1386	cat /tmp/out_command.$$
[623]	1387	IGCM_debug_Exit "IGCM_sys_Put_Out"
[622]	1388	else
[759]	1389
	1390	if [ X${JobType} = XRUN ] ; then
	1391	if [ X${3} = X ] ; then
	1392	[ -f ${2} ] && IGCM_sys_Chmod 444 ${2}
	1393	[ -f ${2}/${1} ] && IGCM_sys_Chmod 444 ${2}/${1}
	1394	fi
	1395	fi
	1396
[938]	1397	\rm /tmp/out_command.$$
[622]	1398	fi
[623]	1399	fi
	1400	IGCM_debug_PopStack "IGCM_sys_Put_Out"
	1401	return 0
[622]	1402	}
	1403
	1404	#D-#==================================================
[623]	1405	#D-function IGCM_sys_PutBuffer_Out
	1406	#D-* Purpose: Copy a file on ${SCRATCHDIR} after having chmod it in readonly
	1407	#D-* Examples:
	1408	#D-
	1409	function IGCM_sys_PutBuffer_Out {
	1410	IGCM_debug_PushStack "IGCM_sys_PutBuffer_Out" $@
	1411	if ( $DEBUG_sys ) ; then
	1412	echo "IGCM_sys_PutBuffer_Out :" $@
	1413	fi
[711]	1414
[773]	1415	typeset NB_ESSAI DELAI status i exist skip
[711]	1416
	1417	# number of tentative
	1418	NB_ESSAI=3
	1419	# time delay between tentative
	1420	DELAI=2
	1421
[623]	1422	if [ $DRYRUN = 0 ]; then
	1423	if [ ! -f ${1} ] ; then
	1424	echo "WARNING : IGCM_sys_PutBuffer_Out ${1} DOES NOT EXIST ."
	1425	IGCM_debug_PopStack "IGCM_sys_PutBuffer_Out"
	1426	return 1
	1427	fi
	1428	#
	1429	IGCM_sys_Mkdir $( dirname $2 )
	1430	#
	1431
[711]	1432	exist=false
	1433	skip=false
	1434	if [ -f $2 ] ; then
	1435	IGCM_debug_Print 1 "$2 already exist"
	1436	exist=true
	1437	if [ "X$( diff $1 $2 )" = X ] ; then
	1438	IGCM_debug_Print 2 "$1 and $2 are the same file, we skip the copy"
[811]	1439	status=0
[711]	1440	skip=true
	1441	else
	1442	IGCM_debug_Print 2 "$1 and $2 are not the same file, we force the copy"
	1443	skip=false
	1444	fi
	1445	fi
[623]	1446	#
[711]	1447	if ( [ X${exist} = Xtrue ] && [ X${skip} = Xfalse ] ) ; then
	1448	IGCM_sys_Chmod u+w $2
	1449	fi
[623]	1450
[711]	1451	if [ X${skip} = Xfalse ] ; then
	1452	i=0
	1453	while [ $i -lt $NB_ESSAI ] ; do
[1037]	1454	if [ $( stat -c %d $1 ) -ne $( stat -c %d $( dirname $2 ) ) ] ; then
	1455	# USUAL WAY
	1456	\cp $1 $2 > /tmp/out_command.$$ 2>&1
	1457	status=$?
	1458	else
	1459	# NOT SO USUAL WAY
	1460	\mv $1 $2 > /tmp/out_command.$$ 2>&1
	1461	status=$?
	1462	fi
[711]	1463	if [ ${status} -gt 0 ]; then
[713]	1464	IGCM_debug_Print 2 "IGCM_sys_PutBuffer_Out : cp failed error code ${status} ${i}/${NB_ESSAI}"
[711]	1465	IGCM_debug_Print 2 "IGCM_sys_PutBuffer_Out : sleep ${DELAI} seconds and try again."
[759]	1466	[ -f ${2} ] && ls -l ${2}
	1467	[ -f ${2}/${1} ] && ls -l ${2}/${1}
[711]	1468	sleep $DELAI
	1469	else
	1470	break
	1471	fi
	1472	(( i = i + 1 ))
	1473	done
	1474	fi
	1475
	1476	if [ ${status} -gt 0 ] ; then
[623]	1477	echo "IGCM_sys_PutBuffer_Out : error."
[759]	1478	[ -f ${2} ] && ls -l ${2}
	1479	[ -f ${2}/${1} ] && ls -l ${2}/${1}
[938]	1480	cat /tmp/out_command.$$
[623]	1481	IGCM_debug_Exit "IGCM_sys_PutBuffer_Out"
	1482	else
[759]	1483
	1484	if [ X${JobType} = XRUN ] ; then
	1485	if [ X${3} = X ] ; then
	1486	[ -f ${2} ] && IGCM_sys_Chmod 444 ${2}
	1487	[ -f ${2}/${1} ] && IGCM_sys_Chmod 444 ${2}/${1}
	1488	fi
	1489	fi
	1490
[938]	1491	\rm /tmp/out_command.$$
[623]	1492	fi
	1493	fi
	1494	IGCM_debug_PopStack "IGCM_sys_PutBuffer_Out"
	1495	return 0
	1496	}
	1497
	1498	#D-#==================================================
[622]	1499	#D-function IGCM_sys_Get
	1500	#D-* Purpose: Get a file from ${ARCHIVE}
	1501	#D-* Examples: IGCM_sys_Get myfile /destpath/myfile_with_PREFIX
	1502	#D- IGCM_sys_Get /l Array_contain_myfiles /destpath/
	1503	function IGCM_sys_Get {
[623]	1504	IGCM_debug_PushStack "IGCM_sys_Get" $@
[622]	1505
[773]	1506	typeset DEST dm_liste target file_work
[667]	1507	typeset NB_ESSAI DELAI status i
[622]	1508
[623]	1509	if ( $DEBUG_sys ) ; then
	1510	echo "IGCM_sys_Get :" $@
	1511	fi
[667]	1512
	1513	# number of tentative
[668]	1514	NB_ESSAI=3
[667]	1515	# time delay between tentative
[668]	1516	DELAI=2
[667]	1517
[623]	1518	if [ $DRYRUN -le 2 ]; then
	1519	if [ X${1} = X'/l' ] ; then
	1520	eval set +A dm_liste \${${2}}
	1521	else
	1522	eval set +A dm_liste ${1}
[622]	1523	fi
[623]	1524	eval DEST=\${${#}}
[622]	1525
[623]	1526	#=====================================================
	1527	# COMMENT OUT DOUBLE COPY ON SCRATCHDIR
	1528	#=====================================================
[622]	1529
[623]	1530	# Is it an R_OUT file (not R_IN) ?
	1531	#echo ${dm_liste[0]} \| grep "${R_OUT}" > /dev/null 2>&1
	1532	#if [ $? -eq 0 ] ; then
[785]	1533	# # Yes ? then we try to get it in SCRATCHDIR
	1534	# set +A file_work $( echo ${dm_liste[*]} \| sed -e "s\|${R_OUT}\|${R_BUF}\|g" )
	1535	# if [ -f ${file_work[0]} ] ; then
	1536	# IGCM_sys_Cp ${file_work[*]} ${DEST}
	1537	# IGCM_debug_PopStack "IGCM_sys_Get"
	1538	# return
	1539	# fi
[623]	1540	#fi
[622]	1541
[938]	1542	ccc_hsm get ${dm_liste[*]} > /tmp/out_command.$$ 2>&1
[715]	1543	status=$?
	1544	if [ ${status} -gt 0 ] ; then
	1545	echo "WARNING IGCM_sys_Get : error code ${status}"
[938]	1546	cat /tmp/out_command.$$
[623]	1547	echo "WARNING IGCM_sys_Get : will stop later if the cp fails."
	1548	fi
[622]	1549
[715]	1550	#if [ ${status} -gt 0 ] ; then
[938]	1551	# if [ ! "X$( grep "Lost dmusrcmd connection" /tmp/out_command.$$ )" = "X" ] ; then
	1552	# cat /tmp/out_command.$$
[785]	1553	# echo "WARNING IGCM_sys_Get : Lost dmusrcmd connection : "
	1554	# sleep 30
	1555	# echo "We try another time"
[938]	1556	## dmget ${dm_liste[*]} > /tmp/out_command.$$ 2>&1
	1557	# ccc_hsm get ${dm_liste[*]} > /tmp/out_command.$$ 2>&1
[785]	1558	# status=$?
	1559	# if [ ${status} -gt 0 ] ; then
	1560	# echo "ERROR IGCM_sys_Get : again demigration error :"
[938]	1561	# cat /tmp/out_command.$$
[785]	1562	# IGCM_debug_Exit "IGCM_sys_Get"
[623]	1563	# fi
[785]	1564	# else
	1565	# echo "ERROR IGCM_sys_Get : demigration error :"
[938]	1566	# cat /tmp/out_command.$$
[785]	1567	# IGCM_debug_Exit "IGCM_sys_Get"
	1568	# fi
[623]	1569	#fi
[622]	1570
[719]	1571	# #RSYNC WITH NETWORK SSH CALL
[938]	1572	# echo ${RSYNC} ${RSYNC_opt} -e ssh ${STOREHOST}:"${dm_liste}" ${STOREHOST}:${RUN_DIR}/${DEST} > /tmp/out_command.$$ 2>&1
	1573	# ${RSYNC} ${RSYNC_opt} -e ssh ${STOREHOST}:"${dm_liste}" ${STOREHOST}:${RUN_DIR}/${DEST} >> /tmp/out_command.$$ 2>&1
[719]	1574
	1575	# #RSYNC WITH NFS USE
[938]	1576	# echo ${RSYNC} ${RSYNC_opt} ${dm_liste} ${RUN_DIR}/${DEST} > /tmp/out_command.$$ 2>&1
	1577	# ${RSYNC} ${RSYNC_opt} ${dm_liste} ${RUN_DIR}/${DEST} >> /tmp/out_command.$$ 2>&1
[719]	1578
	1579	# status=$?
	1580	# IGCM_sys_Rsync_out $status
	1581
[938]	1582	# ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk /tmp/out_command.$$
[719]	1583	# (( status=status+$? ))
	1584
[623]	1585	#USUAL WAY
	1586	if [ X${1} = X'/l' ] ; then
	1587	for target in ${dm_liste[*]} ; do
	1588	local_file=$( basename ${target} )
[668]	1589	# test if the target file is present before the loop
	1590	IGCM_sys_TestFileArchive ${target}
[715]	1591	status=$?
	1592	if [ ${status} -gt 0 ] ; then
[701]	1593	echo "IGCM_sys_Get, ERROR : regular file ${target} DOES NOT EXIST ."
[668]	1594	IGCM_debug_Exit "IGCM_sys_Get"
	1595	else
[719]	1596	i=0
[668]	1597	while [ $i -lt $NB_ESSAI ] ; do
[1037]	1598	#if [ X${DoLink} = Xtrue ] ; then
	1599	# \ln -s ${target} ${DEST}/${local_file} >> /tmp/out_command.$$ 2>&1
	1600	# status=$?
	1601	# else
	1602	# \cp ${target} ${DEST}/${local_file} >> /tmp/out_command.$$ 2>&1
	1603	# status=$?
	1604	#fi
	1605	\ln -s ${target} ${DEST}/${local_file} >> /tmp/out_command.$$ 2>&1
[668]	1606	status=$?
	1607	if [ ${status} -gt 0 ]; then
[713]	1608	IGCM_debug_Print 2 "IGCM_sys_Get : cp failed error code ${status} ${i}/${NB_ESSAI}"
[676]	1609	IGCM_debug_Print 2 "IGCM_sys_Get : sleep ${DELAI} seconds and try again."
	1610	sleep $DELAI
[668]	1611	else
[676]	1612	break
[668]	1613	fi
	1614	(( i = i + 1 ))
	1615	done
[719]	1616	if [ ${status} -gt 0 ] ; then
	1617	echo "IGCM_sys_Get : error"
[938]	1618	cat /tmp/out_command.$$
	1619	\rm /tmp/out_command.$$
[719]	1620	IGCM_debug_Exit "IGCM_sys_Get"
	1621	else
[938]	1622	\rm /tmp/out_command.$$
[719]	1623	fi
[668]	1624	fi
[623]	1625	done
	1626	else
[719]	1627	i=0
	1628	while [ $i -lt $NB_ESSAI ] ; do
[938]	1629	\cp ${dm_liste} ${DEST} >> /tmp/out_command.$$ 2>&1
[719]	1630	status=$?
	1631	if [ ${status} -gt 0 ]; then
	1632	IGCM_debug_Print 2 "IGCM_sys_Get : cp failed error code ${status} ${i}/${NB_ESSAI}"
	1633	IGCM_debug_Print 2 "IGCM_sys_Get : sleep ${DELAI} seconds and try again."
	1634	sleep $DELAI
	1635	else
	1636	break
	1637	fi
	1638	(( i = i + 1 ))
	1639	done
	1640	if [ ${status} -gt 0 ] ; then
	1641	echo "IGCM_sys_Get : error"
[938]	1642	cat /tmp/out_command.$$
	1643	\rm /tmp/out_command.$$
[719]	1644	IGCM_debug_Exit "IGCM_sys_Get"
	1645	else
[938]	1646	\rm /tmp/out_command.$$
[719]	1647	fi
[623]	1648	fi
	1649	fi
	1650	IGCM_debug_PopStack "IGCM_sys_Get"
	1651	}
[622]	1652
[623]	1653	#D-#==================================================
	1654	#D-function IGCM_sys_GetBuffer
	1655	#D-* Purpose: Get a file from ${SCRATCHDIR}
	1656	#D-* Examples: IGCM_sys_GetBuffer myfile /destpath/myfile_with_PREFIX
	1657	#D- IGCM_sys_GetBuffer /l Array_contain_myfiles /destpath/
	1658	function IGCM_sys_GetBuffer {
	1659	IGCM_debug_PushStack "IGCM_sys_GetBuffer" $@
	1660
[773]	1661	typeset DEST buf_liste target file_work
[719]	1662	typeset NB_ESSAI DELAI status i
[623]	1663
	1664	if ( $DEBUG_sys ) ; then
	1665	echo "IGCM_sys_GetBuffer :" $@
	1666	fi
[719]	1667
	1668	# number of tentative
	1669	NB_ESSAI=3
	1670	# time delay between tentative
	1671	DELAI=2
	1672
[623]	1673	if [ $DRYRUN -le 2 ]; then
	1674	if [ X${1} = X'/l' ] ; then
	1675	eval set +A buf_liste \${${2}}
[622]	1676	else
[623]	1677	eval set +A buf_liste ${1}
[622]	1678	fi
[623]	1679	eval DEST=\${${#}}
	1680
	1681	#USUAL WAY
	1682	if [ X${1} = X'/l' ] ; then
	1683	for target in ${buf_liste[*]} ; do
	1684	local_file=$( basename ${target} )
[719]	1685	i=0
	1686	while [ $i -lt $NB_ESSAI ] ; do
[938]	1687	\cp ${target} ${DEST}/${local_file} >> /tmp/out_command.$$ 2>&1
[719]	1688	status=$?
	1689	if [ ${status} -gt 0 ]; then
	1690	IGCM_debug_Print 2 "IGCM_sys_GetBuffer : cp failed error code ${status} ${i}/${NB_ESSAI}"
	1691	IGCM_debug_Print 2 "IGCM_sys_GetBuffer : sleep ${DELAI} seconds and try again."
	1692	sleep $DELAI
	1693	else
	1694	break
	1695	fi
	1696	(( i = i + 1 ))
	1697	done
	1698	if [ ${status} -gt 0 ] ; then
	1699	echo "IGCM_sys_Get : error"
[938]	1700	cat /tmp/out_command.$$
	1701	\rm /tmp/out_command.$$
[719]	1702	IGCM_debug_Exit "IGCM_sys_GetBuffer"
	1703	else
[938]	1704	\rm /tmp/out_command.$$
[719]	1705	fi
[623]	1706	done
	1707	else
[719]	1708	i=0
	1709	while [ $i -lt $NB_ESSAI ] ; do
[938]	1710	\cp ${buf_liste} ${DEST} >> /tmp/out_command.$$ 2>&1
[719]	1711	status=$?
	1712	if [ ${status} -gt 0 ]; then
	1713	IGCM_debug_Print 2 "IGCM_sys_GetBuffer : cp failed error code ${status} ${i}/${NB_ESSAI}"
	1714	IGCM_debug_Print 2 "IGCM_sys_GetBuffer : sleep ${DELAI} seconds and try again."
	1715	sleep $DELAI
	1716	else
	1717	break
	1718	fi
	1719	(( i = i + 1 ))
	1720	done
	1721	if [ ${status} -gt 0 ] ; then
	1722	echo "IGCM_sys_Get : error"
[938]	1723	cat /tmp/out_command.$$
	1724	\rm /tmp/out_command.$$
[719]	1725	IGCM_debug_Exit "IGCM_sys_GetBuffer"
	1726	else
[938]	1727	\rm /tmp/out_command.$$
[719]	1728	fi
[623]	1729	fi
	1730	fi
	1731	IGCM_debug_PopStack "IGCM_sys_GetBuffer"
[622]	1732	}
	1733
	1734	#D-#==================================================
	1735	#D-function IGCM_sys_GetDate_FichWork
	1736	#D-* Purpose: donne la date filesys d'un fichier sur le filesystem WORK
	1737	#D-* Examples:
	1738	#D-
	1739	function IGCM_sys_GetDate_FichWork {
[623]	1740	IGCM_debug_PushStack "IGCM_sys_FichWork" $@
	1741	if ( $DEBUG_sys ) ; then
	1742	echo "IGCM_sys_GetDate_FichWork :" $@
	1743	fi
[955]	1744
	1745	if [ $# -ge 3 ] ; then
	1746	mode=$3
	1747	TimeStyle=$4
	1748	else
	1749	mode="default"
	1750	TimeStyle="%Y%m%d%H%M%S"
	1751	fi
	1752
[623]	1753	typeset dateF
[955]	1754	set +A dateF -- $( ls -l --full-time --time-style=+"${TimeStyle}" ${1} )
[623]	1755
[955]	1756	case $mode in
	1757	"default")
	1758	eval ${2}=${dateF[5]}
	1759	;;
	1760	"SplitFields")
	1761	eval ${2}="${dateF[5]}\ ${dateF[6]}"
	1762	;;
	1763	esac
	1764
	1765
[773]	1766	# donne la date filesys d'un fichier sur la machine work
[623]	1767	IGCM_debug_PopStack "IGCM_sys_FichWork"
[622]	1768	}
	1769
	1770	#D-#==================================================
	1771	#D-function IGCM_sys_GetDate_FichArchive
[623]	1772	#D-* Purpose: donne la date filesys d'un fichier sur le filesystem ARCHIVE
[622]	1773	#D-* Examples:
	1774	#D-
	1775	function IGCM_sys_GetDate_FichArchive {
[623]	1776	IGCM_debug_PushStack "IGCM_sys_FichArchive" $@
	1777	if ( $DEBUG_sys ) ; then
	1778	echo "IGCM_sys_GetDate_FichArchive :" $@
	1779	fi
	1780	typeset dateF
	1781	set +A dateF -- $( ls -l --full-time --time-style='+%Y%m%d%H%M%S' ${1} )
	1782	eval ${2}=${dateF[5]}
	1783
	1784	IGCM_debug_PopStack "IGCM_sys_FichArchive"
	1785	}
	1786
	1787	#D-#==================================================
[835]	1788	#D-function IGCM_sys_GetDate_Monitoring
	1789	#D-* Purpose: get the last year for which the monitoring has been computed
	1790	#D-* Examples:
	1791	#D-
	1792	function IGCM_sys_GetDate_Monitoring {
	1793	IGCM_debug_PushStack "IGCM_sys_GetDate_Monitoring" $@
	1794	if ( $DEBUG_sys ) ; then
	1795	echo "IGCM_sys_GetDate_Monitoring :" $@
	1796	fi
	1797
	1798	eval ${2}=$( cdo showyear ${1} 2> /dev/null \| gawk '{ print $NF }' )
	1799
	1800	IGCM_debug_PopStack "IGCM_sys_GetDate_Monitoring"
	1801	}
	1802
	1803	#D-#==================================================
[623]	1804	#D-function IGCM_sys_Dods_Rm
	1805	#D-* Purpose: DO NOTHING ! Put ${ARCHIVE} files on DODS internet protocole.
	1806	#D-* Examples:
	1807	#D-
	1808	function IGCM_sys_Dods_Rm {
	1809	if ( $DEBUG_sys ) ; then
	1810	echo "IGCM_sys_Dods_Rm :" $@
	1811	fi
[715]	1812	typeset status
[623]	1813	if [ $DRYRUN = 0 ]; then
	1814
	1815	# if [ ! -d /dmnfs/cont003/dods/public/${LOGIN}/${R_DODS}/${1} ] ; then
	1816	# echo "WARNING : IGCM_sys_Dods_Rm /dmnfs/cont003/dods/public/${LOGIN}/${R_DODS}/${1} DOES NOT EXIST ."
	1817	# echo "Nothing has been done."
	1818	# return
	1819	# fi
	1820
[649]	1821	/ccc/cont003/home/dsm/p86ipsl/bin/dods_rm public/${LOGIN}/${R_DODS}/${1} # > out_dods_rm 2>&1
[715]	1822	status=$?
[623]	1823
[785]	1824	# if [ ${status} -gt 0 ] ; then
	1825	# echo "IGCM_sys_Dods_Rm : error."
	1826	# cat out_dods_rm
	1827	# IGCM_debug_Exit "IGCM_sys_Dods_Rm"
	1828	# else
	1829	# rm out_dods_rm
	1830	# fi
[623]	1831
	1832	fi
[715]	1833	return $status
[623]	1834	}
	1835
	1836	#D-#==================================================
	1837	#D-function IGCM_sys_Dods_Cp
	1838	#D-* Purpose: Copy $(ARCHIVE) files on DODS internet protocole.
	1839	#D-* Examples:
	1840	#D-
	1841	function IGCM_sys_Dods_Cp {
	1842	if ( $DEBUG_sys ) ; then
	1843	echo "IGCM_sys_Dods_Cp :" $@
	1844	fi
[715]	1845	typeset status
[623]	1846	if [ $DRYRUN = 0 ]; then
	1847
	1848	# if [ ! -d ${R_SAVE}/${1} ] ; then
	1849	# echo "WARNING : IGCM_sys_Dods_Cp ${R_SAVE}/${1} DOES NOT EXIST ."
	1850	# echo "Nothing has been done."
	1851	# return
	1852	# fi
	1853
[649]	1854	/ccc/cont003/home/dsm/p86ipsl/bin/dods_cp ${1} public/${LOGIN}/${R_DODS} # > out_dods_cp 2>&1
[715]	1855	status=$?
[623]	1856
[715]	1857	# if [ ${status} -gt 0 ] ; then
[623]	1858	# echo "IGCM_sys_Dods_Cp : error."
	1859	# cat out_dods_cp
	1860	# IGCM_debug_Exit "IGCM_sys_Dods_Cp"
	1861	# else
	1862	# rm out_dods_cp
	1863	# fi
	1864
	1865	fi
[715]	1866	return $status
[623]	1867	}
	1868
	1869	#D-#==================================================
	1870	#D-function IGCM_sys_Put_Dods
	1871	#D-* Purpose: Put $(ARCHIVE) files on DODS internet protocole.
	1872	#D-* Examples:
	1873	#D-
	1874	function IGCM_sys_Put_Dods {
	1875	IGCM_debug_PushStack "IGCM_sys_Put_Dods" $@
	1876	if ( $DEBUG_sys ) ; then
	1877	echo "IGCM_sys_Put_Dods :" $@
	1878	fi
[711]	1879	#set -vx
[715]	1880	typeset status
[623]	1881	if [ $DRYRUN = 0 ]; then
	1882	if ( [ ! -d ${R_SAVE}/${1} ] && [ ! -d ${R_FIGR}/${1} ] ) ; then
	1883	echo "WARNING IGCM_sys_Put_Dods : None of the following directories exist. Exactly one should."
	1884	echo "WARNING IGCM_sys_Put_Dods : ${R_SAVE}/${1} DOES NOT EXIST."
	1885	echo "WARNING IGCM_sys_Put_Dods : ${R_FIGR}/${1} DOES NOT EXIST."
	1886	IGCM_debug_PopStack "IGCM_sys_Put_Dods"
	1887	return
[622]	1888	fi
[623]	1889
	1890	if ( [ -d ${R_SAVE}/${1} ] && [ -d ${R_FIGR}/${1} ] ) ; then
	1891	echo "WARNING IGCM_sys_Put_Dods : Both of the following directories exist. Exactly one should."
	1892	echo "WARNING IGCM_sys_Put_Dods : ${R_SAVE}/${1} EXISTS."
	1893	echo "WARNING IGCM_sys_Put_Dods : ${R_FIGR}/${1} EXISTS."
	1894	IGCM_debug_PopStack "IGCM_sys_Put_Dods"
	1895	return
	1896	fi
	1897	#
	1898	if [ -d ${R_SAVE}/${1} ] ; then
	1899	cd ${R_SAVE}
	1900	elif [ -d ${R_FIGR}/${1} ] ; then
	1901	cd ${R_FIGR}
	1902	fi
	1903
	1904	IGCM_sys_Dods_Rm ${1}
	1905	IGCM_sys_Dods_Cp ${1}
[715]	1906	status=0
[623]	1907
[715]	1908	if [ ${status} -gt 0 ] ; then
[623]	1909	echo "IGCM_sys_Put_Dods : error."
	1910	IGCM_debug_Exit "IGCM_sys_Put_Dods"
	1911	fi
	1912	fi
	1913	IGCM_debug_PopStack "IGCM_sys_Put_Dods"
[622]	1914	}
	1915
	1916	##############################################################
	1917	# REBUILD OPERATOR
	1918
[811]	1919	#D-#==================================================
[1163]	1920	#D-function IGCM_sys_sync
	1921	#D-* Purpose: flush buffer on disk
	1922	#D-* Examples:
	1923	#D-
	1924	function IGCM_sys_sync {
	1925	IGCM_debug_PushStack "IGCM_sys_sync" $@
	1926	if ( $DEBUG_sys ) ; then
	1927	echo "IGCM_sys_sync :" $@
	1928	fi
	1929
	1930	/bin/sync
	1931
	1932	IGCM_debug_PopStack "IGCM_sys_sync"
	1933	}
	1934
	1935	#D-#==================================================
[811]	1936	#D-function IGCM_sys_rebuild
	1937	#D-* Purpose: rebuild parallel files
	1938	#D-* Examples:
	1939	#D-
[622]	1940	function IGCM_sys_rebuild {
[1057]	1941	IGCM_debug_PushStack "IGCM_sys_rebuild" $@
[623]	1942	if ( $DEBUG_sys ) ; then
	1943	echo "IGCM_sys_rebuild :" $@
	1944	fi
[716]	1945
[734]	1946	typeset NB_ESSAI DELAI status i firstArg
[716]	1947	# number of tentative
	1948	NB_ESSAI=3
	1949	# time delay between tentative
	1950	DELAI=2
	1951
	1952	i=0
	1953	while [ $i -lt $NB_ESSAI ] ; do
[977]	1954	~p86ipsl/rebuild/src_X64_CURIE/modipsl_v2_2_2_netcdf4.2/bin/rebuild -f -o $@ > /tmp/out_command.$$ 2>&1
[716]	1955	status=$?
	1956	if [ ${status} -gt 0 ] ; then
	1957	IGCM_debug_Print 2 "IGCM_sys_rebuild : error code ${status}"
[938]	1958	cat /tmp/out_command.$$
	1959	\rm /tmp/out_command.$$
[716]	1960	IGCM_debug_Print 2 "IGCM_sys_rebuild : ${i}/${NB_ESSAI} sleep ${DELAI} seconds and try again."
[734]	1961	firstArg=${1}
	1962	\rm ${firstArg}
[733]	1963	sleep $DELAI
[716]	1964	else
[938]	1965	\rm /tmp/out_command.$$
[716]	1966	break
	1967	fi
	1968	(( i = i + 1 ))
	1969	done
	1970
[725]	1971	if [ ${status} -gt 0 ] ; then
[773]	1972	echo "IGCM_sys_rebuild : rebuild error code is ${status}"
[623]	1973	IGCM_debug_Exit "rebuild"
	1974	fi
[622]	1975
[623]	1976	IGCM_debug_PopStack "IGCM_sys_rebuild"
[622]	1977	}
	1978
[811]	1979	#D-#==================================================
	1980	#D-function IGCM_sys_rebuild_station
[912]	1981	#D-* Purpose: rebuild parallel files describing station
[811]	1982	#D-* Examples:
	1983	#D-
[697]	1984	function IGCM_sys_rebuild_station {
[1057]	1985	IGCM_debug_PushStack "IGCM_sys_rebuild_station" $@
[697]	1986	typeset i list_opt file_in file_out prefix_invert list_invert
	1987	if ( $DEBUG_sys ) ; then
	1988	echo "IGCM_sys_rebuild_station :" $@
	1989	fi
	1990	list_opt=$@
	1991
[773]	1992	# Invert Axis : t,x -> x,t
[697]	1993	# t,pres,x -> x,t,pres
	1994	# So that we can concatenate along x
	1995	i=0
	1996	for file_in in ${list_opt} ; do
	1997	(( i = i + 1))
	1998	[ ${i} = 1 ] && file_out=${file_in} && continue
[1046]	1999	# detect time counter and do the job only if present
[1147]	2000	var_unlim=$(ncdump -h ${file_in} \| grep UNLIMITED \| cut -d ' ' -f 1 \| sed -e 's/^[ \t]//' -e 's/[ \t]$//')
[1046]	2001	if [ X${var_unlim} = Xtime_counter ] ; then
	2002	prefix_invert=$( basename ${file_in} .nc )
	2003	IGCM_sys_ncpdq -a x,time_counter -a x,time_counter,presnivs ${file_in} ${prefix_invert}_xt.nc
	2004	list_invert[${#list_invert[*]}]=${prefix_invert}_xt.nc
	2005	fi
[697]	2006	done
	2007
	2008	# Concatenate
	2009	IGCM_sys_ncrcat ${list_invert[*]} histstn_xt.nc
	2010
	2011	# Re-ivert file
	2012	IGCM_sys_ncpdq -a time_counter,x -a time_counter,presnivs,x histstn_xt.nc ${file_out}
	2013
[922]	2014	# Station re-ordering is too expansive to be run within libIGCM
[697]	2015	# This is due to (ncpdq - nrcat - ncpdq) I/O sequence.
	2016	# This re-ordering must be done "in memory" by the cmorization process
	2017	# Anyway this is the best sequence using (ncpdq - nrcat - ncpdq)
	2018	# BEGIN reordering
	2019
	2020	# Only LMDZ text output contains the exact ordering of the station.
	2021	# We isolate this in the code below:
	2022	# 0 38 -157.5000000000000 70.98591549295774
	2023	# 0 54 27.49999999999999 67.18309859154928
	2024	# 0 56 -62.50000000000001 82.39436619718309
	2025	# 0 79 12.49999999999999 78.59154929577466
	2026	# 0 116 -165.0000000000000 76.05633802816901
	2027	# 0 117 130.0000000000000 70.98591549295774
	2028	# 0 118 110.0000000000000 87.46478873239437
	2029	# 1 40 4.999999999999995 51.97183098591550
[1056]	2030
[697]	2031	# typeset iStation iProc list_opt file_in file_out prefix_invert
	2032	# typeset -Z4 j4
	2033	# typeset -Z3 j3
	2034
	2035	# unset list_opt
	2036	# set +A list_opt $@
	2037
	2038	# Filename after rebuild
	2039	# file_out=${list_opt[0]}
	2040	# Prefix of output files
	2041	# prefix_invert=$( basename ${file_out} .nc )
	2042	# Number of procs
	2043	# num_proc=$( grep -i mpi_size ${PREFIX}_${Exe_Output} \| wc -l )
	2044
	2045	# iProc=0
	2046	# while [ ${iProc} -lt ${num_proc} ] ; do
	2047	# Array containing Station as a number
	2048	# unset proc_stn
[912]	2049	# set +A proc_stn $( grep "iophy_mpi rank ip lon lat $iProc" ${PREFIX}_${Exe_Output} \| sed -e "s/iophy_mpi rank ip lon lat //g" \| gawk ' {print $2}' )
[697]	2050	# Number of stations produced by processor proc
	2051	# stationLast=${#proc_stn[*]}
	2052	# Proc number on 4 digits
	2053	# j4=${iProc}
	2054	# Init
	2055	# iStation=0
	2056	# while [ ${iStation} -lt ${stationLast} ] ; do
	2057	# Station number on 3 digits
	2058	# j3=${proc_stn[${iStation}]}
	2059	# Extract station
[773]	2060	# Invert Axis : t,x -> x,t
[697]	2061	# t,pres,x -> x,t,pres
	2062	# So that we can concatenate along x
	2063	# IGCM_sys_ncpdq -a x,time_counter -a x,time_counter,presnivs -d x,$iStation,$iStation ${prefix_invert}_${j4}.nc ${prefix_invert}_stn_${j3}.nc
	2064	# (( iStation = iStation + 1 ))
	2065	# done
	2066	# (( iProc = iProc + 1 ))
	2067	# done
	2068
	2069	# Concatenate all station along x
	2070	# IGCM_sys_ncrcat ${prefix_invert}_stn_???.nc ${prefix_invert}_xt.nc
	2071
	2072	# Re-invert file
	2073	# IGCM_sys_ncpdq -a time_counter,x -a time_counter,presnivs,x ${prefix_invert}_xt.nc ${file_out}
	2074
	2075	# END reordering
	2076
	2077	IGCM_debug_PopStack "IGCM_sys_rebuild_station"
	2078	}
	2079
[622]	2080	############################################################
	2081	# Activate Running Environnment Variables
	2082
[811]	2083	#D-#==================================================
	2084	#D-function IGCM_sys_desactiv_variables
	2085	#D-* Purpose: set environement variables prior to execution
	2086	#D-* Examples:
	2087	#D-
[622]	2088	function IGCM_sys_activ_variables {
[623]	2089	IGCM_debug_PushStack "IGCM_sys_activ_variables"
	2090	if ( $DEBUG_sys ) ; then
	2091	echo "IGCM_sys_activ_variables"
	2092	fi
[622]	2093
[811]	2094	# --------------------------------------------------------------------
	2095	#D- MPI specifications
	2096	# --------------------------------------------------------------------
	2097
	2098	# --------------------------------------------------------------------
	2099	#D- Other specifications
	2100	# --------------------------------------------------------------------
	2101
[623]	2102	ulimit -s unlimited
[622]	2103
[623]	2104	IGCM_debug_PopStack "IGCM_sys_activ_variables"
[622]	2105	}
	2106
	2107	############################################################
	2108	# Desactivate Running Environnment Variables
	2109
[811]	2110	#D-#==================================================
	2111	#D-function IGCM_sys_desactiv_variables
	2112	#D-* Purpose: unset environement variables after execution
	2113	#D-* Examples:
	2114	#D-
[622]	2115	function IGCM_sys_desactiv_variables {
[623]	2116	IGCM_debug_PushStack "IGCM_sys_desactiv_variables"
	2117	if ( $DEBUG_sys ) ; then
	2118	echo "IGCM_sys_desactiv_variables"
	2119	fi
[811]	2120	# --------------------------------------------------------------------
	2121	#D- MPI specifications
	2122	# --------------------------------------------------------------------
	2123
	2124	# --------------------------------------------------------------------
	2125	#D- Other specifications
	2126	# --------------------------------------------------------------------
	2127
[623]	2128	IGCM_debug_PopStack "IGCM_sys_desactiv_variables"
[622]	2129	}
	2130
[623]	2131	############################################################
	2132	# Build MPI/OMP scripts run file (dummy function)
[622]	2133
[811]	2134	#D-#==================================================
	2135	#D-function IGCM_sys_build_run_file
	2136	#D-* Purpose: build run file (deprecated)
	2137	#D-* Examples:
	2138	#D-
[622]	2139	function IGCM_sys_build_run_file {
	2140
[811]	2141	IGCM_debug_Print 3 " dummy function : IGCM_sys_build_run_file "
[622]	2142
	2143	}
	2144
	2145	############################################################
	2146	# Build MPI/OMP scripts
[811]	2147
	2148	#D-#==================================================
	2149	#D-function IGCM_sys_build_execution_scripts
	2150	#D-* Purpose: build execution scripts to be launch by ${HOST_MPIRUN_COMMAND}
	2151	#D-* Examples:
	2152	#D-
[623]	2153	function IGCM_sys_build_execution_scripts
[622]	2154	{
[623]	2155	IGCM_debug_PushStack "IGCM_sys_build_execution_scripts" $@
	2156	if ( $DEBUG_sys ) ; then
	2157	echo "IGCM_sys_build_execution_scripts " $@
	2158	fi
[622]	2159
[623]	2160	EXECUTION=${HOST_MPIRUN_COMMAND}
[622]	2161
[1078]	2162	# MPMD mode
[623]	2163	if ( ${OK_PARA_MPMD} ) ; then
[622]	2164
[1078]	2165	# Only MPI (MPMD)
	2166	if ( ! ${OK_PARA_OMP} ) ; then
[622]	2167
[1078]	2168	if [ -f run_file ] ; then
	2169	IGCM_sys_Rm -f run_file
	2170	fi
	2171	touch run_file
[622]	2172
[1078]	2173	# Build run_file
[622]	2174
[1078]	2175	# First loop on the components for the coupler ie oasis (only if oasis3)
	2176	# the coupler ie oasis3 must be the first one
	2177	for comp in ${config_ListOfComponents[*]} ; do
[622]	2178
[1078]	2179	eval ExeNameIn=\${config_Executable_${comp}[0]}
	2180	eval ExeNameOut=\${config_Executable_${comp}[1]}
[622]	2181
[1078]	2182	# for CPL component only
	2183	if [ "X${comp}" = "XCPL" ] && [ "X${ExeNameOut}" != X\"\" ] ; then
[1000]	2184
[1078]	2185	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
	2186	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
	2187	echo "${comp_proc_mpi_loc} ./${ExeNameOut}" >> run_file
	2188	fi
	2189	done
[622]	2190
[1078]	2191	# Then second loop on the components
	2192	for comp in ${config_ListOfComponents[*]} ; do
[622]	2193
[1078]	2194	eval ExeNameIn=\${config_Executable_${comp}[0]}
	2195	eval ExeNameOut=\${config_Executable_${comp}[1]}
[622]	2196
[1078]	2197	# Only if we really have an executable for the component and not the coupler ie oasis:
	2198	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${comp}" != "XCPL" ] ) ; then
[622]	2199
[1078]	2200	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
	2201	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
	2202	echo "${comp_proc_mpi_loc} ./${ExeNameOut}" >> run_file
	2203	fi
	2204	done
[622]	2205
[1078]	2206	EXECUTION="${HOST_MPIRUN_COMMAND} -f ./run_file"
[622]	2207
[1078]	2208	IGCM_sys_Chmod u+x run_file
	2209	if ( $DEBUG_sys ) ; then
	2210	echo "run_file contains : "
	2211	cat run_file
	2212	fi
[622]	2213
[1078]	2214	# MPI-OpenMP (MPMD)
	2215	else
[622]	2216
[1078]	2217	# Use of mpirun instead of ccc_mprun
	2218	EXECUTION="time mpirun"
[622]	2219
[1078]	2220	# Hosts treatment
	2221	${EXECUTION} hostname \| sort \| uniq > hosts.tmp
[622]	2222
[1078]	2223	i=0
	2224	rm -f hosts rankfile
	2225	IGCM_debug_Print 1 "sys Curie, Hosts available :"
	2226	for nodes in `cat hosts.tmp`
	2227	do
	2228	host[$i]=$nodes
	2229	echo "${host[$i]}" >> hosts
	2230	IGCM_debug_Print 1 ${host[$i]}
	2231	i=$((i+1))
	2232	done
	2233	rm -f hosts.tmp
[1000]	2234
[1078]	2235	listnodes=${host[*]}
[1000]	2236
[1078]	2237	EXECUTION="${EXECUTION} -hostfile hosts -rankfile rankfile"
[1000]	2238
[1078]	2239	# Initialisation
	2240	rank=0
	2241	current_core=0
	2242	core_per_node=16
	2243	init_exec=n
[1000]	2244
[1078]	2245	# Loop on the components
	2246	for comp in ${config_ListOfComponents[*]} ; do
[1000]	2247
[1078]	2248	eval ExeNameIn=\${config_Executable_${comp}[0]}
	2249	eval ExeNameOut=\${config_Executable_${comp}[1]}
[1000]	2250
[1078]	2251	# Not possible if oasis has an executable (i.e old version of oasis3)
	2252	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${comp}" = "XCPL" ] ) ; then
	2253	IGCM_debug_Exit "ERROR MPMD with hybrid MPI-OpenMP is not available with oasis3 version"
	2254	IGCM_debug_Print 2 "Only available with oasis3-MCT version coupler"
	2255	IGCM_debug_Verif_Exit
	2256	fi
[1000]	2257
[1078]	2258	# Only if we really have an executable for the component :
	2259	if [ "X${ExeNameOut}" != X\"\" ] ; then
[1000]	2260
[1078]	2261	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
	2262	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
[1000]	2263
[1078]	2264	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
	2265	echo "" >> script_${ExeNameOut}.ksh
	2266	if [ ${comp_proc_omp_loc} -gt 1 ] ; then
	2267
	2268	# Check if the number of threads is correct
	2269	case ${comp_proc_omp_loc} in
	2270	2\|4\|8\|16)
	2271	IGCM_debug_Print 1 "You run ${ExeNameOut} on ${comp_proc_omp_loc} OMP threads"
	2272	;;
	2273	*)
	2274	IGCM_debug_Exit "ERROR with OMP parameters !"
	2275	IGCM_debug_Print 2 "${comp_proc_omp_loc} is not possible as number of OMP threads"
	2276	IGCM_debug_Print 2 "Only 2,4,8,16 as number of OMP threads are possible "
	2277	IGCM_debug_Verif_Exit
	2278	;;
	2279	esac
	2280	echo "export KMP_STACKSIZE=3g" >> script_${ExeNameOut}.ksh
	2281	echo "export KMP_LIBRARY=turnaround" >> script_${ExeNameOut}.ksh
	2282	echo "export MKL_SERIAL=YES" >> script_${ExeNameOut}.ksh
	2283	echo "OMP_NUM_THREADS=${comp_proc_omp_loc}" >> script_${ExeNameOut}.ksh
	2284	fi
	2285	#echo "(( MYMPIRANK = OMPI_COMM_WORLD_RANK - ${start_num})) " >> script_${ExeNameOut}.ksh
[1157]	2286	#echo "./${ExeNameOut} > out_${ExeNameOut}.out.\${MYMPIRANK} 2>out_${ExeNameOut}.err.\${MYMPIRANK}" >> script_${ExeNameOut}.ksh
[1078]	2287	echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
	2288	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
	2289
	2290	if [ ${init_exec} = y ] ; then
	2291	EXECUTION="${EXECUTION} : -np ${comp_proc_mpi_loc} ./script_${ExeNameOut}.ksh"
	2292	else
	2293	EXECUTION="${EXECUTION} -np ${comp_proc_mpi_loc} ./script_${ExeNameOut}.ksh"
	2294	init_exec=y
	2295	fi
	2296
[1000]	2297	# Build rankfile : method used to assign cores and nodes for the MPI process
[1078]	2298	# Ex :
[1173]	2299	#rank 0=curie5296 slot=0,2,4,6
	2300	#rank 1=curie5296 slot=8,10,12,14
[1000]	2301	# Example of final command :
	2302	# mpirun -hostfile hosts -rankfile rankfile -np 27 ./script_lmdz.x.ksh : -np 5 ./script_opa.xx.ksh
	2303	# with script_lmdz.x.ksh :
	2304	# #!/bin/ksh
	2305	#export KMP_STACKSIZE=3g
	2306	#export KMP_LIBRARY=turnaround
	2307	#export MKL_SERIAL=YES
	2308	#OMP_NUM_THREADS=4
	2309	#./lmdz.x
	2310	#
	2311
[1078]	2312	for nb_proc_mpi in `seq 0 $(($comp_proc_mpi_loc-1))`; do
	2313	(( index_host = current_core / core_per_node ))
	2314	host_value=${host[${index_host}]}
	2315	(( slot = current_core % core_per_node ))
	2316	virg=","
	2317	string_final=""
	2318	for index in `seq $slot $(($slot+$comp_proc_omp_loc-1))`; do
[1173]	2319	(( index_slot = index * 2 ))
	2320	string=$index_slot$virg
[1078]	2321	string_final=$string_final$string
	2322	done
	2323	string_final=$( echo $string_final \| sed "s/.$//" )
	2324	echo "rank $rank=$host_value slot=$string_final" >> rankfile
	2325	(( rank = rank + 1 ))
	2326	(( current_core = current_core + comp_proc_omp_loc ))
	2327	done
	2328	fi
[1000]	2329
[1078]	2330	done
	2331	fi
[1000]	2332
[1078]	2333	# Only one executable (SPMD mode).
	2334	else
[1000]	2335
[622]	2336	for comp in ${config_ListOfComponents[*]} ; do
	2337
[685]	2338	# Only if we really have an executable for the component :
[623]	2339	eval ExeNameOut=\${config_Executable_${comp}[1]}
[685]	2340	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${ExeNameOut}" != "Xinca.dat" ] ) ; then
[622]	2341
[685]	2342	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
	2343	echo "" >> script_${ExeNameOut}.ksh
	2344	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
	2345
	2346	if ( ${OK_PARA_OMP} ) ; then
	2347	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
	2348	echo "" >> script_${ExeNameOut}.ksh
	2349	echo "export KMP_STACKSIZE=3g" >> script_${ExeNameOut}.ksh
	2350	echo "export KMP_LIBRARY=turnaround" >> script_${ExeNameOut}.ksh
	2351	echo "export MKL_SERIAL=YES" >> script_${ExeNameOut}.ksh
	2352	echo "OMP_NUM_THREADS=${comp_proc_omp_loc}" >> script_${ExeNameOut}.ksh
	2353	fi
	2354
	2355	if ( ${OK_PARA_MPI} ) ; then
[623]	2356	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
[685]	2357	# Default : ccc_mprun used if nb_proc gt 1
	2358	# to have out/err per process on different files
	2359	# echo "./${ExeNameOut} > out_${ExeNameOut}.out.\${SLURM_PROCID} 2>out_${ExeNameOut}.err.\${SLURM_PROCID}" >> script_${ExeNameOut}.ksh
	2360	echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
	2361	EXECUTION="${HOST_MPIRUN_COMMAND} -n ${comp_proc_mpi_loc} ./script_${ExeNameOut}.ksh"
[623]	2362	else
[685]	2363	# Default : ccc_mprun is NOT used if nb_proc eq 1
	2364	# to have out/err per process on different files
	2365	# echo "./${ExeNameOut} > out_${ExeNameOut}.out 2>out_${ExeNameOut}.err" >> script_${ExeNameOut}.ksh
	2366	echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
	2367	EXECUTION="/usr/bin/time ./script_${ExeNameOut}.ksh"
[622]	2368	fi
[685]	2369
	2370	IGCM_debug_Print 1 "sys Curie : script_${ExeNameOut}.ksh contains"
	2371	cat script_${ExeNameOut}.ksh
	2372
[623]	2373	fi
[622]	2374
	2375	done
	2376
[623]	2377	fi
[622]	2378
[623]	2379	IGCM_debug_Print 1 "sys Curie : execution command is "
	2380	IGCM_debug_Print 1 "$EXECUTION"
[622]	2381
[623]	2382	IGCM_debug_PopStack "IGCM_sys_build_execution_scripts"
[622]	2383	}
	2384
[933]	2385	#D-#==================================================
	2386	#D-function IGCM_sys_check_path
	2387	#D-* Purpose: check that RUN_DIR_PATH that will be removed on some machine
	2388	#D-* do not point to an important use directory. Stop immediately in that case.
	2389	#D-* Examples:
	2390	#D-
	2391	function IGCM_sys_check_path {
	2392	IGCM_debug_PushStack "IGCM_sys_check_path"
	2393	if ( $DEBUG_sys ) ; then
	2394	echo "IGCM_sys_check_path"
	2395	fi
[811]	2396
[933]	2397	if ( [ X${RUN_DIR_PATH} = X${HOME} ] \|\| [ X${RUN_DIR_PATH} = X${WORKDIR} ] \|\| [ X${RUN_DIR_PATH} = X${SCRATCHDIR} ] \|\| [ X${RUN_DIR_PATH} = X${CCCWORKDIR} ] \|\| [ X${RUN_DIR_PATH} = X${CCCSTOREDIR} ] ) ; then
	2398	IGCM_debug_Print 1 "Variable RUN_DIR_PATH is pointing to an important directory : ${RUN_DIR_PATH}"
	2399	IGCM_debug_Print 1 "Please check the RUN_DIR_PATH definition in your Job : Job_${config_UserChoices_JobName}"
[936]	2400	IGCM_debug_Exit "This will stop the job"
[933]	2401	fi
	2402	IGCM_debug_PopStack "IGCM_sys_check_path"
	2403	}
	2404
[811]	2405	#D-#==================================================
	2406	#D-function IGCM_sys_check_quota
	2407	#D-* Purpose: check user quota. Stop the simulation if quota above 90%
	2408	#D-* Examples:
	2409	#D-
[714]	2410	function IGCM_sys_check_quota {
[776]	2411	IGCM_debug_PushStack "IGCM_sys_check_quota"
	2412	if ( $DEBUG_sys ) ; then
	2413	echo "IGCM_sys_check_quota"
	2414	fi
	2415	# Limit of quota (in %)
	2416	limit_quota=90
	2417
	2418	# Check of the volume
[912]	2419	volume_quota=$(ccc_quota \| grep ' scratch' \| gawk '{print $2}')
	2420	volume_avail=$(ccc_quota \| grep ' scratch' \| gawk '{print $3}')
[776]	2421
	2422	if ( [ ! X${volume_quota} = X ] && [ ! ${volume_quota} = "-" ] ) ; then
	2423
	2424	unit_avail=${volume_avail: -1}
	2425	unit_quota=${volume_quota: -1}
	2426
	2427	if [ "${unit_quota}" = "*" ] ; then
[811]	2428	IGCM_debug_Print 1 "Please, check your quota of volume on scratch"
	2429	IGCM_debug_Print 1 "More than 100% of your quota is used"
	2430	IGCM_debug_Print 1 "Use the ccc_quota command to check"
	2431	IGCM_debug_Print 1 "You must have more than 10% available to run"
	2432	IGCM_debug_Exit "Not enough space to run ! STOP HERE"
	2433	IGCM_debug_Verif_Exit
[714]	2434	fi
[776]	2435
	2436	temp_avail=${volume_avail%%${volume_avail: -1}*}
	2437	temp_quota=${volume_quota%%${volume_quota: -1}*}
	2438
	2439	if [ ! ${unit_avail} = ${unit_quota} ] ; then
	2440
	2441	# Convertion
	2442	if [ ${volume_avail: -1} = "T" ] ; then
	2443	(( temp_avail = temp_avail * 1000000000000 ))
	2444	elif [ ${volume_avail: -1} = "G" ] ; then
	2445	(( temp_avail = temp_avail * 1000000000 ))
	2446	elif [ ${volume_avail: -1} = "M" ] ; then
	2447	(( temp_avail = temp_avail * 1000000 ))
	2448	elif [ ${volume_avail: -1} = "k" ] ; then
	2449	(( temp_avail = temp_avail * 1000 ))
	2450	else
	2451	(( temp_avail = volume_avail ))
	2452	fi
	2453	if [ ${volume_quota: -1} = "T" ] ; then
	2454	(( temp_quota = temp_quota * 1000000000000 ))
	2455	elif [ ${volume_quota: -1} = "G" ] ; then
	2456	(( temp_quota = temp_quota * 1000000000 ))
	2457	elif [ ${volume_quota: -1} = "M" ] ; then
	2458	(( temp_quota = temp_quota * 1000000 ))
	2459	elif [ ${volume_quota: -1} = "k" ] ; then
	2460	(( temp_quota = temp_quota * 1000 ))
	2461	else
	2462	(( temp_quota = volume_quota ))
	2463	fi
	2464	fi
	2465
	2466	quota_volume=$(echo "scale=2 ; $temp_quota/$temp_avail*100" \| bc)
	2467	# echo "volume ratio is " $quota_volume
	2468
	2469	if [ ${quota_volume} -ge ${limit_quota} ] ; then
	2470	IGCM_debug_Print 1 "Please, check your quota of volume on scratch"
	2471	IGCM_debug_Print 1 "${quota_volume}% of your quota is used"
	2472	IGCM_debug_Print 1 "Use the ccc_quota command to check"
	2473	IGCM_debug_Print 1 "You must have more than 10% available to run"
	2474	IGCM_debug_Exit "Not enough space to run ! STOP HERE"
	2475	IGCM_debug_Verif_Exit
	2476	fi
	2477
	2478	fi
	2479
	2480	# Check of the number of inodes
	2481
[912]	2482	inode_quota=$(ccc_quota \| grep ' scratch' \| gawk '{print $6}')
	2483	inode_avail=$(ccc_quota \| grep ' scratch' \| gawk '{print $7}')
[776]	2484
	2485	if ( [ ! X${inode_quota} = X ] && [ ! ${inode_quota} = "-" ] ) ; then
	2486
	2487	unit_avail=${inode_avail: -1}
	2488	unit_quota=${inode_quota: -1}
	2489
	2490	if [ "${unit_quota}" = "*" ] ; then
[811]	2491	IGCM_debug_Print 1 "Please, check your quota of inode on scratch"
	2492	IGCM_debug_Print 1 "More than 100% of your quota is used"
	2493	IGCM_debug_Print 1 "Use the ccc_quota command to check"
	2494	IGCM_debug_Print 1 "You must have more than 10% available to run"
	2495	IGCM_debug_Exit "Not enough space to run ! STOP HERE"
	2496	IGCM_debug_Verif_Exit
[776]	2497	fi
	2498
	2499	temp_avail=${inode_avail%%${inode_avail: -1}*}
	2500	temp_quota=${inode_quota%%${inode_quota: -1}*}
	2501
	2502	if [ ! ${unit_avail} = ${unit_quota} ] ; then
	2503
	2504	# Convertion
	2505	if [ ${inode_avail: -1} = "T" ] ; then
	2506	(( temp_avail = temp_avail * 1000000000000 ))
	2507	elif [ ${inode_avail: -1} = "G" ] ; then
	2508	(( temp_avail = temp_avail * 1000000000 ))
	2509	elif [ ${inode_avail: -1} = "M" ] ; then
	2510	(( temp_avail = temp_avail * 1000000 ))
	2511	elif [ ${inode_avail: -1} = "k" ] ; then
	2512	(( temp_avail = temp_avail * 1000 ))
	2513	else
	2514	(( temp_avail = inode_avail ))
	2515	fi
	2516
	2517	if [ ${inode_quota: -1} = "T" ] ; then
	2518	(( temp_quota = temp_quota * 1000000000000 ))
	2519	elif [ ${inode_quota: -1} = "G" ] ; then
	2520	(( temp_quota = temp_quota * 1000000000 ))
	2521	elif [ ${inode_quota: -1} = "M" ] ; then
	2522	(( temp_quota = temp_quota * 1000000 ))
	2523	elif [ ${inode_quota: -1} = "k" ] ; then
	2524	(( temp_quota = temp_quota * 1000 ))
	2525	else
	2526	(( temp_quota = inode_quota ))
	2527	fi
	2528	fi
	2529	quota_inode=$(echo "scale=2 ; $temp_quota/$temp_avail*100" \| bc)
	2530	# echo "inode ratio is " $quota_inode
	2531
	2532	if [ ${quota_inode} -ge ${limit_quota} ] ; then
	2533	IGCM_debug_Print 1 "Please, check your quota of inode on scratch"
	2534	IGCM_debug_Print 1 "${quota_inode}% of your quota is used"
	2535	IGCM_debug_Print 1 "Use the ccc_quota command to check"
	2536	IGCM_debug_Print 1 "You must have more than 10% available to run"
	2537	IGCM_debug_Exit "Not enough space to run ! STOP HERE"
	2538	IGCM_debug_Verif_Exit
	2539	fi
	2540	fi
	2541	IGCM_debug_PopStack "IGCM_sys_check_quota"
[714]	2542	}
	2543
[773]	2544	#D-#==================================================
[958]	2545	#D-function IGCM_sys_GetJobID
	2546	#D-* Purpose: Check if job_name is currently
	2547	#D- running or in queue
	2548	#D-* Examples: IGCM_sys_GetJobID ${JobName} ${TargetUsr} JobID
	2549	#D-
	2550	function IGCM_sys_GetJobID {
	2551	IGCM_debug_PushStack "IGCM_sys_GetJobID"
	2552	if ( $DEBUG_sys ) ; then
	2553	echo "IGCM_sys_GetJobID"
	2554	fi
	2555
	2556	# With -f option, the full job name is given in the last column
	2557	ID=$( ccc_mstat -f -u $2 \| \
	2558	gawk -v JobName=$1 '( $NF ~ JobName ) { print $1 }' )
	2559
	2560	eval ${3}=${ID}
	2561
	2562	IGCM_debug_PopStack "IGCM_sys_GetJobID"
	2563	}
	2564
	2565	#D-#==================================================
[773]	2566	#D-function IGCM_sys_CountJobInQueue
	2567	#D-* Purpose: Check if job_name is currently
	2568	#D- running or in queue
	2569	#D-* Examples: IGCM_sys_CountJobInQueue ${JobName} NbRun
	2570	#D-
	2571	function IGCM_sys_CountJobInQueue {
	2572	IGCM_debug_PushStack "IGCM_sys_CountJobInQueue"
	2573	if ( $DEBUG_sys ) ; then
	2574	echo "IGCM_sys_CountJobInQueue"
	2575	fi
	2576
	2577	#NbRun=$( ccc_mstat -f \| grep -c ${JobName} )
	2578
	2579	# With -f option, the full job name is given in the last column
[834]	2580	NbRun=$( ccc_mstat -f \| gawk -v JobName=$1 'BEGIN { x=0 } ( $NF ~ JobName ) { x=x+1 } END { print x }' )
[773]	2581
	2582	eval ${2}=${NbRun}
	2583
	2584	IGCM_debug_PopStack "IGCM_sys_CountJobInQueue"
	2585	}
	2586
[955]	2587	#D-#==================================================
	2588	#D-function IGCM_sys_ListJobInQueue
	2589	#D-* Purpose: Check if job_name is currently
	2590	#D- running or in queue
	2591	#D-* Examples: IGCM_sys_ListJobInQueue ${User} JobNameList
	2592	#D-
	2593	function IGCM_sys_ListJobInQueue {
	2594	IGCM_debug_PushStack "IGCM_sys_ListJobInQueue"
	2595	if ( $DEBUG_sys ) ; then
	2596	echo "IGCM_sys_ListJobInQueue"
	2597	fi
	2598
	2599	# With -f option, the full job name is given in the last column
	2600	set -A JobList $( ccc_mstat -f \| gawk -v User=$1 \
	2601	'( $2 == User && \
	2602	$NF != /TS/ && \
	2603	$NF !~ /PACK/ && \
	2604	$NF !~ /REBUILD/ && \
	2605	$NF !~ /pack/ ) \
	2606	{ print $NF }' \| sed -e "s/$.$\.[0-9]/\1/" )
	2607
	2608	eval set -A ${2} ${JobList[*]}
	2609
	2610	IGCM_debug_PopStack "IGCM_sys_ListJobInQueue"
	2611	}
	2612
[622]	2613	##############################################################
	2614	# NCO OPERATOR
	2615
[811]	2616	#D-#==================================================
	2617	#D-function IGCM_sys_ncap2
	2618	#D-* Purpose: encapsulate ncap2 call so as to manage error code and retry
	2619	#D-* Examples:
	2620	#D-
[623]	2621	function IGCM_sys_ncap2 {
[1057]	2622	IGCM_debug_PushStack "IGCM_sys_ncap2" $@
[623]	2623	if ( $DEBUG_sys ) ; then
	2624	echo "IGCM_sys_ncap2 :" $@
	2625	fi
[713]	2626
	2627	typeset NB_ESSAI DELAI status i
	2628	# number of tentative
	2629	NB_ESSAI=3
	2630	# time delay between tentative
	2631	DELAI=2
	2632
	2633	i=0
	2634	while [ $i -lt $NB_ESSAI ] ; do
[1091]	2635	ncap2 -C "$@" > /tmp/out_command.$$ 2>&1
[713]	2636	status=$?
	2637	if [ ${status} -gt 0 ] ; then
	2638	IGCM_debug_Print 2 "IGCM_sys_ncap2 : error code ${status}"
[938]	2639	cat /tmp/out_command.$$
	2640	\rm /tmp/out_command.$$
[713]	2641	IGCM_debug_Print 2 "IGCM_sys_ncap2 : ${i}/${NB_ESSAI} sleep ${DELAI} seconds and try again."
[733]	2642	sleep $DELAI
[713]	2643	else
[938]	2644	\rm /tmp/out_command.$$
[713]	2645	break
	2646	fi
	2647	(( i = i + 1 ))
	2648	done
	2649
	2650	if [ ${status} -gt 0 ] ; then
[811]	2651	echo "IGCM_sys_ncap2 : ncap2 error"
	2652	IGCM_debug_Exit "ncap2"
[623]	2653	fi
	2654
	2655	IGCM_debug_PopStack "IGCM_sys_ncap2"
	2656	}
	2657
[811]	2658	#D-#==================================================
	2659	#D-function IGCM_sys_ncatted
	2660	#D-* Purpose: encapsulate ncatted call so as to manage error code and retry
	2661	#D-* Examples:
	2662	#D-
[622]	2663	function IGCM_sys_ncatted {
[1057]	2664	IGCM_debug_PushStack "IGCM_sys_ncatted" $@
[623]	2665	if ( $DEBUG_sys ) ; then
	2666	echo "IGCM_sys_ncatted :" $@
	2667	fi
[713]	2668
	2669	typeset NB_ESSAI DELAI status i
	2670	# number of tentative
	2671	NB_ESSAI=3
	2672	# time delay between tentative
	2673	DELAI=2
	2674
	2675	i=0
	2676	while [ $i -lt $NB_ESSAI ] ; do
[938]	2677	ncatted "$@" > /tmp/out_command.$$ 2>&1
[713]	2678	status=$?
	2679	if [ ${status} -gt 0 ] ; then
	2680	IGCM_debug_Print 2 "IGCM_sys_ncatted : error code ${status}"
[938]	2681	cat /tmp/out_command.$$
	2682	\rm /tmp/out_command.$$
[713]	2683	IGCM_debug_Print 2 "IGCM_sys_ncatted : ${i}/${NB_ESSAI} sleep ${DELAI} seconds and try again."
[733]	2684	sleep $DELAI
[713]	2685	else
[938]	2686	\rm /tmp/out_command.$$
[713]	2687	break
	2688	fi
	2689	(( i = i + 1 ))
	2690	done
	2691
	2692	if [ ${status} -gt 0 ] ; then
[811]	2693	echo "IGCM_sys_ncatted : ncatted error"
	2694	IGCM_debug_Exit "ncatted"
[623]	2695	fi
[622]	2696
[623]	2697	IGCM_debug_PopStack "IGCM_sys_ncatted"
[622]	2698	}
	2699
[811]	2700	#D-#==================================================
	2701	#D-function IGCM_sys_ncbo
	2702	#D-* Purpose: encapsulate ncbo call so as to manage error code and retry
	2703	#D-* Examples:
	2704	#D-
[622]	2705	function IGCM_sys_ncbo {
[1057]	2706	IGCM_debug_PushStack "IGCM_sys_ncbo" $@
[623]	2707	if ( $DEBUG_sys ) ; then
	2708	echo "IGCM_sys_ncbo :" $@
	2709	fi
[713]	2710
	2711	typeset NB_ESSAI DELAI status i
	2712	# number of tentative
	2713	NB_ESSAI=3
	2714	# time delay between tentative
	2715	DELAI=2
	2716
	2717	i=0
	2718	while [ $i -lt $NB_ESSAI ] ; do
[1091]	2719	ncbo -C $@ > /tmp/out_command.$$ 2>&1
[713]	2720	status=$?
	2721	if [ ${status} -gt 0 ] ; then
	2722	IGCM_debug_Print 2 "IGCM_sys_ncbo : error code ${status}"
[938]	2723	cat /tmp/out_command.$$
	2724	\rm /tmp/out_command.$$
[713]	2725	IGCM_debug_Print 2 "IGCM_sys_ncbo : ${i}/${NB_ESSAI} sleep ${DELAI} seconds and try again."
[733]	2726	sleep $DELAI
[713]	2727	else
[938]	2728	\rm /tmp/out_command.$$
[713]	2729	break
	2730	fi
	2731	(( i = i + 1 ))
	2732	done
	2733
	2734	if [ ${status} -gt 0 ] ; then
[811]	2735	echo "IGCM_sys_ncbo : ncbo error"
	2736	IGCM_debug_Exit "ncbo"
[623]	2737	fi
[622]	2738
[623]	2739	IGCM_debug_PopStack "IGCM_sys_ncbo"
[622]	2740	}
	2741
[811]	2742	#D-#==================================================
	2743	#D-function IGCM_sys_ncdif
	2744	#D-* Purpose: encapsulate ncdiff call so as to manage error code and retry
	2745	#D-* Examples:
	2746	#D-
[622]	2747	function IGCM_sys_ncdiff {
[1057]	2748	IGCM_debug_PushStack "IGCM_sys_ncdiff" $@
[623]	2749	if ( $DEBUG_sys ) ; then
	2750	echo "IGCM_sys_ncdiff :" $@
	2751	fi
[713]	2752
	2753	typeset NB_ESSAI DELAI status i
	2754	# number of tentative
	2755	NB_ESSAI=3
	2756	# time delay between tentative
	2757	DELAI=2
	2758
	2759	i=0
	2760	while [ $i -lt $NB_ESSAI ] ; do
[1091]	2761	ncdiff -C $@ > /tmp/out_command.$$ 2>&1
[713]	2762	status=$?
	2763	if [ ${status} -gt 0 ] ; then
	2764	IGCM_debug_Print 2 "IGCM_sys_ncdiff : error code ${status}"
[938]	2765	cat /tmp/out_command.$$
	2766	\rm /tmp/out_command.$$
[713]	2767	IGCM_debug_Print 2 "IGCM_sys_ncdiff : ${i}/${NB_ESSAI} sleep ${DELAI} seconds and try again."
[733]	2768	sleep $DELAI
[713]	2769	else
[938]	2770	\rm /tmp/out_command.$$
[713]	2771	break
	2772	fi
	2773	(( i = i + 1 ))
	2774	done
	2775
	2776	if [ ${status} -gt 0 ] ; then
[811]	2777	echo "IGCM_sys_ncdiff : ncdiff error"
	2778	IGCM_debug_Exit "ncdiff"
[623]	2779	fi
[622]	2780
[623]	2781	IGCM_debug_PopStack "IGCM_sys_ncdiff"
[622]	2782	}
	2783
[811]	2784	#D-#==================================================
	2785	#D-function IGCM_sys_ncea
	2786	#D-* Purpose: encapsulate ncea call so as to manage error code and retry
	2787	#D-* Examples:
	2788	#D-
[622]	2789	function IGCM_sys_ncea {
[1057]	2790	IGCM_debug_PushStack "IGCM_sys_ncea" $@
[623]	2791	if ( $DEBUG_sys ) ; then
	2792	echo "IGCM_sys_ncea :" $@
	2793	fi
[713]	2794
	2795	typeset NB_ESSAI DELAI status i
	2796	# number of tentative
	2797	NB_ESSAI=3
	2798	# time delay between tentative
	2799	DELAI=2
	2800
	2801	i=0
	2802	while [ $i -lt $NB_ESSAI ] ; do
[1091]	2803	ncea -C $@ > /tmp/out_command.$$ 2>&1
[713]	2804	status=$?
	2805	if [ ${status} -gt 0 ] ; then
	2806	IGCM_debug_Print 2 "IGCM_sys_ncea : error code ${status}"
[938]	2807	cat /tmp/out_command.$$
	2808	\rm /tmp/out_command.$$
[713]	2809	IGCM_debug_Print 2 "IGCM_sys_ncea : ${i}/${NB_ESSAI} sleep ${DELAI} seconds and try again."
[733]	2810	sleep $DELAI
[713]	2811	else
[938]	2812	\rm /tmp/out_command.$$
[713]	2813	break
	2814	fi
	2815	(( i = i + 1 ))
	2816	done
	2817
	2818	if [ ${status} -gt 0 ] ; then
[811]	2819	echo "IGCM_sys_ncea : ncea error"
	2820	IGCM_debug_Exit "ncea"
[623]	2821	fi
[622]	2822
[623]	2823	IGCM_debug_PopStack "IGCM_sys_ncea"
[622]	2824	}
	2825
[811]	2826	#D-#==================================================
	2827	#D-function IGCM_sys_ncecat
	2828	#D-* Purpose: encapsulate ncecat call so as to manage error code and retry
	2829	#D-* Examples:
	2830	#D-
[622]	2831	function IGCM_sys_ncecat {
[1057]	2832	IGCM_debug_PushStack "IGCM_sys_ncecat" $@
[623]	2833	if ( $DEBUG_sys ) ; then
	2834	echo "IGCM_sys_ncecat :" $@
	2835	fi
[713]	2836
	2837	typeset NB_ESSAI DELAI status i
	2838	# number of tentative
	2839	NB_ESSAI=3
	2840	# time delay between tentative
	2841	DELAI=2
	2842
	2843	i=0
	2844	while [ $i -lt $NB_ESSAI ] ; do
[1091]	2845	ncecat -C $@ > /tmp/out_command.$$ 2>&1
[713]	2846	status=$?
	2847	if [ ${status} -gt 0 ] ; then
	2848	IGCM_debug_Print 2 "IGCM_sys_ncecat : error code ${status}"
[938]	2849	cat /tmp/out_command.$$
	2850	\rm /tmp/out_command.$$
[713]	2851	IGCM_debug_Print 2 "IGCM_sys_ncecat : ${i}/${NB_ESSAI} sleep ${DELAI} seconds and try again."
[733]	2852	sleep $DELAI
[713]	2853	else
[938]	2854	\rm /tmp/out_command.$$
[713]	2855	break
	2856	fi
	2857	(( i = i + 1 ))
	2858	done
	2859
	2860	if [ ${status} -gt 0 ] ; then
[811]	2861	echo "IGCM_sys_ncecat : ncecat error"
	2862	IGCM_debug_Exit "ncecat"
[623]	2863	fi
[622]	2864
[623]	2865	IGCM_debug_PopStack "IGCM_sys_ncecat"
[622]	2866	}
	2867
[811]	2868	#D-#==================================================
	2869	#D-function IGCM_sys_ncflint
	2870	#D-* Purpose: encapsulate ncflint call so as to manage error code and retry
	2871	#D-* Examples:
	2872	#D-
[622]	2873	function IGCM_sys_ncflint {
[1057]	2874	IGCM_debug_PushStack "IGCM_sys_ncflint" $@
[623]	2875	if ( $DEBUG_sys ) ; then
	2876	echo "IGCM_sys_ncflint :" $@
	2877	fi
[713]	2878
	2879	typeset NB_ESSAI DELAI status i
	2880	# number of tentative
	2881	NB_ESSAI=3
	2882	# time delay between tentative
	2883	DELAI=2
	2884
	2885	i=0
	2886	while [ $i -lt $NB_ESSAI ] ; do
[1091]	2887	ncflint -C $@ > /tmp/out_command.$$ 2>&1
[713]	2888	status=$?
	2889	if [ ${status} -gt 0 ] ; then
	2890	IGCM_debug_Print 2 "IGCM_sys_ncflint : error code ${status}"
[938]	2891	cat /tmp/out_command.$$
	2892	\rm /tmp/out_command.$$
[713]	2893	IGCM_debug_Print 2 "IGCM_sys_ncflint : ${i}/${NB_ESSAI} sleep ${DELAI} seconds and try again."
[733]	2894	sleep $DELAI
[713]	2895	else
[938]	2896	\rm /tmp/out_command.$$
[713]	2897	break
	2898	fi
	2899	(( i = i + 1 ))
	2900	done
	2901
	2902	if [ ${status} -gt 0 ] ; then
[811]	2903	echo "IGCM_sys_ncflint : ncflint error"
	2904	IGCM_debug_Exit "ncflint"
[623]	2905	fi
[622]	2906
[623]	2907	IGCM_debug_PopStack "IGCM_sys_ncflint"
[622]	2908	}
	2909
[811]	2910	#D-#==================================================
	2911	#D-function IGCM_sys_ncks
	2912	#D-* Purpose: encapsulate ncks call so as to manage error code and retry
	2913	#D-* Examples:
	2914	#D-
[622]	2915	function IGCM_sys_ncks {
[1057]	2916	IGCM_debug_PushStack "IGCM_sys_ncks" $@
[623]	2917	if ( $DEBUG_sys ) ; then
	2918	echo "IGCM_sys_ncks :" $@
	2919	fi
[713]	2920
	2921	typeset NB_ESSAI DELAI status i
	2922	# number of tentative
	2923	NB_ESSAI=3
	2924	# time delay between tentative
	2925	DELAI=2
	2926
	2927	i=0
	2928	while [ $i -lt $NB_ESSAI ] ; do
[1091]	2929	ncks -C $@ > /tmp/out_command.$$ 2>&1
[713]	2930	status=$?
	2931	if [ ${status} -gt 0 ] ; then
	2932	IGCM_debug_Print 2 "IGCM_sys_ncks : error code ${status}"
[938]	2933	cat /tmp/out_command.$$
	2934	\rm /tmp/out_command.$$
[713]	2935	IGCM_debug_Print 2 "IGCM_sys_ncks : ${i}/${NB_ESSAI} sleep ${DELAI} seconds and try again."
[733]	2936	sleep $DELAI
[713]	2937	else
[938]	2938	\rm /tmp/out_command.$$
[713]	2939	break
	2940	fi
	2941	(( i = i + 1 ))
	2942	done
	2943
	2944	if [ ${status} -gt 0 ] ; then
[811]	2945	echo "IGCM_sys_ncks : ncks error"
	2946	IGCM_debug_Exit "ncks"
[623]	2947	fi
[622]	2948
[623]	2949	IGCM_debug_PopStack "IGCM_sys_ncks"
[622]	2950	}
	2951
[811]	2952	#D-#==================================================
	2953	#D-function IGCM_sys_ncpdq
	2954	#D-* Purpose: encapsulate ncpdq call so as to manage error code and retry
	2955	#D-* Examples:
	2956	#D-
[622]	2957	function IGCM_sys_ncpdq {
[1057]	2958	IGCM_debug_PushStack "IGCM_sys_ncpdq" $@
[623]	2959	if ( $DEBUG_sys ) ; then
	2960	echo "IGCM_sys_ncpdq :" $@
	2961	fi
[713]	2962
	2963	typeset NB_ESSAI DELAI status i
	2964	# number of tentative
	2965	NB_ESSAI=3
	2966	# time delay between tentative
	2967	DELAI=2
	2968
	2969	i=0
	2970	while [ $i -lt $NB_ESSAI ] ; do
[1091]	2971	ncpdq -C $@ > /tmp/out_command.$$ 2>&1
[713]	2972	status=$?
	2973	if [ ${status} -gt 0 ] ; then
	2974	IGCM_debug_Print 2 "IGCM_sys_ncpdq : error code ${status}"
[938]	2975	cat /tmp/out_command.$$
	2976	\rm /tmp/out_command.$$
[713]	2977	IGCM_debug_Print 2 "IGCM_sys_ncpdq : ${i}/${NB_ESSAI} sleep ${DELAI} seconds and try again."
[733]	2978	sleep $DELAI
[713]	2979	else
[938]	2980	\rm /tmp/out_command.$$
[713]	2981	break
	2982	fi
	2983	(( i = i + 1 ))
	2984	done
	2985
	2986	if [ ${status} -gt 0 ] ; then
[811]	2987	echo "IGCM_sys_ncpdq : ncpdq error"
	2988	IGCM_debug_Exit "ncpdq"
[623]	2989	fi
[622]	2990
[623]	2991	IGCM_debug_PopStack "IGCM_sys_ncpdq"
[622]	2992	}
	2993
[811]	2994	#D-#==================================================
	2995	#D-function IGCM_sys_ncra
	2996	#D-* Purpose: encapsulate ncra call so as to manage error code and retry
	2997	#D-* Examples:
	2998	#D-
[622]	2999	function IGCM_sys_ncra {
[1057]	3000	IGCM_debug_PushStack "IGCM_sys_ncra" $@
[623]	3001	if ( $DEBUG_sys ) ; then
	3002	echo "IGCM_sys_ncra :" $@
	3003	fi
[713]	3004
	3005	typeset NB_ESSAI DELAI status i
	3006	# number of tentative
	3007	NB_ESSAI=3
	3008	# time delay between tentative
	3009	DELAI=2
	3010
	3011	i=0
	3012	while [ $i -lt $NB_ESSAI ] ; do
[1091]	3013	ncra -C $@ > /tmp/out_command.$$ 2>&1
[713]	3014	status=$?
	3015	if [ ${status} -gt 0 ] ; then
	3016	IGCM_debug_Print 2 "IGCM_sys_ncra : error code ${status}"
[938]	3017	cat /tmp/out_command.$$
	3018	\rm /tmp/out_command.$$
[713]	3019	IGCM_debug_Print 2 "IGCM_sys_ncra : ${i}/${NB_ESSAI} sleep ${DELAI} seconds and try again."
[733]	3020	sleep $DELAI
[713]	3021	else
[938]	3022	\rm /tmp/out_command.$$
[713]	3023	break
	3024	fi
	3025	(( i = i + 1 ))
	3026	done
	3027
	3028	if [ ${status} -gt 0 ] ; then
[811]	3029	echo "IGCM_sys_ncra : ncra error"
	3030	IGCM_debug_Exit "ncra"
[623]	3031	fi
[622]	3032
[623]	3033	IGCM_debug_PopStack "IGCM_sys_ncra"
[622]	3034	}
	3035
[811]	3036	#D-#==================================================
	3037	#D-function IGCM_sys_ncrcat
	3038	#D-* Purpose: encapsulate ncrcat call so as to manage error code and retry
	3039	#D-* Examples:
	3040	#D-
[622]	3041	function IGCM_sys_ncrcat {
[1057]	3042	IGCM_debug_PushStack "IGCM_sys_ncrcat" $@
[623]	3043	if ( $DEBUG_sys ) ; then
	3044	echo "IGCM_sys_ncrcat :" $@
	3045	fi
[713]	3046
[725]	3047	typeset NB_ESSAI DELAI status i lastArg
[713]	3048	# number of tentative
	3049	NB_ESSAI=3
	3050	# time delay between tentative
	3051	DELAI=2
	3052
	3053	i=0
	3054	while [ $i -lt $NB_ESSAI ] ; do
[1091]	3055	ncrcat -C $@ > /tmp/out_command.$$ 2>&1
[713]	3056	status=$?
	3057	if [ ${status} -gt 0 ] ; then
	3058	IGCM_debug_Print 2 "IGCM_sys_ncrcat : error code ${status}"
[938]	3059	cat /tmp/out_command.$$
	3060	\rm /tmp/out_command.$$
[713]	3061	IGCM_debug_Print 2 "IGCM_sys_ncrcat : ${i}/${NB_ESSAI} sleep ${DELAI} seconds and try again."
[733]	3062	sleep $DELAI
[938]	3063	elif [ ! "X$( grep "WARNING Intra-file non-monotonicity" /tmp/out_command.$$ )" = "X" ] ; then
[721]	3064	IGCM_debug_Print 2 "IGCM_sys_ncrcat : WARNING Intra-file non-monotonicity"
[938]	3065	cat /tmp/out_command.$$
[725]	3066	# remove files having corrupted time axis
[734]	3067	eval lastArg=\${$#}
[725]	3068	IGCM_debug_Print 2 "IGCM_sys_ncrcat : Delete ${lastArg}"
	3069	\rm ${lastArg}
[938]	3070	\rm /tmp/out_command.$$
[721]	3071	IGCM_debug_Print 2 "IGCM_sys_ncrcat : ${i}/${NB_ESSAI} sleep ${DELAI} seconds and try again."
[733]	3072	sleep $DELAI
[713]	3073	else
[938]	3074	\rm /tmp/out_command.$$
[713]	3075	break
	3076	fi
	3077	(( i = i + 1 ))
	3078	done
	3079
	3080	if [ ${status} -gt 0 ] ; then
[811]	3081	echo "IGCM_sys_ncrcat : ncrcat error"
	3082	#IGCM_debug_Exit "ncrcat"
[623]	3083	fi
[622]	3084
[623]	3085	IGCM_debug_PopStack "IGCM_sys_ncrcat"
[622]	3086	}
	3087
[811]	3088	#D-#==================================================
	3089	#D-function IGCM_sys_ncrename
	3090	#D-* Purpose: encapsulate ncrename call so as to manage error code and retry
	3091	#D-* Examples:
	3092	#D-
[622]	3093	function IGCM_sys_ncrename {
[1057]	3094	IGCM_debug_PushStack "IGCM_sys_ncrename" $@
[623]	3095	if ( $DEBUG_sys ) ; then
	3096	echo "IGCM_sys_ncrename :" $@
	3097	fi
[713]	3098
	3099	typeset NB_ESSAI DELAI status i
	3100	# number of tentative
	3101	NB_ESSAI=3
	3102	# time delay between tentative
	3103	DELAI=2
	3104
	3105	i=0
	3106	while [ $i -lt $NB_ESSAI ] ; do
[938]	3107	ncrename $@ > /tmp/out_command.$$ 2>&1
[713]	3108	status=$?
	3109	if [ ${status} -gt 0 ] ; then
	3110	IGCM_debug_Print 2 "IGCM_sys_ncrename : error code ${status}"
[938]	3111	cat /tmp/out_command.$$
	3112	\rm /tmp/out_command.$$
[713]	3113	IGCM_debug_Print 2 "IGCM_sys_ncrename : ${i}/${NB_ESSAI} sleep ${DELAI} seconds and try again."
[733]	3114	sleep $DELAI
[713]	3115	else
[938]	3116	\rm /tmp/out_command.$$
[713]	3117	break
	3118	fi
	3119	(( i = i + 1 ))
	3120	done
	3121
	3122	if [ ${status} -gt 0 ] ; then
[811]	3123	echo "IGCM_sys_ncrename : ncrename error"
	3124	IGCM_debug_Exit "ncrename"
[623]	3125	fi
[622]	3126
[623]	3127	IGCM_debug_PopStack "IGCM_sys_ncrename"
[622]	3128	}
	3129
[811]	3130	#D-#==================================================
	3131	#D-function IGCM_sys_ncwa
	3132	#D-* Purpose: encapsulate ncwa call so as to manage error code and retry
	3133	#D-* Examples:
	3134	#D-
[622]	3135	function IGCM_sys_ncwa {
[1057]	3136	IGCM_debug_PushStack "IGCM_sys_ncwa" $@
[623]	3137	if ( $DEBUG_sys ) ; then
	3138	echo "IGCM_sys_ncwa :" $@
	3139	fi
[713]	3140
	3141	typeset NB_ESSAI DELAI status i
	3142	# number of tentative
	3143	NB_ESSAI=3
	3144	# time delay between tentative
	3145	DELAI=2
	3146
	3147	i=0
	3148	while [ $i -lt $NB_ESSAI ] ; do
[1091]	3149	ncwa -C $@ > /tmp/out_command.$$ 2>&1
[713]	3150	status=$?
	3151	if [ ${status} -gt 0 ] ; then
	3152	IGCM_debug_Print 2 "IGCM_sys_ncwa : error code ${status}"
[938]	3153	cat /tmp/out_command.$$
	3154	\rm /tmp/out_command.$$
[713]	3155	IGCM_debug_Print 2 "IGCM_sys_ncwa : ${i}/${NB_ESSAI} sleep ${DELAI} seconds and try again."
[733]	3156	sleep $DELAI
[713]	3157	else
[938]	3158	\rm /tmp/out_command.$$
[713]	3159	break
	3160	fi
	3161	(( i = i + 1 ))
	3162	done
	3163
	3164	if [ ${status} -gt 0 ] ; then
[811]	3165	echo "IGCM_sys_ncwa : ncwa error"
	3166	IGCM_debug_Exit "ncwa"
[623]	3167	fi
[622]	3168
[623]	3169	IGCM_debug_PopStack "IGCM_sys_ncwa"
[622]	3170	}
[623]	3171
	3172	##############################################################
	3173	# CDO OPERATOR
	3174
[811]	3175	#D-#==================================================
	3176	#D-function IGCM_sys_cdo
	3177	#D-* Purpose: encapsulate cdo call so as to manage error code and retry
	3178	#D-* Examples:
	3179	#D-
[623]	3180	function IGCM_sys_cdo {
[1057]	3181	IGCM_debug_PushStack "IGCM_sys_cdo" $@
[623]	3182	if ( $DEBUG_sys ) ; then
	3183	echo "IGCM_sys_cdo :" $@
	3184	fi
[713]	3185
	3186	typeset status
	3187
[938]	3188	\cdo $@ > /tmp/out_command.$$ 2>&1
[713]	3189	status=$?
	3190	if [ ${status} -gt 0 ] ; then
	3191	echo "IGCM_sys_cdo : error code ${status}"
[938]	3192	cat /tmp/out_command.$$
	3193	\rm /tmp/out_command.$$
[623]	3194	IGCM_debug_PopStack "IGCM_sys_cdo"
	3195	return 1
	3196	else
	3197	IGCM_debug_PopStack "IGCM_sys_cdo"
	3198	return 0
	3199	fi
	3200
	3201	IGCM_debug_PopStack "IGCM_sys_cdo"
	3202	}

Note: See TracBrowser for help on using the repository browser.

Context Navigation

source: trunk/libIGCM/libIGCM_sys/libIGCM_sys_curie.ksh @ 1173

Download in other formats: