Context Navigation

source: trunk/libIGCM/libIGCM_sys/libIGCM_sys_default.ksh @ 1134

Last change on this file since 1134 was 1134, checked in by sdipsl, 10 years ago
Bugfix due to a side effect of the IDRIS /tmp workaround
Property licence set to The following licence information concerns ONLY the libIGCM tools ================================================================== Copyright © Centre National de la Recherche Scientifique CNRS Commissariat à l'Énergie Atomique CEA libIGCM : Library for Portable Models Computation of IGCM Group. IGCM Group is the french IPSL Global Climate Model Group. This library is a set of shell scripts and functions whose purpose is the management of the initialization, the launch, the transfer of output files, the post-processing and the monitoring of datas produce by any numerical program on any plateforme. This software is governed by the CeCILL license under French law and abiding by the rules of distribution of free software. You can use, modify and/ or redistribute the software under the terms of the CeCILL license as circulated by CEA, CNRS and INRIA at the following URL "http://www.cecill.info". As a counterpart to the access to the source code and rights to copy, modify and redistribute granted by the license, users are provided only with a limited warranty and the software's author, the holder of the economic rights, and the successive licensors have only limited liability. In this respect, the user's attention is drawn to the risks associated with loading, using, modifying and/or developing or reproducing the software by the user in light of its specific status of free software, that may mean that it is complicated to manipulate, and that also therefore means that it is reserved for developers and experienced professionals having in-depth computer knowledge. Users are therefore encouraged to load and test the software's suitability as regards their requirements in conditions enabling the security of their systems and/or data to be ensured and, more generally, to use and operate it in the same conditions as regards security. The fact that you are presently reading this means that you have had knowledge of the CeCILL license and that you accept its terms. Property svn:keywords set to `Revision Author Date`
File size: 71.8 KB

Rev	Line
[664]	1	#!/bin/ksh
[2]	2
	3	#**************************************************************
	4	# Author: Sebastien Denvil, Martial Mancip
[373]	5	# Contact: Martial.Mancip__at__ipsl.jussieu.fr
	6	# $Revision:: $ Revision of last commit
	7	# $Author:: $ Author of last commit
	8	# $Date:: $ Date of last commit
[2]	9	# IPSL (2006)
	10	# This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
	11	#
	12	#**************************************************************
	13
	14	#=========================================================
	15	# The documentation of this file can be automatically generated
[664]	16	# if you use the prefix #D- for comments to be extracted.
[2]	17	# Extract with command: cat lib* \| grep "^#D-" \| cut -c "4-"
	18	#=========================================================
[664]	19
[2]	20	#D-#==================================================
	21	#D-LibIGCM_sys Default host
	22	#D-#==================================================
	23	#D-
[664]	24	#D- This ksh library if a layer under some usefull
[800]	25	#D-environment variables and shell commands.
	26	#D-All those definitions depend on host particularities.
	27	#D-It manages a stack mechanism and test validity of operations.
	28	#D-All function described bellow must be prefixed by IGCM_sys.
[2]	29
	30	#====================================================
	31	# libIGCM_sys PARAMETERS
	32	#====================================================
	33
[117]	34	#====================================================
[800]	35	# set DEBUG_sys to true to output calls of function
[2]	36	typeset -r DEBUG_sys=${DEBUG_sys:=true}
	37
[117]	38	#====================================================
[2]	39	# Turn in dry run mode ? (sys_Put_Rest, sys_Put_Out, sys_Get)
	40	typeset -r DRYRUN=${DRYRUN:=0}
	41
	42	# YOU MUST COMPILE YOUR EXE FILES FOR DRYRUN MODE !
	43	# -------------------------------------------------------------------------------------
	44	# \| DRYRUN= \| Date computations, \| sys_Get \| Exe \| sys_Put_Out; sys_Put_Rest \|
[809]	45	# \| \| Cp/Exe/param/files \| \| \| \|
	46	# \| \| Chmod Qsub \| \| \| \|
[2]	47	# -------------------------------------------------------------------------------------
	48	# \| 0 \| yes \| yes \| yes \| yes \|
	49	# -------------------------------------------------------------------------------------
	50	# \| 1 \| yes \| yes \| yes \| no \|
	51	# -------------------------------------------------------------------------------------
	52	# \| 2 \| yes \| yes \| no \| no \|
	53	# -------------------------------------------------------------------------------------
	54	# \| 3 \| yes \| no \| no \| no \|
	55	# -------------------------------------------------------------------------------------
	56
	57	#=====================================================
[800]	58	# Global Variables :
[2]	59	#=====================================================
	60	# Language : "fr" or "en"
	61	typeset -r MYLANG="fr"
	62
	63	#=====================================================
[664]	64	# Host and user names
[217]	65	# $hostname ou hostname
[228]	66	typeset HOST=${HOST:=$( hostname )}
[2]	67	# $username ou whoami
[228]	68	typeset LOGIN=${LOGIN:=$( whoami )}
[71]	69	# $hostname of the MASTER job
	70	typeset -r MASTER=${HOST}
[2]	71
	72	#D-
	73	#D-#==================================================
	74	#D-Program used in libIGCM
	75	#D-#==================================================
	76
[742]	77	# Submit command
	78	typeset SUBMIT=${SUBMIT:=qsub}
[2]	79	# rsync with path
[664]	80	typeset -r RSYNC=/usr/bin/rsync
[800]	81	# RSYNC_opt args to rsync
[117]	82	typeset -r RSYNC_opt="-va"
[664]	83	# ie storage filesystem
	84	typeset -r STOREHOST=${MASTER}
[305]	85
[2]	86	#====================================================
[305]	87	# Set environment tools (ferret, nco, cdo)
	88	#====================================================
	89	#. /home/${LOGIN}/.atlas_env_${HOST}_ksh
[2]	90
[855]	91	export PATH=${PATH}:/home/jripsl/snapshot/Monitoring/CNClient/
	92
[117]	93	#====================================================
[2]	94	# Host specific DIRECTORIES
	95	#====================================================
	96
	97	#====================================================
[664]	98	#- MirrorlibIGCM for frontend
	99	typeset -r MirrorlibIGCM=${MirrorlibIGCM:=false}
[2]	100
	101	#====================================================
[800]	102	#- libIGCM_POST for frontend
[343]	103	typeset -r libIGCM_POST=${libIGCM}
	104
	105	#====================================================
[664]	106	#- R_EXE (==> BIN_DIR = ${MODIPSL}/bin )
	107	typeset -r R_EXE="${MODIPSL}/bin"
	108
	109	#====================================================
[2]	110	#- SUBMIT_DIR : submission dir
[1124]	111	typeset -x SUBMIT_DIR=${SUBMIT_DIR:=${PWD}}
[2]	112
	113	#====================================================
[725]	114	#- IN
	115	typeset -r R_IN=${R_IN:=/home/${LOGIN}/IGCM}
[800]	116	typeset -r R_IN_ECMWF=${R_IN_ECMWF:=/home/${LOGIN}/IGCM}
[725]	117
	118	#====================================================
[2]	119	#- RUN_DIR_PATH : Temporary working directory (=> TMP)
[1064]	120	typeset -r RUN_DIR_PATH=${RUN_DIR_PATH:=/home/${LOGIN}/RUN_DIR/tmp$$}
[1134]	121	# Due to a workaround at IDRIS. /tmp is not functionning correctly there.
	122	IGCM_sys_MkdirWork ${RUN_DIR_PATH}
[2]	123
	124	#====================================================
	125	#- HOST_MPIRUN_COMMAND
	126	typeset -r HOST_MPIRUN_COMMAND=${HOST_MPIRUN_COMMAND:="time mpirun"}
	127
[61]	128	#====================================================
	129	#- Max number of arguments passed to nco operator or demigration command
	130	UNIX_MAX_LIMIT=360
	131
[664]	132	#====================================================
	133	#- Number of core per node (max number of OpenMP task)
	134	NUM_COREPERNODE=1
	135
	136	#====================================================
	137	#- Default number of MPI task for IPSL coupled model
	138	#- required for backward compatibility
	139	#-
	140	DEFAULT_NUM_PROC_OCE=1
	141	DEFAULT_NUM_PROC_CPL=1
	142	(( DEFAULT_NUM_PROC_ATM = BATCH_NUM_PROC_TOT - DEFAULT_NUM_PROC_OCE - DEFAULT_NUM_PROC_CPL ))
	143	DEFAULT_NUM_PROC_TOTAL=${BATCH_NUM_PROC_TOT}
	144
[2]	145	#D-#==================================================
[1057]	146	#D-function IGCM_sys_defineArchives
	147	#D-* Purpose:
	148	#D-* Define ARCHIVE : Dedicated to large files
	149	#D-* Define STORAGE : Dedicated to small/medium files
	150	#D-* Define R_OUT : Output tree located on ARCHIVE
	151	#D-* Define R_FIG : Output tree located on STORAGE hosting figures (monitoring and atlas, and/or small files)
	152	#D-* Define R_BUF : Output tree located on STORAGE hosting files waiting for rebuild or pack processes
	153	#D-* if SpaceName=TEST nothing special will hapen
[725]	154	#D-* Examples:
	155	#D-
[1057]	156	function IGCM_sys_defineArchives {
	157	IGCM_debug_PushStack "IGCM_sys_defineArchives"
	158
[1060]	159	if [ ! X${config_UserChoices_ARCHIVE} = X ]; then
[1057]	160	#====================================================
	161	#- ARCHIVE (dedicated to large files)
[1063]	162	ARCHIVE=${config_UserChoices_ARCHIVE}
[1057]	163	else
	164	#====================================================
	165	#- ARCHIVE (dedicated to large files)
[1063]	166	ARCHIVE=${ARCHIVE:=/home/${LOGIN}}
[1057]	167	fi
	168
[1060]	169	if [ ! X${config_UserChoices_STORAGE} = X ]; then
[1057]	170	#====================================================
	171	#- STORAGE (dedicated to small/medium files)
[1063]	172	STORAGE=${config_UserChoices_STORAGE}
[1057]	173	else
	174	#====================================================
	175	#- STORAGE (dedicated to small/medium files)
[1063]	176	STORAGE=${ARCHIVE}
[1057]	177	fi
	178
	179	# ON OBELIX NO SPECIAL CASE WHEN X${config_UserChoices_SpaceName} = XTEST
	180
	181	#====================================================
	182	#- R_OUT
[1063]	183	R_OUT=${ARCHIVE}/IGCM_OUT
[1057]	184
	185	#====================================================
	186	#- R_FIG (hosting figures : monitoring and atlas, and/or small files)
[1063]	187	R_FIG=${STORAGE}/IGCM_OUT
[1057]	188
	189	#====================================================
	190	#- R_BUF (ONLY FOR double copy an scratch)
[1063]	191	R_BUF=${STORAGE}/IGCM_OUT
[1057]	192
	193	IGCM_debug_Print 1 "R_OUT has been defined = ${R_OUT}"
	194	IGCM_debug_Print 1 "R_BUF has been defined = ${R_BUF}"
	195	IGCM_debug_Print 1 "R_FIG has been defined = ${R_FIG}"
	196
	197	IGCM_debug_PopStack "IGCM_sys_defineArchives"
[725]	198	}
	199
	200	#D-#==================================================
[2]	201	#D-function IGCM_sys_RshMaster
[800]	202	#D-* Purpose: Connection to frontend machine.
[2]	203	#D-* Examples:
	204	#D-
	205	function IGCM_sys_RshMaster {
[664]	206	IGCM_debug_PushStack "IGCM_sys_RshMaster" $@
[800]	207	/bin/ksh <<-EOF
[2]	208	export libIGCM=${libIGCM}
[98]	209	export DEBUG_debug=${DEBUG_debug}
[2]	210	. ${libIGCM}/libIGCM_debug/libIGCM_debug.ksh
	211	. ${libIGCM}/libIGCM_card/libIGCM_card.ksh
	212	${@}
	213	EOF
[664]	214	if [ $? -gt 0 ] ; then
	215	echo "IGCM_sys_RshMaster : erreur."
	216	IGCM_debug_Exit "IGCM_sys_RshMaster"
	217	fi
	218	IGCM_debug_PopStack "IGCM_sys_RshMaster"
[2]	219	}
	220
	221	#D-#==================================================
[37]	222	#D-function IGCM_sys_RshArchive
	223	#D-* Purpose: Archive rsh command
	224	#D-* Examples:
	225	#D-
	226	function IGCM_sys_RshArchive {
[664]	227	IGCM_debug_PushStack "IGCM_sys_RshArchive" $@
[800]	228	/bin/ksh <<-EOF
[40]	229	${@}
	230	EOF
[811]	231	status=$?
	232	if [ ${status} -gt 0 ] ; then
	233	IGCM_debug_Print 2 "IGCM_sys_RshArchive : command failed error code ${status}"
[664]	234	IGCM_debug_Exit "IGCM_sys_RshArchive"
	235	fi
	236	IGCM_debug_PopStack "IGCM_sys_RshArchive"
[37]	237	}
	238
	239	#D-#==================================================
[2]	240	#D-function IGCM_sys_RshPost
[117]	241	#D-* Purpose: Post-process rsh command
[2]	242	#D-* Examples:
	243	#D-
	244	function IGCM_sys_RshPost {
[664]	245	IGCM_debug_PushStack "IGCM_sys_RshPost" $@
	246	if ( $DEBUG_sys ) ; then
	247	echo "IGCM_sys_RshPost :" $@
	248	fi
[800]	249
	250	# keep standard input (stdin) for the loop onto temporary file
	251	cat >tmp_IGCM_sys_RshPost_$$
	252
	253	/bin/ksh <tmp_IGCM_sys_RshPost_$$
[664]	254	if [ $? -gt 0 ] ; then
	255	echo "IGCM_sys_RshPost : erreur."
	256	IGCM_debug_Exit "IGCM_sys_RshPost"
	257	fi
[800]	258	# delete temporary file
	259	\rm tmp_IGCM_sys_RshPost_$$
[664]	260	IGCM_debug_PopStack "IGCM_sys_RshPost"
[2]	261	}
	262
	263	#D-#==================================================
[153]	264	#D-function IGCM_sys_SendMail
	265	#D-* Purpose: Send mail when simulation is over
	266	#D-* Examples:
	267	#D-
	268	function IGCM_sys_SendMail {
[664]	269	IGCM_debug_PushStack "IGCM_sys_SendMail" $@
	270	if ( $DEBUG_sys ) ; then
	271	echo "IGCM_sys_SendMail :" $@
	272	fi
[153]	273
[922]	274	if [ X${1} = XAccounting ] ; then
	275	status=Accounting
	276	mailText=jobAccounting.mail
	277	elif ( ${ExitFlag} ) ; then
[664]	278	status=failed
[922]	279	mailText=jobEnd.mail
[664]	280	else
	281	status=completed
[922]	282	mailText=jobEnd.mail
[664]	283	fi
[800]	284
[922]	285	# Update selected mail template
	286	while read -r line; do
	287	eval echo $line >> mail.txt ;
	288	done < ${libIGCM}/libIGCM_sys/${mailText}
[437]	289
[664]	290	if [ ! -z ${config_UserChoices_MailName} ] ; then
[922]	291	mailx -s "${config_UserChoices_JobName} ${status}" ${config_UserChoices_MailName} < mail.txt
[664]	292	elif [ -f ~/.forward ] ; then
[922]	293	mailx -s "${config_UserChoices_JobName} ${status}" $( cat ~/.forward ) < mail.txt
[664]	294	fi
[153]	295
[811]	296	sleep 10
[922]	297	rm -f mail.txt
[811]	298
[664]	299	IGCM_debug_PopStack "IGCM_sys_SendMail"
[153]	300	}
	301
	302	#D-#==================================================
[2]	303	#D-function IGCM_sys_Mkdir
	304	#D-* Purpose: Master locale mkdir command
	305	#D-* Examples:
	306	#D-
	307	function IGCM_sys_Mkdir {
[664]	308	IGCM_debug_PushStack "IGCM_sys_Mkdir" $@
	309	if ( $DEBUG_sys ) ; then
	310	echo "IGCM_sys_Mkdir :" $@
	311	fi
	312	if [ ! -d ${1} ]; then
	313	\mkdir -p $1
	314	if [ $? -gt 0 ] ; then
	315	echo "IGCM_sys_Mkdir : erreur."
	316	IGCM_debug_Exit "IGCM_sys_Mkdir"
[2]	317	fi
[664]	318	fi
[800]	319	# vérification :
[664]	320	if [ ! -d ${1} ] ; then
	321	echo "IGCM_sys_Mkdir : erreur."
	322	IGCM_debug_Exit "IGCM_sys_Mkdir"
	323	fi
	324	IGCM_debug_PopStack "IGCM_sys_Mkdir"
[2]	325	}
	326
	327	#D-#==================================================
	328	#D-function IGCM_sys_MkdirArchive
	329	#D-* Purpose: Mkdir on Archive
	330	#D-* Examples:
	331	#D-
	332	function IGCM_sys_MkdirArchive {
[664]	333	IGCM_debug_PushStack "IGCM_sys_MkdirArchive" $@
	334	if ( $DEBUG_sys ) ; then
	335	echo "IGCM_sys_MkdirArchive :" $@
	336	fi
[800]	337	#- creation de repertoire sur le serveur fichier
[811]	338	mkdir -p $1
	339	status=$?
	340
	341	if [ ${status} -gt 0 ] ; then
	342	IGCM_debug_Print 2 "IGCM_sys_MkdirArchive : mkdir failed error code ${status}"
	343	IGCM_debug_Exit "IGCM_sys_MkdirArchive"
[664]	344	fi
	345	IGCM_debug_PopStack "IGCM_sys_MkdirArchive"
[2]	346	}
	347
	348	#D-#==================================================
	349	#D-function IGCM_sys_MkdirWork
	350	#D-* Purpose: Mkdir on Work
	351	#D-* Examples:
	352	#D-
	353	function IGCM_sys_MkdirWork {
[664]	354	IGCM_debug_PushStack "IGCM_sys_MkdirWork" $@
	355	if ( $DEBUG_sys ) ; then
	356	echo "IGCM_sys_MkdirWork :" $@
	357	fi
[800]	358	#- creation de repertoire sur le serveur fichier
[664]	359	if [ ! -d ${1} ]; then
	360	\mkdir -p $1
	361	if [ $? -gt 0 ] ; then
	362	echo "IGCM_sys_MkdirWork : erreur."
	363	IGCM_debug_Exit "IGCM_sys_MkdirWork"
[2]	364	fi
[664]	365	fi
	366	IGCM_debug_PopStack "IGCM_sys_MkdirWork"
[2]	367	}
	368
	369	#D-#==================================================
	370	#D-function IGCM_sys_Cd
	371	#D-* Purpose: master cd command
	372	#D-* Examples:
	373	#D-
	374	function IGCM_sys_Cd {
[664]	375	IGCM_debug_PushStack "IGCM_sys_Cd" $@
	376	if ( $DEBUG_sys ) ; then
	377	echo "IGCM_sys_Cd :" $@
	378	fi
	379	\cd $1
	380	if [ $? -gt 0 ] ; then
	381	echo "IGCM_sys_Cd : erreur."
	382	IGCM_debug_Exit "IGCM_sys_Cd"
	383	fi
	384	IGCM_debug_PopStack "IGCM_sys_Cd"
[2]	385	}
	386
	387	#D-#==================================================
	388	#D-function IGCM_sys_Chmod
	389	#D-* Purpose: Chmod
	390	#D-* Examples:
	391	#D-
	392	function IGCM_sys_Chmod {
[1057]	393	IGCM_debug_PushStack "IGCM_sys_Chmod" $@
[664]	394	if ( $DEBUG_sys ) ; then
	395	echo "IGCM_sys_Chmod :" $@
	396	fi
[809]	397	\chmod $@
	398	if [ $? -gt 0 ] ; then
	399	echo "IGCM_sys_Chmod : erreur."
	400	IGCM_debug_Exit "IGCM_sys_Chmod"
[664]	401	fi
	402	IGCM_debug_PopStack "IGCM_sys_Chmod"
[2]	403	}
	404
	405	#D-#==================================================
	406	#D-function IGCM_sys_FileSize
	407	#D-* Purpose: Filesize
	408	#D-* Examples:
	409	#D-
	410	function IGCM_sys_FileSize {
[664]	411	IGCM_debug_PushStack "IGCM_sys_FileSize" $@
[2]	412
[664]	413	typeset sizeF
	414	set +A sizeF -- $( ls -la ${1} )
	415	if [ $? -gt 0 ] ; then
	416	IGCM_debug_Exit "IGCM_sys_FileSize"
	417	fi
	418	eval ${2}=${sizeF[4]}
[2]	419
[664]	420	IGCM_debug_PopStack "IGCM_sys_FileSize"
[2]	421	}
	422
	423	#D-#==================================================
	424	#D-function IGCM_sys_TestDir
	425	#D-* Purpose: Test Directory that must exists
	426	#D-* Examples:
	427	#D-
	428	function IGCM_sys_TestDir {
[664]	429	IGCM_debug_PushStack "IGCM_sys_TestDir" $@
	430	if ( $DEBUG_sys ) ; then
	431	echo "IGCM_sys_TestDir :" $@
	432	fi
	433	typeset ExistFlag
	434	ExistFlag=$( [ -d $1 ] && echo 0 \|\| echo 1 )
	435	IGCM_debug_PopStack "IGCM_sys_TestDir"
[67]	436
[664]	437	return ${ExistFlag}
[2]	438	}
	439
	440	#D-#==================================================
	441	#D-function IGCM_sys_TestDirArchive
	442	#D-* Purpose: Test Directory that must exists on Archive
	443	#D-* Examples:
	444	#D-
	445	function IGCM_sys_TestDirArchive {
[664]	446	IGCM_debug_PushStack "IGCM_sys_TestDirArchive" $@
	447	if ( $DEBUG_sys ) ; then
	448	echo "IGCM_sys_TestDirArchive :" $@
	449	fi
	450	typeset ExistFlag
	451	ExistFlag=$( [ -d $1 ] && echo 0 \|\| echo 1 )
	452	IGCM_debug_PopStack "IGCM_sys_TestDirArchive"
	453	return ${ExistFlag}
[2]	454	}
	455
	456	#D-#==================================================
[800]	457	#D-function IGCM_sys_IsFileArchived
	458	#D-* Purpose: Test file that must NOT EXISTS on Archive
	459	#D-* Examples:
	460	#D-
	461	function IGCM_sys_IsFileArchived {
	462	IGCM_debug_PushStack "IGCM_sys_IsFileArchived" $@
	463	if ( $DEBUG_sys ) ; then
	464	echo "IGCM_sys_IsFileArchived :" $@
	465	fi
	466	typeset IsArchivedFlag
	467	# Never archived for this system
	468	IsArchivedFlag=1
	469	IGCM_debug_PopStack "IGCM_sys_IsFileArchived"
	470
	471	return ${IsArchivedFlag}
	472	}
	473
	474	#D-#==================================================
[2]	475	#D-function IGCM_sys_TestFileArchive
	476	#D-* Purpose: Test file that must NOT EXISTS on Archive
	477	#D-* Examples:
	478	#D-
	479	function IGCM_sys_TestFileArchive {
[664]	480	IGCM_debug_PushStack "IGCM_sys_TestFileArchive" $@
	481	typeset ExistFlag
	482	ExistFlag=$( [ -f $1 ] && echo 0 \|\| echo 1 )
	483	IGCM_debug_PopStack "IGCM_sys_TestFileArchive"
[67]	484
[664]	485	return ${ExistFlag}
[67]	486	}
	487
	488	#D-#==================================================
[664]	489	#D-function IGCM_sys_TestFileBuffer
	490	#D-* Purpose: Test file that must NOT EXISTS on Buffer
	491	#D-* Examples:
	492	#D-
	493	function IGCM_sys_TestFileBuffer {
	494	IGCM_debug_PushStack "IGCM_sys_TestFileBuffer" $@
	495	typeset ExistFlag
	496	ExistFlag=$( [ -f $1 ] && echo 0 \|\| echo 1 )
	497	IGCM_debug_PopStack "IGCM_sys_TestFileBuffer"
	498
	499	return ${ExistFlag}
	500	}
	501
	502	#D-#==================================================
[67]	503	#D-function IGCM_sys_CountFileArchive
	504	#D-* Purpose: Count files on Archive filesystem
	505	#D-* Examples:
	506	#D-
	507	function IGCM_sys_CountFileArchive {
[664]	508	IGCM_debug_PushStack "IGCM_sys_CountFileArchive" $@
	509	ls ${@} 2>/dev/null \| wc -l
	510	if [ $? -gt 0 ] ; then
	511	echo "IGCM_sys_CountFileArchive : erreur."
	512	fi
	513	IGCM_debug_PopStack "IGCM_sys_CountFileArchive"
[2]	514	}
	515
	516	#D-#==================================================
[664]	517	#D-function IGCM_sys_CountFileBuffer
	518	#D-* Purpose: Count files on Scratch filesystem
	519	#D-* Examples:
	520	#D-
	521	function IGCM_sys_CountFileBuffer {
	522	IGCM_debug_PushStack "IGCM_sys_CountFileBuffer" $@
	523	ls ${@} 2>/dev/null \| wc -l
	524	if [ $? -gt 0 ] ; then
	525	echo "IGCM_sys_CountFileBuffer : erreur."
	526	fi
	527	IGCM_debug_PopStack "IGCM_sys_CountFileBuffer"
	528	}
	529
	530	#D-#==================================================
[2]	531	#D-function IGCM_sys_Tree
	532	#D-* Purpose: Tree directories with files on ${ARCHIVE}
	533	#D-* Examples: IGCM_sys_Tree ${R_IN} ${R_OUT}
	534	#D-
	535	function IGCM_sys_Tree {
[664]	536	IGCM_debug_PushStack "IGCM_sys_Tree" $@
	537	if ( $DEBUG_sys ) ; then
	538	echo "IGCM_sys_Tree :" $@
	539	fi
[2]	540
[664]	541	\tree -f $@
[2]	542
[664]	543	IGCM_debug_PopStack "IGCM_sys_Tree"
[2]	544	}
	545
	546	#D-#==================================================
	547	#D-function IGCM_sys_Tar
[664]	548	#D-* Purpose: master tar command
[2]	549	#D-* Examples:
	550	#D-
	551	function IGCM_sys_Tar {
[664]	552	IGCM_debug_PushStack "IGCM_sys_Tar" $@
	553	if ( $DEBUG_sys ) ; then
	554	echo "IGCM_sys_Tar :" $@
	555	fi
	556	\tar cf $@
	557	if [ $? -gt 0 ] ; then
	558	echo "IGCM_sys_Tar : erreur."
	559	IGCM_debug_Exit "IGCM_sys_Tar"
	560	fi
	561	IGCM_debug_PopStack "IGCM_sys_Tar"
[2]	562	}
	563
	564	#D-#==================================================
	565	#D-function IGCM_sys_UnTar
	566	#D-* Purpose: master un-tar command
	567	#D-* Examples:
	568	#D-
	569	function IGCM_sys_UnTar {
[664]	570	IGCM_debug_PushStack "IGCM_sys_UnTar" $@
	571	if ( $DEBUG_sys ) ; then
	572	echo "IGCM_sys_UnTar :" $@
	573	fi
	574	\tar xvf $1
	575	if [ $? -gt 0 ] ; then
	576	echo "IGCM_sys_UnTar : erreur."
	577	IGCM_debug_Exit "IGCM_sys_UnTar"
	578	fi
	579	IGCM_debug_PopStack "IGCM_sys_UnTar"
[2]	580	}
	581
	582	#D-#==================================================
	583	#D-function IGCM_sys_Qsub
	584	#D-* Purpose: Qsub new job
	585	#D-* Examples:
	586	#D-
	587	function IGCM_sys_Qsub {
[664]	588	IGCM_debug_PushStack "IGCM_sys_Qsub" $@
	589	if ( $DEBUG_sys ) ; then
	590	echo "IGCM_sys_Qsub :" $@
	591	fi
[800]	592	echo "${@} > ${SUBMIT_DIR}/${Script_Output} 2>&1 &"
	593	${@} > ${SUBMIT_DIR}/${Script_Output} 2>&1 &
[664]	594	if [ $? -gt 0 ] ; then
[800]	595	echo "IGCM_sys_Qsub : erreur ${options} $1"
[664]	596	IGCM_debug_Exit "IGCM_sys_Qsub"
	597	fi
	598	IGCM_debug_PopStack "IGCM_sys_Qsub"
[2]	599	}
	600
	601	#D-#==================================================
	602	#D-function IGCM_sys_QsubPost
	603	#D-* Purpose: Qsub new job on scalaire
	604	#D-* Examples:
	605	#D-
	606	function IGCM_sys_QsubPost {
[664]	607	IGCM_debug_PushStack "IGCM_sys_QsubPost" $@
	608	if ( $DEBUG_sys ) ; then
	609	echo "IGCM_sys_QsubPost :" $@
	610	fi
	611	echo "${libIGCM_POST}/$1.job > ${POST_DIR}/$1.${PeriodDateEnd}.out 2>&1 &"
	612	${libIGCM_POST}/$1.job > ${POST_DIR}/${Script_Post_Output}.out 2>&1 &
	613	if [ $? -gt 0 ] ; then
	614	echo "IGCM_sys_QsubPost : erreur " $@
	615	IGCM_debug_Exit "IGCM_sys_QsubPost"
	616	fi
	617	IGCM_debug_PopStack "IGCM_sys_QsubPost"
[2]	618	}
	619
	620	#D-*************************
[664]	621	#D- File transfer functions
[2]	622	#D-*************************
	623	#D-
	624
	625	#D-#==================================================
	626	#D-function IGCM_sys_Rsync_out
	627	#D-* Purpose: treat return val of rsync
[800]	628	#D-* Examples: IGCM_sys_Rsync_out out_RET_rsync
[2]	629	#D- Error values and explanations can depend on your system version.
	630	function IGCM_sys_Rsync_out {
[800]	631	status=$1
	632	if [ ! $status ] ; then
[664]	633	echo "rsync error !"
	634	fi
[2]	635
[664]	636	if [ $MYLANG = "fr" ]; then
[800]	637	case $status in
[664]	638	0) return ;;
	639	1) echo "Erreur de rsync ; RERR_SYNTAX : "
	640	echo "Erreur de syntaxe ou d'utilisation."
	641	return;;
	642	2) echo "Erreur de rsync ; RERR_PROTOCOL : "
	643	echo "Incompatibilité de protocole."
	644	return;;
	645	3) echo "Erreur de rsync ; RERR_FILESELECT 3"
	646	echo "Erreurs lors de la sélection des fichiers d'entrée sortie et"
	647	echo "répertoires"
	648	return;;
	649	4) echo "Erreur de rsync ; RERR_UNSUPPORTED"
	650	echo "Action demandée non supportée : une tentative de manipulation de"
	651	echo "fichiers 64-bits sur une plate-forme qui ne les supporte pas a"
	652	echo "été faite ; ou une option qui est supportée par le client mais"
	653	echo "pas par le serveur a été spécifiée."
	654	return;;
	655	10) echo "Erreur de rsync ; RERR_SOCKETIO"
	656	echo "Erreur dans le socket d'entrée sortie"
	657	return;;
	658	11) echo "Erreur de rsync ; RERR_FILEIO"
	659	echo "Erreur d'entrée sortie fichier"
	660	return;;
	661	12) echo "Erreur de rsync ; RERR_STREAMIO"
	662	echo "Erreur dans flux de donnée du protocole rsync"
	663	return;;
	664	13) echo "Erreur de rsync ; RERR_MESSAGEIO"
	665	echo "Erreur avec les diagnostics du programme"
	666	return;;
	667	14) echo "Erreur de rsync ; RERR_IPC"
	668	echo "Erreur dans le code IPC"
	669	return;;
	670	20) echo "Erreur de rsync ; RERR_SIGNAL"
	671	echo "SIGUSR1 ou SIGINT reçu"
	672	return;;
	673	21) echo "Erreur de rsync ; RERR_WAITCHILD"
	674	echo "Une erreur retournée par waitpid()"
	675	return;;
	676	22) echo "Erreur de rsync ; RERR_MALLOC"
	677	echo "Erreur lors de l'allocation des tampons de mémoire de coeur"
	678	return;;
	679	23) echo ""
	680	echo "Erreur fichier inexistant"
	681	return;;
	682	30) echo "Erreur de rsync ; RERR_TIMEOUT"
	683	echo "Temps d'attente écoulé dans l'envoi/réception de données"
	684	return;;
[800]	685	*) echo "Erreur de rsync : code de retour de rsync inconnu :" $status
[664]	686	return;;
	687	esac
	688	elif [ $MYLANG = "en" ] ; then
[800]	689	case $status in
[664]	690	0) return;;
	691	1) echo "rsync error : Syntax or usage error "
	692	return;;
	693	2) echo "rsync error : Protocol incompatibility "
	694	return;;
	695	3) echo "rsync error : Errors selecting input/output files, dirs"
	696	return;;
	697	4) echo "rsync error : Requested action not supported: an attempt"
	698	echo "was made to manipulate 64-bit files on a platform that cannot support"
	699	echo "them; or an option was specified that is supported by the client and"
	700	echo "not by the server."
	701	return;;
	702	5) echo "rsync error : Error starting client-server protocol"
	703	return;;
	704	10) echo "rsync error : Error in socket I/O "
	705	return;;
	706	11) echo "rsync error : Error in file I/O "
	707	return;;
	708	12) echo "rsync error : Error in rsync protocol data stream "
	709	return;;
	710	13) echo "rsync error : Errors with program diagnostics "
	711	return;;
	712	14) echo "rsync error : Error in IPC code "
	713	return;;
	714	20) echo "rsync error : Received SIGUSR1 or SIGINT "
	715	return;;
	716	21) echo "rsync error : Some error returned by waitpid() "
	717	return;;
	718	22) echo "rsync error : Error allocating core memory buffers "
	719	return;;
	720	23) echo "rsync error : Partial transfer due to error"
	721	return;;
	722	24) echo "rsync error : Partial transfer due to vanished source files"
	723	return;;
	724	30) echo "rsync error : Timeout in data send/receive "
	725	return;;
[800]	726	*) echo "rsync error : return code of rsync unknown :" $status
[664]	727	return;;
	728	esac
	729	else
	730	echo "unknown language $MYLANG."
	731	return
	732	fi
[2]	733	}
[664]	734
[2]	735	#D-#==================================================
[811]	736	#D-function IGCM_sys_Miror_libIGCM
	737	#D-* Purpose: Mirror libIGCM PATH and lib to frontend
	738	#D-* Examples:
	739	#D-
	740	function IGCM_sys_Mirror_libIGCM {
	741	IGCM_debug_PushStack "IGCM_sys_Mirror_libIGCM"
	742	if ( $DEBUG_sys ) ; then
	743	echo "IGCM_sys_Mirror_libIGCM"
	744	fi
	745
	746	typeset status
	747
	748	mkdir -p ${HOME}/MIRROR/${PATHlibIGCM}
	749
[938]	750	echo ${RSYNC} ${RSYNC_opt} ${libIGCM} ${HOME}/MIRROR/${PATHlibIGCM} > /tmp/out_command.$$ 2>&1
	751	${RSYNC} ${RSYNC_opt} ${libIGCM} ${HOME}/MIRROR/${PATHlibIGCM} >> /tmp/out_command.$$ 2>&1
[811]	752	status=$?
	753
	754	if [ ${status} -gt 0 ] ; then
	755	echo "IGCM_sys_Mirror_libIGCM Warning : no libIGCM on frontend."
[938]	756	cat /tmp/out_command.$$
[811]	757	fi
	758	IGCM_debug_PopStack "IGCM_sys_Mirror_libIGCM"
	759	}
	760
	761	#D-#==================================================
[2]	762	#D-function IGCM_sys_Cp
	763	#D-* Purpose: generic cp
	764	#D-* Examples:
	765	#D-
	766	function IGCM_sys_Cp {
[664]	767	IGCM_debug_PushStack "IGCM_sys_Cp" $@
	768	if ( $DEBUG_sys ) ; then
	769	echo "IGCM_sys_Cp :" $@
	770	fi
[2]	771
[717]	772	typeset status
[2]	773
[938]	774	echo cp $@ > /tmp/out_command.$$ 2>&1
	775	\cp $@ >> /tmp/out_command.$$ 2>&1
[717]	776	status=$?
[664]	777
[717]	778	if [ ${status} -gt 0 ] ; then
	779	echo "IGCM_sys_Cp : error code ${status}"
[938]	780	cat /tmp/out_command.$$
[717]	781	IGCM_debug_Exit "IGCM_sys_Cp"
[664]	782	else
[938]	783	\rm /tmp/out_command.$$
[664]	784	fi
	785	IGCM_debug_PopStack "IGCM_sys_Cp"
[2]	786	}
	787
	788	#D-#==================================================
	789	#D-function IGCM_sys_Rm
	790	#D-* Purpose: generic rm
	791	#D-* Examples:
	792	#D-
	793	function IGCM_sys_Rm {
[1057]	794	IGCM_debug_PushStack "IGCM_sys_Rm" $@
[664]	795	if ( $DEBUG_sys ) ; then
	796	echo "IGCM_sys_Rm :" $@
	797	fi
[2]	798
[717]	799	typeset status
[2]	800
[938]	801	echo rm $@ > /tmp/out_command.$$ 2>&1
	802	\rm $@ >> /tmp/out_command.$$ 2>&1
[717]	803	status=$?
[664]	804
[717]	805	if [ ${status} -gt 0 ] ; then
	806	echo "IGCM_sys_Rm : error code ${status}"
[938]	807	cat /tmp/out_command.$$
[664]	808	IGCM_debug_Exit "IGCM_sys_Rm"
	809	else
[938]	810	\rm /tmp/out_command.$$
[664]	811	fi
	812	IGCM_debug_PopStack "IGCM_sys_Rm"
[2]	813	}
	814
	815	#D-#==================================================
[203]	816	#D-function IGCM_sys_RmRunDir
	817	#D-* Purpose: rm tmpdir (dummy function most of the time batch
	818	#D- scheduler will do the job)
	819	#D-* Examples:
	820	#D-
	821	function IGCM_sys_RmRunDir {
[1057]	822	IGCM_debug_PushStack "IGCM_sys_RmRunDir" $@
[664]	823	if ( $DEBUG_sys ) ; then
	824	echo "IGCM_sys_RmRunDir :" $@
[800]	825	echo "Dummy call, let the user do that."
[785]	826	# IGCM_sys_Rm -rf ${RUN_DIR}
[664]	827	fi
	828	IGCM_debug_PopStack "IGCM_sys_RmRunDir"
[203]	829	}
	830
	831	#D-#==================================================
[2]	832	#D-function IGCM_sys_Mv
	833	#D-* Purpose: generic move
	834	#D-* Examples:
	835	#D-
	836	function IGCM_sys_Mv {
[664]	837	IGCM_debug_PushStack "IGCM_sys_Mv" $@
	838	if ( $DEBUG_sys ) ; then
	839	echo "IGCM_sys_Mv :" $@
	840	fi
[2]	841
[664]	842	if [ $DRYRUN = 0 ]; then
[2]	843
[717]	844	typeset status
[664]	845
[938]	846	echo mv $@ > /tmp/out_command.$$ 2>&1
	847	\mv $@ >> /tmp/out_command.$$ 2>&1
[717]	848	status=$?
[664]	849
[717]	850	if [ ${status} -gt 0 ] ; then
	851	echo "IGCM_sys_Mv : error code ${status}"
[938]	852	cat /tmp/out_command.$$
[664]	853	IGCM_debug_Exit "IGCM_sys_Mv"
[173]	854	else
[938]	855	\rm /tmp/out_command.$$
[2]	856	fi
[664]	857	fi
[2]	858
[664]	859	IGCM_debug_PopStack "IGCM_sys_Mv"
[2]	860	}
	861
	862	#D-#==================================================
	863	#D-function IGCM_sys_Put_Dir
	864	#D-* Purpose: Copy a complete directory on $(ARCHIVE)
	865	#D-* Examples:
	866	#D-
	867	function IGCM_sys_Put_Dir {
[664]	868	IGCM_debug_PushStack "IGCM_sys_Put_Dir" $@
	869	if ( $DEBUG_sys ) ; then
	870	echo "IGCM_sys_Put_Dir :" $@
	871	fi
	872	if [ $DRYRUN = 0 ]; then
	873	if [ ! -d ${1} ] ; then
	874	echo "WARNING : IGCM_sys_Put_Dir ${1} DOES NOT EXIST ."
	875	IGCM_debug_PopStack "IGCM_sys_Put_Dir"
	876	return
[2]	877	fi
	878
[717]	879	typeset status
[2]	880
[717]	881	# Only if we use rsync
	882	#IGCM_sys_TestDirArchive $( dirname $2 )
	883	#
	884	#USUAL WAY
[938]	885	\cp -r $1 $2 > /tmp/out_command.$$ 2>&1
[717]	886	status=$?
[2]	887
[717]	888	if [ ${status} -gt 0 ] ; then
[811]	889	IGCM_debug_Print 2 "IGCM_sys_Put_Dir : cp failed error code ${status}"
[938]	890	cat /tmp/out_command.$$
[664]	891	IGCM_debug_Exit "IGCM_sys_Put_Dir"
[2]	892	else
[938]	893	\rm /tmp/out_command.$$
[2]	894	fi
[664]	895	fi
	896	IGCM_debug_PopStack "IGCM_sys_Put_Dir"
[2]	897	}
	898
	899	#D-#==================================================
	900	#D-function IGCM_sys_Get_Dir
[717]	901	#D-* Purpose: Copy a complete directory from ${ARCHIVE}
[2]	902	#D-* Examples:
	903	#D-
	904	function IGCM_sys_Get_Dir {
[664]	905	IGCM_debug_PushStack "IGCM_sys_Get_Dir" $@
	906	if ( $DEBUG_sys ) ; then
	907	echo "IGCM_sys_Get_Dir :" $@
	908	fi
	909	if [ $DRYRUN = 0 ]; then
	910	if [ ! -d ${1} ] ; then
	911	echo "WARNING : IGCM_sys_Get_Dir ${1} DOES NOT EXIST ."
	912	IGCM_debug_PopStack "IGCM_sys_Get_Dir"
	913	return
[2]	914	fi
	915
[811]	916	typeset status
[2]	917
[800]	918	#USUAL WAY
[938]	919	\cp -ur $1 $2 > /tmp/out_command.$$ 2>&1
[811]	920	status=$?
[2]	921
[811]	922	if [ ${status} -gt 0 ] ; then
	923	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : cp failed error code ${status}"
[938]	924	cat /tmp/out_command.$$
[664]	925	IGCM_debug_Exit "IGCM_sys_Get_Dir"
[2]	926	else
[938]	927	\rm /tmp/out_command.$$
[2]	928	fi
[664]	929	fi
	930	IGCM_debug_PopStack "IGCM_sys_Get_Dir"
[2]	931	}
	932
	933	#D-#==================================================
	934	#D-function IGCM_sys_Get_Master
	935	#D-* Purpose: Copy a complete directory from MASTER filesystem
	936	#D-* Examples:
	937	#D-
	938	function IGCM_sys_Get_Master {
[664]	939	IGCM_debug_PushStack "IGCM_sys_Get_Master" $@
	940	if ( $DEBUG_sys ) ; then
	941	echo "IGCM_sys_Get_Master :" $@
	942	fi
	943	if [ $DRYRUN = 0 ]; then
	944	if ( [ ! -d ${1} ] && [ ! -f ${1} ] ) ; then
	945	echo "WARNING : IGCM_sys_Get_Master ${1} DOES NOT EXIST ."
	946	IGCM_debug_PopStack "IGCM_sys_Get_Master"
	947	return
[2]	948	fi
	949
[811]	950	typeset NB_ESSAI DELAI status i
	951	# number of tentative
	952	NB_ESSAI=3
	953	# time delay between tentative
	954	DELAI=2
[2]	955
[811]	956	i=0
	957	while [ $i -lt $NB_ESSAI ] ; do
[938]	958	\cp -urL $1 $2 > /tmp/out_command.$$ 2>&1
[811]	959	status=$?
	960	if [ ${status} -gt 0 ]; then
	961	IGCM_debug_Print 2 "IGCM_sys_Get_Master : cp failed error code ${status} ${i}/${NB_ESSAI}"
	962	IGCM_debug_Print 2 "IGCM_sys_Get_Master : sleep ${DELAI} seconds and try again."
	963	sleep $DELAI
	964	else
	965	break
	966	fi
	967	(( i = i + 1 ))
	968	done
[2]	969
[811]	970	if [ ${status} -gt 0 ] ; then
[664]	971	echo "IGCM_sys_Get_Master : error."
[938]	972	cat /tmp/out_command.$$
[664]	973	IGCM_debug_Exit "IGCM_sys_Get_Master"
[2]	974	else
[938]	975	\rm /tmp/out_command.$$
[2]	976	fi
[664]	977	fi
	978	IGCM_debug_PopStack "IGCM_sys_Get_Master"
[2]	979	}
	980
[811]	981	#====================================================
	982	#- Call IGCM_sys_Mirror_libIGCM now !
	983	if ( $MirrorlibIGCM ) ; then
	984	IGCM_sys_Mirror_libIGCM
	985	fi
	986
[2]	987	#D-#==================================================
	988	#D-function IGCM_sys_Put_Rest
[717]	989	#D-* Purpose: Put computied restarts on ${ARCHIVE}.
[117]	990	#D- File and target directory must exist.
[2]	991	#D-* Examples:
	992	#D-
	993	function IGCM_sys_Put_Rest {
[664]	994	IGCM_debug_PushStack "IGCM_sys_Put_Rest" $@
	995	if ( $DEBUG_sys ) ; then
	996	echo "IGCM_sys_Put_Rest :" $@
	997	fi
	998	if [ $DRYRUN = 0 ]; then
	999	if [ ! -f ${1} ] ; then
	1000	echo "ERROR : IGCM_sys_Put_Rest ${1} DOES NOT EXIST ."
	1001	IGCM_debug_Exit "IGCM_sys_Put_Rest"
[2]	1002	fi
[717]	1003
	1004	typeset status
	1005	#
[664]	1006	if [ X${JobType} = XRUN ] ; then
	1007	IGCM_sys_Chmod 444 ${1}
	1008	fi
[717]	1009	#
[811]	1010	# Only if we use rsync
	1011	#IGCM_sys_MkdirArchive $( dirname $2 )
	1012	#
	1013	#USUAL WAY
[938]	1014	\cp $1 $2 > /tmp/out_command.$$ 2>&1
[717]	1015	status=$?
[2]	1016
[717]	1017	# #RSYNC WITH NETWORK SSH CALL
[938]	1018	# echo ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} > /tmp/out_command.$$ 2>&1
	1019	# ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} >> /tmp/out_command.$$ 2>&1
[2]	1020
[717]	1021	# #RSYNC WITH NFS USE
[938]	1022	# echo ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} > /tmp/out_command.$$ 2>&1
	1023	# ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} >> /tmp/out_command.$$ 2>&1
[2]	1024
[717]	1025	# status=$?
	1026	# IGCM_sys_Rsync_out $status
	1027
[938]	1028	# ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk /tmp/out_command.$$
[717]	1029	# (( status=status+$? ))
	1030
	1031	if [ ${status} -gt 0 ] ; then
	1032	echo "IGCM_sys_Put_Rest : cp failed error code ${status}"
[938]	1033	cat /tmp/out_command.$$
[664]	1034	IGCM_debug_Exit "IGCM_sys_Put_Rest"
	1035	else
[938]	1036	\rm /tmp/out_command.$$
[664]	1037	fi
	1038	fi
	1039	IGCM_debug_PopStack "IGCM_sys_Put_Rest"
	1040	}
[2]	1041
[664]	1042	#D-#==================================================
	1043	#D-function IGCM_sys_PutBuffer_Rest
	1044	#D-* Purpose: Put computied restarts on ${SCRATCHDIR}.
	1045	#D- File and target directory must exist.
	1046	#D-* Examples:
	1047	#D-
	1048	function IGCM_sys_PutBuffer_Rest {
	1049	IGCM_debug_PushStack "IGCM_sys_PutBuffer_Rest" $@
	1050	if ( $DEBUG_sys ) ; then
	1051	echo "IGCM_sys_PutBuffer_Rest :" $@
	1052	fi
	1053	if [ $DRYRUN = 0 ]; then
	1054	if [ ! -f ${1} ] ; then
	1055	echo "ERROR : IGCM_sys_PutBuffer_Rest ${1} DOES NOT EXIST ."
	1056	IGCM_debug_Exit "IGCM_sys_PutBuffer_Rest"
	1057	fi
[2]	1058
[717]	1059	typeset status
[664]	1060	#
	1061	if [ X${JobType} = XRUN ] ; then
	1062	IGCM_sys_Chmod 444 ${1}
	1063	fi
	1064
	1065	#
	1066	# USUAL WAY
[938]	1067	\cp $1 $2 > /tmp/out_command.$$ 2>&1
[717]	1068	status=$?
[664]	1069
[717]	1070	if [ ${status} -gt 0 ] ; then
	1071	echo "IGCM_sys_PutBuffer_Rest : error code ${status}"
[811]	1072	[ -f ${2} ] && ls -l ${2}
	1073	[ -f ${2}/${1} ] && ls -l ${2}/${1}
[938]	1074	cat /tmp/out_command.$$
[664]	1075	IGCM_debug_Exit "IGCM_sys_PutBuffer_Rest"
[2]	1076	else
[938]	1077	\rm /tmp/out_command.$$
[2]	1078	fi
[664]	1079	fi
	1080	IGCM_debug_PopStack "IGCM_sys_PutBuffer_Rest"
[2]	1081	}
	1082
	1083	#D-#==================================================
[801]	1084	#D-function IGCM_sys_PrepareTaredRestart
	1085	#D-* Purpose: Prepare tared restart to be access by computing job. Identity here.
	1086	#D-* Examples:
	1087	#D-
	1088	function IGCM_sys_PrepareTaredRestart {
	1089	IGCM_debug_PushStack "IGCM_sys_PrepareTaredRestart" $@
[918]	1090
[801]	1091	IGCM_debug_PopStack "IGCM_sys_PrepareTaredRestart"
	1092	}
	1093
	1094	#D-#==================================================
[2]	1095	#D-function IGCM_sys_Put_Out
[717]	1096	#D-* Purpose: Copy a file on ${ARCHIVE} after having chmod it in readonly
[2]	1097	#D-* Examples:
	1098	#D-
	1099	function IGCM_sys_Put_Out {
[664]	1100	IGCM_debug_PushStack "IGCM_sys_Put_Out" $@
	1101	if ( $DEBUG_sys ) ; then
	1102	echo "IGCM_sys_Put_Out :" $@
	1103	fi
[811]	1104
	1105	typeset status
	1106
[664]	1107	if [ $DRYRUN = 0 ]; then
[811]	1108	if [ ! -f ${1} ] ; then
[664]	1109	echo "WARNING : IGCM_sys_Put_Out ${1} DOES NOT EXIST ."
	1110	IGCM_debug_PopStack "IGCM_sys_Put_Out"
	1111	return 1
[2]	1112	fi
[800]	1113	#
[811]	1114	IGCM_sys_MkdirArchive $( dirname $2 )
	1115	#
[664]	1116	if [ X${JobType} = XRUN ] ; then
	1117	if [ X${3} = X ] ; then
	1118	IGCM_sys_Chmod 444 ${1}
	1119	fi
	1120	fi
[800]	1121	#
[2]	1122
[938]	1123	echo ${RSYNC} ${RSYNC_opt} $1 $2 > /tmp/out_command.$$ 2>&1
	1124	${RSYNC} ${RSYNC_opt} $1 $2 >> /tmp/out_command.$$ 2>&1
[811]	1125	status=$?
	1126	IGCM_sys_Rsync_out $status
[2]	1127
[938]	1128	${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk /tmp/out_command.$$
[811]	1129	(( status=status+$? ))
[2]	1130
[811]	1131	if [ ${status} -gt 0 ] ; then
	1132	IGCM_debug_Print 2 "IGCM_sys_Put_Out : rsync failed error code ${status}"
[938]	1133	cat /tmp/out_command.$$
[664]	1134	IGCM_debug_Exit "IGCM_sys_Put_Out"
[2]	1135	else
[938]	1136	\rm /tmp/out_command.$$
[2]	1137	fi
[664]	1138	fi
	1139	IGCM_debug_PopStack "IGCM_sys_Put_Out"
	1140	return 0
[2]	1141	}
	1142
	1143	#D-#==================================================
[664]	1144	#D-function IGCM_sys_PutBuffer_Out
	1145	#D-* Purpose: Copy a file on ${SCRATCHDIR} after having chmod it in readonly
	1146	#D-* Examples:
	1147	#D-
	1148	function IGCM_sys_PutBuffer_Out {
	1149	IGCM_debug_PushStack "IGCM_sys_PutBuffer_Out" $@
	1150	if ( $DEBUG_sys ) ; then
	1151	echo "IGCM_sys_PutBuffer_Out :" $@
	1152	fi
[811]	1153
	1154	typeset NB_ESSAI DELAI status i exist skip
	1155
	1156	# number of tentative
	1157	NB_ESSAI=3
	1158	# time delay between tentative
	1159	DELAI=2
	1160
[664]	1161	if [ $DRYRUN = 0 ]; then
	1162	if [ ! -f ${1} ] ; then
	1163	echo "WARNING : IGCM_sys_PutBuffer_Out ${1} DOES NOT EXIST ."
	1164	IGCM_debug_PopStack "IGCM_sys_PutBuffer_Out"
	1165	return 1
	1166	fi
	1167	#
	1168	IGCM_sys_Mkdir $( dirname $2 )
	1169	#
	1170
[811]	1171	exist=false
	1172	skip=false
	1173	if [ -f $2 ] ; then
	1174	IGCM_debug_Print 1 "$2 already exist"
	1175	exist=true
	1176	if [ "X$( diff $1 $2 )" = X ] ; then
	1177	IGCM_debug_Print 2 "$1 and $2 are the same file, we skip the copy"
	1178	status=0
	1179	skip=true
	1180	else
	1181	IGCM_debug_Print 2 "$1 and $2 are not the same file, we force the copy"
	1182	skip=false
[664]	1183	fi
	1184	fi
	1185	#
[811]	1186	if ( [ X${exist} = Xtrue ] && [ X${skip} = Xfalse ] ) ; then
	1187	IGCM_sys_Chmod u+w $2
	1188	fi
[664]	1189
[811]	1190	if [ X${skip} = Xfalse ] ; then
	1191	i=0
	1192	while [ $i -lt $NB_ESSAI ] ; do
	1193	# USUAL WAY
[938]	1194	\cp $1 $2 > /tmp/out_command.$$ 2>&1
[811]	1195	status=$?
	1196	if [ ${status} -gt 0 ]; then
	1197	IGCM_debug_Print 2 "IGCM_sys_PutBuffer_Out : cp failed error code ${status} ${i}/${NB_ESSAI}"
	1198	IGCM_debug_Print 2 "IGCM_sys_PutBuffer_Out : sleep ${DELAI} seconds and try again."
	1199	[ -f ${2} ] && ls -l ${2}
	1200	[ -f ${2}/${1} ] && ls -l ${2}/${1}
	1201	sleep $DELAI
	1202	else
	1203	break
	1204	fi
	1205	(( i = i + 1 ))
	1206	done
	1207	fi
	1208
	1209	if [ ${status} -gt 0 ] ; then
[664]	1210	echo "IGCM_sys_PutBuffer_Out : error."
[811]	1211	[ -f ${2} ] && ls -l ${2}
	1212	[ -f ${2}/${1} ] && ls -l ${2}/${1}
[938]	1213	cat /tmp/out_command.$$
[664]	1214	IGCM_debug_Exit "IGCM_sys_PutBuffer_Out"
	1215	else
[811]	1216
	1217	if [ X${JobType} = XRUN ] ; then
	1218	if [ X${3} = X ] ; then
	1219	[ -f ${2} ] && IGCM_sys_Chmod 444 ${2}
	1220	[ -f ${2}/${1} ] && IGCM_sys_Chmod 444 ${2}/${1}
	1221	fi
	1222	fi
	1223
[938]	1224	\rm /tmp/out_command.$$
[664]	1225	fi
	1226	fi
	1227	IGCM_debug_PopStack "IGCM_sys_PutBuffer_Out"
	1228	return 0
	1229	}
	1230
	1231	#D-#==================================================
[2]	1232	#D-function IGCM_sys_Get
	1233	#D-* Purpose: Get a file from ${ARCHIVE}
	1234	#D-* Examples: IGCM_sys_Get myfile /destpath/myfile_with_PREFIX
	1235	#D- IGCM_sys_Get /l Array_contain_myfiles /destpath/
	1236	function IGCM_sys_Get {
[664]	1237	IGCM_debug_PushStack "IGCM_sys_Get" $@
[2]	1238
[811]	1239	typeset DEST status dm_liste
[2]	1240
[664]	1241	if ( $DEBUG_sys ) ; then
	1242	echo "IGCM_sys_Get :" $@
	1243	fi
	1244	if [ $DRYRUN -le 2 ]; then
	1245	if [ X${1} = X'/l' ] ; then
	1246	eval set +A dm_liste \${${2}}
	1247	else
	1248	dm_liste=${1}
[2]	1249	fi
[664]	1250	eval DEST=\${${#}}
[2]	1251
[800]	1252	# test if the (first) file is present in the old computation :
[664]	1253	IGCM_sys_TestFileArchive ${dm_liste[0]}
[811]	1254	status=$?
	1255	if [ ${status} -gt 0 ] ; then
[664]	1256	echo "IGCM_sys_Get, ERROR : regular file ${dm_liste[0]} DOES NOT EXIST ."
	1257	IGCM_debug_Exit "IGCM_sys_Get"
[811]	1258	return
[664]	1259	fi
[2]	1260
[811]	1261	#USUAL WAY
[938]	1262	\cp ${dm_liste[*]} ${DEST} > /tmp/out_command.$$ 2>&1
[811]	1263	status=$?
[2]	1264
[811]	1265	if [ ${status} -gt 0 ] ; then
	1266	IGCM_debug_Print 2 "IGCM_sys_Get : cp failed error code ${status}"
[938]	1267	cat /tmp/out_command.$$
[664]	1268	IGCM_debug_Exit "IGCM_sys_Get"
[2]	1269	else
[938]	1270	\rm /tmp/out_command.$$
[2]	1271	fi
[664]	1272	fi
	1273	IGCM_debug_PopStack "IGCM_sys_Get"
[2]	1274	}
	1275
	1276	#D-#==================================================
[664]	1277	#D-function IGCM_sys_GetBuffer
	1278	#D-* Purpose: Get a file from ${SCRATCHDIR}
	1279	#D-* Examples: IGCM_sys_GetBuffer myfile /destpath/myfile_with_PREFIX
	1280	#D- IGCM_sys_GetBuffer /l Array_contain_myfiles /destpath/
	1281	function IGCM_sys_GetBuffer {
	1282	IGCM_debug_PushStack "IGCM_sys_GetBuffer" $@
	1283
[811]	1284	typeset DEST buf_liste target file_work
	1285	typeset NB_ESSAI DELAI status i
[664]	1286
	1287	if ( $DEBUG_sys ) ; then
	1288	echo "IGCM_sys_GetBuffer :" $@
	1289	fi
[811]	1290
	1291	# number of tentative
	1292	NB_ESSAI=3
	1293	# time delay between tentative
	1294	DELAI=2
	1295
[664]	1296	if [ $DRYRUN -le 2 ]; then
	1297	if [ X${1} = X'/l' ] ; then
	1298	eval set +A buf_liste \${${2}}
	1299	else
	1300	eval set +A buf_liste ${1}
	1301	fi
	1302	eval DEST=\${${#}}
	1303
	1304	#USUAL WAY
	1305	if [ X${1} = X'/l' ] ; then
	1306	for target in ${buf_liste[*]} ; do
	1307	local_file=$( basename ${target} )
[811]	1308	i=0
	1309	while [ $i -lt $NB_ESSAI ] ; do
[938]	1310	\cp ${target} ${DEST}/${local_file} >> /tmp/out_command.$$ 2>&1
[811]	1311	status=$?
	1312	if [ ${status} -gt 0 ]; then
	1313	IGCM_debug_Print 2 "IGCM_sys_GetBuffer : cp failed error code ${status} ${i}/${NB_ESSAI}"
	1314	IGCM_debug_Print 2 "IGCM_sys_GetBuffer : sleep ${DELAI} seconds and try again."
	1315	sleep $DELAI
	1316	else
	1317	break
	1318	fi
	1319	(( i = i + 1 ))
	1320	done
	1321	if [ ${status} -gt 0 ] ; then
	1322	echo "IGCM_sys_Get : error"
[938]	1323	cat /tmp/out_command.$$
	1324	\rm /tmp/out_command.$$
[811]	1325	IGCM_debug_Exit "IGCM_sys_GetBuffer"
	1326	else
[938]	1327	\rm /tmp/out_command.$$
[811]	1328	fi
[664]	1329	done
	1330	else
[811]	1331	i=0
	1332	while [ $i -lt $NB_ESSAI ] ; do
[938]	1333	\cp ${buf_liste} ${DEST} >> /tmp/out_command.$$ 2>&1
[811]	1334	status=$?
	1335	if [ ${status} -gt 0 ]; then
	1336	IGCM_debug_Print 2 "IGCM_sys_GetBuffer : cp failed error code ${status} ${i}/${NB_ESSAI}"
	1337	IGCM_debug_Print 2 "IGCM_sys_GetBuffer : sleep ${DELAI} seconds and try again."
	1338	sleep $DELAI
	1339	else
	1340	break
	1341	fi
	1342	(( i = i + 1 ))
	1343	done
	1344	if [ ${status} -gt 0 ] ; then
	1345	echo "IGCM_sys_Get : error"
[938]	1346	cat /tmp/out_command.$$
	1347	\rm /tmp/out_command.$$
[811]	1348	IGCM_debug_Exit "IGCM_sys_GetBuffer"
	1349	else
[938]	1350	\rm /tmp/out_command.$$
[811]	1351	fi
[664]	1352	fi
	1353	fi
	1354	IGCM_debug_PopStack "IGCM_sys_GetBuffer"
	1355	}
	1356
	1357	#D-#==================================================
	1358	#D-function IGCM_sys_GetDate_FichWork
	1359	#D-* Purpose: donne la date filesys d'un fichier sur le filesystem WORK
	1360	#D-* Examples:
	1361	#D-
	1362	function IGCM_sys_GetDate_FichWork {
	1363	IGCM_debug_PushStack "IGCM_sys_FichWork" $@
	1364	if ( $DEBUG_sys ) ; then
	1365	echo "IGCM_sys_GetDate_FichWork :" $@
	1366	fi
	1367	typeset dateF
	1368	set +A dateF -- $( ls -l --full-time --time-style='+%Y%m%d%H%M%S' ${1} )
	1369	eval ${2}=${dateF[5]}
	1370
[800]	1371	# donne la date filesys d'un fichier sur la machine work
[664]	1372	IGCM_debug_PopStack "IGCM_sys_FichWork"
	1373	}
	1374
	1375	#D-#==================================================
	1376	#D-function IGCM_sys_GetDate_FichArchive
	1377	#D-* Purpose: donne la date filesys d'un fichier sur le filesystem ARCHIVE
	1378	#D-* Examples:
	1379	#D-
	1380	function IGCM_sys_GetDate_FichArchive {
	1381	IGCM_debug_PushStack "IGCM_sys_FichArchive" $@
	1382	if ( $DEBUG_sys ) ; then
	1383	echo "IGCM_sys_GetDate_FichArchive :" $@
	1384	fi
	1385	typeset dateF
	1386	set +A dateF -- $( ls -l --full-time --time-style='+%Y%m%d%H%M%S' ${1} )
	1387	eval ${2}=${dateF[5]}
	1388
	1389	IGCM_debug_PopStack "IGCM_sys_FichArchive"
	1390	}
	1391
	1392	#D-#==================================================
[343]	1393	#D-function IGCM_sys_Dods_Rm
[717]	1394	#D-* Purpose: DO NOTHING ! Put ${ARCHIVE} files on DODS internet protocole.
[343]	1395	#D-* Examples:
	1396	#D-
	1397	function IGCM_sys_Dods_Rm {
[664]	1398	if ( $DEBUG_sys ) ; then
	1399	echo "IGCM_sys_Dods_Rm :" $@
	1400	fi
	1401	return 0
[343]	1402	}
	1403
	1404	#D-#==================================================
	1405	#D-function IGCM_sys_Dods_Cp
[811]	1406	#D-* Purpose: Copy $(ARCHIVE) files on DODS internet protocole.
[343]	1407	#D-* Examples:
	1408	#D-
	1409	function IGCM_sys_Dods_Cp {
[664]	1410	if ( $DEBUG_sys ) ; then
	1411	echo "IGCM_sys_Dods_Cp :" $@
	1412	fi
	1413	return 0
[343]	1414	}
	1415
	1416	#D-#==================================================
[2]	1417	#D-function IGCM_sys_Put_Dods
[811]	1418	#D-* Purpose: Put ${ARCHIVE} files on DODS internet protocole. Dummy function here
[2]	1419	#D-* Examples:
	1420	#D-
	1421	function IGCM_sys_Put_Dods {
[664]	1422	IGCM_debug_PushStack "IGCM_sys_Put_Dods" $@
	1423	if ( $DEBUG_sys ) ; then
	1424	echo "IGCM_sys_Put_Dods :" $@
	1425	fi
	1426	IGCM_debug_PopStack "IGCM_sys_Put_Dods"
[2]	1427	}
	1428
[664]	1429	##############################################################
	1430	# REBUILD OPERATOR
[2]	1431
[811]	1432	#D-#==================================================
	1433	#D-function IGCM_sys_rebuild
	1434	#D-* Purpose: rebuild parallel files
	1435	#D-* Examples:
	1436	#D-
[664]	1437	function IGCM_sys_rebuild {
[1057]	1438	IGCM_debug_PushStack "IGCM_sys_rebuild" $@
[664]	1439	if ( $DEBUG_sys ) ; then
	1440	echo "IGCM_sys_rebuild :" $@
	1441	fi
[811]	1442
	1443	typeset NB_ESSAI DELAI status i firstArg
	1444	# number of tentative
	1445	NB_ESSAI=3
	1446	# time delay between tentative
	1447	DELAI=2
	1448
	1449	i=0
	1450	while [ $i -lt $NB_ESSAI ] ; do
[938]	1451	rebuild -f -o $@ > /tmp/out_command.$$ 2>&1
[811]	1452	status=$?
	1453	if [ ${status} -gt 0 ] ; then
	1454	IGCM_debug_Print 2 "IGCM_sys_rebuild : error code ${status}"
[938]	1455	cat /tmp/out_command.$$
	1456	\rm /tmp/out_command.$$
[811]	1457	IGCM_debug_Print 2 "IGCM_sys_rebuild : ${i}/${NB_ESSAI} sleep ${DELAI} seconds and try again."
	1458	firstArg=${1}
	1459	\rm ${firstArg}
	1460	sleep $DELAI
	1461	else
[938]	1462	\rm /tmp/out_command.$$
[811]	1463	break
	1464	fi
	1465	(( i = i + 1 ))
	1466	done
	1467
	1468	if [ ${status} -gt 0 ] ; then
	1469	echo "IGCM_sys_rebuild : rebuild error code is ${status}"
[664]	1470	IGCM_debug_Exit "rebuild"
	1471	fi
	1472
	1473	IGCM_debug_PopStack "IGCM_sys_rebuild"
[2]	1474	}
	1475
[811]	1476	#D-#==================================================
	1477	#D-function IGCM_sys_rebuild_station
[922]	1478	#D-* Purpose: rebuild parallel files describing station
[811]	1479	#D-* Examples:
	1480	#D-
[697]	1481	function IGCM_sys_rebuild_station {
[1057]	1482	IGCM_debug_PushStack "IGCM_sys_rebuild_station" $@
[697]	1483	typeset i list_opt file_in file_out prefix_invert list_invert
	1484	if ( $DEBUG_sys ) ; then
	1485	echo "IGCM_sys_rebuild_station :" $@
	1486	fi
	1487	list_opt=$@
	1488
[800]	1489	# Invert Axis : t,x -> x,t
[697]	1490	# t,pres,x -> x,t,pres
	1491	# So that we can concatenate along x
	1492	i=0
	1493	for file_in in ${list_opt} ; do
	1494	(( i = i + 1))
	1495	[ ${i} = 1 ] && file_out=${file_in} && continue
[1046]	1496	# detect time counter and do the job only if present
	1497	var_unlim=$(ncdump -h ${file_in} \| grep UNLIMITED \| cut -d ' ' -f 1)
	1498	if [ X${var_unlim} = Xtime_counter ] ; then
	1499	prefix_invert=$( basename ${file_in} .nc )
	1500	IGCM_sys_ncpdq -a x,time_counter -a x,time_counter,presnivs ${file_in} ${prefix_invert}_xt.nc
	1501	list_invert[${#list_invert[*]}]=${prefix_invert}_xt.nc
	1502	fi
[697]	1503	done
	1504
	1505	# Concatenate
	1506	IGCM_sys_ncrcat ${list_invert[*]} histstn_xt.nc
	1507
	1508	# Re-ivert file
	1509	IGCM_sys_ncpdq -a time_counter,x -a time_counter,presnivs,x histstn_xt.nc ${file_out}
	1510
[922]	1511	# Station re-ordering is too expansive to be run within libIGCM
[697]	1512	# This is due to (ncpdq - nrcat - ncpdq) I/O sequence.
	1513	# This re-ordering must be done "in memory" by the cmorization process
	1514	# Anyway this is the best sequence using (ncpdq - nrcat - ncpdq)
	1515	# BEGIN reordering
	1516
	1517	# Only LMDZ text output contains the exact ordering of the station.
	1518	# We isolate this in the code below:
	1519	# 0 38 -157.5000000000000 70.98591549295774
	1520	# 0 54 27.49999999999999 67.18309859154928
	1521	# 0 56 -62.50000000000001 82.39436619718309
	1522	# 0 79 12.49999999999999 78.59154929577466
	1523	# 0 116 -165.0000000000000 76.05633802816901
	1524	# 0 117 130.0000000000000 70.98591549295774
	1525	# 0 118 110.0000000000000 87.46478873239437
	1526	# 1 40 4.999999999999995 51.97183098591550
	1527	# typeset iStation iProc list_opt file_in file_out prefix_invert
	1528	# typeset -Z4 j4
	1529	# typeset -Z3 j3
	1530
	1531	# unset list_opt
	1532	# set +A list_opt $@
	1533
	1534	# Filename after rebuild
	1535	# file_out=${list_opt[0]}
	1536	# Prefix of output files
	1537	# prefix_invert=$( basename ${file_out} .nc )
	1538	# Number of procs
	1539	# num_proc=$( grep -i mpi_size ${PREFIX}_${Exe_Output} \| wc -l )
	1540
	1541	# iProc=0
	1542	# while [ ${iProc} -lt ${num_proc} ] ; do
	1543	# Array containing Station as a number
	1544	# unset proc_stn
[912]	1545	# set +A proc_stn $( grep "iophy_mpi rank ip lon lat $iProc" ${PREFIX}_${Exe_Output} \| sed -e "s/iophy_mpi rank ip lon lat //g" \| gawk ' {print $2}' )
[697]	1546	# Number of stations produced by processor proc
	1547	# stationLast=${#proc_stn[*]}
	1548	# Proc number on 4 digits
	1549	# j4=${iProc}
	1550	# Init
	1551	# iStation=0
	1552	# while [ ${iStation} -lt ${stationLast} ] ; do
	1553	# Station number on 3 digits
	1554	# j3=${proc_stn[${iStation}]}
	1555	# Extract station
[800]	1556	# Invert Axis : t,x -> x,t
[697]	1557	# t,pres,x -> x,t,pres
	1558	# So that we can concatenate along x
	1559	# IGCM_sys_ncpdq -a x,time_counter -a x,time_counter,presnivs -d x,$iStation,$iStation ${prefix_invert}_${j4}.nc ${prefix_invert}_stn_${j3}.nc
	1560	# (( iStation = iStation + 1 ))
	1561	# done
	1562	# (( iProc = iProc + 1 ))
	1563	# done
	1564
	1565	# Concatenate all station along x
	1566	# IGCM_sys_ncrcat ${prefix_invert}_stn_???.nc ${prefix_invert}_xt.nc
	1567
	1568	# Re-invert file
	1569	# IGCM_sys_ncpdq -a time_counter,x -a time_counter,presnivs,x ${prefix_invert}_xt.nc ${file_out}
	1570
	1571	# END reordering
	1572
	1573	IGCM_debug_PopStack "IGCM_sys_rebuild_station"
	1574	}
	1575
[664]	1576	############################################################
	1577	# Activate Running Environnment Variables
	1578
[811]	1579	#D-#==================================================
	1580	#D-function IGCM_sys_desactiv_variables
	1581	#D-* Purpose: set environement variables prior to execution
	1582	#D-* Examples:
	1583	#D-
[664]	1584	function IGCM_sys_activ_variables {
	1585	IGCM_debug_PushStack "IGCM_sys_activ_variables"
	1586	if ( $DEBUG_sys ) ; then
	1587	echo "IGCM_sys_activ_variables"
	1588	fi
[811]	1589
	1590	# --------------------------------------------------------------------
	1591	#D- MPI specifications
	1592	# --------------------------------------------------------------------
	1593
	1594	# --------------------------------------------------------------------
	1595	#D- Other specifications
	1596	# --------------------------------------------------------------------
	1597
[664]	1598	IGCM_debug_PopStack "IGCM_sys_activ_variables"
[2]	1599	}
	1600
[664]	1601	############################################################
	1602	# Desactivate Running Environnment Variables
[2]	1603
[811]	1604	#D-#==================================================
	1605	#D-function IGCM_sys_desactiv_variables
	1606	#D-* Purpose: unset environement variables after execution
	1607	#D-* Examples:
	1608	#D-
[664]	1609	function IGCM_sys_desactiv_variables {
	1610	IGCM_debug_PushStack "IGCM_sys_desactiv_variables"
	1611	if ( $DEBUG_sys ) ; then
	1612	echo "IGCM_sys_desactiv_variables"
	1613	fi
[811]	1614	# --------------------------------------------------------------------
	1615	#D- MPI specifications
	1616	# --------------------------------------------------------------------
	1617
	1618	# --------------------------------------------------------------------
	1619	#D- Other specifications
	1620	# --------------------------------------------------------------------
	1621
[664]	1622	IGCM_debug_PopStack "IGCM_sys_desactiv_variables"
	1623	}
	1624
	1625	############################################################
[800]	1626	# Build MPI/OMP scripts run file (dummy function)
[664]	1627
[811]	1628	#D-#==================================================
	1629	#D-function IGCM_sys_build_run_file
	1630	#D-* Purpose: build run file (deprecated)
	1631	#D-* Examples:
	1632	#D-
[664]	1633	function IGCM_sys_build_run_file {
	1634
	1635	IGCM_debug_Print 3 " dummy function : IGCM_sys_build_run_file "
	1636
	1637	}
	1638
	1639	############################################################
	1640	# Build MPI/OMP scripts
[811]	1641
	1642	#D-#==================================================
	1643	#D-function IGCM_sys_build_execution_scripts
	1644	#D-* Purpose: build execution scripts to be launch by ${HOST_MPIRUN_COMMAND}
	1645	#D-* Examples:
	1646	#D-
[664]	1647	function IGCM_sys_build_execution_scripts
	1648	{
	1649	IGCM_debug_PushStack "IGCM_sys_build_execution_scripts" $@
	1650	if ( $DEBUG_sys ) ; then
	1651	echo "IGCM_sys_build_execution_scripts " $@
	1652	fi
	1653	typeset nodes listnodes init_node start_num init_exec comp ExeNameIn ExeNameOut
	1654	typeset node_num_current node_current comp_proc_mpi_loc comp_proc_omp_loc
	1655	typeset num_corempi nombre_restant_node nombre_restant_comp
	1656
	1657	if [ ! -f ${SUBMIT_DIR}/Job_${config_UserChoices_JobName} ] ; then
[811]	1658	IGCM_debug_Exit "IGCM_sys_default build_execution_scripts : Job_${config_UserChoices_JobName} don't exist in SUBMIT_DIR : ${SUBMIT_DIR} "
[664]	1659	fi
	1660
	1661	if ( ${OK_PARA_MPMD} ) ; then
	1662
	1663	if [ -f run_file ] ; then
	1664	IGCM_sys_Rm -f run_file
[2]	1665	fi
[664]	1666	touch run_file
	1667
	1668	if ( ${OK_PARA_OMP} ) ; then
	1669
	1670	# Hosts treatment
	1671
	1672	${HOST_MPIRUN_COMMAND} hostname \| sort \| uniq > hosts.tmp
	1673
	1674	i=0
	1675	rm -f hosts
	1676	IGCM_debug_Print 1 "sys $( hostname ), Hosts avaible :"
	1677	for nodes in `cat hosts.tmp` ; do
	1678	host[$i]=$nodes
	1679	echo "${host[$i]} slots=1 max_slots=1" >> hosts
	1680	IGCM_debug_Print 1 ${host[$i]}
	1681	i=$((i+1))
	1682	done
	1683	rm -f hosts.tmp
	1684
	1685	listnodes=${host[*]}
	1686
	1687	EXECUTION="${HOST_MPIRUN_COMMAND} -hostfile hosts"
	1688
	1689	# Initialisation
	1690
	1691	init_node=y
	1692	node_num_current=0
	1693	start_num=0
	1694	init_exec=n
	1695
	1696	# Test : if oasis is there, we put it at the first position
	1697
	1698	for comp in ${config_ListOfComponents[*]} ; do
	1699
	1700	if [ "X${comp}" = "XCPL" ] ; then
	1701
	1702	eval ExeNameIn=\${config_Executable_${comp}[0]}
	1703	eval ExeNameOut=\${config_Executable_${comp}[1]}
	1704
	1705	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
	1706	echo "" >> script_${ExeNameOut}.ksh
	1707	#echo "export KMP_STACKSIZE=3g" >> script_${ExeNameOut}.ksh
	1708	#echo "export KMP_LIBRARY=turnaround" >> script_${ExeNameOut}.ksh
	1709	#echo "export MKL_SERIAL=YES" >> script_${ExeNameOut}.ksh
	1710	echo "./${ExeNameOut} > out_${ExeNameOut}.out 2>out_${ExeNameOut}.err" >> script_${ExeNameOut}.ksh
	1711	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
	1712
	1713	init_node=n
	1714
	1715	(( nombre_restant_node = NUM_COREPERNODE - 1 ))
	1716	node_num_current=0
	1717	node_current=${host[${node_num_current}]}
	1718
	1719	EXECUTION="${EXECUTION} -H ${node_current} -np 1 ./script_${ExeNameOut}.ksh"
	1720
	1721	init_exec=y
	1722	start_num=1
	1723
	1724	fi
	1725
	1726	done
	1727
	1728	# Then loop on the components (except for oasis)
	1729
	1730	for comp in ${config_ListOfComponents[*]} ; do
	1731
	1732	eval ExeNameIn=\${config_Executable_${comp}[0]}
	1733	eval ExeNameOut=\${config_Executable_${comp}[1]}
	1734
	1735	# Only if we really have an executable for the component :
	1736	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${comp}" != "XCPL" ] ) ; then
	1737
	1738	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
	1739	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
	1740
	1741	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
	1742	# echo "set -vx" >> script_${ExeNameOut}.ksh
	1743	echo "" >> script_${ExeNameOut}.ksh
	1744	#echo "export KMP_STACKSIZE=3g" >> script_${ExeNameOut}.ksh
	1745	#echo "export KMP_LIBRARY=turnaround" >> script_${ExeNameOut}.ksh
	1746	#echo "export MKL_SERIAL=YES" >> script_${ExeNameOut}.ksh
	1747	echo "OMP_NUM_THREADS=${comp_proc_omp_loc}" >> script_${ExeNameOut}.ksh
	1748	echo "(( MYMPIRANK = OMPI_COMM_WORLD_RANK - ${start_num})) " >> script_${ExeNameOut}.ksh
	1749	echo "./${ExeNameOut} > out_${ExeNameOut}.out.\${MYMPIRANK} 2>out_${ExeNameOut}.err.\${MYMPIRANK}" >> script_${ExeNameOut}.ksh
	1750	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
	1751
	1752	node_num=0
	1753
	1754	# We define the number of MPI process to be assigned for the component
	1755
	1756	nombre_restant_comp=${comp_proc_mpi_loc}
	1757
	1758	# Loop on the allocated nodes
	1759
	1760	for node in ${listnodes} ; do
	1761
	1762	# We go to the current node
	1763	if [ ${node_num} = ${node_num_current} ] ; then
	1764
	1765	node_current=${host[${node_num_current}]}
	1766
	1767	# If first time on the node : initialisation
	1768
	1769	if [ ${init_node} = y ] ; then
	1770	nombre_restant_node=${NUM_COREPERNODE}
	1771	fi
	1772
	1773	# Test on the number of OMP threads
	1774
	1775	if [ ${comp_proc_omp_loc} -gt ${nombre_restant_node} ] ; then
	1776	(( node_num = node_num + 1 ))
	1777	node_num_current=${node_num}
	1778	init_node=y
	1779	continue
	1780	fi
	1781
	1782	# Number of MPI process to assign
	1783
	1784	(( num_corempi = nombre_restant_node / comp_proc_omp_loc ))
	1785
	1786	if [ ${num_corempi} -gt ${nombre_restant_comp} ] ; then
	1787	num_corempi=${nombre_restant_comp}
	1788	fi
	1789
	1790	(( nombre_restant_node = nombre_restant_node - num_corempi * comp_proc_omp_loc ))
	1791	(( nombre_restant_comp = nombre_restant_comp - num_corempi ))
	1792
	1793	if [ ${init_exec} = y ] ; then
	1794	EXECUTION="${EXECUTION} : -H ${node_current} -np ${num_corempi} ./script_${ExeNameOut}.ksh"
	1795	else
	1796	EXECUTION="${EXECUTION} -H ${node_current} -np ${num_corempi} ./script_${ExeNameOut}.ksh"
	1797	init_exec=y
	1798	fi
	1799
	1800	(( start_num = num_corempi + start_num ))
	1801
	1802	else
	1803
	1804	(( node_num = node_num + 1 ))
	1805	continue
	1806	fi
	1807
	1808	# Test on the number of core/process remaining on the node/component
	1809
	1810	if [ ${nombre_restant_node} = 0 ] ; then
	1811	(( node_num = node_num + 1 ))
	1812	node_num_current=${node_num}
	1813	init_node=y
	1814
	1815	if [ ${nombre_restant_comp} = 0 ] ; then
	1816	break 1
	1817	fi
	1818	else
	1819
	1820	node_num_current=${node_num}
	1821	init_node=n
	1822
	1823	if [ ${nombre_restant_comp} = 0 ] ; then
	1824	break 1
	1825	fi
	1826	fi
	1827	done
	1828	fi
	1829	done
	1830
	1831	else
	1832
	1833	# Then first loop on the components for the coupler ie oasis
	1834
	1835	## the coupler ie oasis must be the first one
	1836	for comp in ${config_ListOfComponents[*]} ; do
	1837
	1838	eval ExeNameOut=\${config_Executable_${comp}[1]}
	1839
	1840	# for CPL component only
	1841	if [ "X${comp}" = "XCPL" ] ; then
	1842	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
	1843	echo "-np ${comp_proc_mpi_loc} ./${ExeNameOut} " >> run_file
	1844	fi
	1845	done
	1846
	1847	# Then second loop on the components
	1848
	1849	for comp in ${config_ListOfComponents[*]} ; do
	1850
	1851	eval ExeNameOut=\${config_Executable_${comp}[1]}
	1852
	1853	# Only if we really have an executable for the component and not the coupler ie oasis:
	1854	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${comp}" != "XCPL" ] ) ; then
	1855	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
	1856	echo "-np ${comp_proc_mpi_loc} ./${ExeNameOut}" >> run_file
	1857	fi
	1858	done
	1859	IGCM_sys_Chmod u+x run_file
	1860
	1861	EXECUTION="${HOST_MPIRUN_COMMAND} --app ./run_file"
	1862
[2]	1863	fi
	1864
[664]	1865	else # Only one executable. launch it.
	1866
	1867	for comp in ${config_ListOfComponents[*]} ; do
	1868
	1869	# Only if we really have an executable for the component :
	1870	eval ExeNameOut=\${config_Executable_${comp}[1]}
	1871	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${ExeNameOut}" != "Xinca.dat" ] ) ; then
	1872
	1873	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
	1874	echo "" >> script_${ExeNameOut}.ksh
	1875	if ( ${OK_PARA_OMP} ) ; then
	1876	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
	1877	echo "OMP_NUM_THREADS=${comp_proc_omp_loc}" >> script_${ExeNameOut}.ksh
	1878	fi
	1879	if ( ${OK_PARA_MPI} ) ; then
	1880	# Default : mpirun used if nb_proc gt 1
	1881	# pour sortie out/err par process
	1882	# echo "./${ExeNameOut} > out_${ExeNameOut}.out.\${OMPI_COMM_WORLD_RANK} 2>out_${ExeNameOut}.err.\${OMPI_COMM_WORLD_RANK}" >> script_${ExeNameOut}.ksh
	1883	echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
	1884	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
	1885	EXECUTION="${HOST_MPIRUN_COMMAND} ./script_${ExeNameOut}.ksh"
	1886	else
	1887	# Default : mpirun is NOT used if nb_proc eq 1
	1888	# pour sortie out/err par process
	1889	# echo "./${ExeNameOut} > out_${ExeNameOut}.out 2>out_${ExeNameOut}.err" >> script_${ExeNameOut}.ksh
	1890	echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
	1891	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
	1892	EXECUTION="time ./script_${ExeNameOut}.ksh"
	1893	fi
	1894	fi
	1895	done
	1896
	1897	fi
	1898
[811]	1899	IGCM_debug_Print 1 "sys $( hostname ) : execution command is"
	1900	IGCM_debug_Print 1 "$EXECUTION"
[664]	1901
	1902	IGCM_debug_PopStack "IGCM_sys_build_execution_scripts"
[2]	1903	}
	1904
[933]	1905	#D-#==================================================
	1906	#D-function IGCM_sys_check_path
	1907	#D-* Purpose: check that RUN_DIR_PATH that will be removed on some machine
	1908	#D-* do not point to an important use directory. Stop immediately in that case.
	1909	#D-* Examples:
	1910	#D-
	1911	function IGCM_sys_check_path {
	1912	IGCM_debug_PushStack "IGCM_sys_check_path"
	1913	if ( $DEBUG_sys ) ; then
	1914	echo "IGCM_sys_check_path"
	1915	fi
[811]	1916
[933]	1917	if ( [ X${RUN_DIR_PATH} = X${HOME} ] \|\| [ X${RUN_DIR_PATH} = X${ARCHIVE} ] ) ; then
	1918	IGCM_debug_Print 1 "Variable RUN_DIR_PATH is pointing to an important directory : ${RUN_DIR_PATH}"
	1919	IGCM_debug_Print 1 "Please check the RUN_DIR_PATH definition in your Job : Job_${config_UserChoices_JobName}"
[936]	1920	IGCM_debug_Exit "This will stop the job"
[933]	1921	fi
	1922	IGCM_debug_PopStack "IGCM_sys_check_path"
	1923	}
	1924
[811]	1925	#D-#==================================================
	1926	#D-function IGCM_sys_check_quota. Dummy call here
	1927	#D-* Purpose: check user quota. Stop the simulation if quota above 90%
	1928	#D-* Examples:
	1929	#D-
[664]	1930	function IGCM_sys_check_quota {
	1931	IGCM_debug_PushStack "IGCM_sys_check_quota"
	1932	if ( $DEBUG_sys ) ; then
	1933	echo "IGCM_sys_check_quota"
	1934	fi
	1935	IGCM_debug_PopStack "IGCM_sys_check_quota"
	1936	}
	1937
[811]	1938	#D-#==================================================
	1939	#D-function IGCM_sys_CountJobInQueue
	1940	#D-* Purpose: Check if job_name is currently
	1941	#D- running or in queue
	1942	#D-* Examples: IGCM_sys_CountJobInQueue ${JobName} NbRun
	1943	#D-
	1944	function IGCM_sys_CountJobInQueue {
	1945	IGCM_debug_PushStack "IGCM_sys_CountJobInQueue"
	1946	if ( $DEBUG_sys ) ; then
	1947	echo "IGCM_sys_CountJobInQueue"
	1948	fi
	1949	IGCM_debug_PopStack "IGCM_sys_CountJobInQueue"
	1950	}
	1951
[2]	1952	##############################################################
	1953	# NCO OPERATOR
	1954
[811]	1955	#D-#==================================================
	1956	#D-function IGCM_sys_ncap2
	1957	#D-* Purpose: encapsulate ncap2 call so as to manage error code and retry
	1958	#D-* Examples:
	1959	#D-
[377]	1960	function IGCM_sys_ncap2 {
[1057]	1961	IGCM_debug_PushStack "IGCM_sys_ncap2" $@
[664]	1962	if ( $DEBUG_sys ) ; then
	1963	echo "IGCM_sys_ncap2 :" $@
	1964	fi
[714]	1965
	1966	typeset NB_ESSAI DELAI status i
	1967	# number of tentative
	1968	NB_ESSAI=3
	1969	# time delay between tentative
	1970	DELAI=2
	1971
	1972	i=0
	1973	while [ $i -lt $NB_ESSAI ] ; do
[1091]	1974	ncap2 -C "$@" > /tmp/out_command.$$ 2>&1
[714]	1975	status=$?
	1976	if [ ${status} -gt 0 ] ; then
	1977	IGCM_debug_Print 2 "IGCM_sys_ncap2 : error code ${status}"
[938]	1978	cat /tmp/out_command.$$
	1979	\rm /tmp/out_command.$$
[714]	1980	IGCM_debug_Print 2 "IGCM_sys_ncap2 : ${i}/${NB_ESSAI} sleep ${DELAI} seconds and try again."
[800]	1981	sleep $DELAI
[714]	1982	else
[938]	1983	\rm /tmp/out_command.$$
[714]	1984	break
	1985	fi
	1986	(( i = i + 1 ))
	1987	done
	1988
	1989	if [ ${status} -gt 0 ] ; then
[811]	1990	echo "IGCM_sys_ncap2 : ncap2 error"
	1991	IGCM_debug_Exit "ncap2"
[664]	1992	fi
[2]	1993
[664]	1994	IGCM_debug_PopStack "IGCM_sys_ncap2"
[2]	1995	}
	1996
[811]	1997	#D-#==================================================
	1998	#D-function IGCM_sys_ncatted
	1999	#D-* Purpose: encapsulate ncatted call so as to manage error code and retry
	2000	#D-* Examples:
	2001	#D-
[375]	2002	function IGCM_sys_ncatted {
[1057]	2003	IGCM_debug_PushStack "IGCM_sys_ncatted" $@
[664]	2004	if ( $DEBUG_sys ) ; then
	2005	echo "IGCM_sys_ncatted :" $@
	2006	fi
[714]	2007
	2008	typeset NB_ESSAI DELAI status i
	2009	# number of tentative
	2010	NB_ESSAI=3
	2011	# time delay between tentative
	2012	DELAI=2
	2013
	2014	i=0
	2015	while [ $i -lt $NB_ESSAI ] ; do
[938]	2016	ncatted "$@" > /tmp/out_command.$$ 2>&1
[714]	2017	status=$?
	2018	if [ ${status} -gt 0 ] ; then
	2019	IGCM_debug_Print 2 "IGCM_sys_ncatted : error code ${status}"
[938]	2020	cat /tmp/out_command.$$
	2021	\rm /tmp/out_command.$$
[714]	2022	IGCM_debug_Print 2 "IGCM_sys_ncatted : ${i}/${NB_ESSAI} sleep ${DELAI} seconds and try again."
[800]	2023	sleep $DELAI
[714]	2024	else
[938]	2025	\rm /tmp/out_command.$$
[714]	2026	break
	2027	fi
	2028	(( i = i + 1 ))
	2029	done
	2030
	2031	if [ ${status} -gt 0 ] ; then
[811]	2032	echo "IGCM_sys_ncatted : ncatted error"
	2033	IGCM_debug_Exit "ncatted"
[664]	2034	fi
[2]	2035
[664]	2036	IGCM_debug_PopStack "IGCM_sys_ncatted"
[375]	2037	}
[2]	2038
[811]	2039	#D-#==================================================
	2040	#D-function IGCM_sys_ncbo
	2041	#D-* Purpose: encapsulate ncbo call so as to manage error code and retry
	2042	#D-* Examples:
	2043	#D-
[2]	2044	function IGCM_sys_ncbo {
[1057]	2045	IGCM_debug_PushStack "IGCM_sys_ncbo" $@
[664]	2046	if ( $DEBUG_sys ) ; then
	2047	echo "IGCM_sys_ncbo :" $@
	2048	fi
[714]	2049
	2050	typeset NB_ESSAI DELAI status i
	2051	# number of tentative
	2052	NB_ESSAI=3
	2053	# time delay between tentative
	2054	DELAI=2
	2055
	2056	i=0
	2057	while [ $i -lt $NB_ESSAI ] ; do
[1091]	2058	ncbo -C $@ > /tmp/out_command.$$ 2>&1
[714]	2059	status=$?
	2060	if [ ${status} -gt 0 ] ; then
	2061	IGCM_debug_Print 2 "IGCM_sys_ncbo : error code ${status}"
[938]	2062	cat /tmp/out_command.$$
	2063	\rm /tmp/out_command.$$
[714]	2064	IGCM_debug_Print 2 "IGCM_sys_ncbo : ${i}/${NB_ESSAI} sleep ${DELAI} seconds and try again."
[800]	2065	sleep $DELAI
[714]	2066	else
[938]	2067	\rm /tmp/out_command.$$
[714]	2068	break
	2069	fi
	2070	(( i = i + 1 ))
	2071	done
	2072
	2073	if [ ${status} -gt 0 ] ; then
[811]	2074	echo "IGCM_sys_ncbo : ncbo error"
	2075	IGCM_debug_Exit "ncbo"
[664]	2076	fi
[2]	2077
[664]	2078	IGCM_debug_PopStack "IGCM_sys_ncbo"
[2]	2079	}
	2080
[811]	2081	#D-#==================================================
	2082	#D-function IGCM_sys_ncdif
	2083	#D-* Purpose: encapsulate ncdiff call so as to manage error code and retry
	2084	#D-* Examples:
	2085	#D-
[2]	2086	function IGCM_sys_ncdiff {
[1057]	2087	IGCM_debug_PushStack "IGCM_sys_ncdiff" $@
[664]	2088	if ( $DEBUG_sys ) ; then
	2089	echo "IGCM_sys_ncdiff :" $@
	2090	fi
[714]	2091
	2092	typeset NB_ESSAI DELAI status i
	2093	# number of tentative
	2094	NB_ESSAI=3
	2095	# time delay between tentative
	2096	DELAI=2
	2097
	2098	i=0
	2099	while [ $i -lt $NB_ESSAI ] ; do
[1091]	2100	ncdiff -C $@ > /tmp/out_command.$$ 2>&1
[714]	2101	status=$?
	2102	if [ ${status} -gt 0 ] ; then
	2103	IGCM_debug_Print 2 "IGCM_sys_ncdiff : error code ${status}"
[938]	2104	cat /tmp/out_command.$$
	2105	\rm /tmp/out_command.$$
[714]	2106	IGCM_debug_Print 2 "IGCM_sys_ncdiff : ${i}/${NB_ESSAI} sleep ${DELAI} seconds and try again."
[800]	2107	sleep $DELAI
[714]	2108	else
[938]	2109	\rm /tmp/out_command.$$
[714]	2110	break
	2111	fi
	2112	(( i = i + 1 ))
	2113	done
	2114
	2115	if [ ${status} -gt 0 ] ; then
[811]	2116	echo "IGCM_sys_ncdiff : ncdiff error"
	2117	IGCM_debug_Exit "ncdiff"
[664]	2118	fi
[2]	2119
[664]	2120	IGCM_debug_PopStack "IGCM_sys_ncdiff"
[2]	2121	}
	2122
[811]	2123	#D-#==================================================
	2124	#D-function IGCM_sys_ncea
	2125	#D-* Purpose: encapsulate ncea call so as to manage error code and retry
	2126	#D-* Examples:
	2127	#D-
[2]	2128	function IGCM_sys_ncea {
[1057]	2129	IGCM_debug_PushStack "IGCM_sys_ncea" $@
[664]	2130	if ( $DEBUG_sys ) ; then
	2131	echo "IGCM_sys_ncea :" $@
	2132	fi
[714]	2133
	2134	typeset NB_ESSAI DELAI status i
	2135	# number of tentative
	2136	NB_ESSAI=3
	2137	# time delay between tentative
	2138	DELAI=2
	2139
	2140	i=0
	2141	while [ $i -lt $NB_ESSAI ] ; do
[1091]	2142	ncea -C $@ > /tmp/out_command.$$ 2>&1
[714]	2143	status=$?
	2144	if [ ${status} -gt 0 ] ; then
	2145	IGCM_debug_Print 2 "IGCM_sys_ncea : error code ${status}"
[938]	2146	cat /tmp/out_command.$$
	2147	\rm /tmp/out_command.$$
[714]	2148	IGCM_debug_Print 2 "IGCM_sys_ncea : ${i}/${NB_ESSAI} sleep ${DELAI} seconds and try again."
[800]	2149	sleep $DELAI
[714]	2150	else
[938]	2151	\rm /tmp/out_command.$$
[714]	2152	break
	2153	fi
	2154	(( i = i + 1 ))
	2155	done
	2156
	2157	if [ ${status} -gt 0 ] ; then
[811]	2158	echo "IGCM_sys_ncea : ncea error"
	2159	IGCM_debug_Exit "ncea"
[664]	2160	fi
[2]	2161
[664]	2162	IGCM_debug_PopStack "IGCM_sys_ncea"
[2]	2163	}
	2164
[811]	2165	#D-#==================================================
	2166	#D-function IGCM_sys_ncecat
	2167	#D-* Purpose: encapsulate ncecat call so as to manage error code and retry
	2168	#D-* Examples:
	2169	#D-
[2]	2170	function IGCM_sys_ncecat {
[1057]	2171	IGCM_debug_PushStack "IGCM_sys_ncecat" $@
[664]	2172	if ( $DEBUG_sys ) ; then
	2173	echo "IGCM_sys_ncecat :" $@
	2174	fi
[714]	2175
	2176	typeset NB_ESSAI DELAI status i
	2177	# number of tentative
	2178	NB_ESSAI=3
	2179	# time delay between tentative
	2180	DELAI=2
	2181
	2182	i=0
	2183	while [ $i -lt $NB_ESSAI ] ; do
[1091]	2184	ncecat -C $@ > /tmp/out_command.$$ 2>&1
[714]	2185	status=$?
	2186	if [ ${status} -gt 0 ] ; then
	2187	IGCM_debug_Print 2 "IGCM_sys_ncecat : error code ${status}"
[938]	2188	cat /tmp/out_command.$$
	2189	\rm /tmp/out_command.$$
[714]	2190	IGCM_debug_Print 2 "IGCM_sys_ncecat : ${i}/${NB_ESSAI} sleep ${DELAI} seconds and try again."
[800]	2191	sleep $DELAI
[714]	2192	else
[938]	2193	\rm /tmp/out_command.$$
[714]	2194	break
	2195	fi
	2196	(( i = i + 1 ))
	2197	done
	2198
	2199	if [ ${status} -gt 0 ] ; then
[811]	2200	echo "IGCM_sys_ncecat : ncecat error"
	2201	IGCM_debug_Exit "ncecat"
[664]	2202	fi
[2]	2203
[664]	2204	IGCM_debug_PopStack "IGCM_sys_ncecat"
[2]	2205	}
	2206
[811]	2207	#D-#==================================================
	2208	#D-function IGCM_sys_ncflint
	2209	#D-* Purpose: encapsulate ncflint call so as to manage error code and retry
	2210	#D-* Examples:
	2211	#D-
[2]	2212	function IGCM_sys_ncflint {
[1057]	2213	IGCM_debug_PushStack "IGCM_sys_ncflint" $@
[664]	2214	if ( $DEBUG_sys ) ; then
	2215	echo "IGCM_sys_ncflint :" $@
	2216	fi
[714]	2217
	2218	typeset NB_ESSAI DELAI status i
	2219	# number of tentative
	2220	NB_ESSAI=3
	2221	# time delay between tentative
	2222	DELAI=2
	2223
	2224	i=0
	2225	while [ $i -lt $NB_ESSAI ] ; do
[1091]	2226	ncflint -C $@ > /tmp/out_command.$$ 2>&1
[714]	2227	status=$?
	2228	if [ ${status} -gt 0 ] ; then
	2229	IGCM_debug_Print 2 "IGCM_sys_ncflint : error code ${status}"
[938]	2230	cat /tmp/out_command.$$
	2231	\rm /tmp/out_command.$$
[714]	2232	IGCM_debug_Print 2 "IGCM_sys_ncflint : ${i}/${NB_ESSAI} sleep ${DELAI} seconds and try again."
[800]	2233	sleep $DELAI
[714]	2234	else
[938]	2235	\rm /tmp/out_command.$$
[714]	2236	break
	2237	fi
	2238	(( i = i + 1 ))
	2239	done
	2240
	2241	if [ ${status} -gt 0 ] ; then
[811]	2242	echo "IGCM_sys_ncflint : ncflint error"
	2243	IGCM_debug_Exit "ncflint"
[664]	2244	fi
[2]	2245
[664]	2246	IGCM_debug_PopStack "IGCM_sys_ncflint"
[2]	2247	}
	2248
[811]	2249	#D-#==================================================
	2250	#D-function IGCM_sys_ncks
	2251	#D-* Purpose: encapsulate ncks call so as to manage error code and retry
	2252	#D-* Examples:
	2253	#D-
[2]	2254	function IGCM_sys_ncks {
[1057]	2255	IGCM_debug_PushStack "IGCM_sys_ncks" $@
[664]	2256	if ( $DEBUG_sys ) ; then
	2257	echo "IGCM_sys_ncks :" $@
	2258	fi
[714]	2259
	2260	typeset NB_ESSAI DELAI status i
	2261	# number of tentative
	2262	NB_ESSAI=3
	2263	# time delay between tentative
	2264	DELAI=2
	2265
	2266	i=0
	2267	while [ $i -lt $NB_ESSAI ] ; do
[1091]	2268	ncks -C $@ > /tmp/out_command.$$ 2>&1
[714]	2269	status=$?
	2270	if [ ${status} -gt 0 ] ; then
	2271	IGCM_debug_Print 2 "IGCM_sys_ncks : error code ${status}"
[938]	2272	cat /tmp/out_command.$$
	2273	\rm /tmp/out_command.$$
[714]	2274	IGCM_debug_Print 2 "IGCM_sys_ncks : ${i}/${NB_ESSAI} sleep ${DELAI} seconds and try again."
[800]	2275	sleep $DELAI
[714]	2276	else
[938]	2277	\rm /tmp/out_command.$$
[714]	2278	break
	2279	fi
	2280	(( i = i + 1 ))
	2281	done
	2282
	2283	if [ ${status} -gt 0 ] ; then
[811]	2284	echo "IGCM_sys_ncks : ncks error"
	2285	IGCM_debug_Exit "ncks"
[664]	2286	fi
[2]	2287
[664]	2288	IGCM_debug_PopStack "IGCM_sys_ncks"
[2]	2289	}
	2290
[811]	2291	#D-#==================================================
	2292	#D-function IGCM_sys_ncpdq
	2293	#D-* Purpose: encapsulate ncpdq call so as to manage error code and retry
	2294	#D-* Examples:
	2295	#D-
[2]	2296	function IGCM_sys_ncpdq {
[1057]	2297	IGCM_debug_PushStack "IGCM_sys_ncpdq" $@
[664]	2298	if ( $DEBUG_sys ) ; then
	2299	echo "IGCM_sys_ncpdq :" $@
	2300	fi
[714]	2301
	2302	typeset NB_ESSAI DELAI status i
	2303	# number of tentative
	2304	NB_ESSAI=3
	2305	# time delay between tentative
	2306	DELAI=2
	2307
	2308	i=0
	2309	while [ $i -lt $NB_ESSAI ] ; do
[1091]	2310	ncpdq -C $@ > /tmp/out_command.$$ 2>&1
[714]	2311	status=$?
	2312	if [ ${status} -gt 0 ] ; then
	2313	IGCM_debug_Print 2 "IGCM_sys_ncpdq : error code ${status}"
[938]	2314	cat /tmp/out_command.$$
	2315	\rm /tmp/out_command.$$
[714]	2316	IGCM_debug_Print 2 "IGCM_sys_ncpdq : ${i}/${NB_ESSAI} sleep ${DELAI} seconds and try again."
[800]	2317	sleep $DELAI
[714]	2318	else
[938]	2319	\rm /tmp/out_command.$$
[714]	2320	break
	2321	fi
	2322	(( i = i + 1 ))
	2323	done
	2324
	2325	if [ ${status} -gt 0 ] ; then
[811]	2326	echo "IGCM_sys_ncpdq : ncpdq error"
	2327	IGCM_debug_Exit "ncpdq"
[664]	2328	fi
[2]	2329
[664]	2330	IGCM_debug_PopStack "IGCM_sys_ncpdq"
[2]	2331	}
	2332
[811]	2333	#D-#==================================================
	2334	#D-function IGCM_sys_ncra
	2335	#D-* Purpose: encapsulate ncra call so as to manage error code and retry
	2336	#D-* Examples:
	2337	#D-
[2]	2338	function IGCM_sys_ncra {
[1057]	2339	IGCM_debug_PushStack "IGCM_sys_ncra" $@
[664]	2340	if ( $DEBUG_sys ) ; then
	2341	echo "IGCM_sys_ncra :" $@
	2342	fi
[714]	2343
	2344	typeset NB_ESSAI DELAI status i
	2345	# number of tentative
	2346	NB_ESSAI=3
	2347	# time delay between tentative
	2348	DELAI=2
	2349
	2350	i=0
	2351	while [ $i -lt $NB_ESSAI ] ; do
[1091]	2352	ncra -C $@ > /tmp/out_command.$$ 2>&1
[714]	2353	status=$?
	2354	if [ ${status} -gt 0 ] ; then
	2355	IGCM_debug_Print 2 "IGCM_sys_ncra : error code ${status}"
[938]	2356	cat /tmp/out_command.$$
	2357	\rm /tmp/out_command.$$
[714]	2358	IGCM_debug_Print 2 "IGCM_sys_ncra : ${i}/${NB_ESSAI} sleep ${DELAI} seconds and try again."
[800]	2359	sleep $DELAI
[714]	2360	else
[938]	2361	\rm /tmp/out_command.$$
[714]	2362	break
	2363	fi
	2364	(( i = i + 1 ))
	2365	done
	2366
	2367	if [ ${status} -gt 0 ] ; then
[811]	2368	echo "IGCM_sys_ncra : ncra error"
	2369	IGCM_debug_Exit "ncra"
[664]	2370	fi
[2]	2371
[664]	2372	IGCM_debug_PopStack "IGCM_sys_ncra"
[2]	2373	}
	2374
[811]	2375	#D-#==================================================
	2376	#D-function IGCM_sys_ncrcat
	2377	#D-* Purpose: encapsulate ncrcat call so as to manage error code and retry
	2378	#D-* Examples:
	2379	#D-
[2]	2380	function IGCM_sys_ncrcat {
[1057]	2381	IGCM_debug_PushStack "IGCM_sys_ncrcat" $@
[664]	2382	if ( $DEBUG_sys ) ; then
	2383	echo "IGCM_sys_ncrcat :" $@
	2384	fi
[714]	2385
[811]	2386	typeset NB_ESSAI DELAI status i lastArg
[714]	2387	# number of tentative
	2388	NB_ESSAI=3
	2389	# time delay between tentative
	2390	DELAI=2
	2391
	2392	i=0
	2393	while [ $i -lt $NB_ESSAI ] ; do
[1091]	2394	ncrcat -C $@ > /tmp/out_command.$$ 2>&1
[714]	2395	status=$?
	2396	if [ ${status} -gt 0 ] ; then
	2397	IGCM_debug_Print 2 "IGCM_sys_ncrcat : error code ${status}"
[938]	2398	cat /tmp/out_command.$$
	2399	\rm /tmp/out_command.$$
[714]	2400	IGCM_debug_Print 2 "IGCM_sys_ncrcat : ${i}/${NB_ESSAI} sleep ${DELAI} seconds and try again."
[800]	2401	sleep $DELAI
[938]	2402	elif [ ! "X$( grep "WARNING Intra-file non-monotonicity" /tmp/out_command.$$ )" = "X" ] ; then
[811]	2403	IGCM_debug_Print 2 "IGCM_sys_ncrcat : WARNING Intra-file non-monotonicity"
[938]	2404	cat /tmp/out_command.$$
[811]	2405	# remove files having corrupted time axis
	2406	eval lastArg=\${$#}
	2407	IGCM_debug_Print 2 "IGCM_sys_ncrcat : Delete ${lastArg}"
	2408	\rm ${lastArg}
[938]	2409	\rm /tmp/out_command.$$
[811]	2410	IGCM_debug_Print 2 "IGCM_sys_ncrcat : ${i}/${NB_ESSAI} sleep ${DELAI} seconds and try again."
	2411	sleep $DELAI
[714]	2412	else
[938]	2413	\rm /tmp/out_command.$$
[714]	2414	break
	2415	fi
	2416	(( i = i + 1 ))
	2417	done
	2418
	2419	if [ ${status} -gt 0 ] ; then
[811]	2420	echo "IGCM_sys_ncrcat : ncrcat error"
	2421	#IGCM_debug_Exit "ncrcat"
[664]	2422	fi
[2]	2423
[664]	2424	IGCM_debug_PopStack "IGCM_sys_ncrcat"
[2]	2425	}
	2426
[811]	2427	#D-#==================================================
	2428	#D-function IGCM_sys_ncrename
	2429	#D-* Purpose: encapsulate ncrename call so as to manage error code and retry
	2430	#D-* Examples:
	2431	#D-
[2]	2432	function IGCM_sys_ncrename {
[1057]	2433	IGCM_debug_PushStack "IGCM_sys_ncrename" $@
[664]	2434	if ( $DEBUG_sys ) ; then
	2435	echo "IGCM_sys_ncrename :" $@
	2436	fi
[714]	2437
	2438	typeset NB_ESSAI DELAI status i
	2439	# number of tentative
	2440	NB_ESSAI=3
	2441	# time delay between tentative
	2442	DELAI=2
	2443
	2444	i=0
	2445	while [ $i -lt $NB_ESSAI ] ; do
[938]	2446	ncrename $@ > /tmp/out_command.$$ 2>&1
[714]	2447	status=$?
	2448	if [ ${status} -gt 0 ] ; then
	2449	IGCM_debug_Print 2 "IGCM_sys_ncrename : error code ${status}"
[938]	2450	cat /tmp/out_command.$$
	2451	\rm /tmp/out_command.$$
[714]	2452	IGCM_debug_Print 2 "IGCM_sys_ncrename : ${i}/${NB_ESSAI} sleep ${DELAI} seconds and try again."
[800]	2453	sleep $DELAI
[714]	2454	else
[938]	2455	\rm /tmp/out_command.$$
[714]	2456	break
	2457	fi
	2458	(( i = i + 1 ))
	2459	done
	2460
	2461	if [ ${status} -gt 0 ] ; then
[811]	2462	echo "IGCM_sys_ncrename : ncrename error"
	2463	IGCM_debug_Exit "ncrename"
[664]	2464	fi
[2]	2465
[664]	2466	IGCM_debug_PopStack "IGCM_sys_ncrename"
[2]	2467	}
	2468
[811]	2469	#D-#==================================================
	2470	#D-function IGCM_sys_ncwa
	2471	#D-* Purpose: encapsulate ncwa call so as to manage error code and retry
	2472	#D-* Examples:
	2473	#D-
[2]	2474	function IGCM_sys_ncwa {
[1057]	2475	IGCM_debug_PushStack "IGCM_sys_ncwa" $@
[664]	2476	if ( $DEBUG_sys ) ; then
	2477	echo "IGCM_sys_ncwa :" $@
	2478	fi
[714]	2479
	2480	typeset NB_ESSAI DELAI status i
	2481	# number of tentative
	2482	NB_ESSAI=3
	2483	# time delay between tentative
	2484	DELAI=2
	2485
	2486	i=0
	2487	while [ $i -lt $NB_ESSAI ] ; do
[1091]	2488	ncwa -C $@ > /tmp/out_command.$$ 2>&1
[714]	2489	status=$?
	2490	if [ ${status} -gt 0 ] ; then
	2491	IGCM_debug_Print 2 "IGCM_sys_ncwa : error code ${status}"
[938]	2492	cat /tmp/out_command.$$
	2493	\rm /tmp/out_command.$$
[714]	2494	IGCM_debug_Print 2 "IGCM_sys_ncwa : ${i}/${NB_ESSAI} sleep ${DELAI} seconds and try again."
[800]	2495	sleep $DELAI
[714]	2496	else
[938]	2497	\rm /tmp/out_command.$$
[714]	2498	break
	2499	fi
	2500	(( i = i + 1 ))
	2501	done
	2502
	2503	if [ ${status} -gt 0 ] ; then
[811]	2504	echo "IGCM_sys_ncwa : ncwa error"
	2505	IGCM_debug_Exit "ncwa"
[664]	2506	fi
[2]	2507
[664]	2508	IGCM_debug_PopStack "IGCM_sys_ncwa"
[2]	2509	}
	2510
[343]	2511	##############################################################
	2512	# CDO OPERATOR
	2513
[811]	2514	#D-#==================================================
	2515	#D-function IGCM_sys_cdo
	2516	#D-* Purpose: encapsulate cdo call so as to manage error code and retry
	2517	#D-* Examples:
	2518	#D-
[343]	2519	function IGCM_sys_cdo {
[1057]	2520	IGCM_debug_PushStack "IGCM_sys_cdo" $@
[714]	2521	if ( $DEBUG_sys ) ; then
	2522	echo "IGCM_sys_cdo :" $@
	2523	fi
[343]	2524
[714]	2525	typeset status
	2526
[938]	2527	\cdo $@ > /tmp/out_command.$$ 2>&1
[714]	2528	status=$?
	2529	if [ ${status} -gt 0 ] ; then
	2530	echo "IGCM_sys_cdo : error code ${status}"
[938]	2531	cat /tmp/out_command.$$
	2532	\rm /tmp/out_command.$$
[343]	2533	IGCM_debug_PopStack "IGCM_sys_cdo"
[664]	2534	return 1
	2535	else
	2536	IGCM_debug_PopStack "IGCM_sys_cdo"
	2537	return 0
	2538	fi
[343]	2539
[664]	2540	IGCM_debug_PopStack "IGCM_sys_cdo"
[2]	2541	}

Note: See TracBrowser for help on using the repository browser.

Download in other formats: