Context Navigation

libIGCM_sys_irene.ksh @ 1624

Last change on this file since 1624 was 1624, checked in by jgipsl, 5 months ago

Added possibility to declare a restart file as optional which means that libIGCM want stop if the file is not found. Only a warning will be written.

For example, in the following case, if the routing_restart file is not available when starting the model or if it is not produced, libIGCM will not stop. But if the file sechiba_rest is not available, libIGCM will stop as usual.

[RestartFiles]
List=   (sechiba_rest_out.nc, sechiba_rest.nc, sechiba_rest_in.nc) ,\
        (routing_restart.nc, routing_restart.nc, routing_start.nc, OPTIONAL)

Read optional argument OPTIONAL on the line where the restart file is declared.
Each time when a restart file is not found for coping in or out, if the argument OPTIONAL is set, then just continue. If not set, the file is considered as MANDATORY and libIGCM will exist if it is missing as before.
on irene-amd and jean-zay, corrected a bug for the case in config.card DataProject?=DEFAULT

Property licence set to
The following licence information concerns ONLY the libIGCM tools
==================================================================

Copyright © Centre National de la Recherche Scientifique CNRS
Commissariat à l'Énergie Atomique CEA

libIGCM : Library for Portable Models Computation of IGCM Group.

IGCM Group is the french IPSL Global Climate Model Group.

This library is a set of shell scripts and functions whose purpose is
the management of the initialization, the launch, the transfer of
output files, the post-processing and the monitoring of datas produce
by any numerical program on any plateforme.

This software is governed by the CeCILL license under French law and
abiding by the rules of distribution of free software. You can use,
modify and/ or redistribute the software under the terms of the CeCILL
license as circulated by CEA, CNRS and INRIA at the following URL
"http://www.cecill.info".

As a counterpart to the access to the source code and rights to copy,
modify and redistribute granted by the license, users are provided only
with a limited warranty and the software's author, the holder of the
economic rights, and the successive licensors have only limited
liability.

In this respect, the user's attention is drawn to the risks associated
with loading, using, modifying and/or developing or reproducing the
software by the user in light of its specific status of free software,
that may mean that it is complicated to manipulate, and that also
therefore means that it is reserved for developers and experienced
professionals having in-depth computer knowledge. Users are therefore
encouraged to load and test the software's suitability as regards their
requirements in conditions enabling the security of their systems and/or
data to be ensured and, more generally, to use and operate it in the
same conditions as regards security.

The fact that you are presently reading this means that you have had
knowledge of the CeCILL license and that you accept its terms.
Property svn:keywords set to Revision Author Date

File size: 64.3 KB

Rev	Line
[622]	1	#!/bin/ksh
	2
	3	#**************************************************************
	4	# Author: Sebastien Denvil, Martial Mancip, Arnaud Caubel
	5	# Contact: Arnaud.Caubel__at__lsce.ipsl.fr
	6	# $Revision:: $ Revision of last commit
	7	# $Author:: $ Author of last commit
	8	# $Date:: $ Date of last commit
	9	# IPSL (2006)
	10	# This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
	11	#
	12	#**************************************************************
	13
	14	#=========================================================
	15	# The documentation of this file can be automatically generated
[623]	16	# if you use the prefix #D- for comments to be extracted.
[622]	17	# Extract with command: cat lib* \| grep "^#D-" \| cut -c "4-"
	18	#=========================================================
	19
	20	#D-#==================================================
[1457]	21	#D-LibIGCM_sys for Irene
[622]	22	#D-#==================================================
	23	#D-
[623]	24	#D- This ksh library if a layer under some usefull
[622]	25	#D-environment variables and shell commands.
	26	#D-All those definitions depend on host particularities.
	27	#D-It manages a stack mechanism and test validity of operations.
	28	#D-All function described bellow must be prefixed by IGCM_sys.
	29
	30	#====================================================
	31	# libIGCM_sys PARAMETERS
	32	#====================================================
	33
	34	#====================================================
	35	# set DEBUG_sys to true to output calls of function
	36	typeset -r DEBUG_sys=${DEBUG_sys:=true}
	37
	38	#====================================================
	39	# Turn in dry run mode ? (sys_Put_Rest, sys_Put_Out, sys_Get)
[1569]	40	if ([ X${DRYRUN} != X ] && [ ${DRYRUN} = 4 ]) ; then
	41	typeset -r DRYRUN=0
	42	typeset -r DRYRUN_DEBUG=4
	43	else
	44	typeset -r DRYRUN=${DRYRUN:=0}
	45	typeset -r DRYRUN_DEBUG=0
	46	fi
[622]	47
[1569]	48
[622]	49	# YOU MUST COMPILE YOUR EXE FILES FOR DRYRUN MODE !
	50	# -------------------------------------------------------------------------------------
	51	# \| DRYRUN= \| Date computations, \| sys_Get \| Exe \| sys_Put_Out; sys_Put_Rest \|
[809]	52	# \| \| Cp/Exe/param/files \| \| \| \|
	53	# \| \| Chmod Qsub \| \| \| \|
[622]	54	# -------------------------------------------------------------------------------------
	55	# \| 0 \| yes \| yes \| yes \| yes \|
	56	# -------------------------------------------------------------------------------------
	57	# \| 1 \| yes \| yes \| yes \| no \|
	58	# -------------------------------------------------------------------------------------
	59	# \| 2 \| yes \| yes \| no \| no \|
	60	# -------------------------------------------------------------------------------------
	61	# \| 3 \| yes \| no \| no \| no \|
	62	# -------------------------------------------------------------------------------------
[1569]	63	# \| 4 \| yes \| yes \| creation of RUN_DIR and Job_debug \|
	64	# -------------------------------------------------------------------------------------
[622]	65
[1569]	66
[622]	67	#=====================================================
	68	# Global Variables :
	69	#=====================================================
	70	# Language : "fr" or "en"
	71	typeset -r MYLANG="fr"
	72
	73	#=====================================================
[1201]	74	# Host user names project maxCpuTime
[622]	75	# $hostname ou hostname
[1158]	76	typeset HOST=${HOST:=$( hostname )}
[622]	77	# $username ou whoami
[1158]	78	typeset LOGIN=${LOGIN:=$( whoami )}
[622]	79	# $hostname of the MASTER job
[1441]	80	typeset MASTER=irene
[1457]	81	# add default project on irene
[1473]	82	typeset PROJECT=$(echo ${BRIDGE_MSUB_PROJECT:=::default_project::} \| cut -d@ -f1 )
[1201]	83	# jobWarningDelay in seconds
	84	typeset jobWarningDelay=${BRIDGE_MSUB_MAXTIME}
[622]	85
	86	#D-
	87	#D-#==================================================
	88	#D-Program used in libIGCM
	89	#D-#==================================================
	90
[742]	91	# Submit command
[747]	92	typeset SUBMIT=${SUBMIT:=ccc_msub}
[622]	93	# rsync with path
	94	typeset -r RSYNC=/usr/bin/rsync
	95	# RSYNC_opt args to rsync
	96	typeset -r RSYNC_opt="-va"
	97	# ie storage filesystem
[623]	98	typeset -r STOREHOST=${MASTER}
[1180]	99	typeset -r REMOTE_RSYNC=/usr/bin/rsync
[622]	100
	101	#====================================================
[1180]	102	# Set environment tools (ferret, nco, cdo, rebuild, ...)
[622]	103	#====================================================
[1579]	104	typeset CCCOS=${CCCOS:=$( ccc_os )}
	105
[996]	106	if [ X${TaskType} = Xcomputing ] ; then
[1485]	107	IGCM_debug_Print 1 "Modules will be loaded later in IGCM_sys_activ_variables."
[996]	108	else
[1413]	109	module purge > /dev/null 2>&1
[1579]	110	if [ X${CCCOS} = X"Rhel_8__x86_64" ] ; then
[1581]	111	. $( ccc_home -u igcmg)/MachineEnvironment/irene/env_atlas_irene > /dev/null 2>&1
[1579]	112	else
[1581]	113	. $( ccc_home -u igcmg)/MachineEnvironment/irene_prev/env_irene > /dev/null 2>&1
	114	. $( ccc_home -u igcmg)/MachineEnvironment/irene_prev/env_atlas_irene > /dev/null 2>&1
[1579]	115	fi
[1433]	116
[1358]	117	export PCMDI_MP=/ccc/work/cont003/igcmg/igcmg/PCMDI-MP
[1364]	118	export UVCDAT_ANONYMOUS_LOG=FALSE
[996]	119	fi
[1315]	120
	121	# FYI
[1210]	122	[ ! X${TaskType} = Xchecking ] && IGCM_debug_Print 1 "List of loaded modules:"
	123	[ ! X${TaskType} = Xchecking ] && module list
[1208]	124
[1457]	125	# For AddNoise and AddPertu3DOCE
	126	##export PATH=${PATH}:/ccc/cont003/home/igcmg/igcmg/Tools/curie/bin
[1462]	127	export PATH=${PATH}:$(ccc_home -u igcmg)/Tools/irene/bin
[622]	128
[1457]	129	# For rebuild
	130	##export PATH=${PATH}:/ccc/cont003/home/igcmg/igcmg/Tools/irene/rebuild/modipsl_IOIPSL_PLUS_v2_2_4/bin
	131	export PATH=${PATH}:$(ccc_home -u igcmg)/Tools/irene/rebuild/modipsl_IOIPSL_PLUS_v2_2_4/bin
[622]	132
	133	#====================================================
	134	# Host specific DIRECTORIES
	135	#====================================================
	136
[623]	137	# ============ CESIUM START ============ #
	138
[622]	139	#====================================================
[623]	140	#- Mirror libIGCM from titane to cesium if needed
	141	#ROOTSYS=$( echo ${libIGCM} \| gawk -F"/" '{print $3}' )
	142	#if [ ! ${ROOTSYS} = "home" ] ; then
	143	# typeset -r MirrorlibIGCM=${MirrorlibIGCM:=true}
	144	#else
	145	# typeset -r MirrorlibIGCM=${MirrorlibIGCM:=false}
	146	#fi
	147
	148	#====================================================
	149	#- libIGCM_POST
	150	#if ( ${MirrorlibIGCM} ) ; then
	151	# PATHlibIGCM=$( echo ${libIGCM} \| gawk -F"${LOGIN}/" '{print $2}' \| sed -e "s&/libIGCM&&" )
	152	# typeset -r libIGCM_POST=${HOME}/MIRROR/${PATHlibIGCM}/libIGCM
	153	#else
	154	# typeset -r libIGCM_POST=${libIGCM}
	155	#fi
	156
	157	# ============ CESIUM END ============ #
	158
	159	#====================================================
	160	#- MirrorlibIGCM for frontend
	161	typeset -r MirrorlibIGCM=${MirrorlibIGCM:=false}
	162
	163	#====================================================
	164	#- libIGCM_POST for frontend
	165	typeset -r libIGCM_POST=${libIGCM}
	166
	167	#====================================================
[622]	168	#- R_EXE (==> BIN_DIR = ${MODIPSL}/bin )
	169	typeset -r R_EXE="${MODIPSL}/bin"
	170
	171	#====================================================
	172	#- SUBMIT_DIR : submission dir
[1010]	173	typeset -x SUBMIT_DIR=${SUBMIT_DIR:=${BRIDGE_MSUB_PWD}}
[622]	174
	175	#====================================================
	176	#- IN
[1276]	177	typeset -r R_IN=${R_IN:=/ccc/work/cont003/igcmg/igcmg/IGCM}
[622]	178
	179	#====================================================
	180	#- RUN_DIR_PATH : Temporary working directory (=> TMP)
[1476]	181	typeset -r RUN_DIR_PATH=${RUN_DIR_PATH:=${CCCSCRATCHDIR}/RUN_DIR/${BRIDGE_MSUB_JOBID}_${$}}
[622]	182
	183	#====================================================
[1179]	184	#- OUTCOMMAND_PATH : tmp place to store command lines standard error and outputs
[1186]	185	typeset -r OUTCOMMAND_PATH=/tmp
[1179]	186
	187	#====================================================
[622]	188	#- HOST_MPIRUN_COMMAND
[1382]	189	typeset -r HOST_MPIRUN_COMMAND=${HOST_MPIRUN_COMMAND:="/usr/bin/time ccc_mprun -E-K1 "}
[622]	190
	191	#====================================================
	192	#- Max number of arguments passed to nco operator or demigration command
	193	UNIX_MAX_LIMIT=360
	194
[623]	195	#====================================================
[1457]	196	#- set PackDefault to true on irene
[623]	197	PackDefault=true
	198
[664]	199	#====================================================
[623]	200	#- Default number of MPI task for IPSL coupled model
	201	#- required for backward compatibility
	202	#-
	203	DEFAULT_NUM_PROC_OCE=5
	204	DEFAULT_NUM_PROC_CPL=1
[637]	205	(( DEFAULT_NUM_PROC_ATM = BATCH_NUM_PROC_TOT - DEFAULT_NUM_PROC_OCE - DEFAULT_NUM_PROC_CPL ))
[623]	206
[1426]	207	#====================================================
	208	#- Number of computing cores per node
	209	#-
[1477]	210	NB_CORE_PER_NODE=48
[1426]	211
[725]	212	#D-#==================================================
[1057]	213	#D-function IGCM_sys_defineArchives
	214	#D-* Purpose:
[1461]	215	#D-* Load dfldatadir module : According to project used for submission (default) or set in config.card (optional)
[1057]	216	#D-* Define ARCHIVE : Dedicated to large files
	217	#D-* Define STORAGE : Dedicated to small/medium files
	218	#D-* Define R_OUT : Output tree located on ARCHIVE
	219	#D-* Define R_FIG : Output tree located on STORAGE hosting figures (monitoring and atlas, and/or small files)
[1432]	220	#D-* Define R_BUF : Output tree located on CCCSCRATCHDIR hosting files waiting for rebuild or pack processes
	221	#D-* if SpaceName=TEST everything is stored on CCCSCRATCHDIR
[725]	222	#D-* Examples:
	223	#D-
[1057]	224	function IGCM_sys_defineArchives {
	225	IGCM_debug_PushStack "IGCM_sys_defineArchives"
[676]	226
[1461]	227	# Load dfldatadir depending on the project used for submission (default) or set in config.card (optional)
[1598]	228	if [ X${config_UserChoices_DataProject} = X ] \|\| [ X${config_UserChoices_DataProject} = XDEFAULT ]; then
[1461]	229	# Default option: Change the dfldatadir according to the project used for submission
	230	# The variable DataProject in section UserChoices in config.card is not set or is set to DEFAULT
[1474]	231	if [ X"$( echo ${PROJECT} \| grep cmip6 )" != "X" ] ; then
	232	module switch dfldatadir dfldatadir/gencmip6
	233	IGCM_debug_Print 1 "Load dfldatadir for forced gencmip6 project"
	234	else
	235	module switch dfldatadir dfldatadir/${PROJECT}
	236	IGCM_debug_Print 1 "Load dfldatadir for project ${PROJECT}"
	237	fi
[1461]	238	else
	239	# Use the project set by the variable DataProject in section UserChoices in config.card
	240	module switch dfldatadir dfldatadir/${config_UserChoices_DataProject}
[1598]	241	IGCM_debug_Print 1 "Load dfldatadir for DataProject read in config.card : ${config_UserChoices_DataProject}"
[1461]	242	fi
	243
[1474]	244	#====================================================
	245	#- RUN_DIR_PATH : Temporary working directory (=> TMP)
	246	typeset -r RUN_DIR_PATH=${RUN_DIR_PATH:=${CCCSCRATCHDIR}/RUN_DIR/${BRIDGE_MSUB_JOBID}_${$}}
	247
[1060]	248	if [ ! X${config_UserChoices_ARCHIVE} = X ]; then
[1057]	249	#====================================================
	250	#- ARCHIVE (dedicated to large files)
[1063]	251	ARCHIVE=${config_UserChoices_ARCHIVE}
[1057]	252	else
	253	#====================================================
	254	#- ARCHIVE (dedicated to large files)
[1063]	255	ARCHIVE=${CCCSTOREDIR}
[1057]	256	fi
[725]	257
[1060]	258	if [ ! X${config_UserChoices_STORAGE} = X ]; then
[1057]	259	#====================================================
	260	#- STORAGE (dedicated to small/medium files)
[1063]	261	STORAGE=${config_UserChoices_STORAGE}
[1057]	262	else
	263	#====================================================
	264	#- STORAGE (dedicated to small/medium files)
[1063]	265	STORAGE=${CCCWORKDIR}
[1057]	266	fi
[725]	267
[1057]	268	if [ X${config_UserChoices_SpaceName} = XTEST ]; then
	269	#====================================================
	270	#- R_OUT
[1512]	271	R_OUT=${CCCSCRATCHDIR}/IGCM_OUT
[1057]	272
	273	#====================================================
	274	#- R_FIG (hosting figures : monitoring and atlas, and/or small files)
[1512]	275	R_FIG=${CCCSCRATCHDIR}/IGCM_OUT
[1057]	276
	277	IGCM_debug_Print 1 "SpaceName=TEST ==> OVERRULE destination path directories"
	278
	279	else
	280	#====================================================
	281	#- R_OUT
[1063]	282	R_OUT=${ARCHIVE}/IGCM_OUT
[1057]	283
	284	#====================================================
	285	#- R_FIG (hosting figures : monitoring and atlas, and/or small files)
[1063]	286	R_FIG=${STORAGE}/IGCM_OUT
[1057]	287	fi
	288
	289	#====================================================
[1431]	290	#- CMIP6 (hosting CMIP6 files produced by XIOS2 and configured by dr2xml)
	291	CMIP6_BUF=${STORAGE}/IGCM_OUT
	292
	293	#====================================================
[1057]	294	#- R_BUF (ONLY FOR double copy an scratch)
[1512]	295	R_BUF=${CCCSCRATCHDIR}/IGCM_OUT
[1057]	296
	297	IGCM_debug_Print 1 "R_OUT has been defined = ${R_OUT}"
	298	IGCM_debug_Print 1 "R_BUF has been defined = ${R_BUF}"
	299	IGCM_debug_Print 1 "R_FIG has been defined = ${R_FIG}"
	300
	301	IGCM_debug_PopStack "IGCM_sys_defineArchives"
[725]	302	}
	303
[622]	304	#D-#==================================================
	305	#D-function IGCM_sys_RshArchive
	306	#D-* Purpose: Archive rsh command
	307	#D-* Examples:
	308	#D-
	309	function IGCM_sys_RshArchive {
[623]	310	IGCM_debug_PushStack "IGCM_sys_RshArchive" $@
	311	/bin/ksh <<-EOF
[622]	312	${@}
	313	EOF
[811]	314	status=$?
	315	if [ ${status} -gt 0 ] ; then
	316	IGCM_debug_Print 2 "IGCM_sys_RshArchive : command failed error code ${status}"
[623]	317	IGCM_debug_Exit "IGCM_sys_RshArchive"
	318	fi
	319	IGCM_debug_PopStack "IGCM_sys_RshArchive"
[622]	320	}
	321
	322	#D-#==================================================
[845]	323	#D-function IGCM_sys_RshArchive_NoError
	324	#D-* Purpose: Archive rsh command, without error
	325	#D-* used only in monitoring.job
	326	#D-* Examples:
	327	#D-
	328	function IGCM_sys_RshArchive_NoError {
	329	IGCM_debug_PushStack "IGCM_sys_RshArchive_NoError" $@
	330	/bin/ksh <<-EOF
[1181]	331	${@} 2> /dev/null
[845]	332	EOF
	333	IGCM_debug_PopStack "IGCM_sys_RshArchive_NoError"
	334	}
	335
	336	#D-#==================================================
[622]	337	#D-function IGCM_sys_MkdirArchive
	338	#D-* Purpose: Mkdir on Archive
	339	#D-* Examples:
	340	#D-
	341	function IGCM_sys_MkdirArchive {
[623]	342	IGCM_debug_PushStack "IGCM_sys_MkdirArchive" $@
	343	if ( $DEBUG_sys ) ; then
	344	echo "IGCM_sys_MkdirArchive :" $@
	345	fi
	346	#- creation de repertoire sur le serveur fichier
	347	if [ ! -d ${1} ]; then
	348	\mkdir -p $1
[811]	349	status=$?
	350
	351	if [ ${status} -gt 0 ] ; then
	352	IGCM_debug_Print 2 "IGCM_sys_MkdirArchive : mkdir failed error code ${status}"
[623]	353	IGCM_debug_Exit "IGCM_sys_MkdirArchive"
[622]	354	fi
[623]	355	fi
	356	IGCM_debug_PopStack "IGCM_sys_MkdirArchive"
[622]	357	}
	358
	359	#D-#==================================================
	360	#D-function IGCM_sys_TestDirArchive
	361	#D-* Purpose: Test Directory that must exists on Archive
	362	#D-* Examples:
	363	#D-
	364	function IGCM_sys_TestDirArchive {
[623]	365	IGCM_debug_PushStack "IGCM_sys_TestDirArchive" $@
	366	if ( $DEBUG_sys ) ; then
	367	echo "IGCM_sys_TestDirArchive :" $@
	368	fi
	369	typeset ExistFlag
	370	ExistFlag=$( [ -d $1 ] && echo 0 \|\| echo 1 )
	371	IGCM_debug_PopStack "IGCM_sys_TestDirArchive"
	372	return ${ExistFlag}
[622]	373	}
	374
	375	#D-#==================================================
[623]	376	#D-function IGCM_sys_IsFileArchived
[1179]	377	#D-* Purpose: Test file that must NOT EXISTS on Archive based on filename only
[623]	378	#D-* Examples:
	379	#D-
	380	function IGCM_sys_IsFileArchived {
	381	IGCM_debug_PushStack "IGCM_sys_IsFileArchived" $@
	382	if ( $DEBUG_sys ) ; then
	383	echo "IGCM_sys_IsFileArchived :" $@
	384	fi
	385	typeset IsArchivedFlag
[818]	386	IsArchivedFlag=$( [ X$( echo $@ \| grep ^\/ccc\/store ) != X ] && echo 0 \|\| echo 1 )
[623]	387	IGCM_debug_PopStack "IGCM_sys_IsFileArchived"
	388
	389	return ${IsArchivedFlag}
	390	}
	391
	392	#D-#==================================================
[622]	393	#D-function IGCM_sys_TestFileArchive
	394	#D-* Purpose: Test file that must NOT EXISTS on Archive
	395	#D-* Examples:
	396	#D-
	397	function IGCM_sys_TestFileArchive {
[623]	398	IGCM_debug_PushStack "IGCM_sys_TestFileArchive" $@
	399	typeset ExistFlag
	400	ExistFlag=$( [ -f $1 ] && echo 0 \|\| echo 1 )
	401	IGCM_debug_PopStack "IGCM_sys_TestFileArchive"
[622]	402
[623]	403	return ${ExistFlag}
[622]	404	}
	405
	406	#D-#==================================================
	407	#D-function IGCM_sys_CountFileArchive
	408	#D-* Purpose: Count files on Archive filesystem
	409	#D-* Examples:
	410	#D-
	411	function IGCM_sys_CountFileArchive {
[623]	412	IGCM_debug_PushStack "IGCM_sys_CountFileArchive" $@
	413	ls ${@} 2>/dev/null \| wc -l
	414	if [ $? -gt 0 ] ; then
	415	echo "IGCM_sys_CountFileArchive : erreur."
	416	fi
	417	IGCM_debug_PopStack "IGCM_sys_CountFileArchive"
[622]	418	}
	419
	420	#D-#==================================================
	421	#D-function IGCM_sys_Tree
	422	#D-* Purpose: Tree directories with files on ${ARCHIVE}
	423	#D-* Examples: IGCM_sys_Tree ${R_IN} ${R_OUT}
	424	#D-
	425	function IGCM_sys_Tree {
[623]	426	IGCM_debug_PushStack "IGCM_sys_Tree" $@
	427	if ( $DEBUG_sys ) ; then
	428	echo "IGCM_sys_Tree :" $@
	429	fi
[622]	430
[623]	431	\ls -lR ${@}
[622]	432
[623]	433	IGCM_debug_PopStack "IGCM_sys_Tree"
[622]	434	}
	435
	436	#D-#==================================================
	437	#D-function IGCM_sys_Qsub
	438	#D-* Purpose: Qsub new job
	439	#D-* Examples:
	440	#D-
	441	function IGCM_sys_Qsub {
[623]	442	IGCM_debug_PushStack "IGCM_sys_Qsub" $@
	443	if ( $DEBUG_sys ) ; then
	444	echo "IGCM_sys_Qsub :" $@
	445	fi
[811]	446	typeset options status
[923]	447	options="-o ${SUBMIT_DIR}/${Script_Output} -e ${SUBMIT_DIR}/${Script_Output}"
[622]	448
[1179]	449	/usr/bin/ccc_msub ${options} $1 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[811]	450	status=$?
[948]	451
[1179]	452	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[811]	453	if [ ${status} -gt 0 ] ; then
	454	IGCM_debug_Print 2 "IGCM_sys_Qsub ${options} $1 : error code ${status}"
[623]	455	IGCM_debug_Exit "IGCM_sys_Qsub"
[948]	456	else
[1179]	457	JobID=$( gawk {'print $4'} ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ )
[623]	458	fi
	459	IGCM_debug_PopStack "IGCM_sys_Qsub"
[622]	460	}
	461
	462	#D-#==================================================
	463	#D-function IGCM_sys_QsubPost
	464	#D-* Purpose: Qsub new job on scalaire
	465	#D-* Examples:
	466	#D-
	467	function IGCM_sys_QsubPost {
[623]	468	IGCM_debug_PushStack "IGCM_sys_QsubPost" $@
	469	if ( $DEBUG_sys ) ; then
	470	echo "IGCM_sys_QsubPost :" $@
	471	fi
[923]	472	typeset options status
[1468]	473	options="-Q normal -o ${POST_DIR}/${Script_Post_Output}.out -e ${POST_DIR}/${Script_Post_Output}.out"
[923]	474
[1179]	475	/usr/bin/ccc_msub ${options} ${libIGCM_POST}/$1.job > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[811]	476	status=$?
[948]	477
[1179]	478	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[811]	479	if [ ${status} -gt 0 ] ; then
[923]	480	IGCM_debug_Print 2 "IGCM_sys_QsubPost ${options} ${libIGCM_POST}/$1.job : error code ${status}"
[623]	481	IGCM_debug_Exit "IGCM_sys_QsubPost"
[948]	482	else
[1179]	483	JobID=$( gawk {'print $4'} ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ )
[623]	484	fi
	485	IGCM_debug_PopStack "IGCM_sys_QsubPost"
[622]	486	}
	487
	488	#D-*************************
[623]	489	#D- File transfer functions
[622]	490	#D-*************************
	491	#D-
	492
	493	#D-#==================================================
	494	#D-function IGCM_sys_RmRunDir
	495	#D-* Purpose: rm tmpdir (dummy function most of the time batch
	496	#D- scheduler will do the job)
	497	#D-* Examples:
	498	#D-
	499	function IGCM_sys_RmRunDir {
[1057]	500	IGCM_debug_PushStack "IGCM_sys_RmRunDir" $@
[623]	501	if ( $DEBUG_sys ) ; then
	502	echo "IGCM_sys_RmRunDir :" $@
	503	fi
[622]	504
[715]	505	typeset status
[622]	506
[1179]	507	echo rm $@ > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
	508	\rm $@ >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[715]	509	status=$?
[622]	510
[715]	511	if [ ${status} -gt 0 ] ; then
[1183]	512	IGCM_debug_Print 1 "IGCM_sys_RmRunDir : rm error code is ${status}."
[1179]	513	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[623]	514	IGCM_debug_Exit "IGCM_sys_RmRunDir"
	515	else
[1179]	516	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[623]	517	fi
	518	IGCM_debug_PopStack "IGCM_sys_RmRunDir"
[622]	519	}
	520
	521	#D-#==================================================
	522	#D-function IGCM_sys_Put_Dir
	523	#D-* Purpose: Copy a complete directory on $(ARCHIVE)
	524	#D-* Examples:
	525	#D-
	526	function IGCM_sys_Put_Dir {
[623]	527	IGCM_debug_PushStack "IGCM_sys_Put_Dir" $@
	528	if ( $DEBUG_sys ) ; then
	529	echo "IGCM_sys_Put_Dir :" $@
	530	fi
	531	if [ $DRYRUN = 0 ]; then
	532	if [ ! -d ${1} ] ; then
	533	echo "WARNING : IGCM_sys_Put_Dir ${1} DOES NOT EXIST ."
	534	IGCM_debug_PopStack "IGCM_sys_Put_Dir"
	535	return
[622]	536	fi
	537
[715]	538	typeset status
[622]	539
[623]	540	# Only if we use rsync
	541	#IGCM_sys_TestDirArchive $( dirname $2 )
	542	#
	543	#USUAL WAY
[1179]	544	\cp -r $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[715]	545	status=$?
[622]	546
[715]	547	if [ ${status} -gt 0 ] ; then
[811]	548	IGCM_debug_Print 2 "IGCM_sys_Put_Dir : cp failed error code ${status}"
[1179]	549	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[623]	550	IGCM_debug_Exit "IGCM_sys_Put_Dir"
[622]	551	else
[1179]	552	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[622]	553	fi
[623]	554	fi
	555	IGCM_debug_PopStack "IGCM_sys_Put_Dir"
[622]	556	}
	557
	558	#D-#==================================================
	559	#D-function IGCM_sys_Get_Dir
[623]	560	#D-* Purpose: Copy a complete directory from ${ARCHIVE}
[622]	561	#D-* Examples:
	562	#D-
	563	function IGCM_sys_Get_Dir {
[623]	564	IGCM_debug_PushStack "IGCM_sys_Get_Dir" $@
	565	if ( $DEBUG_sys ) ; then
	566	echo "IGCM_sys_Get_Dir :" $@
	567	fi
	568	if [ $DRYRUN = 0 ]; then
[708]	569	typeset NB_ESSAI DELAI status i
	570	# number of tentative
	571	NB_ESSAI=3
	572	# time delay between tentative
	573	DELAI=2
[622]	574
[676]	575	#
[708]	576	# USUAL WAY
	577	# add 'ccc_hsm get' (to demigrate all offline files) to reduce time of this command :
[713]	578	ccc_hsm get -r $1
[622]	579
[709]	580	i=0
[708]	581	while [ $i -lt $NB_ESSAI ] ; do
[1179]	582	\cp -ur $1 $2 >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[708]	583	status=$?
[811]	584	if [ ${status} -gt 0 ] ; then
[713]	585	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : cp failed error code ${status} ${i}/${NB_ESSAI}"
[708]	586	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : sleep ${DELAI} seconds and try again."
	587	sleep $DELAI
	588	else
	589	break
	590	fi
	591	(( i = i + 1 ))
	592	done
	593
	594	if [ ${status} -gt 0 ] ; then
[1178]	595	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : cp failed error code ${status}"
[1179]	596	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[623]	597	IGCM_debug_Exit "IGCM_sys_Get_Dir"
[622]	598	else
[1179]	599	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[622]	600	fi
[623]	601	fi
	602	IGCM_debug_PopStack "IGCM_sys_Get_Dir"
[622]	603	}
	604
	605	#D-#==================================================
	606	#D-function IGCM_sys_Put_Rest
[623]	607	#D-* Purpose: Put computied restarts on ${ARCHIVE}.
[622]	608	#D- File and target directory must exist.
	609	#D-* Examples:
	610	#D-
	611	function IGCM_sys_Put_Rest {
[623]	612	IGCM_debug_PushStack "IGCM_sys_Put_Rest" $@
	613	if ( $DEBUG_sys ) ; then
	614	echo "IGCM_sys_Put_Rest :" $@
	615	fi
	616	if [ $DRYRUN = 0 ]; then
	617	if [ ! -f ${1} ] ; then
[1624]	618	# The file is missing
	619	if [ X${3} = X'OPTIONAL' ] ; then
	620	echo "WARINING : IGCM_sys_Put_Rest ${1} DOES NOT EXIST ."
	621	IGCM_debug_Print 1 "IGCM_sys_Put_Rest: Optional restart file ${1} is missing, now continue for the next"
	622	else
	623	echo "ERROR : IGCM_sys_Put_Rest ${1} DOES NOT EXIST ."
	624	IGCM_debug_Exit "IGCM_sys_Put_Rest"
	625	fi
	626
	627	else
	628	# The file exist
[622]	629
[715]	630	typeset status
[623]	631	#
	632	# USUAL WAY
[1179]	633	\cp $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[715]	634	status=$?
[622]	635
[623]	636	# #RSYNC WITH NETWORK SSH CALL
[1179]	637	# echo ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
	638	# ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[622]	639
[623]	640	# #RSYNC WITH NFS USE
[1179]	641	# echo ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
	642	# ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[622]	643
[715]	644	# status=$?
	645	# IGCM_sys_Rsync_out $status
[622]	646
[1179]	647	# ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[715]	648	# (( status=status+$? ))
[623]	649
[715]	650	if [ ${status} -gt 0 ] ; then
[1179]	651	IGCM_debug_Print 2 "IGCM_sys_Put_Rest : cp failed error code ${status}"
[765]	652	[ -f ${1} ] && ls -l ${1}
[759]	653	[ -f ${2} ] && ls -l ${2}
	654	[ -f ${2}/${1} ] && ls -l ${2}/${1}
[1179]	655	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[623]	656	IGCM_debug_Exit "IGCM_sys_Put_Rest"
[622]	657	else
[759]	658
	659	if [ X${JobType} = XRUN ] ; then
	660	[ -f ${2} ] && IGCM_sys_Chmod 444 ${2}
	661	[ -f ${2}/${1} ] && IGCM_sys_Chmod 444 ${2}/${1}
	662	fi
	663
[1179]	664	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[622]	665	fi
[1624]	666
	667	fi
	668
[623]	669	fi
	670	IGCM_debug_PopStack "IGCM_sys_Put_Rest"
[622]	671	}
	672
	673	#D-#==================================================
	674	#D-function IGCM_sys_Put_Out
[623]	675	#D-* Purpose: Copy a file on ${ARCHIVE} after having chmod it in readonly
[622]	676	#D-* Examples:
	677	#D-
	678	function IGCM_sys_Put_Out {
[623]	679	IGCM_debug_PushStack "IGCM_sys_Put_Out" $@
	680	if ( $DEBUG_sys ) ; then
	681	echo "IGCM_sys_Put_Out :" $@
	682	fi
[667]	683
[711]	684	typeset NB_ESSAI DELAI status i exist skip
[1234]	685	typeset fileDeviceNumberInHex directoryDeviceNumberInHex
[667]	686
	687	# number of tentative
[668]	688	NB_ESSAI=3
[667]	689	# time delay between tentative
[668]	690	DELAI=2
[667]	691
[623]	692	if [ $DRYRUN = 0 ]; then
	693	if [ ! -f ${1} ] ; then
	694	echo "WARNING : IGCM_sys_Put_Out ${1} DOES NOT EXIST ."
	695	IGCM_debug_PopStack "IGCM_sys_Put_Out"
	696	return 1
[622]	697	fi
[623]	698	#
	699	IGCM_sys_MkdirArchive $( dirname $2 )
	700	#
[711]	701	exist=false
	702	skip=false
	703	if [ -f $2 ] ; then
	704	IGCM_debug_Print 1 "$2 already exist"
	705	ccc_hsm get $2
	706	exist=true
	707	if [ "X$( diff $1 $2 )" = X ] ; then
	708	IGCM_debug_Print 2 "$1 and $2 are the same file, we skip the copy"
	709	skip=true
[667]	710	else
[711]	711	IGCM_debug_Print 2 "$1 and $2 are not the same file, we force the copy"
	712	skip=false
[667]	713	fi
[711]	714	fi
	715	#
	716	if ( [ X${exist} = Xtrue ] && [ X${skip} = Xfalse ] ) ; then
	717	IGCM_sys_Chmod u+w $2
	718	fi
[622]	719
[711]	720	if [ X${skip} = Xfalse ] ; then
[719]	721	i=0
[711]	722	while [ $i -lt $NB_ESSAI ] ; do
[1234]	723	# Identify file system
	724	fileDeviceNumberInHex=$( stat -c %d $1 )
	725	status=$?
	726	if [ ${status} -gt 0 ] ; then
	727	IGCM_debug_Exit "IGCM_sys_Put_Out"
	728	fi
	729	# Identify file system
	730	directoryDeviceNumberInHex=$( stat -c %d $( dirname $2 ) )
	731	status=$?
	732	if [ ${status} -gt 0 ] ; then
	733	IGCM_debug_Exit "IGCM_sys_Put_Out"
	734	fi
	735
[1412]	736	if ( [ ${fileDeviceNumberInHex} -ne ${directoryDeviceNumberInHex} ] \|\| [ X$3 = XNOMOVE ] ) ; then
[1234]	737	# They are not on the same device. USUAL WAY
[1179]	738	\cp $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[1037]	739	status=$?
	740	else
[1234]	741	# They are on the same device. NOT SO USUAL WAY
[1179]	742	\mv $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[1037]	743	status=$?
	744	fi
[711]	745	if [ ${status} -gt 0 ]; then
[713]	746	IGCM_debug_Print 2 "IGCM_sys_Put_Out : cp failed error code ${status} ${i}/${NB_ESSAI}"
[711]	747	IGCM_debug_Print 2 "IGCM_sys_Put_Out : sleep ${DELAI} seconds and try again."
[765]	748	[ -f ${1} ] && ls -l ${1}
[759]	749	[ -f ${2} ] && ls -l ${2}
	750	[ -f ${2}/${1} ] && ls -l ${2}/${1}
[711]	751	sleep $DELAI
	752	else
	753	break
	754	fi
	755	(( i = i + 1 ))
	756	done
	757	fi
	758
[623]	759	# #RSYNC WITH NETWORK SSH CALL
[1179]	760	# echo ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
	761	# ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[622]	762
[623]	763	# #RSYNC WITH NFS USE
[1179]	764	# echo ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
	765	# ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[622]	766
[711]	767	# status=$?
	768	# IGCM_sys_Rsync_out $status
[622]	769
[1179]	770	# ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[711]	771	# (( status=status+$? ))
[622]	772
[711]	773	if [ ${status} -gt 0 ] ; then
[1179]	774	IGCM_debug_Print 2 "IGCM_sys_Put_Out : cp failed error code ${status}"
[765]	775	[ -f ${1} ] && ls -l ${1}
[759]	776	[ -f ${2} ] && ls -l ${2}
	777	[ -f ${2}/${1} ] && ls -l ${2}/${1}
[1179]	778	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[623]	779	IGCM_debug_Exit "IGCM_sys_Put_Out"
[622]	780	else
[759]	781
	782	if [ X${JobType} = XRUN ] ; then
	783	if [ X${3} = X ] ; then
	784	[ -f ${2} ] && IGCM_sys_Chmod 444 ${2}
	785	[ -f ${2}/${1} ] && IGCM_sys_Chmod 444 ${2}/${1}
	786	fi
	787	fi
	788
[1179]	789	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[622]	790	fi
[623]	791	fi
	792	IGCM_debug_PopStack "IGCM_sys_Put_Out"
	793	return 0
[622]	794	}
	795
	796	#D-#==================================================
	797	#D-function IGCM_sys_Get
	798	#D-* Purpose: Get a file from ${ARCHIVE}
	799	#D-* Examples: IGCM_sys_Get myfile /destpath/myfile_with_PREFIX
	800	#D- IGCM_sys_Get /l Array_contain_myfiles /destpath/
	801	function IGCM_sys_Get {
[623]	802	IGCM_debug_PushStack "IGCM_sys_Get" $@
[622]	803
[773]	804	typeset DEST dm_liste target file_work
[667]	805	typeset NB_ESSAI DELAI status i
[622]	806
[623]	807	if ( $DEBUG_sys ) ; then
	808	echo "IGCM_sys_Get :" $@
	809	fi
[667]	810
	811	# number of tentative
[668]	812	NB_ESSAI=3
[667]	813	# time delay between tentative
[668]	814	DELAI=2
[667]	815
[623]	816	if [ $DRYRUN -le 2 ]; then
	817	if [ X${1} = X'/l' ] ; then
	818	eval set +A dm_liste \${${2}}
	819	else
	820	eval set +A dm_liste ${1}
[622]	821	fi
[623]	822	eval DEST=\${${#}}
[1179]	823	ccc_hsm get ${dm_liste[*]} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[715]	824	status=$?
	825	if [ ${status} -gt 0 ] ; then
	826	echo "WARNING IGCM_sys_Get : error code ${status}"
[1179]	827	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[623]	828	echo "WARNING IGCM_sys_Get : will stop later if the cp fails."
	829	fi
[622]	830
[715]	831	#if [ ${status} -gt 0 ] ; then
[1179]	832	# if [ ! "X$( grep "Lost dmusrcmd connection" ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ )" = "X" ] ; then
	833	# cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[785]	834	# echo "WARNING IGCM_sys_Get : Lost dmusrcmd connection : "
	835	# sleep 30
	836	# echo "We try another time"
[1179]	837	## dmget ${dm_liste[*]} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
	838	# ccc_hsm get ${dm_liste[*]} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[785]	839	# status=$?
	840	# if [ ${status} -gt 0 ] ; then
	841	# echo "ERROR IGCM_sys_Get : again demigration error :"
[1179]	842	# cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[785]	843	# IGCM_debug_Exit "IGCM_sys_Get"
[623]	844	# fi
[785]	845	# else
	846	# echo "ERROR IGCM_sys_Get : demigration error :"
[1179]	847	# cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[785]	848	# IGCM_debug_Exit "IGCM_sys_Get"
	849	# fi
[623]	850	#fi
[622]	851
[719]	852	# #RSYNC WITH NETWORK SSH CALL
[1179]	853	# echo ${RSYNC} ${RSYNC_opt} -e ssh ${STOREHOST}:"${dm_liste}" ${STOREHOST}:${RUN_DIR}/${DEST} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
	854	# ${RSYNC} ${RSYNC_opt} -e ssh ${STOREHOST}:"${dm_liste}" ${STOREHOST}:${RUN_DIR}/${DEST} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[719]	855
	856	# #RSYNC WITH NFS USE
[1179]	857	# echo ${RSYNC} ${RSYNC_opt} ${dm_liste} ${RUN_DIR}/${DEST} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
	858	# ${RSYNC} ${RSYNC_opt} ${dm_liste} ${RUN_DIR}/${DEST} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[719]	859
	860	# status=$?
	861	# IGCM_sys_Rsync_out $status
	862
[1179]	863	# ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[719]	864	# (( status=status+$? ))
	865
[623]	866	#USUAL WAY
	867	if [ X${1} = X'/l' ] ; then
	868	for target in ${dm_liste[*]} ; do
	869	local_file=$( basename ${target} )
[668]	870	# test if the target file is present before the loop
	871	IGCM_sys_TestFileArchive ${target}
[715]	872	status=$?
	873	if [ ${status} -gt 0 ] ; then
[701]	874	echo "IGCM_sys_Get, ERROR : regular file ${target} DOES NOT EXIST ."
[668]	875	IGCM_debug_Exit "IGCM_sys_Get"
	876	else
[719]	877	i=0
[668]	878	while [ $i -lt $NB_ESSAI ] ; do
[1037]	879	#if [ X${DoLink} = Xtrue ] ; then
[1179]	880	# \ln -s ${target} ${DEST}/${local_file} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[1037]	881	# status=$?
	882	# else
[1179]	883	# \cp ${target} ${DEST}/${local_file} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[1037]	884	# status=$?
	885	#fi
[1179]	886	\ln -s ${target} ${DEST}/${local_file} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[668]	887	status=$?
	888	if [ ${status} -gt 0 ]; then
[713]	889	IGCM_debug_Print 2 "IGCM_sys_Get : cp failed error code ${status} ${i}/${NB_ESSAI}"
[676]	890	IGCM_debug_Print 2 "IGCM_sys_Get : sleep ${DELAI} seconds and try again."
	891	sleep $DELAI
[668]	892	else
[676]	893	break
[668]	894	fi
	895	(( i = i + 1 ))
	896	done
[719]	897	if [ ${status} -gt 0 ] ; then
	898	echo "IGCM_sys_Get : error"
[1179]	899	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
	900	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[719]	901	IGCM_debug_Exit "IGCM_sys_Get"
	902	else
[1179]	903	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[719]	904	fi
[668]	905	fi
[623]	906	done
	907	else
[719]	908	i=0
	909	while [ $i -lt $NB_ESSAI ] ; do
[1179]	910	\cp ${dm_liste} ${DEST} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[719]	911	status=$?
	912	if [ ${status} -gt 0 ]; then
	913	IGCM_debug_Print 2 "IGCM_sys_Get : cp failed error code ${status} ${i}/${NB_ESSAI}"
	914	IGCM_debug_Print 2 "IGCM_sys_Get : sleep ${DELAI} seconds and try again."
	915	sleep $DELAI
	916	else
	917	break
	918	fi
	919	(( i = i + 1 ))
	920	done
	921	if [ ${status} -gt 0 ] ; then
	922	echo "IGCM_sys_Get : error"
[1179]	923	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
	924	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[719]	925	IGCM_debug_Exit "IGCM_sys_Get"
	926	else
[1179]	927	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[719]	928	fi
[623]	929	fi
	930	fi
	931	IGCM_debug_PopStack "IGCM_sys_Get"
	932	}
[622]	933
[623]	934	#D-#==================================================
[835]	935	#D-function IGCM_sys_GetDate_Monitoring
	936	#D-* Purpose: get the last year for which the monitoring has been computed
	937	#D-* Examples:
	938	#D-
	939	function IGCM_sys_GetDate_Monitoring {
	940	IGCM_debug_PushStack "IGCM_sys_GetDate_Monitoring" $@
	941	if ( $DEBUG_sys ) ; then
	942	echo "IGCM_sys_GetDate_Monitoring :" $@
	943	fi
	944
	945	eval ${2}=$( cdo showyear ${1} 2> /dev/null \| gawk '{ print $NF }' )
	946
	947	IGCM_debug_PopStack "IGCM_sys_GetDate_Monitoring"
	948	}
	949
	950	#D-#==================================================
[623]	951	#D-function IGCM_sys_Dods_Rm
	952	#D-* Purpose: DO NOTHING ! Put ${ARCHIVE} files on DODS internet protocole.
	953	#D-* Examples:
	954	#D-
	955	function IGCM_sys_Dods_Rm {
	956	if ( $DEBUG_sys ) ; then
	957	echo "IGCM_sys_Dods_Rm :" $@
	958	fi
[715]	959	typeset status
[623]	960	if [ $DRYRUN = 0 ]; then
	961
	962	# if [ ! -d /dmnfs/cont003/dods/public/${LOGIN}/${R_DODS}/${1} ] ; then
	963	# echo "WARNING : IGCM_sys_Dods_Rm /dmnfs/cont003/dods/public/${LOGIN}/${R_DODS}/${1} DOES NOT EXIST ."
	964	# echo "Nothing has been done."
	965	# return
	966	# fi
	967
[1392]	968	if [ "$#" -eq 1 ]; then
[1457]	969	## /ccc/cont003/home/igcmg/igcmg/Tools/irene/thredds_rm public/${LOGIN}/${R_DODS}/${1} # > out_dods_rm 2>&1
	970	$(ccc_home -u igcmg)/Tools/irene/thredds_rm public/${LOGIN}/${R_DODS}/${1} # > out_dods_rm 2>&1
[1392]	971	status=$?
	972	else
[1457]	973	## /ccc/cont003/home/igcmg/igcmg/Tools/irene/thredds_rm public/${LOGIN}/${R_DODS} # > out_dods_rm 2>&1
	974	$(ccc_home -u igcmg)/Tools/irene/thredds_rm public/${LOGIN}/${R_DODS} # > out_dods_rm 2>&1
[1392]	975	status=$?
	976	fi
[623]	977
[785]	978	# if [ ${status} -gt 0 ] ; then
	979	# echo "IGCM_sys_Dods_Rm : error."
	980	# cat out_dods_rm
	981	# IGCM_debug_Exit "IGCM_sys_Dods_Rm"
	982	# else
	983	# rm out_dods_rm
	984	# fi
[623]	985
	986	fi
[715]	987	return $status
[623]	988	}
	989
	990	#D-#==================================================
	991	#D-function IGCM_sys_Dods_Cp
	992	#D-* Purpose: Copy $(ARCHIVE) files on DODS internet protocole.
	993	#D-* Examples:
	994	#D-
	995	function IGCM_sys_Dods_Cp {
	996	if ( $DEBUG_sys ) ; then
	997	echo "IGCM_sys_Dods_Cp :" $@
	998	fi
[715]	999	typeset status
[623]	1000	if [ $DRYRUN = 0 ]; then
	1001
	1002	# if [ ! -d ${R_SAVE}/${1} ] ; then
	1003	# echo "WARNING : IGCM_sys_Dods_Cp ${R_SAVE}/${1} DOES NOT EXIST ."
	1004	# echo "Nothing has been done."
	1005	# return
	1006	# fi
	1007
[1457]	1008	## /ccc/cont003/home/igcmg/igcmg/Tools/irene/bin/thredds_cp ${1} public/${LOGIN}/${R_DODS} # > out_dods_cp 2>&1
	1009	$(ccc_home -u igcmg)/Tools/irene/thredds_cp ${1} public/${LOGIN}/${R_DODS} # > out_dods_cp 2>&1
[715]	1010	status=$?
[623]	1011
[715]	1012	# if [ ${status} -gt 0 ] ; then
[623]	1013	# echo "IGCM_sys_Dods_Cp : error."
	1014	# cat out_dods_cp
	1015	# IGCM_debug_Exit "IGCM_sys_Dods_Cp"
	1016	# else
	1017	# rm out_dods_cp
	1018	# fi
	1019
	1020	fi
[715]	1021	return $status
[623]	1022	}
	1023
	1024	#D-#==================================================
	1025	#D-function IGCM_sys_Put_Dods
[1178]	1026	#D-* Purpose: Put ${ARCHIVE} files on DODS internet protocole.
[623]	1027	#D-* Examples:
	1028	#D-
	1029	function IGCM_sys_Put_Dods {
	1030	IGCM_debug_PushStack "IGCM_sys_Put_Dods" $@
	1031	if ( $DEBUG_sys ) ; then
	1032	echo "IGCM_sys_Put_Dods :" $@
	1033	fi
[715]	1034	typeset status
[623]	1035	if [ $DRYRUN = 0 ]; then
	1036	if ( [ ! -d ${R_SAVE}/${1} ] && [ ! -d ${R_FIGR}/${1} ] ) ; then
	1037	echo "WARNING IGCM_sys_Put_Dods : None of the following directories exist. Exactly one should."
	1038	echo "WARNING IGCM_sys_Put_Dods : ${R_SAVE}/${1} DOES NOT EXIST."
	1039	echo "WARNING IGCM_sys_Put_Dods : ${R_FIGR}/${1} DOES NOT EXIST."
	1040	IGCM_debug_PopStack "IGCM_sys_Put_Dods"
	1041	return
[622]	1042	fi
[623]	1043
	1044	if ( [ -d ${R_SAVE}/${1} ] && [ -d ${R_FIGR}/${1} ] ) ; then
	1045	echo "WARNING IGCM_sys_Put_Dods : Both of the following directories exist. Exactly one should."
	1046	echo "WARNING IGCM_sys_Put_Dods : ${R_SAVE}/${1} EXISTS."
	1047	echo "WARNING IGCM_sys_Put_Dods : ${R_FIGR}/${1} EXISTS."
	1048	IGCM_debug_PopStack "IGCM_sys_Put_Dods"
	1049	return
	1050	fi
	1051	#
	1052	if [ -d ${R_SAVE}/${1} ] ; then
	1053	cd ${R_SAVE}
	1054	elif [ -d ${R_FIGR}/${1} ] ; then
	1055	cd ${R_FIGR}
	1056	fi
	1057
	1058	IGCM_sys_Dods_Rm ${1}
	1059	IGCM_sys_Dods_Cp ${1}
[715]	1060	status=0
[623]	1061
[715]	1062	if [ ${status} -gt 0 ] ; then
[623]	1063	echo "IGCM_sys_Put_Dods : error."
	1064	IGCM_debug_Exit "IGCM_sys_Put_Dods"
	1065	fi
	1066	fi
	1067	IGCM_debug_PopStack "IGCM_sys_Put_Dods"
[622]	1068	}
	1069
	1070	##############################################################
	1071	# REBUILD OPERATOR
	1072
[811]	1073	#D-#==================================================
[1163]	1074	#D-function IGCM_sys_sync
	1075	#D-* Purpose: flush buffer on disk
	1076	#D-* Examples:
	1077	#D-
	1078	function IGCM_sys_sync {
	1079	IGCM_debug_PushStack "IGCM_sys_sync" $@
	1080	if ( $DEBUG_sys ) ; then
	1081	echo "IGCM_sys_sync :" $@
	1082	fi
	1083
	1084	/bin/sync
	1085
	1086	IGCM_debug_PopStack "IGCM_sys_sync"
	1087	}
	1088
[622]	1089	############################################################
	1090	# Activate Running Environnment Variables
	1091
[811]	1092	#D-#==================================================
[1239]	1093	#D-function IGCM_sys_activ_variables
[811]	1094	#D-* Purpose: set environement variables prior to execution
	1095	#D-* Examples:
	1096	#D-
[622]	1097	function IGCM_sys_activ_variables {
[623]	1098	IGCM_debug_PushStack "IGCM_sys_activ_variables"
	1099	if ( $DEBUG_sys ) ; then
	1100	echo "IGCM_sys_activ_variables"
	1101	fi
[622]	1102
[811]	1103	# --------------------------------------------------------------------
[1409]	1104	#D- MODULE specifications
	1105	# --------------------------------------------------------------------
[1479]	1106	# Source the file EnvFile containing all module load needed to run the model.
[1485]	1107	# EnvFile can be specified in config.card. If this is the case and if the file exists,
	1108	# it will be used. Otherwise a default file will be used.
[1482]	1109	if [ ! X${config_UserChoices_EnvFile} = X ] && [ -f ${config_UserChoices_EnvFile} ] ; then
	1110	# EnvFile is set in config.card and the file exists
	1111	IGCM_debug_Print 1 "EnvFile set in config.card will be used"
	1112	EnvFile=${config_UserChoices_EnvFile}
	1113	else
[1479]	1114	IGCM_debug_Print 1 "IGCM_sys_active_variables : Default modules will be used"
	1115	module purge > /dev/null 2>&1
	1116	if [ X${compilerVersion} = XV17 ] ; then
	1117	IGCM_debug_Print 1 "IGCM_sys_active_variables : Using compiler version V17"
	1118	## EnvFile=/ccc/cont003/home/igcmg/igcmg/MachineEnvironment/irene/env_irene > /dev/null 2>&1
	1119	EnvFile=$( ccc_home -u igcmg)/MachineEnvironment/irene/env_irene > /dev/null 2>&1
	1120	else
	1121	IGCM_debug_Exit "This compiler version is not yet supported : ${compilerVersion}"
	1122	fi
[1409]	1123	fi
[1479]	1124	IGCM_debug_Print 1 "IGCM_sys_active_variables : Following EnvFile will be used :"
	1125	IGCM_debug_Print 1 ${EnvFile}
	1126	. ${EnvFile}
	1127
[1457]	1128	IGCM_debug_Print 1 "IGCM_sys_active_variables : Now loaded modules for Irene. "
[1433]	1129	module list
[1409]	1130
	1131	# --------------------------------------------------------------------
[811]	1132	#D- MPI specifications
	1133	# --------------------------------------------------------------------
[1191]	1134
[811]	1135	# --------------------------------------------------------------------
	1136	#D- Other specifications
	1137	# --------------------------------------------------------------------
	1138
[623]	1139	ulimit -s unlimited
[622]	1140
[623]	1141	IGCM_debug_PopStack "IGCM_sys_activ_variables"
[622]	1142	}
	1143
	1144	############################################################
	1145	# Desactivate Running Environnment Variables
	1146
[811]	1147	#D-#==================================================
	1148	#D-function IGCM_sys_desactiv_variables
	1149	#D-* Purpose: unset environement variables after execution
	1150	#D-* Examples:
	1151	#D-
[622]	1152	function IGCM_sys_desactiv_variables {
[623]	1153	IGCM_debug_PushStack "IGCM_sys_desactiv_variables"
	1154	if ( $DEBUG_sys ) ; then
	1155	echo "IGCM_sys_desactiv_variables"
	1156	fi
[811]	1157	# --------------------------------------------------------------------
	1158	#D- MPI specifications
	1159	# --------------------------------------------------------------------
	1160
	1161	# --------------------------------------------------------------------
	1162	#D- Other specifications
	1163	# --------------------------------------------------------------------
	1164
[623]	1165	IGCM_debug_PopStack "IGCM_sys_desactiv_variables"
[622]	1166	}
	1167
[623]	1168	############################################################
[1238]	1169	# Update job headers to be used by the scheduler
	1170
	1171	#D-#==================================================
	1172	#D-function IGCM_sys_updateHeaders
	1173	#D-* Purpose: Update job headers to be used by the scheduler
	1174	#D-* Examples: IGCM_sys_updateHeaders /path/to/Job_MYEXP
	1175	#D-
	1176	function IGCM_sys_updateHeaders {
	1177	IGCM_debug_PushStack "IGCM_sys_updateHeaders"
	1178	if ( $DEBUG_sys ) ; then
	1179	echo "IGCM_sys_updateHeaders"
	1180	fi
	1181	typeset file
	1182	file=$1
	1183
	1184	if [ ${executionType} -eq 1 ] ; then
	1185	# MPMD + MPI
	1186	sed -e "/::openMPthreads::/d" \
	1187	-e "s/::JobNumProcTot::/${coreNumber}/" \
[1256]	1188	-e "/#MSUB -x/d" \
	1189	-e "/--cpu_bind=none/d" \
[1238]	1190	${file} > ${file}.tmp
	1191
	1192	elif [ ${executionType} -eq 2 ] ; then
[1390]	1193	# MPMD + MPI + OMP : mpirun/ccc_mprun/error
[1432]	1194	if ( [ "X${config_UserChoices_ExecutionModeOnCurie}" = "Xmpirun" ] ) ; then
[1390]	1195	sed -e "/::openMPthreads::/d" \
	1196	-e "s/::JobNumProcTot::/${coreNumber}/" \
	1197	${file} > ${file}.tmp
[1432]	1198	elif ( [ "X${config_UserChoices_ExecutionModeOnCurie}" = "X" ] \|\| [ "X${config_UserChoices_ExecutionModeOnCurie}" = "Xccc_mprun" ] ) ; then
[1390]	1199	sed -e "/::openMPthreads::/d" \
	1200	-e "s/::JobNumProcTot::/${coreNumber}/" \
	1201	-e "/--cpu_bind=none/d" \
	1202	${file} > ${file}.tmp
	1203	else
	1204	IGCM_debug_Print 1 "You have to set ExecutionModeOnCurie=ccc_mprun or mpirun in config.card"
	1205	IGCM_debug_Exit "IGCM_sys_updateHeaders"
	1206	fi
[1238]	1207	elif [ ${executionType} -eq 3 ] ; then
	1208	# SPMD + MPI/OMP
	1209	sed -e "s/::openMPthreads::/${openMPthreads}/" \
	1210	-e "s/::JobNumProcTot::/${mpiTasks}/" \
[1240]	1211	-e "/#MSUB -x/d" \
[1238]	1212	-e "/--cpu_bind=none/d" \
	1213	${file} > ${file}.tmp
	1214
	1215	elif [ ${executionType} -eq 4 ] ; then
	1216	# SPMD + MPI only
	1217	sed -e "s/::JobNumProcTot::/${mpiTasks}/" \
	1218	-e "/::openMPthreads::/d" \
[1240]	1219	-e "/#MSUB -x/d" \
[1238]	1220	-e "/--cpu_bind=none/d" \
	1221	${file} > ${file}.tmp
	1222
	1223	elif [ ${executionType} -eq 5 ] ; then
	1224	# SPMD + OMP only
	1225	sed -e "s/::openMPthreads::/${openMPthreads}/" \
	1226	-e "/::JobNumProcTot::/d" \
[1240]	1227	-e "/#MSUB -x/d" \
[1238]	1228	-e "/--cpu_bind=none/d" \
	1229	${file} > ${file}.tmp
	1230
	1231	elif [ ${executionType} -eq 6 ] ; then
	1232	# SEQUENTIAL THEN
	1233	sed -e "s/::JobNumProcTot::/1/" \
	1234	-e "/::openMPthreads::/d" \
[1240]	1235	-e "/#MSUB -x/d" \
[1238]	1236	-e "/--cpu_bind=none/d" \
	1237	${file} > ${file}.tmp
	1238
	1239	fi
	1240
	1241	IGCM_sys_Mv ${file}.tmp ${file}
	1242
	1243	IGCM_debug_PopStack "IGCM_sys_updateHeaders"
	1244	}
	1245
	1246	############################################################
[623]	1247	# Build MPI/OMP scripts run file (dummy function)
[622]	1248
[811]	1249	#D-#==================================================
	1250	#D-function IGCM_sys_build_run_file
	1251	#D-* Purpose: build run file (deprecated)
	1252	#D-* Examples:
	1253	#D-
[622]	1254	function IGCM_sys_build_run_file {
[811]	1255	IGCM_debug_Print 3 " dummy function : IGCM_sys_build_run_file "
[622]	1256	}
	1257
	1258	############################################################
	1259	# Build MPI/OMP scripts
[811]	1260
	1261	#D-#==================================================
	1262	#D-function IGCM_sys_build_execution_scripts
	1263	#D-* Purpose: build execution scripts to be launch by ${HOST_MPIRUN_COMMAND}
	1264	#D-* Examples:
	1265	#D-
[623]	1266	function IGCM_sys_build_execution_scripts
[622]	1267	{
[623]	1268	IGCM_debug_PushStack "IGCM_sys_build_execution_scripts" $@
	1269	if ( $DEBUG_sys ) ; then
	1270	echo "IGCM_sys_build_execution_scripts " $@
	1271	fi
[622]	1272
[623]	1273	EXECUTION=${HOST_MPIRUN_COMMAND}
[622]	1274
[623]	1275	if ( ${OK_PARA_MPMD} ) ; then
[622]	1276
[1390]	1277	# MPMD mode
	1278	# 1 MPI only
	1279	# 2 MPI/OpenMP mpirun method
	1280	# 3 MPI/OpenMP ccc_mprun method
	1281
[1381]	1282	if [ -f run_file ] ; then
	1283	IGCM_sys_Rm -f run_file
	1284	fi
	1285	touch run_file
	1286
[1390]	1287	# case 1 : Only MPI (MPMD)
[1078]	1288	if ( ! ${OK_PARA_OMP} ) ; then
[622]	1289
[1078]	1290	# Build run_file
[622]	1291
[1078]	1292	# First loop on the components for the coupler ie oasis (only if oasis3)
	1293	# the coupler ie oasis3 must be the first one
	1294	for comp in ${config_ListOfComponents[*]} ; do
[622]	1295
[1078]	1296	eval ExeNameIn=\${config_Executable_${comp}[0]}
	1297	eval ExeNameOut=\${config_Executable_${comp}[1]}
[622]	1298
[1078]	1299	# for CPL component only
	1300	if [ "X${comp}" = "XCPL" ] && [ "X${ExeNameOut}" != X\"\" ] ; then
[1000]	1301
[1078]	1302	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
	1303	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
	1304	echo "${comp_proc_mpi_loc} ./${ExeNameOut}" >> run_file
	1305	fi
	1306	done
[622]	1307
[1078]	1308	# Then second loop on the components
	1309	for comp in ${config_ListOfComponents[*]} ; do
[622]	1310
[1078]	1311	eval ExeNameIn=\${config_Executable_${comp}[0]}
	1312	eval ExeNameOut=\${config_Executable_${comp}[1]}
[622]	1313
[1078]	1314	# Only if we really have an executable for the component and not the coupler ie oasis:
	1315	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${comp}" != "XCPL" ] ) ; then
[622]	1316
[1078]	1317	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
	1318	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
	1319	echo "${comp_proc_mpi_loc} ./${ExeNameOut}" >> run_file
	1320	fi
	1321	done
[622]	1322
[1390]	1323	EXECUTION="${HOST_MPIRUN_COMMAND} -f ./run_file"
	1324
	1325	IGCM_sys_Chmod u+x run_file
	1326	if ( $DEBUG_sys ) ; then
	1327	echo "run_file contains : "
	1328	cat run_file
	1329	fi
	1330
[1432]	1331	elif ( [ "X${config_UserChoices_ExecutionModeOnCurie}" = "Xmpirun" ] ) ; then
[1390]	1332
	1333	# 2 MPI/OpenMP mpirun method
	1334
	1335	# Use of mpirun instead of ccc_mprun
	1336	EXECUTION="time mpirun"
	1337
	1338	# Hosts treatment
	1339	${EXECUTION} hostname \| sort \| uniq > hosts.tmp
	1340
	1341	i=0
	1342	rm -f hosts rankfile
[1462]	1343	IGCM_debug_Print 1 "sys Irene, Hosts available :"
[1390]	1344	for nodes in `cat hosts.tmp`
	1345	do
	1346	host[$i]=$nodes
	1347	echo "${host[$i]}" >> hosts
	1348	IGCM_debug_Print 1 ${host[$i]}
	1349	i=$((i+1))
	1350	done
	1351	rm -f hosts.tmp
	1352
	1353	listnodes=${host[*]}
	1354
	1355	EXECUTION="${EXECUTION} -hostfile hosts -rankfile rankfile"
	1356
	1357	# Initialisation
	1358	rank=0
	1359	current_core=0
	1360	init_exec=n
	1361
	1362	# Loop on the components
	1363	for comp in ${config_ListOfComponents[*]} ; do
	1364
	1365	eval ExeNameIn=\${config_Executable_${comp}[0]}
	1366	eval ExeNameOut=\${config_Executable_${comp}[1]}
	1367
	1368	# Not possible if oasis has an executable (i.e old version of oasis3)
	1369	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${comp}" = "XCPL" ] ) ; then
	1370	IGCM_debug_Exit "ERROR MPMD with hybrid MPI-OpenMP is not available with oasis3 version"
	1371	IGCM_debug_Print 2 "Only available with oasis3-MCT version coupler"
	1372	IGCM_debug_Verif_Exit
	1373	fi
	1374
	1375	# Only if we really have an executable for the component :
	1376	if [ "X${ExeNameOut}" != X\"\" ] ; then
	1377
	1378	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
	1379	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
[1426]	1380	eval comp_proc_nod_loc=\${${comp}_PROC_NOD}
[1390]	1381
	1382	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
	1383	echo "" >> script_${ExeNameOut}.ksh
	1384	if [ ${comp_proc_omp_loc} -gt 1 ] ; then
	1385
	1386	# Check if the number of threads is correct
	1387	case ${comp_proc_omp_loc} in
	1388	2\|4\|8\|16)
	1389	IGCM_debug_Print 1 "You run ${ExeNameOut} on ${comp_proc_omp_loc} OMP threads"
	1390	;;
	1391	*)
	1392	IGCM_debug_Exit "ERROR with OMP parameters !"
	1393	IGCM_debug_Print 2 "${comp_proc_omp_loc} is not possible as number of OMP threads"
	1394	IGCM_debug_Print 2 "Only 2,4,8,16 as number of OMP threads are possible "
	1395	IGCM_debug_Verif_Exit
	1396	;;
	1397	esac
	1398	echo "export KMP_STACKSIZE=3g" >> script_${ExeNameOut}.ksh
	1399	echo "export KMP_LIBRARY=turnaround" >> script_${ExeNameOut}.ksh
	1400	echo "export MKL_SERIAL=YES" >> script_${ExeNameOut}.ksh
	1401	echo "OMP_NUM_THREADS=${comp_proc_omp_loc}" >> script_${ExeNameOut}.ksh
	1402	fi
	1403
	1404	#echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
	1405	echo "(( MYMPIRANK = OMPI_COMM_WORLD_RANK )) " >> script_${ExeNameOut}.ksh
	1406	echo "MYMPIRANK=\$(printf '%3.3d\n' \${MYMPIRANK})" >> script_${ExeNameOut}.ksh
	1407	echo "./${ExeNameOut} > out_${ExeNameOut}.out.\${MYMPIRANK} 2>out_${ExeNameOut}.err.\${MYMPIRANK}" >> script_${ExeNameOut}.ksh
	1408	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
	1409
	1410	if [ ${init_exec} = y ] ; then
	1411	EXECUTION="${EXECUTION} : -np ${comp_proc_mpi_loc} ./script_${ExeNameOut}.ksh"
	1412	else
	1413	EXECUTION="${EXECUTION} -np ${comp_proc_mpi_loc} ./script_${ExeNameOut}.ksh"
	1414	init_exec=y
	1415	fi
	1416
	1417	# Build rankfile : method used to assign cores and nodes for the MPI process
	1418	# Ex :
	1419	#rank 0=curie5296 slot=0,1,2,3
	1420	#rank 1=curie5296 slot=4,5,6,7
	1421	# Example of final command :
	1422	# mpirun -hostfile hosts -rankfile rankfile -np 27 ./script_lmdz.x.ksh : -np 5 ./script_opa.xx.ksh
	1423	# with script_lmdz.x.ksh :
	1424	# #!/bin/ksh
	1425	#export KMP_STACKSIZE=3g
	1426	#export KMP_LIBRARY=turnaround
	1427	#export MKL_SERIAL=YES
	1428	#OMP_NUM_THREADS=4
	1429	#./lmdz.x
[1426]	1430	if [ ${comp_proc_nod_loc} -gt 1 ] ; then
[1430]	1431	(( offset_comp_proc_loc = NB_CORE_PER_NODE / (comp_proc_mpi_loc / comp_proc_nod_loc) ))
[1426]	1432	else
	1433	(( offset_comp_proc_loc = comp_proc_omp_loc ))
	1434	fi
[1390]	1435	for nb_proc_mpi in `seq 0 $(($comp_proc_mpi_loc-1))`; do
[1426]	1436	(( index_host = current_core / NB_CORE_PER_NODE ))
	1437	host_value=${host[${index_host}]}
	1438	(( slot = current_core % NB_CORE_PER_NODE ))
	1439	virg=","
	1440	string_final=""
	1441	for index in `seq $slot $(($slot+$comp_proc_omp_loc-1))`; do
	1442	string=$index$virg
	1443	string_final=$string_final$string
	1444	done
	1445	string_final=$( echo $string_final \| sed "s/.$//" )
	1446	echo "rank $rank=$host_value slot=$string_final" >> rankfile
	1447	(( rank = rank + 1 ))
	1448	(( current_core = current_core + offset_comp_proc_loc ))
[1390]	1449	done
	1450	fi
	1451	done
	1452
[1432]	1453	elif ( [ "X${config_UserChoices_ExecutionModeOnCurie}" = "X" ] \|\| [ "X${config_UserChoices_ExecutionModeOnCurie}" = "Xccc_mprun" ] ) ; then
[1390]	1454
	1455	# 3 MPI/OpenMP ccc_mprun method
	1456
[1381]	1457	# MPI-OpenMP (MPMD)
	1458	# example of run_file
	1459	# 71-8 env OMP_NUM_THREADS=8 ./script_lmdz.x.ksh
	1460	# 480-1 env OMP_NUM_THREADS=1 ./script_opa.xx.ksh
	1461	# 1-1 env OMP_NUM_THREADS=1 ./script_xios.x.ksh
[622]	1462
[1078]	1463	# Loop on the components
	1464	for comp in ${config_ListOfComponents[*]} ; do
[1000]	1465
[1078]	1466	eval ExeNameIn=\${config_Executable_${comp}[0]}
	1467	eval ExeNameOut=\${config_Executable_${comp}[1]}
[1000]	1468
[1078]	1469	# Not possible if oasis has an executable (i.e old version of oasis3)
	1470	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${comp}" = "XCPL" ] ) ; then
	1471	IGCM_debug_Exit "ERROR MPMD with hybrid MPI-OpenMP is not available with oasis3 version"
	1472	IGCM_debug_Print 2 "Only available with oasis3-MCT version coupler"
	1473	IGCM_debug_Verif_Exit
	1474	fi
[1000]	1475
[1078]	1476	# Only if we really have an executable for the component :
	1477	if [ "X${ExeNameOut}" != X\"\" ] ; then
[1000]	1478
[1078]	1479	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
	1480	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
[1000]	1481
[1381]	1482	# Build script files
	1483
[1078]	1484	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
	1485	echo "" >> script_${ExeNameOut}.ksh
	1486	if [ ${comp_proc_omp_loc} -gt 1 ] ; then
	1487
	1488	# Check if the number of threads is correct
	1489	case ${comp_proc_omp_loc} in
[1440]	1490	2\|4\|8\|12\|16\|24)
[1078]	1491	IGCM_debug_Print 1 "You run ${ExeNameOut} on ${comp_proc_omp_loc} OMP threads"
	1492	;;
	1493	*)
	1494	IGCM_debug_Exit "ERROR with OMP parameters !"
	1495	IGCM_debug_Print 2 "${comp_proc_omp_loc} is not possible as number of OMP threads"
[1440]	1496	IGCM_debug_Print 2 "Only 2,4,8,12,16,24 as number of OMP threads are possible "
[1078]	1497	IGCM_debug_Verif_Exit
	1498	;;
	1499	esac
[1381]	1500	### only for LMDZ?
[1078]	1501	echo "export KMP_STACKSIZE=3g" >> script_${ExeNameOut}.ksh
	1502	echo "export KMP_LIBRARY=turnaround" >> script_${ExeNameOut}.ksh
	1503	echo "export MKL_SERIAL=YES" >> script_${ExeNameOut}.ksh
	1504	echo "OMP_NUM_THREADS=${comp_proc_omp_loc}" >> script_${ExeNameOut}.ksh
[1381]	1505
[1078]	1506	fi
[1318]	1507
[1381]	1508	# to have out/err per process on different files
	1509	echo "./${ExeNameOut} > out_${ExeNameOut}.out.\${SLURM_PROCID} 2>out_${ExeNameOut}.err.\${SLURM_PROCID}" >> script_${ExeNameOut}.ksh
[1317]	1510	#echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
[1078]	1511
[1381]	1512	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
[1078]	1513
[1381]	1514	# Complete run_file
[1000]	1515
[1381]	1516	echo "${comp_proc_mpi_loc}-${comp_proc_omp_loc} env OMP_NUM_THREADS=${comp_proc_omp_loc} ./script_${ExeNameOut}.ksh " >>run_file
	1517
[1078]	1518	fi
	1519	done
[1381]	1520
[1385]	1521	## variable added to stop after 60s instead of 600s by default.
	1522	## This is used when no error comes from executables and when something stopped an executable without notice.
	1523	export SLURM_WAIT=60
[1381]	1524
[1385]	1525	## module advised by TGCC (instead of 2 variables)
	1526	module load feature/bridge/heterogenous_mpmd
[1381]	1527
[1390]	1528	# set EXECUTION for ccc_mprun case (similar to #1 : MPI only MPMD method)
[1000]	1529
[1390]	1530	EXECUTION="${HOST_MPIRUN_COMMAND} -f ./run_file"
[1381]	1531
[1390]	1532	IGCM_sys_Chmod u+x run_file
	1533	if ( $DEBUG_sys ) ; then
	1534	echo "run_file contains : "
	1535	cat run_file
	1536	fi
[1381]	1537
[1390]	1538	else
[1381]	1539
[1390]	1540	IGCM_debug_Print 1 "sys Curie : choose mpirun or ccc_mprun in config.card for ExecutionModeOnCurie"
	1541
	1542	fi # 1 2 or 3 if ${OK_PARA_MPMD}
	1543
	1544	else
[1078]	1545	# Only one executable (SPMD mode).
[1000]	1546
[622]	1547	for comp in ${config_ListOfComponents[*]} ; do
	1548
[685]	1549	# Only if we really have an executable for the component :
[623]	1550	eval ExeNameOut=\${config_Executable_${comp}[1]}
[685]	1551	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${ExeNameOut}" != "Xinca.dat" ] ) ; then
[622]	1552
[1381]	1553	# Build script files
	1554
[685]	1555	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
	1556	echo "" >> script_${ExeNameOut}.ksh
	1557	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
	1558
	1559	if ( ${OK_PARA_OMP} ) ; then
	1560	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
	1561	echo "" >> script_${ExeNameOut}.ksh
	1562	echo "export KMP_STACKSIZE=3g" >> script_${ExeNameOut}.ksh
	1563	echo "export KMP_LIBRARY=turnaround" >> script_${ExeNameOut}.ksh
	1564	echo "export MKL_SERIAL=YES" >> script_${ExeNameOut}.ksh
	1565	echo "OMP_NUM_THREADS=${comp_proc_omp_loc}" >> script_${ExeNameOut}.ksh
	1566	fi
	1567
	1568	if ( ${OK_PARA_MPI} ) ; then
[623]	1569	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
[685]	1570	# Default : ccc_mprun used if nb_proc gt 1
	1571	# to have out/err per process on different files
[1317]	1572	echo "./${ExeNameOut} > out_${ExeNameOut}.out.\${SLURM_PROCID} 2>out_${ExeNameOut}.err.\${SLURM_PROCID}" >> script_${ExeNameOut}.ksh
	1573	#echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
[685]	1574	EXECUTION="${HOST_MPIRUN_COMMAND} -n ${comp_proc_mpi_loc} ./script_${ExeNameOut}.ksh"
[623]	1575	else
[685]	1576	# Default : ccc_mprun is NOT used if nb_proc eq 1
	1577	# to have out/err per process on different files
[1317]	1578	echo "./${ExeNameOut} > out_${ExeNameOut}.out 2>out_${ExeNameOut}.err" >> script_${ExeNameOut}.ksh
	1579	#echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
[685]	1580	EXECUTION="/usr/bin/time ./script_${ExeNameOut}.ksh"
[622]	1581	fi
[685]	1582
[1462]	1583	IGCM_debug_Print 1 "sys Irene : script_${ExeNameOut}.ksh contains"
[685]	1584	cat script_${ExeNameOut}.ksh
	1585
[623]	1586	fi
[622]	1587
	1588	done
	1589
[1390]	1590	fi # ${OK_PARA_MPMD}
[622]	1591
[1462]	1592	IGCM_debug_Print 1 "sys Irene : execution command is "
[623]	1593	IGCM_debug_Print 1 "$EXECUTION"
[622]	1594
[623]	1595	IGCM_debug_PopStack "IGCM_sys_build_execution_scripts"
[622]	1596	}
	1597
[933]	1598	#D-#==================================================
	1599	#D-function IGCM_sys_check_path
	1600	#D-* Purpose: check that RUN_DIR_PATH that will be removed on some machine
	1601	#D-* do not point to an important use directory. Stop immediately in that case.
	1602	#D-* Examples:
	1603	#D-
	1604	function IGCM_sys_check_path {
	1605	IGCM_debug_PushStack "IGCM_sys_check_path"
	1606	if ( $DEBUG_sys ) ; then
	1607	echo "IGCM_sys_check_path"
	1608	fi
[811]	1609
[1432]	1610	if ( [ X${RUN_DIR_PATH} = X${HOME} ] \|\| [ X${RUN_DIR_PATH} = X${CCCWORKDIR} ] \|\| [ X${RUN_DIR_PATH} = X${CCCSCRATCHDIR} ] \|\| [ X${RUN_DIR_PATH} = X${CCCSTOREDIR} ] ) ; then
[933]	1611	IGCM_debug_Print 1 "Variable RUN_DIR_PATH is pointing to an important directory : ${RUN_DIR_PATH}"
	1612	IGCM_debug_Print 1 "Please check the RUN_DIR_PATH definition in your Job : Job_${config_UserChoices_JobName}"
[936]	1613	IGCM_debug_Exit "This will stop the job"
[933]	1614	fi
	1615	IGCM_debug_PopStack "IGCM_sys_check_path"
	1616	}
	1617
[811]	1618	#D-#==================================================
	1619	#D-function IGCM_sys_check_quota
	1620	#D-* Purpose: check user quota. Stop the simulation if quota above 90%
	1621	#D-* Examples:
	1622	#D-
[714]	1623	function IGCM_sys_check_quota {
[776]	1624	IGCM_debug_PushStack "IGCM_sys_check_quota"
	1625	if ( $DEBUG_sys ) ; then
	1626	echo "IGCM_sys_check_quota"
	1627	fi
	1628	# Limit of quota (in %)
	1629	limit_quota=90
	1630
	1631	# Check of the volume
[912]	1632	volume_quota=$(ccc_quota \| grep ' scratch' \| gawk '{print $2}')
	1633	volume_avail=$(ccc_quota \| grep ' scratch' \| gawk '{print $3}')
[776]	1634
	1635	if ( [ ! X${volume_quota} = X ] && [ ! ${volume_quota} = "-" ] ) ; then
	1636
	1637	unit_avail=${volume_avail: -1}
	1638	unit_quota=${volume_quota: -1}
	1639
	1640	if [ "${unit_quota}" = "*" ] ; then
[811]	1641	IGCM_debug_Print 1 "Please, check your quota of volume on scratch"
	1642	IGCM_debug_Print 1 "More than 100% of your quota is used"
	1643	IGCM_debug_Print 1 "Use the ccc_quota command to check"
	1644	IGCM_debug_Print 1 "You must have more than 10% available to run"
	1645	IGCM_debug_Exit "Not enough space to run ! STOP HERE"
	1646	IGCM_debug_Verif_Exit
[714]	1647	fi
[776]	1648
	1649	temp_avail=${volume_avail%%${volume_avail: -1}*}
	1650	temp_quota=${volume_quota%%${volume_quota: -1}*}
	1651
	1652	if [ ! ${unit_avail} = ${unit_quota} ] ; then
	1653
	1654	# Convertion
	1655	if [ ${volume_avail: -1} = "T" ] ; then
	1656	(( temp_avail = temp_avail * 1000000000000 ))
	1657	elif [ ${volume_avail: -1} = "G" ] ; then
	1658	(( temp_avail = temp_avail * 1000000000 ))
	1659	elif [ ${volume_avail: -1} = "M" ] ; then
	1660	(( temp_avail = temp_avail * 1000000 ))
	1661	elif [ ${volume_avail: -1} = "k" ] ; then
	1662	(( temp_avail = temp_avail * 1000 ))
	1663	else
	1664	(( temp_avail = volume_avail ))
	1665	fi
	1666	if [ ${volume_quota: -1} = "T" ] ; then
	1667	(( temp_quota = temp_quota * 1000000000000 ))
	1668	elif [ ${volume_quota: -1} = "G" ] ; then
	1669	(( temp_quota = temp_quota * 1000000000 ))
	1670	elif [ ${volume_quota: -1} = "M" ] ; then
	1671	(( temp_quota = temp_quota * 1000000 ))
	1672	elif [ ${volume_quota: -1} = "k" ] ; then
	1673	(( temp_quota = temp_quota * 1000 ))
	1674	else
	1675	(( temp_quota = volume_quota ))
	1676	fi
	1677	fi
	1678
	1679	quota_volume=$(echo "scale=2 ; $temp_quota/$temp_avail*100" \| bc)
	1680	# echo "volume ratio is " $quota_volume
	1681
	1682	if [ ${quota_volume} -ge ${limit_quota} ] ; then
	1683	IGCM_debug_Print 1 "Please, check your quota of volume on scratch"
	1684	IGCM_debug_Print 1 "${quota_volume}% of your quota is used"
	1685	IGCM_debug_Print 1 "Use the ccc_quota command to check"
	1686	IGCM_debug_Print 1 "You must have more than 10% available to run"
	1687	IGCM_debug_Exit "Not enough space to run ! STOP HERE"
	1688	IGCM_debug_Verif_Exit
	1689	fi
	1690
	1691	fi
	1692
	1693	# Check of the number of inodes
	1694
[912]	1695	inode_quota=$(ccc_quota \| grep ' scratch' \| gawk '{print $6}')
	1696	inode_avail=$(ccc_quota \| grep ' scratch' \| gawk '{print $7}')
[776]	1697
	1698	if ( [ ! X${inode_quota} = X ] && [ ! ${inode_quota} = "-" ] ) ; then
	1699
	1700	unit_avail=${inode_avail: -1}
	1701	unit_quota=${inode_quota: -1}
	1702
	1703	if [ "${unit_quota}" = "*" ] ; then
[811]	1704	IGCM_debug_Print 1 "Please, check your quota of inode on scratch"
	1705	IGCM_debug_Print 1 "More than 100% of your quota is used"
	1706	IGCM_debug_Print 1 "Use the ccc_quota command to check"
	1707	IGCM_debug_Print 1 "You must have more than 10% available to run"
	1708	IGCM_debug_Exit "Not enough space to run ! STOP HERE"
	1709	IGCM_debug_Verif_Exit
[776]	1710	fi
	1711
	1712	temp_avail=${inode_avail%%${inode_avail: -1}*}
	1713	temp_quota=${inode_quota%%${inode_quota: -1}*}
	1714
	1715	if [ ! ${unit_avail} = ${unit_quota} ] ; then
	1716
	1717	# Convertion
	1718	if [ ${inode_avail: -1} = "T" ] ; then
	1719	(( temp_avail = temp_avail * 1000000000000 ))
	1720	elif [ ${inode_avail: -1} = "G" ] ; then
	1721	(( temp_avail = temp_avail * 1000000000 ))
	1722	elif [ ${inode_avail: -1} = "M" ] ; then
	1723	(( temp_avail = temp_avail * 1000000 ))
	1724	elif [ ${inode_avail: -1} = "k" ] ; then
	1725	(( temp_avail = temp_avail * 1000 ))
	1726	else
	1727	(( temp_avail = inode_avail ))
	1728	fi
	1729
	1730	if [ ${inode_quota: -1} = "T" ] ; then
	1731	(( temp_quota = temp_quota * 1000000000000 ))
	1732	elif [ ${inode_quota: -1} = "G" ] ; then
	1733	(( temp_quota = temp_quota * 1000000000 ))
	1734	elif [ ${inode_quota: -1} = "M" ] ; then
	1735	(( temp_quota = temp_quota * 1000000 ))
	1736	elif [ ${inode_quota: -1} = "k" ] ; then
	1737	(( temp_quota = temp_quota * 1000 ))
	1738	else
	1739	(( temp_quota = inode_quota ))
	1740	fi
	1741	fi
	1742	quota_inode=$(echo "scale=2 ; $temp_quota/$temp_avail*100" \| bc)
	1743	# echo "inode ratio is " $quota_inode
	1744
	1745	if [ ${quota_inode} -ge ${limit_quota} ] ; then
	1746	IGCM_debug_Print 1 "Please, check your quota of inode on scratch"
	1747	IGCM_debug_Print 1 "${quota_inode}% of your quota is used"
	1748	IGCM_debug_Print 1 "Use the ccc_quota command to check"
	1749	IGCM_debug_Print 1 "You must have more than 10% available to run"
	1750	IGCM_debug_Exit "Not enough space to run ! STOP HERE"
	1751	IGCM_debug_Verif_Exit
	1752	fi
	1753	fi
	1754	IGCM_debug_PopStack "IGCM_sys_check_quota"
[714]	1755	}
	1756
[773]	1757	#D-#==================================================
[1286]	1758	#D-function IGCM_sys_projectAccounting
	1759	#D-* Purpose: store project accounting information in a file
	1760	#D-* Examples:
	1761	#D-
	1762	function IGCM_sys_projectAccounting {
	1763	IGCM_debug_PushStack "IGCM_sys_projectAccounting"
	1764	if ( $DEBUG_sys ) ; then
	1765	echo "IGCM_sys_check_quota"
	1766	fi
	1767
[1462]	1768	ssh irene191 /usr/bin/ccc_myproject > $1
[1286]	1769
	1770	IGCM_debug_PopStack "IGCM_sys_projectAccounting"
	1771	}
	1772
	1773	#D-#==================================================
[1244]	1774	#D-function IGCM_sys_getJobSchedulerID
	1775	#D-* Purpose: Get the job ID during execution
	1776	#D-* Examples: IGCM_sys_getJobSchedulerID jobSchedulerID
	1777	#D-
	1778	function IGCM_sys_getJobSchedulerID {
	1779	IGCM_debug_PushStack "IGCM_sys_getJobSchedulerID"
	1780	if ( $DEBUG_sys ) ; then
	1781	echo "IGCM_sys_getJobSchedulerID"
	1782	fi
	1783
	1784	eval ${1}=${BRIDGE_MSUB_JOBID}
	1785
	1786	IGCM_debug_PopStack "IGCM_sys_getJobSchedulerID"
	1787	}
	1788
	1789	#D-#==================================================
[958]	1790	#D-function IGCM_sys_GetJobID
[1244]	1791	#D-* Purpose: Get the job ID from the JobName
[958]	1792	#D-* Examples: IGCM_sys_GetJobID ${JobName} ${TargetUsr} JobID
	1793	#D-
	1794	function IGCM_sys_GetJobID {
	1795	IGCM_debug_PushStack "IGCM_sys_GetJobID"
	1796	if ( $DEBUG_sys ) ; then
	1797	echo "IGCM_sys_GetJobID"
	1798	fi
	1799
	1800	# With -f option, the full job name is given in the last column
	1801	ID=$( ccc_mstat -f -u $2 \| \
	1802	gawk -v JobName=$1 '( $NF ~ JobName ) { print $1 }' )
	1803
	1804	eval ${3}=${ID}
	1805	IGCM_debug_PopStack "IGCM_sys_GetJobID"
	1806	}
	1807
	1808	#D-#==================================================
[773]	1809	#D-function IGCM_sys_CountJobInQueue
[1244]	1810	#D-* Purpose: Count number of users job
[773]	1811	#D-* Examples: IGCM_sys_CountJobInQueue ${JobName} NbRun
	1812	#D-
	1813	function IGCM_sys_CountJobInQueue {
	1814	IGCM_debug_PushStack "IGCM_sys_CountJobInQueue"
	1815	if ( $DEBUG_sys ) ; then
	1816	echo "IGCM_sys_CountJobInQueue"
	1817	fi
	1818
	1819	# With -f option, the full job name is given in the last column
[834]	1820	NbRun=$( ccc_mstat -f \| gawk -v JobName=$1 'BEGIN { x=0 } ( $NF ~ JobName ) { x=x+1 } END { print x }' )
[773]	1821
	1822	eval ${2}=${NbRun}
	1823
	1824	IGCM_debug_PopStack "IGCM_sys_CountJobInQueue"
	1825	}
	1826
[955]	1827	#D-#==================================================
	1828	#D-function IGCM_sys_ListJobInQueue
[1244]	1829	#D-* Purpose: Produce a list of users computing jobs (excluding post-processing)
[955]	1830	#D-* Examples: IGCM_sys_ListJobInQueue ${User} JobNameList
	1831	#D-
	1832	function IGCM_sys_ListJobInQueue {
	1833	IGCM_debug_PushStack "IGCM_sys_ListJobInQueue"
	1834	if ( $DEBUG_sys ) ; then
	1835	echo "IGCM_sys_ListJobInQueue"
	1836	fi
	1837
	1838	# With -f option, the full job name is given in the last column
	1839	set -A JobList $( ccc_mstat -f \| gawk -v User=$1 \
	1840	'( $2 == User && \
	1841	$NF != /TS/ && \
	1842	$NF !~ /PACK/ && \
	1843	$NF !~ /REBUILD/ && \
	1844	$NF !~ /pack/ ) \
	1845	{ print $NF }' \| sed -e "s/$.$\.[0-9]/\1/" )
	1846
	1847	eval set -A ${2} ${JobList[*]}
	1848
	1849	IGCM_debug_PopStack "IGCM_sys_ListJobInQueue"
	1850	}
[1272]	1851
	1852	#D-#==================================================
	1853	#D-function IGCM_sys_atlas
	1854	#D-* Purpose: encapsulate atlas call so as to manage error code and curie specificity
	1855	#D-* Examples:
	1856	#D-
	1857	function IGCM_sys_atlas {
	1858	IGCM_debug_PushStack "IGCM_sys_atlas" $@
	1859	if ( $DEBUG_sys ) ; then
	1860	echo "IGCM_sys_atlas :" $@
	1861	fi
	1862
	1863	typeset status
	1864
	1865	\ccc_mprun atlas $@ > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
	1866	status=$?
	1867	if [ ${status} -gt 0 ] ; then
	1868	echo "IGCM_sys_atlas : error code ${status}"
	1869	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
	1870	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
	1871	IGCM_debug_PopStack "IGCM_sys_atlas"
	1872	return 1
	1873	else
	1874	IGCM_debug_PopStack "IGCM_sys_atlas"
	1875	return 0
	1876	fi
	1877
	1878	IGCM_debug_PopStack "IGCM_sys_atlas"
	1879	}
	1880
[1380]	1881	#D-#==================================================
	1882	#D-function IGCM_sys_rebuild_nemo
	1883	#D-* Purpose: rebuild nemo parallel files with nemo specific rebuild on curie
	1884	#D-* Examples:
	1885	#D-
	1886
	1887	function IGCM_sys_rebuild_nemo {
	1888	IGCM_debug_PushStack "IGCM_sys_rebuild_nemo" $@
	1889	if ( $DEBUG_sys ) ; then
	1890	echo "IGCM_sys_rebuild_nemo :" $@
	1891	fi
	1892
[1462]	1893	$( ccc_home -u igcmg)/Tools/irene/bin/rebuild_nemo ${1} ${2}
[1380]	1894
	1895	IGCM_debug_PopStack "IGCM_sys_rebuild_nemo"
	1896	}
	1897

Note: See TracBrowser for help on using the repository browser.

Context Navigation

source: trunk/libIGCM/libIGCM_sys/libIGCM_sys_irene.ksh @ 1624

Download in other formats: