Context Navigation

libIGCM_sys_jeanzay.ksh @ 1510

Last change on this file since 1510 was 1508, checked in by jgipsl, 5 years ago

Correct switch of multi-project using variable DataProject? or by the project set in the main job:
Use full path instead of variable $STORE, $WORK and $SCRATCH.

/gpfsstore/rech/${DataProject?}/${LOGIN}
/gpfswork/rech/${DataProject?}/${LOGIN}
/gpfsscratch/rech/${DataProject?}/${LOGIN}

This is done because "eval $(idrenv -d xxx)" can not be used in a job on computing nodes.

Property licence set to
The following licence information concerns ONLY the libIGCM tools
==================================================================

Copyright © Centre National de la Recherche Scientifique CNRS
Commissariat à l'Énergie Atomique CEA

libIGCM : Library for Portable Models Computation of IGCM Group.

IGCM Group is the french IPSL Global Climate Model Group.

This library is a set of shell scripts and functions whose purpose is
the management of the initialization, the launch, the transfer of
output files, the post-processing and the monitoring of datas produce
by any numerical program on any plateforme.

This software is governed by the CeCILL license under French law and
abiding by the rules of distribution of free software. You can use,
modify and/ or redistribute the software under the terms of the CeCILL
license as circulated by CEA, CNRS and INRIA at the following URL
"http://www.cecill.info".

As a counterpart to the access to the source code and rights to copy,
modify and redistribute granted by the license, users are provided only
with a limited warranty and the software's author, the holder of the
economic rights, and the successive licensors have only limited
liability.

In this respect, the user's attention is drawn to the risks associated
with loading, using, modifying and/or developing or reproducing the
software by the user in light of its specific status of free software,
that may mean that it is complicated to manipulate, and that also
therefore means that it is reserved for developers and experienced
professionals having in-depth computer knowledge. Users are therefore
encouraged to load and test the software's suitability as regards their
requirements in conditions enabling the security of their systems and/or
data to be ensured and, more generally, to use and operate it in the
same conditions as regards security.

The fact that you are presently reading this means that you have had
knowledge of the CeCILL license and that you accept its terms.
Property svn:keywords set to Revision Author Date

File size: 57.8 KB

Rev	Line
[622]	1	#!/bin/ksh
	2
	3	#**************************************************************
	4	# Author: Sebastien Denvil, Martial Mancip, Arnaud Caubel
	5	# Contact: Arnaud.Caubel__at__lsce.ipsl.fr
	6	# $Revision:: $ Revision of last commit
	7	# $Author:: $ Author of last commit
	8	# $Date:: $ Date of last commit
	9	# IPSL (2006)
	10	# This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
	11	#
	12	#**************************************************************
	13
	14	#=========================================================
	15	# The documentation of this file can be automatically generated
[623]	16	# if you use the prefix #D- for comments to be extracted.
[622]	17	# Extract with command: cat lib* \| grep "^#D-" \| cut -c "4-"
	18	#=========================================================
	19
	20	#D-#==================================================
[1490]	21	#D-LibIGCM_sys for Jean-Zay
[622]	22	#D-#==================================================
	23	#D-
[623]	24	#D- This ksh library if a layer under some usefull
[622]	25	#D-environment variables and shell commands.
	26	#D-All those definitions depend on host particularities.
	27	#D-It manages a stack mechanism and test validity of operations.
	28	#D-All function described bellow must be prefixed by IGCM_sys.
	29
	30	#====================================================
	31	# libIGCM_sys PARAMETERS
	32	#====================================================
	33
	34	#====================================================
	35	# set DEBUG_sys to true to output calls of function
	36	typeset -r DEBUG_sys=${DEBUG_sys:=true}
	37
	38	#====================================================
	39	# Turn in dry run mode ? (sys_Put_Rest, sys_Put_Out, sys_Get)
	40	typeset -r DRYRUN=${DRYRUN:=0}
	41
	42	# YOU MUST COMPILE YOUR EXE FILES FOR DRYRUN MODE !
	43	# -------------------------------------------------------------------------------------
	44	# \| DRYRUN= \| Date computations, \| sys_Get \| Exe \| sys_Put_Out; sys_Put_Rest \|
[809]	45	# \| \| Cp/Exe/param/files \| \| \| \|
	46	# \| \| Chmod Qsub \| \| \| \|
[622]	47	# -------------------------------------------------------------------------------------
	48	# \| 0 \| yes \| yes \| yes \| yes \|
	49	# -------------------------------------------------------------------------------------
	50	# \| 1 \| yes \| yes \| yes \| no \|
	51	# -------------------------------------------------------------------------------------
	52	# \| 2 \| yes \| yes \| no \| no \|
	53	# -------------------------------------------------------------------------------------
	54	# \| 3 \| yes \| no \| no \| no \|
	55	# -------------------------------------------------------------------------------------
	56
	57	#=====================================================
	58	# Global Variables :
	59	#=====================================================
	60	# Language : "fr" or "en"
	61	typeset -r MYLANG="fr"
	62
	63	#=====================================================
[1201]	64	# Host user names project maxCpuTime
[622]	65	# $hostname ou hostname
[1158]	66	typeset HOST=${HOST:=$( hostname )}
[622]	67	# $username ou whoami
[1158]	68	typeset LOGIN=${LOGIN:=$( whoami )}
[622]	69	# $hostname of the MASTER job
[1490]	70	typeset MASTER=jeanzay
	71	# add default project set by ins_job
	72	typeset PROJECT=$( echo ${SLURM_JOB_ACCOUNT:=::default_project::} \| cut -c "1-3" )
	73	### jobWarningDelay in seconds
	74	##typeset jobWarningDelay=${BRIDGE_MSUB_MAXTIME}
	75
[622]	76	#D-
	77	#D-#==================================================
	78	#D-Program used in libIGCM
	79	#D-#==================================================
	80
[742]	81	# Submit command
[1490]	82	typeset SUBMIT=${SUBMIT:=sbatch}
[622]	83	# rsync with path
	84	typeset -r RSYNC=/usr/bin/rsync
	85	# RSYNC_opt args to rsync
	86	typeset -r RSYNC_opt="-va"
	87	# ie storage filesystem
[623]	88	typeset -r STOREHOST=${MASTER}
[1180]	89	typeset -r REMOTE_RSYNC=/usr/bin/rsync
[622]	90
	91	#====================================================
[1490]	92	# Access to module command
	93	#====================================================
	94	. /usr/share/Modules/init/ksh
	95
	96	#====================================================
[1180]	97	# Set environment tools (ferret, nco, cdo, rebuild, ...)
[622]	98	#====================================================
[996]	99	if [ X${TaskType} = Xcomputing ] ; then
[1485]	100	IGCM_debug_Print 1 "Modules will be loaded later in IGCM_sys_activ_variables."
[996]	101	else
[1413]	102	module purge > /dev/null 2>&1
[1490]	103	. /gpfswork/rech/psl/commun/MachineEnvironment/jeanzay/env_jeanzay > /dev/null 2>&1
[1433]	104
[1502]	105	. /gpfswork/rech/psl/commun/MachineEnvironment/jeanzay/env_atlas_jeanzay > /dev/null 2>&1
	106
[1490]	107	# export PCMDI_MP=/ccc/work/cont003/igcmg/igcmg/PCMDI-MP
[1364]	108	export UVCDAT_ANONYMOUS_LOG=FALSE
[996]	109	fi
[1315]	110
	111	# FYI
[1210]	112	[ ! X${TaskType} = Xchecking ] && IGCM_debug_Print 1 "List of loaded modules:"
	113	[ ! X${TaskType} = Xchecking ] && module list
[1208]	114
[1495]	115	# For AddNoise (not yet AddPertu3DOCE)
	116	export PATH=${PATH}:/gpfswork/rech/psl/commun/Tools/AddNoise/bin
[622]	117
[1457]	118	# For rebuild
[1495]	119	export PATH=${PATH}:/gpfswork/rech/psl/commun/Tools/rebuild/modipsl_IOIPSL_PLUS_v2_2_4/bin
[622]	120
	121	#====================================================
	122	# Host specific DIRECTORIES
	123	#====================================================
	124
	125	#====================================================
[623]	126	#- MirrorlibIGCM for frontend
	127	typeset -r MirrorlibIGCM=${MirrorlibIGCM:=false}
	128
	129	#====================================================
	130	#- libIGCM_POST for frontend
	131	typeset -r libIGCM_POST=${libIGCM}
	132
	133	#====================================================
[622]	134	#- R_EXE (==> BIN_DIR = ${MODIPSL}/bin )
	135	typeset -r R_EXE="${MODIPSL}/bin"
	136
	137	#====================================================
	138	#- SUBMIT_DIR : submission dir
[1490]	139	typeset -x SUBMIT_DIR=${SUBMIT_DIR:=${SLURM_SUBMIT_DIR}}
[622]	140
	141	#====================================================
	142	#- IN
[1490]	143	typeset -r R_IN=${R_IN:=/gpfswork/rech/psl/commun/IGCM}
[622]	144
	145	#====================================================
	146	#- RUN_DIR_PATH : Temporary working directory (=> TMP)
[1490]	147	typeset -r RUN_DIR_PATH=${RUN_DIR_PATH:=${SCRATCH}/RUN_DIR/${SLURM_JOBID}_${$}}
[622]	148
	149	#====================================================
[1179]	150	#- OUTCOMMAND_PATH : tmp place to store command lines standard error and outputs
[1186]	151	typeset -r OUTCOMMAND_PATH=/tmp
[1179]	152
	153	#====================================================
[622]	154	#- HOST_MPIRUN_COMMAND
[1492]	155	typeset -r HOST_MPIRUN_COMMAND=${HOST_MPIRUN_COMMAND:="/usr/bin/time srun "}
[622]	156
	157	#====================================================
	158	#- Max number of arguments passed to nco operator or demigration command
	159	UNIX_MAX_LIMIT=360
	160
[623]	161	#====================================================
[1490]	162	#- set PackDefault to true
[623]	163	PackDefault=true
	164
[664]	165	#====================================================
[623]	166	#- Default number of MPI task for IPSL coupled model
	167	#- required for backward compatibility
	168	#-
	169	DEFAULT_NUM_PROC_OCE=5
	170	DEFAULT_NUM_PROC_CPL=1
[637]	171	(( DEFAULT_NUM_PROC_ATM = BATCH_NUM_PROC_TOT - DEFAULT_NUM_PROC_OCE - DEFAULT_NUM_PROC_CPL ))
[623]	172
[1426]	173	#====================================================
	174	#- Number of computing cores per node
	175	#-
[1492]	176	NB_CORE_PER_NODE=40
[1426]	177
[725]	178	#D-#==================================================
[1057]	179	#D-function IGCM_sys_defineArchives
	180	#D-* Purpose:
[1461]	181	#D-* Load dfldatadir module : According to project used for submission (default) or set in config.card (optional)
[1057]	182	#D-* Define ARCHIVE : Dedicated to large files
	183	#D-* Define STORAGE : Dedicated to small/medium files
	184	#D-* Define R_OUT : Output tree located on ARCHIVE
[1490]	185	#D-* Define R_FIG : Output tree located on STORAGE hosting figures (monsitoring and atlas, and/or small files)
	186	#D-* Define R_BUF : Output tree located on SCRATCH hosting files waiting for rebuild or pack processes
	187	#D-* if SpaceName=TEST everything is stored on SCRATCH
[725]	188	#D-* Examples:
	189	#D-
[1057]	190	function IGCM_sys_defineArchives {
	191	IGCM_debug_PushStack "IGCM_sys_defineArchives"
[676]	192
[1461]	193	# Load dfldatadir depending on the project used for submission (default) or set in config.card (optional)
	194	if [ X${config_UserChoices_DataProject} = X ] \|\| [ X${config_UserChoices_DataProject} = DEFAULT ]; then
[1490]	195	# The variable DataProject is not set in in section UserChoices in config.card or it is set to DEFAULT
[1508]	196	# Default option: Set DataProject according to the project used for submission
	197	DataProject=${PROJECT}
[1461]	198	else
	199	# Use the project set by the variable DataProject in section UserChoices in config.card
[1508]	200	IGCM_debug_Print 1 "Use DataProject as set in config.card : ${config_UserChoices_DataProject}"
	201	DataProject=${config_UserChoices_DataProject}
[1461]	202	fi
[1508]	203	IGCM_debug_Print 1 "Data project for output to be used: ${DataProject}"
[1461]	204
[1490]	205	#====================================================
	206	#- RUN_DIR_PATH : Temporary working directory (=> TMP)
[1508]	207	typeset -r RUN_DIR_PATH=${RUN_DIR_PATH:=/gpfsscratch/rech/${DataProject}/${LOGIN}/RUN_DIR/${SLURM_JOBID}_${$}}
[1474]	208
[1060]	209	if [ ! X${config_UserChoices_ARCHIVE} = X ]; then
[1057]	210	#====================================================
	211	#- ARCHIVE (dedicated to large files)
[1063]	212	ARCHIVE=${config_UserChoices_ARCHIVE}
[1057]	213	else
	214	#====================================================
	215	#- ARCHIVE (dedicated to large files)
[1508]	216	ARCHIVE=/gpfsstore/rech/${DataProject}/${LOGIN}
[1057]	217	fi
[725]	218
[1060]	219	if [ ! X${config_UserChoices_STORAGE} = X ]; then
[1057]	220	#====================================================
	221	#- STORAGE (dedicated to small/medium files)
[1063]	222	STORAGE=${config_UserChoices_STORAGE}
[1057]	223	else
	224	#====================================================
	225	#- STORAGE (dedicated to small/medium files)
[1508]	226	STORAGE=/gpfswork/rech/${DataProject}/${LOGIN}
[1057]	227	fi
[725]	228
[1057]	229	if [ X${config_UserChoices_SpaceName} = XTEST ]; then
	230	#====================================================
	231	#- R_OUT
[1508]	232	R_OUT=/gpfsscratch/rech/${DataProject}/${LOGIN}/IGCM_OUT
[1057]	233
	234	#====================================================
	235	#- R_FIG (hosting figures : monitoring and atlas, and/or small files)
[1508]	236	R_FIG=/gpfsscratch/rech/${DataProject}/${LOGIN}/IGCM_OUT
[1057]	237
	238	IGCM_debug_Print 1 "SpaceName=TEST ==> OVERRULE destination path directories"
	239
	240	else
	241	#====================================================
	242	#- R_OUT
[1063]	243	R_OUT=${ARCHIVE}/IGCM_OUT
[1057]	244
	245	#====================================================
	246	#- R_FIG (hosting figures : monitoring and atlas, and/or small files)
[1063]	247	R_FIG=${STORAGE}/IGCM_OUT
[1057]	248	fi
	249
	250	#====================================================
[1431]	251	#- CMIP6 (hosting CMIP6 files produced by XIOS2 and configured by dr2xml)
	252	CMIP6_BUF=${STORAGE}/IGCM_OUT
	253
	254	#====================================================
[1057]	255	#- R_BUF (ONLY FOR double copy an scratch)
[1508]	256	R_BUF=/gpfsscratch/rech/${DataProject}/${LOGIN}/IGCM_OUT
[1057]	257
	258	IGCM_debug_Print 1 "R_OUT has been defined = ${R_OUT}"
	259	IGCM_debug_Print 1 "R_BUF has been defined = ${R_BUF}"
	260	IGCM_debug_Print 1 "R_FIG has been defined = ${R_FIG}"
	261
	262	IGCM_debug_PopStack "IGCM_sys_defineArchives"
[725]	263	}
	264
[622]	265	#D-#==================================================
	266	#D-function IGCM_sys_RshArchive
	267	#D-* Purpose: Archive rsh command
	268	#D-* Examples:
	269	#D-
	270	function IGCM_sys_RshArchive {
[623]	271	IGCM_debug_PushStack "IGCM_sys_RshArchive" $@
	272	/bin/ksh <<-EOF
[622]	273	${@}
	274	EOF
[811]	275	status=$?
	276	if [ ${status} -gt 0 ] ; then
	277	IGCM_debug_Print 2 "IGCM_sys_RshArchive : command failed error code ${status}"
[623]	278	IGCM_debug_Exit "IGCM_sys_RshArchive"
	279	fi
	280	IGCM_debug_PopStack "IGCM_sys_RshArchive"
[622]	281	}
	282
	283	#D-#==================================================
[845]	284	#D-function IGCM_sys_RshArchive_NoError
	285	#D-* Purpose: Archive rsh command, without error
	286	#D-* used only in monitoring.job
	287	#D-* Examples:
	288	#D-
	289	function IGCM_sys_RshArchive_NoError {
	290	IGCM_debug_PushStack "IGCM_sys_RshArchive_NoError" $@
	291	/bin/ksh <<-EOF
[1181]	292	${@} 2> /dev/null
[845]	293	EOF
	294	IGCM_debug_PopStack "IGCM_sys_RshArchive_NoError"
	295	}
	296
	297	#D-#==================================================
[622]	298	#D-function IGCM_sys_MkdirArchive
	299	#D-* Purpose: Mkdir on Archive
	300	#D-* Examples:
	301	#D-
	302	function IGCM_sys_MkdirArchive {
[623]	303	IGCM_debug_PushStack "IGCM_sys_MkdirArchive" $@
	304	if ( $DEBUG_sys ) ; then
	305	echo "IGCM_sys_MkdirArchive :" $@
	306	fi
	307	#- creation de repertoire sur le serveur fichier
	308	if [ ! -d ${1} ]; then
	309	\mkdir -p $1
[811]	310	status=$?
	311
	312	if [ ${status} -gt 0 ] ; then
	313	IGCM_debug_Print 2 "IGCM_sys_MkdirArchive : mkdir failed error code ${status}"
[623]	314	IGCM_debug_Exit "IGCM_sys_MkdirArchive"
[622]	315	fi
[623]	316	fi
	317	IGCM_debug_PopStack "IGCM_sys_MkdirArchive"
[622]	318	}
	319
	320	#D-#==================================================
	321	#D-function IGCM_sys_TestDirArchive
	322	#D-* Purpose: Test Directory that must exists on Archive
	323	#D-* Examples:
	324	#D-
	325	function IGCM_sys_TestDirArchive {
[623]	326	IGCM_debug_PushStack "IGCM_sys_TestDirArchive" $@
	327	if ( $DEBUG_sys ) ; then
	328	echo "IGCM_sys_TestDirArchive :" $@
	329	fi
	330	typeset ExistFlag
	331	ExistFlag=$( [ -d $1 ] && echo 0 \|\| echo 1 )
	332	IGCM_debug_PopStack "IGCM_sys_TestDirArchive"
	333	return ${ExistFlag}
[622]	334	}
	335
	336	#D-#==================================================
[623]	337	#D-function IGCM_sys_IsFileArchived
[1179]	338	#D-* Purpose: Test file that must NOT EXISTS on Archive based on filename only
[623]	339	#D-* Examples:
	340	#D-
	341	function IGCM_sys_IsFileArchived {
	342	IGCM_debug_PushStack "IGCM_sys_IsFileArchived" $@
	343	if ( $DEBUG_sys ) ; then
	344	echo "IGCM_sys_IsFileArchived :" $@
	345	fi
	346	typeset IsArchivedFlag
[1502]	347	# To be implemented on Jean Zay
	348	# IsArchivedFlag=$( [ X$( echo $@ \| grep ^\/ccc\/store ) != X ] && echo 0 \|\| echo 1 )
[623]	349	IGCM_debug_PopStack "IGCM_sys_IsFileArchived"
	350
	351	return ${IsArchivedFlag}
	352	}
	353
	354	#D-#==================================================
[622]	355	#D-function IGCM_sys_TestFileArchive
	356	#D-* Purpose: Test file that must NOT EXISTS on Archive
	357	#D-* Examples:
	358	#D-
	359	function IGCM_sys_TestFileArchive {
[623]	360	IGCM_debug_PushStack "IGCM_sys_TestFileArchive" $@
	361	typeset ExistFlag
	362	ExistFlag=$( [ -f $1 ] && echo 0 \|\| echo 1 )
	363	IGCM_debug_PopStack "IGCM_sys_TestFileArchive"
[622]	364
[623]	365	return ${ExistFlag}
[622]	366	}
	367
	368	#D-#==================================================
	369	#D-function IGCM_sys_CountFileArchive
	370	#D-* Purpose: Count files on Archive filesystem
	371	#D-* Examples:
	372	#D-
	373	function IGCM_sys_CountFileArchive {
[623]	374	IGCM_debug_PushStack "IGCM_sys_CountFileArchive" $@
	375	ls ${@} 2>/dev/null \| wc -l
	376	if [ $? -gt 0 ] ; then
	377	echo "IGCM_sys_CountFileArchive : erreur."
	378	fi
	379	IGCM_debug_PopStack "IGCM_sys_CountFileArchive"
[622]	380	}
	381
	382	#D-#==================================================
	383	#D-function IGCM_sys_Tree
	384	#D-* Purpose: Tree directories with files on ${ARCHIVE}
	385	#D-* Examples: IGCM_sys_Tree ${R_IN} ${R_OUT}
	386	#D-
	387	function IGCM_sys_Tree {
[623]	388	IGCM_debug_PushStack "IGCM_sys_Tree" $@
	389	if ( $DEBUG_sys ) ; then
	390	echo "IGCM_sys_Tree :" $@
	391	fi
[622]	392
[623]	393	\ls -lR ${@}
[622]	394
[623]	395	IGCM_debug_PopStack "IGCM_sys_Tree"
[622]	396	}
	397
	398	#D-#==================================================
	399	#D-function IGCM_sys_Qsub
	400	#D-* Purpose: Qsub new job
	401	#D-* Examples:
	402	#D-
	403	function IGCM_sys_Qsub {
[623]	404	IGCM_debug_PushStack "IGCM_sys_Qsub" $@
	405	if ( $DEBUG_sys ) ; then
	406	echo "IGCM_sys_Qsub :" $@
	407	fi
[811]	408	typeset options status
[923]	409	options="-o ${SUBMIT_DIR}/${Script_Output} -e ${SUBMIT_DIR}/${Script_Output}"
[622]	410
[1490]	411	/usr/bin/time sbatch ${options} $1 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[811]	412	status=$?
[948]	413
[1179]	414	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[811]	415	if [ ${status} -gt 0 ] ; then
	416	IGCM_debug_Print 2 "IGCM_sys_Qsub ${options} $1 : error code ${status}"
[623]	417	IGCM_debug_Exit "IGCM_sys_Qsub"
[948]	418	else
[1179]	419	JobID=$( gawk {'print $4'} ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ )
[623]	420	fi
	421	IGCM_debug_PopStack "IGCM_sys_Qsub"
[622]	422	}
	423
	424	#D-#==================================================
	425	#D-function IGCM_sys_QsubPost
	426	#D-* Purpose: Qsub new job on scalaire
	427	#D-* Examples:
	428	#D-
	429	function IGCM_sys_QsubPost {
[623]	430	IGCM_debug_PushStack "IGCM_sys_QsubPost" $@
	431	if ( $DEBUG_sys ) ; then
	432	echo "IGCM_sys_QsubPost :" $@
	433	fi
[923]	434	typeset options status
[1490]	435	options="-o ${POST_DIR}/${Script_Post_Output}.out -e ${POST_DIR}/${Script_Post_Output}.out"
[923]	436
[1490]	437	sbatch ${options} ${libIGCM_POST}/$1.job > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[811]	438	status=$?
[948]	439
[1179]	440	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[811]	441	if [ ${status} -gt 0 ] ; then
[923]	442	IGCM_debug_Print 2 "IGCM_sys_QsubPost ${options} ${libIGCM_POST}/$1.job : error code ${status}"
[623]	443	IGCM_debug_Exit "IGCM_sys_QsubPost"
[948]	444	else
[1179]	445	JobID=$( gawk {'print $4'} ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ )
[623]	446	fi
	447	IGCM_debug_PopStack "IGCM_sys_QsubPost"
[622]	448	}
	449
	450	#D-*************************
[623]	451	#D- File transfer functions
[622]	452	#D-*************************
	453	#D-
	454
	455	#D-#==================================================
	456	#D-function IGCM_sys_RmRunDir
	457	#D-* Purpose: rm tmpdir (dummy function most of the time batch
	458	#D- scheduler will do the job)
	459	#D-* Examples:
	460	#D-
	461	function IGCM_sys_RmRunDir {
[1057]	462	IGCM_debug_PushStack "IGCM_sys_RmRunDir" $@
[623]	463	if ( $DEBUG_sys ) ; then
	464	echo "IGCM_sys_RmRunDir :" $@
	465	fi
[622]	466
[715]	467	typeset status
[622]	468
[1179]	469	echo rm $@ > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
	470	\rm $@ >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[715]	471	status=$?
[622]	472
[715]	473	if [ ${status} -gt 0 ] ; then
[1183]	474	IGCM_debug_Print 1 "IGCM_sys_RmRunDir : rm error code is ${status}."
[1179]	475	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[623]	476	IGCM_debug_Exit "IGCM_sys_RmRunDir"
	477	else
[1179]	478	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[623]	479	fi
	480	IGCM_debug_PopStack "IGCM_sys_RmRunDir"
[622]	481	}
	482
	483	#D-#==================================================
	484	#D-function IGCM_sys_Put_Dir
	485	#D-* Purpose: Copy a complete directory on $(ARCHIVE)
	486	#D-* Examples:
	487	#D-
	488	function IGCM_sys_Put_Dir {
[623]	489	IGCM_debug_PushStack "IGCM_sys_Put_Dir" $@
	490	if ( $DEBUG_sys ) ; then
	491	echo "IGCM_sys_Put_Dir :" $@
	492	fi
	493	if [ $DRYRUN = 0 ]; then
	494	if [ ! -d ${1} ] ; then
	495	echo "WARNING : IGCM_sys_Put_Dir ${1} DOES NOT EXIST ."
	496	IGCM_debug_PopStack "IGCM_sys_Put_Dir"
	497	return
[622]	498	fi
	499
[715]	500	typeset status
[622]	501
[623]	502	# Only if we use rsync
	503	#IGCM_sys_TestDirArchive $( dirname $2 )
	504	#
	505	#USUAL WAY
[1179]	506	\cp -r $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[715]	507	status=$?
[622]	508
[715]	509	if [ ${status} -gt 0 ] ; then
[811]	510	IGCM_debug_Print 2 "IGCM_sys_Put_Dir : cp failed error code ${status}"
[1179]	511	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[623]	512	IGCM_debug_Exit "IGCM_sys_Put_Dir"
[622]	513	else
[1179]	514	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[622]	515	fi
[623]	516	fi
	517	IGCM_debug_PopStack "IGCM_sys_Put_Dir"
[622]	518	}
	519
	520	#D-#==================================================
	521	#D-function IGCM_sys_Get_Dir
[623]	522	#D-* Purpose: Copy a complete directory from ${ARCHIVE}
[622]	523	#D-* Examples:
	524	#D-
	525	function IGCM_sys_Get_Dir {
[623]	526	IGCM_debug_PushStack "IGCM_sys_Get_Dir" $@
	527	if ( $DEBUG_sys ) ; then
	528	echo "IGCM_sys_Get_Dir :" $@
	529	fi
	530	if [ $DRYRUN = 0 ]; then
[708]	531	typeset NB_ESSAI DELAI status i
	532	# number of tentative
	533	NB_ESSAI=3
	534	# time delay between tentative
	535	DELAI=2
[622]	536
[676]	537	#
[708]	538	# USUAL WAY
[1502]	539	# add here command to demigrate all offline files to reduce time of this command
	540	# actually on Jean Zay files are not archive on bands
[622]	541
[709]	542	i=0
[708]	543	while [ $i -lt $NB_ESSAI ] ; do
[1179]	544	\cp -ur $1 $2 >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[708]	545	status=$?
[811]	546	if [ ${status} -gt 0 ] ; then
[713]	547	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : cp failed error code ${status} ${i}/${NB_ESSAI}"
[708]	548	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : sleep ${DELAI} seconds and try again."
	549	sleep $DELAI
	550	else
	551	break
	552	fi
	553	(( i = i + 1 ))
	554	done
	555
	556	if [ ${status} -gt 0 ] ; then
[1178]	557	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : cp failed error code ${status}"
[1179]	558	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[623]	559	IGCM_debug_Exit "IGCM_sys_Get_Dir"
[622]	560	else
[1179]	561	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[622]	562	fi
[623]	563	fi
	564	IGCM_debug_PopStack "IGCM_sys_Get_Dir"
[622]	565	}
	566
	567	#D-#==================================================
	568	#D-function IGCM_sys_Put_Rest
[623]	569	#D-* Purpose: Put computied restarts on ${ARCHIVE}.
[622]	570	#D- File and target directory must exist.
	571	#D-* Examples:
	572	#D-
	573	function IGCM_sys_Put_Rest {
[623]	574	IGCM_debug_PushStack "IGCM_sys_Put_Rest" $@
	575	if ( $DEBUG_sys ) ; then
	576	echo "IGCM_sys_Put_Rest :" $@
	577	fi
	578	if [ $DRYRUN = 0 ]; then
	579	if [ ! -f ${1} ] ; then
	580	echo "ERROR : IGCM_sys_Put_Rest ${1} DOES NOT EXIST ."
	581	IGCM_debug_Exit "IGCM_sys_Put_Rest"
[622]	582	fi
	583
[715]	584	typeset status
[623]	585	#
	586	# USUAL WAY
[1179]	587	\cp $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[715]	588	status=$?
[622]	589
[623]	590	# #RSYNC WITH NETWORK SSH CALL
[1179]	591	# echo ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
	592	# ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[622]	593
[623]	594	# #RSYNC WITH NFS USE
[1179]	595	# echo ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
	596	# ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[622]	597
[715]	598	# status=$?
	599	# IGCM_sys_Rsync_out $status
[622]	600
[1179]	601	# ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[715]	602	# (( status=status+$? ))
[623]	603
[715]	604	if [ ${status} -gt 0 ] ; then
[1179]	605	IGCM_debug_Print 2 "IGCM_sys_Put_Rest : cp failed error code ${status}"
[765]	606	[ -f ${1} ] && ls -l ${1}
[759]	607	[ -f ${2} ] && ls -l ${2}
	608	[ -f ${2}/${1} ] && ls -l ${2}/${1}
[1179]	609	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[623]	610	IGCM_debug_Exit "IGCM_sys_Put_Rest"
[622]	611	else
[759]	612
	613	if [ X${JobType} = XRUN ] ; then
	614	[ -f ${2} ] && IGCM_sys_Chmod 444 ${2}
	615	[ -f ${2}/${1} ] && IGCM_sys_Chmod 444 ${2}/${1}
	616	fi
	617
[1179]	618	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[622]	619	fi
[623]	620	fi
	621	IGCM_debug_PopStack "IGCM_sys_Put_Rest"
[622]	622	}
	623
	624	#D-#==================================================
	625	#D-function IGCM_sys_Put_Out
[623]	626	#D-* Purpose: Copy a file on ${ARCHIVE} after having chmod it in readonly
[622]	627	#D-* Examples:
	628	#D-
	629	function IGCM_sys_Put_Out {
[623]	630	IGCM_debug_PushStack "IGCM_sys_Put_Out" $@
	631	if ( $DEBUG_sys ) ; then
	632	echo "IGCM_sys_Put_Out :" $@
	633	fi
[667]	634
[711]	635	typeset NB_ESSAI DELAI status i exist skip
[1234]	636	typeset fileDeviceNumberInHex directoryDeviceNumberInHex
[667]	637
	638	# number of tentative
[668]	639	NB_ESSAI=3
[667]	640	# time delay between tentative
[668]	641	DELAI=2
[667]	642
[623]	643	if [ $DRYRUN = 0 ]; then
	644	if [ ! -f ${1} ] ; then
	645	echo "WARNING : IGCM_sys_Put_Out ${1} DOES NOT EXIST ."
	646	IGCM_debug_PopStack "IGCM_sys_Put_Out"
	647	return 1
[622]	648	fi
[623]	649	#
	650	IGCM_sys_MkdirArchive $( dirname $2 )
	651	#
[711]	652	exist=false
	653	skip=false
	654	if [ -f $2 ] ; then
	655	IGCM_debug_Print 1 "$2 already exist"
[1502]	656	# add here command to demigrate all offline files to reduce time of this command
	657	# actually on Jean Zay files are not archive on bands
	658
[711]	659	exist=true
	660	if [ "X$( diff $1 $2 )" = X ] ; then
	661	IGCM_debug_Print 2 "$1 and $2 are the same file, we skip the copy"
	662	skip=true
[667]	663	else
[711]	664	IGCM_debug_Print 2 "$1 and $2 are not the same file, we force the copy"
	665	skip=false
[667]	666	fi
[711]	667	fi
	668	#
	669	if ( [ X${exist} = Xtrue ] && [ X${skip} = Xfalse ] ) ; then
	670	IGCM_sys_Chmod u+w $2
	671	fi
[622]	672
[711]	673	if [ X${skip} = Xfalse ] ; then
[719]	674	i=0
[711]	675	while [ $i -lt $NB_ESSAI ] ; do
[1234]	676	# Identify file system
	677	fileDeviceNumberInHex=$( stat -c %d $1 )
	678	status=$?
	679	if [ ${status} -gt 0 ] ; then
	680	IGCM_debug_Exit "IGCM_sys_Put_Out"
	681	fi
	682	# Identify file system
	683	directoryDeviceNumberInHex=$( stat -c %d $( dirname $2 ) )
	684	status=$?
	685	if [ ${status} -gt 0 ] ; then
	686	IGCM_debug_Exit "IGCM_sys_Put_Out"
	687	fi
	688
[1412]	689	if ( [ ${fileDeviceNumberInHex} -ne ${directoryDeviceNumberInHex} ] \|\| [ X$3 = XNOMOVE ] ) ; then
[1234]	690	# They are not on the same device. USUAL WAY
[1179]	691	\cp $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[1037]	692	status=$?
	693	else
[1234]	694	# They are on the same device. NOT SO USUAL WAY
[1179]	695	\mv $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[1037]	696	status=$?
	697	fi
[711]	698	if [ ${status} -gt 0 ]; then
[713]	699	IGCM_debug_Print 2 "IGCM_sys_Put_Out : cp failed error code ${status} ${i}/${NB_ESSAI}"
[711]	700	IGCM_debug_Print 2 "IGCM_sys_Put_Out : sleep ${DELAI} seconds and try again."
[765]	701	[ -f ${1} ] && ls -l ${1}
[759]	702	[ -f ${2} ] && ls -l ${2}
	703	[ -f ${2}/${1} ] && ls -l ${2}/${1}
[711]	704	sleep $DELAI
	705	else
	706	break
	707	fi
	708	(( i = i + 1 ))
	709	done
	710	fi
	711
[623]	712	# #RSYNC WITH NETWORK SSH CALL
[1179]	713	# echo ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
	714	# ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[622]	715
[623]	716	# #RSYNC WITH NFS USE
[1179]	717	# echo ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
	718	# ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[622]	719
[711]	720	# status=$?
	721	# IGCM_sys_Rsync_out $status
[622]	722
[1179]	723	# ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[711]	724	# (( status=status+$? ))
[622]	725
[711]	726	if [ ${status} -gt 0 ] ; then
[1179]	727	IGCM_debug_Print 2 "IGCM_sys_Put_Out : cp failed error code ${status}"
[765]	728	[ -f ${1} ] && ls -l ${1}
[759]	729	[ -f ${2} ] && ls -l ${2}
	730	[ -f ${2}/${1} ] && ls -l ${2}/${1}
[1179]	731	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[623]	732	IGCM_debug_Exit "IGCM_sys_Put_Out"
[622]	733	else
[759]	734
	735	if [ X${JobType} = XRUN ] ; then
	736	if [ X${3} = X ] ; then
	737	[ -f ${2} ] && IGCM_sys_Chmod 444 ${2}
	738	[ -f ${2}/${1} ] && IGCM_sys_Chmod 444 ${2}/${1}
	739	fi
	740	fi
	741
[1179]	742	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[622]	743	fi
[623]	744	fi
	745	IGCM_debug_PopStack "IGCM_sys_Put_Out"
	746	return 0
[622]	747	}
	748
	749	#D-#==================================================
	750	#D-function IGCM_sys_Get
	751	#D-* Purpose: Get a file from ${ARCHIVE}
	752	#D-* Examples: IGCM_sys_Get myfile /destpath/myfile_with_PREFIX
	753	#D- IGCM_sys_Get /l Array_contain_myfiles /destpath/
	754	function IGCM_sys_Get {
[623]	755	IGCM_debug_PushStack "IGCM_sys_Get" $@
[622]	756
[773]	757	typeset DEST dm_liste target file_work
[667]	758	typeset NB_ESSAI DELAI status i
[622]	759
[623]	760	if ( $DEBUG_sys ) ; then
	761	echo "IGCM_sys_Get :" $@
	762	fi
[667]	763
	764	# number of tentative
[668]	765	NB_ESSAI=3
[667]	766	# time delay between tentative
[668]	767	DELAI=2
[667]	768
[623]	769	if [ $DRYRUN -le 2 ]; then
	770	if [ X${1} = X'/l' ] ; then
	771	eval set +A dm_liste \${${2}}
	772	else
	773	eval set +A dm_liste ${1}
[622]	774	fi
[623]	775	eval DEST=\${${#}}
[1502]	776	# add here command to demigrate all offline files to reduce time of this command
	777	# actually on Jean Zay files are not archive on bands
	778
[715]	779	status=$?
	780	if [ ${status} -gt 0 ] ; then
	781	echo "WARNING IGCM_sys_Get : error code ${status}"
[1179]	782	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[623]	783	echo "WARNING IGCM_sys_Get : will stop later if the cp fails."
	784	fi
[622]	785
[715]	786	#if [ ${status} -gt 0 ] ; then
[1179]	787	# if [ ! "X$( grep "Lost dmusrcmd connection" ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ )" = "X" ] ; then
	788	# cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[785]	789	# echo "WARNING IGCM_sys_Get : Lost dmusrcmd connection : "
	790	# sleep 30
	791	# echo "We try another time"
[1179]	792	## dmget ${dm_liste[*]} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
	793	# ccc_hsm get ${dm_liste[*]} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[785]	794	# status=$?
	795	# if [ ${status} -gt 0 ] ; then
	796	# echo "ERROR IGCM_sys_Get : again demigration error :"
[1179]	797	# cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[785]	798	# IGCM_debug_Exit "IGCM_sys_Get"
[623]	799	# fi
[785]	800	# else
	801	# echo "ERROR IGCM_sys_Get : demigration error :"
[1179]	802	# cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[785]	803	# IGCM_debug_Exit "IGCM_sys_Get"
	804	# fi
[623]	805	#fi
[622]	806
[719]	807	# #RSYNC WITH NETWORK SSH CALL
[1179]	808	# echo ${RSYNC} ${RSYNC_opt} -e ssh ${STOREHOST}:"${dm_liste}" ${STOREHOST}:${RUN_DIR}/${DEST} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
	809	# ${RSYNC} ${RSYNC_opt} -e ssh ${STOREHOST}:"${dm_liste}" ${STOREHOST}:${RUN_DIR}/${DEST} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[719]	810
	811	# #RSYNC WITH NFS USE
[1179]	812	# echo ${RSYNC} ${RSYNC_opt} ${dm_liste} ${RUN_DIR}/${DEST} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
	813	# ${RSYNC} ${RSYNC_opt} ${dm_liste} ${RUN_DIR}/${DEST} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[719]	814
	815	# status=$?
	816	# IGCM_sys_Rsync_out $status
	817
[1179]	818	# ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[719]	819	# (( status=status+$? ))
	820
[623]	821	#USUAL WAY
	822	if [ X${1} = X'/l' ] ; then
	823	for target in ${dm_liste[*]} ; do
	824	local_file=$( basename ${target} )
[668]	825	# test if the target file is present before the loop
	826	IGCM_sys_TestFileArchive ${target}
[715]	827	status=$?
	828	if [ ${status} -gt 0 ] ; then
[701]	829	echo "IGCM_sys_Get, ERROR : regular file ${target} DOES NOT EXIST ."
[668]	830	IGCM_debug_Exit "IGCM_sys_Get"
	831	else
[719]	832	i=0
[668]	833	while [ $i -lt $NB_ESSAI ] ; do
[1037]	834	#if [ X${DoLink} = Xtrue ] ; then
[1179]	835	# \ln -s ${target} ${DEST}/${local_file} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[1037]	836	# status=$?
	837	# else
[1179]	838	# \cp ${target} ${DEST}/${local_file} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[1037]	839	# status=$?
	840	#fi
[1179]	841	\ln -s ${target} ${DEST}/${local_file} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[668]	842	status=$?
	843	if [ ${status} -gt 0 ]; then
[713]	844	IGCM_debug_Print 2 "IGCM_sys_Get : cp failed error code ${status} ${i}/${NB_ESSAI}"
[676]	845	IGCM_debug_Print 2 "IGCM_sys_Get : sleep ${DELAI} seconds and try again."
	846	sleep $DELAI
[668]	847	else
[676]	848	break
[668]	849	fi
	850	(( i = i + 1 ))
	851	done
[719]	852	if [ ${status} -gt 0 ] ; then
	853	echo "IGCM_sys_Get : error"
[1179]	854	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
	855	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[719]	856	IGCM_debug_Exit "IGCM_sys_Get"
	857	else
[1179]	858	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[719]	859	fi
[668]	860	fi
[623]	861	done
	862	else
[719]	863	i=0
	864	while [ $i -lt $NB_ESSAI ] ; do
[1179]	865	\cp ${dm_liste} ${DEST} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[719]	866	status=$?
	867	if [ ${status} -gt 0 ]; then
	868	IGCM_debug_Print 2 "IGCM_sys_Get : cp failed error code ${status} ${i}/${NB_ESSAI}"
	869	IGCM_debug_Print 2 "IGCM_sys_Get : sleep ${DELAI} seconds and try again."
	870	sleep $DELAI
	871	else
	872	break
	873	fi
	874	(( i = i + 1 ))
	875	done
	876	if [ ${status} -gt 0 ] ; then
	877	echo "IGCM_sys_Get : error"
[1179]	878	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
	879	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[719]	880	IGCM_debug_Exit "IGCM_sys_Get"
	881	else
[1179]	882	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[719]	883	fi
[623]	884	fi
	885	fi
	886	IGCM_debug_PopStack "IGCM_sys_Get"
	887	}
[622]	888
[623]	889	#D-#==================================================
[835]	890	#D-function IGCM_sys_GetDate_Monitoring
	891	#D-* Purpose: get the last year for which the monitoring has been computed
	892	#D-* Examples:
	893	#D-
	894	function IGCM_sys_GetDate_Monitoring {
	895	IGCM_debug_PushStack "IGCM_sys_GetDate_Monitoring" $@
	896	if ( $DEBUG_sys ) ; then
	897	echo "IGCM_sys_GetDate_Monitoring :" $@
	898	fi
	899
	900	eval ${2}=$( cdo showyear ${1} 2> /dev/null \| gawk '{ print $NF }' )
	901
	902	IGCM_debug_PopStack "IGCM_sys_GetDate_Monitoring"
	903	}
	904
	905	#D-#==================================================
[623]	906	#D-function IGCM_sys_Dods_Rm
	907	#D-* Purpose: DO NOTHING ! Put ${ARCHIVE} files on DODS internet protocole.
	908	#D-* Examples:
	909	#D-
	910	function IGCM_sys_Dods_Rm {
	911	if ( $DEBUG_sys ) ; then
	912	echo "IGCM_sys_Dods_Rm :" $@
	913	fi
[715]	914	typeset status
[623]	915	if [ $DRYRUN = 0 ]; then
	916
	917	# if [ ! -d /dmnfs/cont003/dods/public/${LOGIN}/${R_DODS}/${1} ] ; then
	918	# echo "WARNING : IGCM_sys_Dods_Rm /dmnfs/cont003/dods/public/${LOGIN}/${R_DODS}/${1} DOES NOT EXIST ."
	919	# echo "Nothing has been done."
	920	# return
	921	# fi
	922
[1392]	923	if [ "$#" -eq 1 ]; then
[1502]	924	# $(ccc_home -u igcmg)/Tools/irene/thredds_rm public/${LOGIN}/${R_DODS}/${1} # > out_dods_rm 2>&1
[1392]	925	status=$?
	926	else
[1502]	927	# $(ccc_home -u igcmg)/Tools/irene/thredds_rm public/${LOGIN}/${R_DODS} # > out_dods_rm 2>&1
[1392]	928	status=$?
	929	fi
[623]	930
[785]	931	# if [ ${status} -gt 0 ] ; then
	932	# echo "IGCM_sys_Dods_Rm : error."
	933	# cat out_dods_rm
	934	# IGCM_debug_Exit "IGCM_sys_Dods_Rm"
	935	# else
	936	# rm out_dods_rm
	937	# fi
[623]	938
	939	fi
[715]	940	return $status
[623]	941	}
	942
	943	#D-#==================================================
	944	#D-function IGCM_sys_Dods_Cp
	945	#D-* Purpose: Copy $(ARCHIVE) files on DODS internet protocole.
	946	#D-* Examples:
	947	#D-
	948	function IGCM_sys_Dods_Cp {
	949	if ( $DEBUG_sys ) ; then
	950	echo "IGCM_sys_Dods_Cp :" $@
	951	fi
[715]	952	typeset status
[623]	953	if [ $DRYRUN = 0 ]; then
	954
	955	# if [ ! -d ${R_SAVE}/${1} ] ; then
	956	# echo "WARNING : IGCM_sys_Dods_Cp ${R_SAVE}/${1} DOES NOT EXIST ."
	957	# echo "Nothing has been done."
	958	# return
	959	# fi
	960
[1502]	961	# $(ccc_home -u igcmg)/Tools/irene/thredds_cp ${1} public/${LOGIN}/${R_DODS} # > out_dods_cp 2>&1
	962
[715]	963	status=$?
[623]	964
[715]	965	# if [ ${status} -gt 0 ] ; then
[623]	966	# echo "IGCM_sys_Dods_Cp : error."
	967	# cat out_dods_cp
	968	# IGCM_debug_Exit "IGCM_sys_Dods_Cp"
	969	# else
	970	# rm out_dods_cp
	971	# fi
	972
	973	fi
[715]	974	return $status
[623]	975	}
	976
	977	#D-#==================================================
	978	#D-function IGCM_sys_Put_Dods
[1178]	979	#D-* Purpose: Put ${ARCHIVE} files on DODS internet protocole.
[623]	980	#D-* Examples:
	981	#D-
	982	function IGCM_sys_Put_Dods {
	983	IGCM_debug_PushStack "IGCM_sys_Put_Dods" $@
	984	if ( $DEBUG_sys ) ; then
	985	echo "IGCM_sys_Put_Dods :" $@
	986	fi
[715]	987	typeset status
[623]	988	if [ $DRYRUN = 0 ]; then
	989	if ( [ ! -d ${R_SAVE}/${1} ] && [ ! -d ${R_FIGR}/${1} ] ) ; then
	990	echo "WARNING IGCM_sys_Put_Dods : None of the following directories exist. Exactly one should."
	991	echo "WARNING IGCM_sys_Put_Dods : ${R_SAVE}/${1} DOES NOT EXIST."
	992	echo "WARNING IGCM_sys_Put_Dods : ${R_FIGR}/${1} DOES NOT EXIST."
	993	IGCM_debug_PopStack "IGCM_sys_Put_Dods"
	994	return
[622]	995	fi
[623]	996
	997	if ( [ -d ${R_SAVE}/${1} ] && [ -d ${R_FIGR}/${1} ] ) ; then
	998	echo "WARNING IGCM_sys_Put_Dods : Both of the following directories exist. Exactly one should."
	999	echo "WARNING IGCM_sys_Put_Dods : ${R_SAVE}/${1} EXISTS."
	1000	echo "WARNING IGCM_sys_Put_Dods : ${R_FIGR}/${1} EXISTS."
	1001	IGCM_debug_PopStack "IGCM_sys_Put_Dods"
	1002	return
	1003	fi
	1004	#
	1005	if [ -d ${R_SAVE}/${1} ] ; then
	1006	cd ${R_SAVE}
	1007	elif [ -d ${R_FIGR}/${1} ] ; then
	1008	cd ${R_FIGR}
	1009	fi
	1010
	1011	IGCM_sys_Dods_Rm ${1}
	1012	IGCM_sys_Dods_Cp ${1}
[715]	1013	status=0
[623]	1014
[715]	1015	if [ ${status} -gt 0 ] ; then
[623]	1016	echo "IGCM_sys_Put_Dods : error."
	1017	IGCM_debug_Exit "IGCM_sys_Put_Dods"
	1018	fi
	1019	fi
	1020	IGCM_debug_PopStack "IGCM_sys_Put_Dods"
[622]	1021	}
	1022
	1023	##############################################################
	1024	# REBUILD OPERATOR
	1025
[811]	1026	#D-#==================================================
[1163]	1027	#D-function IGCM_sys_sync
	1028	#D-* Purpose: flush buffer on disk
	1029	#D-* Examples:
	1030	#D-
	1031	function IGCM_sys_sync {
	1032	IGCM_debug_PushStack "IGCM_sys_sync" $@
	1033	if ( $DEBUG_sys ) ; then
	1034	echo "IGCM_sys_sync :" $@
	1035	fi
	1036
	1037	/bin/sync
	1038
	1039	IGCM_debug_PopStack "IGCM_sys_sync"
	1040	}
	1041
[622]	1042	############################################################
	1043	# Activate Running Environnment Variables
	1044
[811]	1045	#D-#==================================================
[1239]	1046	#D-function IGCM_sys_activ_variables
[811]	1047	#D-* Purpose: set environement variables prior to execution
	1048	#D-* Examples:
	1049	#D-
[622]	1050	function IGCM_sys_activ_variables {
[623]	1051	IGCM_debug_PushStack "IGCM_sys_activ_variables"
	1052	if ( $DEBUG_sys ) ; then
	1053	echo "IGCM_sys_activ_variables"
	1054	fi
[622]	1055
[811]	1056	# --------------------------------------------------------------------
[1409]	1057	#D- MODULE specifications
	1058	# --------------------------------------------------------------------
[1479]	1059	# Source the file EnvFile containing all module load needed to run the model.
[1485]	1060	# EnvFile can be specified in config.card. If this is the case and if the file exists,
	1061	# it will be used. Otherwise a default file will be used.
[1482]	1062	if [ ! X${config_UserChoices_EnvFile} = X ] && [ -f ${config_UserChoices_EnvFile} ] ; then
	1063	# EnvFile is set in config.card and the file exists
	1064	IGCM_debug_Print 1 "EnvFile set in config.card will be used"
	1065	EnvFile=${config_UserChoices_EnvFile}
	1066	else
[1479]	1067	IGCM_debug_Print 1 "IGCM_sys_active_variables : Default modules will be used"
	1068	module purge > /dev/null 2>&1
[1490]	1069	EnvFile=/gpfswork/rech/psl/commun/MachineEnvironment/jeanzay/env_jeanzay
[1409]	1070	fi
[1479]	1071	IGCM_debug_Print 1 "IGCM_sys_active_variables : Following EnvFile will be used :"
	1072	IGCM_debug_Print 1 ${EnvFile}
	1073	. ${EnvFile}
	1074
[1490]	1075	IGCM_debug_Print 1 "IGCM_sys_active_variables : Now loaded modules for Jean-Zay. "
[1433]	1076	module list
[1409]	1077
	1078	# --------------------------------------------------------------------
[811]	1079	#D- MPI specifications
	1080	# --------------------------------------------------------------------
[1191]	1081
[811]	1082	# --------------------------------------------------------------------
	1083	#D- Other specifications
	1084	# --------------------------------------------------------------------
	1085
[623]	1086	ulimit -s unlimited
[1506]	1087	if [ ${executionType} -eq 2 ] ; then
	1088	export SLURM_HOSTFILE=./hostlist
	1089	fi
[622]	1090
[1506]	1091
[623]	1092	IGCM_debug_PopStack "IGCM_sys_activ_variables"
[622]	1093	}
	1094
	1095	############################################################
	1096	# Desactivate Running Environnment Variables
	1097
[811]	1098	#D-#==================================================
	1099	#D-function IGCM_sys_desactiv_variables
	1100	#D-* Purpose: unset environement variables after execution
	1101	#D-* Examples:
	1102	#D-
[622]	1103	function IGCM_sys_desactiv_variables {
[623]	1104	IGCM_debug_PushStack "IGCM_sys_desactiv_variables"
	1105	if ( $DEBUG_sys ) ; then
	1106	echo "IGCM_sys_desactiv_variables"
	1107	fi
[811]	1108	# --------------------------------------------------------------------
	1109	#D- MPI specifications
	1110	# --------------------------------------------------------------------
[1506]	1111	if [ ${executionType} -eq 2 ] ; then
	1112	unset SLURM_HOSTFILE
	1113	fi
[811]	1114	# --------------------------------------------------------------------
	1115	#D- Other specifications
	1116	# --------------------------------------------------------------------
	1117
[623]	1118	IGCM_debug_PopStack "IGCM_sys_desactiv_variables"
[622]	1119	}
	1120
[623]	1121	############################################################
[1238]	1122	# Update job headers to be used by the scheduler
	1123
	1124	#D-#==================================================
	1125	#D-function IGCM_sys_updateHeaders
	1126	#D-* Purpose: Update job headers to be used by the scheduler
	1127	#D-* Examples: IGCM_sys_updateHeaders /path/to/Job_MYEXP
	1128	#D-
	1129	function IGCM_sys_updateHeaders {
	1130	IGCM_debug_PushStack "IGCM_sys_updateHeaders"
	1131	if ( $DEBUG_sys ) ; then
	1132	echo "IGCM_sys_updateHeaders"
	1133	fi
	1134	typeset file
	1135	file=$1
	1136
	1137	if [ ${executionType} -eq 1 ] ; then
	1138	# MPMD + MPI
	1139	sed -e "/::openMPthreads::/d" \
	1140	-e "s/::JobNumProcTot::/${coreNumber}/" \
	1141	${file} > ${file}.tmp
	1142
	1143	elif [ ${executionType} -eq 2 ] ; then
[1493]	1144	# MPMD + MPI + OMP
[1500]	1145	(( nodeNumber = coreNumber / NB_CORE_PER_NODE ))
	1146	[ $(( ${coreNumber} % ${NB_CORE_PER_NODE} )) -ne 0 ] && (( nodeNumber = nodeNumber + 1 ))
	1147	sed -e "/::openMPthreads::/d" \
	1148	-e "s/::JobNumProcTot::/${mpiTasks}/" \
	1149	-e "/ntasks/i\#SBATCH --nodes=${nodeNumber}"\
	1150	-e "/ntasks/i\#SBATCH --exclusive" \
	1151	${file} > ${file}.tmp
[1238]	1152	elif [ ${executionType} -eq 3 ] ; then
	1153	# SPMD + MPI/OMP
	1154	sed -e "s/::openMPthreads::/${openMPthreads}/" \
	1155	-e "s/::JobNumProcTot::/${mpiTasks}/" \
	1156	${file} > ${file}.tmp
	1157
	1158	elif [ ${executionType} -eq 4 ] ; then
	1159	# SPMD + MPI only
	1160	sed -e "s/::JobNumProcTot::/${mpiTasks}/" \
	1161	-e "/::openMPthreads::/d" \
	1162	${file} > ${file}.tmp
	1163
	1164	elif [ ${executionType} -eq 5 ] ; then
	1165	# SPMD + OMP only
	1166	sed -e "s/::openMPthreads::/${openMPthreads}/" \
	1167	-e "/::JobNumProcTot::/d" \
	1168	${file} > ${file}.tmp
	1169
	1170	elif [ ${executionType} -eq 6 ] ; then
	1171	# SEQUENTIAL THEN
	1172	sed -e "s/::JobNumProcTot::/1/" \
	1173	-e "/::openMPthreads::/d" \
	1174	${file} > ${file}.tmp
	1175
	1176	fi
	1177
	1178	IGCM_sys_Mv ${file}.tmp ${file}
	1179
	1180	IGCM_debug_PopStack "IGCM_sys_updateHeaders"
	1181	}
	1182
	1183	############################################################
[623]	1184	# Build MPI/OMP scripts run file (dummy function)
[622]	1185
[811]	1186	#D-#==================================================
	1187	#D-function IGCM_sys_build_run_file
	1188	#D-* Purpose: build run file (deprecated)
	1189	#D-* Examples:
	1190	#D-
[622]	1191	function IGCM_sys_build_run_file {
[811]	1192	IGCM_debug_Print 3 " dummy function : IGCM_sys_build_run_file "
[622]	1193	}
	1194
	1195	############################################################
	1196	# Build MPI/OMP scripts
[811]	1197
	1198	#D-#==================================================
	1199	#D-function IGCM_sys_build_execution_scripts
	1200	#D-* Purpose: build execution scripts to be launch by ${HOST_MPIRUN_COMMAND}
	1201	#D-* Examples:
	1202	#D-
[623]	1203	function IGCM_sys_build_execution_scripts
[622]	1204	{
[623]	1205	IGCM_debug_PushStack "IGCM_sys_build_execution_scripts" $@
	1206	if ( $DEBUG_sys ) ; then
	1207	echo "IGCM_sys_build_execution_scripts " $@
	1208	fi
[1493]	1209	IGCM_debug_Print 1 "executionType= ${executionType}"
[623]	1210	EXECUTION=${HOST_MPIRUN_COMMAND}
[622]	1211
[623]	1212	if ( ${OK_PARA_MPMD} ) ; then
[622]	1213
[1390]	1214	# MPMD mode
[1493]	1215	# 1 MPI only : executionType=1
	1216	# 2 MPI/OpenMP : executionType=2
[1390]	1217
[1381]	1218	if [ -f run_file ] ; then
	1219	IGCM_sys_Rm -f run_file
	1220	fi
	1221	touch run_file
	1222
[1390]	1223	# case 1 : Only MPI (MPMD)
[1078]	1224	if ( ! ${OK_PARA_OMP} ) ; then
	1225	# Build run_file
[1492]	1226	current_core=0
[1078]	1227	# First loop on the components for the coupler ie oasis (only if oasis3)
	1228	# the coupler ie oasis3 must be the first one
	1229	for comp in ${config_ListOfComponents[*]} ; do
[622]	1230
[1078]	1231	eval ExeNameIn=\${config_Executable_${comp}[0]}
	1232	eval ExeNameOut=\${config_Executable_${comp}[1]}
[622]	1233
[1492]	1234	# Only if the component has an executable
	1235	if ( [ "X${ExeNameOut}" != X\"\" ] ) ; then
[1000]	1236
[1078]	1237	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
[1492]	1238	(( end_core = ${current_core} + ${comp_proc_mpi_loc} - 1 ))
	1239	echo "${current_core}-${end_core} ./${ExeNameOut}" >> run_file
	1240	(( current_core = ${end_core} + 1 ))
[1078]	1241	fi
	1242	done
[622]	1243
[1492]	1244	EXECUTION="${HOST_MPIRUN_COMMAND} --multi-prog ./run_file"
[622]	1245
[1390]	1246	IGCM_sys_Chmod u+x run_file
	1247	if ( $DEBUG_sys ) ; then
	1248	echo "run_file contains : "
	1249	cat run_file
	1250	fi
	1251
[1493]	1252	else
[1390]	1253
[1493]	1254	# 2 MPI/OpenMP : executionType=2
[1390]	1255
[1381]	1256	# MPI-OpenMP (MPMD)
[1500]	1257	# export SLURM_HOSTFILE=./hostlist
	1258	# srun --cpu-bind=none --distribution=arbitrary --multi-prog ./run_file
	1259	# example of hostlist file :
	1260	# r3i3n33
	1261	# r3i3n33
	1262	# ...
	1263	# example of run_file :
	1264	# 0 taskset -c 0-4 ./script_lmdz.x.ksh
	1265	# 1 taskset -c 5-9 ./script_lmdz.x.ksh
	1266	#...
	1267	# 32 taskset -c 0 ./script_opa.xx.ksh
	1268	# 33 taskset -c 1 ./script_opa.xx.ksh
	1269	#...
	1270	# 62 taskset -c 30 ./script_xios.x.ksh
[622]	1271
[1500]	1272	# Hosts treatment
	1273	IFS=$'\n'; set -f
	1274	listnodes=($(< <( scontrol show hostnames $SLURM_JOB_NODELIST )))
	1275	unset IFS; set +f
	1276	rm -f hostlist
	1277
	1278	# Loop on the components to build run_file and script_exec files
	1279	rank=0
	1280	current_core=0
	1281
[1078]	1282	for comp in ${config_ListOfComponents[*]} ; do
[1000]	1283
[1078]	1284	eval ExeNameIn=\${config_Executable_${comp}[0]}
	1285	eval ExeNameOut=\${config_Executable_${comp}[1]}
[1000]	1286
[1078]	1287	# Not possible if oasis has an executable (i.e old version of oasis3)
	1288	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${comp}" = "XCPL" ] ) ; then
	1289	IGCM_debug_Exit "ERROR MPMD with hybrid MPI-OpenMP is not available with oasis3 version"
	1290	IGCM_debug_Print 2 "Only available with oasis3-MCT version coupler"
	1291	IGCM_debug_Verif_Exit
	1292	fi
[1000]	1293
[1078]	1294	# Only if we really have an executable for the component :
	1295	if [ "X${ExeNameOut}" != X\"\" ] ; then
[1000]	1296
[1078]	1297	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
	1298	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
[1500]	1299	eval comp_proc_nod_loc=\${${comp}_PROC_NOD}
	1300
[1000]	1301
[1381]	1302	# Build script files
	1303
[1078]	1304	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
	1305	echo "" >> script_${ExeNameOut}.ksh
	1306	if [ ${comp_proc_omp_loc} -gt 1 ] ; then
[1500]	1307	# Check if the number of threads is correct
	1308	case ${comp_proc_omp_loc} in
	1309	2\|4\|5\|10\|20)
	1310	IGCM_debug_Print 1 "You run ${ExeNameOut} on ${comp_proc_omp_loc} OMP threads"
	1311	;;
	1312	*)
	1313	IGCM_debug_Exit "ERROR with OMP parameters !"
	1314	IGCM_debug_Print 2 "${comp_proc_omp_loc} is not possible as number of OMP threads"
	1315	IGCM_debug_Print 2 "Only 2,4,5,10,20 as number of OMP threads are possible "
	1316	IGCM_debug_Verif_Exit
	1317	;;
	1318	esac
[1493]	1319	echo "export OMP_STACKSIZE=3g" >> script_${ExeNameOut}.ksh
	1320	echo "export OMP_PLACES=cores" >> script_${ExeNameOut}.ksh
[1500]	1321	echo "export OMP_NUM_THREADS=${comp_proc_omp_loc}" >> script_${ExeNameOut}.ksh
	1322
[1078]	1323	fi
[1318]	1324
[1381]	1325	# to have out/err per process on different files
	1326	echo "./${ExeNameOut} > out_${ExeNameOut}.out.\${SLURM_PROCID} 2>out_${ExeNameOut}.err.\${SLURM_PROCID}" >> script_${ExeNameOut}.ksh
[1078]	1327
[1381]	1328	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
[1078]	1329
[1381]	1330	# Complete run_file
[1500]	1331	if [ ${comp_proc_nod_loc} -gt 1 ] ; then
	1332	(( offset_comp_proc_loc = NB_CORE_PER_NODE / (comp_proc_mpi_loc / comp_proc_nod_loc) ))
	1333	else
	1334	(( offset_comp_proc_loc = comp_proc_omp_loc ))
	1335	fi
[1000]	1336
[1500]	1337	for nb_proc_mpi in `seq 0 $(($comp_proc_mpi_loc-1))`; do
	1338	(( index_host = current_core / NB_CORE_PER_NODE ))
	1339	host_value=${listnodes[${index_host}]}
	1340	(( slot = current_core % NB_CORE_PER_NODE ))
	1341	echo "$host_value" >> hostlist
	1342	echo "$rank taskset -c $slot-$((slot+comp_proc_omp_loc-1)) ./script_${ExeNameOut}.ksh" >> run_file
	1343	(( rank = rank + 1 ))
	1344	(( current_core = current_core + offset_comp_proc_loc ))
	1345	done
[1078]	1346	fi
	1347	done
[1381]	1348
[1385]	1349	## variable added to stop after 60s instead of 600s by default.
	1350	## This is used when no error comes from executables and when something stopped an executable without notice.
	1351	export SLURM_WAIT=60
[1381]	1352
[1500]	1353	EXECUTION="${HOST_MPIRUN_COMMAND} --cpu-bind=none --distribution=arbitrary --multi-prog ./run_file"
[1381]	1354
[1390]	1355	IGCM_sys_Chmod u+x run_file
	1356	if ( $DEBUG_sys ) ; then
	1357	echo "run_file contains : "
	1358	cat run_file
	1359	fi
[1381]	1360
[1493]	1361	fi # if ${OK_PARA_MPMD}
[1381]	1362
[1390]	1363	else
[1493]	1364	# Only one executable (SPMD mode): executionType=3, 4, 5 and 6
[1000]	1365
[622]	1366	for comp in ${config_ListOfComponents[*]} ; do
	1367
[685]	1368	# Only if we really have an executable for the component :
[623]	1369	eval ExeNameOut=\${config_Executable_${comp}[1]}
[685]	1370	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${ExeNameOut}" != "Xinca.dat" ] ) ; then
[622]	1371
[1381]	1372	# Build script files
	1373
[685]	1374	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
	1375	echo "" >> script_${ExeNameOut}.ksh
	1376	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
	1377
	1378	if ( ${OK_PARA_OMP} ) ; then
[1507]	1379	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
	1380	# Check if the number of threads is correct
	1381	case ${comp_proc_omp_loc} in
	1382	2\|4\|5\|10\|20)
	1383	IGCM_debug_Print 1 "You run ${ExeNameOut} on ${comp_proc_omp_loc} OMP threads"
	1384	;;
	1385	*)
	1386	IGCM_debug_Exit "ERROR with OMP parameters !"
	1387	IGCM_debug_Print 2 "${comp_proc_omp_loc} is not possible as number of OMP threads"
	1388	IGCM_debug_Print 2 "Only 2,4,5,10,20 as number of OMP threads are possible "
	1389	IGCM_debug_Verif_Exit
	1390	;;
	1391	esac
	1392	echo "" >> script_${ExeNameOut}.ksh
	1393	echo "export OMP_STACKSIZE=3g" >> script_${ExeNameOut}.ksh
	1394	echo "export OMP_PLACES=cores" >> script_${ExeNameOut}.ksh
	1395	echo "OMP_NUM_THREADS=${comp_proc_omp_loc}" >> script_${ExeNameOut}.ksh
[685]	1396	fi
	1397
[1493]	1398	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
	1399
	1400	# To have out/err per process on different files
	1401	echo "./${ExeNameOut} > out_${ExeNameOut}.out.\${SLURM_PROCID} 2>out_${ExeNameOut}.err.\${SLURM_PROCID}" >> script_${ExeNameOut}.ksh
	1402	EXECUTION="${HOST_MPIRUN_COMMAND} ./script_${ExeNameOut}.ksh"
[685]	1403
[1462]	1404	IGCM_debug_Print 1 "sys Irene : script_${ExeNameOut}.ksh contains"
[685]	1405	cat script_${ExeNameOut}.ksh
	1406
[623]	1407	fi
[622]	1408
	1409	done
	1410
[1390]	1411	fi # ${OK_PARA_MPMD}
[622]	1412
[1462]	1413	IGCM_debug_Print 1 "sys Irene : execution command is "
[623]	1414	IGCM_debug_Print 1 "$EXECUTION"
[622]	1415
[623]	1416	IGCM_debug_PopStack "IGCM_sys_build_execution_scripts"
[622]	1417	}
	1418
[933]	1419	#D-#==================================================
	1420	#D-function IGCM_sys_check_path
	1421	#D-* Purpose: check that RUN_DIR_PATH that will be removed on some machine
	1422	#D-* do not point to an important use directory. Stop immediately in that case.
	1423	#D-* Examples:
	1424	#D-
	1425	function IGCM_sys_check_path {
	1426	IGCM_debug_PushStack "IGCM_sys_check_path"
	1427	if ( $DEBUG_sys ) ; then
	1428	echo "IGCM_sys_check_path"
	1429	fi
[811]	1430
[1490]	1431	if ( [ X${RUN_DIR_PATH} = X${HOME} ] \|\| [ X${RUN_DIR_PATH} = X${WORK} ] \|\| [ X${RUN_DIR_PATH} = X${SCRATCH} ] \|\| [ X${RUN_DIR_PATH} = X${STORE} ] ) ; then
[933]	1432	IGCM_debug_Print 1 "Variable RUN_DIR_PATH is pointing to an important directory : ${RUN_DIR_PATH}"
	1433	IGCM_debug_Print 1 "Please check the RUN_DIR_PATH definition in your Job : Job_${config_UserChoices_JobName}"
[936]	1434	IGCM_debug_Exit "This will stop the job"
[1508]	1435	elif ( [ X${RUN_DIR_PATH} = X${HOME} ] \|\| [ X${RUN_DIR_PATH} = X/gpfswork/rech/${DataProject}/${LOGIN} ] \|\| [ X${RUN_DIR_PATH} = X/gpfsscratch/rech/${DataProject}/${LOGIN} ] \|\| [ X${RUN_DIR_PATH} = X/gpfsstore/rech/${DataProject}/${LOGIN} ] ) ; then
	1436	IGCM_debug_Print 1 "Variable RUN_DIR_PATH is pointing to an important directory : ${RUN_DIR_PATH}"
	1437	IGCM_debug_Print 1 "Please check the RUN_DIR_PATH definition in your Job : Job_${config_UserChoices_JobName}"
	1438	IGCM_debug_Exit "This will stop the job"
[933]	1439	fi
	1440	IGCM_debug_PopStack "IGCM_sys_check_path"
	1441	}
	1442
[811]	1443	#D-#==================================================
	1444	#D-function IGCM_sys_check_quota
	1445	#D-* Purpose: check user quota. Stop the simulation if quota above 90%
	1446	#D-* Examples:
	1447	#D-
[714]	1448	function IGCM_sys_check_quota {
[776]	1449	IGCM_debug_PushStack "IGCM_sys_check_quota"
[1502]	1450	echo "IGCM_sys_check_quota is desactivated on JeanZay because there is actually no way to check scratch quota on this computing center (2019.11.19)"
	1451	# if ( $DEBUG_sys ) ; then
	1452	# echo "IGCM_sys_check_quota"
	1453	# fi
	1454	# # Limit of quota (in %)
	1455	# limit_quota=90
	1456	#
	1457	# # Check of the volume
	1458	# volume_quota=$(ccc_quota \| grep ' scratch' \| gawk '{print $2}')
	1459	# volume_avail=$(ccc_quota \| grep ' scratch' \| gawk '{print $3}')
	1460	#
	1461	# if ( [ ! X${volume_quota} = X ] && [ ! ${volume_quota} = "-" ] ) ; then
	1462	#
	1463	# unit_avail=${volume_avail: -1}
	1464	# unit_quota=${volume_quota: -1}
	1465	#
	1466	# if [ "${unit_quota}" = "*" ] ; then
	1467	# IGCM_debug_Print 1 "Please, check your quota of volume on scratch"
	1468	# IGCM_debug_Print 1 "More than 100% of your quota is used"
	1469	# IGCM_debug_Print 1 "Use the ccc_quota command to check"
	1470	# IGCM_debug_Print 1 "You must have more than 10% available to run"
	1471	# IGCM_debug_Exit "Not enough space to run ! STOP HERE"
	1472	# IGCM_debug_Verif_Exit
	1473	# fi
	1474	#
	1475	# temp_avail=${volume_avail%%${volume_avail: -1}*}
	1476	# temp_quota=${volume_quota%%${volume_quota: -1}*}
	1477	#
	1478	# if [ ! ${unit_avail} = ${unit_quota} ] ; then
	1479	#
	1480	# # Convertion
	1481	# if [ ${volume_avail: -1} = "T" ] ; then
	1482	# (( temp_avail = temp_avail * 1000000000000 ))
	1483	# elif [ ${volume_avail: -1} = "G" ] ; then
	1484	# (( temp_avail = temp_avail * 1000000000 ))
	1485	# elif [ ${volume_avail: -1} = "M" ] ; then
	1486	# (( temp_avail = temp_avail * 1000000 ))
	1487	# elif [ ${volume_avail: -1} = "k" ] ; then
	1488	# (( temp_avail = temp_avail * 1000 ))
	1489	# else
	1490	# (( temp_avail = volume_avail ))
	1491	# fi
	1492	# if [ ${volume_quota: -1} = "T" ] ; then
	1493	# (( temp_quota = temp_quota * 1000000000000 ))
	1494	# elif [ ${volume_quota: -1} = "G" ] ; then
	1495	# (( temp_quota = temp_quota * 1000000000 ))
	1496	# elif [ ${volume_quota: -1} = "M" ] ; then
	1497	# (( temp_quota = temp_quota * 1000000 ))
	1498	# elif [ ${volume_quota: -1} = "k" ] ; then
	1499	# (( temp_quota = temp_quota * 1000 ))
	1500	# else
	1501	# (( temp_quota = volume_quota ))
	1502	# fi
	1503	# fi
	1504	#
	1505	# quota_volume=$(echo "scale=2 ; $temp_quota/$temp_avail*100" \| bc)
	1506	## echo "volume ratio is " $quota_volume
	1507	#
	1508	# if [ ${quota_volume} -ge ${limit_quota} ] ; then
	1509	# IGCM_debug_Print 1 "Please, check your quota of volume on scratch"
	1510	# IGCM_debug_Print 1 "${quota_volume}% of your quota is used"
	1511	# IGCM_debug_Print 1 "Use the ccc_quota command to check"
	1512	# IGCM_debug_Print 1 "You must have more than 10% available to run"
	1513	# IGCM_debug_Exit "Not enough space to run ! STOP HERE"
	1514	# IGCM_debug_Verif_Exit
	1515	# fi
	1516	#
	1517	# fi
	1518	#
	1519	## Check of the number of inodes
	1520	#
	1521	# inode_quota=$(ccc_quota \| grep ' scratch' \| gawk '{print $6}')
	1522	# inode_avail=$(ccc_quota \| grep ' scratch' \| gawk '{print $7}')
	1523	#
	1524	# if ( [ ! X${inode_quota} = X ] && [ ! ${inode_quota} = "-" ] ) ; then
	1525	#
	1526	# unit_avail=${inode_avail: -1}
	1527	# unit_quota=${inode_quota: -1}
	1528	#
	1529	# if [ "${unit_quota}" = "*" ] ; then
	1530	# IGCM_debug_Print 1 "Please, check your quota of inode on scratch"
	1531	# IGCM_debug_Print 1 "More than 100% of your quota is used"
	1532	# IGCM_debug_Print 1 "Use the ccc_quota command to check"
	1533	# IGCM_debug_Print 1 "You must have more than 10% available to run"
	1534	# IGCM_debug_Exit "Not enough space to run ! STOP HERE"
	1535	# IGCM_debug_Verif_Exit
	1536	# fi
	1537	#
	1538	# temp_avail=${inode_avail%%${inode_avail: -1}*}
	1539	# temp_quota=${inode_quota%%${inode_quota: -1}*}
	1540	#
	1541	# if [ ! ${unit_avail} = ${unit_quota} ] ; then
	1542	#
	1543	# # Convertion
	1544	# if [ ${inode_avail: -1} = "T" ] ; then
	1545	# (( temp_avail = temp_avail * 1000000000000 ))
	1546	# elif [ ${inode_avail: -1} = "G" ] ; then
	1547	# (( temp_avail = temp_avail * 1000000000 ))
	1548	# elif [ ${inode_avail: -1} = "M" ] ; then
	1549	# (( temp_avail = temp_avail * 1000000 ))
	1550	# elif [ ${inode_avail: -1} = "k" ] ; then
	1551	# (( temp_avail = temp_avail * 1000 ))
	1552	# else
	1553	# (( temp_avail = inode_avail ))
	1554	# fi
	1555	#
	1556	# if [ ${inode_quota: -1} = "T" ] ; then
	1557	# (( temp_quota = temp_quota * 1000000000000 ))
	1558	# elif [ ${inode_quota: -1} = "G" ] ; then
	1559	# (( temp_quota = temp_quota * 1000000000 ))
	1560	# elif [ ${inode_quota: -1} = "M" ] ; then
	1561	# (( temp_quota = temp_quota * 1000000 ))
	1562	# elif [ ${inode_quota: -1} = "k" ] ; then
	1563	# (( temp_quota = temp_quota * 1000 ))
	1564	# else
	1565	# (( temp_quota = inode_quota ))
	1566	# fi
	1567	# fi
	1568	# quota_inode=$(echo "scale=2 ; $temp_quota/$temp_avail*100" \| bc)
	1569	## echo "inode ratio is " $quota_inode
	1570	#
	1571	# if [ ${quota_inode} -ge ${limit_quota} ] ; then
	1572	# IGCM_debug_Print 1 "Please, check your quota of inode on scratch"
	1573	# IGCM_debug_Print 1 "${quota_inode}% of your quota is used"
	1574	# IGCM_debug_Print 1 "Use the ccc_quota command to check"
	1575	# IGCM_debug_Print 1 "You must have more than 10% available to run"
	1576	# IGCM_debug_Exit "Not enough space to run ! STOP HERE"
	1577	# IGCM_debug_Verif_Exit
	1578	# fi
	1579	# fi
[776]	1580	IGCM_debug_PopStack "IGCM_sys_check_quota"
[714]	1581	}
	1582
[773]	1583	#D-#==================================================
[1286]	1584	#D-function IGCM_sys_projectAccounting
	1585	#D-* Purpose: store project accounting information in a file
	1586	#D-* Examples:
	1587	#D-
	1588	function IGCM_sys_projectAccounting {
	1589	IGCM_debug_PushStack "IGCM_sys_projectAccounting"
	1590	if ( $DEBUG_sys ) ; then
	1591	echo "IGCM_sys_check_quota"
	1592	fi
[1502]	1593	#need to implemented on Jean Zay
	1594	touch $1
	1595	# ssh irene191 /usr/bin/ccc_myproject > $1
[1286]	1596
	1597	IGCM_debug_PopStack "IGCM_sys_projectAccounting"
	1598	}
	1599
	1600	#D-#==================================================
[1244]	1601	#D-function IGCM_sys_getJobSchedulerID
	1602	#D-* Purpose: Get the job ID during execution
	1603	#D-* Examples: IGCM_sys_getJobSchedulerID jobSchedulerID
	1604	#D-
	1605	function IGCM_sys_getJobSchedulerID {
	1606	IGCM_debug_PushStack "IGCM_sys_getJobSchedulerID"
	1607	if ( $DEBUG_sys ) ; then
	1608	echo "IGCM_sys_getJobSchedulerID"
	1609	fi
	1610
[1503]	1611	eval ${1}=${SLURM_JOBID}
[1244]	1612
	1613	IGCM_debug_PopStack "IGCM_sys_getJobSchedulerID"
	1614	}
	1615
	1616	#D-#==================================================
[958]	1617	#D-function IGCM_sys_GetJobID
[1244]	1618	#D-* Purpose: Get the job ID from the JobName
[958]	1619	#D-* Examples: IGCM_sys_GetJobID ${JobName} ${TargetUsr} JobID
	1620	#D-
	1621	function IGCM_sys_GetJobID {
	1622	IGCM_debug_PushStack "IGCM_sys_GetJobID"
	1623	if ( $DEBUG_sys ) ; then
	1624	echo "IGCM_sys_GetJobID"
	1625	fi
	1626
[1502]	1627	ID=$( squeue -u $user \| grep Job_${config_UserChoices_JobName} \| gawk ' { print $1}' )
[958]	1628
[1502]	1629
[958]	1630	eval ${3}=${ID}
	1631	IGCM_debug_PopStack "IGCM_sys_GetJobID"
	1632	}
	1633
	1634	#D-#==================================================
[773]	1635	#D-function IGCM_sys_CountJobInQueue
[1244]	1636	#D-* Purpose: Count number of users job
[773]	1637	#D-* Examples: IGCM_sys_CountJobInQueue ${JobName} NbRun
	1638	#D-
	1639	function IGCM_sys_CountJobInQueue {
	1640	IGCM_debug_PushStack "IGCM_sys_CountJobInQueue"
	1641	if ( $DEBUG_sys ) ; then
	1642	echo "IGCM_sys_CountJobInQueue"
	1643	fi
	1644
	1645	# With -f option, the full job name is given in the last column
[1502]	1646	NbRun=$(squeue -u $user \| \
	1647	grep -v JOBID \| gawk 'BEGIN { x=0 } ( $NF ~ JobName ) { x=x+1 } END { print x }' )
[773]	1648
	1649	eval ${2}=${NbRun}
	1650
	1651	IGCM_debug_PopStack "IGCM_sys_CountJobInQueue"
	1652	}
	1653
[955]	1654	#D-#==================================================
	1655	#D-function IGCM_sys_ListJobInQueue
[1244]	1656	#D-* Purpose: Produce a list of users computing jobs (excluding post-processing)
[955]	1657	#D-* Examples: IGCM_sys_ListJobInQueue ${User} JobNameList
	1658	#D-
	1659	function IGCM_sys_ListJobInQueue {
	1660	IGCM_debug_PushStack "IGCM_sys_ListJobInQueue"
	1661	if ( $DEBUG_sys ) ; then
	1662	echo "IGCM_sys_ListJobInQueue"
	1663	fi
	1664
[1502]	1665	# to be implemented on Jean Zay
	1666	set -A JobList $( squeue -u $user \| gawk '{print $3}' \| grep -v NAME)
	1667	# # With -f option, the full job name is given in the last column
	1668	# set -A JobList $( ccc_mstat -f \| gawk -v User=$1 \
	1669	# '( $2 == User && \
	1670	# $NF != /TS/ && \
	1671	# $NF !~ /PACK/ && \
	1672	# $NF !~ /REBUILD/ && \
	1673	# $NF !~ /pack/ ) \
	1674	# { print $NF }' \| sed -e "s/$.$\.[0-9]/\1/" )
	1675	#
[955]	1676	eval set -A ${2} ${JobList[*]}
	1677
	1678	IGCM_debug_PopStack "IGCM_sys_ListJobInQueue"
	1679	}
[1272]	1680
	1681	#D-#==================================================
	1682	#D-function IGCM_sys_atlas
	1683	#D-* Purpose: encapsulate atlas call so as to manage error code and curie specificity
	1684	#D-* Examples:
	1685	#D-
	1686	function IGCM_sys_atlas {
	1687	IGCM_debug_PushStack "IGCM_sys_atlas" $@
	1688	if ( $DEBUG_sys ) ; then
	1689	echo "IGCM_sys_atlas :" $@
	1690	fi
	1691
	1692	typeset status
[1502]	1693	# To be implemented on Jean Zay
	1694	#
	1695	# \ccc_mprun atlas $@ > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
	1696	# status=$?
	1697	# if [ ${status} -gt 0 ] ; then
	1698	# echo "IGCM_sys_atlas : error code ${status}"
	1699	# cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
	1700	# \rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
	1701	# IGCM_debug_PopStack "IGCM_sys_atlas"
	1702	# return 1
	1703	# else
	1704	# IGCM_debug_PopStack "IGCM_sys_atlas"
	1705	# return 0
	1706	# fi
	1707	#
[1272]	1708	IGCM_debug_PopStack "IGCM_sys_atlas"
	1709	}
	1710
[1380]	1711	#D-#==================================================
	1712	#D-function IGCM_sys_rebuild_nemo
	1713	#D-* Purpose: rebuild nemo parallel files with nemo specific rebuild on curie
	1714	#D-* Examples:
	1715	#D-
	1716
	1717	function IGCM_sys_rebuild_nemo {
	1718	IGCM_debug_PushStack "IGCM_sys_rebuild_nemo" $@
	1719	if ( $DEBUG_sys ) ; then
	1720	echo "IGCM_sys_rebuild_nemo :" $@
	1721	fi
	1722
[1496]	1723	/gpfswork/rech/psl/commun/Tools/rebuild_nemo/bin/rebuild_nemo ${1} ${2}
[1380]	1724
	1725	IGCM_debug_PopStack "IGCM_sys_rebuild_nemo"
	1726	}
	1727

Note: See TracBrowser for help on using the repository browser.

Context Navigation

source: trunk/libIGCM/libIGCM_sys/libIGCM_sys_jeanzay.ksh @ 1510

Download in other formats: