Context Navigation

source: trunk/libIGCM/libIGCM_sys/libIGCM_sys_jeanzay.ksh @ 1491

Last change on this file since 1491 was 1490, checked in by jgipsl, 5 years ago
Added first version of sys_jeanzay. Still lots to do. Only case done for now is one executable mpi mode. Only create_ts done among post-treatments and only from main job (not done via TimeSeriesChecker?). Change of data project is not working (follwoing is not working when launched from job: eval $(idrenv -d ${PROJECT}) ).
Property licence set to The following licence information concerns ONLY the libIGCM tools ================================================================== Copyright © Centre National de la Recherche Scientifique CNRS Commissariat à l'Énergie Atomique CEA libIGCM : Library for Portable Models Computation of IGCM Group. IGCM Group is the french IPSL Global Climate Model Group. This library is a set of shell scripts and functions whose purpose is the management of the initialization, the launch, the transfer of output files, the post-processing and the monitoring of datas produce by any numerical program on any plateforme. This software is governed by the CeCILL license under French law and abiding by the rules of distribution of free software. You can use, modify and/ or redistribute the software under the terms of the CeCILL license as circulated by CEA, CNRS and INRIA at the following URL "http://www.cecill.info". As a counterpart to the access to the source code and rights to copy, modify and redistribute granted by the license, users are provided only with a limited warranty and the software's author, the holder of the economic rights, and the successive licensors have only limited liability. In this respect, the user's attention is drawn to the risks associated with loading, using, modifying and/or developing or reproducing the software by the user in light of its specific status of free software, that may mean that it is complicated to manipulate, and that also therefore means that it is reserved for developers and experienced professionals having in-depth computer knowledge. Users are therefore encouraged to load and test the software's suitability as regards their requirements in conditions enabling the security of their systems and/or data to be ensured and, more generally, to use and operate it in the same conditions as regards security. The fact that you are presently reading this means that you have had knowledge of the CeCILL license and that you accept its terms. Property svn:keywords set to `Revision Author Date`
File size: 62.5 KB

Rev	Line
[622]	1	#!/bin/ksh
	2
	3	#**************************************************************
	4	# Author: Sebastien Denvil, Martial Mancip, Arnaud Caubel
	5	# Contact: Arnaud.Caubel__at__lsce.ipsl.fr
	6	# $Revision:: $ Revision of last commit
	7	# $Author:: $ Author of last commit
	8	# $Date:: $ Date of last commit
	9	# IPSL (2006)
	10	# This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
	11	#
	12	#**************************************************************
	13
	14	#=========================================================
	15	# The documentation of this file can be automatically generated
[623]	16	# if you use the prefix #D- for comments to be extracted.
[622]	17	# Extract with command: cat lib* \| grep "^#D-" \| cut -c "4-"
	18	#=========================================================
	19
	20	#D-#==================================================
[1490]	21	#D-LibIGCM_sys for Jean-Zay
[622]	22	#D-#==================================================
	23	#D-
[623]	24	#D- This ksh library if a layer under some usefull
[622]	25	#D-environment variables and shell commands.
	26	#D-All those definitions depend on host particularities.
	27	#D-It manages a stack mechanism and test validity of operations.
	28	#D-All function described bellow must be prefixed by IGCM_sys.
	29
	30	#====================================================
	31	# libIGCM_sys PARAMETERS
	32	#====================================================
	33
	34	#====================================================
	35	# set DEBUG_sys to true to output calls of function
	36	typeset -r DEBUG_sys=${DEBUG_sys:=true}
	37
	38	#====================================================
	39	# Turn in dry run mode ? (sys_Put_Rest, sys_Put_Out, sys_Get)
	40	typeset -r DRYRUN=${DRYRUN:=0}
	41
	42	# YOU MUST COMPILE YOUR EXE FILES FOR DRYRUN MODE !
	43	# -------------------------------------------------------------------------------------
	44	# \| DRYRUN= \| Date computations, \| sys_Get \| Exe \| sys_Put_Out; sys_Put_Rest \|
[809]	45	# \| \| Cp/Exe/param/files \| \| \| \|
	46	# \| \| Chmod Qsub \| \| \| \|
[622]	47	# -------------------------------------------------------------------------------------
	48	# \| 0 \| yes \| yes \| yes \| yes \|
	49	# -------------------------------------------------------------------------------------
	50	# \| 1 \| yes \| yes \| yes \| no \|
	51	# -------------------------------------------------------------------------------------
	52	# \| 2 \| yes \| yes \| no \| no \|
	53	# -------------------------------------------------------------------------------------
	54	# \| 3 \| yes \| no \| no \| no \|
	55	# -------------------------------------------------------------------------------------
	56
	57	#=====================================================
	58	# Global Variables :
	59	#=====================================================
	60	# Language : "fr" or "en"
	61	typeset -r MYLANG="fr"
	62
	63	#=====================================================
[1201]	64	# Host user names project maxCpuTime
[622]	65	# $hostname ou hostname
[1158]	66	typeset HOST=${HOST:=$( hostname )}
[622]	67	# $username ou whoami
[1158]	68	typeset LOGIN=${LOGIN:=$( whoami )}
[622]	69	# $hostname of the MASTER job
[1490]	70	typeset MASTER=jeanzay
	71	# add default project set by ins_job
	72	typeset PROJECT=$( echo ${SLURM_JOB_ACCOUNT:=::default_project::} \| cut -c "1-3" )
	73	### jobWarningDelay in seconds
	74	##typeset jobWarningDelay=${BRIDGE_MSUB_MAXTIME}
	75
[622]	76	#D-
	77	#D-#==================================================
	78	#D-Program used in libIGCM
	79	#D-#==================================================
	80
[742]	81	# Submit command
[1490]	82	typeset SUBMIT=${SUBMIT:=sbatch}
[622]	83	# rsync with path
	84	typeset -r RSYNC=/usr/bin/rsync
	85	# RSYNC_opt args to rsync
	86	typeset -r RSYNC_opt="-va"
	87	# ie storage filesystem
[623]	88	typeset -r STOREHOST=${MASTER}
[1180]	89	typeset -r REMOTE_RSYNC=/usr/bin/rsync
[622]	90
	91	#====================================================
[1490]	92	# Access to module command
	93	#====================================================
	94	. /usr/share/Modules/init/ksh
	95
	96	#====================================================
[1180]	97	# Set environment tools (ferret, nco, cdo, rebuild, ...)
[622]	98	#====================================================
[996]	99	if [ X${TaskType} = Xcomputing ] ; then
[1485]	100	IGCM_debug_Print 1 "Modules will be loaded later in IGCM_sys_activ_variables."
[996]	101	else
[1413]	102	module purge > /dev/null 2>&1
[1490]	103	. /gpfswork/rech/psl/commun/MachineEnvironment/jeanzay/env_jeanzay > /dev/null 2>&1
	104	#currently no atlas
	105	# . /gpfswork/rech/psl/commun/MachineEnvironment/jeanzay/env_atlas_jeanzay > /dev/null 2>&1
[1433]	106
[1490]	107	# export PCMDI_MP=/ccc/work/cont003/igcmg/igcmg/PCMDI-MP
[1364]	108	export UVCDAT_ANONYMOUS_LOG=FALSE
[996]	109	fi
[1315]	110
	111	# FYI
[1210]	112	[ ! X${TaskType} = Xchecking ] && IGCM_debug_Print 1 "List of loaded modules:"
	113	[ ! X${TaskType} = Xchecking ] && module list
[1208]	114
[1457]	115	# For AddNoise and AddPertu3DOCE
	116	##export PATH=${PATH}:/ccc/cont003/home/igcmg/igcmg/Tools/curie/bin
[1490]	117	#jg no addnoise
	118	#export PATH=${PATH}:$(ccc_home -u igcmg)/Tools/irene/bin
[622]	119
[1457]	120	# For rebuild
	121	##export PATH=${PATH}:/ccc/cont003/home/igcmg/igcmg/Tools/irene/rebuild/modipsl_IOIPSL_PLUS_v2_2_4/bin
[1490]	122	#jg no yet any rebuild
	123	#export PATH=${PATH}:$(ccc_home -u igcmg)/Tools/irene/rebuild/modipsl_IOIPSL_PLUS_v2_2_4/bin
[622]	124
	125	#====================================================
	126	# Host specific DIRECTORIES
	127	#====================================================
	128
	129	#====================================================
[623]	130	#- MirrorlibIGCM for frontend
	131	typeset -r MirrorlibIGCM=${MirrorlibIGCM:=false}
	132
	133	#====================================================
	134	#- libIGCM_POST for frontend
	135	typeset -r libIGCM_POST=${libIGCM}
	136
	137	#====================================================
[622]	138	#- R_EXE (==> BIN_DIR = ${MODIPSL}/bin )
	139	typeset -r R_EXE="${MODIPSL}/bin"
	140
	141	#====================================================
	142	#- SUBMIT_DIR : submission dir
[1490]	143	typeset -x SUBMIT_DIR=${SUBMIT_DIR:=${SLURM_SUBMIT_DIR}}
[622]	144
	145	#====================================================
	146	#- IN
[1490]	147	typeset -r R_IN=${R_IN:=/gpfswork/rech/psl/commun/IGCM}
[622]	148
	149	#====================================================
	150	#- RUN_DIR_PATH : Temporary working directory (=> TMP)
[1490]	151	typeset -r RUN_DIR_PATH=${RUN_DIR_PATH:=${SCRATCH}/RUN_DIR/${SLURM_JOBID}_${$}}
[622]	152
	153	#====================================================
[1179]	154	#- OUTCOMMAND_PATH : tmp place to store command lines standard error and outputs
[1186]	155	typeset -r OUTCOMMAND_PATH=/tmp
[1179]	156
	157	#====================================================
[622]	158	#- HOST_MPIRUN_COMMAND
[1490]	159	typeset -r HOST_MPIRUN_COMMAND=${HOST_MPIRUN_COMMAND:="/usr/bin/time srun --mpi=pmi2 "}
[622]	160
	161	#====================================================
	162	#- Max number of arguments passed to nco operator or demigration command
	163	UNIX_MAX_LIMIT=360
	164
[623]	165	#====================================================
[1490]	166	#- set PackDefault to true
[623]	167	PackDefault=true
	168
[664]	169	#====================================================
[623]	170	#- Default number of MPI task for IPSL coupled model
	171	#- required for backward compatibility
	172	#-
	173	DEFAULT_NUM_PROC_OCE=5
	174	DEFAULT_NUM_PROC_CPL=1
[637]	175	(( DEFAULT_NUM_PROC_ATM = BATCH_NUM_PROC_TOT - DEFAULT_NUM_PROC_OCE - DEFAULT_NUM_PROC_CPL ))
[623]	176
[1426]	177	#====================================================
	178	#- Number of computing cores per node
	179	#-
[1477]	180	NB_CORE_PER_NODE=48
[1426]	181
[725]	182	#D-#==================================================
[1057]	183	#D-function IGCM_sys_defineArchives
	184	#D-* Purpose:
[1461]	185	#D-* Load dfldatadir module : According to project used for submission (default) or set in config.card (optional)
[1057]	186	#D-* Define ARCHIVE : Dedicated to large files
	187	#D-* Define STORAGE : Dedicated to small/medium files
	188	#D-* Define R_OUT : Output tree located on ARCHIVE
[1490]	189	#D-* Define R_FIG : Output tree located on STORAGE hosting figures (monsitoring and atlas, and/or small files)
	190	#D-* Define R_BUF : Output tree located on SCRATCH hosting files waiting for rebuild or pack processes
	191	#D-* if SpaceName=TEST everything is stored on SCRATCH
[725]	192	#D-* Examples:
	193	#D-
[1057]	194	function IGCM_sys_defineArchives {
	195	IGCM_debug_PushStack "IGCM_sys_defineArchives"
[676]	196
[1461]	197	# Load dfldatadir depending on the project used for submission (default) or set in config.card (optional)
	198	if [ X${config_UserChoices_DataProject} = X ] \|\| [ X${config_UserChoices_DataProject} = DEFAULT ]; then
[1490]	199	# The variable DataProject is not set in in section UserChoices in config.card or it is set to DEFAULT
	200	# Default option: Change IDRPROJ according to the project used for submission
	201
	202	# jg following line does not work in a submitted job:
	203	eval $(idrenv -d ${PROJECT})
[1461]	204	else
	205	# Use the project set by the variable DataProject in section UserChoices in config.card
[1490]	206	eval $(idrenv -d ${config_UserChoices_DataProject})
	207	IGCM_debug_Print 1 "Change DataProject red in config.card : ${config_UserChoices_DataProject}"
[1461]	208	fi
[1490]	209	IGCM_debug_Print 1 "Data project for output to be used: ${IDRPROJ}"
[1461]	210
[1490]	211	#====================================================
	212	#- RUN_DIR_PATH : Temporary working directory (=> TMP)
	213	typeset -r RUN_DIR_PATH=${RUN_DIR_PATH:=${SCRATCH}/RUN_DIR/${SLURM_JOBID}_${$}}
[1474]	214
[1060]	215	if [ ! X${config_UserChoices_ARCHIVE} = X ]; then
[1057]	216	#====================================================
	217	#- ARCHIVE (dedicated to large files)
[1063]	218	ARCHIVE=${config_UserChoices_ARCHIVE}
[1057]	219	else
	220	#====================================================
	221	#- ARCHIVE (dedicated to large files)
[1490]	222	ARCHIVE=${STORE}
[1057]	223	fi
[725]	224
[1060]	225	if [ ! X${config_UserChoices_STORAGE} = X ]; then
[1057]	226	#====================================================
	227	#- STORAGE (dedicated to small/medium files)
[1063]	228	STORAGE=${config_UserChoices_STORAGE}
[1057]	229	else
	230	#====================================================
	231	#- STORAGE (dedicated to small/medium files)
[1490]	232	STORAGE=${WORK}
[1057]	233	fi
[725]	234
[1057]	235	if [ X${config_UserChoices_SpaceName} = XTEST ]; then
	236	#====================================================
	237	#- R_OUT
[1490]	238	R_OUT=${SCRATCH}/IGCM_OUT
[1057]	239
	240	#====================================================
	241	#- R_FIG (hosting figures : monitoring and atlas, and/or small files)
[1490]	242	R_FIG=${SCRATCH}/IGCM_OUT
[1057]	243
	244	IGCM_debug_Print 1 "SpaceName=TEST ==> OVERRULE destination path directories"
	245
	246	else
	247	#====================================================
	248	#- R_OUT
[1063]	249	R_OUT=${ARCHIVE}/IGCM_OUT
[1057]	250
	251	#====================================================
	252	#- R_FIG (hosting figures : monitoring and atlas, and/or small files)
[1063]	253	R_FIG=${STORAGE}/IGCM_OUT
[1057]	254	fi
	255
	256	#====================================================
[1431]	257	#- CMIP6 (hosting CMIP6 files produced by XIOS2 and configured by dr2xml)
	258	CMIP6_BUF=${STORAGE}/IGCM_OUT
	259
	260	#====================================================
[1057]	261	#- R_BUF (ONLY FOR double copy an scratch)
[1490]	262	R_BUF=${SCRATCH}/IGCM_OUT
[1057]	263
	264	IGCM_debug_Print 1 "R_OUT has been defined = ${R_OUT}"
	265	IGCM_debug_Print 1 "R_BUF has been defined = ${R_BUF}"
	266	IGCM_debug_Print 1 "R_FIG has been defined = ${R_FIG}"
	267
	268	IGCM_debug_PopStack "IGCM_sys_defineArchives"
[725]	269	}
	270
[622]	271	#D-#==================================================
	272	#D-function IGCM_sys_RshArchive
	273	#D-* Purpose: Archive rsh command
	274	#D-* Examples:
	275	#D-
	276	function IGCM_sys_RshArchive {
[623]	277	IGCM_debug_PushStack "IGCM_sys_RshArchive" $@
	278	/bin/ksh <<-EOF
[622]	279	${@}
	280	EOF
[811]	281	status=$?
	282	if [ ${status} -gt 0 ] ; then
	283	IGCM_debug_Print 2 "IGCM_sys_RshArchive : command failed error code ${status}"
[623]	284	IGCM_debug_Exit "IGCM_sys_RshArchive"
	285	fi
	286	IGCM_debug_PopStack "IGCM_sys_RshArchive"
[622]	287	}
	288
	289	#D-#==================================================
[845]	290	#D-function IGCM_sys_RshArchive_NoError
	291	#D-* Purpose: Archive rsh command, without error
	292	#D-* used only in monitoring.job
	293	#D-* Examples:
	294	#D-
	295	function IGCM_sys_RshArchive_NoError {
	296	IGCM_debug_PushStack "IGCM_sys_RshArchive_NoError" $@
	297	/bin/ksh <<-EOF
[1181]	298	${@} 2> /dev/null
[845]	299	EOF
	300	IGCM_debug_PopStack "IGCM_sys_RshArchive_NoError"
	301	}
	302
	303	#D-#==================================================
[622]	304	#D-function IGCM_sys_MkdirArchive
	305	#D-* Purpose: Mkdir on Archive
	306	#D-* Examples:
	307	#D-
	308	function IGCM_sys_MkdirArchive {
[623]	309	IGCM_debug_PushStack "IGCM_sys_MkdirArchive" $@
	310	if ( $DEBUG_sys ) ; then
	311	echo "IGCM_sys_MkdirArchive :" $@
	312	fi
	313	#- creation de repertoire sur le serveur fichier
	314	if [ ! -d ${1} ]; then
	315	\mkdir -p $1
[811]	316	status=$?
	317
	318	if [ ${status} -gt 0 ] ; then
	319	IGCM_debug_Print 2 "IGCM_sys_MkdirArchive : mkdir failed error code ${status}"
[623]	320	IGCM_debug_Exit "IGCM_sys_MkdirArchive"
[622]	321	fi
[623]	322	fi
	323	IGCM_debug_PopStack "IGCM_sys_MkdirArchive"
[622]	324	}
	325
	326	#D-#==================================================
	327	#D-function IGCM_sys_TestDirArchive
	328	#D-* Purpose: Test Directory that must exists on Archive
	329	#D-* Examples:
	330	#D-
	331	function IGCM_sys_TestDirArchive {
[623]	332	IGCM_debug_PushStack "IGCM_sys_TestDirArchive" $@
	333	if ( $DEBUG_sys ) ; then
	334	echo "IGCM_sys_TestDirArchive :" $@
	335	fi
	336	typeset ExistFlag
	337	ExistFlag=$( [ -d $1 ] && echo 0 \|\| echo 1 )
	338	IGCM_debug_PopStack "IGCM_sys_TestDirArchive"
	339	return ${ExistFlag}
[622]	340	}
	341
	342	#D-#==================================================
[623]	343	#D-function IGCM_sys_IsFileArchived
[1179]	344	#D-* Purpose: Test file that must NOT EXISTS on Archive based on filename only
[623]	345	#D-* Examples:
	346	#D-
	347	function IGCM_sys_IsFileArchived {
	348	IGCM_debug_PushStack "IGCM_sys_IsFileArchived" $@
	349	if ( $DEBUG_sys ) ; then
	350	echo "IGCM_sys_IsFileArchived :" $@
	351	fi
	352	typeset IsArchivedFlag
[818]	353	IsArchivedFlag=$( [ X$( echo $@ \| grep ^\/ccc\/store ) != X ] && echo 0 \|\| echo 1 )
[623]	354	IGCM_debug_PopStack "IGCM_sys_IsFileArchived"
	355
	356	return ${IsArchivedFlag}
	357	}
	358
	359	#D-#==================================================
[622]	360	#D-function IGCM_sys_TestFileArchive
	361	#D-* Purpose: Test file that must NOT EXISTS on Archive
	362	#D-* Examples:
	363	#D-
	364	function IGCM_sys_TestFileArchive {
[623]	365	IGCM_debug_PushStack "IGCM_sys_TestFileArchive" $@
	366	typeset ExistFlag
	367	ExistFlag=$( [ -f $1 ] && echo 0 \|\| echo 1 )
	368	IGCM_debug_PopStack "IGCM_sys_TestFileArchive"
[622]	369
[623]	370	return ${ExistFlag}
[622]	371	}
	372
	373	#D-#==================================================
	374	#D-function IGCM_sys_CountFileArchive
	375	#D-* Purpose: Count files on Archive filesystem
	376	#D-* Examples:
	377	#D-
	378	function IGCM_sys_CountFileArchive {
[623]	379	IGCM_debug_PushStack "IGCM_sys_CountFileArchive" $@
	380	ls ${@} 2>/dev/null \| wc -l
	381	if [ $? -gt 0 ] ; then
	382	echo "IGCM_sys_CountFileArchive : erreur."
	383	fi
	384	IGCM_debug_PopStack "IGCM_sys_CountFileArchive"
[622]	385	}
	386
	387	#D-#==================================================
	388	#D-function IGCM_sys_Tree
	389	#D-* Purpose: Tree directories with files on ${ARCHIVE}
	390	#D-* Examples: IGCM_sys_Tree ${R_IN} ${R_OUT}
	391	#D-
	392	function IGCM_sys_Tree {
[623]	393	IGCM_debug_PushStack "IGCM_sys_Tree" $@
	394	if ( $DEBUG_sys ) ; then
	395	echo "IGCM_sys_Tree :" $@
	396	fi
[622]	397
[623]	398	\ls -lR ${@}
[622]	399
[623]	400	IGCM_debug_PopStack "IGCM_sys_Tree"
[622]	401	}
	402
	403	#D-#==================================================
	404	#D-function IGCM_sys_Qsub
	405	#D-* Purpose: Qsub new job
	406	#D-* Examples:
	407	#D-
	408	function IGCM_sys_Qsub {
[623]	409	IGCM_debug_PushStack "IGCM_sys_Qsub" $@
	410	if ( $DEBUG_sys ) ; then
	411	echo "IGCM_sys_Qsub :" $@
	412	fi
[811]	413	typeset options status
[923]	414	options="-o ${SUBMIT_DIR}/${Script_Output} -e ${SUBMIT_DIR}/${Script_Output}"
[622]	415
[1490]	416	/usr/bin/time sbatch ${options} $1 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[811]	417	status=$?
[948]	418
[1179]	419	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[811]	420	if [ ${status} -gt 0 ] ; then
	421	IGCM_debug_Print 2 "IGCM_sys_Qsub ${options} $1 : error code ${status}"
[623]	422	IGCM_debug_Exit "IGCM_sys_Qsub"
[948]	423	else
[1179]	424	JobID=$( gawk {'print $4'} ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ )
[623]	425	fi
	426	IGCM_debug_PopStack "IGCM_sys_Qsub"
[622]	427	}
	428
	429	#D-#==================================================
	430	#D-function IGCM_sys_QsubPost
	431	#D-* Purpose: Qsub new job on scalaire
	432	#D-* Examples:
	433	#D-
	434	function IGCM_sys_QsubPost {
[623]	435	IGCM_debug_PushStack "IGCM_sys_QsubPost" $@
	436	if ( $DEBUG_sys ) ; then
	437	echo "IGCM_sys_QsubPost :" $@
	438	fi
[923]	439	typeset options status
[1490]	440	options="-o ${POST_DIR}/${Script_Post_Output}.out -e ${POST_DIR}/${Script_Post_Output}.out"
[923]	441
[1490]	442	sbatch ${options} ${libIGCM_POST}/$1.job > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[811]	443	status=$?
[948]	444
[1179]	445	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[811]	446	if [ ${status} -gt 0 ] ; then
[923]	447	IGCM_debug_Print 2 "IGCM_sys_QsubPost ${options} ${libIGCM_POST}/$1.job : error code ${status}"
[623]	448	IGCM_debug_Exit "IGCM_sys_QsubPost"
[948]	449	else
[1179]	450	JobID=$( gawk {'print $4'} ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ )
[623]	451	fi
	452	IGCM_debug_PopStack "IGCM_sys_QsubPost"
[622]	453	}
	454
	455	#D-*************************
[623]	456	#D- File transfer functions
[622]	457	#D-*************************
	458	#D-
	459
	460	#D-#==================================================
	461	#D-function IGCM_sys_RmRunDir
	462	#D-* Purpose: rm tmpdir (dummy function most of the time batch
	463	#D- scheduler will do the job)
	464	#D-* Examples:
	465	#D-
	466	function IGCM_sys_RmRunDir {
[1057]	467	IGCM_debug_PushStack "IGCM_sys_RmRunDir" $@
[623]	468	if ( $DEBUG_sys ) ; then
	469	echo "IGCM_sys_RmRunDir :" $@
	470	fi
[622]	471
[715]	472	typeset status
[622]	473
[1179]	474	echo rm $@ > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
	475	\rm $@ >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[715]	476	status=$?
[622]	477
[715]	478	if [ ${status} -gt 0 ] ; then
[1183]	479	IGCM_debug_Print 1 "IGCM_sys_RmRunDir : rm error code is ${status}."
[1179]	480	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[623]	481	IGCM_debug_Exit "IGCM_sys_RmRunDir"
	482	else
[1179]	483	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[623]	484	fi
	485	IGCM_debug_PopStack "IGCM_sys_RmRunDir"
[622]	486	}
	487
	488	#D-#==================================================
	489	#D-function IGCM_sys_Put_Dir
	490	#D-* Purpose: Copy a complete directory on $(ARCHIVE)
	491	#D-* Examples:
	492	#D-
	493	function IGCM_sys_Put_Dir {
[623]	494	IGCM_debug_PushStack "IGCM_sys_Put_Dir" $@
	495	if ( $DEBUG_sys ) ; then
	496	echo "IGCM_sys_Put_Dir :" $@
	497	fi
	498	if [ $DRYRUN = 0 ]; then
	499	if [ ! -d ${1} ] ; then
	500	echo "WARNING : IGCM_sys_Put_Dir ${1} DOES NOT EXIST ."
	501	IGCM_debug_PopStack "IGCM_sys_Put_Dir"
	502	return
[622]	503	fi
	504
[715]	505	typeset status
[622]	506
[623]	507	# Only if we use rsync
	508	#IGCM_sys_TestDirArchive $( dirname $2 )
	509	#
	510	#USUAL WAY
[1179]	511	\cp -r $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[715]	512	status=$?
[622]	513
[715]	514	if [ ${status} -gt 0 ] ; then
[811]	515	IGCM_debug_Print 2 "IGCM_sys_Put_Dir : cp failed error code ${status}"
[1179]	516	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[623]	517	IGCM_debug_Exit "IGCM_sys_Put_Dir"
[622]	518	else
[1179]	519	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[622]	520	fi
[623]	521	fi
	522	IGCM_debug_PopStack "IGCM_sys_Put_Dir"
[622]	523	}
	524
	525	#D-#==================================================
	526	#D-function IGCM_sys_Get_Dir
[623]	527	#D-* Purpose: Copy a complete directory from ${ARCHIVE}
[622]	528	#D-* Examples:
	529	#D-
	530	function IGCM_sys_Get_Dir {
[623]	531	IGCM_debug_PushStack "IGCM_sys_Get_Dir" $@
	532	if ( $DEBUG_sys ) ; then
	533	echo "IGCM_sys_Get_Dir :" $@
	534	fi
	535	if [ $DRYRUN = 0 ]; then
[708]	536	typeset NB_ESSAI DELAI status i
	537	# number of tentative
	538	NB_ESSAI=3
	539	# time delay between tentative
	540	DELAI=2
[622]	541
[676]	542	#
[708]	543	# USUAL WAY
	544	# add 'ccc_hsm get' (to demigrate all offline files) to reduce time of this command :
[713]	545	ccc_hsm get -r $1
[622]	546
[709]	547	i=0
[708]	548	while [ $i -lt $NB_ESSAI ] ; do
[1179]	549	\cp -ur $1 $2 >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[708]	550	status=$?
[811]	551	if [ ${status} -gt 0 ] ; then
[713]	552	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : cp failed error code ${status} ${i}/${NB_ESSAI}"
[708]	553	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : sleep ${DELAI} seconds and try again."
	554	sleep $DELAI
	555	else
	556	break
	557	fi
	558	(( i = i + 1 ))
	559	done
	560
	561	if [ ${status} -gt 0 ] ; then
[1178]	562	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : cp failed error code ${status}"
[1179]	563	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[623]	564	IGCM_debug_Exit "IGCM_sys_Get_Dir"
[622]	565	else
[1179]	566	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[622]	567	fi
[623]	568	fi
	569	IGCM_debug_PopStack "IGCM_sys_Get_Dir"
[622]	570	}
	571
	572	#D-#==================================================
	573	#D-function IGCM_sys_Put_Rest
[623]	574	#D-* Purpose: Put computied restarts on ${ARCHIVE}.
[622]	575	#D- File and target directory must exist.
	576	#D-* Examples:
	577	#D-
	578	function IGCM_sys_Put_Rest {
[623]	579	IGCM_debug_PushStack "IGCM_sys_Put_Rest" $@
	580	if ( $DEBUG_sys ) ; then
	581	echo "IGCM_sys_Put_Rest :" $@
	582	fi
	583	if [ $DRYRUN = 0 ]; then
	584	if [ ! -f ${1} ] ; then
	585	echo "ERROR : IGCM_sys_Put_Rest ${1} DOES NOT EXIST ."
	586	IGCM_debug_Exit "IGCM_sys_Put_Rest"
[622]	587	fi
	588
[715]	589	typeset status
[623]	590	#
	591	# USUAL WAY
[1179]	592	\cp $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[715]	593	status=$?
[622]	594
[623]	595	# #RSYNC WITH NETWORK SSH CALL
[1179]	596	# echo ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
	597	# ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[622]	598
[623]	599	# #RSYNC WITH NFS USE
[1179]	600	# echo ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
	601	# ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[622]	602
[715]	603	# status=$?
	604	# IGCM_sys_Rsync_out $status
[622]	605
[1179]	606	# ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[715]	607	# (( status=status+$? ))
[623]	608
[715]	609	if [ ${status} -gt 0 ] ; then
[1179]	610	IGCM_debug_Print 2 "IGCM_sys_Put_Rest : cp failed error code ${status}"
[765]	611	[ -f ${1} ] && ls -l ${1}
[759]	612	[ -f ${2} ] && ls -l ${2}
	613	[ -f ${2}/${1} ] && ls -l ${2}/${1}
[1179]	614	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[623]	615	IGCM_debug_Exit "IGCM_sys_Put_Rest"
[622]	616	else
[759]	617
	618	if [ X${JobType} = XRUN ] ; then
	619	[ -f ${2} ] && IGCM_sys_Chmod 444 ${2}
	620	[ -f ${2}/${1} ] && IGCM_sys_Chmod 444 ${2}/${1}
	621	fi
	622
[1179]	623	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[622]	624	fi
[623]	625	fi
	626	IGCM_debug_PopStack "IGCM_sys_Put_Rest"
[622]	627	}
	628
	629	#D-#==================================================
	630	#D-function IGCM_sys_Put_Out
[623]	631	#D-* Purpose: Copy a file on ${ARCHIVE} after having chmod it in readonly
[622]	632	#D-* Examples:
	633	#D-
	634	function IGCM_sys_Put_Out {
[623]	635	IGCM_debug_PushStack "IGCM_sys_Put_Out" $@
	636	if ( $DEBUG_sys ) ; then
	637	echo "IGCM_sys_Put_Out :" $@
	638	fi
[667]	639
[711]	640	typeset NB_ESSAI DELAI status i exist skip
[1234]	641	typeset fileDeviceNumberInHex directoryDeviceNumberInHex
[667]	642
	643	# number of tentative
[668]	644	NB_ESSAI=3
[667]	645	# time delay between tentative
[668]	646	DELAI=2
[667]	647
[623]	648	if [ $DRYRUN = 0 ]; then
	649	if [ ! -f ${1} ] ; then
	650	echo "WARNING : IGCM_sys_Put_Out ${1} DOES NOT EXIST ."
	651	IGCM_debug_PopStack "IGCM_sys_Put_Out"
	652	return 1
[622]	653	fi
[623]	654	#
	655	IGCM_sys_MkdirArchive $( dirname $2 )
	656	#
[711]	657	exist=false
	658	skip=false
	659	if [ -f $2 ] ; then
	660	IGCM_debug_Print 1 "$2 already exist"
	661	ccc_hsm get $2
	662	exist=true
	663	if [ "X$( diff $1 $2 )" = X ] ; then
	664	IGCM_debug_Print 2 "$1 and $2 are the same file, we skip the copy"
	665	skip=true
[667]	666	else
[711]	667	IGCM_debug_Print 2 "$1 and $2 are not the same file, we force the copy"
	668	skip=false
[667]	669	fi
[711]	670	fi
	671	#
	672	if ( [ X${exist} = Xtrue ] && [ X${skip} = Xfalse ] ) ; then
	673	IGCM_sys_Chmod u+w $2
	674	fi
[622]	675
[711]	676	if [ X${skip} = Xfalse ] ; then
[719]	677	i=0
[711]	678	while [ $i -lt $NB_ESSAI ] ; do
[1234]	679	# Identify file system
	680	fileDeviceNumberInHex=$( stat -c %d $1 )
	681	status=$?
	682	if [ ${status} -gt 0 ] ; then
	683	IGCM_debug_Exit "IGCM_sys_Put_Out"
	684	fi
	685	# Identify file system
	686	directoryDeviceNumberInHex=$( stat -c %d $( dirname $2 ) )
	687	status=$?
	688	if [ ${status} -gt 0 ] ; then
	689	IGCM_debug_Exit "IGCM_sys_Put_Out"
	690	fi
	691
[1412]	692	if ( [ ${fileDeviceNumberInHex} -ne ${directoryDeviceNumberInHex} ] \|\| [ X$3 = XNOMOVE ] ) ; then
[1234]	693	# They are not on the same device. USUAL WAY
[1179]	694	\cp $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[1037]	695	status=$?
	696	else
[1234]	697	# They are on the same device. NOT SO USUAL WAY
[1179]	698	\mv $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[1037]	699	status=$?
	700	fi
[711]	701	if [ ${status} -gt 0 ]; then
[713]	702	IGCM_debug_Print 2 "IGCM_sys_Put_Out : cp failed error code ${status} ${i}/${NB_ESSAI}"
[711]	703	IGCM_debug_Print 2 "IGCM_sys_Put_Out : sleep ${DELAI} seconds and try again."
[765]	704	[ -f ${1} ] && ls -l ${1}
[759]	705	[ -f ${2} ] && ls -l ${2}
	706	[ -f ${2}/${1} ] && ls -l ${2}/${1}
[711]	707	sleep $DELAI
	708	else
	709	break
	710	fi
	711	(( i = i + 1 ))
	712	done
	713	fi
	714
[623]	715	# #RSYNC WITH NETWORK SSH CALL
[1179]	716	# echo ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
	717	# ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[622]	718
[623]	719	# #RSYNC WITH NFS USE
[1179]	720	# echo ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
	721	# ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[622]	722
[711]	723	# status=$?
	724	# IGCM_sys_Rsync_out $status
[622]	725
[1179]	726	# ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[711]	727	# (( status=status+$? ))
[622]	728
[711]	729	if [ ${status} -gt 0 ] ; then
[1179]	730	IGCM_debug_Print 2 "IGCM_sys_Put_Out : cp failed error code ${status}"
[765]	731	[ -f ${1} ] && ls -l ${1}
[759]	732	[ -f ${2} ] && ls -l ${2}
	733	[ -f ${2}/${1} ] && ls -l ${2}/${1}
[1179]	734	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[623]	735	IGCM_debug_Exit "IGCM_sys_Put_Out"
[622]	736	else
[759]	737
	738	if [ X${JobType} = XRUN ] ; then
	739	if [ X${3} = X ] ; then
	740	[ -f ${2} ] && IGCM_sys_Chmod 444 ${2}
	741	[ -f ${2}/${1} ] && IGCM_sys_Chmod 444 ${2}/${1}
	742	fi
	743	fi
	744
[1179]	745	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[622]	746	fi
[623]	747	fi
	748	IGCM_debug_PopStack "IGCM_sys_Put_Out"
	749	return 0
[622]	750	}
	751
	752	#D-#==================================================
	753	#D-function IGCM_sys_Get
	754	#D-* Purpose: Get a file from ${ARCHIVE}
	755	#D-* Examples: IGCM_sys_Get myfile /destpath/myfile_with_PREFIX
	756	#D- IGCM_sys_Get /l Array_contain_myfiles /destpath/
	757	function IGCM_sys_Get {
[623]	758	IGCM_debug_PushStack "IGCM_sys_Get" $@
[622]	759
[773]	760	typeset DEST dm_liste target file_work
[667]	761	typeset NB_ESSAI DELAI status i
[622]	762
[623]	763	if ( $DEBUG_sys ) ; then
	764	echo "IGCM_sys_Get :" $@
	765	fi
[667]	766
	767	# number of tentative
[668]	768	NB_ESSAI=3
[667]	769	# time delay between tentative
[668]	770	DELAI=2
[667]	771
[623]	772	if [ $DRYRUN -le 2 ]; then
	773	if [ X${1} = X'/l' ] ; then
	774	eval set +A dm_liste \${${2}}
	775	else
	776	eval set +A dm_liste ${1}
[622]	777	fi
[623]	778	eval DEST=\${${#}}
[1179]	779	ccc_hsm get ${dm_liste[*]} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[715]	780	status=$?
	781	if [ ${status} -gt 0 ] ; then
	782	echo "WARNING IGCM_sys_Get : error code ${status}"
[1179]	783	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[623]	784	echo "WARNING IGCM_sys_Get : will stop later if the cp fails."
	785	fi
[622]	786
[715]	787	#if [ ${status} -gt 0 ] ; then
[1179]	788	# if [ ! "X$( grep "Lost dmusrcmd connection" ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ )" = "X" ] ; then
	789	# cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[785]	790	# echo "WARNING IGCM_sys_Get : Lost dmusrcmd connection : "
	791	# sleep 30
	792	# echo "We try another time"
[1179]	793	## dmget ${dm_liste[*]} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
	794	# ccc_hsm get ${dm_liste[*]} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[785]	795	# status=$?
	796	# if [ ${status} -gt 0 ] ; then
	797	# echo "ERROR IGCM_sys_Get : again demigration error :"
[1179]	798	# cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[785]	799	# IGCM_debug_Exit "IGCM_sys_Get"
[623]	800	# fi
[785]	801	# else
	802	# echo "ERROR IGCM_sys_Get : demigration error :"
[1179]	803	# cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[785]	804	# IGCM_debug_Exit "IGCM_sys_Get"
	805	# fi
[623]	806	#fi
[622]	807
[719]	808	# #RSYNC WITH NETWORK SSH CALL
[1179]	809	# echo ${RSYNC} ${RSYNC_opt} -e ssh ${STOREHOST}:"${dm_liste}" ${STOREHOST}:${RUN_DIR}/${DEST} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
	810	# ${RSYNC} ${RSYNC_opt} -e ssh ${STOREHOST}:"${dm_liste}" ${STOREHOST}:${RUN_DIR}/${DEST} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[719]	811
	812	# #RSYNC WITH NFS USE
[1179]	813	# echo ${RSYNC} ${RSYNC_opt} ${dm_liste} ${RUN_DIR}/${DEST} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
	814	# ${RSYNC} ${RSYNC_opt} ${dm_liste} ${RUN_DIR}/${DEST} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[719]	815
	816	# status=$?
	817	# IGCM_sys_Rsync_out $status
	818
[1179]	819	# ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[719]	820	# (( status=status+$? ))
	821
[623]	822	#USUAL WAY
	823	if [ X${1} = X'/l' ] ; then
	824	for target in ${dm_liste[*]} ; do
	825	local_file=$( basename ${target} )
[668]	826	# test if the target file is present before the loop
	827	IGCM_sys_TestFileArchive ${target}
[715]	828	status=$?
	829	if [ ${status} -gt 0 ] ; then
[701]	830	echo "IGCM_sys_Get, ERROR : regular file ${target} DOES NOT EXIST ."
[668]	831	IGCM_debug_Exit "IGCM_sys_Get"
	832	else
[719]	833	i=0
[668]	834	while [ $i -lt $NB_ESSAI ] ; do
[1037]	835	#if [ X${DoLink} = Xtrue ] ; then
[1179]	836	# \ln -s ${target} ${DEST}/${local_file} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[1037]	837	# status=$?
	838	# else
[1179]	839	# \cp ${target} ${DEST}/${local_file} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[1037]	840	# status=$?
	841	#fi
[1179]	842	\ln -s ${target} ${DEST}/${local_file} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[668]	843	status=$?
	844	if [ ${status} -gt 0 ]; then
[713]	845	IGCM_debug_Print 2 "IGCM_sys_Get : cp failed error code ${status} ${i}/${NB_ESSAI}"
[676]	846	IGCM_debug_Print 2 "IGCM_sys_Get : sleep ${DELAI} seconds and try again."
	847	sleep $DELAI
[668]	848	else
[676]	849	break
[668]	850	fi
	851	(( i = i + 1 ))
	852	done
[719]	853	if [ ${status} -gt 0 ] ; then
	854	echo "IGCM_sys_Get : error"
[1179]	855	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
	856	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[719]	857	IGCM_debug_Exit "IGCM_sys_Get"
	858	else
[1179]	859	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[719]	860	fi
[668]	861	fi
[623]	862	done
	863	else
[719]	864	i=0
	865	while [ $i -lt $NB_ESSAI ] ; do
[1179]	866	\cp ${dm_liste} ${DEST} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[719]	867	status=$?
	868	if [ ${status} -gt 0 ]; then
	869	IGCM_debug_Print 2 "IGCM_sys_Get : cp failed error code ${status} ${i}/${NB_ESSAI}"
	870	IGCM_debug_Print 2 "IGCM_sys_Get : sleep ${DELAI} seconds and try again."
	871	sleep $DELAI
	872	else
	873	break
	874	fi
	875	(( i = i + 1 ))
	876	done
	877	if [ ${status} -gt 0 ] ; then
	878	echo "IGCM_sys_Get : error"
[1179]	879	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
	880	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[719]	881	IGCM_debug_Exit "IGCM_sys_Get"
	882	else
[1179]	883	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[719]	884	fi
[623]	885	fi
	886	fi
	887	IGCM_debug_PopStack "IGCM_sys_Get"
	888	}
[622]	889
[623]	890	#D-#==================================================
[835]	891	#D-function IGCM_sys_GetDate_Monitoring
	892	#D-* Purpose: get the last year for which the monitoring has been computed
	893	#D-* Examples:
	894	#D-
	895	function IGCM_sys_GetDate_Monitoring {
	896	IGCM_debug_PushStack "IGCM_sys_GetDate_Monitoring" $@
	897	if ( $DEBUG_sys ) ; then
	898	echo "IGCM_sys_GetDate_Monitoring :" $@
	899	fi
	900
	901	eval ${2}=$( cdo showyear ${1} 2> /dev/null \| gawk '{ print $NF }' )
	902
	903	IGCM_debug_PopStack "IGCM_sys_GetDate_Monitoring"
	904	}
	905
	906	#D-#==================================================
[623]	907	#D-function IGCM_sys_Dods_Rm
	908	#D-* Purpose: DO NOTHING ! Put ${ARCHIVE} files on DODS internet protocole.
	909	#D-* Examples:
	910	#D-
	911	function IGCM_sys_Dods_Rm {
	912	if ( $DEBUG_sys ) ; then
	913	echo "IGCM_sys_Dods_Rm :" $@
	914	fi
[715]	915	typeset status
[623]	916	if [ $DRYRUN = 0 ]; then
	917
	918	# if [ ! -d /dmnfs/cont003/dods/public/${LOGIN}/${R_DODS}/${1} ] ; then
	919	# echo "WARNING : IGCM_sys_Dods_Rm /dmnfs/cont003/dods/public/${LOGIN}/${R_DODS}/${1} DOES NOT EXIST ."
	920	# echo "Nothing has been done."
	921	# return
	922	# fi
	923
[1392]	924	if [ "$#" -eq 1 ]; then
[1457]	925	## /ccc/cont003/home/igcmg/igcmg/Tools/irene/thredds_rm public/${LOGIN}/${R_DODS}/${1} # > out_dods_rm 2>&1
	926	$(ccc_home -u igcmg)/Tools/irene/thredds_rm public/${LOGIN}/${R_DODS}/${1} # > out_dods_rm 2>&1
[1392]	927	status=$?
	928	else
[1457]	929	## /ccc/cont003/home/igcmg/igcmg/Tools/irene/thredds_rm public/${LOGIN}/${R_DODS} # > out_dods_rm 2>&1
	930	$(ccc_home -u igcmg)/Tools/irene/thredds_rm public/${LOGIN}/${R_DODS} # > out_dods_rm 2>&1
[1392]	931	status=$?
	932	fi
[623]	933
[785]	934	# if [ ${status} -gt 0 ] ; then
	935	# echo "IGCM_sys_Dods_Rm : error."
	936	# cat out_dods_rm
	937	# IGCM_debug_Exit "IGCM_sys_Dods_Rm"
	938	# else
	939	# rm out_dods_rm
	940	# fi
[623]	941
	942	fi
[715]	943	return $status
[623]	944	}
	945
	946	#D-#==================================================
	947	#D-function IGCM_sys_Dods_Cp
	948	#D-* Purpose: Copy $(ARCHIVE) files on DODS internet protocole.
	949	#D-* Examples:
	950	#D-
	951	function IGCM_sys_Dods_Cp {
	952	if ( $DEBUG_sys ) ; then
	953	echo "IGCM_sys_Dods_Cp :" $@
	954	fi
[715]	955	typeset status
[623]	956	if [ $DRYRUN = 0 ]; then
	957
	958	# if [ ! -d ${R_SAVE}/${1} ] ; then
	959	# echo "WARNING : IGCM_sys_Dods_Cp ${R_SAVE}/${1} DOES NOT EXIST ."
	960	# echo "Nothing has been done."
	961	# return
	962	# fi
	963
[1457]	964	## /ccc/cont003/home/igcmg/igcmg/Tools/irene/bin/thredds_cp ${1} public/${LOGIN}/${R_DODS} # > out_dods_cp 2>&1
	965	$(ccc_home -u igcmg)/Tools/irene/thredds_cp ${1} public/${LOGIN}/${R_DODS} # > out_dods_cp 2>&1
[715]	966	status=$?
[623]	967
[715]	968	# if [ ${status} -gt 0 ] ; then
[623]	969	# echo "IGCM_sys_Dods_Cp : error."
	970	# cat out_dods_cp
	971	# IGCM_debug_Exit "IGCM_sys_Dods_Cp"
	972	# else
	973	# rm out_dods_cp
	974	# fi
	975
	976	fi
[715]	977	return $status
[623]	978	}
	979
	980	#D-#==================================================
	981	#D-function IGCM_sys_Put_Dods
[1178]	982	#D-* Purpose: Put ${ARCHIVE} files on DODS internet protocole.
[623]	983	#D-* Examples:
	984	#D-
	985	function IGCM_sys_Put_Dods {
	986	IGCM_debug_PushStack "IGCM_sys_Put_Dods" $@
	987	if ( $DEBUG_sys ) ; then
	988	echo "IGCM_sys_Put_Dods :" $@
	989	fi
[715]	990	typeset status
[623]	991	if [ $DRYRUN = 0 ]; then
	992	if ( [ ! -d ${R_SAVE}/${1} ] && [ ! -d ${R_FIGR}/${1} ] ) ; then
	993	echo "WARNING IGCM_sys_Put_Dods : None of the following directories exist. Exactly one should."
	994	echo "WARNING IGCM_sys_Put_Dods : ${R_SAVE}/${1} DOES NOT EXIST."
	995	echo "WARNING IGCM_sys_Put_Dods : ${R_FIGR}/${1} DOES NOT EXIST."
	996	IGCM_debug_PopStack "IGCM_sys_Put_Dods"
	997	return
[622]	998	fi
[623]	999
	1000	if ( [ -d ${R_SAVE}/${1} ] && [ -d ${R_FIGR}/${1} ] ) ; then
	1001	echo "WARNING IGCM_sys_Put_Dods : Both of the following directories exist. Exactly one should."
	1002	echo "WARNING IGCM_sys_Put_Dods : ${R_SAVE}/${1} EXISTS."
	1003	echo "WARNING IGCM_sys_Put_Dods : ${R_FIGR}/${1} EXISTS."
	1004	IGCM_debug_PopStack "IGCM_sys_Put_Dods"
	1005	return
	1006	fi
	1007	#
	1008	if [ -d ${R_SAVE}/${1} ] ; then
	1009	cd ${R_SAVE}
	1010	elif [ -d ${R_FIGR}/${1} ] ; then
	1011	cd ${R_FIGR}
	1012	fi
	1013
	1014	IGCM_sys_Dods_Rm ${1}
	1015	IGCM_sys_Dods_Cp ${1}
[715]	1016	status=0
[623]	1017
[715]	1018	if [ ${status} -gt 0 ] ; then
[623]	1019	echo "IGCM_sys_Put_Dods : error."
	1020	IGCM_debug_Exit "IGCM_sys_Put_Dods"
	1021	fi
	1022	fi
	1023	IGCM_debug_PopStack "IGCM_sys_Put_Dods"
[622]	1024	}
	1025
	1026	##############################################################
	1027	# REBUILD OPERATOR
	1028
[811]	1029	#D-#==================================================
[1163]	1030	#D-function IGCM_sys_sync
	1031	#D-* Purpose: flush buffer on disk
	1032	#D-* Examples:
	1033	#D-
	1034	function IGCM_sys_sync {
	1035	IGCM_debug_PushStack "IGCM_sys_sync" $@
	1036	if ( $DEBUG_sys ) ; then
	1037	echo "IGCM_sys_sync :" $@
	1038	fi
	1039
	1040	/bin/sync
	1041
	1042	IGCM_debug_PopStack "IGCM_sys_sync"
	1043	}
	1044
[622]	1045	############################################################
	1046	# Activate Running Environnment Variables
	1047
[811]	1048	#D-#==================================================
[1239]	1049	#D-function IGCM_sys_activ_variables
[811]	1050	#D-* Purpose: set environement variables prior to execution
	1051	#D-* Examples:
	1052	#D-
[622]	1053	function IGCM_sys_activ_variables {
[623]	1054	IGCM_debug_PushStack "IGCM_sys_activ_variables"
	1055	if ( $DEBUG_sys ) ; then
	1056	echo "IGCM_sys_activ_variables"
	1057	fi
[622]	1058
[811]	1059	# --------------------------------------------------------------------
[1409]	1060	#D- MODULE specifications
	1061	# --------------------------------------------------------------------
[1479]	1062	# Source the file EnvFile containing all module load needed to run the model.
[1485]	1063	# EnvFile can be specified in config.card. If this is the case and if the file exists,
	1064	# it will be used. Otherwise a default file will be used.
[1482]	1065	if [ ! X${config_UserChoices_EnvFile} = X ] && [ -f ${config_UserChoices_EnvFile} ] ; then
	1066	# EnvFile is set in config.card and the file exists
	1067	IGCM_debug_Print 1 "EnvFile set in config.card will be used"
	1068	EnvFile=${config_UserChoices_EnvFile}
	1069	else
[1479]	1070	IGCM_debug_Print 1 "IGCM_sys_active_variables : Default modules will be used"
	1071	module purge > /dev/null 2>&1
[1490]	1072	EnvFile=/gpfswork/rech/psl/commun/MachineEnvironment/jeanzay/env_jeanzay
[1409]	1073	fi
[1479]	1074	IGCM_debug_Print 1 "IGCM_sys_active_variables : Following EnvFile will be used :"
	1075	IGCM_debug_Print 1 ${EnvFile}
	1076	. ${EnvFile}
	1077
[1490]	1078	IGCM_debug_Print 1 "IGCM_sys_active_variables : Now loaded modules for Jean-Zay. "
[1433]	1079	module list
[1409]	1080
	1081	# --------------------------------------------------------------------
[811]	1082	#D- MPI specifications
	1083	# --------------------------------------------------------------------
[1191]	1084
[811]	1085	# --------------------------------------------------------------------
	1086	#D- Other specifications
	1087	# --------------------------------------------------------------------
	1088
[623]	1089	ulimit -s unlimited
[622]	1090
[623]	1091	IGCM_debug_PopStack "IGCM_sys_activ_variables"
[622]	1092	}
	1093
	1094	############################################################
	1095	# Desactivate Running Environnment Variables
	1096
[811]	1097	#D-#==================================================
	1098	#D-function IGCM_sys_desactiv_variables
	1099	#D-* Purpose: unset environement variables after execution
	1100	#D-* Examples:
	1101	#D-
[622]	1102	function IGCM_sys_desactiv_variables {
[623]	1103	IGCM_debug_PushStack "IGCM_sys_desactiv_variables"
	1104	if ( $DEBUG_sys ) ; then
	1105	echo "IGCM_sys_desactiv_variables"
	1106	fi
[811]	1107	# --------------------------------------------------------------------
	1108	#D- MPI specifications
	1109	# --------------------------------------------------------------------
	1110
	1111	# --------------------------------------------------------------------
	1112	#D- Other specifications
	1113	# --------------------------------------------------------------------
	1114
[623]	1115	IGCM_debug_PopStack "IGCM_sys_desactiv_variables"
[622]	1116	}
	1117
[623]	1118	############################################################
[1238]	1119	# Update job headers to be used by the scheduler
	1120
	1121	#D-#==================================================
	1122	#D-function IGCM_sys_updateHeaders
	1123	#D-* Purpose: Update job headers to be used by the scheduler
	1124	#D-* Examples: IGCM_sys_updateHeaders /path/to/Job_MYEXP
	1125	#D-
	1126	function IGCM_sys_updateHeaders {
	1127	IGCM_debug_PushStack "IGCM_sys_updateHeaders"
	1128	if ( $DEBUG_sys ) ; then
	1129	echo "IGCM_sys_updateHeaders"
	1130	fi
	1131	typeset file
	1132	file=$1
	1133
	1134	if [ ${executionType} -eq 1 ] ; then
	1135	# MPMD + MPI
	1136	sed -e "/::openMPthreads::/d" \
	1137	-e "s/::JobNumProcTot::/${coreNumber}/" \
[1256]	1138	-e "/#MSUB -x/d" \
	1139	-e "/--cpu_bind=none/d" \
[1238]	1140	${file} > ${file}.tmp
	1141
	1142	elif [ ${executionType} -eq 2 ] ; then
[1390]	1143	# MPMD + MPI + OMP : mpirun/ccc_mprun/error
[1432]	1144	if ( [ "X${config_UserChoices_ExecutionModeOnCurie}" = "Xmpirun" ] ) ; then
[1390]	1145	sed -e "/::openMPthreads::/d" \
	1146	-e "s/::JobNumProcTot::/${coreNumber}/" \
	1147	${file} > ${file}.tmp
[1432]	1148	elif ( [ "X${config_UserChoices_ExecutionModeOnCurie}" = "X" ] \|\| [ "X${config_UserChoices_ExecutionModeOnCurie}" = "Xccc_mprun" ] ) ; then
[1390]	1149	sed -e "/::openMPthreads::/d" \
	1150	-e "s/::JobNumProcTot::/${coreNumber}/" \
	1151	-e "/--cpu_bind=none/d" \
	1152	${file} > ${file}.tmp
	1153	else
	1154	IGCM_debug_Print 1 "You have to set ExecutionModeOnCurie=ccc_mprun or mpirun in config.card"
	1155	IGCM_debug_Exit "IGCM_sys_updateHeaders"
	1156	fi
[1238]	1157	elif [ ${executionType} -eq 3 ] ; then
	1158	# SPMD + MPI/OMP
	1159	sed -e "s/::openMPthreads::/${openMPthreads}/" \
	1160	-e "s/::JobNumProcTot::/${mpiTasks}/" \
[1240]	1161	-e "/#MSUB -x/d" \
[1238]	1162	-e "/--cpu_bind=none/d" \
	1163	${file} > ${file}.tmp
	1164
	1165	elif [ ${executionType} -eq 4 ] ; then
	1166	# SPMD + MPI only
	1167	sed -e "s/::JobNumProcTot::/${mpiTasks}/" \
	1168	-e "/::openMPthreads::/d" \
[1240]	1169	-e "/#MSUB -x/d" \
[1238]	1170	-e "/--cpu_bind=none/d" \
	1171	${file} > ${file}.tmp
	1172
	1173	elif [ ${executionType} -eq 5 ] ; then
	1174	# SPMD + OMP only
	1175	sed -e "s/::openMPthreads::/${openMPthreads}/" \
	1176	-e "/::JobNumProcTot::/d" \
[1240]	1177	-e "/#MSUB -x/d" \
[1238]	1178	-e "/--cpu_bind=none/d" \
	1179	${file} > ${file}.tmp
	1180
	1181	elif [ ${executionType} -eq 6 ] ; then
	1182	# SEQUENTIAL THEN
	1183	sed -e "s/::JobNumProcTot::/1/" \
	1184	-e "/::openMPthreads::/d" \
[1240]	1185	-e "/#MSUB -x/d" \
[1238]	1186	-e "/--cpu_bind=none/d" \
	1187	${file} > ${file}.tmp
	1188
	1189	fi
	1190
	1191	IGCM_sys_Mv ${file}.tmp ${file}
	1192
	1193	IGCM_debug_PopStack "IGCM_sys_updateHeaders"
	1194	}
	1195
	1196	############################################################
[623]	1197	# Build MPI/OMP scripts run file (dummy function)
[622]	1198
[811]	1199	#D-#==================================================
	1200	#D-function IGCM_sys_build_run_file
	1201	#D-* Purpose: build run file (deprecated)
	1202	#D-* Examples:
	1203	#D-
[622]	1204	function IGCM_sys_build_run_file {
[811]	1205	IGCM_debug_Print 3 " dummy function : IGCM_sys_build_run_file "
[622]	1206	}
	1207
	1208	############################################################
	1209	# Build MPI/OMP scripts
[811]	1210
	1211	#D-#==================================================
	1212	#D-function IGCM_sys_build_execution_scripts
	1213	#D-* Purpose: build execution scripts to be launch by ${HOST_MPIRUN_COMMAND}
	1214	#D-* Examples:
	1215	#D-
[623]	1216	function IGCM_sys_build_execution_scripts
[622]	1217	{
[623]	1218	IGCM_debug_PushStack "IGCM_sys_build_execution_scripts" $@
	1219	if ( $DEBUG_sys ) ; then
	1220	echo "IGCM_sys_build_execution_scripts " $@
	1221	fi
[622]	1222
[623]	1223	EXECUTION=${HOST_MPIRUN_COMMAND}
[622]	1224
[623]	1225	if ( ${OK_PARA_MPMD} ) ; then
[622]	1226
[1390]	1227	# MPMD mode
	1228	# 1 MPI only
	1229	# 2 MPI/OpenMP mpirun method
	1230	# 3 MPI/OpenMP ccc_mprun method
	1231
[1381]	1232	if [ -f run_file ] ; then
	1233	IGCM_sys_Rm -f run_file
	1234	fi
	1235	touch run_file
	1236
[1390]	1237	# case 1 : Only MPI (MPMD)
[1078]	1238	if ( ! ${OK_PARA_OMP} ) ; then
[622]	1239
[1078]	1240	# Build run_file
[622]	1241
[1078]	1242	# First loop on the components for the coupler ie oasis (only if oasis3)
	1243	# the coupler ie oasis3 must be the first one
	1244	for comp in ${config_ListOfComponents[*]} ; do
[622]	1245
[1078]	1246	eval ExeNameIn=\${config_Executable_${comp}[0]}
	1247	eval ExeNameOut=\${config_Executable_${comp}[1]}
[622]	1248
[1078]	1249	# for CPL component only
	1250	if [ "X${comp}" = "XCPL" ] && [ "X${ExeNameOut}" != X\"\" ] ; then
[1000]	1251
[1078]	1252	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
	1253	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
	1254	echo "${comp_proc_mpi_loc} ./${ExeNameOut}" >> run_file
	1255	fi
	1256	done
[622]	1257
[1078]	1258	# Then second loop on the components
	1259	for comp in ${config_ListOfComponents[*]} ; do
[622]	1260
[1078]	1261	eval ExeNameIn=\${config_Executable_${comp}[0]}
	1262	eval ExeNameOut=\${config_Executable_${comp}[1]}
[622]	1263
[1078]	1264	# Only if we really have an executable for the component and not the coupler ie oasis:
	1265	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${comp}" != "XCPL" ] ) ; then
[622]	1266
[1078]	1267	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
	1268	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
	1269	echo "${comp_proc_mpi_loc} ./${ExeNameOut}" >> run_file
	1270	fi
	1271	done
[622]	1272
[1390]	1273	EXECUTION="${HOST_MPIRUN_COMMAND} -f ./run_file"
	1274
	1275	IGCM_sys_Chmod u+x run_file
	1276	if ( $DEBUG_sys ) ; then
	1277	echo "run_file contains : "
	1278	cat run_file
	1279	fi
	1280
[1432]	1281	elif ( [ "X${config_UserChoices_ExecutionModeOnCurie}" = "Xmpirun" ] ) ; then
[1390]	1282
	1283	# 2 MPI/OpenMP mpirun method
	1284
	1285	# Use of mpirun instead of ccc_mprun
	1286	EXECUTION="time mpirun"
	1287
	1288	# Hosts treatment
	1289	${EXECUTION} hostname \| sort \| uniq > hosts.tmp
	1290
	1291	i=0
	1292	rm -f hosts rankfile
[1462]	1293	IGCM_debug_Print 1 "sys Irene, Hosts available :"
[1390]	1294	for nodes in `cat hosts.tmp`
	1295	do
	1296	host[$i]=$nodes
	1297	echo "${host[$i]}" >> hosts
	1298	IGCM_debug_Print 1 ${host[$i]}
	1299	i=$((i+1))
	1300	done
	1301	rm -f hosts.tmp
	1302
	1303	listnodes=${host[*]}
	1304
	1305	EXECUTION="${EXECUTION} -hostfile hosts -rankfile rankfile"
	1306
	1307	# Initialisation
	1308	rank=0
	1309	current_core=0
	1310	init_exec=n
	1311
	1312	# Loop on the components
	1313	for comp in ${config_ListOfComponents[*]} ; do
	1314
	1315	eval ExeNameIn=\${config_Executable_${comp}[0]}
	1316	eval ExeNameOut=\${config_Executable_${comp}[1]}
	1317
	1318	# Not possible if oasis has an executable (i.e old version of oasis3)
	1319	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${comp}" = "XCPL" ] ) ; then
	1320	IGCM_debug_Exit "ERROR MPMD with hybrid MPI-OpenMP is not available with oasis3 version"
	1321	IGCM_debug_Print 2 "Only available with oasis3-MCT version coupler"
	1322	IGCM_debug_Verif_Exit
	1323	fi
	1324
	1325	# Only if we really have an executable for the component :
	1326	if [ "X${ExeNameOut}" != X\"\" ] ; then
	1327
	1328	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
	1329	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
[1426]	1330	eval comp_proc_nod_loc=\${${comp}_PROC_NOD}
[1390]	1331
	1332	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
	1333	echo "" >> script_${ExeNameOut}.ksh
	1334	if [ ${comp_proc_omp_loc} -gt 1 ] ; then
	1335
	1336	# Check if the number of threads is correct
	1337	case ${comp_proc_omp_loc} in
	1338	2\|4\|8\|16)
	1339	IGCM_debug_Print 1 "You run ${ExeNameOut} on ${comp_proc_omp_loc} OMP threads"
	1340	;;
	1341	*)
	1342	IGCM_debug_Exit "ERROR with OMP parameters !"
	1343	IGCM_debug_Print 2 "${comp_proc_omp_loc} is not possible as number of OMP threads"
	1344	IGCM_debug_Print 2 "Only 2,4,8,16 as number of OMP threads are possible "
	1345	IGCM_debug_Verif_Exit
	1346	;;
	1347	esac
	1348	echo "export KMP_STACKSIZE=3g" >> script_${ExeNameOut}.ksh
	1349	echo "export KMP_LIBRARY=turnaround" >> script_${ExeNameOut}.ksh
	1350	echo "export MKL_SERIAL=YES" >> script_${ExeNameOut}.ksh
	1351	echo "OMP_NUM_THREADS=${comp_proc_omp_loc}" >> script_${ExeNameOut}.ksh
	1352	fi
	1353
	1354	#echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
	1355	echo "(( MYMPIRANK = OMPI_COMM_WORLD_RANK )) " >> script_${ExeNameOut}.ksh
	1356	echo "MYMPIRANK=\$(printf '%3.3d\n' \${MYMPIRANK})" >> script_${ExeNameOut}.ksh
	1357	echo "./${ExeNameOut} > out_${ExeNameOut}.out.\${MYMPIRANK} 2>out_${ExeNameOut}.err.\${MYMPIRANK}" >> script_${ExeNameOut}.ksh
	1358	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
	1359
	1360	if [ ${init_exec} = y ] ; then
	1361	EXECUTION="${EXECUTION} : -np ${comp_proc_mpi_loc} ./script_${ExeNameOut}.ksh"
	1362	else
	1363	EXECUTION="${EXECUTION} -np ${comp_proc_mpi_loc} ./script_${ExeNameOut}.ksh"
	1364	init_exec=y
	1365	fi
	1366
	1367	# Build rankfile : method used to assign cores and nodes for the MPI process
	1368	# Ex :
	1369	#rank 0=curie5296 slot=0,1,2,3
	1370	#rank 1=curie5296 slot=4,5,6,7
	1371	# Example of final command :
	1372	# mpirun -hostfile hosts -rankfile rankfile -np 27 ./script_lmdz.x.ksh : -np 5 ./script_opa.xx.ksh
	1373	# with script_lmdz.x.ksh :
	1374	# #!/bin/ksh
	1375	#export KMP_STACKSIZE=3g
	1376	#export KMP_LIBRARY=turnaround
	1377	#export MKL_SERIAL=YES
	1378	#OMP_NUM_THREADS=4
	1379	#./lmdz.x
[1426]	1380	if [ ${comp_proc_nod_loc} -gt 1 ] ; then
[1430]	1381	(( offset_comp_proc_loc = NB_CORE_PER_NODE / (comp_proc_mpi_loc / comp_proc_nod_loc) ))
[1426]	1382	else
	1383	(( offset_comp_proc_loc = comp_proc_omp_loc ))
	1384	fi
[1390]	1385	for nb_proc_mpi in `seq 0 $(($comp_proc_mpi_loc-1))`; do
[1426]	1386	(( index_host = current_core / NB_CORE_PER_NODE ))
	1387	host_value=${host[${index_host}]}
	1388	(( slot = current_core % NB_CORE_PER_NODE ))
	1389	virg=","
	1390	string_final=""
	1391	for index in `seq $slot $(($slot+$comp_proc_omp_loc-1))`; do
	1392	string=$index$virg
	1393	string_final=$string_final$string
	1394	done
	1395	string_final=$( echo $string_final \| sed "s/.$//" )
	1396	echo "rank $rank=$host_value slot=$string_final" >> rankfile
	1397	(( rank = rank + 1 ))
	1398	(( current_core = current_core + offset_comp_proc_loc ))
[1390]	1399	done
	1400	fi
	1401	done
	1402
[1432]	1403	elif ( [ "X${config_UserChoices_ExecutionModeOnCurie}" = "X" ] \|\| [ "X${config_UserChoices_ExecutionModeOnCurie}" = "Xccc_mprun" ] ) ; then
[1390]	1404
	1405	# 3 MPI/OpenMP ccc_mprun method
	1406
[1381]	1407	# MPI-OpenMP (MPMD)
	1408	# example of run_file
	1409	# 71-8 env OMP_NUM_THREADS=8 ./script_lmdz.x.ksh
	1410	# 480-1 env OMP_NUM_THREADS=1 ./script_opa.xx.ksh
	1411	# 1-1 env OMP_NUM_THREADS=1 ./script_xios.x.ksh
[622]	1412
[1078]	1413	# Loop on the components
	1414	for comp in ${config_ListOfComponents[*]} ; do
[1000]	1415
[1078]	1416	eval ExeNameIn=\${config_Executable_${comp}[0]}
	1417	eval ExeNameOut=\${config_Executable_${comp}[1]}
[1000]	1418
[1078]	1419	# Not possible if oasis has an executable (i.e old version of oasis3)
	1420	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${comp}" = "XCPL" ] ) ; then
	1421	IGCM_debug_Exit "ERROR MPMD with hybrid MPI-OpenMP is not available with oasis3 version"
	1422	IGCM_debug_Print 2 "Only available with oasis3-MCT version coupler"
	1423	IGCM_debug_Verif_Exit
	1424	fi
[1000]	1425
[1078]	1426	# Only if we really have an executable for the component :
	1427	if [ "X${ExeNameOut}" != X\"\" ] ; then
[1000]	1428
[1078]	1429	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
	1430	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
[1000]	1431
[1381]	1432	# Build script files
	1433
[1078]	1434	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
	1435	echo "" >> script_${ExeNameOut}.ksh
	1436	if [ ${comp_proc_omp_loc} -gt 1 ] ; then
	1437
	1438	# Check if the number of threads is correct
	1439	case ${comp_proc_omp_loc} in
[1440]	1440	2\|4\|8\|12\|16\|24)
[1078]	1441	IGCM_debug_Print 1 "You run ${ExeNameOut} on ${comp_proc_omp_loc} OMP threads"
	1442	;;
	1443	*)
	1444	IGCM_debug_Exit "ERROR with OMP parameters !"
	1445	IGCM_debug_Print 2 "${comp_proc_omp_loc} is not possible as number of OMP threads"
[1440]	1446	IGCM_debug_Print 2 "Only 2,4,8,12,16,24 as number of OMP threads are possible "
[1078]	1447	IGCM_debug_Verif_Exit
	1448	;;
	1449	esac
[1381]	1450	### only for LMDZ?
[1078]	1451	echo "export KMP_STACKSIZE=3g" >> script_${ExeNameOut}.ksh
	1452	echo "export KMP_LIBRARY=turnaround" >> script_${ExeNameOut}.ksh
	1453	echo "export MKL_SERIAL=YES" >> script_${ExeNameOut}.ksh
	1454	echo "OMP_NUM_THREADS=${comp_proc_omp_loc}" >> script_${ExeNameOut}.ksh
[1381]	1455
[1078]	1456	fi
[1318]	1457
[1381]	1458	# to have out/err per process on different files
	1459	echo "./${ExeNameOut} > out_${ExeNameOut}.out.\${SLURM_PROCID} 2>out_${ExeNameOut}.err.\${SLURM_PROCID}" >> script_${ExeNameOut}.ksh
[1317]	1460	#echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
[1078]	1461
[1381]	1462	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
[1078]	1463
[1381]	1464	# Complete run_file
[1000]	1465
[1381]	1466	echo "${comp_proc_mpi_loc}-${comp_proc_omp_loc} env OMP_NUM_THREADS=${comp_proc_omp_loc} ./script_${ExeNameOut}.ksh " >>run_file
	1467
[1078]	1468	fi
	1469	done
[1381]	1470
[1385]	1471	## variable added to stop after 60s instead of 600s by default.
	1472	## This is used when no error comes from executables and when something stopped an executable without notice.
	1473	export SLURM_WAIT=60
[1381]	1474
[1385]	1475	## module advised by TGCC (instead of 2 variables)
	1476	module load feature/bridge/heterogenous_mpmd
[1381]	1477
[1390]	1478	# set EXECUTION for ccc_mprun case (similar to #1 : MPI only MPMD method)
[1000]	1479
[1390]	1480	EXECUTION="${HOST_MPIRUN_COMMAND} -f ./run_file"
[1381]	1481
[1390]	1482	IGCM_sys_Chmod u+x run_file
	1483	if ( $DEBUG_sys ) ; then
	1484	echo "run_file contains : "
	1485	cat run_file
	1486	fi
[1381]	1487
[1390]	1488	else
[1381]	1489
[1390]	1490	IGCM_debug_Print 1 "sys Curie : choose mpirun or ccc_mprun in config.card for ExecutionModeOnCurie"
	1491
	1492	fi # 1 2 or 3 if ${OK_PARA_MPMD}
	1493
	1494	else
[1078]	1495	# Only one executable (SPMD mode).
[1000]	1496
[622]	1497	for comp in ${config_ListOfComponents[*]} ; do
	1498
[685]	1499	# Only if we really have an executable for the component :
[623]	1500	eval ExeNameOut=\${config_Executable_${comp}[1]}
[685]	1501	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${ExeNameOut}" != "Xinca.dat" ] ) ; then
[622]	1502
[1381]	1503	# Build script files
	1504
[685]	1505	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
	1506	echo "" >> script_${ExeNameOut}.ksh
	1507	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
	1508
	1509	if ( ${OK_PARA_OMP} ) ; then
	1510	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
	1511	echo "" >> script_${ExeNameOut}.ksh
	1512	echo "export KMP_STACKSIZE=3g" >> script_${ExeNameOut}.ksh
	1513	echo "export KMP_LIBRARY=turnaround" >> script_${ExeNameOut}.ksh
	1514	echo "export MKL_SERIAL=YES" >> script_${ExeNameOut}.ksh
	1515	echo "OMP_NUM_THREADS=${comp_proc_omp_loc}" >> script_${ExeNameOut}.ksh
	1516	fi
	1517
	1518	if ( ${OK_PARA_MPI} ) ; then
[623]	1519	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
[685]	1520	# Default : ccc_mprun used if nb_proc gt 1
	1521	# to have out/err per process on different files
[1317]	1522	echo "./${ExeNameOut} > out_${ExeNameOut}.out.\${SLURM_PROCID} 2>out_${ExeNameOut}.err.\${SLURM_PROCID}" >> script_${ExeNameOut}.ksh
	1523	#echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
[685]	1524	EXECUTION="${HOST_MPIRUN_COMMAND} -n ${comp_proc_mpi_loc} ./script_${ExeNameOut}.ksh"
[623]	1525	else
[685]	1526	# Default : ccc_mprun is NOT used if nb_proc eq 1
	1527	# to have out/err per process on different files
[1317]	1528	echo "./${ExeNameOut} > out_${ExeNameOut}.out 2>out_${ExeNameOut}.err" >> script_${ExeNameOut}.ksh
	1529	#echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
[685]	1530	EXECUTION="/usr/bin/time ./script_${ExeNameOut}.ksh"
[622]	1531	fi
[685]	1532
[1462]	1533	IGCM_debug_Print 1 "sys Irene : script_${ExeNameOut}.ksh contains"
[685]	1534	cat script_${ExeNameOut}.ksh
	1535
[623]	1536	fi
[622]	1537
	1538	done
	1539
[1390]	1540	fi # ${OK_PARA_MPMD}
[622]	1541
[1462]	1542	IGCM_debug_Print 1 "sys Irene : execution command is "
[623]	1543	IGCM_debug_Print 1 "$EXECUTION"
[622]	1544
[623]	1545	IGCM_debug_PopStack "IGCM_sys_build_execution_scripts"
[622]	1546	}
	1547
[933]	1548	#D-#==================================================
	1549	#D-function IGCM_sys_check_path
	1550	#D-* Purpose: check that RUN_DIR_PATH that will be removed on some machine
	1551	#D-* do not point to an important use directory. Stop immediately in that case.
	1552	#D-* Examples:
	1553	#D-
	1554	function IGCM_sys_check_path {
	1555	IGCM_debug_PushStack "IGCM_sys_check_path"
	1556	if ( $DEBUG_sys ) ; then
	1557	echo "IGCM_sys_check_path"
	1558	fi
[811]	1559
[1490]	1560	if ( [ X${RUN_DIR_PATH} = X${HOME} ] \|\| [ X${RUN_DIR_PATH} = X${WORK} ] \|\| [ X${RUN_DIR_PATH} = X${SCRATCH} ] \|\| [ X${RUN_DIR_PATH} = X${STORE} ] ) ; then
[933]	1561	IGCM_debug_Print 1 "Variable RUN_DIR_PATH is pointing to an important directory : ${RUN_DIR_PATH}"
	1562	IGCM_debug_Print 1 "Please check the RUN_DIR_PATH definition in your Job : Job_${config_UserChoices_JobName}"
[936]	1563	IGCM_debug_Exit "This will stop the job"
[933]	1564	fi
	1565	IGCM_debug_PopStack "IGCM_sys_check_path"
	1566	}
	1567
[811]	1568	#D-#==================================================
	1569	#D-function IGCM_sys_check_quota
	1570	#D-* Purpose: check user quota. Stop the simulation if quota above 90%
	1571	#D-* Examples:
	1572	#D-
[714]	1573	function IGCM_sys_check_quota {
[776]	1574	IGCM_debug_PushStack "IGCM_sys_check_quota"
	1575	if ( $DEBUG_sys ) ; then
	1576	echo "IGCM_sys_check_quota"
	1577	fi
	1578	# Limit of quota (in %)
	1579	limit_quota=90
	1580
	1581	# Check of the volume
[912]	1582	volume_quota=$(ccc_quota \| grep ' scratch' \| gawk '{print $2}')
	1583	volume_avail=$(ccc_quota \| grep ' scratch' \| gawk '{print $3}')
[776]	1584
	1585	if ( [ ! X${volume_quota} = X ] && [ ! ${volume_quota} = "-" ] ) ; then
	1586
	1587	unit_avail=${volume_avail: -1}
	1588	unit_quota=${volume_quota: -1}
	1589
	1590	if [ "${unit_quota}" = "*" ] ; then
[811]	1591	IGCM_debug_Print 1 "Please, check your quota of volume on scratch"
	1592	IGCM_debug_Print 1 "More than 100% of your quota is used"
	1593	IGCM_debug_Print 1 "Use the ccc_quota command to check"
	1594	IGCM_debug_Print 1 "You must have more than 10% available to run"
	1595	IGCM_debug_Exit "Not enough space to run ! STOP HERE"
	1596	IGCM_debug_Verif_Exit
[714]	1597	fi
[776]	1598
	1599	temp_avail=${volume_avail%%${volume_avail: -1}*}
	1600	temp_quota=${volume_quota%%${volume_quota: -1}*}
	1601
	1602	if [ ! ${unit_avail} = ${unit_quota} ] ; then
	1603
	1604	# Convertion
	1605	if [ ${volume_avail: -1} = "T" ] ; then
	1606	(( temp_avail = temp_avail * 1000000000000 ))
	1607	elif [ ${volume_avail: -1} = "G" ] ; then
	1608	(( temp_avail = temp_avail * 1000000000 ))
	1609	elif [ ${volume_avail: -1} = "M" ] ; then
	1610	(( temp_avail = temp_avail * 1000000 ))
	1611	elif [ ${volume_avail: -1} = "k" ] ; then
	1612	(( temp_avail = temp_avail * 1000 ))
	1613	else
	1614	(( temp_avail = volume_avail ))
	1615	fi
	1616	if [ ${volume_quota: -1} = "T" ] ; then
	1617	(( temp_quota = temp_quota * 1000000000000 ))
	1618	elif [ ${volume_quota: -1} = "G" ] ; then
	1619	(( temp_quota = temp_quota * 1000000000 ))
	1620	elif [ ${volume_quota: -1} = "M" ] ; then
	1621	(( temp_quota = temp_quota * 1000000 ))
	1622	elif [ ${volume_quota: -1} = "k" ] ; then
	1623	(( temp_quota = temp_quota * 1000 ))
	1624	else
	1625	(( temp_quota = volume_quota ))
	1626	fi
	1627	fi
	1628
	1629	quota_volume=$(echo "scale=2 ; $temp_quota/$temp_avail*100" \| bc)
	1630	# echo "volume ratio is " $quota_volume
	1631
	1632	if [ ${quota_volume} -ge ${limit_quota} ] ; then
	1633	IGCM_debug_Print 1 "Please, check your quota of volume on scratch"
	1634	IGCM_debug_Print 1 "${quota_volume}% of your quota is used"
	1635	IGCM_debug_Print 1 "Use the ccc_quota command to check"
	1636	IGCM_debug_Print 1 "You must have more than 10% available to run"
	1637	IGCM_debug_Exit "Not enough space to run ! STOP HERE"
	1638	IGCM_debug_Verif_Exit
	1639	fi
	1640
	1641	fi
	1642
	1643	# Check of the number of inodes
	1644
[912]	1645	inode_quota=$(ccc_quota \| grep ' scratch' \| gawk '{print $6}')
	1646	inode_avail=$(ccc_quota \| grep ' scratch' \| gawk '{print $7}')
[776]	1647
	1648	if ( [ ! X${inode_quota} = X ] && [ ! ${inode_quota} = "-" ] ) ; then
	1649
	1650	unit_avail=${inode_avail: -1}
	1651	unit_quota=${inode_quota: -1}
	1652
	1653	if [ "${unit_quota}" = "*" ] ; then
[811]	1654	IGCM_debug_Print 1 "Please, check your quota of inode on scratch"
	1655	IGCM_debug_Print 1 "More than 100% of your quota is used"
	1656	IGCM_debug_Print 1 "Use the ccc_quota command to check"
	1657	IGCM_debug_Print 1 "You must have more than 10% available to run"
	1658	IGCM_debug_Exit "Not enough space to run ! STOP HERE"
	1659	IGCM_debug_Verif_Exit
[776]	1660	fi
	1661
	1662	temp_avail=${inode_avail%%${inode_avail: -1}*}
	1663	temp_quota=${inode_quota%%${inode_quota: -1}*}
	1664
	1665	if [ ! ${unit_avail} = ${unit_quota} ] ; then
	1666
	1667	# Convertion
	1668	if [ ${inode_avail: -1} = "T" ] ; then
	1669	(( temp_avail = temp_avail * 1000000000000 ))
	1670	elif [ ${inode_avail: -1} = "G" ] ; then
	1671	(( temp_avail = temp_avail * 1000000000 ))
	1672	elif [ ${inode_avail: -1} = "M" ] ; then
	1673	(( temp_avail = temp_avail * 1000000 ))
	1674	elif [ ${inode_avail: -1} = "k" ] ; then
	1675	(( temp_avail = temp_avail * 1000 ))
	1676	else
	1677	(( temp_avail = inode_avail ))
	1678	fi
	1679
	1680	if [ ${inode_quota: -1} = "T" ] ; then
	1681	(( temp_quota = temp_quota * 1000000000000 ))
	1682	elif [ ${inode_quota: -1} = "G" ] ; then
	1683	(( temp_quota = temp_quota * 1000000000 ))
	1684	elif [ ${inode_quota: -1} = "M" ] ; then
	1685	(( temp_quota = temp_quota * 1000000 ))
	1686	elif [ ${inode_quota: -1} = "k" ] ; then
	1687	(( temp_quota = temp_quota * 1000 ))
	1688	else
	1689	(( temp_quota = inode_quota ))
	1690	fi
	1691	fi
	1692	quota_inode=$(echo "scale=2 ; $temp_quota/$temp_avail*100" \| bc)
	1693	# echo "inode ratio is " $quota_inode
	1694
	1695	if [ ${quota_inode} -ge ${limit_quota} ] ; then
	1696	IGCM_debug_Print 1 "Please, check your quota of inode on scratch"
	1697	IGCM_debug_Print 1 "${quota_inode}% of your quota is used"
	1698	IGCM_debug_Print 1 "Use the ccc_quota command to check"
	1699	IGCM_debug_Print 1 "You must have more than 10% available to run"
	1700	IGCM_debug_Exit "Not enough space to run ! STOP HERE"
	1701	IGCM_debug_Verif_Exit
	1702	fi
	1703	fi
	1704	IGCM_debug_PopStack "IGCM_sys_check_quota"
[714]	1705	}
	1706
[773]	1707	#D-#==================================================
[1286]	1708	#D-function IGCM_sys_projectAccounting
	1709	#D-* Purpose: store project accounting information in a file
	1710	#D-* Examples:
	1711	#D-
	1712	function IGCM_sys_projectAccounting {
	1713	IGCM_debug_PushStack "IGCM_sys_projectAccounting"
	1714	if ( $DEBUG_sys ) ; then
	1715	echo "IGCM_sys_check_quota"
	1716	fi
	1717
[1462]	1718	ssh irene191 /usr/bin/ccc_myproject > $1
[1286]	1719
	1720	IGCM_debug_PopStack "IGCM_sys_projectAccounting"
	1721	}
	1722
	1723	#D-#==================================================
[1244]	1724	#D-function IGCM_sys_getJobSchedulerID
	1725	#D-* Purpose: Get the job ID during execution
	1726	#D-* Examples: IGCM_sys_getJobSchedulerID jobSchedulerID
	1727	#D-
	1728	function IGCM_sys_getJobSchedulerID {
	1729	IGCM_debug_PushStack "IGCM_sys_getJobSchedulerID"
	1730	if ( $DEBUG_sys ) ; then
	1731	echo "IGCM_sys_getJobSchedulerID"
	1732	fi
	1733
	1734	eval ${1}=${BRIDGE_MSUB_JOBID}
	1735
	1736	IGCM_debug_PopStack "IGCM_sys_getJobSchedulerID"
	1737	}
	1738
	1739	#D-#==================================================
[958]	1740	#D-function IGCM_sys_GetJobID
[1244]	1741	#D-* Purpose: Get the job ID from the JobName
[958]	1742	#D-* Examples: IGCM_sys_GetJobID ${JobName} ${TargetUsr} JobID
	1743	#D-
	1744	function IGCM_sys_GetJobID {
	1745	IGCM_debug_PushStack "IGCM_sys_GetJobID"
	1746	if ( $DEBUG_sys ) ; then
	1747	echo "IGCM_sys_GetJobID"
	1748	fi
	1749
	1750	# With -f option, the full job name is given in the last column
	1751	ID=$( ccc_mstat -f -u $2 \| \
	1752	gawk -v JobName=$1 '( $NF ~ JobName ) { print $1 }' )
	1753
	1754	eval ${3}=${ID}
	1755	IGCM_debug_PopStack "IGCM_sys_GetJobID"
	1756	}
	1757
	1758	#D-#==================================================
[773]	1759	#D-function IGCM_sys_CountJobInQueue
[1244]	1760	#D-* Purpose: Count number of users job
[773]	1761	#D-* Examples: IGCM_sys_CountJobInQueue ${JobName} NbRun
	1762	#D-
	1763	function IGCM_sys_CountJobInQueue {
	1764	IGCM_debug_PushStack "IGCM_sys_CountJobInQueue"
	1765	if ( $DEBUG_sys ) ; then
	1766	echo "IGCM_sys_CountJobInQueue"
	1767	fi
	1768
	1769	# With -f option, the full job name is given in the last column
[834]	1770	NbRun=$( ccc_mstat -f \| gawk -v JobName=$1 'BEGIN { x=0 } ( $NF ~ JobName ) { x=x+1 } END { print x }' )
[773]	1771
	1772	eval ${2}=${NbRun}
	1773
	1774	IGCM_debug_PopStack "IGCM_sys_CountJobInQueue"
	1775	}
	1776
[955]	1777	#D-#==================================================
	1778	#D-function IGCM_sys_ListJobInQueue
[1244]	1779	#D-* Purpose: Produce a list of users computing jobs (excluding post-processing)
[955]	1780	#D-* Examples: IGCM_sys_ListJobInQueue ${User} JobNameList
	1781	#D-
	1782	function IGCM_sys_ListJobInQueue {
	1783	IGCM_debug_PushStack "IGCM_sys_ListJobInQueue"
	1784	if ( $DEBUG_sys ) ; then
	1785	echo "IGCM_sys_ListJobInQueue"
	1786	fi
	1787
	1788	# With -f option, the full job name is given in the last column
	1789	set -A JobList $( ccc_mstat -f \| gawk -v User=$1 \
	1790	'( $2 == User && \
	1791	$NF != /TS/ && \
	1792	$NF !~ /PACK/ && \
	1793	$NF !~ /REBUILD/ && \
	1794	$NF !~ /pack/ ) \
	1795	{ print $NF }' \| sed -e "s/\(.\)\.[0-9]/\1/" )
	1796
	1797	eval set -A ${2} ${JobList[*]}
	1798
	1799	IGCM_debug_PopStack "IGCM_sys_ListJobInQueue"
	1800	}
[1272]	1801
	1802	#D-#==================================================
	1803	#D-function IGCM_sys_atlas
	1804	#D-* Purpose: encapsulate atlas call so as to manage error code and curie specificity
	1805	#D-* Examples:
	1806	#D-
	1807	function IGCM_sys_atlas {
	1808	IGCM_debug_PushStack "IGCM_sys_atlas" $@
	1809	if ( $DEBUG_sys ) ; then
	1810	echo "IGCM_sys_atlas :" $@
	1811	fi
	1812
	1813	typeset status
	1814
	1815	\ccc_mprun atlas $@ > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
	1816	status=$?
	1817	if [ ${status} -gt 0 ] ; then
	1818	echo "IGCM_sys_atlas : error code ${status}"
	1819	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
	1820	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
	1821	IGCM_debug_PopStack "IGCM_sys_atlas"
	1822	return 1
	1823	else
	1824	IGCM_debug_PopStack "IGCM_sys_atlas"
	1825	return 0
	1826	fi
	1827
	1828	IGCM_debug_PopStack "IGCM_sys_atlas"
	1829	}
	1830
[1380]	1831	#D-#==================================================
	1832	#D-function IGCM_sys_rebuild_nemo
	1833	#D-* Purpose: rebuild nemo parallel files with nemo specific rebuild on curie
	1834	#D-* Examples:
	1835	#D-
	1836
	1837	function IGCM_sys_rebuild_nemo {
	1838	IGCM_debug_PushStack "IGCM_sys_rebuild_nemo" $@
	1839	if ( $DEBUG_sys ) ; then
	1840	echo "IGCM_sys_rebuild_nemo :" $@
	1841	fi
	1842
[1462]	1843	$( ccc_home -u igcmg)/Tools/irene/bin/rebuild_nemo ${1} ${2}
[1380]	1844
	1845	IGCM_debug_PopStack "IGCM_sys_rebuild_nemo"
	1846	}
	1847

Note: See TracBrowser for help on using the repository browser.

Download in other formats: