Context Navigation

libIGCM_sys_obelix.ksh @ 1601

Last change on this file since 1601 was 1601, checked in by jgipsl, 9 months ago

Update for obelix:

always create run directory in /home/scratch01/LOGIN/RUN_DIR. This is necessary when running on more than 1 node.
Update function IGCM_sys_RmRunDir to be identic with the one at irene, jean-zay and mesoipsl.

Property licence set to
The following licence information concerns ONLY the libIGCM tools
==================================================================

Copyright © Centre National de la Recherche Scientifique CNRS
Commissariat à l'Énergie Atomique CEA

libIGCM : Library for Portable Models Computation of IGCM Group.

IGCM Group is the french IPSL Global Climate Model Group.

This library is a set of shell scripts and functions whose purpose is
the management of the initialization, the launch, the transfer of
output files, the post-processing and the monitoring of datas produce
by any numerical program on any plateforme.

This software is governed by the CeCILL license under French law and
abiding by the rules of distribution of free software. You can use,
modify and/ or redistribute the software under the terms of the CeCILL
license as circulated by CEA, CNRS and INRIA at the following URL
"http://www.cecill.info".

As a counterpart to the access to the source code and rights to copy,
modify and redistribute granted by the license, users are provided only
with a limited warranty and the software's author, the holder of the
economic rights, and the successive licensors have only limited
liability.

In this respect, the user's attention is drawn to the risks associated
with loading, using, modifying and/or developing or reproducing the
software by the user in light of its specific status of free software,
that may mean that it is complicated to manipulate, and that also
therefore means that it is reserved for developers and experienced
professionals having in-depth computer knowledge. Users are therefore
encouraged to load and test the software's suitability as regards their
requirements in conditions enabling the security of their systems and/or
data to be ensured and, more generally, to use and operate it in the
same conditions as regards security.

The fact that you are presently reading this means that you have had
knowledge of the CeCILL license and that you accept its terms.
Property svn:keywords set to Revision Author Date

File size: 41.2 KB

Rev	Line
[657]	1	#!/bin/ksh
[16]	2
	3	#**************************************************************
	4	# Author: Sebastien Denvil, Martial Mancip, Christian Laguerre
[373]	5	# Contact: Martial.Mancip__at__ipsl.jussieu.fr
	6	# $Revision:: $ Revision of last commit
	7	# $Author:: $ Author of last commit
	8	# $Date:: $ Date of last commit
[16]	9	# IPSL (2006)
	10	# This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
[657]	11	#
[16]	12	#**************************************************************
	13
	14	#=========================================================
	15	# The documentation of this file can be automatically generated
[657]	16	# if you use the prefix #D- for comments to be extracted.
[16]	17	# Extract with command: cat lib* \| grep "^#D-" \| cut -c "4-"
	18	#=========================================================
	19
	20	#D-#==================================================
	21	#D-LibIGCM_sys for obelix
	22	#D-#==================================================
	23	#D-
[657]	24	#D- This ksh library if a layer under some usefull
[16]	25	#D-environment variables and shell commands.
	26	#D-All those definitions depend on host particularities.
	27	#D-It manages a stack mechanism and test validity of operations.
	28	#D-All function described bellow must be prefixed by IGCM_sys.
	29
	30	#====================================================
	31	# libIGCM_sys PARAMETERS
	32	#====================================================
	33
[343]	34	#====================================================
[16]	35	# set DEBUG_sys to true to output calls of function
	36	typeset -r DEBUG_sys=${DEBUG_sys:=true}
	37
[343]	38	#====================================================
[16]	39	# Turn in dry run mode ? (sys_Put_Rest, sys_Put_Out, sys_Get)
[1569]	40	if ([ X${DRYRUN} != X ] && [ ${DRYRUN} = 4 ]) ; then
	41	typeset -r DRYRUN=0
	42	typeset -r DRYRUN_DEBUG=4
	43	else
	44	typeset -r DRYRUN=${DRYRUN:=0}
	45	typeset -r DRYRUN_DEBUG=0
	46	fi
[16]	47
	48	# YOU MUST COMPILE YOUR EXE FILES FOR DRYRUN MODE !
	49	# -------------------------------------------------------------------------------------
	50	# \| DRYRUN= \| Date computations, \| sys_Get \| Exe \| sys_Put_Out; sys_Put_Rest \|
[809]	51	# \| \| Cp/Exe/param/files \| \| \| \|
	52	# \| \| Chmod Qsub \| \| \| \|
[16]	53	# -------------------------------------------------------------------------------------
	54	# \| 0 \| yes \| yes \| yes \| yes \|
	55	# -------------------------------------------------------------------------------------
	56	# \| 1 \| yes \| yes \| yes \| no \|
	57	# -------------------------------------------------------------------------------------
	58	# \| 2 \| yes \| yes \| no \| no \|
	59	# -------------------------------------------------------------------------------------
	60	# \| 3 \| yes \| no \| no \| no \|
	61	# -------------------------------------------------------------------------------------
[1569]	62	# \| 4 \| yes \| yes \| creation of RUN_DIR and Job_debug \|
	63	# -------------------------------------------------------------------------------------
[16]	64
[1569]	65
[16]	66	#=====================================================
	67	# Global Variables :
	68	#=====================================================
	69	# Language : "fr" or "en"
	70	typeset -r MYLANG="fr"
	71
	72	#=====================================================
[1300]	73	# Host user names project maxCpuTime
[217]	74	# $hostname ou hostname
[1180]	75	typeset HOST=${HOST:=$( hostname )}
[16]	76	# $username ou whoami
[1180]	77	typeset LOGIN=${LOGIN:=$( whoami )}
[657]	78	# $hostname of the MASTER job
[742]	79	typeset MASTER=${MASTER:=$( hostname )}
[1201]	80	# project name
	81	typeset PROJECT=NONE
	82	# jobWarningDelay in seconds
	83	typeset jobWarningDelay=${PBS_WALLTIME}
[16]	84
	85	#D-
	86	#D-#==================================================
	87	#D-Program used in libIGCM
	88	#D-#==================================================
	89
[742]	90	# Submit command
	91	typeset SUBMIT=${SUBMIT:=qsub}
[16]	92	# rsync with path
	93	typeset -r RSYNC=/usr/bin/rsync
	94	# RSYNC_opt args to rsync
[282]	95	typeset -r RSYNC_opt="-va"
[657]	96	# ie storage filesystem
	97	typeset -r STOREHOST=${MASTER}
[16]	98
	99	#====================================================
[1485]	100	# Access to module command
	101	#====================================================
	102	. /usr/share/Modules/init/ksh
	103
	104	#====================================================
[1180]	105	# Set environment tools (ferret, nco, cdo, rebuild, ...)
[305]	106	#====================================================
[1485]	107	if [ X${TaskType} = Xcomputing ] ; then
	108	IGCM_debug_Print 1 "Modules will be loaded later in IGCM_sys_activ_variables."
	109	else
	110	. /home/orchideeshare/igcmg/MachineEnvironment/obelix/env_atlas_obelix
	111	fi
	112
	113	[ ! X${TaskType} = Xchecking ] && IGCM_debug_Print 1 "List of loaded modules:"
	114	[ ! X${TaskType} = Xchecking ] && module list
	115
	116	# Load python
[1358]	117	module load python/2.7.5
[16]	118
[343]	119	#====================================================
[16]	120	# Host specific DIRECTORIES
	121	#====================================================
[1485]	122	# For rebuild
[1594]	123	export PATH=${PATH}:${MODIPSL}/modeles/IOIPSL/bin:/home/orchideeshare/igcmg/IOIPSL_PLUS/modipsl.tagv2_2_4.28082019/bin/
[16]	124
	125	#====================================================
[657]	126	#- MirrorlibIGCM for frontend
	127	typeset -r MirrorlibIGCM=${MirrorlibIGCM:=false}
	128
	129	#====================================================
	130	#- libIGCM_POST for frontend
	131	typeset -r libIGCM_POST=${libIGCM}
	132
	133	#====================================================
[16]	134	#- R_EXE (==> BIN_DIR = ${MODIPSL}/bin )
	135	typeset -r R_EXE="${MODIPSL}/bin"
	136
	137	#====================================================
	138	#- SUBMIT_DIR : submission dir
[256]	139	if [ X${PBS_O_WORKDIR} != X ] ; then
[1124]	140	typeset -x SUBMIT_DIR=${SUBMIT_DIR:=${PBS_O_WORKDIR}}
[256]	141	else
[1124]	142	typeset -x SUBMIT_DIR=${SUBMIT_DIR:=${PWD}}
[256]	143	fi
[657]	144
[16]	145	#====================================================
[725]	146	#- IN
[978]	147	typeset -r R_IN=${R_IN:=/home/orchideeshare/igcmg/IGCM}
[725]	148
	149	#====================================================
[16]	150	#- RUN_DIR_PATH : Temporary working directory (=> TMP)
[1601]	151	if [ X${PBS_JOBID} != X ] ; then
	152	typeset -r RUN_DIR_PATH=${RUN_DIR_PATH:=/home/scratch01/$LOGIN/RUN_DIR/$PBS_O_LOGNAME.$PBS_JOBID}
[838]	153	else
[1601]	154	typeset -r RUN_DIR_PATH=${RUN_DIR_PATH:=/home/scratch01/$LOGIN/RUN_DIR/tmp$$}
[838]	155	fi
[16]	156
[1601]	157
[16]	158	#====================================================
[1179]	159	#- OUTCOMMAND_PATH : tmp place to store command lines standard error and outputs
[1186]	160	typeset -r OUTCOMMAND_PATH=/tmp
[1179]	161
	162	#====================================================
[16]	163	#- HOST_MPIRUN_COMMAND
	164	typeset -r HOST_MPIRUN_COMMAND=${HOST_MPIRUN_COMMAND:="time mpirun"}
	165
[61]	166	#====================================================
	167	#- Max number of arguments passed to nco operator or demigration command
	168	UNIX_MAX_LIMIT=360
	169
[657]	170	#====================================================
[1180]	171	#- set PackDefault to false on obelix
	172	PackDefault=false
	173
	174	#====================================================
[657]	175	#- Default number of MPI task for IPSL coupled model
	176	#- required for backward compatibility
	177	#-
[1300]	178	DEFAULT_NUM_PROC_OCE=5
[657]	179	DEFAULT_NUM_PROC_CPL=1
	180	(( DEFAULT_NUM_PROC_ATM = BATCH_NUM_PROC_TOT - DEFAULT_NUM_PROC_OCE - DEFAULT_NUM_PROC_CPL ))
	181
[16]	182	#D-#==================================================
[1057]	183	#D-function IGCM_sys_defineArchives
	184	#D-* Purpose:
	185	#D-* Define ARCHIVE : Dedicated to large files
	186	#D-* Define STORAGE : Dedicated to small/medium files
	187	#D-* Define R_OUT : Output tree located on ARCHIVE
	188	#D-* Define R_FIG : Output tree located on STORAGE hosting figures (monitoring and atlas, and/or small files)
	189	#D-* Define R_BUF : Output tree located on STORAGE hosting files waiting for rebuild or pack processes
	190	#D-* if SpaceName=TEST nothing special will hapen
[725]	191	#D-* Examples:
	192	#D-
[1057]	193	function IGCM_sys_defineArchives {
	194	IGCM_debug_PushStack "IGCM_sys_defineArchives"
	195
[1060]	196	if [ ! X${config_UserChoices_ARCHIVE} = X ]; then
[1057]	197	#====================================================
	198	#- ARCHIVE (dedicated to large files)
[1063]	199	ARCHIVE=${config_UserChoices_ARCHIVE}
[1057]	200	else
	201	#====================================================
	202	#- ARCHIVE (dedicated to large files)
[1063]	203	ARCHIVE=${ARCHIVE:=/home/scratch01/${LOGIN}}
[1057]	204	fi
	205
[1060]	206	if [ ! X${config_UserChoices_STORAGE} = X ]; then
[1057]	207	#====================================================
	208	#- STORAGE (dedicated to small/medium files)
[1063]	209	STORAGE=${config_UserChoices_STORAGE}
[1057]	210	else
	211	#====================================================
	212	#- STORAGE (dedicated to small/medium files)
[1063]	213	STORAGE=${ARCHIVE}
[1057]	214	fi
	215
	216	# ON OBELIX NO SPECIAL CASE WHEN X${config_UserChoices_SpaceName} = XTEST
	217
	218	#====================================================
	219	#- R_OUT
[1063]	220	R_OUT=${ARCHIVE}/IGCM_OUT
[1057]	221
	222	#====================================================
	223	#- R_FIG (hosting figures : monitoring and atlas, and/or small files)
[1063]	224	R_FIG=${STORAGE}/IGCM_OUT
[1057]	225
	226	#====================================================
[1431]	227	#- CMIP6 (hosting CMIP6 files produced by XIOS2 and configured by dr2xml)
	228	CMIP6_BUF=${STORAGE}/IGCM_OUT
	229
	230	#====================================================
[1057]	231	#- R_BUF (ONLY FOR double copy an scratch)
[1063]	232	R_BUF=${STORAGE}/IGCM_OUT
[1057]	233
	234	IGCM_debug_Print 1 "R_OUT has been defined = ${R_OUT}"
	235	IGCM_debug_Print 1 "R_BUF has been defined = ${R_BUF}"
	236	IGCM_debug_Print 1 "R_FIG has been defined = ${R_FIG}"
	237
	238	IGCM_debug_PopStack "IGCM_sys_defineArchives"
[725]	239	}
	240
	241	#D-#==================================================
[37]	242	#D-function IGCM_sys_RshArchive
	243	#D-* Purpose: Archive rsh command
	244	#D-* Examples:
	245	#D-
	246	function IGCM_sys_RshArchive {
[657]	247	IGCM_debug_PushStack "IGCM_sys_RshArchive" $@
	248	/bin/ksh <<-EOF
[40]	249	${@}
	250	EOF
[811]	251	status=$?
	252	if [ ${status} -gt 0 ] ; then
	253	IGCM_debug_Print 2 "IGCM_sys_RshArchive : command failed error code ${status}"
[657]	254	IGCM_debug_Exit "IGCM_sys_RshArchive"
	255	fi
	256	IGCM_debug_PopStack "IGCM_sys_RshArchive"
[37]	257	}
	258
	259	#D-#==================================================
[1178]	260	#D-function IGCM_sys_RshArchive_NoError
	261	#D-* Purpose: Archive rsh command, without error
	262	#D-* used only in monitoring.job
	263	#D-* Examples:
	264	#D-
	265	function IGCM_sys_RshArchive_NoError {
	266	IGCM_debug_PushStack "IGCM_sys_RshArchive_NoError" $@
	267	/bin/ksh <<-EOF
[1181]	268	${@} 2> /dev/null
[1178]	269	EOF
	270	IGCM_debug_PopStack "IGCM_sys_RshArchive_NoError"
	271	}
	272
	273	#D-#==================================================
[16]	274	#D-function IGCM_sys_MkdirArchive
	275	#D-* Purpose: Mkdir on Archive
	276	#D-* Examples:
	277	#D-
	278	function IGCM_sys_MkdirArchive {
[657]	279	IGCM_debug_PushStack "IGCM_sys_MkdirArchive" $@
	280	if ( $DEBUG_sys ) ; then
	281	echo "IGCM_sys_MkdirArchive :" $@
	282	fi
[717]	283	#- creation de repertoire sur le serveur fichier
[657]	284	if [ ! -d ${1} ]; then
[1178]	285	\mkdir -p $1
[811]	286	status=$?
	287
	288	if [ ${status} -gt 0 ] ; then
	289	IGCM_debug_Print 2 "IGCM_sys_MkdirArchive : mkdir failed error code ${status}"
[657]	290	IGCM_debug_Exit "IGCM_sys_MkdirArchive"
[16]	291	fi
[657]	292	fi
	293	IGCM_debug_PopStack "IGCM_sys_MkdirArchive"
[16]	294	}
	295
	296	#D-#==================================================
	297	#D-function IGCM_sys_TestDirArchive
	298	#D-* Purpose: Test Directory that must exists on Archive
	299	#D-* Examples:
	300	#D-
	301	function IGCM_sys_TestDirArchive {
[657]	302	IGCM_debug_PushStack "IGCM_sys_TestDirArchive" $@
	303	if ( $DEBUG_sys ) ; then
	304	echo "IGCM_sys_TestDirArchive :" $@
	305	fi
	306	typeset ExistFlag
	307	ExistFlag=$( [ -d $1 ] && echo 0 \|\| echo 1 )
	308	IGCM_debug_PopStack "IGCM_sys_TestDirArchive"
	309	return ${ExistFlag}
[16]	310	}
	311
	312	#D-#==================================================
[800]	313	#D-function IGCM_sys_IsFileArchived
[1180]	314	#D-* Purpose: Test file that must NOT EXISTS on Archive based on filename only
[800]	315	#D-* Examples:
	316	#D-
	317	function IGCM_sys_IsFileArchived {
	318	IGCM_debug_PushStack "IGCM_sys_IsFileArchived" $@
	319	if ( $DEBUG_sys ) ; then
	320	echo "IGCM_sys_IsFileArchived :" $@
	321	fi
	322	typeset IsArchivedFlag
	323	# Never archived for this system
	324	IsArchivedFlag=1
	325	IGCM_debug_PopStack "IGCM_sys_IsFileArchived"
	326
	327	return ${IsArchivedFlag}
	328	}
	329
	330	#D-#==================================================
[16]	331	#D-function IGCM_sys_TestFileArchive
	332	#D-* Purpose: Test file that must NOT EXISTS on Archive
	333	#D-* Examples:
	334	#D-
	335	function IGCM_sys_TestFileArchive {
[657]	336	IGCM_debug_PushStack "IGCM_sys_TestFileArchive" $@
	337	typeset ExistFlag
	338	ExistFlag=$( [ -f $1 ] && echo 0 \|\| echo 1 )
	339	IGCM_debug_PopStack "IGCM_sys_TestFileArchive"
[67]	340
[657]	341	return ${ExistFlag}
[67]	342	}
	343
	344	#D-#==================================================
	345	#D-function IGCM_sys_CountFileArchive
	346	#D-* Purpose: Count files on Archive filesystem
	347	#D-* Examples:
	348	#D-
	349	function IGCM_sys_CountFileArchive {
[657]	350	IGCM_debug_PushStack "IGCM_sys_CountFileArchive" $@
	351	ls ${@} 2>/dev/null \| wc -l
	352	if [ $? -gt 0 ] ; then
	353	echo "IGCM_sys_CountFileArchive : erreur."
	354	fi
	355	IGCM_debug_PopStack "IGCM_sys_CountFileArchive"
[16]	356	}
	357
	358	#D-#==================================================
	359	#D-function IGCM_sys_Tree
	360	#D-* Purpose: Tree directories with files on ${ARCHIVE}
	361	#D-* Examples: IGCM_sys_Tree ${R_IN} ${R_OUT}
	362	#D-
	363	function IGCM_sys_Tree {
[657]	364	IGCM_debug_PushStack "IGCM_sys_Tree" $@
	365	if ( $DEBUG_sys ) ; then
	366	echo "IGCM_sys_Tree :" $@
	367	fi
[16]	368
[1300]	369	\ls -lR ${@}
[16]	370
[657]	371	IGCM_debug_PopStack "IGCM_sys_Tree"
[16]	372	}
	373
	374	#D-#==================================================
	375	#D-function IGCM_sys_Qsub
	376	#D-* Purpose: Qsub new job
	377	#D-* Examples:
	378	#D-
	379	function IGCM_sys_Qsub {
[657]	380	IGCM_debug_PushStack "IGCM_sys_Qsub" $@
	381	if ( $DEBUG_sys ) ; then
	382	echo "IGCM_sys_Qsub :" $@
	383	fi
[923]	384	typeset options status
	385	options="-o ${SUBMIT_DIR}/${Script_Output}"
[1179]	386	/usr/local/bin/qsub ${options} $1 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[811]	387	status=$?
[949]	388
[1179]	389	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[811]	390	if [ ${status} -gt 0 ] ; then
[1180]	391	IGCM_debug_Print 2 "IGCM_sys_Qsub ${options} $1 : error code ${status}"
[657]	392	IGCM_debug_Exit "IGCM_sys_Qsub"
[949]	393	else
[1179]	394	JobID=$( gawk {'print $1'} ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ )
[657]	395	fi
	396	IGCM_debug_PopStack "IGCM_sys_Qsub"
[16]	397	}
	398
	399	#D-#==================================================
	400	#D-function IGCM_sys_QsubPost
	401	#D-* Purpose: Qsub new job on scalaire
	402	#D-* Examples:
	403	#D-
	404	function IGCM_sys_QsubPost {
[657]	405	IGCM_debug_PushStack "IGCM_sys_QsubPost" $@
	406	if ( $DEBUG_sys ) ; then
	407	echo "IGCM_sys_QsubPost :" $@
	408	fi
[923]	409	typeset options status
	410	options="-o ${POST_DIR}/${Script_Post_Output}.out -v ${listVarEnv}"
[1179]	411	/usr/local/bin/qsub ${options} ${libIGCM_POST}/$1.job > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[811]	412	status=$?
[949]	413
[1179]	414	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[811]	415	if [ ${status} -gt 0 ] ; then
[923]	416	IGCM_debug_Print 2 "IGCM_sys_QsubPost ${options} ${libIGCM_POST}/$1.job : error code ${status}"
[657]	417	IGCM_debug_Exit "IGCM_sys_QsubPost"
[949]	418	else
[1179]	419	JobID=$( gawk {'print $1'} ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ )
[657]	420	fi
	421	IGCM_debug_PopStack "IGCM_sys_QsubPost"
[16]	422	}
	423
	424	#D-*************************
[657]	425	#D- File transfer functions
[16]	426	#D-*************************
	427	#D-
	428
	429	#D-#==================================================
[203]	430	#D-function IGCM_sys_RmRunDir
	431	#D-* Purpose: rm tmpdir (dummy function most of the time batch
	432	#D- scheduler will do the job)
	433	#D-* Examples:
	434	#D-
	435	function IGCM_sys_RmRunDir {
[1057]	436	IGCM_debug_PushStack "IGCM_sys_RmRunDir" $@
[657]	437	if ( $DEBUG_sys ) ; then
	438	echo "IGCM_sys_RmRunDir :" $@
	439	fi
[1601]	440
	441	typeset status
	442
	443	echo rm $@ > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
	444	\rm $@ >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
	445	status=$?
	446
	447	if [ ${status} -gt 0 ] ; then
	448	IGCM_debug_Print 1 "IGCM_sys_RmRunDir : rm error code is ${status}."
	449	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
	450	IGCM_debug_Exit "IGCM_sys_RmRunDir"
	451	else
	452	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
	453	fi
[657]	454	IGCM_debug_PopStack "IGCM_sys_RmRunDir"
[203]	455	}
	456
	457	#D-#==================================================
[16]	458	#D-function IGCM_sys_Put_Dir
	459	#D-* Purpose: Copy a complete directory on $(ARCHIVE)
	460	#D-* Examples:
	461	#D-
	462	function IGCM_sys_Put_Dir {
[657]	463	IGCM_debug_PushStack "IGCM_sys_Put_Dir" $@
	464	if ( $DEBUG_sys ) ; then
	465	echo "IGCM_sys_Put_Dir :" $@
	466	fi
	467	if [ $DRYRUN = 0 ]; then
	468	if [ ! -d ${1} ] ; then
	469	echo "WARNING : IGCM_sys_Put_Dir ${1} DOES NOT EXIST ."
	470	IGCM_debug_PopStack "IGCM_sys_Put_Dir"
	471	return
[16]	472	fi
	473
[717]	474	typeset status
[16]	475
[717]	476	# Only if we use rsync
	477	#IGCM_sys_TestDirArchive $( dirname $2 )
	478	#
	479	#USUAL WAY
[1179]	480	\cp -r $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[717]	481	status=$?
[16]	482
[717]	483	if [ ${status} -gt 0 ] ; then
[811]	484	IGCM_debug_Print 2 "IGCM_sys_Put_Dir : cp failed error code ${status}"
[1179]	485	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[657]	486	IGCM_debug_Exit "IGCM_sys_Put_Dir"
[16]	487	else
[1179]	488	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[16]	489	fi
[657]	490	fi
	491	IGCM_debug_PopStack "IGCM_sys_Put_Dir"
[16]	492	}
	493
	494	#D-#==================================================
	495	#D-function IGCM_sys_Get_Dir
[717]	496	#D-* Purpose: Copy a complete directory from ${ARCHIVE}
[16]	497	#D-* Examples:
	498	#D-
	499	function IGCM_sys_Get_Dir {
[657]	500	IGCM_debug_PushStack "IGCM_sys_Get_Dir" $@
	501	if ( $DEBUG_sys ) ; then
	502	echo "IGCM_sys_Get_Dir :" $@
	503	fi
	504	if [ $DRYRUN = 0 ]; then
[1180]	505	typeset NB_ESSAI DELAI status i
	506	# number of tentative
	507	NB_ESSAI=3
	508	# time delay between tentative
	509	DELAI=2
[16]	510
[1180]	511	#
	512	# USUAL WAY
	513	i=0
	514	while [ $i -lt $NB_ESSAI ] ; do
	515	\cp -ur $1 $2 >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
	516	status=$?
	517	if [ ${status} -gt 0 ] ; then
	518	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : cp failed error code ${status} ${i}/${NB_ESSAI}"
	519	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : sleep ${DELAI} seconds and try again."
	520	sleep $DELAI
	521	else
	522	break
	523	fi
	524	(( i = i + 1 ))
	525	done
[16]	526
[717]	527	if [ ${status} -gt 0 ] ; then
[811]	528	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : cp failed error code ${status}"
[1179]	529	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[657]	530	IGCM_debug_Exit "IGCM_sys_Get_Dir"
[16]	531	else
[1179]	532	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[16]	533	fi
[657]	534	fi
	535	IGCM_debug_PopStack "IGCM_sys_Get_Dir"
[16]	536	}
	537
	538	#D-#==================================================
	539	#D-function IGCM_sys_Put_Rest
[717]	540	#D-* Purpose: Put computied restarts on ${ARCHIVE}.
[282]	541	#D- File and target directory must exist.
[16]	542	#D-* Examples:
	543	#D-
	544	function IGCM_sys_Put_Rest {
[657]	545	IGCM_debug_PushStack "IGCM_sys_Put_Rest" $@
	546	if ( $DEBUG_sys ) ; then
	547	echo "IGCM_sys_Put_Rest :" $@
	548	fi
	549	if [ $DRYRUN = 0 ]; then
	550	if [ ! -f ${1} ] ; then
	551	echo "ERROR : IGCM_sys_Put_Rest ${1} DOES NOT EXIST ."
	552	IGCM_debug_Exit "IGCM_sys_Put_Rest"
[16]	553	fi
[717]	554
	555	typeset status
	556	#
[1178]	557	# USUAL WAY
[1179]	558	\cp $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[717]	559	status=$?
[16]	560
[717]	561	# #RSYNC WITH NETWORK SSH CALL
[1179]	562	# echo ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
	563	# ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[16]	564
[717]	565	# #RSYNC WITH NFS USE
[1179]	566	# echo ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
	567	# ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[16]	568
[717]	569	# status=$?
	570	# IGCM_sys_Rsync_out $status
	571
[1179]	572	# ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[717]	573	# (( status=status+$? ))
	574
	575	if [ ${status} -gt 0 ] ; then
[1180]	576	IGCM_debug_Print 2 "IGCM_sys_Put_Rest : cp failed error code ${status}"
[1178]	577	[ -f ${1} ] && ls -l ${1}
	578	[ -f ${2} ] && ls -l ${2}
	579	[ -f ${2}/${1} ] && ls -l ${2}/${1}
[1179]	580	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[657]	581	IGCM_debug_Exit "IGCM_sys_Put_Rest"
	582	else
[1178]	583
	584	if [ X${JobType} = XRUN ] ; then
	585	[ -f ${2} ] && IGCM_sys_Chmod 444 ${2}
	586	[ -f ${2}/${1} ] && IGCM_sys_Chmod 444 ${2}/${1}
	587	fi
	588
[1179]	589	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[657]	590	fi
	591	fi
	592	IGCM_debug_PopStack "IGCM_sys_Put_Rest"
	593	}
[16]	594
[657]	595	#D-#==================================================
[16]	596	#D-function IGCM_sys_Put_Out
[717]	597	#D-* Purpose: Copy a file on ${ARCHIVE} after having chmod it in readonly
[16]	598	#D-* Examples:
	599	#D-
	600	function IGCM_sys_Put_Out {
[657]	601	IGCM_debug_PushStack "IGCM_sys_Put_Out" $@
	602	if ( $DEBUG_sys ) ; then
	603	echo "IGCM_sys_Put_Out :" $@
	604	fi
[811]	605
	606	typeset status
	607
[657]	608	if [ $DRYRUN = 0 ]; then
[811]	609	if [ ! -f ${1} ] ; then
[657]	610	echo "WARNING : IGCM_sys_Put_Out ${1} DOES NOT EXIST ."
	611	IGCM_debug_PopStack "IGCM_sys_Put_Out"
	612	return 1
[16]	613	fi
[719]	614	#
[811]	615	IGCM_sys_MkdirArchive $( dirname $2 )
	616	#
[657]	617	if [ X${JobType} = XRUN ] ; then
	618	if [ X${3} = X ] ; then
	619	IGCM_sys_Chmod 444 ${1}
	620	fi
	621	fi
[717]	622	#
[16]	623
[1179]	624	echo ${RSYNC} ${RSYNC_opt} $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
	625	${RSYNC} ${RSYNC_opt} $1 $2 >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[717]	626	status=$?
	627	IGCM_sys_Rsync_out $status
[16]	628
[1179]	629	${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[717]	630	(( status=status+$? ))
[16]	631
[717]	632	if [ ${status} -gt 0 ] ; then
[811]	633	IGCM_debug_Print 2 "IGCM_sys_Put_Out : rsync failed error code ${status}"
[1179]	634	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[657]	635	IGCM_debug_Exit "IGCM_sys_Put_Out"
[16]	636	else
[1179]	637	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[16]	638	fi
[657]	639	fi
	640	IGCM_debug_PopStack "IGCM_sys_Put_Out"
	641	return 0
[16]	642	}
	643
	644	#D-#==================================================
	645	#D-function IGCM_sys_Get
	646	#D-* Purpose: Get a file from ${ARCHIVE}
	647	#D-* Examples: IGCM_sys_Get myfile /destpath/myfile_with_PREFIX
	648	#D- IGCM_sys_Get /l Array_contain_myfiles /destpath/
	649	function IGCM_sys_Get {
[657]	650	IGCM_debug_PushStack "IGCM_sys_Get" $@
[16]	651
[811]	652	typeset DEST status dm_liste
[16]	653
[657]	654	if ( $DEBUG_sys ) ; then
	655	echo "IGCM_sys_Get :" $@
	656	fi
	657	if [ $DRYRUN -le 2 ]; then
	658	if [ X${1} = X'/l' ] ; then
	659	eval set +A dm_liste \${${2}}
	660	else
	661	dm_liste=${1}
[16]	662	fi
[657]	663	eval DEST=\${${#}}
[16]	664
[717]	665	# test if the (first) file is present in the old computation :
[657]	666	IGCM_sys_TestFileArchive ${dm_liste[0]}
[717]	667	status=$?
	668	if [ ${status} -gt 0 ] ; then
[657]	669	echo "IGCM_sys_Get, ERROR : regular file ${dm_liste[0]} DOES NOT EXIST ."
	670	IGCM_debug_Exit "IGCM_sys_Get"
[811]	671	return
[657]	672	fi
[16]	673
[811]	674	#USUAL WAY
[1179]	675	\cp ${dm_liste[*]} ${DEST} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[811]	676	status=$?
[16]	677
[811]	678	if [ ${status} -gt 0 ] ; then
	679	IGCM_debug_Print 2 "IGCM_sys_Get : cp failed error code ${status}"
[1179]	680	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[657]	681	IGCM_debug_Exit "IGCM_sys_Get"
[16]	682	else
[1179]	683	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[16]	684	fi
[657]	685	fi
	686	IGCM_debug_PopStack "IGCM_sys_Get"
[16]	687	}
	688
	689	#D-#==================================================
[1131]	690	#D-function IGCM_sys_GetDate_Monitoring
	691	#D-* Purpose: get the last year for which the monitoring has been computed
	692	#D-* Examples:
	693	#D-
	694	function IGCM_sys_GetDate_Monitoring {
	695	IGCM_debug_PushStack "IGCM_sys_GetDate_Monitoring" $@
	696	if ( $DEBUG_sys ) ; then
	697	echo "IGCM_sys_GetDate_Monitoring :" $@
	698	fi
	699
	700	eval ${2}=$( cdo showyear ${1} 2> /dev/null \| gawk '{ print $NF }' )
	701
	702	IGCM_debug_PopStack "IGCM_sys_GetDate_Monitoring"
	703	}
	704
	705	#D-#==================================================
[343]	706	#D-function IGCM_sys_Dods_Rm
[717]	707	#D-* Purpose: DO NOTHING ! Put ${ARCHIVE} files on DODS internet protocole.
[343]	708	#D-* Examples:
	709	#D-
	710	function IGCM_sys_Dods_Rm {
[657]	711	if ( $DEBUG_sys ) ; then
	712	echo "IGCM_sys_Dods_Rm :" $@
	713	fi
	714	return 0
[343]	715	}
	716
	717	#D-#==================================================
	718	#D-function IGCM_sys_Dods_Cp
[717]	719	#D-* Purpose: Copy $(ARCHIVE) files on DODS internet protocole.
[343]	720	#D-* Examples:
	721	#D-
	722	function IGCM_sys_Dods_Cp {
[657]	723	if ( $DEBUG_sys ) ; then
	724	echo "IGCM_sys_Dods_Cp :" $@
	725	fi
	726	return 0
[343]	727	}
	728
	729	#D-#==================================================
[16]	730	#D-function IGCM_sys_Put_Dods
[811]	731	#D-* Purpose: Put ${ARCHIVE} files on DODS internet protocole. Dummy function here
[16]	732	#D-* Examples:
	733	#D-
	734	function IGCM_sys_Put_Dods {
[657]	735	IGCM_debug_PushStack "IGCM_sys_Put_Dods" $@
	736	if ( $DEBUG_sys ) ; then
	737	echo "IGCM_sys_Put_Dods :" $@
	738	fi
	739	IGCM_debug_PopStack "IGCM_sys_Put_Dods"
[16]	740	}
	741
[657]	742	##############################################################
	743	# REBUILD OPERATOR
[536]	744
[811]	745	#D-#==================================================
[1163]	746	#D-function IGCM_sys_sync
[1300]	747	#D-* Purpose: flush buffer on disk (dummy function on Obelix)
[1163]	748	#D-* Examples:
	749	#D-
	750	function IGCM_sys_sync {
	751	IGCM_debug_PushStack "IGCM_sys_sync" $@
	752	if ( $DEBUG_sys ) ; then
	753	echo "IGCM_sys_sync :" $@
	754	echo "Dummy call, let the system do that."
	755	fi
	756	IGCM_debug_PopStack "IGCM_sys_sync"
	757	}
	758
[657]	759	############################################################
	760	# Activate Running Environnment Variables
[536]	761
[811]	762	#D-#==================================================
[1239]	763	#D-function IGCM_sys_activ_variables
[811]	764	#D-* Purpose: set environement variables prior to execution
	765	#D-* Examples:
	766	#D-
[657]	767	function IGCM_sys_activ_variables {
	768	IGCM_debug_PushStack "IGCM_sys_activ_variables"
	769	if ( $DEBUG_sys ) ; then
	770	echo "IGCM_sys_activ_variables"
	771	fi
[811]	772
	773	# --------------------------------------------------------------------
[1485]	774	#D- MODULE specifications
	775	# --------------------------------------------------------------------
	776	# Source the file EnvFile containing all module load needed to run the model.
	777	# EnvFile can be specified in config.card. If this is the case and if the file exists,
	778	# it will be used. Otherwise a default file will be used.
	779	if [ ! X${config_UserChoices_EnvFile} = X ] && [ -f ${config_UserChoices_EnvFile} ] ; then
	780	# EnvFile is set in config.card and the file exists
	781	IGCM_debug_Print 1 "EnvFile set in config.card will be used"
	782	EnvFile=${config_UserChoices_EnvFile}
	783	else
	784	IGCM_debug_Print 1 "IGCM_sys_active_variables : Default modules will be used"
	785	EnvFile=/home/orchideeshare/igcmg/MachineEnvironment/obelix/env_atlas_obelix
	786	fi
	787	IGCM_debug_Print 1 "IGCM_sys_active_variables : Following EnvFile will be used :"
	788	IGCM_debug_Print 1 ${EnvFile}
	789	. ${EnvFile}
	790
	791	IGCM_debug_Print 1 "IGCM_sys_active_variables : Now loaded modules for obelix. "
	792	module list
	793
	794	# --------------------------------------------------------------------
[811]	795	#D- MPI specifications
	796	# --------------------------------------------------------------------
	797
	798	# --------------------------------------------------------------------
	799	#D- Other specifications
	800	# --------------------------------------------------------------------
	801
[1300]	802	ulimit -s unlimited
	803
[657]	804	IGCM_debug_PopStack "IGCM_sys_activ_variables"
[16]	805	}
	806
[657]	807	############################################################
	808	# Desactivate Running Environnment Variables
[16]	809
[811]	810	#D-#==================================================
	811	#D-function IGCM_sys_desactiv_variables
	812	#D-* Purpose: unset environement variables after execution
	813	#D-* Examples:
	814	#D-
[657]	815	function IGCM_sys_desactiv_variables {
	816	IGCM_debug_PushStack "IGCM_sys_desactiv_variables"
	817	if ( $DEBUG_sys ) ; then
	818	echo "IGCM_sys_desactiv_variables"
	819	fi
[811]	820	# --------------------------------------------------------------------
	821	#D- MPI specifications
	822	# --------------------------------------------------------------------
	823
	824	# --------------------------------------------------------------------
	825	#D- Other specifications
	826	# --------------------------------------------------------------------
	827
[657]	828	IGCM_debug_PopStack "IGCM_sys_desactiv_variables"
	829	}
	830
	831	############################################################
[1239]	832	# Update job headers to be used by the scheduler
	833
	834	#D-#==================================================
	835	#D-function IGCM_sys_updateHeaders
	836	#D-* Purpose: Update job headers to be used by the scheduler
	837	#D-* Examples: IGCM_sys_updateHeaders /path/to/Job_MYEXP
	838	#D-
	839	function IGCM_sys_updateHeaders {
	840	IGCM_debug_PushStack "IGCM_sys_updateHeaders"
	841	if ( $DEBUG_sys ) ; then
	842	echo "IGCM_sys_updateHeaders"
	843	fi
	844	typeset file
	845	file=$1
	846
	847	if [ ${executionType} -eq 1 ] ; then
	848	# MPMD + MPI
	849	sed -e "/::openMPthreads::/d" \
	850	-e "s/::JobNumProcTot::/${coreNumber}/" \
	851	${file} > ${file}.tmp
	852
	853	elif [ ${executionType} -eq 2 ] ; then
	854	# MPMD + MPI + OMP
	855	sed -e "s/::openMPthreads::/${openMPthreads}/" \
	856	-e "s/::JobNumProcTot::/${coreNumber}/" \
	857	${file} > ${file}.tmp
	858
	859	elif [ ${executionType} -eq 3 ] ; then
	860	# SPMD + MPI/OMP
	861	sed -e "s/::openMPthreads::/${openMPthreads}/" \
	862	-e "s/::JobNumProcTot::/${mpiTasks}/" \
	863	${file} > ${file}.tmp
	864
	865	elif [ ${executionType} -eq 4 ] ; then
	866	# SPMD + MPI only
	867	sed -e "s/::JobNumProcTot::/${mpiTasks}/" \
	868	-e "/::openMPthreads::/d" \
	869	${file} > ${file}.tmp
	870
	871	elif [ ${executionType} -eq 5 ] ; then
	872	# SPMD + OMP only
	873	sed -e "s/::openMPthreads::/${openMPthreads}/" \
	874	-e "/::JobNumProcTot::/d" \
	875	${file} > ${file}.tmp
	876
	877	elif [ ${executionType} -eq 6 ] ; then
	878	# SEQUENTIAL THEN
	879	sed -e "s/::JobNumProcTot::/1/" \
	880	-e "/::openMPthreads::/d" \
	881	${file} > ${file}.tmp
	882
	883	fi
	884
	885	IGCM_sys_Mv ${file}.tmp ${file}
	886
	887	IGCM_debug_PopStack "IGCM_sys_updateHeaders"
	888	}
	889
	890	############################################################
[800]	891	# Build MPI/OMP scripts run file (dummy function)
[657]	892
[811]	893	#D-#==================================================
	894	#D-function IGCM_sys_build_run_file
	895	#D-* Purpose: build run file (deprecated)
	896	#D-* Examples:
	897	#D-
[657]	898	function IGCM_sys_build_run_file {
[664]	899	IGCM_debug_Print 3 " dummy function : IGCM_sys_build_run_file "
[657]	900	}
	901
	902	############################################################
	903	# Build MPI/OMP scripts
[811]	904
	905	#D-#==================================================
	906	#D-function IGCM_sys_build_execution_scripts
	907	#D-* Purpose: build execution scripts to be launch by ${HOST_MPIRUN_COMMAND}
	908	#D-* Examples:
	909	#D-
[657]	910	function IGCM_sys_build_execution_scripts
	911	{
	912	IGCM_debug_PushStack "IGCM_sys_build_execution_scripts" $@
	913	if ( $DEBUG_sys ) ; then
	914	echo "IGCM_sys_build_execution_scripts " $@
	915	fi
[1180]	916
[1300]	917	EXECUTION=${HOST_MPIRUN_COMMAND}
[657]	918
[1300]	919	# MPMD mode
[657]	920	if ( ${OK_PARA_MPMD} ) ; then
	921
[1300]	922	# Only MPI (MPMD)
	923	if ( ! ${OK_PARA_OMP} ) ; then
[657]	924
[1300]	925	if [ -f run_file ] ; then
	926	IGCM_sys_Rm -f run_file
	927	fi
	928	touch run_file
[657]	929
[1300]	930	# Build run_file
	931
	932	# First loop on the components for the coupler ie oasis (only if oasis3)
	933	# the coupler ie oasis3 must be the first one
	934	for comp in ${config_ListOfComponents[*]} ; do
	935
	936	eval ExeNameIn=\${config_Executable_${comp}[0]}
	937	eval ExeNameOut=\${config_Executable_${comp}[1]}
	938
	939	# for CPL component only
	940	if [ "X${comp}" = "XCPL" ] && [ "X${ExeNameOut}" != X\"\" ] ; then
	941	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
	942	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
	943	echo "-np ${comp_proc_mpi_loc} ./${ExeNameOut} " >> run_file
	944	fi
	945	done
	946
	947	# Then second loop on the components
	948	for comp in ${config_ListOfComponents[*]} ; do
	949
	950	eval ExeNameIn=\${config_Executable_${comp}[0]}
	951	eval ExeNameOut=\${config_Executable_${comp}[1]}
	952
	953	# Only if we really have an executable for the component and not the coupler ie oasis:
	954	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${comp}" != "XCPL" ] ) ; then
	955
	956	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
	957	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
	958	echo "-np ${comp_proc_mpi_loc} ./${ExeNameOut}" >> run_file
	959	fi
	960	done
	961
	962	EXECUTION="${HOST_MPIRUN_COMMAND} --app ./run_file"
	963
	964	IGCM_sys_Chmod u+x run_file
	965	if ( $DEBUG_sys ) ; then
	966	echo "run_file contains : "
	967	cat run_file
	968	fi
	969
	970	# MPI-OpenMP (MPMD)
	971	else
	972
[657]	973	# Hosts treatment
	974	${HOST_MPIRUN_COMMAND} hostname \| sort \| uniq > hosts.tmp
	975
	976	i=0
	977	rm -f hosts
[1300]	978	IGCM_debug_Print 1 "sys Obelix, Hosts available :"
[657]	979	for nodes in `cat hosts.tmp` ; do
	980	host[$i]=$nodes
	981	echo "${host[$i]} slots=1 max_slots=1" >> hosts
	982	IGCM_debug_Print 1 ${host[$i]}
	983	i=$((i+1))
	984	done
	985	rm -f hosts.tmp
	986
	987	listnodes=${host[*]}
	988
	989	EXECUTION="${HOST_MPIRUN_COMMAND} -hostfile hosts"
	990
	991	# Initialisation
[1300]	992	rank=0
	993	current_core=0
	994	core_per_node=6
[657]	995	init_exec=n
	996
[1300]	997	# Loop on the components
[657]	998	for comp in ${config_ListOfComponents[*]} ; do
	999
[1300]	1000	eval ExeNameIn=\${config_Executable_${comp}[0]}
	1001	eval ExeNameOut=\${config_Executable_${comp}[1]}
[657]	1002
[1300]	1003	# Not possible if oasis has an executable (i.e old version of oasis3)
	1004	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${comp}" = "XCPL" ] ) ; then
	1005	IGCM_debug_Exit "ERROR MPMD with hybrid MPI-OpenMP is not available with oasis3 version"
	1006	IGCM_debug_Print 2 "Only available with oasis3-MCT version coupler"
	1007	IGCM_debug_Verif_Exit
[657]	1008	fi
	1009
	1010	# Only if we really have an executable for the component :
[1300]	1011	if [ "X${ExeNameOut}" != X\"\" ] ; then
[657]	1012
	1013	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
	1014	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
	1015
	1016	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
	1017	echo "" >> script_${ExeNameOut}.ksh
[1300]	1018	if [ ${comp_proc_omp_loc} -gt 1 ] ; then
	1019
	1020	# Check if the number of threads is correct
	1021	case ${comp_proc_omp_loc} in
	1022	2\|4\|6)
	1023	IGCM_debug_Print 1 "You run ${ExeNameOut} on ${comp_proc_omp_loc} OMP threads"
	1024	;;
	1025	*)
	1026	IGCM_debug_Exit "ERROR with OMP parameters !"
	1027	IGCM_debug_Print 2 "${comp_proc_omp_loc} is not possible as number of OMP threads"
	1028	IGCM_debug_Print 2 "Only 2,4,6 as number of OMP threads are possible "
	1029	IGCM_debug_Verif_Exit
	1030	;;
	1031	esac
	1032	#echo "export KMP_STACKSIZE=3g" >> script_${ExeNameOut}.ksh
	1033	#echo "export KMP_LIBRARY=turnaround" >> script_${ExeNameOut}.ksh
	1034	#echo "export MKL_SERIAL=YES" >> script_${ExeNameOut}.ksh
	1035	echo "export OMP_STACKSIZE=200M" >> script_${ExeNameOut}.ksh
	1036	echo "export OMP_NUM_THREADS=${comp_proc_omp_loc}" >> script_${ExeNameOut}.ksh
	1037	fi
[1318]	1038
[1317]	1039	#echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
[1318]	1040	echo "(( MYMPIRANK = OMPI_COMM_WORLD_RANK )) " >> script_${ExeNameOut}.ksh
	1041	echo "MYMPIRANK=\$(printf '%3.3d\n' \${MYMPIRANK})" >> script_${ExeNameOut}.ksh
	1042	echo "./${ExeNameOut} > out_${ExeNameOut}.out.\${MYMPIRANK} 2>out_${ExeNameOut}.err.\${MYMPIRANK}" >> script_${ExeNameOut}.ksh
[657]	1043	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
	1044
[1300]	1045	if [ ${init_exec} = y ] ; then
	1046	EXECUTION="${EXECUTION} : -np ${comp_proc_mpi_loc} ./script_${ExeNameOut}.ksh"
	1047	else
	1048	EXECUTION="${EXECUTION} -np ${comp_proc_mpi_loc} ./script_${ExeNameOut}.ksh"
	1049	init_exec=y
	1050	fi
[657]	1051
[1300]	1052	# Build rankfile : method used to assign cores and nodes for the MPI process
	1053	# Ex :
	1054	#rank 0=curie5296 slot=0,1,2,3
	1055	#rank 1=curie5296 slot=4,5,6,7
	1056	# Example of final command :
	1057	# mpirun -hostfile hosts -rankfile rankfile -np 27 ./script_lmdz.x.ksh : -np 5 ./script_opa.xx.ksh
	1058	# with script_lmdz.x.ksh :
	1059	# #!/bin/ksh
	1060	#export OMP_STACKSIZE=200M
	1061	#export OMP_NUM_THREADS=4
	1062	#./lmdz.x
[657]	1063
[1300]	1064	for nb_proc_mpi in `seq 0 $(($comp_proc_mpi_loc-1))`; do
	1065	(( index_host = current_core / core_per_node ))
	1066	host_value=${host[${index_host}]}
	1067	(( slot = current_core % core_per_node ))
	1068	virg=","
	1069	string_final=""
	1070	for index in `seq $slot $(($slot+$comp_proc_omp_loc-1))`; do
	1071	string=$index$virg
	1072	string_final=$string_final$string
	1073	done
	1074	string_final=$( echo $string_final \| sed "s/.$//" )
	1075	echo "rank $rank=$host_value slot=$string_final" >> rankfile
	1076	(( rank = rank + 1 ))
	1077	(( current_core = current_core + comp_proc_omp_loc ))
[657]	1078	done
	1079	fi
	1080	done
[16]	1081	fi
	1082
[1300]	1083	# Only one executable (SPMD mode).
	1084	else
[657]	1085
[664]	1086	for comp in ${config_ListOfComponents[*]} ; do
[657]	1087
[664]	1088	# Only if we really have an executable for the component :
	1089	eval ExeNameOut=\${config_Executable_${comp}[1]}
	1090	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${ExeNameOut}" != "Xinca.dat" ] ) ; then
[657]	1091
[664]	1092	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
	1093	echo "" >> script_${ExeNameOut}.ksh
	1094	if ( ${OK_PARA_OMP} ) ; then
	1095	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
[1300]	1096	echo "export OMP_STACKSIZE=200M" >> script_${ExeNameOut}.ksh
	1097	echo "export OMP_NUM_THREADS=${comp_proc_omp_loc}" >> script_${ExeNameOut}.ksh
[664]	1098	fi
[1480]	1099
	1100	if [ ${mpiTasks} -eq 1 ] && [ ${openMPthreads} -eq 1 ] ; then
	1101	# Case with only one proc
	1102	# mpirun should not be used at obelix for only one proc
	1103	# Output text per process
	1104	echo "./${ExeNameOut} > out_${ExeNameOut}.out 2>out_${ExeNameOut}.err" >> script_${ExeNameOut}.ksh
	1105	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
	1106	EXECUTION="time ./script_${ExeNameOut}.ksh"
	1107	elif ( ${OK_PARA_MPI} ) ; then
[664]	1108	# Default : mpirun used if nb_proc gt 1
[1480]	1109	# Output text per process
[1317]	1110	echo "./${ExeNameOut} > out_${ExeNameOut}.out.\${OMPI_COMM_WORLD_RANK} 2>out_${ExeNameOut}.err.\${OMPI_COMM_WORLD_RANK}" >> script_${ExeNameOut}.ksh
[664]	1111	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
	1112	EXECUTION="${HOST_MPIRUN_COMMAND} ./script_${ExeNameOut}.ksh"
[1481]	1113	elif [ ${executionType} -eq 6 ] ; then
	1114	# Sequentiel case: this is the same as the first case.
	1115	echo "./${ExeNameOut} > out_${ExeNameOut}.out 2>out_${ExeNameOut}.err" >> script_${ExeNameOut}.ksh
	1116	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
	1117	EXECUTION="time ./script_${ExeNameOut}.ksh"
[664]	1118	else
[1480]	1119	# Strange case.
	1120	IGCM_debug_Exit "ERROR with OK_PARA_MPI"
	1121	IGCM_debug_Print 2 "sys Obelix : OK_PARA_MPI=False and mpiTask/openMPthreds different from 1"
	1122	IGCM_debug_Verif_Exit
[664]	1123	fi
[1002]	1124
	1125	IGCM_debug_Print 1 "sys Obelix : script_${ExeNameOut}.ksh contains"
	1126	cat script_${ExeNameOut}.ksh
	1127
[664]	1128	fi
[1180]	1129
[664]	1130	done
[657]	1131
	1132	fi
	1133
[811]	1134	IGCM_debug_Print 1 "sys Obelix : execution command is"
	1135	IGCM_debug_Print 1 "$EXECUTION"
[657]	1136
	1137	IGCM_debug_PopStack "IGCM_sys_build_execution_scripts"
[16]	1138	}
	1139
[933]	1140	#D-#==================================================
	1141	#D-function IGCM_sys_check_path
	1142	#D-* Purpose: check that RUN_DIR_PATH that will be removed on some machine
	1143	#D-* do not point to an important use directory. Stop immediately in that case.
	1144	#D-* Examples:
	1145	#D-
	1146	function IGCM_sys_check_path {
	1147	IGCM_debug_PushStack "IGCM_sys_check_path"
	1148	if ( $DEBUG_sys ) ; then
	1149	echo "IGCM_sys_check_path"
	1150	fi
[811]	1151
[933]	1152	if ( [ X${RUN_DIR_PATH} = X${HOME} ] \|\| [ X${RUN_DIR_PATH} = X${ARCHIVE} ] ) ; then
	1153	IGCM_debug_Print 1 "Variable RUN_DIR_PATH is pointing to an important directory : ${RUN_DIR_PATH}"
	1154	IGCM_debug_Print 1 "Please check the RUN_DIR_PATH definition in your Job : Job_${config_UserChoices_JobName}"
[936]	1155	IGCM_debug_Exit "This will stop the job"
[933]	1156	fi
	1157	IGCM_debug_PopStack "IGCM_sys_check_path"
	1158	}
	1159
[811]	1160	#D-#==================================================
	1161	#D-function IGCM_sys_check_quota. Dummy call here
	1162	#D-* Purpose: check user quota. Stop the simulation if quota above 90%
	1163	#D-* Examples:
	1164	#D-
[657]	1165	function IGCM_sys_check_quota {
	1166	IGCM_debug_PushStack "IGCM_sys_check_quota"
	1167	if ( $DEBUG_sys ) ; then
	1168	echo "IGCM_sys_check_quota"
	1169	fi
[1300]	1170
	1171	# TO BE IMPLEMENTED
	1172
[657]	1173	IGCM_debug_PopStack "IGCM_sys_check_quota"
	1174	}
	1175
[811]	1176	#D-#==================================================
[1286]	1177	#D-function IGCM_sys_projectAccounting
	1178	#D-* Purpose: store project accounting information in a file
	1179	#D-* Examples:
	1180	#D-
	1181	function IGCM_sys_projectAccounting {
	1182	IGCM_debug_PushStack "IGCM_sys_projectAccounting"
	1183	if ( $DEBUG_sys ) ; then
	1184	echo "IGCM_sys_check_quota"
	1185	fi
	1186
	1187	touch $1
	1188
	1189	IGCM_debug_PopStack "IGCM_sys_projectAccounting"
	1190	}
	1191
	1192	#D-#==================================================
[1244]	1193	#D-function IGCM_sys_getJobSchedulerID
	1194	#D-* Purpose: Get the job ID during execution
	1195	#D-* Examples: IGCM_sys_getJobSchedulerID jobSchedulerID
	1196	#D-
	1197	function IGCM_sys_getJobSchedulerID {
	1198	IGCM_debug_PushStack "IGCM_sys_getJobSchedulerID"
	1199	if ( $DEBUG_sys ) ; then
	1200	echo "IGCM_sys_getJobSchedulerID"
	1201	fi
	1202
	1203	eval ${1}=$( echo ${PBS_JOBID} \| awk -F. '{print $1}' )
	1204
	1205	IGCM_debug_PopStack "IGCM_sys_getJobSchedulerID"
	1206	}
	1207
	1208	#D-#==================================================
[1124]	1209	#D-function IGCM_sys_GetJobID
[1244]	1210	#D-* Purpose: Get the job ID from the JobName
[1124]	1211	#D-* Examples: IGCM_sys_GetJobID ${JobName} ${TargetUsr} JobID
	1212	#D-
	1213	function IGCM_sys_GetJobID {
	1214	IGCM_debug_PushStack "IGCM_sys_GetJobID"
	1215	if ( $DEBUG_sys ) ; then
	1216	echo "IGCM_sys_GetJobID"
	1217	fi
	1218
	1219	# With -f option, the full job name is given in the last column
[1300]	1220	ID="$( qstat -u $2 \| grep -w $1 \| gawk '-F ' '{print $10}' )"
[1124]	1221
	1222	eval ${3}=${ID}
	1223	IGCM_debug_PopStack "IGCM_sys_GetJobID"
	1224	}
	1225
	1226	#D-#==================================================
[811]	1227	#D-function IGCM_sys_CountJobInQueue
[1244]	1228	#D-* Purpose: Count number of users job
[811]	1229	#D-* Examples: IGCM_sys_CountJobInQueue ${JobName} NbRun
	1230	#D-
	1231	function IGCM_sys_CountJobInQueue {
	1232	IGCM_debug_PushStack "IGCM_sys_CountJobInQueue"
	1233	if ( $DEBUG_sys ) ; then
	1234	echo "IGCM_sys_CountJobInQueue"
	1235	fi
	1236	IGCM_debug_PopStack "IGCM_sys_CountJobInQueue"
	1237	}
[1272]	1238
	1239	#D-#==================================================
	1240	#D-function IGCM_sys_atlas
	1241	#D-* Purpose: encapsulate atlas call so as to manage error code and curie specificity
	1242	#D-* Examples:
	1243	#D-
	1244	function IGCM_sys_atlas {
	1245	IGCM_debug_PushStack "IGCM_sys_atlas" $@
	1246	if ( $DEBUG_sys ) ; then
	1247	echo "IGCM_sys_atlas :" $@
	1248	fi
	1249
	1250	typeset status
	1251
	1252	\atlas $@ > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
	1253	status=$?
	1254	if [ ${status} -gt 0 ] ; then
	1255	echo "IGCM_sys_atlas : error code ${status}"
	1256	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
	1257	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
	1258	IGCM_debug_PopStack "IGCM_sys_atlas"
	1259	return 1
	1260	else
	1261	IGCM_debug_PopStack "IGCM_sys_atlas"
	1262	return 0
	1263	fi
	1264
	1265	IGCM_debug_PopStack "IGCM_sys_atlas"
	1266	}
	1267
[1380]	1268
	1269	#D-#==================================================
	1270	#D-function IGCM_sys_rebuild_nemo
	1271	#D-* Purpose: rebuild nemo parallel files with general rebuild
	1272	#D-* Examples:
	1273	#D-
	1274
	1275	function IGCM_sys_rebuild_nemo {
	1276	IGCM_debug_PushStack "IGCM_sys_rebuild_nemo" $@
	1277	if ( $DEBUG_sys ) ; then
	1278	echo "IGCM_sys_rebuild_nemo :" $@
	1279	fi
	1280
	1281	nemo_generic_restart_file_name_out=${1}
	1282	nemo_extension_out=${3}
	1283	shift ; shift ; shift
	1284
	1285	IGCM_sys_rebuild -o ${nemo_generic_restart_file_name_out}.${nemo_extension_out} $@
	1286
	1287	IGCM_debug_PopStack "IGCM_sys_rebuild_nemo"
	1288	}
	1289

Note: See TracBrowser for help on using the repository browser.

Context Navigation

source: trunk/libIGCM/libIGCM_sys/libIGCM_sys_obelix.ksh @ 1601

Download in other formats: