Context Navigation

source: trunk/libIGCM/libIGCM_sys/libIGCM_sys_obelix.ksh @ 1482

Last change on this file since 1482 was 1481, checked in by jgipsl, 5 years ago
Added case for executionType=6. This is needed for ENSEMBLE configuration.
Property licence set to The following licence information concerns ONLY the libIGCM tools ================================================================== Copyright © Centre National de la Recherche Scientifique CNRS Commissariat à l'Énergie Atomique CEA libIGCM : Library for Portable Models Computation of IGCM Group. IGCM Group is the french IPSL Global Climate Model Group. This library is a set of shell scripts and functions whose purpose is the management of the initialization, the launch, the transfer of output files, the post-processing and the monitoring of datas produce by any numerical program on any plateforme. This software is governed by the CeCILL license under French law and abiding by the rules of distribution of free software. You can use, modify and/ or redistribute the software under the terms of the CeCILL license as circulated by CEA, CNRS and INRIA at the following URL "http://www.cecill.info". As a counterpart to the access to the source code and rights to copy, modify and redistribute granted by the license, users are provided only with a limited warranty and the software's author, the holder of the economic rights, and the successive licensors have only limited liability. In this respect, the user's attention is drawn to the risks associated with loading, using, modifying and/or developing or reproducing the software by the user in light of its specific status of free software, that may mean that it is complicated to manipulate, and that also therefore means that it is reserved for developers and experienced professionals having in-depth computer knowledge. Users are therefore encouraged to load and test the software's suitability as regards their requirements in conditions enabling the security of their systems and/or data to be ensured and, more generally, to use and operate it in the same conditions as regards security. The fact that you are presently reading this means that you have had knowledge of the CeCILL license and that you accept its terms. Property svn:keywords set to `Revision Author Date`
File size: 39.0 KB

Rev	Line
[657]	1	#!/bin/ksh
[16]	2
	3	#**************************************************************
	4	# Author: Sebastien Denvil, Martial Mancip, Christian Laguerre
[373]	5	# Contact: Martial.Mancip__at__ipsl.jussieu.fr
	6	# $Revision:: $ Revision of last commit
	7	# $Author:: $ Author of last commit
	8	# $Date:: $ Date of last commit
[16]	9	# IPSL (2006)
	10	# This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
[657]	11	#
[16]	12	#**************************************************************
	13
	14	#=========================================================
	15	# The documentation of this file can be automatically generated
[657]	16	# if you use the prefix #D- for comments to be extracted.
[16]	17	# Extract with command: cat lib* \| grep "^#D-" \| cut -c "4-"
	18	#=========================================================
	19
	20	#D-#==================================================
	21	#D-LibIGCM_sys for obelix
	22	#D-#==================================================
	23	#D-
[657]	24	#D- This ksh library if a layer under some usefull
[16]	25	#D-environment variables and shell commands.
	26	#D-All those definitions depend on host particularities.
	27	#D-It manages a stack mechanism and test validity of operations.
	28	#D-All function described bellow must be prefixed by IGCM_sys.
	29
	30	#====================================================
	31	# libIGCM_sys PARAMETERS
	32	#====================================================
	33
[343]	34	#====================================================
[16]	35	# set DEBUG_sys to true to output calls of function
	36	typeset -r DEBUG_sys=${DEBUG_sys:=true}
	37
[343]	38	#====================================================
[16]	39	# Turn in dry run mode ? (sys_Put_Rest, sys_Put_Out, sys_Get)
	40	typeset -r DRYRUN=${DRYRUN:=0}
	41
	42	# YOU MUST COMPILE YOUR EXE FILES FOR DRYRUN MODE !
	43	# -------------------------------------------------------------------------------------
	44	# \| DRYRUN= \| Date computations, \| sys_Get \| Exe \| sys_Put_Out; sys_Put_Rest \|
[809]	45	# \| \| Cp/Exe/param/files \| \| \| \|
	46	# \| \| Chmod Qsub \| \| \| \|
[16]	47	# -------------------------------------------------------------------------------------
	48	# \| 0 \| yes \| yes \| yes \| yes \|
	49	# -------------------------------------------------------------------------------------
	50	# \| 1 \| yes \| yes \| yes \| no \|
	51	# -------------------------------------------------------------------------------------
	52	# \| 2 \| yes \| yes \| no \| no \|
	53	# -------------------------------------------------------------------------------------
	54	# \| 3 \| yes \| no \| no \| no \|
	55	# -------------------------------------------------------------------------------------
	56
	57	#=====================================================
	58	# Global Variables :
	59	#=====================================================
	60	# Language : "fr" or "en"
	61	typeset -r MYLANG="fr"
	62
	63	#=====================================================
[1300]	64	# Host user names project maxCpuTime
[217]	65	# $hostname ou hostname
[1180]	66	typeset HOST=${HOST:=$( hostname )}
[16]	67	# $username ou whoami
[1180]	68	typeset LOGIN=${LOGIN:=$( whoami )}
[657]	69	# $hostname of the MASTER job
[742]	70	typeset MASTER=${MASTER:=$( hostname )}
[1201]	71	# project name
	72	typeset PROJECT=NONE
	73	# jobWarningDelay in seconds
	74	typeset jobWarningDelay=${PBS_WALLTIME}
[16]	75
	76	#D-
	77	#D-#==================================================
	78	#D-Program used in libIGCM
	79	#D-#==================================================
	80
[742]	81	# Submit command
	82	typeset SUBMIT=${SUBMIT:=qsub}
[16]	83	# rsync with path
	84	typeset -r RSYNC=/usr/bin/rsync
	85	# RSYNC_opt args to rsync
[282]	86	typeset -r RSYNC_opt="-va"
[657]	87	# ie storage filesystem
	88	typeset -r STOREHOST=${MASTER}
[16]	89
	90	#====================================================
[1180]	91	# Set environment tools (ferret, nco, cdo, rebuild, ...)
[305]	92	#====================================================
[1360]	93	. /usr/share/Modules/init/ksh
[1358]	94	module load python/2.7.5
[1469]	95	. /home/orchideeshare/igcmg/MachineEnvironment/obelix/env_atlas_obelix
	96	export PATH=${PATH}:/home/orchideeshare/igcmg/IOIPSL_PLUS/modipsl.tagv2_2_4/bin/
[16]	97
[343]	98	#====================================================
[16]	99	# Host specific DIRECTORIES
	100	#====================================================
	101
	102	#====================================================
[657]	103	#- MirrorlibIGCM for frontend
	104	typeset -r MirrorlibIGCM=${MirrorlibIGCM:=false}
	105
	106	#====================================================
	107	#- libIGCM_POST for frontend
	108	typeset -r libIGCM_POST=${libIGCM}
	109
	110	#====================================================
[16]	111	#- R_EXE (==> BIN_DIR = ${MODIPSL}/bin )
	112	typeset -r R_EXE="${MODIPSL}/bin"
	113
	114	#====================================================
	115	#- SUBMIT_DIR : submission dir
[256]	116	if [ X${PBS_O_WORKDIR} != X ] ; then
[1124]	117	typeset -x SUBMIT_DIR=${SUBMIT_DIR:=${PBS_O_WORKDIR}}
[256]	118	else
[1124]	119	typeset -x SUBMIT_DIR=${SUBMIT_DIR:=${PWD}}
[256]	120	fi
[657]	121
[16]	122	#====================================================
[725]	123	#- IN
[978]	124	typeset -r R_IN=${R_IN:=/home/orchideeshare/igcmg/IGCM}
[725]	125
	126	#====================================================
[16]	127	#- RUN_DIR_PATH : Temporary working directory (=> TMP)
[838]	128	if [ X${PBS_JOBID} != X ] ; then
	129	typeset -r RUN_DIR_PATH=${RUN_DIR_PATH:=/scratch/$PBS_O_LOGNAME.$PBS_JOBID}
	130	else
[839]	131	typeset -r RUN_DIR_PATH=${RUN_DIR_PATH:=/tmp/tmp$$}
[838]	132	fi
[16]	133
	134	#====================================================
[1179]	135	#- OUTCOMMAND_PATH : tmp place to store command lines standard error and outputs
[1186]	136	typeset -r OUTCOMMAND_PATH=/tmp
[1179]	137
	138	#====================================================
[16]	139	#- HOST_MPIRUN_COMMAND
	140	typeset -r HOST_MPIRUN_COMMAND=${HOST_MPIRUN_COMMAND:="time mpirun"}
	141
[61]	142	#====================================================
	143	#- Max number of arguments passed to nco operator or demigration command
	144	UNIX_MAX_LIMIT=360
	145
[657]	146	#====================================================
[1180]	147	#- set PackDefault to false on obelix
	148	PackDefault=false
	149
	150	#====================================================
[657]	151	#- Default number of MPI task for IPSL coupled model
	152	#- required for backward compatibility
	153	#-
[1300]	154	DEFAULT_NUM_PROC_OCE=5
[657]	155	DEFAULT_NUM_PROC_CPL=1
	156	(( DEFAULT_NUM_PROC_ATM = BATCH_NUM_PROC_TOT - DEFAULT_NUM_PROC_OCE - DEFAULT_NUM_PROC_CPL ))
	157
[16]	158	#D-#==================================================
[1057]	159	#D-function IGCM_sys_defineArchives
	160	#D-* Purpose:
	161	#D-* Define ARCHIVE : Dedicated to large files
	162	#D-* Define STORAGE : Dedicated to small/medium files
	163	#D-* Define R_OUT : Output tree located on ARCHIVE
	164	#D-* Define R_FIG : Output tree located on STORAGE hosting figures (monitoring and atlas, and/or small files)
	165	#D-* Define R_BUF : Output tree located on STORAGE hosting files waiting for rebuild or pack processes
	166	#D-* if SpaceName=TEST nothing special will hapen
[725]	167	#D-* Examples:
	168	#D-
[1057]	169	function IGCM_sys_defineArchives {
	170	IGCM_debug_PushStack "IGCM_sys_defineArchives"
	171
[1060]	172	if [ ! X${config_UserChoices_ARCHIVE} = X ]; then
[1057]	173	#====================================================
	174	#- ARCHIVE (dedicated to large files)
[1063]	175	ARCHIVE=${config_UserChoices_ARCHIVE}
[1057]	176	else
	177	#====================================================
	178	#- ARCHIVE (dedicated to large files)
[1063]	179	ARCHIVE=${ARCHIVE:=/home/scratch01/${LOGIN}}
[1057]	180	fi
	181
[1060]	182	if [ ! X${config_UserChoices_STORAGE} = X ]; then
[1057]	183	#====================================================
	184	#- STORAGE (dedicated to small/medium files)
[1063]	185	STORAGE=${config_UserChoices_STORAGE}
[1057]	186	else
	187	#====================================================
	188	#- STORAGE (dedicated to small/medium files)
[1063]	189	STORAGE=${ARCHIVE}
[1057]	190	fi
	191
	192	# ON OBELIX NO SPECIAL CASE WHEN X${config_UserChoices_SpaceName} = XTEST
	193
	194	#====================================================
	195	#- R_OUT
[1063]	196	R_OUT=${ARCHIVE}/IGCM_OUT
[1057]	197
	198	#====================================================
	199	#- R_FIG (hosting figures : monitoring and atlas, and/or small files)
[1063]	200	R_FIG=${STORAGE}/IGCM_OUT
[1057]	201
	202	#====================================================
[1431]	203	#- CMIP6 (hosting CMIP6 files produced by XIOS2 and configured by dr2xml)
	204	CMIP6_BUF=${STORAGE}/IGCM_OUT
	205
	206	#====================================================
[1057]	207	#- R_BUF (ONLY FOR double copy an scratch)
[1063]	208	R_BUF=${STORAGE}/IGCM_OUT
[1057]	209
	210	IGCM_debug_Print 1 "R_OUT has been defined = ${R_OUT}"
	211	IGCM_debug_Print 1 "R_BUF has been defined = ${R_BUF}"
	212	IGCM_debug_Print 1 "R_FIG has been defined = ${R_FIG}"
	213
	214	IGCM_debug_PopStack "IGCM_sys_defineArchives"
[725]	215	}
	216
	217	#D-#==================================================
[37]	218	#D-function IGCM_sys_RshArchive
	219	#D-* Purpose: Archive rsh command
	220	#D-* Examples:
	221	#D-
	222	function IGCM_sys_RshArchive {
[657]	223	IGCM_debug_PushStack "IGCM_sys_RshArchive" $@
	224	/bin/ksh <<-EOF
[40]	225	${@}
	226	EOF
[811]	227	status=$?
	228	if [ ${status} -gt 0 ] ; then
	229	IGCM_debug_Print 2 "IGCM_sys_RshArchive : command failed error code ${status}"
[657]	230	IGCM_debug_Exit "IGCM_sys_RshArchive"
	231	fi
	232	IGCM_debug_PopStack "IGCM_sys_RshArchive"
[37]	233	}
	234
	235	#D-#==================================================
[1178]	236	#D-function IGCM_sys_RshArchive_NoError
	237	#D-* Purpose: Archive rsh command, without error
	238	#D-* used only in monitoring.job
	239	#D-* Examples:
	240	#D-
	241	function IGCM_sys_RshArchive_NoError {
	242	IGCM_debug_PushStack "IGCM_sys_RshArchive_NoError" $@
	243	/bin/ksh <<-EOF
[1181]	244	${@} 2> /dev/null
[1178]	245	EOF
	246	IGCM_debug_PopStack "IGCM_sys_RshArchive_NoError"
	247	}
	248
	249	#D-#==================================================
[16]	250	#D-function IGCM_sys_MkdirArchive
	251	#D-* Purpose: Mkdir on Archive
	252	#D-* Examples:
	253	#D-
	254	function IGCM_sys_MkdirArchive {
[657]	255	IGCM_debug_PushStack "IGCM_sys_MkdirArchive" $@
	256	if ( $DEBUG_sys ) ; then
	257	echo "IGCM_sys_MkdirArchive :" $@
	258	fi
[717]	259	#- creation de repertoire sur le serveur fichier
[657]	260	if [ ! -d ${1} ]; then
[1178]	261	\mkdir -p $1
[811]	262	status=$?
	263
	264	if [ ${status} -gt 0 ] ; then
	265	IGCM_debug_Print 2 "IGCM_sys_MkdirArchive : mkdir failed error code ${status}"
[657]	266	IGCM_debug_Exit "IGCM_sys_MkdirArchive"
[16]	267	fi
[657]	268	fi
	269	IGCM_debug_PopStack "IGCM_sys_MkdirArchive"
[16]	270	}
	271
	272	#D-#==================================================
	273	#D-function IGCM_sys_TestDirArchive
	274	#D-* Purpose: Test Directory that must exists on Archive
	275	#D-* Examples:
	276	#D-
	277	function IGCM_sys_TestDirArchive {
[657]	278	IGCM_debug_PushStack "IGCM_sys_TestDirArchive" $@
	279	if ( $DEBUG_sys ) ; then
	280	echo "IGCM_sys_TestDirArchive :" $@
	281	fi
	282	typeset ExistFlag
	283	ExistFlag=$( [ -d $1 ] && echo 0 \|\| echo 1 )
	284	IGCM_debug_PopStack "IGCM_sys_TestDirArchive"
	285	return ${ExistFlag}
[16]	286	}
	287
	288	#D-#==================================================
[800]	289	#D-function IGCM_sys_IsFileArchived
[1180]	290	#D-* Purpose: Test file that must NOT EXISTS on Archive based on filename only
[800]	291	#D-* Examples:
	292	#D-
	293	function IGCM_sys_IsFileArchived {
	294	IGCM_debug_PushStack "IGCM_sys_IsFileArchived" $@
	295	if ( $DEBUG_sys ) ; then
	296	echo "IGCM_sys_IsFileArchived :" $@
	297	fi
	298	typeset IsArchivedFlag
	299	# Never archived for this system
	300	IsArchivedFlag=1
	301	IGCM_debug_PopStack "IGCM_sys_IsFileArchived"
	302
	303	return ${IsArchivedFlag}
	304	}
	305
	306	#D-#==================================================
[16]	307	#D-function IGCM_sys_TestFileArchive
	308	#D-* Purpose: Test file that must NOT EXISTS on Archive
	309	#D-* Examples:
	310	#D-
	311	function IGCM_sys_TestFileArchive {
[657]	312	IGCM_debug_PushStack "IGCM_sys_TestFileArchive" $@
	313	typeset ExistFlag
	314	ExistFlag=$( [ -f $1 ] && echo 0 \|\| echo 1 )
	315	IGCM_debug_PopStack "IGCM_sys_TestFileArchive"
[67]	316
[657]	317	return ${ExistFlag}
[67]	318	}
	319
	320	#D-#==================================================
	321	#D-function IGCM_sys_CountFileArchive
	322	#D-* Purpose: Count files on Archive filesystem
	323	#D-* Examples:
	324	#D-
	325	function IGCM_sys_CountFileArchive {
[657]	326	IGCM_debug_PushStack "IGCM_sys_CountFileArchive" $@
	327	ls ${@} 2>/dev/null \| wc -l
	328	if [ $? -gt 0 ] ; then
	329	echo "IGCM_sys_CountFileArchive : erreur."
	330	fi
	331	IGCM_debug_PopStack "IGCM_sys_CountFileArchive"
[16]	332	}
	333
	334	#D-#==================================================
	335	#D-function IGCM_sys_Tree
	336	#D-* Purpose: Tree directories with files on ${ARCHIVE}
	337	#D-* Examples: IGCM_sys_Tree ${R_IN} ${R_OUT}
	338	#D-
	339	function IGCM_sys_Tree {
[657]	340	IGCM_debug_PushStack "IGCM_sys_Tree" $@
	341	if ( $DEBUG_sys ) ; then
	342	echo "IGCM_sys_Tree :" $@
	343	fi
[16]	344
[1300]	345	\ls -lR ${@}
[16]	346
[657]	347	IGCM_debug_PopStack "IGCM_sys_Tree"
[16]	348	}
	349
	350	#D-#==================================================
	351	#D-function IGCM_sys_Qsub
	352	#D-* Purpose: Qsub new job
	353	#D-* Examples:
	354	#D-
	355	function IGCM_sys_Qsub {
[657]	356	IGCM_debug_PushStack "IGCM_sys_Qsub" $@
	357	if ( $DEBUG_sys ) ; then
	358	echo "IGCM_sys_Qsub :" $@
	359	fi
[923]	360	typeset options status
	361	options="-o ${SUBMIT_DIR}/${Script_Output}"
[1179]	362	/usr/local/bin/qsub ${options} $1 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[811]	363	status=$?
[949]	364
[1179]	365	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[811]	366	if [ ${status} -gt 0 ] ; then
[1180]	367	IGCM_debug_Print 2 "IGCM_sys_Qsub ${options} $1 : error code ${status}"
[657]	368	IGCM_debug_Exit "IGCM_sys_Qsub"
[949]	369	else
[1179]	370	JobID=$( gawk {'print $1'} ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ )
[657]	371	fi
	372	IGCM_debug_PopStack "IGCM_sys_Qsub"
[16]	373	}
	374
	375	#D-#==================================================
	376	#D-function IGCM_sys_QsubPost
	377	#D-* Purpose: Qsub new job on scalaire
	378	#D-* Examples:
	379	#D-
	380	function IGCM_sys_QsubPost {
[657]	381	IGCM_debug_PushStack "IGCM_sys_QsubPost" $@
	382	if ( $DEBUG_sys ) ; then
	383	echo "IGCM_sys_QsubPost :" $@
	384	fi
[923]	385	typeset options status
	386	options="-o ${POST_DIR}/${Script_Post_Output}.out -v ${listVarEnv}"
[1179]	387	/usr/local/bin/qsub ${options} ${libIGCM_POST}/$1.job > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[811]	388	status=$?
[949]	389
[1179]	390	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[811]	391	if [ ${status} -gt 0 ] ; then
[923]	392	IGCM_debug_Print 2 "IGCM_sys_QsubPost ${options} ${libIGCM_POST}/$1.job : error code ${status}"
[657]	393	IGCM_debug_Exit "IGCM_sys_QsubPost"
[949]	394	else
[1179]	395	JobID=$( gawk {'print $1'} ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ )
[657]	396	fi
	397	IGCM_debug_PopStack "IGCM_sys_QsubPost"
[16]	398	}
	399
	400	#D-*************************
[657]	401	#D- File transfer functions
[16]	402	#D-*************************
	403	#D-
	404
	405	#D-#==================================================
[203]	406	#D-function IGCM_sys_RmRunDir
	407	#D-* Purpose: rm tmpdir (dummy function most of the time batch
	408	#D- scheduler will do the job)
	409	#D-* Examples:
	410	#D-
	411	function IGCM_sys_RmRunDir {
[1057]	412	IGCM_debug_PushStack "IGCM_sys_RmRunDir" $@
[657]	413	if ( $DEBUG_sys ) ; then
	414	echo "IGCM_sys_RmRunDir :" $@
[964]	415	echo "Dummy call, let the scheduler do that."
[657]	416	fi
	417	IGCM_debug_PopStack "IGCM_sys_RmRunDir"
[203]	418	}
	419
	420	#D-#==================================================
[16]	421	#D-function IGCM_sys_Put_Dir
	422	#D-* Purpose: Copy a complete directory on $(ARCHIVE)
	423	#D-* Examples:
	424	#D-
	425	function IGCM_sys_Put_Dir {
[657]	426	IGCM_debug_PushStack "IGCM_sys_Put_Dir" $@
	427	if ( $DEBUG_sys ) ; then
	428	echo "IGCM_sys_Put_Dir :" $@
	429	fi
	430	if [ $DRYRUN = 0 ]; then
	431	if [ ! -d ${1} ] ; then
	432	echo "WARNING : IGCM_sys_Put_Dir ${1} DOES NOT EXIST ."
	433	IGCM_debug_PopStack "IGCM_sys_Put_Dir"
	434	return
[16]	435	fi
	436
[717]	437	typeset status
[16]	438
[717]	439	# Only if we use rsync
	440	#IGCM_sys_TestDirArchive $( dirname $2 )
	441	#
	442	#USUAL WAY
[1179]	443	\cp -r $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[717]	444	status=$?
[16]	445
[717]	446	if [ ${status} -gt 0 ] ; then
[811]	447	IGCM_debug_Print 2 "IGCM_sys_Put_Dir : cp failed error code ${status}"
[1179]	448	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[657]	449	IGCM_debug_Exit "IGCM_sys_Put_Dir"
[16]	450	else
[1179]	451	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[16]	452	fi
[657]	453	fi
	454	IGCM_debug_PopStack "IGCM_sys_Put_Dir"
[16]	455	}
	456
	457	#D-#==================================================
	458	#D-function IGCM_sys_Get_Dir
[717]	459	#D-* Purpose: Copy a complete directory from ${ARCHIVE}
[16]	460	#D-* Examples:
	461	#D-
	462	function IGCM_sys_Get_Dir {
[657]	463	IGCM_debug_PushStack "IGCM_sys_Get_Dir" $@
	464	if ( $DEBUG_sys ) ; then
	465	echo "IGCM_sys_Get_Dir :" $@
	466	fi
	467	if [ $DRYRUN = 0 ]; then
[1180]	468	typeset NB_ESSAI DELAI status i
	469	# number of tentative
	470	NB_ESSAI=3
	471	# time delay between tentative
	472	DELAI=2
[16]	473
[1180]	474	#
	475	# USUAL WAY
	476	i=0
	477	while [ $i -lt $NB_ESSAI ] ; do
	478	\cp -ur $1 $2 >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
	479	status=$?
	480	if [ ${status} -gt 0 ] ; then
	481	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : cp failed error code ${status} ${i}/${NB_ESSAI}"
	482	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : sleep ${DELAI} seconds and try again."
	483	sleep $DELAI
	484	else
	485	break
	486	fi
	487	(( i = i + 1 ))
	488	done
[16]	489
[717]	490	if [ ${status} -gt 0 ] ; then
[811]	491	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : cp failed error code ${status}"
[1179]	492	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[657]	493	IGCM_debug_Exit "IGCM_sys_Get_Dir"
[16]	494	else
[1179]	495	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[16]	496	fi
[657]	497	fi
	498	IGCM_debug_PopStack "IGCM_sys_Get_Dir"
[16]	499	}
	500
	501	#D-#==================================================
	502	#D-function IGCM_sys_Put_Rest
[717]	503	#D-* Purpose: Put computied restarts on ${ARCHIVE}.
[282]	504	#D- File and target directory must exist.
[16]	505	#D-* Examples:
	506	#D-
	507	function IGCM_sys_Put_Rest {
[657]	508	IGCM_debug_PushStack "IGCM_sys_Put_Rest" $@
	509	if ( $DEBUG_sys ) ; then
	510	echo "IGCM_sys_Put_Rest :" $@
	511	fi
	512	if [ $DRYRUN = 0 ]; then
	513	if [ ! -f ${1} ] ; then
	514	echo "ERROR : IGCM_sys_Put_Rest ${1} DOES NOT EXIST ."
	515	IGCM_debug_Exit "IGCM_sys_Put_Rest"
[16]	516	fi
[717]	517
	518	typeset status
	519	#
[1178]	520	# USUAL WAY
[1179]	521	\cp $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[717]	522	status=$?
[16]	523
[717]	524	# #RSYNC WITH NETWORK SSH CALL
[1179]	525	# echo ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
	526	# ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[16]	527
[717]	528	# #RSYNC WITH NFS USE
[1179]	529	# echo ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
	530	# ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[16]	531
[717]	532	# status=$?
	533	# IGCM_sys_Rsync_out $status
	534
[1179]	535	# ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[717]	536	# (( status=status+$? ))
	537
	538	if [ ${status} -gt 0 ] ; then
[1180]	539	IGCM_debug_Print 2 "IGCM_sys_Put_Rest : cp failed error code ${status}"
[1178]	540	[ -f ${1} ] && ls -l ${1}
	541	[ -f ${2} ] && ls -l ${2}
	542	[ -f ${2}/${1} ] && ls -l ${2}/${1}
[1179]	543	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[657]	544	IGCM_debug_Exit "IGCM_sys_Put_Rest"
	545	else
[1178]	546
	547	if [ X${JobType} = XRUN ] ; then
	548	[ -f ${2} ] && IGCM_sys_Chmod 444 ${2}
	549	[ -f ${2}/${1} ] && IGCM_sys_Chmod 444 ${2}/${1}
	550	fi
	551
[1179]	552	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[657]	553	fi
	554	fi
	555	IGCM_debug_PopStack "IGCM_sys_Put_Rest"
	556	}
[16]	557
[657]	558	#D-#==================================================
[16]	559	#D-function IGCM_sys_Put_Out
[717]	560	#D-* Purpose: Copy a file on ${ARCHIVE} after having chmod it in readonly
[16]	561	#D-* Examples:
	562	#D-
	563	function IGCM_sys_Put_Out {
[657]	564	IGCM_debug_PushStack "IGCM_sys_Put_Out" $@
	565	if ( $DEBUG_sys ) ; then
	566	echo "IGCM_sys_Put_Out :" $@
	567	fi
[811]	568
	569	typeset status
	570
[657]	571	if [ $DRYRUN = 0 ]; then
[811]	572	if [ ! -f ${1} ] ; then
[657]	573	echo "WARNING : IGCM_sys_Put_Out ${1} DOES NOT EXIST ."
	574	IGCM_debug_PopStack "IGCM_sys_Put_Out"
	575	return 1
[16]	576	fi
[719]	577	#
[811]	578	IGCM_sys_MkdirArchive $( dirname $2 )
	579	#
[657]	580	if [ X${JobType} = XRUN ] ; then
	581	if [ X${3} = X ] ; then
	582	IGCM_sys_Chmod 444 ${1}
	583	fi
	584	fi
[717]	585	#
[16]	586
[1179]	587	echo ${RSYNC} ${RSYNC_opt} $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
	588	${RSYNC} ${RSYNC_opt} $1 $2 >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[717]	589	status=$?
	590	IGCM_sys_Rsync_out $status
[16]	591
[1179]	592	${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[717]	593	(( status=status+$? ))
[16]	594
[717]	595	if [ ${status} -gt 0 ] ; then
[811]	596	IGCM_debug_Print 2 "IGCM_sys_Put_Out : rsync failed error code ${status}"
[1179]	597	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[657]	598	IGCM_debug_Exit "IGCM_sys_Put_Out"
[16]	599	else
[1179]	600	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[16]	601	fi
[657]	602	fi
	603	IGCM_debug_PopStack "IGCM_sys_Put_Out"
	604	return 0
[16]	605	}
	606
	607	#D-#==================================================
	608	#D-function IGCM_sys_Get
	609	#D-* Purpose: Get a file from ${ARCHIVE}
	610	#D-* Examples: IGCM_sys_Get myfile /destpath/myfile_with_PREFIX
	611	#D- IGCM_sys_Get /l Array_contain_myfiles /destpath/
	612	function IGCM_sys_Get {
[657]	613	IGCM_debug_PushStack "IGCM_sys_Get" $@
[16]	614
[811]	615	typeset DEST status dm_liste
[16]	616
[657]	617	if ( $DEBUG_sys ) ; then
	618	echo "IGCM_sys_Get :" $@
	619	fi
	620	if [ $DRYRUN -le 2 ]; then
	621	if [ X${1} = X'/l' ] ; then
	622	eval set +A dm_liste \${${2}}
	623	else
	624	dm_liste=${1}
[16]	625	fi
[657]	626	eval DEST=\${${#}}
[16]	627
[717]	628	# test if the (first) file is present in the old computation :
[657]	629	IGCM_sys_TestFileArchive ${dm_liste[0]}
[717]	630	status=$?
	631	if [ ${status} -gt 0 ] ; then
[657]	632	echo "IGCM_sys_Get, ERROR : regular file ${dm_liste[0]} DOES NOT EXIST ."
	633	IGCM_debug_Exit "IGCM_sys_Get"
[811]	634	return
[657]	635	fi
[16]	636
[811]	637	#USUAL WAY
[1179]	638	\cp ${dm_liste[*]} ${DEST} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
[811]	639	status=$?
[16]	640
[811]	641	if [ ${status} -gt 0 ] ; then
	642	IGCM_debug_Print 2 "IGCM_sys_Get : cp failed error code ${status}"
[1179]	643	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[657]	644	IGCM_debug_Exit "IGCM_sys_Get"
[16]	645	else
[1179]	646	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
[16]	647	fi
[657]	648	fi
	649	IGCM_debug_PopStack "IGCM_sys_Get"
[16]	650	}
	651
	652	#D-#==================================================
[1131]	653	#D-function IGCM_sys_GetDate_Monitoring
	654	#D-* Purpose: get the last year for which the monitoring has been computed
	655	#D-* Examples:
	656	#D-
	657	function IGCM_sys_GetDate_Monitoring {
	658	IGCM_debug_PushStack "IGCM_sys_GetDate_Monitoring" $@
	659	if ( $DEBUG_sys ) ; then
	660	echo "IGCM_sys_GetDate_Monitoring :" $@
	661	fi
	662
	663	eval ${2}=$( cdo showyear ${1} 2> /dev/null \| gawk '{ print $NF }' )
	664
	665	IGCM_debug_PopStack "IGCM_sys_GetDate_Monitoring"
	666	}
	667
	668	#D-#==================================================
[343]	669	#D-function IGCM_sys_Dods_Rm
[717]	670	#D-* Purpose: DO NOTHING ! Put ${ARCHIVE} files on DODS internet protocole.
[343]	671	#D-* Examples:
	672	#D-
	673	function IGCM_sys_Dods_Rm {
[657]	674	if ( $DEBUG_sys ) ; then
	675	echo "IGCM_sys_Dods_Rm :" $@
	676	fi
	677	return 0
[343]	678	}
	679
	680	#D-#==================================================
	681	#D-function IGCM_sys_Dods_Cp
[717]	682	#D-* Purpose: Copy $(ARCHIVE) files on DODS internet protocole.
[343]	683	#D-* Examples:
	684	#D-
	685	function IGCM_sys_Dods_Cp {
[657]	686	if ( $DEBUG_sys ) ; then
	687	echo "IGCM_sys_Dods_Cp :" $@
	688	fi
	689	return 0
[343]	690	}
	691
	692	#D-#==================================================
[16]	693	#D-function IGCM_sys_Put_Dods
[811]	694	#D-* Purpose: Put ${ARCHIVE} files on DODS internet protocole. Dummy function here
[16]	695	#D-* Examples:
	696	#D-
	697	function IGCM_sys_Put_Dods {
[657]	698	IGCM_debug_PushStack "IGCM_sys_Put_Dods" $@
	699	if ( $DEBUG_sys ) ; then
	700	echo "IGCM_sys_Put_Dods :" $@
	701	fi
	702	IGCM_debug_PopStack "IGCM_sys_Put_Dods"
[16]	703	}
	704
[657]	705	##############################################################
	706	# REBUILD OPERATOR
[536]	707
[811]	708	#D-#==================================================
[1163]	709	#D-function IGCM_sys_sync
[1300]	710	#D-* Purpose: flush buffer on disk (dummy function on Obelix)
[1163]	711	#D-* Examples:
	712	#D-
	713	function IGCM_sys_sync {
	714	IGCM_debug_PushStack "IGCM_sys_sync" $@
	715	if ( $DEBUG_sys ) ; then
	716	echo "IGCM_sys_sync :" $@
	717	echo "Dummy call, let the system do that."
	718	fi
	719	IGCM_debug_PopStack "IGCM_sys_sync"
	720	}
	721
[657]	722	############################################################
	723	# Activate Running Environnment Variables
[536]	724
[811]	725	#D-#==================================================
[1239]	726	#D-function IGCM_sys_activ_variables
[811]	727	#D-* Purpose: set environement variables prior to execution
	728	#D-* Examples:
	729	#D-
[657]	730	function IGCM_sys_activ_variables {
	731	IGCM_debug_PushStack "IGCM_sys_activ_variables"
	732	if ( $DEBUG_sys ) ; then
	733	echo "IGCM_sys_activ_variables"
	734	fi
[811]	735
	736	# --------------------------------------------------------------------
	737	#D- MPI specifications
	738	# --------------------------------------------------------------------
	739
	740	# --------------------------------------------------------------------
	741	#D- Other specifications
	742	# --------------------------------------------------------------------
	743
[1300]	744	ulimit -s unlimited
	745
[657]	746	IGCM_debug_PopStack "IGCM_sys_activ_variables"
[16]	747	}
	748
[657]	749	############################################################
	750	# Desactivate Running Environnment Variables
[16]	751
[811]	752	#D-#==================================================
	753	#D-function IGCM_sys_desactiv_variables
	754	#D-* Purpose: unset environement variables after execution
	755	#D-* Examples:
	756	#D-
[657]	757	function IGCM_sys_desactiv_variables {
	758	IGCM_debug_PushStack "IGCM_sys_desactiv_variables"
	759	if ( $DEBUG_sys ) ; then
	760	echo "IGCM_sys_desactiv_variables"
	761	fi
[811]	762	# --------------------------------------------------------------------
	763	#D- MPI specifications
	764	# --------------------------------------------------------------------
	765
	766	# --------------------------------------------------------------------
	767	#D- Other specifications
	768	# --------------------------------------------------------------------
	769
[657]	770	IGCM_debug_PopStack "IGCM_sys_desactiv_variables"
	771	}
	772
	773	############################################################
[1239]	774	# Update job headers to be used by the scheduler
	775
	776	#D-#==================================================
	777	#D-function IGCM_sys_updateHeaders
	778	#D-* Purpose: Update job headers to be used by the scheduler
	779	#D-* Examples: IGCM_sys_updateHeaders /path/to/Job_MYEXP
	780	#D-
	781	function IGCM_sys_updateHeaders {
	782	IGCM_debug_PushStack "IGCM_sys_updateHeaders"
	783	if ( $DEBUG_sys ) ; then
	784	echo "IGCM_sys_updateHeaders"
	785	fi
	786	typeset file
	787	file=$1
	788
	789	if [ ${executionType} -eq 1 ] ; then
	790	# MPMD + MPI
	791	sed -e "/::openMPthreads::/d" \
	792	-e "s/::JobNumProcTot::/${coreNumber}/" \
	793	${file} > ${file}.tmp
	794
	795	elif [ ${executionType} -eq 2 ] ; then
	796	# MPMD + MPI + OMP
	797	sed -e "s/::openMPthreads::/${openMPthreads}/" \
	798	-e "s/::JobNumProcTot::/${coreNumber}/" \
	799	${file} > ${file}.tmp
	800
	801	elif [ ${executionType} -eq 3 ] ; then
	802	# SPMD + MPI/OMP
	803	sed -e "s/::openMPthreads::/${openMPthreads}/" \
	804	-e "s/::JobNumProcTot::/${mpiTasks}/" \
	805	${file} > ${file}.tmp
	806
	807	elif [ ${executionType} -eq 4 ] ; then
	808	# SPMD + MPI only
	809	sed -e "s/::JobNumProcTot::/${mpiTasks}/" \
	810	-e "/::openMPthreads::/d" \
	811	${file} > ${file}.tmp
	812
	813	elif [ ${executionType} -eq 5 ] ; then
	814	# SPMD + OMP only
	815	sed -e "s/::openMPthreads::/${openMPthreads}/" \
	816	-e "/::JobNumProcTot::/d" \
	817	${file} > ${file}.tmp
	818
	819	elif [ ${executionType} -eq 6 ] ; then
	820	# SEQUENTIAL THEN
	821	sed -e "s/::JobNumProcTot::/1/" \
	822	-e "/::openMPthreads::/d" \
	823	${file} > ${file}.tmp
	824
	825	fi
	826
	827	IGCM_sys_Mv ${file}.tmp ${file}
	828
	829	IGCM_debug_PopStack "IGCM_sys_updateHeaders"
	830	}
	831
	832	############################################################
[800]	833	# Build MPI/OMP scripts run file (dummy function)
[657]	834
[811]	835	#D-#==================================================
	836	#D-function IGCM_sys_build_run_file
	837	#D-* Purpose: build run file (deprecated)
	838	#D-* Examples:
	839	#D-
[657]	840	function IGCM_sys_build_run_file {
[664]	841	IGCM_debug_Print 3 " dummy function : IGCM_sys_build_run_file "
[657]	842	}
	843
	844	############################################################
	845	# Build MPI/OMP scripts
[811]	846
	847	#D-#==================================================
	848	#D-function IGCM_sys_build_execution_scripts
	849	#D-* Purpose: build execution scripts to be launch by ${HOST_MPIRUN_COMMAND}
	850	#D-* Examples:
	851	#D-
[657]	852	function IGCM_sys_build_execution_scripts
	853	{
	854	IGCM_debug_PushStack "IGCM_sys_build_execution_scripts" $@
	855	if ( $DEBUG_sys ) ; then
	856	echo "IGCM_sys_build_execution_scripts " $@
	857	fi
[1180]	858
[1300]	859	EXECUTION=${HOST_MPIRUN_COMMAND}
[657]	860
[1300]	861	# MPMD mode
[657]	862	if ( ${OK_PARA_MPMD} ) ; then
	863
[1300]	864	# Only MPI (MPMD)
	865	if ( ! ${OK_PARA_OMP} ) ; then
[657]	866
[1300]	867	if [ -f run_file ] ; then
	868	IGCM_sys_Rm -f run_file
	869	fi
	870	touch run_file
[657]	871
[1300]	872	# Build run_file
	873
	874	# First loop on the components for the coupler ie oasis (only if oasis3)
	875	# the coupler ie oasis3 must be the first one
	876	for comp in ${config_ListOfComponents[*]} ; do
	877
	878	eval ExeNameIn=\${config_Executable_${comp}[0]}
	879	eval ExeNameOut=\${config_Executable_${comp}[1]}
	880
	881	# for CPL component only
	882	if [ "X${comp}" = "XCPL" ] && [ "X${ExeNameOut}" != X\"\" ] ; then
	883	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
	884	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
	885	echo "-np ${comp_proc_mpi_loc} ./${ExeNameOut} " >> run_file
	886	fi
	887	done
	888
	889	# Then second loop on the components
	890	for comp in ${config_ListOfComponents[*]} ; do
	891
	892	eval ExeNameIn=\${config_Executable_${comp}[0]}
	893	eval ExeNameOut=\${config_Executable_${comp}[1]}
	894
	895	# Only if we really have an executable for the component and not the coupler ie oasis:
	896	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${comp}" != "XCPL" ] ) ; then
	897
	898	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
	899	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
	900	echo "-np ${comp_proc_mpi_loc} ./${ExeNameOut}" >> run_file
	901	fi
	902	done
	903
	904	EXECUTION="${HOST_MPIRUN_COMMAND} --app ./run_file"
	905
	906	IGCM_sys_Chmod u+x run_file
	907	if ( $DEBUG_sys ) ; then
	908	echo "run_file contains : "
	909	cat run_file
	910	fi
	911
	912	# MPI-OpenMP (MPMD)
	913	else
	914
[657]	915	# Hosts treatment
	916	${HOST_MPIRUN_COMMAND} hostname \| sort \| uniq > hosts.tmp
	917
	918	i=0
	919	rm -f hosts
[1300]	920	IGCM_debug_Print 1 "sys Obelix, Hosts available :"
[657]	921	for nodes in `cat hosts.tmp` ; do
	922	host[$i]=$nodes
	923	echo "${host[$i]} slots=1 max_slots=1" >> hosts
	924	IGCM_debug_Print 1 ${host[$i]}
	925	i=$((i+1))
	926	done
	927	rm -f hosts.tmp
	928
	929	listnodes=${host[*]}
	930
	931	EXECUTION="${HOST_MPIRUN_COMMAND} -hostfile hosts"
	932
	933	# Initialisation
[1300]	934	rank=0
	935	current_core=0
	936	core_per_node=6
[657]	937	init_exec=n
	938
[1300]	939	# Loop on the components
[657]	940	for comp in ${config_ListOfComponents[*]} ; do
	941
[1300]	942	eval ExeNameIn=\${config_Executable_${comp}[0]}
	943	eval ExeNameOut=\${config_Executable_${comp}[1]}
[657]	944
[1300]	945	# Not possible if oasis has an executable (i.e old version of oasis3)
	946	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${comp}" = "XCPL" ] ) ; then
	947	IGCM_debug_Exit "ERROR MPMD with hybrid MPI-OpenMP is not available with oasis3 version"
	948	IGCM_debug_Print 2 "Only available with oasis3-MCT version coupler"
	949	IGCM_debug_Verif_Exit
[657]	950	fi
	951
	952	# Only if we really have an executable for the component :
[1300]	953	if [ "X${ExeNameOut}" != X\"\" ] ; then
[657]	954
	955	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
	956	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
	957
	958	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
	959	echo "" >> script_${ExeNameOut}.ksh
[1300]	960	if [ ${comp_proc_omp_loc} -gt 1 ] ; then
	961
	962	# Check if the number of threads is correct
	963	case ${comp_proc_omp_loc} in
	964	2\|4\|6)
	965	IGCM_debug_Print 1 "You run ${ExeNameOut} on ${comp_proc_omp_loc} OMP threads"
	966	;;
	967	*)
	968	IGCM_debug_Exit "ERROR with OMP parameters !"
	969	IGCM_debug_Print 2 "${comp_proc_omp_loc} is not possible as number of OMP threads"
	970	IGCM_debug_Print 2 "Only 2,4,6 as number of OMP threads are possible "
	971	IGCM_debug_Verif_Exit
	972	;;
	973	esac
	974	#echo "export KMP_STACKSIZE=3g" >> script_${ExeNameOut}.ksh
	975	#echo "export KMP_LIBRARY=turnaround" >> script_${ExeNameOut}.ksh
	976	#echo "export MKL_SERIAL=YES" >> script_${ExeNameOut}.ksh
	977	echo "export OMP_STACKSIZE=200M" >> script_${ExeNameOut}.ksh
	978	echo "export OMP_NUM_THREADS=${comp_proc_omp_loc}" >> script_${ExeNameOut}.ksh
	979	fi
[1318]	980
[1317]	981	#echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
[1318]	982	echo "(( MYMPIRANK = OMPI_COMM_WORLD_RANK )) " >> script_${ExeNameOut}.ksh
	983	echo "MYMPIRANK=\$(printf '%3.3d\n' \${MYMPIRANK})" >> script_${ExeNameOut}.ksh
	984	echo "./${ExeNameOut} > out_${ExeNameOut}.out.\${MYMPIRANK} 2>out_${ExeNameOut}.err.\${MYMPIRANK}" >> script_${ExeNameOut}.ksh
[657]	985	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
	986
[1300]	987	if [ ${init_exec} = y ] ; then
	988	EXECUTION="${EXECUTION} : -np ${comp_proc_mpi_loc} ./script_${ExeNameOut}.ksh"
	989	else
	990	EXECUTION="${EXECUTION} -np ${comp_proc_mpi_loc} ./script_${ExeNameOut}.ksh"
	991	init_exec=y
	992	fi
[657]	993
[1300]	994	# Build rankfile : method used to assign cores and nodes for the MPI process
	995	# Ex :
	996	#rank 0=curie5296 slot=0,1,2,3
	997	#rank 1=curie5296 slot=4,5,6,7
	998	# Example of final command :
	999	# mpirun -hostfile hosts -rankfile rankfile -np 27 ./script_lmdz.x.ksh : -np 5 ./script_opa.xx.ksh
	1000	# with script_lmdz.x.ksh :
	1001	# #!/bin/ksh
	1002	#export OMP_STACKSIZE=200M
	1003	#export OMP_NUM_THREADS=4
	1004	#./lmdz.x
[657]	1005
[1300]	1006	for nb_proc_mpi in `seq 0 $(($comp_proc_mpi_loc-1))`; do
	1007	(( index_host = current_core / core_per_node ))
	1008	host_value=${host[${index_host}]}
	1009	(( slot = current_core % core_per_node ))
	1010	virg=","
	1011	string_final=""
	1012	for index in `seq $slot $(($slot+$comp_proc_omp_loc-1))`; do
	1013	string=$index$virg
	1014	string_final=$string_final$string
	1015	done
	1016	string_final=$( echo $string_final \| sed "s/.$//" )
	1017	echo "rank $rank=$host_value slot=$string_final" >> rankfile
	1018	(( rank = rank + 1 ))
	1019	(( current_core = current_core + comp_proc_omp_loc ))
[657]	1020	done
	1021	fi
	1022	done
[16]	1023	fi
	1024
[1300]	1025	# Only one executable (SPMD mode).
	1026	else
[657]	1027
[664]	1028	for comp in ${config_ListOfComponents[*]} ; do
[657]	1029
[664]	1030	# Only if we really have an executable for the component :
	1031	eval ExeNameOut=\${config_Executable_${comp}[1]}
	1032	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${ExeNameOut}" != "Xinca.dat" ] ) ; then
[657]	1033
[664]	1034	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
	1035	echo "" >> script_${ExeNameOut}.ksh
	1036	if ( ${OK_PARA_OMP} ) ; then
	1037	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
[1300]	1038	echo "export OMP_STACKSIZE=200M" >> script_${ExeNameOut}.ksh
	1039	echo "export OMP_NUM_THREADS=${comp_proc_omp_loc}" >> script_${ExeNameOut}.ksh
[664]	1040	fi
[1480]	1041
	1042	if [ ${mpiTasks} -eq 1 ] && [ ${openMPthreads} -eq 1 ] ; then
	1043	# Case with only one proc
	1044	# mpirun should not be used at obelix for only one proc
	1045	# Output text per process
	1046	echo "./${ExeNameOut} > out_${ExeNameOut}.out 2>out_${ExeNameOut}.err" >> script_${ExeNameOut}.ksh
	1047	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
	1048	EXECUTION="time ./script_${ExeNameOut}.ksh"
	1049	elif ( ${OK_PARA_MPI} ) ; then
[664]	1050	# Default : mpirun used if nb_proc gt 1
[1480]	1051	# Output text per process
[1317]	1052	echo "./${ExeNameOut} > out_${ExeNameOut}.out.\${OMPI_COMM_WORLD_RANK} 2>out_${ExeNameOut}.err.\${OMPI_COMM_WORLD_RANK}" >> script_${ExeNameOut}.ksh
[664]	1053	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
	1054	EXECUTION="${HOST_MPIRUN_COMMAND} ./script_${ExeNameOut}.ksh"
[1481]	1055	elif [ ${executionType} -eq 6 ] ; then
	1056	# Sequentiel case: this is the same as the first case.
	1057	echo "./${ExeNameOut} > out_${ExeNameOut}.out 2>out_${ExeNameOut}.err" >> script_${ExeNameOut}.ksh
	1058	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
	1059	EXECUTION="time ./script_${ExeNameOut}.ksh"
[664]	1060	else
[1480]	1061	# Strange case.
	1062	IGCM_debug_Exit "ERROR with OK_PARA_MPI"
	1063	IGCM_debug_Print 2 "sys Obelix : OK_PARA_MPI=False and mpiTask/openMPthreds different from 1"
	1064	IGCM_debug_Verif_Exit
[664]	1065	fi
[1002]	1066
	1067	IGCM_debug_Print 1 "sys Obelix : script_${ExeNameOut}.ksh contains"
	1068	cat script_${ExeNameOut}.ksh
	1069
[664]	1070	fi
[1180]	1071
[664]	1072	done
[657]	1073
	1074	fi
	1075
[811]	1076	IGCM_debug_Print 1 "sys Obelix : execution command is"
	1077	IGCM_debug_Print 1 "$EXECUTION"
[657]	1078
	1079	IGCM_debug_PopStack "IGCM_sys_build_execution_scripts"
[16]	1080	}
	1081
[933]	1082	#D-#==================================================
	1083	#D-function IGCM_sys_check_path
	1084	#D-* Purpose: check that RUN_DIR_PATH that will be removed on some machine
	1085	#D-* do not point to an important use directory. Stop immediately in that case.
	1086	#D-* Examples:
	1087	#D-
	1088	function IGCM_sys_check_path {
	1089	IGCM_debug_PushStack "IGCM_sys_check_path"
	1090	if ( $DEBUG_sys ) ; then
	1091	echo "IGCM_sys_check_path"
	1092	fi
[811]	1093
[933]	1094	if ( [ X${RUN_DIR_PATH} = X${HOME} ] \|\| [ X${RUN_DIR_PATH} = X${ARCHIVE} ] ) ; then
	1095	IGCM_debug_Print 1 "Variable RUN_DIR_PATH is pointing to an important directory : ${RUN_DIR_PATH}"
	1096	IGCM_debug_Print 1 "Please check the RUN_DIR_PATH definition in your Job : Job_${config_UserChoices_JobName}"
[936]	1097	IGCM_debug_Exit "This will stop the job"
[933]	1098	fi
	1099	IGCM_debug_PopStack "IGCM_sys_check_path"
	1100	}
	1101
[811]	1102	#D-#==================================================
	1103	#D-function IGCM_sys_check_quota. Dummy call here
	1104	#D-* Purpose: check user quota. Stop the simulation if quota above 90%
	1105	#D-* Examples:
	1106	#D-
[657]	1107	function IGCM_sys_check_quota {
	1108	IGCM_debug_PushStack "IGCM_sys_check_quota"
	1109	if ( $DEBUG_sys ) ; then
	1110	echo "IGCM_sys_check_quota"
	1111	fi
[1300]	1112
	1113	# TO BE IMPLEMENTED
	1114
[657]	1115	IGCM_debug_PopStack "IGCM_sys_check_quota"
	1116	}
	1117
[811]	1118	#D-#==================================================
[1286]	1119	#D-function IGCM_sys_projectAccounting
	1120	#D-* Purpose: store project accounting information in a file
	1121	#D-* Examples:
	1122	#D-
	1123	function IGCM_sys_projectAccounting {
	1124	IGCM_debug_PushStack "IGCM_sys_projectAccounting"
	1125	if ( $DEBUG_sys ) ; then
	1126	echo "IGCM_sys_check_quota"
	1127	fi
	1128
	1129	touch $1
	1130
	1131	IGCM_debug_PopStack "IGCM_sys_projectAccounting"
	1132	}
	1133
	1134	#D-#==================================================
[1244]	1135	#D-function IGCM_sys_getJobSchedulerID
	1136	#D-* Purpose: Get the job ID during execution
	1137	#D-* Examples: IGCM_sys_getJobSchedulerID jobSchedulerID
	1138	#D-
	1139	function IGCM_sys_getJobSchedulerID {
	1140	IGCM_debug_PushStack "IGCM_sys_getJobSchedulerID"
	1141	if ( $DEBUG_sys ) ; then
	1142	echo "IGCM_sys_getJobSchedulerID"
	1143	fi
	1144
	1145	eval ${1}=$( echo ${PBS_JOBID} \| awk -F. '{print $1}' )
	1146
	1147	IGCM_debug_PopStack "IGCM_sys_getJobSchedulerID"
	1148	}
	1149
	1150	#D-#==================================================
[1124]	1151	#D-function IGCM_sys_GetJobID
[1244]	1152	#D-* Purpose: Get the job ID from the JobName
[1124]	1153	#D-* Examples: IGCM_sys_GetJobID ${JobName} ${TargetUsr} JobID
	1154	#D-
	1155	function IGCM_sys_GetJobID {
	1156	IGCM_debug_PushStack "IGCM_sys_GetJobID"
	1157	if ( $DEBUG_sys ) ; then
	1158	echo "IGCM_sys_GetJobID"
	1159	fi
	1160
	1161	# With -f option, the full job name is given in the last column
[1300]	1162	ID="$( qstat -u $2 \| grep -w $1 \| gawk '-F ' '{print $10}' )"
[1124]	1163
	1164	eval ${3}=${ID}
	1165	IGCM_debug_PopStack "IGCM_sys_GetJobID"
	1166	}
	1167
	1168	#D-#==================================================
[811]	1169	#D-function IGCM_sys_CountJobInQueue
[1244]	1170	#D-* Purpose: Count number of users job
[811]	1171	#D-* Examples: IGCM_sys_CountJobInQueue ${JobName} NbRun
	1172	#D-
	1173	function IGCM_sys_CountJobInQueue {
	1174	IGCM_debug_PushStack "IGCM_sys_CountJobInQueue"
	1175	if ( $DEBUG_sys ) ; then
	1176	echo "IGCM_sys_CountJobInQueue"
	1177	fi
	1178	IGCM_debug_PopStack "IGCM_sys_CountJobInQueue"
	1179	}
[1272]	1180
	1181	#D-#==================================================
	1182	#D-function IGCM_sys_atlas
	1183	#D-* Purpose: encapsulate atlas call so as to manage error code and curie specificity
	1184	#D-* Examples:
	1185	#D-
	1186	function IGCM_sys_atlas {
	1187	IGCM_debug_PushStack "IGCM_sys_atlas" $@
	1188	if ( $DEBUG_sys ) ; then
	1189	echo "IGCM_sys_atlas :" $@
	1190	fi
	1191
	1192	typeset status
	1193
	1194	\atlas $@ > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
	1195	status=$?
	1196	if [ ${status} -gt 0 ] ; then
	1197	echo "IGCM_sys_atlas : error code ${status}"
	1198	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
	1199	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
	1200	IGCM_debug_PopStack "IGCM_sys_atlas"
	1201	return 1
	1202	else
	1203	IGCM_debug_PopStack "IGCM_sys_atlas"
	1204	return 0
	1205	fi
	1206
	1207	IGCM_debug_PopStack "IGCM_sys_atlas"
	1208	}
	1209
[1380]	1210
	1211	#D-#==================================================
	1212	#D-function IGCM_sys_rebuild_nemo
	1213	#D-* Purpose: rebuild nemo parallel files with general rebuild
	1214	#D-* Examples:
	1215	#D-
	1216
	1217	function IGCM_sys_rebuild_nemo {
	1218	IGCM_debug_PushStack "IGCM_sys_rebuild_nemo" $@
	1219	if ( $DEBUG_sys ) ; then
	1220	echo "IGCM_sys_rebuild_nemo :" $@
	1221	fi
	1222
	1223	nemo_generic_restart_file_name_out=${1}
	1224	nemo_extension_out=${3}
	1225	shift ; shift ; shift
	1226
	1227	IGCM_sys_rebuild -o ${nemo_generic_restart_file_name_out}.${nemo_extension_out} $@
	1228
	1229	IGCM_debug_PopStack "IGCM_sys_rebuild_nemo"
	1230	}
	1231

Note: See TracBrowser for help on using the repository browser.

Download in other formats: