Context Navigation

source: trunk/libIGCM/libIGCM_sys/libIGCM_sys_irene.ksh @ 1477

Last change on this file since 1477 was 1477, checked in by aclsce, 5 years ago
Updated nb of cores per node for irene machine.
Property licence set to The following licence information concerns ONLY the libIGCM tools ================================================================== Copyright © Centre National de la Recherche Scientifique CNRS Commissariat à l'Énergie Atomique CEA libIGCM : Library for Portable Models Computation of IGCM Group. IGCM Group is the french IPSL Global Climate Model Group. This library is a set of shell scripts and functions whose purpose is the management of the initialization, the launch, the transfer of output files, the post-processing and the monitoring of datas produce by any numerical program on any plateforme. This software is governed by the CeCILL license under French law and abiding by the rules of distribution of free software. You can use, modify and/ or redistribute the software under the terms of the CeCILL license as circulated by CEA, CNRS and INRIA at the following URL "http://www.cecill.info". As a counterpart to the access to the source code and rights to copy, modify and redistribute granted by the license, users are provided only with a limited warranty and the software's author, the holder of the economic rights, and the successive licensors have only limited liability. In this respect, the user's attention is drawn to the risks associated with loading, using, modifying and/or developing or reproducing the software by the user in light of its specific status of free software, that may mean that it is complicated to manipulate, and that also therefore means that it is reserved for developers and experienced professionals having in-depth computer knowledge. Users are therefore encouraged to load and test the software's suitability as regards their requirements in conditions enabling the security of their systems and/or data to be ensured and, more generally, to use and operate it in the same conditions as regards security. The fact that you are presently reading this means that you have had knowledge of the CeCILL license and that you accept its terms. Property svn:keywords set to `Revision Author Date`
File size: 63.1 KB

Line
1	#!/bin/ksh
2
3	#**************************************************************
4	# Author: Sebastien Denvil, Martial Mancip, Arnaud Caubel
5	# Contact: Arnaud.Caubel__at__lsce.ipsl.fr
6	# $Revision:: $ Revision of last commit
7	# $Author:: $ Author of last commit
8	# $Date:: $ Date of last commit
9	# IPSL (2006)
10	# This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
11	#
12	#**************************************************************
13
14	#=========================================================
15	# The documentation of this file can be automatically generated
16	# if you use the prefix #D- for comments to be extracted.
17	# Extract with command: cat lib* \| grep "^#D-" \| cut -c "4-"
18	#=========================================================
19
20	#D-#==================================================
21	#D-LibIGCM_sys for Irene
22	#D-#==================================================
23	#D-
24	#D- This ksh library if a layer under some usefull
25	#D-environment variables and shell commands.
26	#D-All those definitions depend on host particularities.
27	#D-It manages a stack mechanism and test validity of operations.
28	#D-All function described bellow must be prefixed by IGCM_sys.
29
30	#====================================================
31	# libIGCM_sys PARAMETERS
32	#====================================================
33
34	#====================================================
35	# set DEBUG_sys to true to output calls of function
36	typeset -r DEBUG_sys=${DEBUG_sys:=true}
37
38	#====================================================
39	# Turn in dry run mode ? (sys_Put_Rest, sys_Put_Out, sys_Get)
40	typeset -r DRYRUN=${DRYRUN:=0}
41
42	# YOU MUST COMPILE YOUR EXE FILES FOR DRYRUN MODE !
43	# -------------------------------------------------------------------------------------
44	# \| DRYRUN= \| Date computations, \| sys_Get \| Exe \| sys_Put_Out; sys_Put_Rest \|
45	# \| \| Cp/Exe/param/files \| \| \| \|
46	# \| \| Chmod Qsub \| \| \| \|
47	# -------------------------------------------------------------------------------------
48	# \| 0 \| yes \| yes \| yes \| yes \|
49	# -------------------------------------------------------------------------------------
50	# \| 1 \| yes \| yes \| yes \| no \|
51	# -------------------------------------------------------------------------------------
52	# \| 2 \| yes \| yes \| no \| no \|
53	# -------------------------------------------------------------------------------------
54	# \| 3 \| yes \| no \| no \| no \|
55	# -------------------------------------------------------------------------------------
56
57	#=====================================================
58	# Global Variables :
59	#=====================================================
60	# Language : "fr" or "en"
61	typeset -r MYLANG="fr"
62
63	#=====================================================
64	# Host user names project maxCpuTime
65	# $hostname ou hostname
66	typeset HOST=${HOST:=$( hostname )}
67	# $username ou whoami
68	typeset LOGIN=${LOGIN:=$( whoami )}
69	# $hostname of the MASTER job
70	typeset MASTER=irene
71	# add default project on irene
72	typeset PROJECT=$(echo ${BRIDGE_MSUB_PROJECT:=::default_project::} \| cut -d@ -f1 )
73	# jobWarningDelay in seconds
74	typeset jobWarningDelay=${BRIDGE_MSUB_MAXTIME}
75
76	#D-
77	#D-#==================================================
78	#D-Program used in libIGCM
79	#D-#==================================================
80
81	# Submit command
82	typeset SUBMIT=${SUBMIT:=ccc_msub}
83	# rsync with path
84	typeset -r RSYNC=/usr/bin/rsync
85	# RSYNC_opt args to rsync
86	typeset -r RSYNC_opt="-va"
87	# ie storage filesystem
88	typeset -r STOREHOST=${MASTER}
89	typeset -r REMOTE_RSYNC=/usr/bin/rsync
90
91	#====================================================
92	# Set environment tools (ferret, nco, cdo, rebuild, ...)
93	#====================================================
94	if [ X${TaskType} = Xcomputing ] ; then
95	IGCM_debug_Print 1 "Module will be loaded once we know the compiler version"
96	else
97	module purge > /dev/null 2>&1
98	##. /ccc/cont003/home/igcmg/igcmg/MachineEnvironment/irene/env_irene > /dev/null 2>&1
99	. $( ccc_home -u igcmg)/MachineEnvironment/irene/env_irene > /dev/null 2>&1
100	##. /ccc/cont003/home/igcmg/igcmg/MachineEnvironment/irene/env_atlas_irene > /dev/null 2>&1
101	. $( ccc_home -u igcmg)/MachineEnvironment/irene/env_atlas_irene > /dev/null 2>&1
102
103	export PCMDI_MP=/ccc/work/cont003/igcmg/igcmg/PCMDI-MP
104	export UVCDAT_ANONYMOUS_LOG=FALSE
105	fi
106
107	# FYI
108	[ ! X${TaskType} = Xchecking ] && IGCM_debug_Print 1 "List of loaded modules:"
109	[ ! X${TaskType} = Xchecking ] && module list
110
111	# For AddNoise and AddPertu3DOCE
112	##export PATH=${PATH}:/ccc/cont003/home/igcmg/igcmg/Tools/curie/bin
113	export PATH=${PATH}:$(ccc_home -u igcmg)/Tools/irene/bin
114
115	# For rebuild
116	##export PATH=${PATH}:/ccc/cont003/home/igcmg/igcmg/Tools/irene/rebuild/modipsl_IOIPSL_PLUS_v2_2_4/bin
117	export PATH=${PATH}:$(ccc_home -u igcmg)/Tools/irene/rebuild/modipsl_IOIPSL_PLUS_v2_2_4/bin
118
119	#====================================================
120	# Host specific DIRECTORIES
121	#====================================================
122
123	# ============ CESIUM START ============ #
124
125	#====================================================
126	#- Mirror libIGCM from titane to cesium if needed
127	#ROOTSYS=$( echo ${libIGCM} \| gawk -F"/" '{print $3}' )
128	#if [ ! ${ROOTSYS} = "home" ] ; then
129	# typeset -r MirrorlibIGCM=${MirrorlibIGCM:=true}
130	#else
131	# typeset -r MirrorlibIGCM=${MirrorlibIGCM:=false}
132	#fi
133
134	#====================================================
135	#- libIGCM_POST
136	#if ( ${MirrorlibIGCM} ) ; then
137	# PATHlibIGCM=$( echo ${libIGCM} \| gawk -F"${LOGIN}/" '{print $2}' \| sed -e "s&/libIGCM&&" )
138	# typeset -r libIGCM_POST=${HOME}/MIRROR/${PATHlibIGCM}/libIGCM
139	#else
140	# typeset -r libIGCM_POST=${libIGCM}
141	#fi
142
143	# ============ CESIUM END ============ #
144
145	#====================================================
146	#- MirrorlibIGCM for frontend
147	typeset -r MirrorlibIGCM=${MirrorlibIGCM:=false}
148
149	#====================================================
150	#- libIGCM_POST for frontend
151	typeset -r libIGCM_POST=${libIGCM}
152
153	#====================================================
154	#- R_EXE (==> BIN_DIR = ${MODIPSL}/bin )
155	typeset -r R_EXE="${MODIPSL}/bin"
156
157	#====================================================
158	#- SUBMIT_DIR : submission dir
159	typeset -x SUBMIT_DIR=${SUBMIT_DIR:=${BRIDGE_MSUB_PWD}}
160
161	#====================================================
162	#- IN
163	typeset -r R_IN=${R_IN:=/ccc/work/cont003/igcmg/igcmg/IGCM}
164
165	#====================================================
166	#- RUN_DIR_PATH : Temporary working directory (=> TMP)
167	typeset -r RUN_DIR_PATH=${RUN_DIR_PATH:=${CCCSCRATCHDIR}/RUN_DIR/${BRIDGE_MSUB_JOBID}_${$}}
168
169	#====================================================
170	#- OUTCOMMAND_PATH : tmp place to store command lines standard error and outputs
171	typeset -r OUTCOMMAND_PATH=/tmp
172
173	#====================================================
174	#- HOST_MPIRUN_COMMAND
175	typeset -r HOST_MPIRUN_COMMAND=${HOST_MPIRUN_COMMAND:="/usr/bin/time ccc_mprun -E-K1 "}
176
177	#====================================================
178	#- Max number of arguments passed to nco operator or demigration command
179	UNIX_MAX_LIMIT=360
180
181	#====================================================
182	#- set PackDefault to true on irene
183	PackDefault=true
184
185	#====================================================
186	#- Default number of MPI task for IPSL coupled model
187	#- required for backward compatibility
188	#-
189	DEFAULT_NUM_PROC_OCE=5
190	DEFAULT_NUM_PROC_CPL=1
191	(( DEFAULT_NUM_PROC_ATM = BATCH_NUM_PROC_TOT - DEFAULT_NUM_PROC_OCE - DEFAULT_NUM_PROC_CPL ))
192
193	#====================================================
194	#- Number of computing cores per node
195	#-
196	NB_CORE_PER_NODE=48
197
198	#D-#==================================================
199	#D-function IGCM_sys_defineArchives
200	#D-* Purpose:
201	#D-* Load dfldatadir module : According to project used for submission (default) or set in config.card (optional)
202	#D-* Define ARCHIVE : Dedicated to large files
203	#D-* Define STORAGE : Dedicated to small/medium files
204	#D-* Define R_OUT : Output tree located on ARCHIVE
205	#D-* Define R_FIG : Output tree located on STORAGE hosting figures (monitoring and atlas, and/or small files)
206	#D-* Define R_BUF : Output tree located on CCCSCRATCHDIR hosting files waiting for rebuild or pack processes
207	#D-* if SpaceName=TEST everything is stored on CCCSCRATCHDIR
208	#D-* Examples:
209	#D-
210	function IGCM_sys_defineArchives {
211	IGCM_debug_PushStack "IGCM_sys_defineArchives"
212
213	# Load dfldatadir depending on the project used for submission (default) or set in config.card (optional)
214	if [ X${config_UserChoices_DataProject} = X ] \|\| [ X${config_UserChoices_DataProject} = DEFAULT ]; then
215	# Default option: Change the dfldatadir according to the project used for submission
216	# The variable DataProject in section UserChoices in config.card is not set or is set to DEFAULT
217	if [ X"$( echo ${PROJECT} \| grep cmip6 )" != "X" ] ; then
218	module switch dfldatadir dfldatadir/gencmip6
219	IGCM_debug_Print 1 "Load dfldatadir for forced gencmip6 project"
220	else
221	module switch dfldatadir dfldatadir/${PROJECT}
222	IGCM_debug_Print 1 "Load dfldatadir for project ${PROJECT}"
223	fi
224	else
225	# Use the project set by the variable DataProject in section UserChoices in config.card
226	module switch dfldatadir dfldatadir/${config_UserChoices_DataProject}
227	IGCM_debug_Print 1 "Load dfldatadir for DataProject red in config.card : ${config_UserChoices_DataProject}"
228	fi
229
230	#====================================================
231	#- RUN_DIR_PATH : Temporary working directory (=> TMP)
232	typeset -r RUN_DIR_PATH=${RUN_DIR_PATH:=${CCCSCRATCHDIR}/RUN_DIR/${BRIDGE_MSUB_JOBID}_${$}}
233
234	if [ ! X${config_UserChoices_ARCHIVE} = X ]; then
235	#====================================================
236	#- ARCHIVE (dedicated to large files)
237	ARCHIVE=${config_UserChoices_ARCHIVE}
238	else
239	#====================================================
240	#- ARCHIVE (dedicated to large files)
241	ARCHIVE=${CCCSTOREDIR}
242	fi
243
244	if [ ! X${config_UserChoices_STORAGE} = X ]; then
245	#====================================================
246	#- STORAGE (dedicated to small/medium files)
247	STORAGE=${config_UserChoices_STORAGE}
248	else
249	#====================================================
250	#- STORAGE (dedicated to small/medium files)
251	STORAGE=${CCCWORKDIR}
252	fi
253
254	if [ X${config_UserChoices_SpaceName} = XTEST ]; then
255	#====================================================
256	#- R_OUT
257	R_OUT=${CCCSCRATCHDIR}/IGCM_OUT
258
259	#====================================================
260	#- R_FIG (hosting figures : monitoring and atlas, and/or small files)
261	R_FIG=${CCCSCRATCHDIR}/IGCM_OUT
262
263	IGCM_debug_Print 1 "SpaceName=TEST ==> OVERRULE destination path directories"
264
265	else
266	#====================================================
267	#- R_OUT
268	R_OUT=${ARCHIVE}/IGCM_OUT
269
270	#====================================================
271	#- R_FIG (hosting figures : monitoring and atlas, and/or small files)
272	R_FIG=${STORAGE}/IGCM_OUT
273	fi
274
275	#====================================================
276	#- CMIP6 (hosting CMIP6 files produced by XIOS2 and configured by dr2xml)
277	CMIP6_BUF=${STORAGE}/IGCM_OUT
278
279	#====================================================
280	#- R_BUF (ONLY FOR double copy an scratch)
281	R_BUF=${CCCSCRATCHDIR}/IGCM_OUT
282
283	IGCM_debug_Print 1 "R_OUT has been defined = ${R_OUT}"
284	IGCM_debug_Print 1 "R_BUF has been defined = ${R_BUF}"
285	IGCM_debug_Print 1 "R_FIG has been defined = ${R_FIG}"
286
287	IGCM_debug_PopStack "IGCM_sys_defineArchives"
288	}
289
290	#D-#==================================================
291	#D-function IGCM_sys_RshArchive
292	#D-* Purpose: Archive rsh command
293	#D-* Examples:
294	#D-
295	function IGCM_sys_RshArchive {
296	IGCM_debug_PushStack "IGCM_sys_RshArchive" $@
297	/bin/ksh <<-EOF
298	${@}
299	EOF
300	status=$?
301	if [ ${status} -gt 0 ] ; then
302	IGCM_debug_Print 2 "IGCM_sys_RshArchive : command failed error code ${status}"
303	IGCM_debug_Exit "IGCM_sys_RshArchive"
304	fi
305	IGCM_debug_PopStack "IGCM_sys_RshArchive"
306	}
307
308	#D-#==================================================
309	#D-function IGCM_sys_RshArchive_NoError
310	#D-* Purpose: Archive rsh command, without error
311	#D-* used only in monitoring.job
312	#D-* Examples:
313	#D-
314	function IGCM_sys_RshArchive_NoError {
315	IGCM_debug_PushStack "IGCM_sys_RshArchive_NoError" $@
316	/bin/ksh <<-EOF
317	${@} 2> /dev/null
318	EOF
319	IGCM_debug_PopStack "IGCM_sys_RshArchive_NoError"
320	}
321
322	#D-#==================================================
323	#D-function IGCM_sys_MkdirArchive
324	#D-* Purpose: Mkdir on Archive
325	#D-* Examples:
326	#D-
327	function IGCM_sys_MkdirArchive {
328	IGCM_debug_PushStack "IGCM_sys_MkdirArchive" $@
329	if ( $DEBUG_sys ) ; then
330	echo "IGCM_sys_MkdirArchive :" $@
331	fi
332	#- creation de repertoire sur le serveur fichier
333	if [ ! -d ${1} ]; then
334	\mkdir -p $1
335	status=$?
336
337	if [ ${status} -gt 0 ] ; then
338	IGCM_debug_Print 2 "IGCM_sys_MkdirArchive : mkdir failed error code ${status}"
339	IGCM_debug_Exit "IGCM_sys_MkdirArchive"
340	fi
341	fi
342	IGCM_debug_PopStack "IGCM_sys_MkdirArchive"
343	}
344
345	#D-#==================================================
346	#D-function IGCM_sys_TestDirArchive
347	#D-* Purpose: Test Directory that must exists on Archive
348	#D-* Examples:
349	#D-
350	function IGCM_sys_TestDirArchive {
351	IGCM_debug_PushStack "IGCM_sys_TestDirArchive" $@
352	if ( $DEBUG_sys ) ; then
353	echo "IGCM_sys_TestDirArchive :" $@
354	fi
355	typeset ExistFlag
356	ExistFlag=$( [ -d $1 ] && echo 0 \|\| echo 1 )
357	IGCM_debug_PopStack "IGCM_sys_TestDirArchive"
358	return ${ExistFlag}
359	}
360
361	#D-#==================================================
362	#D-function IGCM_sys_IsFileArchived
363	#D-* Purpose: Test file that must NOT EXISTS on Archive based on filename only
364	#D-* Examples:
365	#D-
366	function IGCM_sys_IsFileArchived {
367	IGCM_debug_PushStack "IGCM_sys_IsFileArchived" $@
368	if ( $DEBUG_sys ) ; then
369	echo "IGCM_sys_IsFileArchived :" $@
370	fi
371	typeset IsArchivedFlag
372	IsArchivedFlag=$( [ X$( echo $@ \| grep ^\/ccc\/store ) != X ] && echo 0 \|\| echo 1 )
373	IGCM_debug_PopStack "IGCM_sys_IsFileArchived"
374
375	return ${IsArchivedFlag}
376	}
377
378	#D-#==================================================
379	#D-function IGCM_sys_TestFileArchive
380	#D-* Purpose: Test file that must NOT EXISTS on Archive
381	#D-* Examples:
382	#D-
383	function IGCM_sys_TestFileArchive {
384	IGCM_debug_PushStack "IGCM_sys_TestFileArchive" $@
385	typeset ExistFlag
386	ExistFlag=$( [ -f $1 ] && echo 0 \|\| echo 1 )
387	IGCM_debug_PopStack "IGCM_sys_TestFileArchive"
388
389	return ${ExistFlag}
390	}
391
392	#D-#==================================================
393	#D-function IGCM_sys_CountFileArchive
394	#D-* Purpose: Count files on Archive filesystem
395	#D-* Examples:
396	#D-
397	function IGCM_sys_CountFileArchive {
398	IGCM_debug_PushStack "IGCM_sys_CountFileArchive" $@
399	ls ${@} 2>/dev/null \| wc -l
400	if [ $? -gt 0 ] ; then
401	echo "IGCM_sys_CountFileArchive : erreur."
402	fi
403	IGCM_debug_PopStack "IGCM_sys_CountFileArchive"
404	}
405
406	#D-#==================================================
407	#D-function IGCM_sys_Tree
408	#D-* Purpose: Tree directories with files on ${ARCHIVE}
409	#D-* Examples: IGCM_sys_Tree ${R_IN} ${R_OUT}
410	#D-
411	function IGCM_sys_Tree {
412	IGCM_debug_PushStack "IGCM_sys_Tree" $@
413	if ( $DEBUG_sys ) ; then
414	echo "IGCM_sys_Tree :" $@
415	fi
416
417	\ls -lR ${@}
418
419	IGCM_debug_PopStack "IGCM_sys_Tree"
420	}
421
422	#D-#==================================================
423	#D-function IGCM_sys_Qsub
424	#D-* Purpose: Qsub new job
425	#D-* Examples:
426	#D-
427	function IGCM_sys_Qsub {
428	IGCM_debug_PushStack "IGCM_sys_Qsub" $@
429	if ( $DEBUG_sys ) ; then
430	echo "IGCM_sys_Qsub :" $@
431	fi
432	typeset options status
433	options="-o ${SUBMIT_DIR}/${Script_Output} -e ${SUBMIT_DIR}/${Script_Output}"
434
435	/usr/bin/ccc_msub ${options} $1 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
436	status=$?
437
438	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
439	if [ ${status} -gt 0 ] ; then
440	IGCM_debug_Print 2 "IGCM_sys_Qsub ${options} $1 : error code ${status}"
441	IGCM_debug_Exit "IGCM_sys_Qsub"
442	else
443	JobID=$( gawk {'print $4'} ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ )
444	fi
445	IGCM_debug_PopStack "IGCM_sys_Qsub"
446	}
447
448	#D-#==================================================
449	#D-function IGCM_sys_QsubPost
450	#D-* Purpose: Qsub new job on scalaire
451	#D-* Examples:
452	#D-
453	function IGCM_sys_QsubPost {
454	IGCM_debug_PushStack "IGCM_sys_QsubPost" $@
455	if ( $DEBUG_sys ) ; then
456	echo "IGCM_sys_QsubPost :" $@
457	fi
458	typeset options status
459	options="-Q normal -o ${POST_DIR}/${Script_Post_Output}.out -e ${POST_DIR}/${Script_Post_Output}.out"
460
461	/usr/bin/ccc_msub ${options} ${libIGCM_POST}/$1.job > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
462	status=$?
463
464	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
465	if [ ${status} -gt 0 ] ; then
466	IGCM_debug_Print 2 "IGCM_sys_QsubPost ${options} ${libIGCM_POST}/$1.job : error code ${status}"
467	IGCM_debug_Exit "IGCM_sys_QsubPost"
468	else
469	JobID=$( gawk {'print $4'} ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ )
470	fi
471	IGCM_debug_PopStack "IGCM_sys_QsubPost"
472	}
473
474	#D-*************************
475	#D- File transfer functions
476	#D-*************************
477	#D-
478
479	#D-#==================================================
480	#D-function IGCM_sys_RmRunDir
481	#D-* Purpose: rm tmpdir (dummy function most of the time batch
482	#D- scheduler will do the job)
483	#D-* Examples:
484	#D-
485	function IGCM_sys_RmRunDir {
486	IGCM_debug_PushStack "IGCM_sys_RmRunDir" $@
487	if ( $DEBUG_sys ) ; then
488	echo "IGCM_sys_RmRunDir :" $@
489	fi
490
491	typeset status
492
493	echo rm $@ > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
494	\rm $@ >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
495	status=$?
496
497	if [ ${status} -gt 0 ] ; then
498	IGCM_debug_Print 1 "IGCM_sys_RmRunDir : rm error code is ${status}."
499	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
500	IGCM_debug_Exit "IGCM_sys_RmRunDir"
501	else
502	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
503	fi
504	IGCM_debug_PopStack "IGCM_sys_RmRunDir"
505	}
506
507	#D-#==================================================
508	#D-function IGCM_sys_Put_Dir
509	#D-* Purpose: Copy a complete directory on $(ARCHIVE)
510	#D-* Examples:
511	#D-
512	function IGCM_sys_Put_Dir {
513	IGCM_debug_PushStack "IGCM_sys_Put_Dir" $@
514	if ( $DEBUG_sys ) ; then
515	echo "IGCM_sys_Put_Dir :" $@
516	fi
517	if [ $DRYRUN = 0 ]; then
518	if [ ! -d ${1} ] ; then
519	echo "WARNING : IGCM_sys_Put_Dir ${1} DOES NOT EXIST ."
520	IGCM_debug_PopStack "IGCM_sys_Put_Dir"
521	return
522	fi
523
524	typeset status
525
526	# Only if we use rsync
527	#IGCM_sys_TestDirArchive $( dirname $2 )
528	#
529	#USUAL WAY
530	\cp -r $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
531	status=$?
532
533	if [ ${status} -gt 0 ] ; then
534	IGCM_debug_Print 2 "IGCM_sys_Put_Dir : cp failed error code ${status}"
535	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
536	IGCM_debug_Exit "IGCM_sys_Put_Dir"
537	else
538	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
539	fi
540	fi
541	IGCM_debug_PopStack "IGCM_sys_Put_Dir"
542	}
543
544	#D-#==================================================
545	#D-function IGCM_sys_Get_Dir
546	#D-* Purpose: Copy a complete directory from ${ARCHIVE}
547	#D-* Examples:
548	#D-
549	function IGCM_sys_Get_Dir {
550	IGCM_debug_PushStack "IGCM_sys_Get_Dir" $@
551	if ( $DEBUG_sys ) ; then
552	echo "IGCM_sys_Get_Dir :" $@
553	fi
554	if [ $DRYRUN = 0 ]; then
555	typeset NB_ESSAI DELAI status i
556	# number of tentative
557	NB_ESSAI=3
558	# time delay between tentative
559	DELAI=2
560
561	#
562	# USUAL WAY
563	# add 'ccc_hsm get' (to demigrate all offline files) to reduce time of this command :
564	ccc_hsm get -r $1
565
566	i=0
567	while [ $i -lt $NB_ESSAI ] ; do
568	\cp -ur $1 $2 >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
569	status=$?
570	if [ ${status} -gt 0 ] ; then
571	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : cp failed error code ${status} ${i}/${NB_ESSAI}"
572	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : sleep ${DELAI} seconds and try again."
573	sleep $DELAI
574	else
575	break
576	fi
577	(( i = i + 1 ))
578	done
579
580	if [ ${status} -gt 0 ] ; then
581	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : cp failed error code ${status}"
582	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
583	IGCM_debug_Exit "IGCM_sys_Get_Dir"
584	else
585	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
586	fi
587	fi
588	IGCM_debug_PopStack "IGCM_sys_Get_Dir"
589	}
590
591	#D-#==================================================
592	#D-function IGCM_sys_Put_Rest
593	#D-* Purpose: Put computied restarts on ${ARCHIVE}.
594	#D- File and target directory must exist.
595	#D-* Examples:
596	#D-
597	function IGCM_sys_Put_Rest {
598	IGCM_debug_PushStack "IGCM_sys_Put_Rest" $@
599	if ( $DEBUG_sys ) ; then
600	echo "IGCM_sys_Put_Rest :" $@
601	fi
602	if [ $DRYRUN = 0 ]; then
603	if [ ! -f ${1} ] ; then
604	echo "ERROR : IGCM_sys_Put_Rest ${1} DOES NOT EXIST ."
605	IGCM_debug_Exit "IGCM_sys_Put_Rest"
606	fi
607
608	typeset status
609	#
610	# USUAL WAY
611	\cp $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
612	status=$?
613
614	# #RSYNC WITH NETWORK SSH CALL
615	# echo ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
616	# ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
617
618	# #RSYNC WITH NFS USE
619	# echo ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
620	# ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
621
622	# status=$?
623	# IGCM_sys_Rsync_out $status
624
625	# ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
626	# (( status=status+$? ))
627
628	if [ ${status} -gt 0 ] ; then
629	IGCM_debug_Print 2 "IGCM_sys_Put_Rest : cp failed error code ${status}"
630	[ -f ${1} ] && ls -l ${1}
631	[ -f ${2} ] && ls -l ${2}
632	[ -f ${2}/${1} ] && ls -l ${2}/${1}
633	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
634	IGCM_debug_Exit "IGCM_sys_Put_Rest"
635	else
636
637	if [ X${JobType} = XRUN ] ; then
638	[ -f ${2} ] && IGCM_sys_Chmod 444 ${2}
639	[ -f ${2}/${1} ] && IGCM_sys_Chmod 444 ${2}/${1}
640	fi
641
642	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
643	fi
644	fi
645	IGCM_debug_PopStack "IGCM_sys_Put_Rest"
646	}
647
648	#D-#==================================================
649	#D-function IGCM_sys_Put_Out
650	#D-* Purpose: Copy a file on ${ARCHIVE} after having chmod it in readonly
651	#D-* Examples:
652	#D-
653	function IGCM_sys_Put_Out {
654	IGCM_debug_PushStack "IGCM_sys_Put_Out" $@
655	if ( $DEBUG_sys ) ; then
656	echo "IGCM_sys_Put_Out :" $@
657	fi
658
659	typeset NB_ESSAI DELAI status i exist skip
660	typeset fileDeviceNumberInHex directoryDeviceNumberInHex
661
662	# number of tentative
663	NB_ESSAI=3
664	# time delay between tentative
665	DELAI=2
666
667	if [ $DRYRUN = 0 ]; then
668	if [ ! -f ${1} ] ; then
669	echo "WARNING : IGCM_sys_Put_Out ${1} DOES NOT EXIST ."
670	IGCM_debug_PopStack "IGCM_sys_Put_Out"
671	return 1
672	fi
673	#
674	IGCM_sys_MkdirArchive $( dirname $2 )
675	#
676	exist=false
677	skip=false
678	if [ -f $2 ] ; then
679	IGCM_debug_Print 1 "$2 already exist"
680	ccc_hsm get $2
681	exist=true
682	if [ "X$( diff $1 $2 )" = X ] ; then
683	IGCM_debug_Print 2 "$1 and $2 are the same file, we skip the copy"
684	skip=true
685	else
686	IGCM_debug_Print 2 "$1 and $2 are not the same file, we force the copy"
687	skip=false
688	fi
689	fi
690	#
691	if ( [ X${exist} = Xtrue ] && [ X${skip} = Xfalse ] ) ; then
692	IGCM_sys_Chmod u+w $2
693	fi
694
695	if [ X${skip} = Xfalse ] ; then
696	i=0
697	while [ $i -lt $NB_ESSAI ] ; do
698	# Identify file system
699	fileDeviceNumberInHex=$( stat -c %d $1 )
700	status=$?
701	if [ ${status} -gt 0 ] ; then
702	IGCM_debug_Exit "IGCM_sys_Put_Out"
703	fi
704	# Identify file system
705	directoryDeviceNumberInHex=$( stat -c %d $( dirname $2 ) )
706	status=$?
707	if [ ${status} -gt 0 ] ; then
708	IGCM_debug_Exit "IGCM_sys_Put_Out"
709	fi
710
711	if ( [ ${fileDeviceNumberInHex} -ne ${directoryDeviceNumberInHex} ] \|\| [ X$3 = XNOMOVE ] ) ; then
712	# They are not on the same device. USUAL WAY
713	\cp $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
714	status=$?
715	else
716	# They are on the same device. NOT SO USUAL WAY
717	\mv $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
718	status=$?
719	fi
720	if [ ${status} -gt 0 ]; then
721	IGCM_debug_Print 2 "IGCM_sys_Put_Out : cp failed error code ${status} ${i}/${NB_ESSAI}"
722	IGCM_debug_Print 2 "IGCM_sys_Put_Out : sleep ${DELAI} seconds and try again."
723	[ -f ${1} ] && ls -l ${1}
724	[ -f ${2} ] && ls -l ${2}
725	[ -f ${2}/${1} ] && ls -l ${2}/${1}
726	sleep $DELAI
727	else
728	break
729	fi
730	(( i = i + 1 ))
731	done
732	fi
733
734	# #RSYNC WITH NETWORK SSH CALL
735	# echo ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
736	# ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
737
738	# #RSYNC WITH NFS USE
739	# echo ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
740	# ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
741
742	# status=$?
743	# IGCM_sys_Rsync_out $status
744
745	# ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
746	# (( status=status+$? ))
747
748	if [ ${status} -gt 0 ] ; then
749	IGCM_debug_Print 2 "IGCM_sys_Put_Out : cp failed error code ${status}"
750	[ -f ${1} ] && ls -l ${1}
751	[ -f ${2} ] && ls -l ${2}
752	[ -f ${2}/${1} ] && ls -l ${2}/${1}
753	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
754	IGCM_debug_Exit "IGCM_sys_Put_Out"
755	else
756
757	if [ X${JobType} = XRUN ] ; then
758	if [ X${3} = X ] ; then
759	[ -f ${2} ] && IGCM_sys_Chmod 444 ${2}
760	[ -f ${2}/${1} ] && IGCM_sys_Chmod 444 ${2}/${1}
761	fi
762	fi
763
764	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
765	fi
766	fi
767	IGCM_debug_PopStack "IGCM_sys_Put_Out"
768	return 0
769	}
770
771	#D-#==================================================
772	#D-function IGCM_sys_Get
773	#D-* Purpose: Get a file from ${ARCHIVE}
774	#D-* Examples: IGCM_sys_Get myfile /destpath/myfile_with_PREFIX
775	#D- IGCM_sys_Get /l Array_contain_myfiles /destpath/
776	function IGCM_sys_Get {
777	IGCM_debug_PushStack "IGCM_sys_Get" $@
778
779	typeset DEST dm_liste target file_work
780	typeset NB_ESSAI DELAI status i
781
782	if ( $DEBUG_sys ) ; then
783	echo "IGCM_sys_Get :" $@
784	fi
785
786	# number of tentative
787	NB_ESSAI=3
788	# time delay between tentative
789	DELAI=2
790
791	if [ $DRYRUN -le 2 ]; then
792	if [ X${1} = X'/l' ] ; then
793	eval set +A dm_liste \${${2}}
794	else
795	eval set +A dm_liste ${1}
796	fi
797	eval DEST=\${${#}}
798	ccc_hsm get ${dm_liste[*]} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
799	status=$?
800	if [ ${status} -gt 0 ] ; then
801	echo "WARNING IGCM_sys_Get : error code ${status}"
802	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
803	echo "WARNING IGCM_sys_Get : will stop later if the cp fails."
804	fi
805
806	#if [ ${status} -gt 0 ] ; then
807	# if [ ! "X$( grep "Lost dmusrcmd connection" ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ )" = "X" ] ; then
808	# cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
809	# echo "WARNING IGCM_sys_Get : Lost dmusrcmd connection : "
810	# sleep 30
811	# echo "We try another time"
812	## dmget ${dm_liste[*]} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
813	# ccc_hsm get ${dm_liste[*]} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
814	# status=$?
815	# if [ ${status} -gt 0 ] ; then
816	# echo "ERROR IGCM_sys_Get : again demigration error :"
817	# cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
818	# IGCM_debug_Exit "IGCM_sys_Get"
819	# fi
820	# else
821	# echo "ERROR IGCM_sys_Get : demigration error :"
822	# cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
823	# IGCM_debug_Exit "IGCM_sys_Get"
824	# fi
825	#fi
826
827	# #RSYNC WITH NETWORK SSH CALL
828	# echo ${RSYNC} ${RSYNC_opt} -e ssh ${STOREHOST}:"${dm_liste}" ${STOREHOST}:${RUN_DIR}/${DEST} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
829	# ${RSYNC} ${RSYNC_opt} -e ssh ${STOREHOST}:"${dm_liste}" ${STOREHOST}:${RUN_DIR}/${DEST} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
830
831	# #RSYNC WITH NFS USE
832	# echo ${RSYNC} ${RSYNC_opt} ${dm_liste} ${RUN_DIR}/${DEST} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
833	# ${RSYNC} ${RSYNC_opt} ${dm_liste} ${RUN_DIR}/${DEST} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
834
835	# status=$?
836	# IGCM_sys_Rsync_out $status
837
838	# ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
839	# (( status=status+$? ))
840
841	#USUAL WAY
842	if [ X${1} = X'/l' ] ; then
843	for target in ${dm_liste[*]} ; do
844	local_file=$( basename ${target} )
845	# test if the target file is present before the loop
846	IGCM_sys_TestFileArchive ${target}
847	status=$?
848	if [ ${status} -gt 0 ] ; then
849	echo "IGCM_sys_Get, ERROR : regular file ${target} DOES NOT EXIST ."
850	IGCM_debug_Exit "IGCM_sys_Get"
851	else
852	i=0
853	while [ $i -lt $NB_ESSAI ] ; do
854	#if [ X${DoLink} = Xtrue ] ; then
855	# \ln -s ${target} ${DEST}/${local_file} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
856	# status=$?
857	# else
858	# \cp ${target} ${DEST}/${local_file} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
859	# status=$?
860	#fi
861	\ln -s ${target} ${DEST}/${local_file} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
862	status=$?
863	if [ ${status} -gt 0 ]; then
864	IGCM_debug_Print 2 "IGCM_sys_Get : cp failed error code ${status} ${i}/${NB_ESSAI}"
865	IGCM_debug_Print 2 "IGCM_sys_Get : sleep ${DELAI} seconds and try again."
866	sleep $DELAI
867	else
868	break
869	fi
870	(( i = i + 1 ))
871	done
872	if [ ${status} -gt 0 ] ; then
873	echo "IGCM_sys_Get : error"
874	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
875	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
876	IGCM_debug_Exit "IGCM_sys_Get"
877	else
878	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
879	fi
880	fi
881	done
882	else
883	i=0
884	while [ $i -lt $NB_ESSAI ] ; do
885	\cp ${dm_liste} ${DEST} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
886	status=$?
887	if [ ${status} -gt 0 ]; then
888	IGCM_debug_Print 2 "IGCM_sys_Get : cp failed error code ${status} ${i}/${NB_ESSAI}"
889	IGCM_debug_Print 2 "IGCM_sys_Get : sleep ${DELAI} seconds and try again."
890	sleep $DELAI
891	else
892	break
893	fi
894	(( i = i + 1 ))
895	done
896	if [ ${status} -gt 0 ] ; then
897	echo "IGCM_sys_Get : error"
898	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
899	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
900	IGCM_debug_Exit "IGCM_sys_Get"
901	else
902	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
903	fi
904	fi
905	fi
906	IGCM_debug_PopStack "IGCM_sys_Get"
907	}
908
909	#D-#==================================================
910	#D-function IGCM_sys_GetDate_Monitoring
911	#D-* Purpose: get the last year for which the monitoring has been computed
912	#D-* Examples:
913	#D-
914	function IGCM_sys_GetDate_Monitoring {
915	IGCM_debug_PushStack "IGCM_sys_GetDate_Monitoring" $@
916	if ( $DEBUG_sys ) ; then
917	echo "IGCM_sys_GetDate_Monitoring :" $@
918	fi
919
920	eval ${2}=$( cdo showyear ${1} 2> /dev/null \| gawk '{ print $NF }' )
921
922	IGCM_debug_PopStack "IGCM_sys_GetDate_Monitoring"
923	}
924
925	#D-#==================================================
926	#D-function IGCM_sys_Dods_Rm
927	#D-* Purpose: DO NOTHING ! Put ${ARCHIVE} files on DODS internet protocole.
928	#D-* Examples:
929	#D-
930	function IGCM_sys_Dods_Rm {
931	if ( $DEBUG_sys ) ; then
932	echo "IGCM_sys_Dods_Rm :" $@
933	fi
934	typeset status
935	if [ $DRYRUN = 0 ]; then
936
937	# if [ ! -d /dmnfs/cont003/dods/public/${LOGIN}/${R_DODS}/${1} ] ; then
938	# echo "WARNING : IGCM_sys_Dods_Rm /dmnfs/cont003/dods/public/${LOGIN}/${R_DODS}/${1} DOES NOT EXIST ."
939	# echo "Nothing has been done."
940	# return
941	# fi
942
943	if [ "$#" -eq 1 ]; then
944	## /ccc/cont003/home/igcmg/igcmg/Tools/irene/thredds_rm public/${LOGIN}/${R_DODS}/${1} # > out_dods_rm 2>&1
945	$(ccc_home -u igcmg)/Tools/irene/thredds_rm public/${LOGIN}/${R_DODS}/${1} # > out_dods_rm 2>&1
946	status=$?
947	else
948	## /ccc/cont003/home/igcmg/igcmg/Tools/irene/thredds_rm public/${LOGIN}/${R_DODS} # > out_dods_rm 2>&1
949	$(ccc_home -u igcmg)/Tools/irene/thredds_rm public/${LOGIN}/${R_DODS} # > out_dods_rm 2>&1
950	status=$?
951	fi
952
953	# if [ ${status} -gt 0 ] ; then
954	# echo "IGCM_sys_Dods_Rm : error."
955	# cat out_dods_rm
956	# IGCM_debug_Exit "IGCM_sys_Dods_Rm"
957	# else
958	# rm out_dods_rm
959	# fi
960
961	fi
962	return $status
963	}
964
965	#D-#==================================================
966	#D-function IGCM_sys_Dods_Cp
967	#D-* Purpose: Copy $(ARCHIVE) files on DODS internet protocole.
968	#D-* Examples:
969	#D-
970	function IGCM_sys_Dods_Cp {
971	if ( $DEBUG_sys ) ; then
972	echo "IGCM_sys_Dods_Cp :" $@
973	fi
974	typeset status
975	if [ $DRYRUN = 0 ]; then
976
977	# if [ ! -d ${R_SAVE}/${1} ] ; then
978	# echo "WARNING : IGCM_sys_Dods_Cp ${R_SAVE}/${1} DOES NOT EXIST ."
979	# echo "Nothing has been done."
980	# return
981	# fi
982
983	## /ccc/cont003/home/igcmg/igcmg/Tools/irene/bin/thredds_cp ${1} public/${LOGIN}/${R_DODS} # > out_dods_cp 2>&1
984	$(ccc_home -u igcmg)/Tools/irene/thredds_cp ${1} public/${LOGIN}/${R_DODS} # > out_dods_cp 2>&1
985	status=$?
986
987	# if [ ${status} -gt 0 ] ; then
988	# echo "IGCM_sys_Dods_Cp : error."
989	# cat out_dods_cp
990	# IGCM_debug_Exit "IGCM_sys_Dods_Cp"
991	# else
992	# rm out_dods_cp
993	# fi
994
995	fi
996	return $status
997	}
998
999	#D-#==================================================
1000	#D-function IGCM_sys_Put_Dods
1001	#D-* Purpose: Put ${ARCHIVE} files on DODS internet protocole.
1002	#D-* Examples:
1003	#D-
1004	function IGCM_sys_Put_Dods {
1005	IGCM_debug_PushStack "IGCM_sys_Put_Dods" $@
1006	if ( $DEBUG_sys ) ; then
1007	echo "IGCM_sys_Put_Dods :" $@
1008	fi
1009	typeset status
1010	if [ $DRYRUN = 0 ]; then
1011	if ( [ ! -d ${R_SAVE}/${1} ] && [ ! -d ${R_FIGR}/${1} ] ) ; then
1012	echo "WARNING IGCM_sys_Put_Dods : None of the following directories exist. Exactly one should."
1013	echo "WARNING IGCM_sys_Put_Dods : ${R_SAVE}/${1} DOES NOT EXIST."
1014	echo "WARNING IGCM_sys_Put_Dods : ${R_FIGR}/${1} DOES NOT EXIST."
1015	IGCM_debug_PopStack "IGCM_sys_Put_Dods"
1016	return
1017	fi
1018
1019	if ( [ -d ${R_SAVE}/${1} ] && [ -d ${R_FIGR}/${1} ] ) ; then
1020	echo "WARNING IGCM_sys_Put_Dods : Both of the following directories exist. Exactly one should."
1021	echo "WARNING IGCM_sys_Put_Dods : ${R_SAVE}/${1} EXISTS."
1022	echo "WARNING IGCM_sys_Put_Dods : ${R_FIGR}/${1} EXISTS."
1023	IGCM_debug_PopStack "IGCM_sys_Put_Dods"
1024	return
1025	fi
1026	#
1027	if [ -d ${R_SAVE}/${1} ] ; then
1028	cd ${R_SAVE}
1029	elif [ -d ${R_FIGR}/${1} ] ; then
1030	cd ${R_FIGR}
1031	fi
1032
1033	IGCM_sys_Dods_Rm ${1}
1034	IGCM_sys_Dods_Cp ${1}
1035	status=0
1036
1037	if [ ${status} -gt 0 ] ; then
1038	echo "IGCM_sys_Put_Dods : error."
1039	IGCM_debug_Exit "IGCM_sys_Put_Dods"
1040	fi
1041	fi
1042	IGCM_debug_PopStack "IGCM_sys_Put_Dods"
1043	}
1044
1045	##############################################################
1046	# REBUILD OPERATOR
1047
1048	#D-#==================================================
1049	#D-function IGCM_sys_sync
1050	#D-* Purpose: flush buffer on disk
1051	#D-* Examples:
1052	#D-
1053	function IGCM_sys_sync {
1054	IGCM_debug_PushStack "IGCM_sys_sync" $@
1055	if ( $DEBUG_sys ) ; then
1056	echo "IGCM_sys_sync :" $@
1057	fi
1058
1059	/bin/sync
1060
1061	IGCM_debug_PopStack "IGCM_sys_sync"
1062	}
1063
1064	############################################################
1065	# Activate Running Environnment Variables
1066
1067	#D-#==================================================
1068	#D-function IGCM_sys_activ_variables
1069	#D-* Purpose: set environement variables prior to execution
1070	#D-* Examples:
1071	#D-
1072	function IGCM_sys_activ_variables {
1073	IGCM_debug_PushStack "IGCM_sys_activ_variables"
1074	if ( $DEBUG_sys ) ; then
1075	echo "IGCM_sys_activ_variables"
1076	fi
1077
1078	# --------------------------------------------------------------------
1079	#D- MODULE specifications
1080	# --------------------------------------------------------------------
1081	module purge > /dev/null 2>&1
1082	if [ X${compilerVersion} = XV17 ] ; then
1083	IGCM_debug_Print 1 "IGCM_sys_active_variables : Using compiler version V17"
1084	else
1085	IGCM_debug_Exit "This compiler version is not yet supported : ${compilerVersion}"
1086	fi
1087	##. /ccc/cont003/home/igcmg/igcmg/MachineEnvironment/irene/env_irene > /dev/null 2>&1
1088	. $( ccc_home -u igcmg)/MachineEnvironment/irene/env_irene > /dev/null 2>&1
1089	IGCM_debug_Print 1 "IGCM_sys_active_variables : Now loaded modules for Irene. "
1090	module list
1091
1092	# Keep track of this
1093	[ ! X${TaskType} = Xchecking ] && IGCM_debug_Print 1 "List of loaded modules:"
1094	[ ! X${TaskType} = Xchecking ] && module list
1095
1096	# --------------------------------------------------------------------
1097	#D- MPI specifications
1098	# --------------------------------------------------------------------
1099
1100	# --------------------------------------------------------------------
1101	#D- Other specifications
1102	# --------------------------------------------------------------------
1103
1104	ulimit -s unlimited
1105
1106	IGCM_debug_PopStack "IGCM_sys_activ_variables"
1107	}
1108
1109	############################################################
1110	# Desactivate Running Environnment Variables
1111
1112	#D-#==================================================
1113	#D-function IGCM_sys_desactiv_variables
1114	#D-* Purpose: unset environement variables after execution
1115	#D-* Examples:
1116	#D-
1117	function IGCM_sys_desactiv_variables {
1118	IGCM_debug_PushStack "IGCM_sys_desactiv_variables"
1119	if ( $DEBUG_sys ) ; then
1120	echo "IGCM_sys_desactiv_variables"
1121	fi
1122	# --------------------------------------------------------------------
1123	#D- MPI specifications
1124	# --------------------------------------------------------------------
1125
1126	# --------------------------------------------------------------------
1127	#D- Other specifications
1128	# --------------------------------------------------------------------
1129
1130	IGCM_debug_PopStack "IGCM_sys_desactiv_variables"
1131	}
1132
1133	############################################################
1134	# Update job headers to be used by the scheduler
1135
1136	#D-#==================================================
1137	#D-function IGCM_sys_updateHeaders
1138	#D-* Purpose: Update job headers to be used by the scheduler
1139	#D-* Examples: IGCM_sys_updateHeaders /path/to/Job_MYEXP
1140	#D-
1141	function IGCM_sys_updateHeaders {
1142	IGCM_debug_PushStack "IGCM_sys_updateHeaders"
1143	if ( $DEBUG_sys ) ; then
1144	echo "IGCM_sys_updateHeaders"
1145	fi
1146	typeset file
1147	file=$1
1148
1149	if [ ${executionType} -eq 1 ] ; then
1150	# MPMD + MPI
1151	sed -e "/::openMPthreads::/d" \
1152	-e "s/::JobNumProcTot::/${coreNumber}/" \
1153	-e "/#MSUB -x/d" \
1154	-e "/--cpu_bind=none/d" \
1155	${file} > ${file}.tmp
1156
1157	elif [ ${executionType} -eq 2 ] ; then
1158	# MPMD + MPI + OMP : mpirun/ccc_mprun/error
1159	if ( [ "X${config_UserChoices_ExecutionModeOnCurie}" = "Xmpirun" ] ) ; then
1160	sed -e "/::openMPthreads::/d" \
1161	-e "s/::JobNumProcTot::/${coreNumber}/" \
1162	${file} > ${file}.tmp
1163	elif ( [ "X${config_UserChoices_ExecutionModeOnCurie}" = "X" ] \|\| [ "X${config_UserChoices_ExecutionModeOnCurie}" = "Xccc_mprun" ] ) ; then
1164	sed -e "/::openMPthreads::/d" \
1165	-e "s/::JobNumProcTot::/${coreNumber}/" \
1166	-e "/--cpu_bind=none/d" \
1167	${file} > ${file}.tmp
1168	else
1169	IGCM_debug_Print 1 "You have to set ExecutionModeOnCurie=ccc_mprun or mpirun in config.card"
1170	IGCM_debug_Exit "IGCM_sys_updateHeaders"
1171	fi
1172	elif [ ${executionType} -eq 3 ] ; then
1173	# SPMD + MPI/OMP
1174	sed -e "s/::openMPthreads::/${openMPthreads}/" \
1175	-e "s/::JobNumProcTot::/${mpiTasks}/" \
1176	-e "/#MSUB -x/d" \
1177	-e "/--cpu_bind=none/d" \
1178	${file} > ${file}.tmp
1179
1180	elif [ ${executionType} -eq 4 ] ; then
1181	# SPMD + MPI only
1182	sed -e "s/::JobNumProcTot::/${mpiTasks}/" \
1183	-e "/::openMPthreads::/d" \
1184	-e "/#MSUB -x/d" \
1185	-e "/--cpu_bind=none/d" \
1186	${file} > ${file}.tmp
1187
1188	elif [ ${executionType} -eq 5 ] ; then
1189	# SPMD + OMP only
1190	sed -e "s/::openMPthreads::/${openMPthreads}/" \
1191	-e "/::JobNumProcTot::/d" \
1192	-e "/#MSUB -x/d" \
1193	-e "/--cpu_bind=none/d" \
1194	${file} > ${file}.tmp
1195
1196	elif [ ${executionType} -eq 6 ] ; then
1197	# SEQUENTIAL THEN
1198	sed -e "s/::JobNumProcTot::/1/" \
1199	-e "/::openMPthreads::/d" \
1200	-e "/#MSUB -x/d" \
1201	-e "/--cpu_bind=none/d" \
1202	${file} > ${file}.tmp
1203
1204	fi
1205
1206	IGCM_sys_Mv ${file}.tmp ${file}
1207
1208	IGCM_debug_PopStack "IGCM_sys_updateHeaders"
1209	}
1210
1211	############################################################
1212	# Build MPI/OMP scripts run file (dummy function)
1213
1214	#D-#==================================================
1215	#D-function IGCM_sys_build_run_file
1216	#D-* Purpose: build run file (deprecated)
1217	#D-* Examples:
1218	#D-
1219	function IGCM_sys_build_run_file {
1220	IGCM_debug_Print 3 " dummy function : IGCM_sys_build_run_file "
1221	}
1222
1223	############################################################
1224	# Build MPI/OMP scripts
1225
1226	#D-#==================================================
1227	#D-function IGCM_sys_build_execution_scripts
1228	#D-* Purpose: build execution scripts to be launch by ${HOST_MPIRUN_COMMAND}
1229	#D-* Examples:
1230	#D-
1231	function IGCM_sys_build_execution_scripts
1232	{
1233	IGCM_debug_PushStack "IGCM_sys_build_execution_scripts" $@
1234	if ( $DEBUG_sys ) ; then
1235	echo "IGCM_sys_build_execution_scripts " $@
1236	fi
1237
1238	EXECUTION=${HOST_MPIRUN_COMMAND}
1239
1240	if ( ${OK_PARA_MPMD} ) ; then
1241
1242	# MPMD mode
1243	# 1 MPI only
1244	# 2 MPI/OpenMP mpirun method
1245	# 3 MPI/OpenMP ccc_mprun method
1246
1247	if [ -f run_file ] ; then
1248	IGCM_sys_Rm -f run_file
1249	fi
1250	touch run_file
1251
1252	# case 1 : Only MPI (MPMD)
1253	if ( ! ${OK_PARA_OMP} ) ; then
1254
1255	# Build run_file
1256
1257	# First loop on the components for the coupler ie oasis (only if oasis3)
1258	# the coupler ie oasis3 must be the first one
1259	for comp in ${config_ListOfComponents[*]} ; do
1260
1261	eval ExeNameIn=\${config_Executable_${comp}[0]}
1262	eval ExeNameOut=\${config_Executable_${comp}[1]}
1263
1264	# for CPL component only
1265	if [ "X${comp}" = "XCPL" ] && [ "X${ExeNameOut}" != X\"\" ] ; then
1266
1267	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1268	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
1269	echo "${comp_proc_mpi_loc} ./${ExeNameOut}" >> run_file
1270	fi
1271	done
1272
1273	# Then second loop on the components
1274	for comp in ${config_ListOfComponents[*]} ; do
1275
1276	eval ExeNameIn=\${config_Executable_${comp}[0]}
1277	eval ExeNameOut=\${config_Executable_${comp}[1]}
1278
1279	# Only if we really have an executable for the component and not the coupler ie oasis:
1280	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${comp}" != "XCPL" ] ) ; then
1281
1282	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1283	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
1284	echo "${comp_proc_mpi_loc} ./${ExeNameOut}" >> run_file
1285	fi
1286	done
1287
1288	EXECUTION="${HOST_MPIRUN_COMMAND} -f ./run_file"
1289
1290	IGCM_sys_Chmod u+x run_file
1291	if ( $DEBUG_sys ) ; then
1292	echo "run_file contains : "
1293	cat run_file
1294	fi
1295
1296	elif ( [ "X${config_UserChoices_ExecutionModeOnCurie}" = "Xmpirun" ] ) ; then
1297
1298	# 2 MPI/OpenMP mpirun method
1299
1300	# Use of mpirun instead of ccc_mprun
1301	EXECUTION="time mpirun"
1302
1303	# Hosts treatment
1304	${EXECUTION} hostname \| sort \| uniq > hosts.tmp
1305
1306	i=0
1307	rm -f hosts rankfile
1308	IGCM_debug_Print 1 "sys Irene, Hosts available :"
1309	for nodes in `cat hosts.tmp`
1310	do
1311	host[$i]=$nodes
1312	echo "${host[$i]}" >> hosts
1313	IGCM_debug_Print 1 ${host[$i]}
1314	i=$((i+1))
1315	done
1316	rm -f hosts.tmp
1317
1318	listnodes=${host[*]}
1319
1320	EXECUTION="${EXECUTION} -hostfile hosts -rankfile rankfile"
1321
1322	# Initialisation
1323	rank=0
1324	current_core=0
1325	init_exec=n
1326
1327	# Loop on the components
1328	for comp in ${config_ListOfComponents[*]} ; do
1329
1330	eval ExeNameIn=\${config_Executable_${comp}[0]}
1331	eval ExeNameOut=\${config_Executable_${comp}[1]}
1332
1333	# Not possible if oasis has an executable (i.e old version of oasis3)
1334	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${comp}" = "XCPL" ] ) ; then
1335	IGCM_debug_Exit "ERROR MPMD with hybrid MPI-OpenMP is not available with oasis3 version"
1336	IGCM_debug_Print 2 "Only available with oasis3-MCT version coupler"
1337	IGCM_debug_Verif_Exit
1338	fi
1339
1340	# Only if we really have an executable for the component :
1341	if [ "X${ExeNameOut}" != X\"\" ] ; then
1342
1343	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1344	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
1345	eval comp_proc_nod_loc=\${${comp}_PROC_NOD}
1346
1347	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
1348	echo "" >> script_${ExeNameOut}.ksh
1349	if [ ${comp_proc_omp_loc} -gt 1 ] ; then
1350
1351	# Check if the number of threads is correct
1352	case ${comp_proc_omp_loc} in
1353	2\|4\|8\|16)
1354	IGCM_debug_Print 1 "You run ${ExeNameOut} on ${comp_proc_omp_loc} OMP threads"
1355	;;
1356	*)
1357	IGCM_debug_Exit "ERROR with OMP parameters !"
1358	IGCM_debug_Print 2 "${comp_proc_omp_loc} is not possible as number of OMP threads"
1359	IGCM_debug_Print 2 "Only 2,4,8,16 as number of OMP threads are possible "
1360	IGCM_debug_Verif_Exit
1361	;;
1362	esac
1363	echo "export KMP_STACKSIZE=3g" >> script_${ExeNameOut}.ksh
1364	echo "export KMP_LIBRARY=turnaround" >> script_${ExeNameOut}.ksh
1365	echo "export MKL_SERIAL=YES" >> script_${ExeNameOut}.ksh
1366	echo "OMP_NUM_THREADS=${comp_proc_omp_loc}" >> script_${ExeNameOut}.ksh
1367	fi
1368
1369	#echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
1370	echo "(( MYMPIRANK = OMPI_COMM_WORLD_RANK )) " >> script_${ExeNameOut}.ksh
1371	echo "MYMPIRANK=\$(printf '%3.3d\n' \${MYMPIRANK})" >> script_${ExeNameOut}.ksh
1372	echo "./${ExeNameOut} > out_${ExeNameOut}.out.\${MYMPIRANK} 2>out_${ExeNameOut}.err.\${MYMPIRANK}" >> script_${ExeNameOut}.ksh
1373	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
1374
1375	if [ ${init_exec} = y ] ; then
1376	EXECUTION="${EXECUTION} : -np ${comp_proc_mpi_loc} ./script_${ExeNameOut}.ksh"
1377	else
1378	EXECUTION="${EXECUTION} -np ${comp_proc_mpi_loc} ./script_${ExeNameOut}.ksh"
1379	init_exec=y
1380	fi
1381
1382	# Build rankfile : method used to assign cores and nodes for the MPI process
1383	# Ex :
1384	#rank 0=curie5296 slot=0,1,2,3
1385	#rank 1=curie5296 slot=4,5,6,7
1386	# Example of final command :
1387	# mpirun -hostfile hosts -rankfile rankfile -np 27 ./script_lmdz.x.ksh : -np 5 ./script_opa.xx.ksh
1388	# with script_lmdz.x.ksh :
1389	# #!/bin/ksh
1390	#export KMP_STACKSIZE=3g
1391	#export KMP_LIBRARY=turnaround
1392	#export MKL_SERIAL=YES
1393	#OMP_NUM_THREADS=4
1394	#./lmdz.x
1395	if [ ${comp_proc_nod_loc} -gt 1 ] ; then
1396	(( offset_comp_proc_loc = NB_CORE_PER_NODE / (comp_proc_mpi_loc / comp_proc_nod_loc) ))
1397	else
1398	(( offset_comp_proc_loc = comp_proc_omp_loc ))
1399	fi
1400	for nb_proc_mpi in `seq 0 $(($comp_proc_mpi_loc-1))`; do
1401	(( index_host = current_core / NB_CORE_PER_NODE ))
1402	host_value=${host[${index_host}]}
1403	(( slot = current_core % NB_CORE_PER_NODE ))
1404	virg=","
1405	string_final=""
1406	for index in `seq $slot $(($slot+$comp_proc_omp_loc-1))`; do
1407	string=$index$virg
1408	string_final=$string_final$string
1409	done
1410	string_final=$( echo $string_final \| sed "s/.$//" )
1411	echo "rank $rank=$host_value slot=$string_final" >> rankfile
1412	(( rank = rank + 1 ))
1413	(( current_core = current_core + offset_comp_proc_loc ))
1414	done
1415	fi
1416	done
1417
1418	elif ( [ "X${config_UserChoices_ExecutionModeOnCurie}" = "X" ] \|\| [ "X${config_UserChoices_ExecutionModeOnCurie}" = "Xccc_mprun" ] ) ; then
1419
1420	# 3 MPI/OpenMP ccc_mprun method
1421
1422	# MPI-OpenMP (MPMD)
1423	# example of run_file
1424	# 71-8 env OMP_NUM_THREADS=8 ./script_lmdz.x.ksh
1425	# 480-1 env OMP_NUM_THREADS=1 ./script_opa.xx.ksh
1426	# 1-1 env OMP_NUM_THREADS=1 ./script_xios.x.ksh
1427
1428	# Loop on the components
1429	for comp in ${config_ListOfComponents[*]} ; do
1430
1431	eval ExeNameIn=\${config_Executable_${comp}[0]}
1432	eval ExeNameOut=\${config_Executable_${comp}[1]}
1433
1434	# Not possible if oasis has an executable (i.e old version of oasis3)
1435	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${comp}" = "XCPL" ] ) ; then
1436	IGCM_debug_Exit "ERROR MPMD with hybrid MPI-OpenMP is not available with oasis3 version"
1437	IGCM_debug_Print 2 "Only available with oasis3-MCT version coupler"
1438	IGCM_debug_Verif_Exit
1439	fi
1440
1441	# Only if we really have an executable for the component :
1442	if [ "X${ExeNameOut}" != X\"\" ] ; then
1443
1444	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1445	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
1446
1447	# Build script files
1448
1449	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
1450	echo "" >> script_${ExeNameOut}.ksh
1451	if [ ${comp_proc_omp_loc} -gt 1 ] ; then
1452
1453	# Check if the number of threads is correct
1454	case ${comp_proc_omp_loc} in
1455	2\|4\|8\|12\|16\|24)
1456	IGCM_debug_Print 1 "You run ${ExeNameOut} on ${comp_proc_omp_loc} OMP threads"
1457	;;
1458	*)
1459	IGCM_debug_Exit "ERROR with OMP parameters !"
1460	IGCM_debug_Print 2 "${comp_proc_omp_loc} is not possible as number of OMP threads"
1461	IGCM_debug_Print 2 "Only 2,4,8,12,16,24 as number of OMP threads are possible "
1462	IGCM_debug_Verif_Exit
1463	;;
1464	esac
1465	### only for LMDZ?
1466	echo "export KMP_STACKSIZE=3g" >> script_${ExeNameOut}.ksh
1467	echo "export KMP_LIBRARY=turnaround" >> script_${ExeNameOut}.ksh
1468	echo "export MKL_SERIAL=YES" >> script_${ExeNameOut}.ksh
1469	echo "OMP_NUM_THREADS=${comp_proc_omp_loc}" >> script_${ExeNameOut}.ksh
1470
1471	fi
1472
1473	# to have out/err per process on different files
1474	echo "./${ExeNameOut} > out_${ExeNameOut}.out.\${SLURM_PROCID} 2>out_${ExeNameOut}.err.\${SLURM_PROCID}" >> script_${ExeNameOut}.ksh
1475	#echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
1476
1477	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
1478
1479	# Complete run_file
1480
1481	echo "${comp_proc_mpi_loc}-${comp_proc_omp_loc} env OMP_NUM_THREADS=${comp_proc_omp_loc} ./script_${ExeNameOut}.ksh " >>run_file
1482
1483	fi
1484	done
1485
1486	## variable added to stop after 60s instead of 600s by default.
1487	## This is used when no error comes from executables and when something stopped an executable without notice.
1488	export SLURM_WAIT=60
1489
1490	## module advised by TGCC (instead of 2 variables)
1491	module load feature/bridge/heterogenous_mpmd
1492
1493	# set EXECUTION for ccc_mprun case (similar to #1 : MPI only MPMD method)
1494
1495	EXECUTION="${HOST_MPIRUN_COMMAND} -f ./run_file"
1496
1497	IGCM_sys_Chmod u+x run_file
1498	if ( $DEBUG_sys ) ; then
1499	echo "run_file contains : "
1500	cat run_file
1501	fi
1502
1503	else
1504
1505	IGCM_debug_Print 1 "sys Curie : choose mpirun or ccc_mprun in config.card for ExecutionModeOnCurie"
1506
1507	fi # 1 2 or 3 if ${OK_PARA_MPMD}
1508
1509	else
1510	# Only one executable (SPMD mode).
1511
1512	for comp in ${config_ListOfComponents[*]} ; do
1513
1514	# Only if we really have an executable for the component :
1515	eval ExeNameOut=\${config_Executable_${comp}[1]}
1516	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${ExeNameOut}" != "Xinca.dat" ] ) ; then
1517
1518	# Build script files
1519
1520	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
1521	echo "" >> script_${ExeNameOut}.ksh
1522	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
1523
1524	if ( ${OK_PARA_OMP} ) ; then
1525	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
1526	echo "" >> script_${ExeNameOut}.ksh
1527	echo "export KMP_STACKSIZE=3g" >> script_${ExeNameOut}.ksh
1528	echo "export KMP_LIBRARY=turnaround" >> script_${ExeNameOut}.ksh
1529	echo "export MKL_SERIAL=YES" >> script_${ExeNameOut}.ksh
1530	echo "OMP_NUM_THREADS=${comp_proc_omp_loc}" >> script_${ExeNameOut}.ksh
1531	fi
1532
1533	if ( ${OK_PARA_MPI} ) ; then
1534	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1535	# Default : ccc_mprun used if nb_proc gt 1
1536	# to have out/err per process on different files
1537	echo "./${ExeNameOut} > out_${ExeNameOut}.out.\${SLURM_PROCID} 2>out_${ExeNameOut}.err.\${SLURM_PROCID}" >> script_${ExeNameOut}.ksh
1538	#echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
1539	EXECUTION="${HOST_MPIRUN_COMMAND} -n ${comp_proc_mpi_loc} ./script_${ExeNameOut}.ksh"
1540	else
1541	# Default : ccc_mprun is NOT used if nb_proc eq 1
1542	# to have out/err per process on different files
1543	echo "./${ExeNameOut} > out_${ExeNameOut}.out 2>out_${ExeNameOut}.err" >> script_${ExeNameOut}.ksh
1544	#echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
1545	EXECUTION="/usr/bin/time ./script_${ExeNameOut}.ksh"
1546	fi
1547
1548	IGCM_debug_Print 1 "sys Irene : script_${ExeNameOut}.ksh contains"
1549	cat script_${ExeNameOut}.ksh
1550
1551	fi
1552
1553	done
1554
1555	fi # ${OK_PARA_MPMD}
1556
1557	IGCM_debug_Print 1 "sys Irene : execution command is "
1558	IGCM_debug_Print 1 "$EXECUTION"
1559
1560	IGCM_debug_PopStack "IGCM_sys_build_execution_scripts"
1561	}
1562
1563	#D-#==================================================
1564	#D-function IGCM_sys_check_path
1565	#D-* Purpose: check that RUN_DIR_PATH that will be removed on some machine
1566	#D-* do not point to an important use directory. Stop immediately in that case.
1567	#D-* Examples:
1568	#D-
1569	function IGCM_sys_check_path {
1570	IGCM_debug_PushStack "IGCM_sys_check_path"
1571	if ( $DEBUG_sys ) ; then
1572	echo "IGCM_sys_check_path"
1573	fi
1574
1575	if ( [ X${RUN_DIR_PATH} = X${HOME} ] \|\| [ X${RUN_DIR_PATH} = X${CCCWORKDIR} ] \|\| [ X${RUN_DIR_PATH} = X${CCCSCRATCHDIR} ] \|\| [ X${RUN_DIR_PATH} = X${CCCSTOREDIR} ] ) ; then
1576	IGCM_debug_Print 1 "Variable RUN_DIR_PATH is pointing to an important directory : ${RUN_DIR_PATH}"
1577	IGCM_debug_Print 1 "Please check the RUN_DIR_PATH definition in your Job : Job_${config_UserChoices_JobName}"
1578	IGCM_debug_Exit "This will stop the job"
1579	fi
1580	IGCM_debug_PopStack "IGCM_sys_check_path"
1581	}
1582
1583	#D-#==================================================
1584	#D-function IGCM_sys_check_quota
1585	#D-* Purpose: check user quota. Stop the simulation if quota above 90%
1586	#D-* Examples:
1587	#D-
1588	function IGCM_sys_check_quota {
1589	IGCM_debug_PushStack "IGCM_sys_check_quota"
1590	if ( $DEBUG_sys ) ; then
1591	echo "IGCM_sys_check_quota"
1592	fi
1593	# Limit of quota (in %)
1594	limit_quota=90
1595
1596	# Check of the volume
1597	volume_quota=$(ccc_quota \| grep ' scratch' \| gawk '{print $2}')
1598	volume_avail=$(ccc_quota \| grep ' scratch' \| gawk '{print $3}')
1599
1600	if ( [ ! X${volume_quota} = X ] && [ ! ${volume_quota} = "-" ] ) ; then
1601
1602	unit_avail=${volume_avail: -1}
1603	unit_quota=${volume_quota: -1}
1604
1605	if [ "${unit_quota}" = "*" ] ; then
1606	IGCM_debug_Print 1 "Please, check your quota of volume on scratch"
1607	IGCM_debug_Print 1 "More than 100% of your quota is used"
1608	IGCM_debug_Print 1 "Use the ccc_quota command to check"
1609	IGCM_debug_Print 1 "You must have more than 10% available to run"
1610	IGCM_debug_Exit "Not enough space to run ! STOP HERE"
1611	IGCM_debug_Verif_Exit
1612	fi
1613
1614	temp_avail=${volume_avail%%${volume_avail: -1}*}
1615	temp_quota=${volume_quota%%${volume_quota: -1}*}
1616
1617	if [ ! ${unit_avail} = ${unit_quota} ] ; then
1618
1619	# Convertion
1620	if [ ${volume_avail: -1} = "T" ] ; then
1621	(( temp_avail = temp_avail * 1000000000000 ))
1622	elif [ ${volume_avail: -1} = "G" ] ; then
1623	(( temp_avail = temp_avail * 1000000000 ))
1624	elif [ ${volume_avail: -1} = "M" ] ; then
1625	(( temp_avail = temp_avail * 1000000 ))
1626	elif [ ${volume_avail: -1} = "k" ] ; then
1627	(( temp_avail = temp_avail * 1000 ))
1628	else
1629	(( temp_avail = volume_avail ))
1630	fi
1631	if [ ${volume_quota: -1} = "T" ] ; then
1632	(( temp_quota = temp_quota * 1000000000000 ))
1633	elif [ ${volume_quota: -1} = "G" ] ; then
1634	(( temp_quota = temp_quota * 1000000000 ))
1635	elif [ ${volume_quota: -1} = "M" ] ; then
1636	(( temp_quota = temp_quota * 1000000 ))
1637	elif [ ${volume_quota: -1} = "k" ] ; then
1638	(( temp_quota = temp_quota * 1000 ))
1639	else
1640	(( temp_quota = volume_quota ))
1641	fi
1642	fi
1643
1644	quota_volume=$(echo "scale=2 ; $temp_quota/$temp_avail*100" \| bc)
1645	# echo "volume ratio is " $quota_volume
1646
1647	if [ ${quota_volume} -ge ${limit_quota} ] ; then
1648	IGCM_debug_Print 1 "Please, check your quota of volume on scratch"
1649	IGCM_debug_Print 1 "${quota_volume}% of your quota is used"
1650	IGCM_debug_Print 1 "Use the ccc_quota command to check"
1651	IGCM_debug_Print 1 "You must have more than 10% available to run"
1652	IGCM_debug_Exit "Not enough space to run ! STOP HERE"
1653	IGCM_debug_Verif_Exit
1654	fi
1655
1656	fi
1657
1658	# Check of the number of inodes
1659
1660	inode_quota=$(ccc_quota \| grep ' scratch' \| gawk '{print $6}')
1661	inode_avail=$(ccc_quota \| grep ' scratch' \| gawk '{print $7}')
1662
1663	if ( [ ! X${inode_quota} = X ] && [ ! ${inode_quota} = "-" ] ) ; then
1664
1665	unit_avail=${inode_avail: -1}
1666	unit_quota=${inode_quota: -1}
1667
1668	if [ "${unit_quota}" = "*" ] ; then
1669	IGCM_debug_Print 1 "Please, check your quota of inode on scratch"
1670	IGCM_debug_Print 1 "More than 100% of your quota is used"
1671	IGCM_debug_Print 1 "Use the ccc_quota command to check"
1672	IGCM_debug_Print 1 "You must have more than 10% available to run"
1673	IGCM_debug_Exit "Not enough space to run ! STOP HERE"
1674	IGCM_debug_Verif_Exit
1675	fi
1676
1677	temp_avail=${inode_avail%%${inode_avail: -1}*}
1678	temp_quota=${inode_quota%%${inode_quota: -1}*}
1679
1680	if [ ! ${unit_avail} = ${unit_quota} ] ; then
1681
1682	# Convertion
1683	if [ ${inode_avail: -1} = "T" ] ; then
1684	(( temp_avail = temp_avail * 1000000000000 ))
1685	elif [ ${inode_avail: -1} = "G" ] ; then
1686	(( temp_avail = temp_avail * 1000000000 ))
1687	elif [ ${inode_avail: -1} = "M" ] ; then
1688	(( temp_avail = temp_avail * 1000000 ))
1689	elif [ ${inode_avail: -1} = "k" ] ; then
1690	(( temp_avail = temp_avail * 1000 ))
1691	else
1692	(( temp_avail = inode_avail ))
1693	fi
1694
1695	if [ ${inode_quota: -1} = "T" ] ; then
1696	(( temp_quota = temp_quota * 1000000000000 ))
1697	elif [ ${inode_quota: -1} = "G" ] ; then
1698	(( temp_quota = temp_quota * 1000000000 ))
1699	elif [ ${inode_quota: -1} = "M" ] ; then
1700	(( temp_quota = temp_quota * 1000000 ))
1701	elif [ ${inode_quota: -1} = "k" ] ; then
1702	(( temp_quota = temp_quota * 1000 ))
1703	else
1704	(( temp_quota = inode_quota ))
1705	fi
1706	fi
1707	quota_inode=$(echo "scale=2 ; $temp_quota/$temp_avail*100" \| bc)
1708	# echo "inode ratio is " $quota_inode
1709
1710	if [ ${quota_inode} -ge ${limit_quota} ] ; then
1711	IGCM_debug_Print 1 "Please, check your quota of inode on scratch"
1712	IGCM_debug_Print 1 "${quota_inode}% of your quota is used"
1713	IGCM_debug_Print 1 "Use the ccc_quota command to check"
1714	IGCM_debug_Print 1 "You must have more than 10% available to run"
1715	IGCM_debug_Exit "Not enough space to run ! STOP HERE"
1716	IGCM_debug_Verif_Exit
1717	fi
1718	fi
1719	IGCM_debug_PopStack "IGCM_sys_check_quota"
1720	}
1721
1722	#D-#==================================================
1723	#D-function IGCM_sys_projectAccounting
1724	#D-* Purpose: store project accounting information in a file
1725	#D-* Examples:
1726	#D-
1727	function IGCM_sys_projectAccounting {
1728	IGCM_debug_PushStack "IGCM_sys_projectAccounting"
1729	if ( $DEBUG_sys ) ; then
1730	echo "IGCM_sys_check_quota"
1731	fi
1732
1733	ssh irene191 /usr/bin/ccc_myproject > $1
1734
1735	IGCM_debug_PopStack "IGCM_sys_projectAccounting"
1736	}
1737
1738	#D-#==================================================
1739	#D-function IGCM_sys_getJobSchedulerID
1740	#D-* Purpose: Get the job ID during execution
1741	#D-* Examples: IGCM_sys_getJobSchedulerID jobSchedulerID
1742	#D-
1743	function IGCM_sys_getJobSchedulerID {
1744	IGCM_debug_PushStack "IGCM_sys_getJobSchedulerID"
1745	if ( $DEBUG_sys ) ; then
1746	echo "IGCM_sys_getJobSchedulerID"
1747	fi
1748
1749	eval ${1}=${BRIDGE_MSUB_JOBID}
1750
1751	IGCM_debug_PopStack "IGCM_sys_getJobSchedulerID"
1752	}
1753
1754	#D-#==================================================
1755	#D-function IGCM_sys_GetJobID
1756	#D-* Purpose: Get the job ID from the JobName
1757	#D-* Examples: IGCM_sys_GetJobID ${JobName} ${TargetUsr} JobID
1758	#D-
1759	function IGCM_sys_GetJobID {
1760	IGCM_debug_PushStack "IGCM_sys_GetJobID"
1761	if ( $DEBUG_sys ) ; then
1762	echo "IGCM_sys_GetJobID"
1763	fi
1764
1765	# With -f option, the full job name is given in the last column
1766	ID=$( ccc_mstat -f -u $2 \| \
1767	gawk -v JobName=$1 '( $NF ~ JobName ) { print $1 }' )
1768
1769	eval ${3}=${ID}
1770	IGCM_debug_PopStack "IGCM_sys_GetJobID"
1771	}
1772
1773	#D-#==================================================
1774	#D-function IGCM_sys_CountJobInQueue
1775	#D-* Purpose: Count number of users job
1776	#D-* Examples: IGCM_sys_CountJobInQueue ${JobName} NbRun
1777	#D-
1778	function IGCM_sys_CountJobInQueue {
1779	IGCM_debug_PushStack "IGCM_sys_CountJobInQueue"
1780	if ( $DEBUG_sys ) ; then
1781	echo "IGCM_sys_CountJobInQueue"
1782	fi
1783
1784	# With -f option, the full job name is given in the last column
1785	NbRun=$( ccc_mstat -f \| gawk -v JobName=$1 'BEGIN { x=0 } ( $NF ~ JobName ) { x=x+1 } END { print x }' )
1786
1787	eval ${2}=${NbRun}
1788
1789	IGCM_debug_PopStack "IGCM_sys_CountJobInQueue"
1790	}
1791
1792	#D-#==================================================
1793	#D-function IGCM_sys_ListJobInQueue
1794	#D-* Purpose: Produce a list of users computing jobs (excluding post-processing)
1795	#D-* Examples: IGCM_sys_ListJobInQueue ${User} JobNameList
1796	#D-
1797	function IGCM_sys_ListJobInQueue {
1798	IGCM_debug_PushStack "IGCM_sys_ListJobInQueue"
1799	if ( $DEBUG_sys ) ; then
1800	echo "IGCM_sys_ListJobInQueue"
1801	fi
1802
1803	# With -f option, the full job name is given in the last column
1804	set -A JobList $( ccc_mstat -f \| gawk -v User=$1 \
1805	'( $2 == User && \
1806	$NF != /TS/ && \
1807	$NF !~ /PACK/ && \
1808	$NF !~ /REBUILD/ && \
1809	$NF !~ /pack/ ) \
1810	{ print $NF }' \| sed -e "s/\(.\)\.[0-9]/\1/" )
1811
1812	eval set -A ${2} ${JobList[*]}
1813
1814	IGCM_debug_PopStack "IGCM_sys_ListJobInQueue"
1815	}
1816
1817	#D-#==================================================
1818	#D-function IGCM_sys_atlas
1819	#D-* Purpose: encapsulate atlas call so as to manage error code and curie specificity
1820	#D-* Examples:
1821	#D-
1822	function IGCM_sys_atlas {
1823	IGCM_debug_PushStack "IGCM_sys_atlas" $@
1824	if ( $DEBUG_sys ) ; then
1825	echo "IGCM_sys_atlas :" $@
1826	fi
1827
1828	typeset status
1829
1830	\ccc_mprun atlas $@ > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
1831	status=$?
1832	if [ ${status} -gt 0 ] ; then
1833	echo "IGCM_sys_atlas : error code ${status}"
1834	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
1835	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
1836	IGCM_debug_PopStack "IGCM_sys_atlas"
1837	return 1
1838	else
1839	IGCM_debug_PopStack "IGCM_sys_atlas"
1840	return 0
1841	fi
1842
1843	IGCM_debug_PopStack "IGCM_sys_atlas"
1844	}
1845
1846	#D-#==================================================
1847	#D-function IGCM_sys_rebuild_nemo
1848	#D-* Purpose: rebuild nemo parallel files with nemo specific rebuild on curie
1849	#D-* Examples:
1850	#D-
1851
1852	function IGCM_sys_rebuild_nemo {
1853	IGCM_debug_PushStack "IGCM_sys_rebuild_nemo" $@
1854	if ( $DEBUG_sys ) ; then
1855	echo "IGCM_sys_rebuild_nemo :" $@
1856	fi
1857
1858	$( ccc_home -u igcmg)/Tools/irene/bin/rebuild_nemo ${1} ${2}
1859
1860	IGCM_debug_PopStack "IGCM_sys_rebuild_nemo"
1861	}
1862

Note: See TracBrowser for help on using the repository browser.

Download in other formats: