Context Navigation

source: trunk/libIGCM/libIGCM_sys/libIGCM_sys_jeanzay.ksh @ 1491

Last change on this file since 1491 was 1490, checked in by jgipsl, 5 years ago
Added first version of sys_jeanzay. Still lots to do. Only case done for now is one executable mpi mode. Only create_ts done among post-treatments and only from main job (not done via TimeSeriesChecker?). Change of data project is not working (follwoing is not working when launched from job: eval $(idrenv -d ${PROJECT}) ).
Property licence set to The following licence information concerns ONLY the libIGCM tools ================================================================== Copyright © Centre National de la Recherche Scientifique CNRS Commissariat à l'Énergie Atomique CEA libIGCM : Library for Portable Models Computation of IGCM Group. IGCM Group is the french IPSL Global Climate Model Group. This library is a set of shell scripts and functions whose purpose is the management of the initialization, the launch, the transfer of output files, the post-processing and the monitoring of datas produce by any numerical program on any plateforme. This software is governed by the CeCILL license under French law and abiding by the rules of distribution of free software. You can use, modify and/ or redistribute the software under the terms of the CeCILL license as circulated by CEA, CNRS and INRIA at the following URL "http://www.cecill.info". As a counterpart to the access to the source code and rights to copy, modify and redistribute granted by the license, users are provided only with a limited warranty and the software's author, the holder of the economic rights, and the successive licensors have only limited liability. In this respect, the user's attention is drawn to the risks associated with loading, using, modifying and/or developing or reproducing the software by the user in light of its specific status of free software, that may mean that it is complicated to manipulate, and that also therefore means that it is reserved for developers and experienced professionals having in-depth computer knowledge. Users are therefore encouraged to load and test the software's suitability as regards their requirements in conditions enabling the security of their systems and/or data to be ensured and, more generally, to use and operate it in the same conditions as regards security. The fact that you are presently reading this means that you have had knowledge of the CeCILL license and that you accept its terms. Property svn:keywords set to `Revision Author Date`
File size: 62.5 KB

Line
1	#!/bin/ksh
2
3	#**************************************************************
4	# Author: Sebastien Denvil, Martial Mancip, Arnaud Caubel
5	# Contact: Arnaud.Caubel__at__lsce.ipsl.fr
6	# $Revision:: $ Revision of last commit
7	# $Author:: $ Author of last commit
8	# $Date:: $ Date of last commit
9	# IPSL (2006)
10	# This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
11	#
12	#**************************************************************
13
14	#=========================================================
15	# The documentation of this file can be automatically generated
16	# if you use the prefix #D- for comments to be extracted.
17	# Extract with command: cat lib* \| grep "^#D-" \| cut -c "4-"
18	#=========================================================
19
20	#D-#==================================================
21	#D-LibIGCM_sys for Jean-Zay
22	#D-#==================================================
23	#D-
24	#D- This ksh library if a layer under some usefull
25	#D-environment variables and shell commands.
26	#D-All those definitions depend on host particularities.
27	#D-It manages a stack mechanism and test validity of operations.
28	#D-All function described bellow must be prefixed by IGCM_sys.
29
30	#====================================================
31	# libIGCM_sys PARAMETERS
32	#====================================================
33
34	#====================================================
35	# set DEBUG_sys to true to output calls of function
36	typeset -r DEBUG_sys=${DEBUG_sys:=true}
37
38	#====================================================
39	# Turn in dry run mode ? (sys_Put_Rest, sys_Put_Out, sys_Get)
40	typeset -r DRYRUN=${DRYRUN:=0}
41
42	# YOU MUST COMPILE YOUR EXE FILES FOR DRYRUN MODE !
43	# -------------------------------------------------------------------------------------
44	# \| DRYRUN= \| Date computations, \| sys_Get \| Exe \| sys_Put_Out; sys_Put_Rest \|
45	# \| \| Cp/Exe/param/files \| \| \| \|
46	# \| \| Chmod Qsub \| \| \| \|
47	# -------------------------------------------------------------------------------------
48	# \| 0 \| yes \| yes \| yes \| yes \|
49	# -------------------------------------------------------------------------------------
50	# \| 1 \| yes \| yes \| yes \| no \|
51	# -------------------------------------------------------------------------------------
52	# \| 2 \| yes \| yes \| no \| no \|
53	# -------------------------------------------------------------------------------------
54	# \| 3 \| yes \| no \| no \| no \|
55	# -------------------------------------------------------------------------------------
56
57	#=====================================================
58	# Global Variables :
59	#=====================================================
60	# Language : "fr" or "en"
61	typeset -r MYLANG="fr"
62
63	#=====================================================
64	# Host user names project maxCpuTime
65	# $hostname ou hostname
66	typeset HOST=${HOST:=$( hostname )}
67	# $username ou whoami
68	typeset LOGIN=${LOGIN:=$( whoami )}
69	# $hostname of the MASTER job
70	typeset MASTER=jeanzay
71	# add default project set by ins_job
72	typeset PROJECT=$( echo ${SLURM_JOB_ACCOUNT:=::default_project::} \| cut -c "1-3" )
73	### jobWarningDelay in seconds
74	##typeset jobWarningDelay=${BRIDGE_MSUB_MAXTIME}
75
76	#D-
77	#D-#==================================================
78	#D-Program used in libIGCM
79	#D-#==================================================
80
81	# Submit command
82	typeset SUBMIT=${SUBMIT:=sbatch}
83	# rsync with path
84	typeset -r RSYNC=/usr/bin/rsync
85	# RSYNC_opt args to rsync
86	typeset -r RSYNC_opt="-va"
87	# ie storage filesystem
88	typeset -r STOREHOST=${MASTER}
89	typeset -r REMOTE_RSYNC=/usr/bin/rsync
90
91	#====================================================
92	# Access to module command
93	#====================================================
94	. /usr/share/Modules/init/ksh
95
96	#====================================================
97	# Set environment tools (ferret, nco, cdo, rebuild, ...)
98	#====================================================
99	if [ X${TaskType} = Xcomputing ] ; then
100	IGCM_debug_Print 1 "Modules will be loaded later in IGCM_sys_activ_variables."
101	else
102	module purge > /dev/null 2>&1
103	. /gpfswork/rech/psl/commun/MachineEnvironment/jeanzay/env_jeanzay > /dev/null 2>&1
104	#currently no atlas
105	# . /gpfswork/rech/psl/commun/MachineEnvironment/jeanzay/env_atlas_jeanzay > /dev/null 2>&1
106
107	# export PCMDI_MP=/ccc/work/cont003/igcmg/igcmg/PCMDI-MP
108	export UVCDAT_ANONYMOUS_LOG=FALSE
109	fi
110
111	# FYI
112	[ ! X${TaskType} = Xchecking ] && IGCM_debug_Print 1 "List of loaded modules:"
113	[ ! X${TaskType} = Xchecking ] && module list
114
115	# For AddNoise and AddPertu3DOCE
116	##export PATH=${PATH}:/ccc/cont003/home/igcmg/igcmg/Tools/curie/bin
117	#jg no addnoise
118	#export PATH=${PATH}:$(ccc_home -u igcmg)/Tools/irene/bin
119
120	# For rebuild
121	##export PATH=${PATH}:/ccc/cont003/home/igcmg/igcmg/Tools/irene/rebuild/modipsl_IOIPSL_PLUS_v2_2_4/bin
122	#jg no yet any rebuild
123	#export PATH=${PATH}:$(ccc_home -u igcmg)/Tools/irene/rebuild/modipsl_IOIPSL_PLUS_v2_2_4/bin
124
125	#====================================================
126	# Host specific DIRECTORIES
127	#====================================================
128
129	#====================================================
130	#- MirrorlibIGCM for frontend
131	typeset -r MirrorlibIGCM=${MirrorlibIGCM:=false}
132
133	#====================================================
134	#- libIGCM_POST for frontend
135	typeset -r libIGCM_POST=${libIGCM}
136
137	#====================================================
138	#- R_EXE (==> BIN_DIR = ${MODIPSL}/bin )
139	typeset -r R_EXE="${MODIPSL}/bin"
140
141	#====================================================
142	#- SUBMIT_DIR : submission dir
143	typeset -x SUBMIT_DIR=${SUBMIT_DIR:=${SLURM_SUBMIT_DIR}}
144
145	#====================================================
146	#- IN
147	typeset -r R_IN=${R_IN:=/gpfswork/rech/psl/commun/IGCM}
148
149	#====================================================
150	#- RUN_DIR_PATH : Temporary working directory (=> TMP)
151	typeset -r RUN_DIR_PATH=${RUN_DIR_PATH:=${SCRATCH}/RUN_DIR/${SLURM_JOBID}_${$}}
152
153	#====================================================
154	#- OUTCOMMAND_PATH : tmp place to store command lines standard error and outputs
155	typeset -r OUTCOMMAND_PATH=/tmp
156
157	#====================================================
158	#- HOST_MPIRUN_COMMAND
159	typeset -r HOST_MPIRUN_COMMAND=${HOST_MPIRUN_COMMAND:="/usr/bin/time srun --mpi=pmi2 "}
160
161	#====================================================
162	#- Max number of arguments passed to nco operator or demigration command
163	UNIX_MAX_LIMIT=360
164
165	#====================================================
166	#- set PackDefault to true
167	PackDefault=true
168
169	#====================================================
170	#- Default number of MPI task for IPSL coupled model
171	#- required for backward compatibility
172	#-
173	DEFAULT_NUM_PROC_OCE=5
174	DEFAULT_NUM_PROC_CPL=1
175	(( DEFAULT_NUM_PROC_ATM = BATCH_NUM_PROC_TOT - DEFAULT_NUM_PROC_OCE - DEFAULT_NUM_PROC_CPL ))
176
177	#====================================================
178	#- Number of computing cores per node
179	#-
180	NB_CORE_PER_NODE=48
181
182	#D-#==================================================
183	#D-function IGCM_sys_defineArchives
184	#D-* Purpose:
185	#D-* Load dfldatadir module : According to project used for submission (default) or set in config.card (optional)
186	#D-* Define ARCHIVE : Dedicated to large files
187	#D-* Define STORAGE : Dedicated to small/medium files
188	#D-* Define R_OUT : Output tree located on ARCHIVE
189	#D-* Define R_FIG : Output tree located on STORAGE hosting figures (monsitoring and atlas, and/or small files)
190	#D-* Define R_BUF : Output tree located on SCRATCH hosting files waiting for rebuild or pack processes
191	#D-* if SpaceName=TEST everything is stored on SCRATCH
192	#D-* Examples:
193	#D-
194	function IGCM_sys_defineArchives {
195	IGCM_debug_PushStack "IGCM_sys_defineArchives"
196
197	# Load dfldatadir depending on the project used for submission (default) or set in config.card (optional)
198	if [ X${config_UserChoices_DataProject} = X ] \|\| [ X${config_UserChoices_DataProject} = DEFAULT ]; then
199	# The variable DataProject is not set in in section UserChoices in config.card or it is set to DEFAULT
200	# Default option: Change IDRPROJ according to the project used for submission
201
202	# jg following line does not work in a submitted job:
203	eval $(idrenv -d ${PROJECT})
204	else
205	# Use the project set by the variable DataProject in section UserChoices in config.card
206	eval $(idrenv -d ${config_UserChoices_DataProject})
207	IGCM_debug_Print 1 "Change DataProject red in config.card : ${config_UserChoices_DataProject}"
208	fi
209	IGCM_debug_Print 1 "Data project for output to be used: ${IDRPROJ}"
210
211	#====================================================
212	#- RUN_DIR_PATH : Temporary working directory (=> TMP)
213	typeset -r RUN_DIR_PATH=${RUN_DIR_PATH:=${SCRATCH}/RUN_DIR/${SLURM_JOBID}_${$}}
214
215	if [ ! X${config_UserChoices_ARCHIVE} = X ]; then
216	#====================================================
217	#- ARCHIVE (dedicated to large files)
218	ARCHIVE=${config_UserChoices_ARCHIVE}
219	else
220	#====================================================
221	#- ARCHIVE (dedicated to large files)
222	ARCHIVE=${STORE}
223	fi
224
225	if [ ! X${config_UserChoices_STORAGE} = X ]; then
226	#====================================================
227	#- STORAGE (dedicated to small/medium files)
228	STORAGE=${config_UserChoices_STORAGE}
229	else
230	#====================================================
231	#- STORAGE (dedicated to small/medium files)
232	STORAGE=${WORK}
233	fi
234
235	if [ X${config_UserChoices_SpaceName} = XTEST ]; then
236	#====================================================
237	#- R_OUT
238	R_OUT=${SCRATCH}/IGCM_OUT
239
240	#====================================================
241	#- R_FIG (hosting figures : monitoring and atlas, and/or small files)
242	R_FIG=${SCRATCH}/IGCM_OUT
243
244	IGCM_debug_Print 1 "SpaceName=TEST ==> OVERRULE destination path directories"
245
246	else
247	#====================================================
248	#- R_OUT
249	R_OUT=${ARCHIVE}/IGCM_OUT
250
251	#====================================================
252	#- R_FIG (hosting figures : monitoring and atlas, and/or small files)
253	R_FIG=${STORAGE}/IGCM_OUT
254	fi
255
256	#====================================================
257	#- CMIP6 (hosting CMIP6 files produced by XIOS2 and configured by dr2xml)
258	CMIP6_BUF=${STORAGE}/IGCM_OUT
259
260	#====================================================
261	#- R_BUF (ONLY FOR double copy an scratch)
262	R_BUF=${SCRATCH}/IGCM_OUT
263
264	IGCM_debug_Print 1 "R_OUT has been defined = ${R_OUT}"
265	IGCM_debug_Print 1 "R_BUF has been defined = ${R_BUF}"
266	IGCM_debug_Print 1 "R_FIG has been defined = ${R_FIG}"
267
268	IGCM_debug_PopStack "IGCM_sys_defineArchives"
269	}
270
271	#D-#==================================================
272	#D-function IGCM_sys_RshArchive
273	#D-* Purpose: Archive rsh command
274	#D-* Examples:
275	#D-
276	function IGCM_sys_RshArchive {
277	IGCM_debug_PushStack "IGCM_sys_RshArchive" $@
278	/bin/ksh <<-EOF
279	${@}
280	EOF
281	status=$?
282	if [ ${status} -gt 0 ] ; then
283	IGCM_debug_Print 2 "IGCM_sys_RshArchive : command failed error code ${status}"
284	IGCM_debug_Exit "IGCM_sys_RshArchive"
285	fi
286	IGCM_debug_PopStack "IGCM_sys_RshArchive"
287	}
288
289	#D-#==================================================
290	#D-function IGCM_sys_RshArchive_NoError
291	#D-* Purpose: Archive rsh command, without error
292	#D-* used only in monitoring.job
293	#D-* Examples:
294	#D-
295	function IGCM_sys_RshArchive_NoError {
296	IGCM_debug_PushStack "IGCM_sys_RshArchive_NoError" $@
297	/bin/ksh <<-EOF
298	${@} 2> /dev/null
299	EOF
300	IGCM_debug_PopStack "IGCM_sys_RshArchive_NoError"
301	}
302
303	#D-#==================================================
304	#D-function IGCM_sys_MkdirArchive
305	#D-* Purpose: Mkdir on Archive
306	#D-* Examples:
307	#D-
308	function IGCM_sys_MkdirArchive {
309	IGCM_debug_PushStack "IGCM_sys_MkdirArchive" $@
310	if ( $DEBUG_sys ) ; then
311	echo "IGCM_sys_MkdirArchive :" $@
312	fi
313	#- creation de repertoire sur le serveur fichier
314	if [ ! -d ${1} ]; then
315	\mkdir -p $1
316	status=$?
317
318	if [ ${status} -gt 0 ] ; then
319	IGCM_debug_Print 2 "IGCM_sys_MkdirArchive : mkdir failed error code ${status}"
320	IGCM_debug_Exit "IGCM_sys_MkdirArchive"
321	fi
322	fi
323	IGCM_debug_PopStack "IGCM_sys_MkdirArchive"
324	}
325
326	#D-#==================================================
327	#D-function IGCM_sys_TestDirArchive
328	#D-* Purpose: Test Directory that must exists on Archive
329	#D-* Examples:
330	#D-
331	function IGCM_sys_TestDirArchive {
332	IGCM_debug_PushStack "IGCM_sys_TestDirArchive" $@
333	if ( $DEBUG_sys ) ; then
334	echo "IGCM_sys_TestDirArchive :" $@
335	fi
336	typeset ExistFlag
337	ExistFlag=$( [ -d $1 ] && echo 0 \|\| echo 1 )
338	IGCM_debug_PopStack "IGCM_sys_TestDirArchive"
339	return ${ExistFlag}
340	}
341
342	#D-#==================================================
343	#D-function IGCM_sys_IsFileArchived
344	#D-* Purpose: Test file that must NOT EXISTS on Archive based on filename only
345	#D-* Examples:
346	#D-
347	function IGCM_sys_IsFileArchived {
348	IGCM_debug_PushStack "IGCM_sys_IsFileArchived" $@
349	if ( $DEBUG_sys ) ; then
350	echo "IGCM_sys_IsFileArchived :" $@
351	fi
352	typeset IsArchivedFlag
353	IsArchivedFlag=$( [ X$( echo $@ \| grep ^\/ccc\/store ) != X ] && echo 0 \|\| echo 1 )
354	IGCM_debug_PopStack "IGCM_sys_IsFileArchived"
355
356	return ${IsArchivedFlag}
357	}
358
359	#D-#==================================================
360	#D-function IGCM_sys_TestFileArchive
361	#D-* Purpose: Test file that must NOT EXISTS on Archive
362	#D-* Examples:
363	#D-
364	function IGCM_sys_TestFileArchive {
365	IGCM_debug_PushStack "IGCM_sys_TestFileArchive" $@
366	typeset ExistFlag
367	ExistFlag=$( [ -f $1 ] && echo 0 \|\| echo 1 )
368	IGCM_debug_PopStack "IGCM_sys_TestFileArchive"
369
370	return ${ExistFlag}
371	}
372
373	#D-#==================================================
374	#D-function IGCM_sys_CountFileArchive
375	#D-* Purpose: Count files on Archive filesystem
376	#D-* Examples:
377	#D-
378	function IGCM_sys_CountFileArchive {
379	IGCM_debug_PushStack "IGCM_sys_CountFileArchive" $@
380	ls ${@} 2>/dev/null \| wc -l
381	if [ $? -gt 0 ] ; then
382	echo "IGCM_sys_CountFileArchive : erreur."
383	fi
384	IGCM_debug_PopStack "IGCM_sys_CountFileArchive"
385	}
386
387	#D-#==================================================
388	#D-function IGCM_sys_Tree
389	#D-* Purpose: Tree directories with files on ${ARCHIVE}
390	#D-* Examples: IGCM_sys_Tree ${R_IN} ${R_OUT}
391	#D-
392	function IGCM_sys_Tree {
393	IGCM_debug_PushStack "IGCM_sys_Tree" $@
394	if ( $DEBUG_sys ) ; then
395	echo "IGCM_sys_Tree :" $@
396	fi
397
398	\ls -lR ${@}
399
400	IGCM_debug_PopStack "IGCM_sys_Tree"
401	}
402
403	#D-#==================================================
404	#D-function IGCM_sys_Qsub
405	#D-* Purpose: Qsub new job
406	#D-* Examples:
407	#D-
408	function IGCM_sys_Qsub {
409	IGCM_debug_PushStack "IGCM_sys_Qsub" $@
410	if ( $DEBUG_sys ) ; then
411	echo "IGCM_sys_Qsub :" $@
412	fi
413	typeset options status
414	options="-o ${SUBMIT_DIR}/${Script_Output} -e ${SUBMIT_DIR}/${Script_Output}"
415
416	/usr/bin/time sbatch ${options} $1 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
417	status=$?
418
419	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
420	if [ ${status} -gt 0 ] ; then
421	IGCM_debug_Print 2 "IGCM_sys_Qsub ${options} $1 : error code ${status}"
422	IGCM_debug_Exit "IGCM_sys_Qsub"
423	else
424	JobID=$( gawk {'print $4'} ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ )
425	fi
426	IGCM_debug_PopStack "IGCM_sys_Qsub"
427	}
428
429	#D-#==================================================
430	#D-function IGCM_sys_QsubPost
431	#D-* Purpose: Qsub new job on scalaire
432	#D-* Examples:
433	#D-
434	function IGCM_sys_QsubPost {
435	IGCM_debug_PushStack "IGCM_sys_QsubPost" $@
436	if ( $DEBUG_sys ) ; then
437	echo "IGCM_sys_QsubPost :" $@
438	fi
439	typeset options status
440	options="-o ${POST_DIR}/${Script_Post_Output}.out -e ${POST_DIR}/${Script_Post_Output}.out"
441
442	sbatch ${options} ${libIGCM_POST}/$1.job > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
443	status=$?
444
445	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
446	if [ ${status} -gt 0 ] ; then
447	IGCM_debug_Print 2 "IGCM_sys_QsubPost ${options} ${libIGCM_POST}/$1.job : error code ${status}"
448	IGCM_debug_Exit "IGCM_sys_QsubPost"
449	else
450	JobID=$( gawk {'print $4'} ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ )
451	fi
452	IGCM_debug_PopStack "IGCM_sys_QsubPost"
453	}
454
455	#D-*************************
456	#D- File transfer functions
457	#D-*************************
458	#D-
459
460	#D-#==================================================
461	#D-function IGCM_sys_RmRunDir
462	#D-* Purpose: rm tmpdir (dummy function most of the time batch
463	#D- scheduler will do the job)
464	#D-* Examples:
465	#D-
466	function IGCM_sys_RmRunDir {
467	IGCM_debug_PushStack "IGCM_sys_RmRunDir" $@
468	if ( $DEBUG_sys ) ; then
469	echo "IGCM_sys_RmRunDir :" $@
470	fi
471
472	typeset status
473
474	echo rm $@ > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
475	\rm $@ >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
476	status=$?
477
478	if [ ${status} -gt 0 ] ; then
479	IGCM_debug_Print 1 "IGCM_sys_RmRunDir : rm error code is ${status}."
480	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
481	IGCM_debug_Exit "IGCM_sys_RmRunDir"
482	else
483	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
484	fi
485	IGCM_debug_PopStack "IGCM_sys_RmRunDir"
486	}
487
488	#D-#==================================================
489	#D-function IGCM_sys_Put_Dir
490	#D-* Purpose: Copy a complete directory on $(ARCHIVE)
491	#D-* Examples:
492	#D-
493	function IGCM_sys_Put_Dir {
494	IGCM_debug_PushStack "IGCM_sys_Put_Dir" $@
495	if ( $DEBUG_sys ) ; then
496	echo "IGCM_sys_Put_Dir :" $@
497	fi
498	if [ $DRYRUN = 0 ]; then
499	if [ ! -d ${1} ] ; then
500	echo "WARNING : IGCM_sys_Put_Dir ${1} DOES NOT EXIST ."
501	IGCM_debug_PopStack "IGCM_sys_Put_Dir"
502	return
503	fi
504
505	typeset status
506
507	# Only if we use rsync
508	#IGCM_sys_TestDirArchive $( dirname $2 )
509	#
510	#USUAL WAY
511	\cp -r $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
512	status=$?
513
514	if [ ${status} -gt 0 ] ; then
515	IGCM_debug_Print 2 "IGCM_sys_Put_Dir : cp failed error code ${status}"
516	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
517	IGCM_debug_Exit "IGCM_sys_Put_Dir"
518	else
519	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
520	fi
521	fi
522	IGCM_debug_PopStack "IGCM_sys_Put_Dir"
523	}
524
525	#D-#==================================================
526	#D-function IGCM_sys_Get_Dir
527	#D-* Purpose: Copy a complete directory from ${ARCHIVE}
528	#D-* Examples:
529	#D-
530	function IGCM_sys_Get_Dir {
531	IGCM_debug_PushStack "IGCM_sys_Get_Dir" $@
532	if ( $DEBUG_sys ) ; then
533	echo "IGCM_sys_Get_Dir :" $@
534	fi
535	if [ $DRYRUN = 0 ]; then
536	typeset NB_ESSAI DELAI status i
537	# number of tentative
538	NB_ESSAI=3
539	# time delay between tentative
540	DELAI=2
541
542	#
543	# USUAL WAY
544	# add 'ccc_hsm get' (to demigrate all offline files) to reduce time of this command :
545	ccc_hsm get -r $1
546
547	i=0
548	while [ $i -lt $NB_ESSAI ] ; do
549	\cp -ur $1 $2 >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
550	status=$?
551	if [ ${status} -gt 0 ] ; then
552	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : cp failed error code ${status} ${i}/${NB_ESSAI}"
553	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : sleep ${DELAI} seconds and try again."
554	sleep $DELAI
555	else
556	break
557	fi
558	(( i = i + 1 ))
559	done
560
561	if [ ${status} -gt 0 ] ; then
562	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : cp failed error code ${status}"
563	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
564	IGCM_debug_Exit "IGCM_sys_Get_Dir"
565	else
566	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
567	fi
568	fi
569	IGCM_debug_PopStack "IGCM_sys_Get_Dir"
570	}
571
572	#D-#==================================================
573	#D-function IGCM_sys_Put_Rest
574	#D-* Purpose: Put computied restarts on ${ARCHIVE}.
575	#D- File and target directory must exist.
576	#D-* Examples:
577	#D-
578	function IGCM_sys_Put_Rest {
579	IGCM_debug_PushStack "IGCM_sys_Put_Rest" $@
580	if ( $DEBUG_sys ) ; then
581	echo "IGCM_sys_Put_Rest :" $@
582	fi
583	if [ $DRYRUN = 0 ]; then
584	if [ ! -f ${1} ] ; then
585	echo "ERROR : IGCM_sys_Put_Rest ${1} DOES NOT EXIST ."
586	IGCM_debug_Exit "IGCM_sys_Put_Rest"
587	fi
588
589	typeset status
590	#
591	# USUAL WAY
592	\cp $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
593	status=$?
594
595	# #RSYNC WITH NETWORK SSH CALL
596	# echo ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
597	# ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
598
599	# #RSYNC WITH NFS USE
600	# echo ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
601	# ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
602
603	# status=$?
604	# IGCM_sys_Rsync_out $status
605
606	# ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
607	# (( status=status+$? ))
608
609	if [ ${status} -gt 0 ] ; then
610	IGCM_debug_Print 2 "IGCM_sys_Put_Rest : cp failed error code ${status}"
611	[ -f ${1} ] && ls -l ${1}
612	[ -f ${2} ] && ls -l ${2}
613	[ -f ${2}/${1} ] && ls -l ${2}/${1}
614	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
615	IGCM_debug_Exit "IGCM_sys_Put_Rest"
616	else
617
618	if [ X${JobType} = XRUN ] ; then
619	[ -f ${2} ] && IGCM_sys_Chmod 444 ${2}
620	[ -f ${2}/${1} ] && IGCM_sys_Chmod 444 ${2}/${1}
621	fi
622
623	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
624	fi
625	fi
626	IGCM_debug_PopStack "IGCM_sys_Put_Rest"
627	}
628
629	#D-#==================================================
630	#D-function IGCM_sys_Put_Out
631	#D-* Purpose: Copy a file on ${ARCHIVE} after having chmod it in readonly
632	#D-* Examples:
633	#D-
634	function IGCM_sys_Put_Out {
635	IGCM_debug_PushStack "IGCM_sys_Put_Out" $@
636	if ( $DEBUG_sys ) ; then
637	echo "IGCM_sys_Put_Out :" $@
638	fi
639
640	typeset NB_ESSAI DELAI status i exist skip
641	typeset fileDeviceNumberInHex directoryDeviceNumberInHex
642
643	# number of tentative
644	NB_ESSAI=3
645	# time delay between tentative
646	DELAI=2
647
648	if [ $DRYRUN = 0 ]; then
649	if [ ! -f ${1} ] ; then
650	echo "WARNING : IGCM_sys_Put_Out ${1} DOES NOT EXIST ."
651	IGCM_debug_PopStack "IGCM_sys_Put_Out"
652	return 1
653	fi
654	#
655	IGCM_sys_MkdirArchive $( dirname $2 )
656	#
657	exist=false
658	skip=false
659	if [ -f $2 ] ; then
660	IGCM_debug_Print 1 "$2 already exist"
661	ccc_hsm get $2
662	exist=true
663	if [ "X$( diff $1 $2 )" = X ] ; then
664	IGCM_debug_Print 2 "$1 and $2 are the same file, we skip the copy"
665	skip=true
666	else
667	IGCM_debug_Print 2 "$1 and $2 are not the same file, we force the copy"
668	skip=false
669	fi
670	fi
671	#
672	if ( [ X${exist} = Xtrue ] && [ X${skip} = Xfalse ] ) ; then
673	IGCM_sys_Chmod u+w $2
674	fi
675
676	if [ X${skip} = Xfalse ] ; then
677	i=0
678	while [ $i -lt $NB_ESSAI ] ; do
679	# Identify file system
680	fileDeviceNumberInHex=$( stat -c %d $1 )
681	status=$?
682	if [ ${status} -gt 0 ] ; then
683	IGCM_debug_Exit "IGCM_sys_Put_Out"
684	fi
685	# Identify file system
686	directoryDeviceNumberInHex=$( stat -c %d $( dirname $2 ) )
687	status=$?
688	if [ ${status} -gt 0 ] ; then
689	IGCM_debug_Exit "IGCM_sys_Put_Out"
690	fi
691
692	if ( [ ${fileDeviceNumberInHex} -ne ${directoryDeviceNumberInHex} ] \|\| [ X$3 = XNOMOVE ] ) ; then
693	# They are not on the same device. USUAL WAY
694	\cp $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
695	status=$?
696	else
697	# They are on the same device. NOT SO USUAL WAY
698	\mv $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
699	status=$?
700	fi
701	if [ ${status} -gt 0 ]; then
702	IGCM_debug_Print 2 "IGCM_sys_Put_Out : cp failed error code ${status} ${i}/${NB_ESSAI}"
703	IGCM_debug_Print 2 "IGCM_sys_Put_Out : sleep ${DELAI} seconds and try again."
704	[ -f ${1} ] && ls -l ${1}
705	[ -f ${2} ] && ls -l ${2}
706	[ -f ${2}/${1} ] && ls -l ${2}/${1}
707	sleep $DELAI
708	else
709	break
710	fi
711	(( i = i + 1 ))
712	done
713	fi
714
715	# #RSYNC WITH NETWORK SSH CALL
716	# echo ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
717	# ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
718
719	# #RSYNC WITH NFS USE
720	# echo ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
721	# ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
722
723	# status=$?
724	# IGCM_sys_Rsync_out $status
725
726	# ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
727	# (( status=status+$? ))
728
729	if [ ${status} -gt 0 ] ; then
730	IGCM_debug_Print 2 "IGCM_sys_Put_Out : cp failed error code ${status}"
731	[ -f ${1} ] && ls -l ${1}
732	[ -f ${2} ] && ls -l ${2}
733	[ -f ${2}/${1} ] && ls -l ${2}/${1}
734	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
735	IGCM_debug_Exit "IGCM_sys_Put_Out"
736	else
737
738	if [ X${JobType} = XRUN ] ; then
739	if [ X${3} = X ] ; then
740	[ -f ${2} ] && IGCM_sys_Chmod 444 ${2}
741	[ -f ${2}/${1} ] && IGCM_sys_Chmod 444 ${2}/${1}
742	fi
743	fi
744
745	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
746	fi
747	fi
748	IGCM_debug_PopStack "IGCM_sys_Put_Out"
749	return 0
750	}
751
752	#D-#==================================================
753	#D-function IGCM_sys_Get
754	#D-* Purpose: Get a file from ${ARCHIVE}
755	#D-* Examples: IGCM_sys_Get myfile /destpath/myfile_with_PREFIX
756	#D- IGCM_sys_Get /l Array_contain_myfiles /destpath/
757	function IGCM_sys_Get {
758	IGCM_debug_PushStack "IGCM_sys_Get" $@
759
760	typeset DEST dm_liste target file_work
761	typeset NB_ESSAI DELAI status i
762
763	if ( $DEBUG_sys ) ; then
764	echo "IGCM_sys_Get :" $@
765	fi
766
767	# number of tentative
768	NB_ESSAI=3
769	# time delay between tentative
770	DELAI=2
771
772	if [ $DRYRUN -le 2 ]; then
773	if [ X${1} = X'/l' ] ; then
774	eval set +A dm_liste \${${2}}
775	else
776	eval set +A dm_liste ${1}
777	fi
778	eval DEST=\${${#}}
779	ccc_hsm get ${dm_liste[*]} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
780	status=$?
781	if [ ${status} -gt 0 ] ; then
782	echo "WARNING IGCM_sys_Get : error code ${status}"
783	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
784	echo "WARNING IGCM_sys_Get : will stop later if the cp fails."
785	fi
786
787	#if [ ${status} -gt 0 ] ; then
788	# if [ ! "X$( grep "Lost dmusrcmd connection" ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ )" = "X" ] ; then
789	# cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
790	# echo "WARNING IGCM_sys_Get : Lost dmusrcmd connection : "
791	# sleep 30
792	# echo "We try another time"
793	## dmget ${dm_liste[*]} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
794	# ccc_hsm get ${dm_liste[*]} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
795	# status=$?
796	# if [ ${status} -gt 0 ] ; then
797	# echo "ERROR IGCM_sys_Get : again demigration error :"
798	# cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
799	# IGCM_debug_Exit "IGCM_sys_Get"
800	# fi
801	# else
802	# echo "ERROR IGCM_sys_Get : demigration error :"
803	# cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
804	# IGCM_debug_Exit "IGCM_sys_Get"
805	# fi
806	#fi
807
808	# #RSYNC WITH NETWORK SSH CALL
809	# echo ${RSYNC} ${RSYNC_opt} -e ssh ${STOREHOST}:"${dm_liste}" ${STOREHOST}:${RUN_DIR}/${DEST} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
810	# ${RSYNC} ${RSYNC_opt} -e ssh ${STOREHOST}:"${dm_liste}" ${STOREHOST}:${RUN_DIR}/${DEST} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
811
812	# #RSYNC WITH NFS USE
813	# echo ${RSYNC} ${RSYNC_opt} ${dm_liste} ${RUN_DIR}/${DEST} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
814	# ${RSYNC} ${RSYNC_opt} ${dm_liste} ${RUN_DIR}/${DEST} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
815
816	# status=$?
817	# IGCM_sys_Rsync_out $status
818
819	# ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
820	# (( status=status+$? ))
821
822	#USUAL WAY
823	if [ X${1} = X'/l' ] ; then
824	for target in ${dm_liste[*]} ; do
825	local_file=$( basename ${target} )
826	# test if the target file is present before the loop
827	IGCM_sys_TestFileArchive ${target}
828	status=$?
829	if [ ${status} -gt 0 ] ; then
830	echo "IGCM_sys_Get, ERROR : regular file ${target} DOES NOT EXIST ."
831	IGCM_debug_Exit "IGCM_sys_Get"
832	else
833	i=0
834	while [ $i -lt $NB_ESSAI ] ; do
835	#if [ X${DoLink} = Xtrue ] ; then
836	# \ln -s ${target} ${DEST}/${local_file} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
837	# status=$?
838	# else
839	# \cp ${target} ${DEST}/${local_file} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
840	# status=$?
841	#fi
842	\ln -s ${target} ${DEST}/${local_file} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
843	status=$?
844	if [ ${status} -gt 0 ]; then
845	IGCM_debug_Print 2 "IGCM_sys_Get : cp failed error code ${status} ${i}/${NB_ESSAI}"
846	IGCM_debug_Print 2 "IGCM_sys_Get : sleep ${DELAI} seconds and try again."
847	sleep $DELAI
848	else
849	break
850	fi
851	(( i = i + 1 ))
852	done
853	if [ ${status} -gt 0 ] ; then
854	echo "IGCM_sys_Get : error"
855	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
856	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
857	IGCM_debug_Exit "IGCM_sys_Get"
858	else
859	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
860	fi
861	fi
862	done
863	else
864	i=0
865	while [ $i -lt $NB_ESSAI ] ; do
866	\cp ${dm_liste} ${DEST} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
867	status=$?
868	if [ ${status} -gt 0 ]; then
869	IGCM_debug_Print 2 "IGCM_sys_Get : cp failed error code ${status} ${i}/${NB_ESSAI}"
870	IGCM_debug_Print 2 "IGCM_sys_Get : sleep ${DELAI} seconds and try again."
871	sleep $DELAI
872	else
873	break
874	fi
875	(( i = i + 1 ))
876	done
877	if [ ${status} -gt 0 ] ; then
878	echo "IGCM_sys_Get : error"
879	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
880	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
881	IGCM_debug_Exit "IGCM_sys_Get"
882	else
883	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
884	fi
885	fi
886	fi
887	IGCM_debug_PopStack "IGCM_sys_Get"
888	}
889
890	#D-#==================================================
891	#D-function IGCM_sys_GetDate_Monitoring
892	#D-* Purpose: get the last year for which the monitoring has been computed
893	#D-* Examples:
894	#D-
895	function IGCM_sys_GetDate_Monitoring {
896	IGCM_debug_PushStack "IGCM_sys_GetDate_Monitoring" $@
897	if ( $DEBUG_sys ) ; then
898	echo "IGCM_sys_GetDate_Monitoring :" $@
899	fi
900
901	eval ${2}=$( cdo showyear ${1} 2> /dev/null \| gawk '{ print $NF }' )
902
903	IGCM_debug_PopStack "IGCM_sys_GetDate_Monitoring"
904	}
905
906	#D-#==================================================
907	#D-function IGCM_sys_Dods_Rm
908	#D-* Purpose: DO NOTHING ! Put ${ARCHIVE} files on DODS internet protocole.
909	#D-* Examples:
910	#D-
911	function IGCM_sys_Dods_Rm {
912	if ( $DEBUG_sys ) ; then
913	echo "IGCM_sys_Dods_Rm :" $@
914	fi
915	typeset status
916	if [ $DRYRUN = 0 ]; then
917
918	# if [ ! -d /dmnfs/cont003/dods/public/${LOGIN}/${R_DODS}/${1} ] ; then
919	# echo "WARNING : IGCM_sys_Dods_Rm /dmnfs/cont003/dods/public/${LOGIN}/${R_DODS}/${1} DOES NOT EXIST ."
920	# echo "Nothing has been done."
921	# return
922	# fi
923
924	if [ "$#" -eq 1 ]; then
925	## /ccc/cont003/home/igcmg/igcmg/Tools/irene/thredds_rm public/${LOGIN}/${R_DODS}/${1} # > out_dods_rm 2>&1
926	$(ccc_home -u igcmg)/Tools/irene/thredds_rm public/${LOGIN}/${R_DODS}/${1} # > out_dods_rm 2>&1
927	status=$?
928	else
929	## /ccc/cont003/home/igcmg/igcmg/Tools/irene/thredds_rm public/${LOGIN}/${R_DODS} # > out_dods_rm 2>&1
930	$(ccc_home -u igcmg)/Tools/irene/thredds_rm public/${LOGIN}/${R_DODS} # > out_dods_rm 2>&1
931	status=$?
932	fi
933
934	# if [ ${status} -gt 0 ] ; then
935	# echo "IGCM_sys_Dods_Rm : error."
936	# cat out_dods_rm
937	# IGCM_debug_Exit "IGCM_sys_Dods_Rm"
938	# else
939	# rm out_dods_rm
940	# fi
941
942	fi
943	return $status
944	}
945
946	#D-#==================================================
947	#D-function IGCM_sys_Dods_Cp
948	#D-* Purpose: Copy $(ARCHIVE) files on DODS internet protocole.
949	#D-* Examples:
950	#D-
951	function IGCM_sys_Dods_Cp {
952	if ( $DEBUG_sys ) ; then
953	echo "IGCM_sys_Dods_Cp :" $@
954	fi
955	typeset status
956	if [ $DRYRUN = 0 ]; then
957
958	# if [ ! -d ${R_SAVE}/${1} ] ; then
959	# echo "WARNING : IGCM_sys_Dods_Cp ${R_SAVE}/${1} DOES NOT EXIST ."
960	# echo "Nothing has been done."
961	# return
962	# fi
963
964	## /ccc/cont003/home/igcmg/igcmg/Tools/irene/bin/thredds_cp ${1} public/${LOGIN}/${R_DODS} # > out_dods_cp 2>&1
965	$(ccc_home -u igcmg)/Tools/irene/thredds_cp ${1} public/${LOGIN}/${R_DODS} # > out_dods_cp 2>&1
966	status=$?
967
968	# if [ ${status} -gt 0 ] ; then
969	# echo "IGCM_sys_Dods_Cp : error."
970	# cat out_dods_cp
971	# IGCM_debug_Exit "IGCM_sys_Dods_Cp"
972	# else
973	# rm out_dods_cp
974	# fi
975
976	fi
977	return $status
978	}
979
980	#D-#==================================================
981	#D-function IGCM_sys_Put_Dods
982	#D-* Purpose: Put ${ARCHIVE} files on DODS internet protocole.
983	#D-* Examples:
984	#D-
985	function IGCM_sys_Put_Dods {
986	IGCM_debug_PushStack "IGCM_sys_Put_Dods" $@
987	if ( $DEBUG_sys ) ; then
988	echo "IGCM_sys_Put_Dods :" $@
989	fi
990	typeset status
991	if [ $DRYRUN = 0 ]; then
992	if ( [ ! -d ${R_SAVE}/${1} ] && [ ! -d ${R_FIGR}/${1} ] ) ; then
993	echo "WARNING IGCM_sys_Put_Dods : None of the following directories exist. Exactly one should."
994	echo "WARNING IGCM_sys_Put_Dods : ${R_SAVE}/${1} DOES NOT EXIST."
995	echo "WARNING IGCM_sys_Put_Dods : ${R_FIGR}/${1} DOES NOT EXIST."
996	IGCM_debug_PopStack "IGCM_sys_Put_Dods"
997	return
998	fi
999
1000	if ( [ -d ${R_SAVE}/${1} ] && [ -d ${R_FIGR}/${1} ] ) ; then
1001	echo "WARNING IGCM_sys_Put_Dods : Both of the following directories exist. Exactly one should."
1002	echo "WARNING IGCM_sys_Put_Dods : ${R_SAVE}/${1} EXISTS."
1003	echo "WARNING IGCM_sys_Put_Dods : ${R_FIGR}/${1} EXISTS."
1004	IGCM_debug_PopStack "IGCM_sys_Put_Dods"
1005	return
1006	fi
1007	#
1008	if [ -d ${R_SAVE}/${1} ] ; then
1009	cd ${R_SAVE}
1010	elif [ -d ${R_FIGR}/${1} ] ; then
1011	cd ${R_FIGR}
1012	fi
1013
1014	IGCM_sys_Dods_Rm ${1}
1015	IGCM_sys_Dods_Cp ${1}
1016	status=0
1017
1018	if [ ${status} -gt 0 ] ; then
1019	echo "IGCM_sys_Put_Dods : error."
1020	IGCM_debug_Exit "IGCM_sys_Put_Dods"
1021	fi
1022	fi
1023	IGCM_debug_PopStack "IGCM_sys_Put_Dods"
1024	}
1025
1026	##############################################################
1027	# REBUILD OPERATOR
1028
1029	#D-#==================================================
1030	#D-function IGCM_sys_sync
1031	#D-* Purpose: flush buffer on disk
1032	#D-* Examples:
1033	#D-
1034	function IGCM_sys_sync {
1035	IGCM_debug_PushStack "IGCM_sys_sync" $@
1036	if ( $DEBUG_sys ) ; then
1037	echo "IGCM_sys_sync :" $@
1038	fi
1039
1040	/bin/sync
1041
1042	IGCM_debug_PopStack "IGCM_sys_sync"
1043	}
1044
1045	############################################################
1046	# Activate Running Environnment Variables
1047
1048	#D-#==================================================
1049	#D-function IGCM_sys_activ_variables
1050	#D-* Purpose: set environement variables prior to execution
1051	#D-* Examples:
1052	#D-
1053	function IGCM_sys_activ_variables {
1054	IGCM_debug_PushStack "IGCM_sys_activ_variables"
1055	if ( $DEBUG_sys ) ; then
1056	echo "IGCM_sys_activ_variables"
1057	fi
1058
1059	# --------------------------------------------------------------------
1060	#D- MODULE specifications
1061	# --------------------------------------------------------------------
1062	# Source the file EnvFile containing all module load needed to run the model.
1063	# EnvFile can be specified in config.card. If this is the case and if the file exists,
1064	# it will be used. Otherwise a default file will be used.
1065	if [ ! X${config_UserChoices_EnvFile} = X ] && [ -f ${config_UserChoices_EnvFile} ] ; then
1066	# EnvFile is set in config.card and the file exists
1067	IGCM_debug_Print 1 "EnvFile set in config.card will be used"
1068	EnvFile=${config_UserChoices_EnvFile}
1069	else
1070	IGCM_debug_Print 1 "IGCM_sys_active_variables : Default modules will be used"
1071	module purge > /dev/null 2>&1
1072	EnvFile=/gpfswork/rech/psl/commun/MachineEnvironment/jeanzay/env_jeanzay
1073	fi
1074	IGCM_debug_Print 1 "IGCM_sys_active_variables : Following EnvFile will be used :"
1075	IGCM_debug_Print 1 ${EnvFile}
1076	. ${EnvFile}
1077
1078	IGCM_debug_Print 1 "IGCM_sys_active_variables : Now loaded modules for Jean-Zay. "
1079	module list
1080
1081	# --------------------------------------------------------------------
1082	#D- MPI specifications
1083	# --------------------------------------------------------------------
1084
1085	# --------------------------------------------------------------------
1086	#D- Other specifications
1087	# --------------------------------------------------------------------
1088
1089	ulimit -s unlimited
1090
1091	IGCM_debug_PopStack "IGCM_sys_activ_variables"
1092	}
1093
1094	############################################################
1095	# Desactivate Running Environnment Variables
1096
1097	#D-#==================================================
1098	#D-function IGCM_sys_desactiv_variables
1099	#D-* Purpose: unset environement variables after execution
1100	#D-* Examples:
1101	#D-
1102	function IGCM_sys_desactiv_variables {
1103	IGCM_debug_PushStack "IGCM_sys_desactiv_variables"
1104	if ( $DEBUG_sys ) ; then
1105	echo "IGCM_sys_desactiv_variables"
1106	fi
1107	# --------------------------------------------------------------------
1108	#D- MPI specifications
1109	# --------------------------------------------------------------------
1110
1111	# --------------------------------------------------------------------
1112	#D- Other specifications
1113	# --------------------------------------------------------------------
1114
1115	IGCM_debug_PopStack "IGCM_sys_desactiv_variables"
1116	}
1117
1118	############################################################
1119	# Update job headers to be used by the scheduler
1120
1121	#D-#==================================================
1122	#D-function IGCM_sys_updateHeaders
1123	#D-* Purpose: Update job headers to be used by the scheduler
1124	#D-* Examples: IGCM_sys_updateHeaders /path/to/Job_MYEXP
1125	#D-
1126	function IGCM_sys_updateHeaders {
1127	IGCM_debug_PushStack "IGCM_sys_updateHeaders"
1128	if ( $DEBUG_sys ) ; then
1129	echo "IGCM_sys_updateHeaders"
1130	fi
1131	typeset file
1132	file=$1
1133
1134	if [ ${executionType} -eq 1 ] ; then
1135	# MPMD + MPI
1136	sed -e "/::openMPthreads::/d" \
1137	-e "s/::JobNumProcTot::/${coreNumber}/" \
1138	-e "/#MSUB -x/d" \
1139	-e "/--cpu_bind=none/d" \
1140	${file} > ${file}.tmp
1141
1142	elif [ ${executionType} -eq 2 ] ; then
1143	# MPMD + MPI + OMP : mpirun/ccc_mprun/error
1144	if ( [ "X${config_UserChoices_ExecutionModeOnCurie}" = "Xmpirun" ] ) ; then
1145	sed -e "/::openMPthreads::/d" \
1146	-e "s/::JobNumProcTot::/${coreNumber}/" \
1147	${file} > ${file}.tmp
1148	elif ( [ "X${config_UserChoices_ExecutionModeOnCurie}" = "X" ] \|\| [ "X${config_UserChoices_ExecutionModeOnCurie}" = "Xccc_mprun" ] ) ; then
1149	sed -e "/::openMPthreads::/d" \
1150	-e "s/::JobNumProcTot::/${coreNumber}/" \
1151	-e "/--cpu_bind=none/d" \
1152	${file} > ${file}.tmp
1153	else
1154	IGCM_debug_Print 1 "You have to set ExecutionModeOnCurie=ccc_mprun or mpirun in config.card"
1155	IGCM_debug_Exit "IGCM_sys_updateHeaders"
1156	fi
1157	elif [ ${executionType} -eq 3 ] ; then
1158	# SPMD + MPI/OMP
1159	sed -e "s/::openMPthreads::/${openMPthreads}/" \
1160	-e "s/::JobNumProcTot::/${mpiTasks}/" \
1161	-e "/#MSUB -x/d" \
1162	-e "/--cpu_bind=none/d" \
1163	${file} > ${file}.tmp
1164
1165	elif [ ${executionType} -eq 4 ] ; then
1166	# SPMD + MPI only
1167	sed -e "s/::JobNumProcTot::/${mpiTasks}/" \
1168	-e "/::openMPthreads::/d" \
1169	-e "/#MSUB -x/d" \
1170	-e "/--cpu_bind=none/d" \
1171	${file} > ${file}.tmp
1172
1173	elif [ ${executionType} -eq 5 ] ; then
1174	# SPMD + OMP only
1175	sed -e "s/::openMPthreads::/${openMPthreads}/" \
1176	-e "/::JobNumProcTot::/d" \
1177	-e "/#MSUB -x/d" \
1178	-e "/--cpu_bind=none/d" \
1179	${file} > ${file}.tmp
1180
1181	elif [ ${executionType} -eq 6 ] ; then
1182	# SEQUENTIAL THEN
1183	sed -e "s/::JobNumProcTot::/1/" \
1184	-e "/::openMPthreads::/d" \
1185	-e "/#MSUB -x/d" \
1186	-e "/--cpu_bind=none/d" \
1187	${file} > ${file}.tmp
1188
1189	fi
1190
1191	IGCM_sys_Mv ${file}.tmp ${file}
1192
1193	IGCM_debug_PopStack "IGCM_sys_updateHeaders"
1194	}
1195
1196	############################################################
1197	# Build MPI/OMP scripts run file (dummy function)
1198
1199	#D-#==================================================
1200	#D-function IGCM_sys_build_run_file
1201	#D-* Purpose: build run file (deprecated)
1202	#D-* Examples:
1203	#D-
1204	function IGCM_sys_build_run_file {
1205	IGCM_debug_Print 3 " dummy function : IGCM_sys_build_run_file "
1206	}
1207
1208	############################################################
1209	# Build MPI/OMP scripts
1210
1211	#D-#==================================================
1212	#D-function IGCM_sys_build_execution_scripts
1213	#D-* Purpose: build execution scripts to be launch by ${HOST_MPIRUN_COMMAND}
1214	#D-* Examples:
1215	#D-
1216	function IGCM_sys_build_execution_scripts
1217	{
1218	IGCM_debug_PushStack "IGCM_sys_build_execution_scripts" $@
1219	if ( $DEBUG_sys ) ; then
1220	echo "IGCM_sys_build_execution_scripts " $@
1221	fi
1222
1223	EXECUTION=${HOST_MPIRUN_COMMAND}
1224
1225	if ( ${OK_PARA_MPMD} ) ; then
1226
1227	# MPMD mode
1228	# 1 MPI only
1229	# 2 MPI/OpenMP mpirun method
1230	# 3 MPI/OpenMP ccc_mprun method
1231
1232	if [ -f run_file ] ; then
1233	IGCM_sys_Rm -f run_file
1234	fi
1235	touch run_file
1236
1237	# case 1 : Only MPI (MPMD)
1238	if ( ! ${OK_PARA_OMP} ) ; then
1239
1240	# Build run_file
1241
1242	# First loop on the components for the coupler ie oasis (only if oasis3)
1243	# the coupler ie oasis3 must be the first one
1244	for comp in ${config_ListOfComponents[*]} ; do
1245
1246	eval ExeNameIn=\${config_Executable_${comp}[0]}
1247	eval ExeNameOut=\${config_Executable_${comp}[1]}
1248
1249	# for CPL component only
1250	if [ "X${comp}" = "XCPL" ] && [ "X${ExeNameOut}" != X\"\" ] ; then
1251
1252	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1253	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
1254	echo "${comp_proc_mpi_loc} ./${ExeNameOut}" >> run_file
1255	fi
1256	done
1257
1258	# Then second loop on the components
1259	for comp in ${config_ListOfComponents[*]} ; do
1260
1261	eval ExeNameIn=\${config_Executable_${comp}[0]}
1262	eval ExeNameOut=\${config_Executable_${comp}[1]}
1263
1264	# Only if we really have an executable for the component and not the coupler ie oasis:
1265	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${comp}" != "XCPL" ] ) ; then
1266
1267	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1268	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
1269	echo "${comp_proc_mpi_loc} ./${ExeNameOut}" >> run_file
1270	fi
1271	done
1272
1273	EXECUTION="${HOST_MPIRUN_COMMAND} -f ./run_file"
1274
1275	IGCM_sys_Chmod u+x run_file
1276	if ( $DEBUG_sys ) ; then
1277	echo "run_file contains : "
1278	cat run_file
1279	fi
1280
1281	elif ( [ "X${config_UserChoices_ExecutionModeOnCurie}" = "Xmpirun" ] ) ; then
1282
1283	# 2 MPI/OpenMP mpirun method
1284
1285	# Use of mpirun instead of ccc_mprun
1286	EXECUTION="time mpirun"
1287
1288	# Hosts treatment
1289	${EXECUTION} hostname \| sort \| uniq > hosts.tmp
1290
1291	i=0
1292	rm -f hosts rankfile
1293	IGCM_debug_Print 1 "sys Irene, Hosts available :"
1294	for nodes in `cat hosts.tmp`
1295	do
1296	host[$i]=$nodes
1297	echo "${host[$i]}" >> hosts
1298	IGCM_debug_Print 1 ${host[$i]}
1299	i=$((i+1))
1300	done
1301	rm -f hosts.tmp
1302
1303	listnodes=${host[*]}
1304
1305	EXECUTION="${EXECUTION} -hostfile hosts -rankfile rankfile"
1306
1307	# Initialisation
1308	rank=0
1309	current_core=0
1310	init_exec=n
1311
1312	# Loop on the components
1313	for comp in ${config_ListOfComponents[*]} ; do
1314
1315	eval ExeNameIn=\${config_Executable_${comp}[0]}
1316	eval ExeNameOut=\${config_Executable_${comp}[1]}
1317
1318	# Not possible if oasis has an executable (i.e old version of oasis3)
1319	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${comp}" = "XCPL" ] ) ; then
1320	IGCM_debug_Exit "ERROR MPMD with hybrid MPI-OpenMP is not available with oasis3 version"
1321	IGCM_debug_Print 2 "Only available with oasis3-MCT version coupler"
1322	IGCM_debug_Verif_Exit
1323	fi
1324
1325	# Only if we really have an executable for the component :
1326	if [ "X${ExeNameOut}" != X\"\" ] ; then
1327
1328	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1329	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
1330	eval comp_proc_nod_loc=\${${comp}_PROC_NOD}
1331
1332	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
1333	echo "" >> script_${ExeNameOut}.ksh
1334	if [ ${comp_proc_omp_loc} -gt 1 ] ; then
1335
1336	# Check if the number of threads is correct
1337	case ${comp_proc_omp_loc} in
1338	2\|4\|8\|16)
1339	IGCM_debug_Print 1 "You run ${ExeNameOut} on ${comp_proc_omp_loc} OMP threads"
1340	;;
1341	*)
1342	IGCM_debug_Exit "ERROR with OMP parameters !"
1343	IGCM_debug_Print 2 "${comp_proc_omp_loc} is not possible as number of OMP threads"
1344	IGCM_debug_Print 2 "Only 2,4,8,16 as number of OMP threads are possible "
1345	IGCM_debug_Verif_Exit
1346	;;
1347	esac
1348	echo "export KMP_STACKSIZE=3g" >> script_${ExeNameOut}.ksh
1349	echo "export KMP_LIBRARY=turnaround" >> script_${ExeNameOut}.ksh
1350	echo "export MKL_SERIAL=YES" >> script_${ExeNameOut}.ksh
1351	echo "OMP_NUM_THREADS=${comp_proc_omp_loc}" >> script_${ExeNameOut}.ksh
1352	fi
1353
1354	#echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
1355	echo "(( MYMPIRANK = OMPI_COMM_WORLD_RANK )) " >> script_${ExeNameOut}.ksh
1356	echo "MYMPIRANK=\$(printf '%3.3d\n' \${MYMPIRANK})" >> script_${ExeNameOut}.ksh
1357	echo "./${ExeNameOut} > out_${ExeNameOut}.out.\${MYMPIRANK} 2>out_${ExeNameOut}.err.\${MYMPIRANK}" >> script_${ExeNameOut}.ksh
1358	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
1359
1360	if [ ${init_exec} = y ] ; then
1361	EXECUTION="${EXECUTION} : -np ${comp_proc_mpi_loc} ./script_${ExeNameOut}.ksh"
1362	else
1363	EXECUTION="${EXECUTION} -np ${comp_proc_mpi_loc} ./script_${ExeNameOut}.ksh"
1364	init_exec=y
1365	fi
1366
1367	# Build rankfile : method used to assign cores and nodes for the MPI process
1368	# Ex :
1369	#rank 0=curie5296 slot=0,1,2,3
1370	#rank 1=curie5296 slot=4,5,6,7
1371	# Example of final command :
1372	# mpirun -hostfile hosts -rankfile rankfile -np 27 ./script_lmdz.x.ksh : -np 5 ./script_opa.xx.ksh
1373	# with script_lmdz.x.ksh :
1374	# #!/bin/ksh
1375	#export KMP_STACKSIZE=3g
1376	#export KMP_LIBRARY=turnaround
1377	#export MKL_SERIAL=YES
1378	#OMP_NUM_THREADS=4
1379	#./lmdz.x
1380	if [ ${comp_proc_nod_loc} -gt 1 ] ; then
1381	(( offset_comp_proc_loc = NB_CORE_PER_NODE / (comp_proc_mpi_loc / comp_proc_nod_loc) ))
1382	else
1383	(( offset_comp_proc_loc = comp_proc_omp_loc ))
1384	fi
1385	for nb_proc_mpi in `seq 0 $(($comp_proc_mpi_loc-1))`; do
1386	(( index_host = current_core / NB_CORE_PER_NODE ))
1387	host_value=${host[${index_host}]}
1388	(( slot = current_core % NB_CORE_PER_NODE ))
1389	virg=","
1390	string_final=""
1391	for index in `seq $slot $(($slot+$comp_proc_omp_loc-1))`; do
1392	string=$index$virg
1393	string_final=$string_final$string
1394	done
1395	string_final=$( echo $string_final \| sed "s/.$//" )
1396	echo "rank $rank=$host_value slot=$string_final" >> rankfile
1397	(( rank = rank + 1 ))
1398	(( current_core = current_core + offset_comp_proc_loc ))
1399	done
1400	fi
1401	done
1402
1403	elif ( [ "X${config_UserChoices_ExecutionModeOnCurie}" = "X" ] \|\| [ "X${config_UserChoices_ExecutionModeOnCurie}" = "Xccc_mprun" ] ) ; then
1404
1405	# 3 MPI/OpenMP ccc_mprun method
1406
1407	# MPI-OpenMP (MPMD)
1408	# example of run_file
1409	# 71-8 env OMP_NUM_THREADS=8 ./script_lmdz.x.ksh
1410	# 480-1 env OMP_NUM_THREADS=1 ./script_opa.xx.ksh
1411	# 1-1 env OMP_NUM_THREADS=1 ./script_xios.x.ksh
1412
1413	# Loop on the components
1414	for comp in ${config_ListOfComponents[*]} ; do
1415
1416	eval ExeNameIn=\${config_Executable_${comp}[0]}
1417	eval ExeNameOut=\${config_Executable_${comp}[1]}
1418
1419	# Not possible if oasis has an executable (i.e old version of oasis3)
1420	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${comp}" = "XCPL" ] ) ; then
1421	IGCM_debug_Exit "ERROR MPMD with hybrid MPI-OpenMP is not available with oasis3 version"
1422	IGCM_debug_Print 2 "Only available with oasis3-MCT version coupler"
1423	IGCM_debug_Verif_Exit
1424	fi
1425
1426	# Only if we really have an executable for the component :
1427	if [ "X${ExeNameOut}" != X\"\" ] ; then
1428
1429	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1430	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
1431
1432	# Build script files
1433
1434	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
1435	echo "" >> script_${ExeNameOut}.ksh
1436	if [ ${comp_proc_omp_loc} -gt 1 ] ; then
1437
1438	# Check if the number of threads is correct
1439	case ${comp_proc_omp_loc} in
1440	2\|4\|8\|12\|16\|24)
1441	IGCM_debug_Print 1 "You run ${ExeNameOut} on ${comp_proc_omp_loc} OMP threads"
1442	;;
1443	*)
1444	IGCM_debug_Exit "ERROR with OMP parameters !"
1445	IGCM_debug_Print 2 "${comp_proc_omp_loc} is not possible as number of OMP threads"
1446	IGCM_debug_Print 2 "Only 2,4,8,12,16,24 as number of OMP threads are possible "
1447	IGCM_debug_Verif_Exit
1448	;;
1449	esac
1450	### only for LMDZ?
1451	echo "export KMP_STACKSIZE=3g" >> script_${ExeNameOut}.ksh
1452	echo "export KMP_LIBRARY=turnaround" >> script_${ExeNameOut}.ksh
1453	echo "export MKL_SERIAL=YES" >> script_${ExeNameOut}.ksh
1454	echo "OMP_NUM_THREADS=${comp_proc_omp_loc}" >> script_${ExeNameOut}.ksh
1455
1456	fi
1457
1458	# to have out/err per process on different files
1459	echo "./${ExeNameOut} > out_${ExeNameOut}.out.\${SLURM_PROCID} 2>out_${ExeNameOut}.err.\${SLURM_PROCID}" >> script_${ExeNameOut}.ksh
1460	#echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
1461
1462	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
1463
1464	# Complete run_file
1465
1466	echo "${comp_proc_mpi_loc}-${comp_proc_omp_loc} env OMP_NUM_THREADS=${comp_proc_omp_loc} ./script_${ExeNameOut}.ksh " >>run_file
1467
1468	fi
1469	done
1470
1471	## variable added to stop after 60s instead of 600s by default.
1472	## This is used when no error comes from executables and when something stopped an executable without notice.
1473	export SLURM_WAIT=60
1474
1475	## module advised by TGCC (instead of 2 variables)
1476	module load feature/bridge/heterogenous_mpmd
1477
1478	# set EXECUTION for ccc_mprun case (similar to #1 : MPI only MPMD method)
1479
1480	EXECUTION="${HOST_MPIRUN_COMMAND} -f ./run_file"
1481
1482	IGCM_sys_Chmod u+x run_file
1483	if ( $DEBUG_sys ) ; then
1484	echo "run_file contains : "
1485	cat run_file
1486	fi
1487
1488	else
1489
1490	IGCM_debug_Print 1 "sys Curie : choose mpirun or ccc_mprun in config.card for ExecutionModeOnCurie"
1491
1492	fi # 1 2 or 3 if ${OK_PARA_MPMD}
1493
1494	else
1495	# Only one executable (SPMD mode).
1496
1497	for comp in ${config_ListOfComponents[*]} ; do
1498
1499	# Only if we really have an executable for the component :
1500	eval ExeNameOut=\${config_Executable_${comp}[1]}
1501	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${ExeNameOut}" != "Xinca.dat" ] ) ; then
1502
1503	# Build script files
1504
1505	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
1506	echo "" >> script_${ExeNameOut}.ksh
1507	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
1508
1509	if ( ${OK_PARA_OMP} ) ; then
1510	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
1511	echo "" >> script_${ExeNameOut}.ksh
1512	echo "export KMP_STACKSIZE=3g" >> script_${ExeNameOut}.ksh
1513	echo "export KMP_LIBRARY=turnaround" >> script_${ExeNameOut}.ksh
1514	echo "export MKL_SERIAL=YES" >> script_${ExeNameOut}.ksh
1515	echo "OMP_NUM_THREADS=${comp_proc_omp_loc}" >> script_${ExeNameOut}.ksh
1516	fi
1517
1518	if ( ${OK_PARA_MPI} ) ; then
1519	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1520	# Default : ccc_mprun used if nb_proc gt 1
1521	# to have out/err per process on different files
1522	echo "./${ExeNameOut} > out_${ExeNameOut}.out.\${SLURM_PROCID} 2>out_${ExeNameOut}.err.\${SLURM_PROCID}" >> script_${ExeNameOut}.ksh
1523	#echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
1524	EXECUTION="${HOST_MPIRUN_COMMAND} -n ${comp_proc_mpi_loc} ./script_${ExeNameOut}.ksh"
1525	else
1526	# Default : ccc_mprun is NOT used if nb_proc eq 1
1527	# to have out/err per process on different files
1528	echo "./${ExeNameOut} > out_${ExeNameOut}.out 2>out_${ExeNameOut}.err" >> script_${ExeNameOut}.ksh
1529	#echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
1530	EXECUTION="/usr/bin/time ./script_${ExeNameOut}.ksh"
1531	fi
1532
1533	IGCM_debug_Print 1 "sys Irene : script_${ExeNameOut}.ksh contains"
1534	cat script_${ExeNameOut}.ksh
1535
1536	fi
1537
1538	done
1539
1540	fi # ${OK_PARA_MPMD}
1541
1542	IGCM_debug_Print 1 "sys Irene : execution command is "
1543	IGCM_debug_Print 1 "$EXECUTION"
1544
1545	IGCM_debug_PopStack "IGCM_sys_build_execution_scripts"
1546	}
1547
1548	#D-#==================================================
1549	#D-function IGCM_sys_check_path
1550	#D-* Purpose: check that RUN_DIR_PATH that will be removed on some machine
1551	#D-* do not point to an important use directory. Stop immediately in that case.
1552	#D-* Examples:
1553	#D-
1554	function IGCM_sys_check_path {
1555	IGCM_debug_PushStack "IGCM_sys_check_path"
1556	if ( $DEBUG_sys ) ; then
1557	echo "IGCM_sys_check_path"
1558	fi
1559
1560	if ( [ X${RUN_DIR_PATH} = X${HOME} ] \|\| [ X${RUN_DIR_PATH} = X${WORK} ] \|\| [ X${RUN_DIR_PATH} = X${SCRATCH} ] \|\| [ X${RUN_DIR_PATH} = X${STORE} ] ) ; then
1561	IGCM_debug_Print 1 "Variable RUN_DIR_PATH is pointing to an important directory : ${RUN_DIR_PATH}"
1562	IGCM_debug_Print 1 "Please check the RUN_DIR_PATH definition in your Job : Job_${config_UserChoices_JobName}"
1563	IGCM_debug_Exit "This will stop the job"
1564	fi
1565	IGCM_debug_PopStack "IGCM_sys_check_path"
1566	}
1567
1568	#D-#==================================================
1569	#D-function IGCM_sys_check_quota
1570	#D-* Purpose: check user quota. Stop the simulation if quota above 90%
1571	#D-* Examples:
1572	#D-
1573	function IGCM_sys_check_quota {
1574	IGCM_debug_PushStack "IGCM_sys_check_quota"
1575	if ( $DEBUG_sys ) ; then
1576	echo "IGCM_sys_check_quota"
1577	fi
1578	# Limit of quota (in %)
1579	limit_quota=90
1580
1581	# Check of the volume
1582	volume_quota=$(ccc_quota \| grep ' scratch' \| gawk '{print $2}')
1583	volume_avail=$(ccc_quota \| grep ' scratch' \| gawk '{print $3}')
1584
1585	if ( [ ! X${volume_quota} = X ] && [ ! ${volume_quota} = "-" ] ) ; then
1586
1587	unit_avail=${volume_avail: -1}
1588	unit_quota=${volume_quota: -1}
1589
1590	if [ "${unit_quota}" = "*" ] ; then
1591	IGCM_debug_Print 1 "Please, check your quota of volume on scratch"
1592	IGCM_debug_Print 1 "More than 100% of your quota is used"
1593	IGCM_debug_Print 1 "Use the ccc_quota command to check"
1594	IGCM_debug_Print 1 "You must have more than 10% available to run"
1595	IGCM_debug_Exit "Not enough space to run ! STOP HERE"
1596	IGCM_debug_Verif_Exit
1597	fi
1598
1599	temp_avail=${volume_avail%%${volume_avail: -1}*}
1600	temp_quota=${volume_quota%%${volume_quota: -1}*}
1601
1602	if [ ! ${unit_avail} = ${unit_quota} ] ; then
1603
1604	# Convertion
1605	if [ ${volume_avail: -1} = "T" ] ; then
1606	(( temp_avail = temp_avail * 1000000000000 ))
1607	elif [ ${volume_avail: -1} = "G" ] ; then
1608	(( temp_avail = temp_avail * 1000000000 ))
1609	elif [ ${volume_avail: -1} = "M" ] ; then
1610	(( temp_avail = temp_avail * 1000000 ))
1611	elif [ ${volume_avail: -1} = "k" ] ; then
1612	(( temp_avail = temp_avail * 1000 ))
1613	else
1614	(( temp_avail = volume_avail ))
1615	fi
1616	if [ ${volume_quota: -1} = "T" ] ; then
1617	(( temp_quota = temp_quota * 1000000000000 ))
1618	elif [ ${volume_quota: -1} = "G" ] ; then
1619	(( temp_quota = temp_quota * 1000000000 ))
1620	elif [ ${volume_quota: -1} = "M" ] ; then
1621	(( temp_quota = temp_quota * 1000000 ))
1622	elif [ ${volume_quota: -1} = "k" ] ; then
1623	(( temp_quota = temp_quota * 1000 ))
1624	else
1625	(( temp_quota = volume_quota ))
1626	fi
1627	fi
1628
1629	quota_volume=$(echo "scale=2 ; $temp_quota/$temp_avail*100" \| bc)
1630	# echo "volume ratio is " $quota_volume
1631
1632	if [ ${quota_volume} -ge ${limit_quota} ] ; then
1633	IGCM_debug_Print 1 "Please, check your quota of volume on scratch"
1634	IGCM_debug_Print 1 "${quota_volume}% of your quota is used"
1635	IGCM_debug_Print 1 "Use the ccc_quota command to check"
1636	IGCM_debug_Print 1 "You must have more than 10% available to run"
1637	IGCM_debug_Exit "Not enough space to run ! STOP HERE"
1638	IGCM_debug_Verif_Exit
1639	fi
1640
1641	fi
1642
1643	# Check of the number of inodes
1644
1645	inode_quota=$(ccc_quota \| grep ' scratch' \| gawk '{print $6}')
1646	inode_avail=$(ccc_quota \| grep ' scratch' \| gawk '{print $7}')
1647
1648	if ( [ ! X${inode_quota} = X ] && [ ! ${inode_quota} = "-" ] ) ; then
1649
1650	unit_avail=${inode_avail: -1}
1651	unit_quota=${inode_quota: -1}
1652
1653	if [ "${unit_quota}" = "*" ] ; then
1654	IGCM_debug_Print 1 "Please, check your quota of inode on scratch"
1655	IGCM_debug_Print 1 "More than 100% of your quota is used"
1656	IGCM_debug_Print 1 "Use the ccc_quota command to check"
1657	IGCM_debug_Print 1 "You must have more than 10% available to run"
1658	IGCM_debug_Exit "Not enough space to run ! STOP HERE"
1659	IGCM_debug_Verif_Exit
1660	fi
1661
1662	temp_avail=${inode_avail%%${inode_avail: -1}*}
1663	temp_quota=${inode_quota%%${inode_quota: -1}*}
1664
1665	if [ ! ${unit_avail} = ${unit_quota} ] ; then
1666
1667	# Convertion
1668	if [ ${inode_avail: -1} = "T" ] ; then
1669	(( temp_avail = temp_avail * 1000000000000 ))
1670	elif [ ${inode_avail: -1} = "G" ] ; then
1671	(( temp_avail = temp_avail * 1000000000 ))
1672	elif [ ${inode_avail: -1} = "M" ] ; then
1673	(( temp_avail = temp_avail * 1000000 ))
1674	elif [ ${inode_avail: -1} = "k" ] ; then
1675	(( temp_avail = temp_avail * 1000 ))
1676	else
1677	(( temp_avail = inode_avail ))
1678	fi
1679
1680	if [ ${inode_quota: -1} = "T" ] ; then
1681	(( temp_quota = temp_quota * 1000000000000 ))
1682	elif [ ${inode_quota: -1} = "G" ] ; then
1683	(( temp_quota = temp_quota * 1000000000 ))
1684	elif [ ${inode_quota: -1} = "M" ] ; then
1685	(( temp_quota = temp_quota * 1000000 ))
1686	elif [ ${inode_quota: -1} = "k" ] ; then
1687	(( temp_quota = temp_quota * 1000 ))
1688	else
1689	(( temp_quota = inode_quota ))
1690	fi
1691	fi
1692	quota_inode=$(echo "scale=2 ; $temp_quota/$temp_avail*100" \| bc)
1693	# echo "inode ratio is " $quota_inode
1694
1695	if [ ${quota_inode} -ge ${limit_quota} ] ; then
1696	IGCM_debug_Print 1 "Please, check your quota of inode on scratch"
1697	IGCM_debug_Print 1 "${quota_inode}% of your quota is used"
1698	IGCM_debug_Print 1 "Use the ccc_quota command to check"
1699	IGCM_debug_Print 1 "You must have more than 10% available to run"
1700	IGCM_debug_Exit "Not enough space to run ! STOP HERE"
1701	IGCM_debug_Verif_Exit
1702	fi
1703	fi
1704	IGCM_debug_PopStack "IGCM_sys_check_quota"
1705	}
1706
1707	#D-#==================================================
1708	#D-function IGCM_sys_projectAccounting
1709	#D-* Purpose: store project accounting information in a file
1710	#D-* Examples:
1711	#D-
1712	function IGCM_sys_projectAccounting {
1713	IGCM_debug_PushStack "IGCM_sys_projectAccounting"
1714	if ( $DEBUG_sys ) ; then
1715	echo "IGCM_sys_check_quota"
1716	fi
1717
1718	ssh irene191 /usr/bin/ccc_myproject > $1
1719
1720	IGCM_debug_PopStack "IGCM_sys_projectAccounting"
1721	}
1722
1723	#D-#==================================================
1724	#D-function IGCM_sys_getJobSchedulerID
1725	#D-* Purpose: Get the job ID during execution
1726	#D-* Examples: IGCM_sys_getJobSchedulerID jobSchedulerID
1727	#D-
1728	function IGCM_sys_getJobSchedulerID {
1729	IGCM_debug_PushStack "IGCM_sys_getJobSchedulerID"
1730	if ( $DEBUG_sys ) ; then
1731	echo "IGCM_sys_getJobSchedulerID"
1732	fi
1733
1734	eval ${1}=${BRIDGE_MSUB_JOBID}
1735
1736	IGCM_debug_PopStack "IGCM_sys_getJobSchedulerID"
1737	}
1738
1739	#D-#==================================================
1740	#D-function IGCM_sys_GetJobID
1741	#D-* Purpose: Get the job ID from the JobName
1742	#D-* Examples: IGCM_sys_GetJobID ${JobName} ${TargetUsr} JobID
1743	#D-
1744	function IGCM_sys_GetJobID {
1745	IGCM_debug_PushStack "IGCM_sys_GetJobID"
1746	if ( $DEBUG_sys ) ; then
1747	echo "IGCM_sys_GetJobID"
1748	fi
1749
1750	# With -f option, the full job name is given in the last column
1751	ID=$( ccc_mstat -f -u $2 \| \
1752	gawk -v JobName=$1 '( $NF ~ JobName ) { print $1 }' )
1753
1754	eval ${3}=${ID}
1755	IGCM_debug_PopStack "IGCM_sys_GetJobID"
1756	}
1757
1758	#D-#==================================================
1759	#D-function IGCM_sys_CountJobInQueue
1760	#D-* Purpose: Count number of users job
1761	#D-* Examples: IGCM_sys_CountJobInQueue ${JobName} NbRun
1762	#D-
1763	function IGCM_sys_CountJobInQueue {
1764	IGCM_debug_PushStack "IGCM_sys_CountJobInQueue"
1765	if ( $DEBUG_sys ) ; then
1766	echo "IGCM_sys_CountJobInQueue"
1767	fi
1768
1769	# With -f option, the full job name is given in the last column
1770	NbRun=$( ccc_mstat -f \| gawk -v JobName=$1 'BEGIN { x=0 } ( $NF ~ JobName ) { x=x+1 } END { print x }' )
1771
1772	eval ${2}=${NbRun}
1773
1774	IGCM_debug_PopStack "IGCM_sys_CountJobInQueue"
1775	}
1776
1777	#D-#==================================================
1778	#D-function IGCM_sys_ListJobInQueue
1779	#D-* Purpose: Produce a list of users computing jobs (excluding post-processing)
1780	#D-* Examples: IGCM_sys_ListJobInQueue ${User} JobNameList
1781	#D-
1782	function IGCM_sys_ListJobInQueue {
1783	IGCM_debug_PushStack "IGCM_sys_ListJobInQueue"
1784	if ( $DEBUG_sys ) ; then
1785	echo "IGCM_sys_ListJobInQueue"
1786	fi
1787
1788	# With -f option, the full job name is given in the last column
1789	set -A JobList $( ccc_mstat -f \| gawk -v User=$1 \
1790	'( $2 == User && \
1791	$NF != /TS/ && \
1792	$NF !~ /PACK/ && \
1793	$NF !~ /REBUILD/ && \
1794	$NF !~ /pack/ ) \
1795	{ print $NF }' \| sed -e "s/\(.\)\.[0-9]/\1/" )
1796
1797	eval set -A ${2} ${JobList[*]}
1798
1799	IGCM_debug_PopStack "IGCM_sys_ListJobInQueue"
1800	}
1801
1802	#D-#==================================================
1803	#D-function IGCM_sys_atlas
1804	#D-* Purpose: encapsulate atlas call so as to manage error code and curie specificity
1805	#D-* Examples:
1806	#D-
1807	function IGCM_sys_atlas {
1808	IGCM_debug_PushStack "IGCM_sys_atlas" $@
1809	if ( $DEBUG_sys ) ; then
1810	echo "IGCM_sys_atlas :" $@
1811	fi
1812
1813	typeset status
1814
1815	\ccc_mprun atlas $@ > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
1816	status=$?
1817	if [ ${status} -gt 0 ] ; then
1818	echo "IGCM_sys_atlas : error code ${status}"
1819	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
1820	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
1821	IGCM_debug_PopStack "IGCM_sys_atlas"
1822	return 1
1823	else
1824	IGCM_debug_PopStack "IGCM_sys_atlas"
1825	return 0
1826	fi
1827
1828	IGCM_debug_PopStack "IGCM_sys_atlas"
1829	}
1830
1831	#D-#==================================================
1832	#D-function IGCM_sys_rebuild_nemo
1833	#D-* Purpose: rebuild nemo parallel files with nemo specific rebuild on curie
1834	#D-* Examples:
1835	#D-
1836
1837	function IGCM_sys_rebuild_nemo {
1838	IGCM_debug_PushStack "IGCM_sys_rebuild_nemo" $@
1839	if ( $DEBUG_sys ) ; then
1840	echo "IGCM_sys_rebuild_nemo :" $@
1841	fi
1842
1843	$( ccc_home -u igcmg)/Tools/irene/bin/rebuild_nemo ${1} ${2}
1844
1845	IGCM_debug_PopStack "IGCM_sys_rebuild_nemo"
1846	}
1847

Note: See TracBrowser for help on using the repository browser.

Download in other formats: