Context Navigation

source: trunk/libIGCM/libIGCM_sys/libIGCM_sys_irene.ksh @ 1435

Last change on this file since 1435 was 1433, checked in by jgipsl, 6 years ago
Added headers for irene on some post-treatment jobs. Source ~p86ipsl/.env_irene to load all modules. For the moment no separation has been done between netfcdf and intel as done on curie but might be needed later.
Property licence set to The following licence information concerns ONLY the libIGCM tools ================================================================== Copyright © Centre National de la Recherche Scientifique CNRS Commissariat à l'Énergie Atomique CEA libIGCM : Library for Portable Models Computation of IGCM Group. IGCM Group is the french IPSL Global Climate Model Group. This library is a set of shell scripts and functions whose purpose is the management of the initialization, the launch, the transfer of output files, the post-processing and the monitoring of datas produce by any numerical program on any plateforme. This software is governed by the CeCILL license under French law and abiding by the rules of distribution of free software. You can use, modify and/ or redistribute the software under the terms of the CeCILL license as circulated by CEA, CNRS and INRIA at the following URL "http://www.cecill.info". As a counterpart to the access to the source code and rights to copy, modify and redistribute granted by the license, users are provided only with a limited warranty and the software's author, the holder of the economic rights, and the successive licensors have only limited liability. In this respect, the user's attention is drawn to the risks associated with loading, using, modifying and/or developing or reproducing the software by the user in light of its specific status of free software, that may mean that it is complicated to manipulate, and that also therefore means that it is reserved for developers and experienced professionals having in-depth computer knowledge. Users are therefore encouraged to load and test the software's suitability as regards their requirements in conditions enabling the security of their systems and/or data to be ensured and, more generally, to use and operate it in the same conditions as regards security. The fact that you are presently reading this means that you have had knowledge of the CeCILL license and that you accept its terms. Property svn:keywords set to `Revision Author Date`
File size: 62.0 KB

Line
1	#!/bin/ksh
2
3	#**************************************************************
4	# Author: Sebastien Denvil, Martial Mancip, Arnaud Caubel
5	# Contact: Arnaud.Caubel__at__lsce.ipsl.fr
6	# $Revision:: $ Revision of last commit
7	# $Author:: $ Author of last commit
8	# $Date:: $ Date of last commit
9	# IPSL (2006)
10	# This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
11	#
12	#**************************************************************
13
14	#=========================================================
15	# The documentation of this file can be automatically generated
16	# if you use the prefix #D- for comments to be extracted.
17	# Extract with command: cat lib* \| grep "^#D-" \| cut -c "4-"
18	#=========================================================
19
20	#D-#==================================================
21	#D-LibIGCM_sys for Curie
22	#D-#==================================================
23	#D-
24	#D- This ksh library if a layer under some usefull
25	#D-environment variables and shell commands.
26	#D-All those definitions depend on host particularities.
27	#D-It manages a stack mechanism and test validity of operations.
28	#D-All function described bellow must be prefixed by IGCM_sys.
29
30	#====================================================
31	# libIGCM_sys PARAMETERS
32	#====================================================
33
34	#====================================================
35	# set DEBUG_sys to true to output calls of function
36	typeset -r DEBUG_sys=${DEBUG_sys:=true}
37
38	#====================================================
39	# Turn in dry run mode ? (sys_Put_Rest, sys_Put_Out, sys_Get)
40	typeset -r DRYRUN=${DRYRUN:=0}
41
42	# YOU MUST COMPILE YOUR EXE FILES FOR DRYRUN MODE !
43	# -------------------------------------------------------------------------------------
44	# \| DRYRUN= \| Date computations, \| sys_Get \| Exe \| sys_Put_Out; sys_Put_Rest \|
45	# \| \| Cp/Exe/param/files \| \| \| \|
46	# \| \| Chmod Qsub \| \| \| \|
47	# -------------------------------------------------------------------------------------
48	# \| 0 \| yes \| yes \| yes \| yes \|
49	# -------------------------------------------------------------------------------------
50	# \| 1 \| yes \| yes \| yes \| no \|
51	# -------------------------------------------------------------------------------------
52	# \| 2 \| yes \| yes \| no \| no \|
53	# -------------------------------------------------------------------------------------
54	# \| 3 \| yes \| no \| no \| no \|
55	# -------------------------------------------------------------------------------------
56
57	#=====================================================
58	# Global Variables :
59	#=====================================================
60	# Language : "fr" or "en"
61	typeset -r MYLANG="fr"
62
63	#=====================================================
64	# Host user names project maxCpuTime
65	# $hostname ou hostname
66	typeset HOST=${HOST:=$( hostname )}
67	# $username ou whoami
68	typeset LOGIN=${LOGIN:=$( whoami )}
69	# $hostname of the MASTER job
70	typeset MASTER=curie
71	# add default project on curie
72	typeset PROJECT=$(echo ${BRIDGE_MSUB_PROJECT:=gen0826} \| cut -d@ -f1 )
73	# jobWarningDelay in seconds
74	typeset jobWarningDelay=${BRIDGE_MSUB_MAXTIME}
75
76	#D-
77	#D-#==================================================
78	#D-Program used in libIGCM
79	#D-#==================================================
80
81	# Submit command
82	typeset SUBMIT=${SUBMIT:=ccc_msub}
83	# rsync with path
84	typeset -r RSYNC=/usr/bin/rsync
85	# RSYNC_opt args to rsync
86	typeset -r RSYNC_opt="-va"
87	# ie storage filesystem
88	typeset -r STOREHOST=${MASTER}
89	typeset -r REMOTE_RSYNC=/usr/bin/rsync
90
91	#====================================================
92	# Set environment tools (ferret, nco, cdo, rebuild, ...)
93	#====================================================
94	if [ X${TaskType} = Xcomputing ] ; then
95	IGCM_debug_Print 1 "Module will be loaded once we know the compiler version"
96	else
97	module purge > /dev/null 2>&1
98	. /ccc/cont003/home/dsm/p86ipsl/.env_irene > /dev/null 2>&1
99
100	# . /ccc/cont003/home/dsm/p86ipsl/.env_intel17.0.2_curie > /dev/null 2>&1
101	# . /ccc/cont003/home/dsm/p86ipsl/.env_netcdf4.3.3.1_curie > /dev/null 2>&1
102	# . /ccc/cont003/home/dsm/p86ipsl/.env_atlas_curie > /dev/null 2>&1
103	export PCMDI_MP=/ccc/work/cont003/igcmg/igcmg/PCMDI-MP
104	export UVCDAT_ANONYMOUS_LOG=FALSE
105	fi
106
107	# Use CMIP6 storage space when using CMIP6 cpu ressources
108	[[ "X${PROJECT}" = "Xdevcmip6" && ! $(module list --terse 2>&1 \| grep dfldatadir/gencmip6) ]] && module switch dfldatadir dfldatadir/gencmip6
109
110	# FYI
111	[ ! X${TaskType} = Xchecking ] && IGCM_debug_Print 1 "List of loaded modules:"
112	[ ! X${TaskType} = Xchecking ] && module list
113
114	export PATH=${PATH}:/ccc/cont003/home/dsm/p86ipsl/AddNoise/src_X64_CURIE/bin
115	export PATH=${PATH}:/ccc/cont003/home/dsm/p86ipsl/AddPerturbation/src_X64_CURIE/bin
116	export PATH=${PATH}:$( ccc_home -u p86ipsl )/rebuild/src_X64_CURIE/modipsl_v2_2_2_netcdf4.2/bin/
117
118	#====================================================
119	# Specific for ocean additionnal diagnostic
120	export FER_GO="$FER_GO /home/cont003/p86denv/IGCM_POST_UTIL/JNL /home/cont003/p86denv/GRAF /home/cont003/p86denv/GRAF/GO"
121	export FER_PALETTE="$FER_PALETTE /home/cont003/p86denv/GRAF/PALET"
122
123	#====================================================
124	# Host specific DIRECTORIES
125	#====================================================
126
127	# ============ CESIUM START ============ #
128
129	#====================================================
130	#- Mirror libIGCM from titane to cesium if needed
131	#ROOTSYS=$( echo ${libIGCM} \| gawk -F"/" '{print $3}' )
132	#if [ ! ${ROOTSYS} = "home" ] ; then
133	# typeset -r MirrorlibIGCM=${MirrorlibIGCM:=true}
134	#else
135	# typeset -r MirrorlibIGCM=${MirrorlibIGCM:=false}
136	#fi
137
138	#====================================================
139	#- libIGCM_POST
140	#if ( ${MirrorlibIGCM} ) ; then
141	# PATHlibIGCM=$( echo ${libIGCM} \| gawk -F"${LOGIN}/" '{print $2}' \| sed -e "s&/libIGCM&&" )
142	# typeset -r libIGCM_POST=${HOME}/MIRROR/${PATHlibIGCM}/libIGCM
143	#else
144	# typeset -r libIGCM_POST=${libIGCM}
145	#fi
146
147	# ============ CESIUM END ============ #
148
149	#====================================================
150	#- MirrorlibIGCM for frontend
151	typeset -r MirrorlibIGCM=${MirrorlibIGCM:=false}
152
153	#====================================================
154	#- libIGCM_POST for frontend
155	typeset -r libIGCM_POST=${libIGCM}
156
157	#====================================================
158	#- R_EXE (==> BIN_DIR = ${MODIPSL}/bin )
159	typeset -r R_EXE="${MODIPSL}/bin"
160
161	#====================================================
162	#- SUBMIT_DIR : submission dir
163	typeset -x SUBMIT_DIR=${SUBMIT_DIR:=${BRIDGE_MSUB_PWD}}
164
165	#====================================================
166	#- IN
167	typeset -r R_IN=${R_IN:=/ccc/work/cont003/igcmg/igcmg/IGCM}
168	typeset -r R_IN_ECMWF=${R_IN_ECMWF:=/ccc/work/cont003/dsm/p24data}
169
170	#====================================================
171	#- RUN_DIR_PATH : Temporary working directory (=> TMP)
172	typeset -r RUN_DIR_PATH=${RUN_DIR_PATH:=${CCCSCRATCHDIR}/RUN_DIR/${BRIDGE_MSUB_JOBID}_${$}}
173
174	#====================================================
175	#- OUTCOMMAND_PATH : tmp place to store command lines standard error and outputs
176	typeset -r OUTCOMMAND_PATH=/tmp
177
178	#====================================================
179	#- HOST_MPIRUN_COMMAND
180	typeset -r HOST_MPIRUN_COMMAND=${HOST_MPIRUN_COMMAND:="/usr/bin/time ccc_mprun -E-K1 "}
181
182	#====================================================
183	#- Max number of arguments passed to nco operator or demigration command
184	UNIX_MAX_LIMIT=360
185
186	#====================================================
187	#- set PackDefault to true on curie
188	PackDefault=true
189
190	#====================================================
191	#- Default number of MPI task for IPSL coupled model
192	#- required for backward compatibility
193	#-
194	DEFAULT_NUM_PROC_OCE=5
195	DEFAULT_NUM_PROC_CPL=1
196	(( DEFAULT_NUM_PROC_ATM = BATCH_NUM_PROC_TOT - DEFAULT_NUM_PROC_OCE - DEFAULT_NUM_PROC_CPL ))
197
198	#====================================================
199	#- Number of computing cores per node
200	#-
201	NB_CORE_PER_NODE=16
202
203	#D-#==================================================
204	#D-function IGCM_sys_defineArchives
205	#D-* Purpose:
206	#D-* Define ARCHIVE : Dedicated to large files
207	#D-* Define STORAGE : Dedicated to small/medium files
208	#D-* Define R_OUT : Output tree located on ARCHIVE
209	#D-* Define R_FIG : Output tree located on STORAGE hosting figures (monitoring and atlas, and/or small files)
210	#D-* Define R_BUF : Output tree located on CCCSCRATCHDIR hosting files waiting for rebuild or pack processes
211	#D-* if SpaceName=TEST everything is stored on CCCSCRATCHDIR
212	#D-* Examples:
213	#D-
214	function IGCM_sys_defineArchives {
215	IGCM_debug_PushStack "IGCM_sys_defineArchives"
216
217	if [ ! X${config_UserChoices_ARCHIVE} = X ]; then
218	#====================================================
219	#- ARCHIVE (dedicated to large files)
220	ARCHIVE=${config_UserChoices_ARCHIVE}
221	else
222	#====================================================
223	#- ARCHIVE (dedicated to large files)
224	ARCHIVE=${CCCSTOREDIR}
225	fi
226
227	if [ ! X${config_UserChoices_STORAGE} = X ]; then
228	#====================================================
229	#- STORAGE (dedicated to small/medium files)
230	STORAGE=${config_UserChoices_STORAGE}
231	else
232	#====================================================
233	#- STORAGE (dedicated to small/medium files)
234	STORAGE=${CCCWORKDIR}
235	fi
236
237	if [ X${config_UserChoices_SpaceName} = XTEST ]; then
238	#====================================================
239	#- R_OUT
240	R_OUT=${CCCSCRATCHDIR}/IGCM_OUT
241
242	#====================================================
243	#- R_FIG (hosting figures : monitoring and atlas, and/or small files)
244	R_FIG=${CCCSCRATCHDIR}/IGCM_OUT
245
246	IGCM_debug_Print 1 "SpaceName=TEST ==> OVERRULE destination path directories"
247
248	else
249	#====================================================
250	#- R_OUT
251	R_OUT=${ARCHIVE}/IGCM_OUT
252
253	#====================================================
254	#- R_FIG (hosting figures : monitoring and atlas, and/or small files)
255	R_FIG=${STORAGE}/IGCM_OUT
256	fi
257
258	#====================================================
259	#- CMIP6 (hosting CMIP6 files produced by XIOS2 and configured by dr2xml)
260	CMIP6_BUF=${STORAGE}/IGCM_OUT
261
262	#====================================================
263	#- R_BUF (ONLY FOR double copy an scratch)
264	R_BUF=${CCCSCRATCHDIR}/IGCM_OUT
265
266	IGCM_debug_Print 1 "R_OUT has been defined = ${R_OUT}"
267	IGCM_debug_Print 1 "R_BUF has been defined = ${R_BUF}"
268	IGCM_debug_Print 1 "R_FIG has been defined = ${R_FIG}"
269
270	IGCM_debug_PopStack "IGCM_sys_defineArchives"
271	}
272
273	#D-#==================================================
274	#D-function IGCM_sys_RshArchive
275	#D-* Purpose: Archive rsh command
276	#D-* Examples:
277	#D-
278	function IGCM_sys_RshArchive {
279	IGCM_debug_PushStack "IGCM_sys_RshArchive" $@
280	/bin/ksh <<-EOF
281	${@}
282	EOF
283	status=$?
284	if [ ${status} -gt 0 ] ; then
285	IGCM_debug_Print 2 "IGCM_sys_RshArchive : command failed error code ${status}"
286	IGCM_debug_Exit "IGCM_sys_RshArchive"
287	fi
288	IGCM_debug_PopStack "IGCM_sys_RshArchive"
289	}
290
291	#D-#==================================================
292	#D-function IGCM_sys_RshArchive_NoError
293	#D-* Purpose: Archive rsh command, without error
294	#D-* used only in monitoring.job
295	#D-* Examples:
296	#D-
297	function IGCM_sys_RshArchive_NoError {
298	IGCM_debug_PushStack "IGCM_sys_RshArchive_NoError" $@
299	/bin/ksh <<-EOF
300	${@} 2> /dev/null
301	EOF
302	IGCM_debug_PopStack "IGCM_sys_RshArchive_NoError"
303	}
304
305	#D-#==================================================
306	#D-function IGCM_sys_MkdirArchive
307	#D-* Purpose: Mkdir on Archive
308	#D-* Examples:
309	#D-
310	function IGCM_sys_MkdirArchive {
311	IGCM_debug_PushStack "IGCM_sys_MkdirArchive" $@
312	if ( $DEBUG_sys ) ; then
313	echo "IGCM_sys_MkdirArchive :" $@
314	fi
315	#- creation de repertoire sur le serveur fichier
316	if [ ! -d ${1} ]; then
317	\mkdir -p $1
318	status=$?
319
320	if [ ${status} -gt 0 ] ; then
321	IGCM_debug_Print 2 "IGCM_sys_MkdirArchive : mkdir failed error code ${status}"
322	IGCM_debug_Exit "IGCM_sys_MkdirArchive"
323	fi
324	fi
325	IGCM_debug_PopStack "IGCM_sys_MkdirArchive"
326	}
327
328	#D-#==================================================
329	#D-function IGCM_sys_TestDirArchive
330	#D-* Purpose: Test Directory that must exists on Archive
331	#D-* Examples:
332	#D-
333	function IGCM_sys_TestDirArchive {
334	IGCM_debug_PushStack "IGCM_sys_TestDirArchive" $@
335	if ( $DEBUG_sys ) ; then
336	echo "IGCM_sys_TestDirArchive :" $@
337	fi
338	typeset ExistFlag
339	ExistFlag=$( [ -d $1 ] && echo 0 \|\| echo 1 )
340	IGCM_debug_PopStack "IGCM_sys_TestDirArchive"
341	return ${ExistFlag}
342	}
343
344	#D-#==================================================
345	#D-function IGCM_sys_IsFileArchived
346	#D-* Purpose: Test file that must NOT EXISTS on Archive based on filename only
347	#D-* Examples:
348	#D-
349	function IGCM_sys_IsFileArchived {
350	IGCM_debug_PushStack "IGCM_sys_IsFileArchived" $@
351	if ( $DEBUG_sys ) ; then
352	echo "IGCM_sys_IsFileArchived :" $@
353	fi
354	typeset IsArchivedFlag
355	IsArchivedFlag=$( [ X$( echo $@ \| grep ^\/ccc\/store ) != X ] && echo 0 \|\| echo 1 )
356	IGCM_debug_PopStack "IGCM_sys_IsFileArchived"
357
358	return ${IsArchivedFlag}
359	}
360
361	#D-#==================================================
362	#D-function IGCM_sys_TestFileArchive
363	#D-* Purpose: Test file that must NOT EXISTS on Archive
364	#D-* Examples:
365	#D-
366	function IGCM_sys_TestFileArchive {
367	IGCM_debug_PushStack "IGCM_sys_TestFileArchive" $@
368	typeset ExistFlag
369	ExistFlag=$( [ -f $1 ] && echo 0 \|\| echo 1 )
370	IGCM_debug_PopStack "IGCM_sys_TestFileArchive"
371
372	return ${ExistFlag}
373	}
374
375	#D-#==================================================
376	#D-function IGCM_sys_CountFileArchive
377	#D-* Purpose: Count files on Archive filesystem
378	#D-* Examples:
379	#D-
380	function IGCM_sys_CountFileArchive {
381	IGCM_debug_PushStack "IGCM_sys_CountFileArchive" $@
382	ls ${@} 2>/dev/null \| wc -l
383	if [ $? -gt 0 ] ; then
384	echo "IGCM_sys_CountFileArchive : erreur."
385	fi
386	IGCM_debug_PopStack "IGCM_sys_CountFileArchive"
387	}
388
389	#D-#==================================================
390	#D-function IGCM_sys_Tree
391	#D-* Purpose: Tree directories with files on ${ARCHIVE}
392	#D-* Examples: IGCM_sys_Tree ${R_IN} ${R_OUT}
393	#D-
394	function IGCM_sys_Tree {
395	IGCM_debug_PushStack "IGCM_sys_Tree" $@
396	if ( $DEBUG_sys ) ; then
397	echo "IGCM_sys_Tree :" $@
398	fi
399
400	\ls -lR ${@}
401
402	IGCM_debug_PopStack "IGCM_sys_Tree"
403	}
404
405	#D-#==================================================
406	#D-function IGCM_sys_Qsub
407	#D-* Purpose: Qsub new job
408	#D-* Examples:
409	#D-
410	function IGCM_sys_Qsub {
411	IGCM_debug_PushStack "IGCM_sys_Qsub" $@
412	if ( $DEBUG_sys ) ; then
413	echo "IGCM_sys_Qsub :" $@
414	fi
415	typeset options status
416	options="-o ${SUBMIT_DIR}/${Script_Output} -e ${SUBMIT_DIR}/${Script_Output}"
417
418	/usr/bin/ccc_msub ${options} $1 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
419	status=$?
420
421	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
422	if [ ${status} -gt 0 ] ; then
423	IGCM_debug_Print 2 "IGCM_sys_Qsub ${options} $1 : error code ${status}"
424	IGCM_debug_Exit "IGCM_sys_Qsub"
425	else
426	JobID=$( gawk {'print $4'} ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ )
427	fi
428	IGCM_debug_PopStack "IGCM_sys_Qsub"
429	}
430
431	#D-#==================================================
432	#D-function IGCM_sys_QsubPost
433	#D-* Purpose: Qsub new job on scalaire
434	#D-* Examples:
435	#D-
436	function IGCM_sys_QsubPost {
437	IGCM_debug_PushStack "IGCM_sys_QsubPost" $@
438	if ( $DEBUG_sys ) ; then
439	echo "IGCM_sys_QsubPost :" $@
440	fi
441	typeset options status
442
443	# EASIER TO DO THIS INSTEAD OF DUPLICATING libIGCM_sys_curie.ksh
444	case $( hostname -s ) in
445	curie*)
446	options="-Q normal -A ${PROJECT} -o ${POST_DIR}/${Script_Post_Output}.out -e ${POST_DIR}/${Script_Post_Output}.out";;
447	airain*)
448	options="-q ivybridge -A dsm -o ${POST_DIR}/${Script_Post_Output}.out -e ${POST_DIR}/${Script_Post_Output}.out"
449	esac
450
451	/usr/bin/ccc_msub ${options} ${libIGCM_POST}/$1.job > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
452	status=$?
453
454	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
455	if [ ${status} -gt 0 ] ; then
456	IGCM_debug_Print 2 "IGCM_sys_QsubPost ${options} ${libIGCM_POST}/$1.job : error code ${status}"
457	IGCM_debug_Exit "IGCM_sys_QsubPost"
458	else
459	JobID=$( gawk {'print $4'} ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ )
460	fi
461	IGCM_debug_PopStack "IGCM_sys_QsubPost"
462	}
463
464	#D-*************************
465	#D- File transfer functions
466	#D-*************************
467	#D-
468
469	#D-#==================================================
470	#D-function IGCM_sys_RmRunDir
471	#D-* Purpose: rm tmpdir (dummy function most of the time batch
472	#D- scheduler will do the job)
473	#D-* Examples:
474	#D-
475	function IGCM_sys_RmRunDir {
476	IGCM_debug_PushStack "IGCM_sys_RmRunDir" $@
477	if ( $DEBUG_sys ) ; then
478	echo "IGCM_sys_RmRunDir :" $@
479	fi
480
481	typeset status
482
483	echo rm $@ > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
484	\rm $@ >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
485	status=$?
486
487	if [ ${status} -gt 0 ] ; then
488	IGCM_debug_Print 1 "IGCM_sys_RmRunDir : rm error code is ${status}."
489	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
490	IGCM_debug_Exit "IGCM_sys_RmRunDir"
491	else
492	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
493	fi
494	IGCM_debug_PopStack "IGCM_sys_RmRunDir"
495	}
496
497	#D-#==================================================
498	#D-function IGCM_sys_Put_Dir
499	#D-* Purpose: Copy a complete directory on $(ARCHIVE)
500	#D-* Examples:
501	#D-
502	function IGCM_sys_Put_Dir {
503	IGCM_debug_PushStack "IGCM_sys_Put_Dir" $@
504	if ( $DEBUG_sys ) ; then
505	echo "IGCM_sys_Put_Dir :" $@
506	fi
507	if [ $DRYRUN = 0 ]; then
508	if [ ! -d ${1} ] ; then
509	echo "WARNING : IGCM_sys_Put_Dir ${1} DOES NOT EXIST ."
510	IGCM_debug_PopStack "IGCM_sys_Put_Dir"
511	return
512	fi
513
514	typeset status
515
516	# Only if we use rsync
517	#IGCM_sys_TestDirArchive $( dirname $2 )
518	#
519	#USUAL WAY
520	\cp -r $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
521	status=$?
522
523	if [ ${status} -gt 0 ] ; then
524	IGCM_debug_Print 2 "IGCM_sys_Put_Dir : cp failed error code ${status}"
525	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
526	IGCM_debug_Exit "IGCM_sys_Put_Dir"
527	else
528	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
529	fi
530	fi
531	IGCM_debug_PopStack "IGCM_sys_Put_Dir"
532	}
533
534	#D-#==================================================
535	#D-function IGCM_sys_Get_Dir
536	#D-* Purpose: Copy a complete directory from ${ARCHIVE}
537	#D-* Examples:
538	#D-
539	function IGCM_sys_Get_Dir {
540	IGCM_debug_PushStack "IGCM_sys_Get_Dir" $@
541	if ( $DEBUG_sys ) ; then
542	echo "IGCM_sys_Get_Dir :" $@
543	fi
544	if [ $DRYRUN = 0 ]; then
545	typeset NB_ESSAI DELAI status i
546	# number of tentative
547	NB_ESSAI=3
548	# time delay between tentative
549	DELAI=2
550
551	#
552	# USUAL WAY
553	# add 'ccc_hsm get' (to demigrate all offline files) to reduce time of this command :
554	ccc_hsm get -r $1
555
556	i=0
557	while [ $i -lt $NB_ESSAI ] ; do
558	\cp -ur $1 $2 >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
559	status=$?
560	if [ ${status} -gt 0 ] ; then
561	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : cp failed error code ${status} ${i}/${NB_ESSAI}"
562	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : sleep ${DELAI} seconds and try again."
563	sleep $DELAI
564	else
565	break
566	fi
567	(( i = i + 1 ))
568	done
569
570	if [ ${status} -gt 0 ] ; then
571	IGCM_debug_Print 2 "IGCM_sys_Get_Dir : cp failed error code ${status}"
572	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
573	IGCM_debug_Exit "IGCM_sys_Get_Dir"
574	else
575	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
576	fi
577	fi
578	IGCM_debug_PopStack "IGCM_sys_Get_Dir"
579	}
580
581	#D-#==================================================
582	#D-function IGCM_sys_Put_Rest
583	#D-* Purpose: Put computied restarts on ${ARCHIVE}.
584	#D- File and target directory must exist.
585	#D-* Examples:
586	#D-
587	function IGCM_sys_Put_Rest {
588	IGCM_debug_PushStack "IGCM_sys_Put_Rest" $@
589	if ( $DEBUG_sys ) ; then
590	echo "IGCM_sys_Put_Rest :" $@
591	fi
592	if [ $DRYRUN = 0 ]; then
593	if [ ! -f ${1} ] ; then
594	echo "ERROR : IGCM_sys_Put_Rest ${1} DOES NOT EXIST ."
595	IGCM_debug_Exit "IGCM_sys_Put_Rest"
596	fi
597
598	typeset status
599	#
600	# USUAL WAY
601	\cp $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
602	status=$?
603
604	# #RSYNC WITH NETWORK SSH CALL
605	# echo ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
606	# ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
607
608	# #RSYNC WITH NFS USE
609	# echo ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
610	# ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
611
612	# status=$?
613	# IGCM_sys_Rsync_out $status
614
615	# ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
616	# (( status=status+$? ))
617
618	if [ ${status} -gt 0 ] ; then
619	IGCM_debug_Print 2 "IGCM_sys_Put_Rest : cp failed error code ${status}"
620	[ -f ${1} ] && ls -l ${1}
621	[ -f ${2} ] && ls -l ${2}
622	[ -f ${2}/${1} ] && ls -l ${2}/${1}
623	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
624	IGCM_debug_Exit "IGCM_sys_Put_Rest"
625	else
626
627	if [ X${JobType} = XRUN ] ; then
628	[ -f ${2} ] && IGCM_sys_Chmod 444 ${2}
629	[ -f ${2}/${1} ] && IGCM_sys_Chmod 444 ${2}/${1}
630	fi
631
632	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
633	fi
634	fi
635	IGCM_debug_PopStack "IGCM_sys_Put_Rest"
636	}
637
638	#D-#==================================================
639	#D-function IGCM_sys_Put_Out
640	#D-* Purpose: Copy a file on ${ARCHIVE} after having chmod it in readonly
641	#D-* Examples:
642	#D-
643	function IGCM_sys_Put_Out {
644	IGCM_debug_PushStack "IGCM_sys_Put_Out" $@
645	if ( $DEBUG_sys ) ; then
646	echo "IGCM_sys_Put_Out :" $@
647	fi
648
649	typeset NB_ESSAI DELAI status i exist skip
650	typeset fileDeviceNumberInHex directoryDeviceNumberInHex
651
652	# number of tentative
653	NB_ESSAI=3
654	# time delay between tentative
655	DELAI=2
656
657	if [ $DRYRUN = 0 ]; then
658	if [ ! -f ${1} ] ; then
659	echo "WARNING : IGCM_sys_Put_Out ${1} DOES NOT EXIST ."
660	IGCM_debug_PopStack "IGCM_sys_Put_Out"
661	return 1
662	fi
663	#
664	IGCM_sys_MkdirArchive $( dirname $2 )
665	#
666	exist=false
667	skip=false
668	if [ -f $2 ] ; then
669	IGCM_debug_Print 1 "$2 already exist"
670	ccc_hsm get $2
671	exist=true
672	if [ "X$( diff $1 $2 )" = X ] ; then
673	IGCM_debug_Print 2 "$1 and $2 are the same file, we skip the copy"
674	skip=true
675	else
676	IGCM_debug_Print 2 "$1 and $2 are not the same file, we force the copy"
677	skip=false
678	fi
679	fi
680	#
681	if ( [ X${exist} = Xtrue ] && [ X${skip} = Xfalse ] ) ; then
682	IGCM_sys_Chmod u+w $2
683	fi
684
685	if [ X${skip} = Xfalse ] ; then
686	i=0
687	while [ $i -lt $NB_ESSAI ] ; do
688	# Identify file system
689	fileDeviceNumberInHex=$( stat -c %d $1 )
690	status=$?
691	if [ ${status} -gt 0 ] ; then
692	IGCM_debug_Exit "IGCM_sys_Put_Out"
693	fi
694	# Identify file system
695	directoryDeviceNumberInHex=$( stat -c %d $( dirname $2 ) )
696	status=$?
697	if [ ${status} -gt 0 ] ; then
698	IGCM_debug_Exit "IGCM_sys_Put_Out"
699	fi
700
701	if ( [ ${fileDeviceNumberInHex} -ne ${directoryDeviceNumberInHex} ] \|\| [ X$3 = XNOMOVE ] ) ; then
702	# They are not on the same device. USUAL WAY
703	\cp $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
704	status=$?
705	else
706	# They are on the same device. NOT SO USUAL WAY
707	\mv $1 $2 > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
708	status=$?
709	fi
710	if [ ${status} -gt 0 ]; then
711	IGCM_debug_Print 2 "IGCM_sys_Put_Out : cp failed error code ${status} ${i}/${NB_ESSAI}"
712	IGCM_debug_Print 2 "IGCM_sys_Put_Out : sleep ${DELAI} seconds and try again."
713	[ -f ${1} ] && ls -l ${1}
714	[ -f ${2} ] && ls -l ${2}
715	[ -f ${2}/${1} ] && ls -l ${2}/${1}
716	sleep $DELAI
717	else
718	break
719	fi
720	(( i = i + 1 ))
721	done
722	fi
723
724	# #RSYNC WITH NETWORK SSH CALL
725	# echo ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
726	# ${RSYNC} ${RSYNC_opt} -e ssh ${RUN_DIR}/$1 ${STOREHOST}:${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
727
728	# #RSYNC WITH NFS USE
729	# echo ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
730	# ${RSYNC} ${RSYNC_opt} ${RUN_DIR}/$1 ${2} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
731
732	# status=$?
733	# IGCM_sys_Rsync_out $status
734
735	# ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
736	# (( status=status+$? ))
737
738	if [ ${status} -gt 0 ] ; then
739	IGCM_debug_Print 2 "IGCM_sys_Put_Out : cp failed error code ${status}"
740	[ -f ${1} ] && ls -l ${1}
741	[ -f ${2} ] && ls -l ${2}
742	[ -f ${2}/${1} ] && ls -l ${2}/${1}
743	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
744	IGCM_debug_Exit "IGCM_sys_Put_Out"
745	else
746
747	if [ X${JobType} = XRUN ] ; then
748	if [ X${3} = X ] ; then
749	[ -f ${2} ] && IGCM_sys_Chmod 444 ${2}
750	[ -f ${2}/${1} ] && IGCM_sys_Chmod 444 ${2}/${1}
751	fi
752	fi
753
754	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
755	fi
756	fi
757	IGCM_debug_PopStack "IGCM_sys_Put_Out"
758	return 0
759	}
760
761	#D-#==================================================
762	#D-function IGCM_sys_Get
763	#D-* Purpose: Get a file from ${ARCHIVE}
764	#D-* Examples: IGCM_sys_Get myfile /destpath/myfile_with_PREFIX
765	#D- IGCM_sys_Get /l Array_contain_myfiles /destpath/
766	function IGCM_sys_Get {
767	IGCM_debug_PushStack "IGCM_sys_Get" $@
768
769	typeset DEST dm_liste target file_work
770	typeset NB_ESSAI DELAI status i
771
772	if ( $DEBUG_sys ) ; then
773	echo "IGCM_sys_Get :" $@
774	fi
775
776	# number of tentative
777	NB_ESSAI=3
778	# time delay between tentative
779	DELAI=2
780
781	if [ $DRYRUN -le 2 ]; then
782	if [ X${1} = X'/l' ] ; then
783	eval set +A dm_liste \${${2}}
784	else
785	eval set +A dm_liste ${1}
786	fi
787	eval DEST=\${${#}}
788	ccc_hsm get ${dm_liste[*]} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
789	status=$?
790	if [ ${status} -gt 0 ] ; then
791	echo "WARNING IGCM_sys_Get : error code ${status}"
792	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
793	echo "WARNING IGCM_sys_Get : will stop later if the cp fails."
794	fi
795
796	#if [ ${status} -gt 0 ] ; then
797	# if [ ! "X$( grep "Lost dmusrcmd connection" ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ )" = "X" ] ; then
798	# cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
799	# echo "WARNING IGCM_sys_Get : Lost dmusrcmd connection : "
800	# sleep 30
801	# echo "We try another time"
802	## dmget ${dm_liste[*]} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
803	# ccc_hsm get ${dm_liste[*]} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
804	# status=$?
805	# if [ ${status} -gt 0 ] ; then
806	# echo "ERROR IGCM_sys_Get : again demigration error :"
807	# cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
808	# IGCM_debug_Exit "IGCM_sys_Get"
809	# fi
810	# else
811	# echo "ERROR IGCM_sys_Get : demigration error :"
812	# cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
813	# IGCM_debug_Exit "IGCM_sys_Get"
814	# fi
815	#fi
816
817	# #RSYNC WITH NETWORK SSH CALL
818	# echo ${RSYNC} ${RSYNC_opt} -e ssh ${STOREHOST}:"${dm_liste}" ${STOREHOST}:${RUN_DIR}/${DEST} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
819	# ${RSYNC} ${RSYNC_opt} -e ssh ${STOREHOST}:"${dm_liste}" ${STOREHOST}:${RUN_DIR}/${DEST} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
820
821	# #RSYNC WITH NFS USE
822	# echo ${RSYNC} ${RSYNC_opt} ${dm_liste} ${RUN_DIR}/${DEST} > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
823	# ${RSYNC} ${RSYNC_opt} ${dm_liste} ${RUN_DIR}/${DEST} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
824
825	# status=$?
826	# IGCM_sys_Rsync_out $status
827
828	# ${libIGCM}/libIGCM_sys/IGCM_analyse_rsync_out.awk ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
829	# (( status=status+$? ))
830
831	#USUAL WAY
832	if [ X${1} = X'/l' ] ; then
833	for target in ${dm_liste[*]} ; do
834	local_file=$( basename ${target} )
835	# test if the target file is present before the loop
836	IGCM_sys_TestFileArchive ${target}
837	status=$?
838	if [ ${status} -gt 0 ] ; then
839	echo "IGCM_sys_Get, ERROR : regular file ${target} DOES NOT EXIST ."
840	IGCM_debug_Exit "IGCM_sys_Get"
841	else
842	i=0
843	while [ $i -lt $NB_ESSAI ] ; do
844	#if [ X${DoLink} = Xtrue ] ; then
845	# \ln -s ${target} ${DEST}/${local_file} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
846	# status=$?
847	# else
848	# \cp ${target} ${DEST}/${local_file} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
849	# status=$?
850	#fi
851	\ln -s ${target} ${DEST}/${local_file} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
852	status=$?
853	if [ ${status} -gt 0 ]; then
854	IGCM_debug_Print 2 "IGCM_sys_Get : cp failed error code ${status} ${i}/${NB_ESSAI}"
855	IGCM_debug_Print 2 "IGCM_sys_Get : sleep ${DELAI} seconds and try again."
856	sleep $DELAI
857	else
858	break
859	fi
860	(( i = i + 1 ))
861	done
862	if [ ${status} -gt 0 ] ; then
863	echo "IGCM_sys_Get : error"
864	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
865	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
866	IGCM_debug_Exit "IGCM_sys_Get"
867	else
868	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
869	fi
870	fi
871	done
872	else
873	i=0
874	while [ $i -lt $NB_ESSAI ] ; do
875	\cp ${dm_liste} ${DEST} >> ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
876	status=$?
877	if [ ${status} -gt 0 ]; then
878	IGCM_debug_Print 2 "IGCM_sys_Get : cp failed error code ${status} ${i}/${NB_ESSAI}"
879	IGCM_debug_Print 2 "IGCM_sys_Get : sleep ${DELAI} seconds and try again."
880	sleep $DELAI
881	else
882	break
883	fi
884	(( i = i + 1 ))
885	done
886	if [ ${status} -gt 0 ] ; then
887	echo "IGCM_sys_Get : error"
888	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
889	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
890	IGCM_debug_Exit "IGCM_sys_Get"
891	else
892	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
893	fi
894	fi
895	fi
896	IGCM_debug_PopStack "IGCM_sys_Get"
897	}
898
899	#D-#==================================================
900	#D-function IGCM_sys_GetDate_Monitoring
901	#D-* Purpose: get the last year for which the monitoring has been computed
902	#D-* Examples:
903	#D-
904	function IGCM_sys_GetDate_Monitoring {
905	IGCM_debug_PushStack "IGCM_sys_GetDate_Monitoring" $@
906	if ( $DEBUG_sys ) ; then
907	echo "IGCM_sys_GetDate_Monitoring :" $@
908	fi
909
910	eval ${2}=$( cdo showyear ${1} 2> /dev/null \| gawk '{ print $NF }' )
911
912	IGCM_debug_PopStack "IGCM_sys_GetDate_Monitoring"
913	}
914
915	#D-#==================================================
916	#D-function IGCM_sys_Dods_Rm
917	#D-* Purpose: DO NOTHING ! Put ${ARCHIVE} files on DODS internet protocole.
918	#D-* Examples:
919	#D-
920	function IGCM_sys_Dods_Rm {
921	if ( $DEBUG_sys ) ; then
922	echo "IGCM_sys_Dods_Rm :" $@
923	fi
924	typeset status
925	if [ $DRYRUN = 0 ]; then
926
927	# if [ ! -d /dmnfs/cont003/dods/public/${LOGIN}/${R_DODS}/${1} ] ; then
928	# echo "WARNING : IGCM_sys_Dods_Rm /dmnfs/cont003/dods/public/${LOGIN}/${R_DODS}/${1} DOES NOT EXIST ."
929	# echo "Nothing has been done."
930	# return
931	# fi
932
933	if [ "$#" -eq 1 ]; then
934	/ccc/cont003/home/dsm/p86ipsl/bin/dods_rm public/${LOGIN}/${R_DODS}/${1} # > out_dods_rm 2>&1
935	status=$?
936	else
937	/ccc/cont003/home/dsm/p86ipsl/bin/dods_rm public/${LOGIN}/${R_DODS} # > out_dods_rm 2>&1
938	status=$?
939	fi
940
941	# if [ ${status} -gt 0 ] ; then
942	# echo "IGCM_sys_Dods_Rm : error."
943	# cat out_dods_rm
944	# IGCM_debug_Exit "IGCM_sys_Dods_Rm"
945	# else
946	# rm out_dods_rm
947	# fi
948
949	fi
950	return $status
951	}
952
953	#D-#==================================================
954	#D-function IGCM_sys_Dods_Cp
955	#D-* Purpose: Copy $(ARCHIVE) files on DODS internet protocole.
956	#D-* Examples:
957	#D-
958	function IGCM_sys_Dods_Cp {
959	if ( $DEBUG_sys ) ; then
960	echo "IGCM_sys_Dods_Cp :" $@
961	fi
962	typeset status
963	if [ $DRYRUN = 0 ]; then
964
965	# if [ ! -d ${R_SAVE}/${1} ] ; then
966	# echo "WARNING : IGCM_sys_Dods_Cp ${R_SAVE}/${1} DOES NOT EXIST ."
967	# echo "Nothing has been done."
968	# return
969	# fi
970
971	/ccc/cont003/home/dsm/p86ipsl/bin/dods_cp ${1} public/${LOGIN}/${R_DODS} # > out_dods_cp 2>&1
972	status=$?
973
974	# if [ ${status} -gt 0 ] ; then
975	# echo "IGCM_sys_Dods_Cp : error."
976	# cat out_dods_cp
977	# IGCM_debug_Exit "IGCM_sys_Dods_Cp"
978	# else
979	# rm out_dods_cp
980	# fi
981
982	fi
983	return $status
984	}
985
986	#D-#==================================================
987	#D-function IGCM_sys_Put_Dods
988	#D-* Purpose: Put ${ARCHIVE} files on DODS internet protocole.
989	#D-* Examples:
990	#D-
991	function IGCM_sys_Put_Dods {
992	IGCM_debug_PushStack "IGCM_sys_Put_Dods" $@
993	if ( $DEBUG_sys ) ; then
994	echo "IGCM_sys_Put_Dods :" $@
995	fi
996	typeset status
997	if [ $DRYRUN = 0 ]; then
998	if ( [ ! -d ${R_SAVE}/${1} ] && [ ! -d ${R_FIGR}/${1} ] ) ; then
999	echo "WARNING IGCM_sys_Put_Dods : None of the following directories exist. Exactly one should."
1000	echo "WARNING IGCM_sys_Put_Dods : ${R_SAVE}/${1} DOES NOT EXIST."
1001	echo "WARNING IGCM_sys_Put_Dods : ${R_FIGR}/${1} DOES NOT EXIST."
1002	IGCM_debug_PopStack "IGCM_sys_Put_Dods"
1003	return
1004	fi
1005
1006	if ( [ -d ${R_SAVE}/${1} ] && [ -d ${R_FIGR}/${1} ] ) ; then
1007	echo "WARNING IGCM_sys_Put_Dods : Both of the following directories exist. Exactly one should."
1008	echo "WARNING IGCM_sys_Put_Dods : ${R_SAVE}/${1} EXISTS."
1009	echo "WARNING IGCM_sys_Put_Dods : ${R_FIGR}/${1} EXISTS."
1010	IGCM_debug_PopStack "IGCM_sys_Put_Dods"
1011	return
1012	fi
1013	#
1014	if [ -d ${R_SAVE}/${1} ] ; then
1015	cd ${R_SAVE}
1016	elif [ -d ${R_FIGR}/${1} ] ; then
1017	cd ${R_FIGR}
1018	fi
1019
1020	IGCM_sys_Dods_Rm ${1}
1021	IGCM_sys_Dods_Cp ${1}
1022	status=0
1023
1024	if [ ${status} -gt 0 ] ; then
1025	echo "IGCM_sys_Put_Dods : error."
1026	IGCM_debug_Exit "IGCM_sys_Put_Dods"
1027	fi
1028	fi
1029	IGCM_debug_PopStack "IGCM_sys_Put_Dods"
1030	}
1031
1032	##############################################################
1033	# REBUILD OPERATOR
1034
1035	#D-#==================================================
1036	#D-function IGCM_sys_sync
1037	#D-* Purpose: flush buffer on disk
1038	#D-* Examples:
1039	#D-
1040	function IGCM_sys_sync {
1041	IGCM_debug_PushStack "IGCM_sys_sync" $@
1042	if ( $DEBUG_sys ) ; then
1043	echo "IGCM_sys_sync :" $@
1044	fi
1045
1046	/bin/sync
1047
1048	IGCM_debug_PopStack "IGCM_sys_sync"
1049	}
1050
1051	############################################################
1052	# Activate Running Environnment Variables
1053
1054	#D-#==================================================
1055	#D-function IGCM_sys_activ_variables
1056	#D-* Purpose: set environement variables prior to execution
1057	#D-* Examples:
1058	#D-
1059	function IGCM_sys_activ_variables {
1060	IGCM_debug_PushStack "IGCM_sys_activ_variables"
1061	if ( $DEBUG_sys ) ; then
1062	echo "IGCM_sys_activ_variables"
1063	fi
1064
1065	# --------------------------------------------------------------------
1066	#D- MODULE specifications
1067	# --------------------------------------------------------------------
1068	module purge > /dev/null 2>&1
1069	if [ X${compilerVersion} = XV17 ] ; then
1070	IGCM_debug_Print 1 "IGCM_sys_active_variables : Using compiler version V17"
1071	else
1072	IGCM_debug_Exit "This compiler version is not yet supported : ${compilerVersion}"
1073	fi
1074	. /ccc/cont003/home/dsm/p86ipsl/.env_irene > /dev/null 2>&1
1075	IGCM_debug_Print 1 "IGCM_sys_active_variables : Now loaded module for Irene. "
1076	module list
1077
1078	# Keep track of this
1079	[ ! X${TaskType} = Xchecking ] && IGCM_debug_Print 1 "List of loaded modules:"
1080	[ ! X${TaskType} = Xchecking ] && module list
1081
1082	# --------------------------------------------------------------------
1083	#D- MPI specifications
1084	# --------------------------------------------------------------------
1085
1086	# --------------------------------------------------------------------
1087	#D- Other specifications
1088	# --------------------------------------------------------------------
1089
1090	ulimit -s unlimited
1091
1092	IGCM_debug_PopStack "IGCM_sys_activ_variables"
1093	}
1094
1095	############################################################
1096	# Desactivate Running Environnment Variables
1097
1098	#D-#==================================================
1099	#D-function IGCM_sys_desactiv_variables
1100	#D-* Purpose: unset environement variables after execution
1101	#D-* Examples:
1102	#D-
1103	function IGCM_sys_desactiv_variables {
1104	IGCM_debug_PushStack "IGCM_sys_desactiv_variables"
1105	if ( $DEBUG_sys ) ; then
1106	echo "IGCM_sys_desactiv_variables"
1107	fi
1108	# --------------------------------------------------------------------
1109	#D- MPI specifications
1110	# --------------------------------------------------------------------
1111
1112	# --------------------------------------------------------------------
1113	#D- Other specifications
1114	# --------------------------------------------------------------------
1115
1116	IGCM_debug_PopStack "IGCM_sys_desactiv_variables"
1117	}
1118
1119	############################################################
1120	# Update job headers to be used by the scheduler
1121
1122	#D-#==================================================
1123	#D-function IGCM_sys_updateHeaders
1124	#D-* Purpose: Update job headers to be used by the scheduler
1125	#D-* Examples: IGCM_sys_updateHeaders /path/to/Job_MYEXP
1126	#D-
1127	function IGCM_sys_updateHeaders {
1128	IGCM_debug_PushStack "IGCM_sys_updateHeaders"
1129	if ( $DEBUG_sys ) ; then
1130	echo "IGCM_sys_updateHeaders"
1131	fi
1132	typeset file
1133	file=$1
1134
1135	if [ ${executionType} -eq 1 ] ; then
1136	# MPMD + MPI
1137	sed -e "/::openMPthreads::/d" \
1138	-e "s/::JobNumProcTot::/${coreNumber}/" \
1139	-e "/#MSUB -x/d" \
1140	-e "/--cpu_bind=none/d" \
1141	${file} > ${file}.tmp
1142
1143	elif [ ${executionType} -eq 2 ] ; then
1144	# MPMD + MPI + OMP : mpirun/ccc_mprun/error
1145	if ( [ "X${config_UserChoices_ExecutionModeOnCurie}" = "Xmpirun" ] ) ; then
1146	sed -e "/::openMPthreads::/d" \
1147	-e "s/::JobNumProcTot::/${coreNumber}/" \
1148	${file} > ${file}.tmp
1149	elif ( [ "X${config_UserChoices_ExecutionModeOnCurie}" = "X" ] \|\| [ "X${config_UserChoices_ExecutionModeOnCurie}" = "Xccc_mprun" ] ) ; then
1150	sed -e "/::openMPthreads::/d" \
1151	-e "s/::JobNumProcTot::/${coreNumber}/" \
1152	-e "/--cpu_bind=none/d" \
1153	${file} > ${file}.tmp
1154	else
1155	IGCM_debug_Print 1 "You have to set ExecutionModeOnCurie=ccc_mprun or mpirun in config.card"
1156	IGCM_debug_Exit "IGCM_sys_updateHeaders"
1157	fi
1158	elif [ ${executionType} -eq 3 ] ; then
1159	# SPMD + MPI/OMP
1160	sed -e "s/::openMPthreads::/${openMPthreads}/" \
1161	-e "s/::JobNumProcTot::/${mpiTasks}/" \
1162	-e "/#MSUB -x/d" \
1163	-e "/--cpu_bind=none/d" \
1164	${file} > ${file}.tmp
1165
1166	elif [ ${executionType} -eq 4 ] ; then
1167	# SPMD + MPI only
1168	sed -e "s/::JobNumProcTot::/${mpiTasks}/" \
1169	-e "/::openMPthreads::/d" \
1170	-e "/#MSUB -x/d" \
1171	-e "/--cpu_bind=none/d" \
1172	${file} > ${file}.tmp
1173
1174	elif [ ${executionType} -eq 5 ] ; then
1175	# SPMD + OMP only
1176	sed -e "s/::openMPthreads::/${openMPthreads}/" \
1177	-e "/::JobNumProcTot::/d" \
1178	-e "/#MSUB -x/d" \
1179	-e "/--cpu_bind=none/d" \
1180	${file} > ${file}.tmp
1181
1182	elif [ ${executionType} -eq 6 ] ; then
1183	# SEQUENTIAL THEN
1184	sed -e "s/::JobNumProcTot::/1/" \
1185	-e "/::openMPthreads::/d" \
1186	-e "/#MSUB -x/d" \
1187	-e "/--cpu_bind=none/d" \
1188	${file} > ${file}.tmp
1189
1190	fi
1191
1192	IGCM_sys_Mv ${file}.tmp ${file}
1193
1194	IGCM_debug_PopStack "IGCM_sys_updateHeaders"
1195	}
1196
1197	############################################################
1198	# Build MPI/OMP scripts run file (dummy function)
1199
1200	#D-#==================================================
1201	#D-function IGCM_sys_build_run_file
1202	#D-* Purpose: build run file (deprecated)
1203	#D-* Examples:
1204	#D-
1205	function IGCM_sys_build_run_file {
1206	IGCM_debug_Print 3 " dummy function : IGCM_sys_build_run_file "
1207	}
1208
1209	############################################################
1210	# Build MPI/OMP scripts
1211
1212	#D-#==================================================
1213	#D-function IGCM_sys_build_execution_scripts
1214	#D-* Purpose: build execution scripts to be launch by ${HOST_MPIRUN_COMMAND}
1215	#D-* Examples:
1216	#D-
1217	function IGCM_sys_build_execution_scripts
1218	{
1219	IGCM_debug_PushStack "IGCM_sys_build_execution_scripts" $@
1220	if ( $DEBUG_sys ) ; then
1221	echo "IGCM_sys_build_execution_scripts " $@
1222	fi
1223
1224	EXECUTION=${HOST_MPIRUN_COMMAND}
1225
1226	if ( ${OK_PARA_MPMD} ) ; then
1227
1228	# MPMD mode
1229	# 1 MPI only
1230	# 2 MPI/OpenMP mpirun method
1231	# 3 MPI/OpenMP ccc_mprun method
1232
1233	if [ -f run_file ] ; then
1234	IGCM_sys_Rm -f run_file
1235	fi
1236	touch run_file
1237
1238	# case 1 : Only MPI (MPMD)
1239	if ( ! ${OK_PARA_OMP} ) ; then
1240
1241	# Build run_file
1242
1243	# First loop on the components for the coupler ie oasis (only if oasis3)
1244	# the coupler ie oasis3 must be the first one
1245	for comp in ${config_ListOfComponents[*]} ; do
1246
1247	eval ExeNameIn=\${config_Executable_${comp}[0]}
1248	eval ExeNameOut=\${config_Executable_${comp}[1]}
1249
1250	# for CPL component only
1251	if [ "X${comp}" = "XCPL" ] && [ "X${ExeNameOut}" != X\"\" ] ; then
1252
1253	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1254	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
1255	echo "${comp_proc_mpi_loc} ./${ExeNameOut}" >> run_file
1256	fi
1257	done
1258
1259	# Then second loop on the components
1260	for comp in ${config_ListOfComponents[*]} ; do
1261
1262	eval ExeNameIn=\${config_Executable_${comp}[0]}
1263	eval ExeNameOut=\${config_Executable_${comp}[1]}
1264
1265	# Only if we really have an executable for the component and not the coupler ie oasis:
1266	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${comp}" != "XCPL" ] ) ; then
1267
1268	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1269	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
1270	echo "${comp_proc_mpi_loc} ./${ExeNameOut}" >> run_file
1271	fi
1272	done
1273
1274	EXECUTION="${HOST_MPIRUN_COMMAND} -f ./run_file"
1275
1276	IGCM_sys_Chmod u+x run_file
1277	if ( $DEBUG_sys ) ; then
1278	echo "run_file contains : "
1279	cat run_file
1280	fi
1281
1282	elif ( [ "X${config_UserChoices_ExecutionModeOnCurie}" = "Xmpirun" ] ) ; then
1283
1284	# 2 MPI/OpenMP mpirun method
1285
1286	# Use of mpirun instead of ccc_mprun
1287	EXECUTION="time mpirun"
1288
1289	# Hosts treatment
1290	${EXECUTION} hostname \| sort \| uniq > hosts.tmp
1291
1292	i=0
1293	rm -f hosts rankfile
1294	IGCM_debug_Print 1 "sys Curie, Hosts available :"
1295	for nodes in `cat hosts.tmp`
1296	do
1297	host[$i]=$nodes
1298	echo "${host[$i]}" >> hosts
1299	IGCM_debug_Print 1 ${host[$i]}
1300	i=$((i+1))
1301	done
1302	rm -f hosts.tmp
1303
1304	listnodes=${host[*]}
1305
1306	EXECUTION="${EXECUTION} -hostfile hosts -rankfile rankfile"
1307
1308	# Initialisation
1309	rank=0
1310	current_core=0
1311	init_exec=n
1312
1313	# Loop on the components
1314	for comp in ${config_ListOfComponents[*]} ; do
1315
1316	eval ExeNameIn=\${config_Executable_${comp}[0]}
1317	eval ExeNameOut=\${config_Executable_${comp}[1]}
1318
1319	# Not possible if oasis has an executable (i.e old version of oasis3)
1320	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${comp}" = "XCPL" ] ) ; then
1321	IGCM_debug_Exit "ERROR MPMD with hybrid MPI-OpenMP is not available with oasis3 version"
1322	IGCM_debug_Print 2 "Only available with oasis3-MCT version coupler"
1323	IGCM_debug_Verif_Exit
1324	fi
1325
1326	# Only if we really have an executable for the component :
1327	if [ "X${ExeNameOut}" != X\"\" ] ; then
1328
1329	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1330	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
1331	eval comp_proc_nod_loc=\${${comp}_PROC_NOD}
1332
1333	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
1334	echo "" >> script_${ExeNameOut}.ksh
1335	if [ ${comp_proc_omp_loc} -gt 1 ] ; then
1336
1337	# Check if the number of threads is correct
1338	case ${comp_proc_omp_loc} in
1339	2\|4\|8\|16)
1340	IGCM_debug_Print 1 "You run ${ExeNameOut} on ${comp_proc_omp_loc} OMP threads"
1341	;;
1342	*)
1343	IGCM_debug_Exit "ERROR with OMP parameters !"
1344	IGCM_debug_Print 2 "${comp_proc_omp_loc} is not possible as number of OMP threads"
1345	IGCM_debug_Print 2 "Only 2,4,8,16 as number of OMP threads are possible "
1346	IGCM_debug_Verif_Exit
1347	;;
1348	esac
1349	echo "export KMP_STACKSIZE=3g" >> script_${ExeNameOut}.ksh
1350	echo "export KMP_LIBRARY=turnaround" >> script_${ExeNameOut}.ksh
1351	echo "export MKL_SERIAL=YES" >> script_${ExeNameOut}.ksh
1352	echo "OMP_NUM_THREADS=${comp_proc_omp_loc}" >> script_${ExeNameOut}.ksh
1353	fi
1354
1355	#echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
1356	echo "(( MYMPIRANK = OMPI_COMM_WORLD_RANK )) " >> script_${ExeNameOut}.ksh
1357	echo "MYMPIRANK=\$(printf '%3.3d\n' \${MYMPIRANK})" >> script_${ExeNameOut}.ksh
1358	echo "./${ExeNameOut} > out_${ExeNameOut}.out.\${MYMPIRANK} 2>out_${ExeNameOut}.err.\${MYMPIRANK}" >> script_${ExeNameOut}.ksh
1359	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
1360
1361	if [ ${init_exec} = y ] ; then
1362	EXECUTION="${EXECUTION} : -np ${comp_proc_mpi_loc} ./script_${ExeNameOut}.ksh"
1363	else
1364	EXECUTION="${EXECUTION} -np ${comp_proc_mpi_loc} ./script_${ExeNameOut}.ksh"
1365	init_exec=y
1366	fi
1367
1368	# Build rankfile : method used to assign cores and nodes for the MPI process
1369	# Ex :
1370	#rank 0=curie5296 slot=0,1,2,3
1371	#rank 1=curie5296 slot=4,5,6,7
1372	# Example of final command :
1373	# mpirun -hostfile hosts -rankfile rankfile -np 27 ./script_lmdz.x.ksh : -np 5 ./script_opa.xx.ksh
1374	# with script_lmdz.x.ksh :
1375	# #!/bin/ksh
1376	#export KMP_STACKSIZE=3g
1377	#export KMP_LIBRARY=turnaround
1378	#export MKL_SERIAL=YES
1379	#OMP_NUM_THREADS=4
1380	#./lmdz.x
1381	if [ ${comp_proc_nod_loc} -gt 1 ] ; then
1382	(( offset_comp_proc_loc = NB_CORE_PER_NODE / (comp_proc_mpi_loc / comp_proc_nod_loc) ))
1383	else
1384	(( offset_comp_proc_loc = comp_proc_omp_loc ))
1385	fi
1386	for nb_proc_mpi in `seq 0 $(($comp_proc_mpi_loc-1))`; do
1387	(( index_host = current_core / NB_CORE_PER_NODE ))
1388	host_value=${host[${index_host}]}
1389	(( slot = current_core % NB_CORE_PER_NODE ))
1390	virg=","
1391	string_final=""
1392	for index in `seq $slot $(($slot+$comp_proc_omp_loc-1))`; do
1393	string=$index$virg
1394	string_final=$string_final$string
1395	done
1396	string_final=$( echo $string_final \| sed "s/.$//" )
1397	echo "rank $rank=$host_value slot=$string_final" >> rankfile
1398	(( rank = rank + 1 ))
1399	(( current_core = current_core + offset_comp_proc_loc ))
1400	done
1401	fi
1402	done
1403
1404	elif ( [ "X${config_UserChoices_ExecutionModeOnCurie}" = "X" ] \|\| [ "X${config_UserChoices_ExecutionModeOnCurie}" = "Xccc_mprun" ] ) ; then
1405
1406	# 3 MPI/OpenMP ccc_mprun method
1407
1408	# MPI-OpenMP (MPMD)
1409	# example of run_file
1410	# 71-8 env OMP_NUM_THREADS=8 ./script_lmdz.x.ksh
1411	# 480-1 env OMP_NUM_THREADS=1 ./script_opa.xx.ksh
1412	# 1-1 env OMP_NUM_THREADS=1 ./script_xios.x.ksh
1413
1414	# Loop on the components
1415	for comp in ${config_ListOfComponents[*]} ; do
1416
1417	eval ExeNameIn=\${config_Executable_${comp}[0]}
1418	eval ExeNameOut=\${config_Executable_${comp}[1]}
1419
1420	# Not possible if oasis has an executable (i.e old version of oasis3)
1421	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${comp}" = "XCPL" ] ) ; then
1422	IGCM_debug_Exit "ERROR MPMD with hybrid MPI-OpenMP is not available with oasis3 version"
1423	IGCM_debug_Print 2 "Only available with oasis3-MCT version coupler"
1424	IGCM_debug_Verif_Exit
1425	fi
1426
1427	# Only if we really have an executable for the component :
1428	if [ "X${ExeNameOut}" != X\"\" ] ; then
1429
1430	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1431	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
1432
1433	# Build script files
1434
1435	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
1436	echo "" >> script_${ExeNameOut}.ksh
1437	if [ ${comp_proc_omp_loc} -gt 1 ] ; then
1438
1439	# Check if the number of threads is correct
1440	case ${comp_proc_omp_loc} in
1441	2\|4\|8\|16)
1442	IGCM_debug_Print 1 "You run ${ExeNameOut} on ${comp_proc_omp_loc} OMP threads"
1443	;;
1444	*)
1445	IGCM_debug_Exit "ERROR with OMP parameters !"
1446	IGCM_debug_Print 2 "${comp_proc_omp_loc} is not possible as number of OMP threads"
1447	IGCM_debug_Print 2 "Only 2,4,8,16 as number of OMP threads are possible "
1448	IGCM_debug_Verif_Exit
1449	;;
1450	esac
1451	### only for LMDZ?
1452	echo "export KMP_STACKSIZE=3g" >> script_${ExeNameOut}.ksh
1453	echo "export KMP_LIBRARY=turnaround" >> script_${ExeNameOut}.ksh
1454	echo "export MKL_SERIAL=YES" >> script_${ExeNameOut}.ksh
1455	echo "OMP_NUM_THREADS=${comp_proc_omp_loc}" >> script_${ExeNameOut}.ksh
1456
1457	fi
1458
1459	# to have out/err per process on different files
1460	echo "./${ExeNameOut} > out_${ExeNameOut}.out.\${SLURM_PROCID} 2>out_${ExeNameOut}.err.\${SLURM_PROCID}" >> script_${ExeNameOut}.ksh
1461	#echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
1462
1463	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
1464
1465	# Complete run_file
1466
1467	echo "${comp_proc_mpi_loc}-${comp_proc_omp_loc} env OMP_NUM_THREADS=${comp_proc_omp_loc} ./script_${ExeNameOut}.ksh " >>run_file
1468
1469	fi
1470	done
1471
1472	## variable added to stop after 60s instead of 600s by default.
1473	## This is used when no error comes from executables and when something stopped an executable without notice.
1474	export SLURM_WAIT=60
1475
1476	## module advised by TGCC (instead of 2 variables)
1477	module load feature/bridge/heterogenous_mpmd
1478
1479	# set EXECUTION for ccc_mprun case (similar to #1 : MPI only MPMD method)
1480
1481	EXECUTION="${HOST_MPIRUN_COMMAND} -f ./run_file"
1482
1483	IGCM_sys_Chmod u+x run_file
1484	if ( $DEBUG_sys ) ; then
1485	echo "run_file contains : "
1486	cat run_file
1487	fi
1488
1489	else
1490
1491	IGCM_debug_Print 1 "sys Curie : choose mpirun or ccc_mprun in config.card for ExecutionModeOnCurie"
1492
1493	fi # 1 2 or 3 if ${OK_PARA_MPMD}
1494
1495	else
1496	# Only one executable (SPMD mode).
1497
1498	for comp in ${config_ListOfComponents[*]} ; do
1499
1500	# Only if we really have an executable for the component :
1501	eval ExeNameOut=\${config_Executable_${comp}[1]}
1502	if ( [ "X${ExeNameOut}" != X\"\" ] && [ "X${ExeNameOut}" != "Xinca.dat" ] ) ; then
1503
1504	# Build script files
1505
1506	echo "#!/bin/ksh" > script_${ExeNameOut}.ksh
1507	echo "" >> script_${ExeNameOut}.ksh
1508	IGCM_sys_Chmod u+x script_${ExeNameOut}.ksh
1509
1510	if ( ${OK_PARA_OMP} ) ; then
1511	eval comp_proc_omp_loc=\${${comp}_PROC_OMP}
1512	echo "" >> script_${ExeNameOut}.ksh
1513	echo "export KMP_STACKSIZE=3g" >> script_${ExeNameOut}.ksh
1514	echo "export KMP_LIBRARY=turnaround" >> script_${ExeNameOut}.ksh
1515	echo "export MKL_SERIAL=YES" >> script_${ExeNameOut}.ksh
1516	echo "OMP_NUM_THREADS=${comp_proc_omp_loc}" >> script_${ExeNameOut}.ksh
1517	fi
1518
1519	if ( ${OK_PARA_MPI} ) ; then
1520	eval comp_proc_mpi_loc=\${${comp}_PROC_MPI}
1521	# Default : ccc_mprun used if nb_proc gt 1
1522	# to have out/err per process on different files
1523	echo "./${ExeNameOut} > out_${ExeNameOut}.out.\${SLURM_PROCID} 2>out_${ExeNameOut}.err.\${SLURM_PROCID}" >> script_${ExeNameOut}.ksh
1524	#echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
1525	EXECUTION="${HOST_MPIRUN_COMMAND} -n ${comp_proc_mpi_loc} ./script_${ExeNameOut}.ksh"
1526	else
1527	# Default : ccc_mprun is NOT used if nb_proc eq 1
1528	# to have out/err per process on different files
1529	echo "./${ExeNameOut} > out_${ExeNameOut}.out 2>out_${ExeNameOut}.err" >> script_${ExeNameOut}.ksh
1530	#echo "./${ExeNameOut}" >> script_${ExeNameOut}.ksh
1531	EXECUTION="/usr/bin/time ./script_${ExeNameOut}.ksh"
1532	fi
1533
1534	IGCM_debug_Print 1 "sys Curie : script_${ExeNameOut}.ksh contains"
1535	cat script_${ExeNameOut}.ksh
1536
1537	fi
1538
1539	done
1540
1541	fi # ${OK_PARA_MPMD}
1542
1543	IGCM_debug_Print 1 "sys Curie : execution command is "
1544	IGCM_debug_Print 1 "$EXECUTION"
1545
1546	IGCM_debug_PopStack "IGCM_sys_build_execution_scripts"
1547	}
1548
1549	#D-#==================================================
1550	#D-function IGCM_sys_check_path
1551	#D-* Purpose: check that RUN_DIR_PATH that will be removed on some machine
1552	#D-* do not point to an important use directory. Stop immediately in that case.
1553	#D-* Examples:
1554	#D-
1555	function IGCM_sys_check_path {
1556	IGCM_debug_PushStack "IGCM_sys_check_path"
1557	if ( $DEBUG_sys ) ; then
1558	echo "IGCM_sys_check_path"
1559	fi
1560
1561	if ( [ X${RUN_DIR_PATH} = X${HOME} ] \|\| [ X${RUN_DIR_PATH} = X${CCCWORKDIR} ] \|\| [ X${RUN_DIR_PATH} = X${CCCSCRATCHDIR} ] \|\| [ X${RUN_DIR_PATH} = X${CCCSTOREDIR} ] ) ; then
1562	IGCM_debug_Print 1 "Variable RUN_DIR_PATH is pointing to an important directory : ${RUN_DIR_PATH}"
1563	IGCM_debug_Print 1 "Please check the RUN_DIR_PATH definition in your Job : Job_${config_UserChoices_JobName}"
1564	IGCM_debug_Exit "This will stop the job"
1565	fi
1566	IGCM_debug_PopStack "IGCM_sys_check_path"
1567	}
1568
1569	#D-#==================================================
1570	#D-function IGCM_sys_check_quota
1571	#D-* Purpose: check user quota. Stop the simulation if quota above 90%
1572	#D-* Examples:
1573	#D-
1574	function IGCM_sys_check_quota {
1575	IGCM_debug_PushStack "IGCM_sys_check_quota"
1576	if ( $DEBUG_sys ) ; then
1577	echo "IGCM_sys_check_quota"
1578	fi
1579	# Limit of quota (in %)
1580	limit_quota=90
1581
1582	# Check of the volume
1583	volume_quota=$(ccc_quota \| grep ' scratch' \| gawk '{print $2}')
1584	volume_avail=$(ccc_quota \| grep ' scratch' \| gawk '{print $3}')
1585
1586	if ( [ ! X${volume_quota} = X ] && [ ! ${volume_quota} = "-" ] ) ; then
1587
1588	unit_avail=${volume_avail: -1}
1589	unit_quota=${volume_quota: -1}
1590
1591	if [ "${unit_quota}" = "*" ] ; then
1592	IGCM_debug_Print 1 "Please, check your quota of volume on scratch"
1593	IGCM_debug_Print 1 "More than 100% of your quota is used"
1594	IGCM_debug_Print 1 "Use the ccc_quota command to check"
1595	IGCM_debug_Print 1 "You must have more than 10% available to run"
1596	IGCM_debug_Exit "Not enough space to run ! STOP HERE"
1597	IGCM_debug_Verif_Exit
1598	fi
1599
1600	temp_avail=${volume_avail%%${volume_avail: -1}*}
1601	temp_quota=${volume_quota%%${volume_quota: -1}*}
1602
1603	if [ ! ${unit_avail} = ${unit_quota} ] ; then
1604
1605	# Convertion
1606	if [ ${volume_avail: -1} = "T" ] ; then
1607	(( temp_avail = temp_avail * 1000000000000 ))
1608	elif [ ${volume_avail: -1} = "G" ] ; then
1609	(( temp_avail = temp_avail * 1000000000 ))
1610	elif [ ${volume_avail: -1} = "M" ] ; then
1611	(( temp_avail = temp_avail * 1000000 ))
1612	elif [ ${volume_avail: -1} = "k" ] ; then
1613	(( temp_avail = temp_avail * 1000 ))
1614	else
1615	(( temp_avail = volume_avail ))
1616	fi
1617	if [ ${volume_quota: -1} = "T" ] ; then
1618	(( temp_quota = temp_quota * 1000000000000 ))
1619	elif [ ${volume_quota: -1} = "G" ] ; then
1620	(( temp_quota = temp_quota * 1000000000 ))
1621	elif [ ${volume_quota: -1} = "M" ] ; then
1622	(( temp_quota = temp_quota * 1000000 ))
1623	elif [ ${volume_quota: -1} = "k" ] ; then
1624	(( temp_quota = temp_quota * 1000 ))
1625	else
1626	(( temp_quota = volume_quota ))
1627	fi
1628	fi
1629
1630	quota_volume=$(echo "scale=2 ; $temp_quota/$temp_avail*100" \| bc)
1631	# echo "volume ratio is " $quota_volume
1632
1633	if [ ${quota_volume} -ge ${limit_quota} ] ; then
1634	IGCM_debug_Print 1 "Please, check your quota of volume on scratch"
1635	IGCM_debug_Print 1 "${quota_volume}% of your quota is used"
1636	IGCM_debug_Print 1 "Use the ccc_quota command to check"
1637	IGCM_debug_Print 1 "You must have more than 10% available to run"
1638	IGCM_debug_Exit "Not enough space to run ! STOP HERE"
1639	IGCM_debug_Verif_Exit
1640	fi
1641
1642	fi
1643
1644	# Check of the number of inodes
1645
1646	inode_quota=$(ccc_quota \| grep ' scratch' \| gawk '{print $6}')
1647	inode_avail=$(ccc_quota \| grep ' scratch' \| gawk '{print $7}')
1648
1649	if ( [ ! X${inode_quota} = X ] && [ ! ${inode_quota} = "-" ] ) ; then
1650
1651	unit_avail=${inode_avail: -1}
1652	unit_quota=${inode_quota: -1}
1653
1654	if [ "${unit_quota}" = "*" ] ; then
1655	IGCM_debug_Print 1 "Please, check your quota of inode on scratch"
1656	IGCM_debug_Print 1 "More than 100% of your quota is used"
1657	IGCM_debug_Print 1 "Use the ccc_quota command to check"
1658	IGCM_debug_Print 1 "You must have more than 10% available to run"
1659	IGCM_debug_Exit "Not enough space to run ! STOP HERE"
1660	IGCM_debug_Verif_Exit
1661	fi
1662
1663	temp_avail=${inode_avail%%${inode_avail: -1}*}
1664	temp_quota=${inode_quota%%${inode_quota: -1}*}
1665
1666	if [ ! ${unit_avail} = ${unit_quota} ] ; then
1667
1668	# Convertion
1669	if [ ${inode_avail: -1} = "T" ] ; then
1670	(( temp_avail = temp_avail * 1000000000000 ))
1671	elif [ ${inode_avail: -1} = "G" ] ; then
1672	(( temp_avail = temp_avail * 1000000000 ))
1673	elif [ ${inode_avail: -1} = "M" ] ; then
1674	(( temp_avail = temp_avail * 1000000 ))
1675	elif [ ${inode_avail: -1} = "k" ] ; then
1676	(( temp_avail = temp_avail * 1000 ))
1677	else
1678	(( temp_avail = inode_avail ))
1679	fi
1680
1681	if [ ${inode_quota: -1} = "T" ] ; then
1682	(( temp_quota = temp_quota * 1000000000000 ))
1683	elif [ ${inode_quota: -1} = "G" ] ; then
1684	(( temp_quota = temp_quota * 1000000000 ))
1685	elif [ ${inode_quota: -1} = "M" ] ; then
1686	(( temp_quota = temp_quota * 1000000 ))
1687	elif [ ${inode_quota: -1} = "k" ] ; then
1688	(( temp_quota = temp_quota * 1000 ))
1689	else
1690	(( temp_quota = inode_quota ))
1691	fi
1692	fi
1693	quota_inode=$(echo "scale=2 ; $temp_quota/$temp_avail*100" \| bc)
1694	# echo "inode ratio is " $quota_inode
1695
1696	if [ ${quota_inode} -ge ${limit_quota} ] ; then
1697	IGCM_debug_Print 1 "Please, check your quota of inode on scratch"
1698	IGCM_debug_Print 1 "${quota_inode}% of your quota is used"
1699	IGCM_debug_Print 1 "Use the ccc_quota command to check"
1700	IGCM_debug_Print 1 "You must have more than 10% available to run"
1701	IGCM_debug_Exit "Not enough space to run ! STOP HERE"
1702	IGCM_debug_Verif_Exit
1703	fi
1704	fi
1705	IGCM_debug_PopStack "IGCM_sys_check_quota"
1706	}
1707
1708	#D-#==================================================
1709	#D-function IGCM_sys_projectAccounting
1710	#D-* Purpose: store project accounting information in a file
1711	#D-* Examples:
1712	#D-
1713	function IGCM_sys_projectAccounting {
1714	IGCM_debug_PushStack "IGCM_sys_projectAccounting"
1715	if ( $DEBUG_sys ) ; then
1716	echo "IGCM_sys_check_quota"
1717	fi
1718
1719	ssh curie91 /usr/bin/ccc_myproject > $1
1720
1721	IGCM_debug_PopStack "IGCM_sys_projectAccounting"
1722	}
1723
1724	#D-#==================================================
1725	#D-function IGCM_sys_getJobSchedulerID
1726	#D-* Purpose: Get the job ID during execution
1727	#D-* Examples: IGCM_sys_getJobSchedulerID jobSchedulerID
1728	#D-
1729	function IGCM_sys_getJobSchedulerID {
1730	IGCM_debug_PushStack "IGCM_sys_getJobSchedulerID"
1731	if ( $DEBUG_sys ) ; then
1732	echo "IGCM_sys_getJobSchedulerID"
1733	fi
1734
1735	eval ${1}=${BRIDGE_MSUB_JOBID}
1736
1737	IGCM_debug_PopStack "IGCM_sys_getJobSchedulerID"
1738	}
1739
1740	#D-#==================================================
1741	#D-function IGCM_sys_GetJobID
1742	#D-* Purpose: Get the job ID from the JobName
1743	#D-* Examples: IGCM_sys_GetJobID ${JobName} ${TargetUsr} JobID
1744	#D-
1745	function IGCM_sys_GetJobID {
1746	IGCM_debug_PushStack "IGCM_sys_GetJobID"
1747	if ( $DEBUG_sys ) ; then
1748	echo "IGCM_sys_GetJobID"
1749	fi
1750
1751	# With -f option, the full job name is given in the last column
1752	ID=$( ccc_mstat -f -u $2 \| \
1753	gawk -v JobName=$1 '( $NF ~ JobName ) { print $1 }' )
1754
1755	eval ${3}=${ID}
1756	IGCM_debug_PopStack "IGCM_sys_GetJobID"
1757	}
1758
1759	#D-#==================================================
1760	#D-function IGCM_sys_CountJobInQueue
1761	#D-* Purpose: Count number of users job
1762	#D-* Examples: IGCM_sys_CountJobInQueue ${JobName} NbRun
1763	#D-
1764	function IGCM_sys_CountJobInQueue {
1765	IGCM_debug_PushStack "IGCM_sys_CountJobInQueue"
1766	if ( $DEBUG_sys ) ; then
1767	echo "IGCM_sys_CountJobInQueue"
1768	fi
1769
1770	# With -f option, the full job name is given in the last column
1771	NbRun=$( ccc_mstat -f \| gawk -v JobName=$1 'BEGIN { x=0 } ( $NF ~ JobName ) { x=x+1 } END { print x }' )
1772
1773	eval ${2}=${NbRun}
1774
1775	IGCM_debug_PopStack "IGCM_sys_CountJobInQueue"
1776	}
1777
1778	#D-#==================================================
1779	#D-function IGCM_sys_ListJobInQueue
1780	#D-* Purpose: Produce a list of users computing jobs (excluding post-processing)
1781	#D-* Examples: IGCM_sys_ListJobInQueue ${User} JobNameList
1782	#D-
1783	function IGCM_sys_ListJobInQueue {
1784	IGCM_debug_PushStack "IGCM_sys_ListJobInQueue"
1785	if ( $DEBUG_sys ) ; then
1786	echo "IGCM_sys_ListJobInQueue"
1787	fi
1788
1789	# With -f option, the full job name is given in the last column
1790	set -A JobList $( ccc_mstat -f \| gawk -v User=$1 \
1791	'( $2 == User && \
1792	$NF != /TS/ && \
1793	$NF !~ /PACK/ && \
1794	$NF !~ /REBUILD/ && \
1795	$NF !~ /pack/ ) \
1796	{ print $NF }' \| sed -e "s/\(.\)\.[0-9]/\1/" )
1797
1798	eval set -A ${2} ${JobList[*]}
1799
1800	IGCM_debug_PopStack "IGCM_sys_ListJobInQueue"
1801	}
1802
1803	#D-#==================================================
1804	#D-function IGCM_sys_atlas
1805	#D-* Purpose: encapsulate atlas call so as to manage error code and curie specificity
1806	#D-* Examples:
1807	#D-
1808	function IGCM_sys_atlas {
1809	IGCM_debug_PushStack "IGCM_sys_atlas" $@
1810	if ( $DEBUG_sys ) ; then
1811	echo "IGCM_sys_atlas :" $@
1812	fi
1813
1814	typeset status
1815
1816	\ccc_mprun atlas $@ > ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$ 2>&1
1817	status=$?
1818	if [ ${status} -gt 0 ] ; then
1819	echo "IGCM_sys_atlas : error code ${status}"
1820	cat ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
1821	\rm ${OUTCOMMAND_PATH}/out_command_${LOGIN}.$$
1822	IGCM_debug_PopStack "IGCM_sys_atlas"
1823	return 1
1824	else
1825	IGCM_debug_PopStack "IGCM_sys_atlas"
1826	return 0
1827	fi
1828
1829	IGCM_debug_PopStack "IGCM_sys_atlas"
1830	}
1831
1832	#D-#==================================================
1833	#D-function IGCM_sys_rebuild_nemo
1834	#D-* Purpose: rebuild nemo parallel files with nemo specific rebuild on curie
1835	#D-* Examples:
1836	#D-
1837
1838	function IGCM_sys_rebuild_nemo {
1839	IGCM_debug_PushStack "IGCM_sys_rebuild_nemo" $@
1840	if ( $DEBUG_sys ) ; then
1841	echo "IGCM_sys_rebuild_nemo :" $@
1842	fi
1843
1844	$( ccc_home -u p86ipsl )/REBUILD_NEMO/rebuild_nemo ${1} ${2}
1845
1846	IGCM_debug_PopStack "IGCM_sys_rebuild_nemo"
1847	}
1848

Note: See TracBrowser for help on using the repository browser.

Download in other formats: