Context Navigation

source: trunk/libIGCM/libIGCM_debug/libIGCM_debug.ksh @ 929

Last change on this file since 929 was 926, checked in by sdipsl, 11 years ago
Add instrumentation for IGCM_sys_ncrcat IGCM_sys_ncecat IGCM_sys_ncra IGCM_sys_ncks IGCM_sys_cdo see #140 stack files handling : one stack file per TaskType? : computing.stack, post-processsing, checking.stack
Property licence set to The following licence information concerns ONLY the libIGCM tools ================================================================== Copyright © Centre National de la Recherche Scientifique CNRS Commissariat à l'Énergie Atomique CEA libIGCM : Library for Portable Models Computation of IGCM Group. IGCM Group is the french IPSL Global Climate Model Group. This library is a set of shell scripts and functions whose purpose is the management of the initialization, the launch, the transfer of output files, the post-processing and the monitoring of datas produce by any numerical program on any plateforme. This software is governed by the CeCILL license under French law and abiding by the rules of distribution of free software. You can use, modify and/ or redistribute the software under the terms of the CeCILL license as circulated by CEA, CNRS and INRIA at the following URL "http://www.cecill.info". As a counterpart to the access to the source code and rights to copy, modify and redistribute granted by the license, users are provided only with a limited warranty and the software's author, the holder of the economic rights, and the successive licensors have only limited liability. In this respect, the user's attention is drawn to the risks associated with loading, using, modifying and/or developing or reproducing the software by the user in light of its specific status of free software, that may mean that it is complicated to manipulate, and that also therefore means that it is reserved for developers and experienced professionals having in-depth computer knowledge. Users are therefore encouraged to load and test the software's suitability as regards their requirements in conditions enabling the security of their systems and/or data to be ensured and, more generally, to use and operate it in the same conditions as regards security. The fact that you are presently reading this means that you have had knowledge of the CeCILL license and that you accept its terms. Property svn:keywords set to `Revision Author Date`
File size: 24.5 KB

Line
1	#!/bin/ksh
2
3	#**************************************************************
4	# Author: Patrick Brockmann, Martial Mancip
5	# Contact: Patrick.Brockmann__at__cea.fr Martial.Mancip__at__ipsl.jussieu.fr
6	# $Revision:: $ Revision of last commit
7	# $Author:: $ Author of last commit
8	# $Date:: $ Date of last commit
9	# IPSL (2006)
10	# This software is governed by the CeCILL licence see libIGCM/libIGCM_CeCILL.LIC
11	#
12	#**************************************************************
13
14	#==================================================
15	# The documentation of this file can be automatically generated
16	# if you use the prefix #D- for comments to be extracted.
17	# Extract with command: cat lib* \| grep "^#D-" \| cut -c "4-"
18	#==================================================
19
20	#==================================================
21	# Add high level verbosity
22	typeset -i Verbosity=${Verbosity:=3}
23
24	#==================================================
25	# DEBUG_debug
26	# Add low level verbosity
27	typeset DEBUG_debug=${DEBUG_debug:=false}
28
29	#==================================================
30	# GENERATE RANDOM ERROR ; only apply if ( ${DEBUG_debug} )
31	typeset RandomError=false
32
33	# Stack file containing call tree will be stored there.
34	if ( $DEBUG_debug ) ; then
35	if [ X${TaskType} = Xcomputing ]; then
36	typeset StackFileLocation=${StackFileLocation:=${PWD}}
37	typeset StackFileName=computing.stack.$$
38	elif [ X${TaskType} = Xpost-processing ]; then
39	typeset StackFileLocation=${POST_DIR:=${StackFileLocation}}
40	typeset StackFileName=post-processing.stack.$$
41	elif [ X${TaskType} = Xchecking ]; then
42	typeset StackFileLocation=${POST_DIR:=${PWD}}
43	typeset StackFileName=checking.stack.$$
44	fi
45	if [ -f ${StackFileLocation}/${StackFileName} ] ;
46	then
47	echo "Stack of an libIGCM job :" >> ${StackFileLocation}/${StackFileName}
48	else
49	echo "Stack of an libIGCM job :" > ${StackFileLocation}/${StackFileName}
50	fi
51	fi
52
53	#==================================================
54	# NULL_STR
55	# Default null string
56	typeset -r NULL_STR="_0_"
57
58	#==================================================
59	# libIGCM_CurrentTag
60	# Current libIGCM tag, check compatibilty with *.card
61	typeset -r libIGCM_CurrentTag="1.0"
62
63	#==================================================
64	# Exit Flag (internal debug)
65	# When true, end the master loop AFTER SAVES FILES
66	ExitFlag=false
67
68	#==================================================
69	# Declare a stack of functions calls
70	unset IGCM_debug_Stack
71	unset IGCM_debug_StackArgs
72	unset IGCM_debug_StackTiming
73	IGCM_debug_Stack[0]=${NULL_STR}
74	IGCM_debug_StackArgs[0]=${NULL_STR}
75	IGCM_debug_StackTiming[0]=${NULL_STR}
76	IGCM_debug_LenStack=0
77
78	#D-#==================================================================
79	#D-function IGCM_debug_getDate_ms
80	#D- * Purpose: Give number of milliseconds since 01-jan-1970
81	function IGCM_debug_getDate_ms
82	{
83	typeset nanosecs ms
84	# nano secondes since 01-jan-1970
85	nanosecs=$( date +%s%N )
86
87	# truncate the last 6 digits to get milliseconds since 01-jan-1970
88	ms=${nanosecs:0:${#nanosecs}-6}
89
90	echo "$ms"
91	}
92
93	#D-#==================================================================
94	#D-function IGCM_debug_sizeOfTabContent
95	#D- * Purpose: Give sumed size of a list of files
96	#D- * Usage: IGCM_debug_sizeOfTabContent entityList destination
97	#D- * where entityList is a list of files or directory
98	#D- * where dest is either a directory or a file name
99	function IGCM_debug_sizeOfTabContent
100	{
101	typeset entityListe destination iEntity sizeKo sumSizeKo sumSizeMo
102
103	eval set +A entityListe \${${1}}
104	destination=${2}
105	sumSizeKo=0
106
107	# Here we will try to compute size (file or directory size) from local path and not from archive.
108	for ((i = 0; i < ${#entityListe[*]}; i += 1)) ; do
109	if [ -f ${entityListe[$i]##/*/} ] ; then
110	# One file or a bunch of files has been copied without renaming
111	# remove path /home/login/../ from entityListe elements
112	iEntity=${entityListe[$i]##/*/}
113	elif [ -f ${destination} ] ; then
114	# a file has been copied and rename
115	iEntity=${destination}
116	elif [ -f ${destination}/${entityListe[$i]##/*/} ] ; then
117	# a copy in a directory but not in ${PWD}
118	iEntity=${destination}/${entityListe[$i]##/*/}
119	elif [ -d ${entityListe[$i]} ] ; then
120	# directory has been copied from non remote place
121	iEntity=${entityListe[$i]}
122	elif [ -d ${destination}/${entityListe[$i]##/*/} ] ; then
123	# directory has been copied from remote archive and not rename
124	iEntity=${destination}/${entityListe[$i]##/*/}
125	elif [ -d ${destination} ] ; then
126	# directory has been copied from remote archive and rename
127	iEntity=${destination}
128	fi
129	sizeKo=$( du --apparent-size -sk ${iEntity} \| awk {'print $1'} )
130	sumSizeKo=$(( $sumSizeKo + $sizeKo ))
131	done
132	sumSizeMo=$( echo "scale=6;${sumSizeKo}/1024" \| bc )
133	echo "${sumSizeKo}\|${sumSizeMo}"
134
135	}
136
137	#D-#==================================================================
138	#D-function IGCM_debug_SendAMQP
139	#D- * Purpose: Send body; encoded body and config.card to rabbitMQ
140	function IGCM_debug_sendAMQP
141	{
142	typeset decal first additionnalOption encodedBody
143
144	# Encode message Body
145	encodedBody=$( echo "${Body}" \| base64 -w 0 )
146
147	# Send config.card ?
148	if [ X${1} = Xactivate ] ; then
149	# Encode config.card
150	cat ${SUBMIT_DIR}/config.card \| base64 -w 0 > ${SUBMIT_DIR}/config.card.base64
151	# Prepare additionnal option
152	additionnalOption="-f ${SUBMIT_DIR}/config.card.base64"
153	else
154	additionnalOption=
155	fi
156
157	# Only cosmetics : stack file
158	decal=0
159	while [ ${decal} -lt ${IGCM_debug_LenStack} ]; do
160	printf ' ' >> ${StackFileLocation}/${StackFileName}
161	(( decal = decal + 1 ))
162	done
163	# Log to stack file
164	echo sendAMQPMsg -h localhost -p 5672 ${additionnalOption} -b "${Body}" >> ${StackFileLocation}/${StackFileName}
165
166	# Log separately AMQP send message command
167	echo sendAMQPMsg -h localhost -p 5672 ${additionnalOption} -b ${encodedBody} >> /tmp/send.AMQP.${jobid}.history.txt
168
169	# Send the message
170	sendAMQPMsg -h localhost -p 5672 ${additionnalOption} -b ${encodedBody}
171	status=$?
172	if [ ${status} -gt 0 ] ; then
173	IGCM_debug_Print 2 "IGCM_debug_Push/PopStack/ActivateBigBro : command sendAMQPMsg failed error code ${status}"
174	echo sendAMQPMsg -h localhost -p 5672 -b "${Body}"
175	exit
176	fi
177	}
178
179	#D-#==================================================================
180	#D-function IGCM_debug_CallStack
181	#D-* Purpose: Print the call stack tree from the oldest to the youngest (opposite of the display standard)
182	#D-
183	function IGCM_debug_CallStack {
184	if ( $DEBUG_debug ) ; then
185	# Cosmetics
186	typeset i decal
187	i=0
188	until [ $i -eq ${IGCM_debug_LenStack} ]; do
189	decal=0
190	until [ $decal -eq ${i} ]; do
191	printf -- ' '
192	(( decal = decal + 1 ))
193	done
194	echo "$i - ${IGCM_debug_Stack[$(( $IGCM_debug_LenStack-$i-1 ))]}" "(${IGCM_debug_StackArgs[$(( $IGCM_debug_LenStack-$i-1 ))]})"
195	((i = i + 1))
196	done
197	fi
198	}
199
200	#D-#==================================================================
201	#D-function IGCM_debug_PushStack
202	#D-* Purpose: Push a function name in the stack
203	#D-
204	function IGCM_debug_PushStack {
205	if ( $DEBUG_debug ) ; then
206	typeset decal inputs startTime_ms
207
208	# Only cosmetics : stack file
209	echo >> ${StackFileLocation}/${StackFileName}
210	decal=0
211	while [ ${decal} -lt ${IGCM_debug_LenStack} ]; do
212	printf ' ' >> ${StackFileLocation}/${StackFileName}
213	(( decal = decal + 1 ))
214	done
215
216	# Fill the stack file
217	echo "> ${IGCM_debug_LenStack} : ${@}" >> ${StackFileLocation}/${StackFileName}
218
219	# Save input list in an indexed array
220	INPUTS=( $@ )
221
222	# Get timing information
223	startTime_ms=$( IGCM_debug_getDate_ms )
224
225	# We add function call name on beginning of the stack
226	set +A IGCM_debug_Stack -- ${1} ${IGCM_debug_Stack[*]}
227
228	# Save timing in milliseconds in an indexed array
229	set +A IGCM_debug_StackTiming -- ${startTime_ms} ${IGCM_debug_StackTiming[*]}
230
231	# We include the "null" Args in the beginning of the StackArgs
232	set +A IGCM_debug_StackArgs ${NULL_STR} ${IGCM_debug_StackArgs[*]}
233
234	# Then, we shift StackArgs tabular
235	if [ $# -gt 1 ]; then
236	IGCM_debug_StackArgs[0]=$(echo ${INPUTS[*]:1} \| sed -e "s/\ /,/g" )
237	fi
238
239	if [ X${ActivateBigBro} = Xtrue ] ; then
240	# RabbitMQ message code
241	code=2000
242	# RabbitMQ message body
243	Body=$( echo "{\"code\":\"${code}\",\"simuid\":\"${simuid}\",\"jobid\":\"${jobid}\",\"nesting\":\"${IGCM_debug_LenStack}\",\"command\":\"${INPUTS[*]}\",\"timestamp\":\"$( date +"%Y-%m-%d-%T" )\"}" )
244	# Fill the rabbitMQ queue
245	IGCM_debug_sendAMQP
246	fi
247
248	# Increment LenStack
249	(( IGCM_debug_LenStack = IGCM_debug_LenStack + 1 ))
250
251	#IGCM_debug_CallStack
252	fi
253	}
254
255	#D-#==================================================================
256	#D-function IGCM_debug_PopStack
257	#D-* Purpose: Pop a function name in the stack
258	#D-
259	function IGCM_debug_PopStack {
260	if ( $DEBUG_debug ) ; then
261	typeset i decal command arguments startTime_ms endTime_ms
262	typeset instrumentation dest
263	# they are not typeset because they are send "by adress" to son functions
264	# we unset them to avoid "memory effect"
265	unset fileList source
266
267	# INTRODUCE SIMPLE ERROR GENERATOR TO TEST SUPERVISOR
268	# PROBABILITY ERROR IS 0.0001 PER COMMAND OR FUNCTION CALL
269	# THERE ARE ~500 COMMAND OR FUNCTION CALL PER PERIOD
270	if ( ${RandomError} ) ; then
271	if [ $((RANDOM%10000)) -le 10 ] ; then
272	IGCM_debug_Print 1 "Random error has been triggered"
273	echo "RANDOM ERROR" >> ${StackFileLocation}/${StackFileName}
274	ExitFlag=true
275	fi
276	fi
277
278	if [ "${IGCM_debug_Stack[0]}" = "${1}" ]; then
279	# Everything is cool
280
281	# Get timing information
282	endTime_ms=$( IGCM_debug_getDate_ms )
283
284	# Save Stack information before poping the stack
285	command=${IGCM_debug_Stack[0]}
286
287	# Go from comma separated (list) to space separated in an array
288	set -A arguments $( echo ${IGCM_debug_StackArgs[0]} \| sed -e "s/,/\ /g" )
289
290	# Save Stack information before poping the stack
291	startTime_ms=${IGCM_debug_StackTiming[0]}
292
293	# Pop the stack
294	(( IGCM_debug_LenStack = IGCM_debug_LenStack - 1 ))
295	set -A IGCM_debug_Stack -- ${IGCM_debug_Stack[*]:1}
296	set -A IGCM_debug_StackArgs -- ${IGCM_debug_StackArgs[*]:1}
297	set -A IGCM_debug_StackTiming -- ${IGCM_debug_StackTiming[*]:1}
298	else
299	echo 'IGCM_debug_Exit : stack is corrupted ! LenStack =' ${IGCM_debug_LenStack}
300	IGCM_debug_Exit $@
301	fi
302
303	# Special actions depending on command to prepare IGCM_debug_PrintInfosActions call
304	# We are interested in:
305	# 0. Which command performs the work
306	# 1. Size of entity we are working with
307	# 2. Where are we reading
308	# 3. Where are we writing
309	# 4. How long it took
310
311	instrumentation=false
312
313	case ${command} in
314	# Classical copy (only files are given to IGCM_sys_Cp as options)
315	IGCM_sys_Cp)
316	instrumentation=true
317	# All but the latest
318	fileList=${arguments[]:0:${#arguments[]}-1}
319	# just need the first file to get the directory
320	source=${arguments[0]}
321	# Nothing but the latest
322	dest=${arguments[${#arguments[*]}-1]}
323	# Size of file whose name are stored in a list
324	entitySize=$( IGCM_debug_sizeOfTabContent fileList ${dest} )
325	;;
326
327	# Copy from archive machine or from buffer
328	IGCM_sys_Get\|IGCM_sys_GetBuffer)
329	instrumentation=true
330	if [ ${#arguments[*]} -eq 2 ] ; then
331	source=${arguments[0]}
332	dest=${arguments[1]}
333	# Size of file whose name are stored in a variable
334	entitySize=$( IGCM_debug_sizeOfTabContent source ${dest} )
335	elif ( [ ${#arguments[*]} -eq 3 ] && [ ${arguments[0]} = '/l' ] ) ; then
336	# Keep the array name hosting the all list
337	fileList=${arguments[0]}
338	# just need the first file to get the directory
339	eval source=\${${arguments[0]}[0]}
340	dest=${arguments[1]}
341	# Size of file whose name are stored in an array
342	entitySize=$( IGCM_debug_sizeOfTabContent fileList[*] ${dest} )
343	elif [ [ ${#arguments[*]} -ge 3 ] ; then
344	# All but the latest
345	fileList=${arguments[]:0:${#arguments[]}-1}
346	# just need the first file to get the directory
347	source=${arguments[0]}
348	# Nothing but the latest
349	dest=${arguments[${#arguments[*]}-1]}
350	# Size of file whose name are stored in a list
351	entitySize=$( IGCM_debug_sizeOfTabContent fileList ${dest} )
352	fi
353	;;
354
355	# Copy from compute node or copy to archive/buffer
356	IGCM_sys_Get_Master\|IGCM_sys_Get_Dir\|IGCM_sys_Put_Out\|IGCM_sys_PutBuffer_Out)
357	instrumentation=true
358	source=${arguments[0]}
359	dest=${arguments[1]}
360	# Size of file whose name are stored in a variable
361	entitySize=$( IGCM_debug_sizeOfTabContent source ${dest} )
362	;;
363
364	# Rebuild command
365	IGCM_sys_rebuild\|IGCM_sys_rebuild_station)
366	instrumentation=true
367	# All but the first
368	fileList=${arguments[]:1:${#arguments[]}-1}
369	# just need a file to get the directory
370	source=${arguments[1]}
371	# Nothing but the first
372	dest=${arguments[0]}
373	# Size of file whose name are stored in a list
374	entitySize=$( IGCM_debug_sizeOfTabContent fileList ${dest} )
375	;;
376
377	# NCO commands
378	IGCM_sys_ncrcat\|IGCM_sys_ncecat\|IGCM_sys_ncra\|IGCM_sys_ncks\|IGCM_sys_cdo)
379	# Example of what we want to catch : only filenames in those command lines
380	# IGCM_sys_ncrcat -O -v ${list_var_final_ncrcat} ${OUT_SE[*]} ${RESULT_SE}
381	# IGCM_sys_ncrcat --hst -v ${liste_coord}${var} ${file1} ${liste_file_tmp[*]} ${file_out}
382	# IGCM_sys_ncrcat -p ${dir} ${liste_file_tmp} --output ${output}
383	# IGCM_sys_ncrcat -x -v ${list_var} -p ${dir} ${liste_file_tmp} --output ${output}
384	instrumentation=true
385	# All files that are not the latest argument (only input files)
386	i=0
387	keepGoing=true
388	while ( ${keepGoing} ) ; do
389	if [ -f ${arguments[${#arguments[*]}-2-${i}]} ] ; then
390	fileList="${fileList} ${arguments[${#arguments[*]}-2-${i}]}"
391	((i = i + 1))
392	elif [ ${i} -ge 1 ] ; then
393	keepGoing=false
394	else
395	((i = i + 1))
396	fi
397	done
398	# i value is at least 1
399	# just need one file to get the directory
400	source=${arguments[${#arguments[*]}-1-${i}]}
401	# Nothing but the latest
402	dest=${arguments[${#arguments[*]}-1]}
403	# Size of file whose name are stored in a list
404	entitySize=$( IGCM_debug_sizeOfTabContent fileList ${dest} )
405	;;
406	esac
407
408	# Print information related to instrumentation
409	( ${instrumentation} ) && IGCM_debug_PrintInfosActions ${command} ${entitySize} ${startTime_ms} ${endTime_ms} ${dest} ${source}
410
411	# Only cosmetics : stack file
412	decal=0
413	while [ ${decal} -lt ${IGCM_debug_LenStack} ]; do
414	printf ' ' >> ${StackFileLocation}/${StackFileName}
415	(( decal = decal + 1 ))
416	done
417
418	if ( ${ExitFlag} ) ; then
419	# Inform the stack file
420	echo '!!! ExitFlag has been activated !!!' >> ${StackFileLocation}/${StackFileName}
421
422	if [ X${ActivateBigBro} = Xtrue ] ; then
423	# RabbitMQ message code
424	code=9000
425	# RabbitMQ message body
426	Body=$( echo "{\"code\":\"${code}\",\"simuid\":\"${simuid}\",\"jobid\":\"${jobid}\",\"status\":\"NOK\",\"out\":\"true\",\"nesting\":\"${IGCM_debug_LenStack}\",\"command\":\"${INPUTS[*]}\",\"timestamp\":\"$( date +"%Y-%m-%d-%T" )\"}" )
427	# Fill the rabbitMQ queue
428	IGCM_debug_sendAMQP
429	fi
430	else
431	# Inform the stack file
432	echo "< ${IGCM_debug_LenStack} : ${@}" >> ${StackFileLocation}/${StackFileName}
433	if [ X${ActivateBigBro} = Xtrue ] ; then
434	# RabbitMQ message code
435	code=3000
436	# RabbitMQ message body
437	Body=$( echo "{\"code\":\"${code}\",\"simuid\":\"${simuid}\",\"jobid\":\"${jobid}\",\"status\":\"OK\",\"out\":\"true\",\"nesting\":\"${IGCM_debug_LenStack}\",\"command\":\"${INPUTS[*]}\",\"timestamp\":\"$( date +"%Y-%m-%d-%T" )\"}" )
438	# Fill the rabbitMQ queue
439	IGCM_debug_sendAMQP
440	fi
441	fi
442
443	# Reset array if necessary
444	if [ ${IGCM_debug_LenStack} = 0 ]; then
445	#echo
446	#IGCM_debug_Print 3 "Clean stack array"
447	#echo
448	unset IGCM_debug_Stack
449	unset IGCM_debug_StackArgs
450	unset IGCM_debug_StackTiming
451	IGCM_debug_Stack[0]=${NULL_STR}
452	IGCM_debug_StackArgs[0]=${NULL_STR}
453	IGCM_debug_StackTiming[0]=${NULL_STR}
454	fi
455	fi
456	#IGCM_debug_CallStack
457	}
458
459	#D-#==================================================================
460	#D-function IGCM_debug_ActivateBigBro
461	#D-* Purpose: switch rabbitMQ on
462	#D-
463	function IGCM_debug_ActivateBigBro {
464	IGCM_debug_PushStack "IGCM_debug_ActivateBigBro"
465
466	if [ X${BigBrother} = Xtrue ] ; then
467	# ID to identify a simulation
468	simuid=${config_UserChoices_JobName}.${config_UserChoices_ExperimentName}.${config_UserChoices_SpaceName}.${config_UserChoices_TagName}.p86denv.TGCC.CURIE
469
470	# ID to identify a job. Several Jobs are needed to complete a simulation
471	jobid=${config_UserChoices_JobName}.${config_UserChoices_ExperimentName}.${config_UserChoices_SpaceName}.${config_UserChoices_TagName}.p86denv.TGCC.CURIE.${CumulPeriod}
472
473	# RabbitMQ message code
474	if ( ${FirstInitialize} ) ; then
475	code=0000
476	else
477	code=1000
478	fi
479	# RabbitMQ message body
480	Body=$( echo "{\"code\":\"${code}\",\"simuid\":\"${simuid}\",\"jobid\":\"${jobid}\",\"status\":\"OK\",\"out\":\"false\",\"nesting\":\"${IGCM_debug_LenStack}\",\"timestamp\":\"$( date +"%Y-%m-%d-%T" )\"}" )
481	# Fill the rabbitMQ queue (specifying config.card must be send)
482	IGCM_debug_sendAMQP activate
483
484	# Turn the flag on
485	ActivateBigBro=true
486	fi
487	IGCM_debug_PopStack "IGCM_debug_ActivateBigBro"
488	}
489
490	#D-#==================================================================
491	#D-function IGCM_debug_Exit
492	#D-* Purpose: Print Call Stack and set ExitFlag to true
493	#D-
494	function IGCM_debug_Exit {
495	IGCM_debug_PushStack "IGCM_debug_Exit"
496	echo "IGCM_debug_Exit : " "${@}"
497	echo
498	echo "!!!!!!!!!!!!!!!!!!!!!!!!!!"
499	echo "!! ERROR TRIGGERED !!"
500	echo "!! EXIT FLAG SET !!"
501	echo "!------------------------!"
502	echo
503	IGCM_debug_CallStack
504	ExitFlag=true
505	IGCM_debug_PopStack "IGCM_debug_Exit"
506	}
507
508	#D-#==================================================
509	#D-function IGCM_debug_Verif_Exit
510	#D-* Purpose: exit with number 1 if ExitFlag is true
511	#D-
512	function IGCM_debug_Verif_Exit {
513	if ( ${ExitFlag} ) ; then
514	# Plan to send an email here with IGCM_sys_SendMail
515	if [ X${TaskType} != Xchecking ] ; then
516	IGCM_card_WriteOption ${SUBMIT_DIR}/run.card Configuration PeriodState "Fatal"
517	echo "IGCM_debug_Verif_Exit : Something wrong happened previously."
518	echo "IGCM_debug_Verif_Exit : ERROR and EXIT keyword will help find out where."
519	echo " EXIT THE JOB."
520	echo
521	IGCM_debug_CallStack
522	fi
523
524	if [ X${ActivateBigBro} = Xtrue ] ; then
525	# RabbitMQ message code
526	code=9999
527	# RabbitMQ message body
528	Body=$( echo "{\"code\":\"${code}\",\"simuid\":\"${simuid}\",\"jobid\":\"${jobid}\",\"status\":\"FATAL\",\"timestamp\":\"$( date +"%Y-%m-%d-%T" )\"}" )
529	# Fill the rabbitMQ queue
530	IGCM_debug_sendAMQP
531	fi
532
533	# Not sure about that one ...
534	if ( $DEBUG_debug ) ; then
535	echo "Your files on ${R_OUT} :"
536	IGCM_sys_Tree ${R_SAVE}
537	echo
538	fi
539
540	# Mail notification
541	IGCM_sys_SendMail
542
543	# And Good Bye
544	date
545	exit 1
546	fi
547	}
548
549	#D-#==================================================
550	#D-function IGCM_debug_Verif_Exit_Post
551	#D-* Purpose: exit with number 1 if ExitFlag is true for Post-treatment
552	#D-
553	function IGCM_debug_Verif_Exit_Post {
554	if ( ${ExitFlag} ) ; then
555	echo "IGCM_debug_Verif_Exit_Post : Something wrong happened."
556	# If SpaceName is PROD then we stop if post_processing fails
557	# Plan to send an email here with IGCM_sys_SendMail
558	if [ X${config_UserChoices_SpaceName} = XPROD ] ; then
559	echo " EXIT THE JOB."
560	echo
561	# Mail notification
562	#IGCM_sys_SendMailPost
563	# And Good Bye
564	date
565	exit 1
566	else
567	echo "Either inside config.card the variable SpaceName is not in PROD"
568	echo "or inside the main Job the variable JobType is not in RUN mode"
569	echo " SO WE DO NOT EXIT THE JOB."
570	echo
571	date
572	fi
573	fi
574	}
575
576	#D-#==================================================================
577	#D-function IGCM_debug_Print
578	#D-* Purpose: Print arguments according to a level of verbosity.
579	#D-
580	function IGCM_debug_Print
581	{
582	typeset level=$1
583	shift
584
585	if [ X"${1}" = X"-e" ]; then
586	typeset cmd_echo="echo -e"
587	shift
588	else
589	typeset cmd_echo="echo"
590	fi
591
592	if [ ${level} -le ${Verbosity} ] ; then
593	typeset i
594	case "${level}" in
595	1) for i in "$@" ; do
596	${cmd_echo} $(date +"%Y-%m-%d %T") "--Debug1-->" ${i}
597	done ;;
598	2) for i in "$@" ; do
599	${cmd_echo} $(date +"%Y-%m-%d %T") "--------Debug2-->" ${i}
600	done ;;
601	3) for i in "$@" ; do
602	${cmd_echo} $(date +"%Y-%m-%d %T") "--------------Debug3-->" ${i}
603	done ;;
604	esac
605	fi
606	}
607
608	#D-#==================================================================
609	#D-function IGCM_debug_PrintVariables
610	#D-* Purpose: Print arguments when match a pattern
611	#D- according to a level of verbosity.
612	function IGCM_debug_PrintVariables
613	{
614	typeset level=$1
615	shift
616
617	list=$( set \| grep ^$1 \| sed -e "s/'//g" )
618
619	if [ "X${list}" != X ] ; then
620	IGCM_debug_Print ${level} ${list}
621	fi
622	}
623
624	#D-#==================================================================
625	#D-function IGCM_debug_PrintInfosActions
626	#D-* Purpose: Print information related to instrumentation
627	function IGCM_debug_PrintInfosActions
628	{
629	typeset actionType=$1
630	typeset entitySize=$2
631	typeset start_ms=$3
632	typeset end_ms=$4
633
634	typeset dest=$5
635	typeset source=$6
636
637	typeset diff_ms entitySizeKo entitySizeMo flux_Ko_ms flux_Ko_s flux_Mo_s
638	typeset name dirOut dirSource
639
640	name="doNotKnow"
641
642	diff_ms=$(( $end_ms - $start_ms ))
643	# echo "diff_ms=$diff_ms"
644
645	entitySizeKo=$( echo ${entitySize} \| gawk -F"\|" '{print $1}' )
646	# echo "entitySizeKo=$entitySizeKo"
647	entitySizeMo=$( echo ${entitySize} \| gawk -F"\|" '{print $2}' )
648
649	# flux en Ko / ms
650	flux_Ko_ms=$( echo "scale=6;${entitySizeKo}/${diff_ms}" \| bc )
651	# echo "flux_Ko_ms=$flux_Ko_ms"
652
653	# flux en Ko / s
654	flux_Ko_s=$(( $flux_Ko_ms * 1000 ))
655	# echo "flux_Ko_s=$flux_Ko_s"
656
657	# flux en Mo / s
658	flux_Mo_s=$( echo "scale=6;${flux_Ko_s}/1024" \| bc )
659	# echo "flux_Mo_s=$flux_Mo_s"
660
661	if [ -d $dest ] ; then
662	dirOut=$( readlink -f ${dest} )
663	else
664	dirOut=$( readlink -f $( dirname ${dest} ) )
665	fi
666
667	if [ -d $source ] ; then
668	dirSource=$( readlink -f ${source} )
669	else
670	dirSource=$( readlink -f $( dirname ${source} ) )
671	fi
672
673	echo "==>act:${actionType}\|sz:${entitySizeMo}\|ms:${diff_ms}\|fx(ko):${flux_Ko_s}\|fx(mo):${flux_Mo_s}\|nm:${name}\|dirSource:${dirSource}\|dirOut:${dirOut}"
674	}
675
676	#D-#==================================================================
677	#D-function IGCM_debug_Check
678	#D- * Purpose: Check the present file by comparison with a reference file
679	function IGCM_debug_Check
680	{
681	#---------------------
682	if [ ! -n "${libIGCM}" ] ; then
683	echo "Check libIGCM_debug ..........................................[ FAILED ]"
684	echo "--Error--> libIGCM variable is not defined"
685	exit 2
686	fi
687
688	#---------------------
689	if [ ! -n "${Verbosity}" ] ; then
690	echo "Check libIGCM_debug ..........................................[ FAILED ]"
691	echo "--Error--> Verbosity variable is not defined"
692	exit 3
693	fi
694
695	#---------------------
696	${libIGCM}/libIGCM_debug/IGCM_debug_Test.ksh > /tmp/IGCM_debug_Test.ref.failed 2>&1
697	sleep 2
698
699	# Remove date stamp.
700	sed -e "s:[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9] [0-9][0-9]\:[0-9][0-9]\:[0-9][0-9] ::g" /tmp/IGCM_debug_Test.ref.failed > /tmp/IGCM_debug_Test.ref.failed.nodate
701	mv /tmp/IGCM_debug_Test.ref.failed.nodate /tmp/IGCM_debug_Test.ref.failed
702
703	if diff /tmp/IGCM_debug_Test.ref.failed ${libIGCM}/libIGCM_debug/IGCM_debug_Test.ref > /dev/null 2>&1 ; then
704	echo "Check libIGCM_debug ..............................................[ OK ]"
705	rm -f /tmp/IGCM_debug_Test.ref.failed
706	else
707	echo "Check libIGCM_debug ..........................................[ FAILED ]"
708	echo "--Error--> Execution of ${libIGCM}/libIGCM_debug/IGCM_debug_Test.ksh"
709	echo " has produced the file IGCM_debug_Test.ref.failed"
710	echo " Please analyse differences with the reference file by typing:"
711	echo " diff IGCM_debug_Test.ref.failed ${libIGCM}/libIGCM_debug/IGCM_debug_Test.ref"
712	echo " Report errors to the author: Patrick.Brockmann@cea.fr"
713	cat /tmp/IGCM_debug_Test.ref.failed
714	exit 4
715	fi
716	#---------------------
717	}

Note: See TracBrowser for help on using the repository browser.

Download in other formats: