New URL for NEMO forge!   http://forge.nemo-ocean.eu

Since March 2022 along with NEMO 4.2 release, the code development moved to a self-hosted GitLab.
This present forge is now archived and remained online for history.
#2556 (Sette test not catching failed REPRO runs) – NEMO

Opened 4 years ago

Closed 4 years ago

#2556 closed Bug (fixed)

Sette test not catching failed REPRO runs

Reported by: ayoung Owned by: ayoung
Priority: low Milestone:
Component: tools Version: trunk
Severity: minor Keywords:
Cc:

Description

Context

Sette test did not catch failed reproducibility run and reported the task as "passed".

Analysis

The sette script finds the two REPRO jobs to compare by grepping the folder for "REPRO", assuming that there will be two matches.

# check reproducibility
rep1=ls -1rt $vdir/$nam/$mach/$dorv/ | grep REPRO | tail -2l | head -1
rep2=ls -1rt $vdir/$nam/$mach/$dorv/ | grep REPRO | tail -1l

This is fine if both jobs run or if neither job runs. However, just one of the jobs fail, there will only be one "REPRO*" directory in the folder hence rep1 and rep2 will have the same value. The result is that sette will compare the single REPRO job against itself and reports the job as "passed".

Fix

Check that rep1 != rep2 before continuing.

Commit History (1)

ChangesetAuthorTimeChangeLog
13785ayoung2020-11-12T10:38:57+01:00

Catching the scenario where one REPRO run is successful and the other fails. See ticket #2556

Change History (3)

comment:1 Changed 4 years ago by ayoung

I propose the following change:

# check reproducibility
rep1=`ls -1rt $vdir/$nam/$mach/$dorv/ | grep REPRO | tail -2l | head -1 `
rep2=`ls -1rt $vdir/$nam/$mach/$dorv/ | grep REPRO | tail -1l`
+ if [ $rep1 == $rep2 ]; then
+    rep2=''
+ fi
f1o=$vdir/$nam/$mach/$dorv/$rep1/ocean.output
f1s=$vdir/$nam/$mach/$dorv/$rep1/run.stat
f1t=$vdir/$nam/$mach/$dorv/$rep1/tracer.stat
f2o=$vdir/$nam/$mach/$dorv/$rep2/ocean.output
f2s=$vdir/$nam/$mach/$dorv/$rep2/run.stat
f2t=$vdir/$nam/$mach/$dorv/$rep2/tracer.stat

i.e. if the rep1 and rep2 are pointing to the same REPRO directories, set the second one to an empty string.

comment:2 Changed 4 years ago by ayoung

In 13785:

Error: Failed to load processor CommitTicketReference
No macro or processor named 'CommitTicketReference' found

comment:3 Changed 4 years ago by ayoung

  • Resolution set to fixed
  • Status changed from new to closed
Note: See TracTickets for help on using tickets.