Faulty option 1 of the freshwater budget adjustment mechanism (`nn_fwb = 1`, module `sbcfwb`) when both `key_mpp_mpi` and `key_mpi2` are defined

[smueller]

Context

After changing the freshwater budget correction to option 1 in a variant of the ORCA2_ICE_PISCES reference configuration (OCE without icebergs, ICE unchanged, TOP excluded), model outputs from runs which differ in the restart-file output frequency (different values of nn_stock) started to diverge following the first output of a restart file. A test of restartability, however, was found to be successful.

Analysis

An inspection of the code related to restart-file output indicated that the subroutine mpp_delay_rcv of module lib_mpp (called indirectly by subroutine rst_write of module restart via subroutine iom_delay_rst of module iom) could potentially indirectly affect the model forcing during the output of restart files, as it enforces the otherwise delayed evaluation of the freshwater budget in an MPI context (key_mpp_mpi was defined when the problem occurred) and thus reveal an error elsewhere. Further analysis revealed the incorrect implementation of the fallback option of MPI-2 compatibility (key_mpi2 was defined when the problem occurred): an incorrect argument list in the substitute mpi_allreduce calls leads to the storage of inappropriate request-id values in variable ndelayid, and, as a result, subroutine iom_delay_rcv not being called in the subsequent execution of the delayed operation; in the case of iom_delay_sum this omission results in the received value not being propagated to the output variable of subroutine mpi_delay_sum, even if the value has been received in the meantime. An exception to this erroneous behaviour occurs during the first timestep as well as during timesteps that output restart files, during which subroutine mpp_delay_rcv is called irrespective of the request-id value; as a result, the freshwater budget remains uncontrolled between these time steps.

Fix

The argument list of the mpi_allreduce calls that substitute mpi_iallreduce calls when key_mpi2 is defined should be corrected, and, following such mpi_allreduce calls, ndelayid(idvar) could be assigned a positive value to enable the expected mpp_delay_rcv calls at each time step:

src/OCE/LBC/lib_mpp.F90

@@ -401 +401 @@
       ! send y_in into todelay(idvar)%y1d with a non-blocking communication
 # if defined key_mpi2
       IF( ln_timing ) CALL tic_tac( .TRUE., ld_global = .TRUE.)
-      CALL  mpi_allreduce( y_in(:), todelay(idvar)%y1d(:), isz, MPI_DOUBLE_COMPLEX, mpi_sumdd, ilocalcomm, ndelayid(idvar), ierr )
+      CALL  mpi_allreduce( y_in(:), todelay(idvar)%y1d(:), isz, MPI_DOUBLE_COMPLEX, mpi_sumdd, ilocalcomm, ierr )
+      ndelayid(idvar) = 1
       IF( ln_timing ) CALL tic_tac(.FALSE., ld_global = .TRUE.)
 # else
       CALL mpi_iallreduce( y_in(:), todelay(idvar)%y1d(:), isz, MPI_DOUBLE_COMPLEX, mpi_sumdd, ilocalcomm, ndelayid(idvar), ierr )
@@ -468 +469 @@
       ! send p_in into todelay(idvar)%z1d with a non-blocking communication
 # if defined key_mpi2
       IF( ln_timing ) CALL tic_tac( .TRUE., ld_global = .TRUE.)
-      CALL  mpi_allreduce( p_in(:), todelay(idvar)%z1d(:), isz, MPI_DOUBLE_PRECISION, mpi_max, ilocalcomm, ndelayid(idvar), ierr )
+      CALL  mpi_allreduce( p_in(:), todelay(idvar)%z1d(:), isz, MPI_DOUBLE_PRECISION, mpi_max, ilocalcomm, ierr )
+      ndelayid(idvar) = 1
       IF( ln_timing ) CALL tic_tac(.FALSE., ld_global = .TRUE.)
 # else
       CALL mpi_iallreduce( p_in(:), todelay(idvar)%z1d(:), isz, MPI_DOUBLE_PRECISION, mpi_max, ilocalcomm, ndelayid(idvar), ierr )

[smueller]

In 12512:

Error: Failed to load processor CommitTicketReference

No macro or processor named 'CommitTicketReference' found

[smueller]

A variant of the ORCA2_ICE_PISCES restartability test with defined key_mpi2, nn_fwb = 1 in both LONG and SHORT, nn_stock = 496 in LONG, nn_stock = 248 in SHORT, and disabled PISCES component fails just before and succeeds after application of [12512].

Further, source:/NEMO/trunk@12512 passes SETTE.

[smueller]

In 12518:

Error: Failed to load processor CommitTicketReference

No macro or processor named 'CommitTicketReference' found