ln_nnogather option not working for nn_hls=2

[acc]

Context

Default ORCA2 based SETTE tests are still failing SETTE tests with nn_hls=2. There appear to be two issues (at least): one is associated with tiling and the other with the ln_nnogather option. Setting ln_nnogather=.false. allows a full set of SETTE passes if the -t option is used with sette.sh. Even without tiling, ORCA2_ICE_PISCES reproducibility fails when ln_nnogather=.true.. This ticket addresses the ln_nnogather issues first.

Analysis

Adding the following code to stpmlf.F90 provides evidence of a change in results simply arising from the choice of ln_nnogather:

OCE/stpmlf.F90

@@ -93 +93 @@
       !!----------------------------------------------------------------------
       INTEGER ::   ji, jj, jk, jtile   ! dummy loop indice
       REAL(wp), ALLOCATABLE, DIMENSION(:,:,:) ::   zgdept
+      REAL(wp), DIMENSION(jpi,jpj) ::   try1, try2
+      INTEGER :: jtyp
+      CHARACTER(LEN=1), DIMENSION(4) :: ctyp = (/'T','U','V','W'/)
       !! ---------------------------------------------------------------------
 #if defined key_agrif
       IF( nstop > 0 ) RETURN   ! avoid to go further if an error was detected during previous time step (child grid)
@@ -128 +131 @@
          IF( lk_diamlr   )   CALL dia_mlr_iom_init    ! with additional setup for multiple-linear-regression analysis
                              CALL iom_init_closedef
          IF( ln_crs      )   CALL iom_init( TRIM(cxios_context)//"_crs" )  ! for coarse grid
+         DO jtyp=1,4
+         DO_2D(nn_hls, nn_hls, nn_hls, nn_hls )
+          try1(ji,jj) = mig(ji) + (mjg(jj)-1)*jpiglo
+          try2(ji,jj) = mig(ji) + (mjg(jj)-1)*jpiglo
+         END_2D
+         ln_nnogather=.false. ; CALL lbc_lnk('tryme', try1, ctyp(jtyp), 1._wp)
+         ln_nnogather=.true. ; CALL lbc_lnk('tryme', try2, ctyp(jtyp), 1._wp)
+         DO_2D(nn_hls, nn_hls, nn_hls, nn_hls )
+          IF(ABS(try1(ji,jj) - try2(ji,jj)) .gt. 0._wp ) THEN
+           write(*,*) 'NAREA:'//ctyp(jtyp)//' ',narea,ji,jj,try1(ji,jj),try2(ji,jj)
+          ENDIF
+         END_2D
+         END DO
       ENDIF
       IF( kstp == nitrst .AND. lwxios ) THEN
                              CALL iom_swap(                     cw_ocerst_cxt )

which, when run, produces this from the restartability test:

NAREA slurm-421072.out
 NAREA:T  29,  1,  17,  27421.,  27597.
 NAREA:T  29,  2,  17,  27420.,  27598.
 NAREA:U  29,  1,  17,  27420.,  27597.
 NAREA:U  29,  2,  17,  27419.,  27598.
 NAREA:V  29,  1,  17,  27237.,  27597.
 NAREA:V  29,  2,  17,  27236.,  27598.
 NAREA:V  29,  1,  18,  27053.,  27237.
 NAREA:V  29,  2,  18,  27052.,  27236.
 NAREA:V  29,  1,  19,  26869.,  27053.
 NAREA:V  29,  2,  19,  26868.,  27052.
 NAREA:W  29,  1,  17,  27421.,  27597.
 NAREA:W  29,  2,  17,  27420.,  27598.
 NAREA:U  30,  47,  17,  27508.,  27509.
 NAREA:U  30,  48,  17,  27508.,  27509.
 NAREA:U  31,  2,  17,  27508.,  27509.
 NAREA:U  31,  3,  17,  27508.,  27509.
 NAREA:T  32,  49,  17,  27420.,  27598.
 NAREA:U  32,  1,  17,  27465.,  27552.
 NAREA:W  32,  49,  17,  27420.,  27598.

Recommendation

None.
I do not recognise the current implementation of the ln_nnogather option. Hopefully someone in the team will have a better idea where to start digging.

Secondary recommendation would be to add this test, or something similar, to a permanent testing suite.

[acc]

Same test but with the values converted back into i-j indices; i.e.:

           jbi1=MOD(INT(try1(ji,jj)-1),jpiglo) + 1 ; jbj1=(try1(ji,jj)-jbi1)/jpiglo + 1
           jbi2=MOD(INT(try2(ji,jj)-1),jpiglo) + 1 : jbj2=(try2(ji,jj)-jbi2)/jpiglo + 1
           write(*,'(a,3i5,2i7," (",i5,",",i5,")"," (",i5,",",i5,")")') 'NAREA:'//ctyp(jtyp)//' ',narea,ji,jj,INT(try1(ji,jj)),INT(try2(ji,jj)),jbi1,jbj1,jbi2,jbj2

grep NAREA slurm-421639.out
NAREA:T    29    1   17  27421  27597 (    5,  150) (  181,  150)
NAREA:T    29    2   17  27420  27598 (    4,  150) (  182,  150)
NAREA:U    29    1   17  27420  27597 (    4,  150) (  181,  150)
NAREA:U    29    2   17  27419  27598 (    3,  150) (  182,  150)
NAREA:V    29    1   17  27237  27597 (    5,  149) (  181,  150)
NAREA:V    29    2   17  27236  27598 (    4,  149) (  182,  150)
NAREA:V    29    1   18  27053  27237 (    5,  148) (    5,  149)
NAREA:V    29    2   18  27052  27236 (    4,  148) (    4,  149)
NAREA:V    29    1   19  26869  27053 (    5,  147) (    5,  148)
NAREA:V    29    2   19  26868  27052 (    4,  147) (    4,  148)
NAREA:W    29    1   17  27421  27597 (    5,  150) (  181,  150)
NAREA:W    29    2   17  27420  27598 (    4,  150) (  182,  150)
NAREA:U    30   47   17  27508  27509 (   92,  150) (   93,  150)
NAREA:U    30   48   17  27508  27509 (   92,  150) (   93,  150)
NAREA:U    31    2   17  27508  27509 (   92,  150) (   93,  150)
NAREA:U    31    3   17  27508  27509 (   92,  150) (   93,  150)
NAREA:T    32   49   17  27420  27598 (    4,  150) (  182,  150)
NAREA:U    32    1   17  27465  27552 (   49,  150) (  136,  150)
NAREA:W    32   49   17  27420  27598 (    4,  150) (  182,  150)

Any clues there?