Issues with trunk when testing beyond SETTE (ORCA2_ICE_PISCES)

[acc]

Context

Two issues have arisen with the trunk whilst using the SETTE ORCA2_ICE_PISCES configuration for preliminary tests on a new cluster. These both arise when using domain decompositions beyond the standard 32-core, 8x4 or 4x8 used by SETTE.

1. Decompositions with land suppression give different run.stat results. This may only be an issue with this new cluster but all successful, fully-populated partitions up to 105 cores give consistent results. Decompositions with land suppression differ. First example is 8x4 with 30 cores; run.stat values differ from the 32 core equivalent after 651 timesteps.

2. Some larger decompositions fail to run and are stopped by checks on the validity of the specific humidity input in sbcblk.F90:

         ztmp = SUM(tmask(:,:,1)) ! number of ocean points on local proc domain
         IF( ztmp > 8._wp ) THEN ! test only on proc domains with at least 8 ocean points!
            ztmp = SUM(sf(jp_humi)%fnow(:,:,1)*tmask(:,:,1))/ztmp ! mean humidity over ocean on proc
            SELECT CASE( nhumi )
            CASE( np_humi_sph ) ! specific humidity => expect: 0. <= something < 0.065 [kg/kg] (0.061 is saturation at 45degC !!!)
               IF(  (ztmp < 0._wp) .OR. (ztmp > 0.065)  ) ztmp = -1._wp
            CASE( np_humi_dpt ) ! dew-point temperature => expect: 110. <= something < 320. [K]
               IF( (ztmp < 110._wp).OR.(ztmp > 320._wp) ) ztmp = -1._wp
            CASE( np_humi_rlh ) ! relative humidity => expect: 0. <= something < 100. [%]
               IF(  (ztmp < 0._wp) .OR.(ztmp > 100._wp) ) ztmp = -1._wp
            END SELECT
            IF(ztmp < 0._wp) THEN
               IF (lwp) WRITE(numout,'("   Mean humidity value found on proc #",i6.6," is: ",f10.5)') narea, ztmp
               CALL ctl_stop( 'STOP', 'Something is wrong with air humidity!!!', &
                  &   ' ==> check the unit in your input files'       , &
                  &   ' ==> check consistence of namelist choice: specific? relative? dew-point?', &
                  &   ' ==> ln_humi_sph -> [kg/kg] | ln_humi_rlh -> [%] | ln_humi_dpt -> [K] !!!' )
            END IF
         END IF

This is genuine but arises because the CORE2 input data has small regions of negative specific humidity (mainly arctic). With a 19x14, 223 core decomposition, for example, there is one 36 point domain that has a negative ztmp. With the check suppressed, this decomposition will complete the LONG integration in 74 seconds. A 223 decomposition for ORCA2 may look excessive but the new cluster has 128 core nodes and the charging granularity is per node.

Recommendation

The land suppression tests will be repeated on a different platform to confirm any issue.

The validation of specific humidity should be given an optional override.

...