This wiki relates to the ticket on NEMO reproducibility fails with land domains exclusion (#1704)
Author: Tomas Lovato
Institution: CMCC
Proposed actions, after the confirmation from other ST members:
- revise the code and the log of mppini subroutines
- evaluate the modified tool for offline mpp decomposition
Additional details not included in the ticket
I tried to use configuration 4 (ORCA2_LIM_PISCES) to test the reproducibility for a run with a total of 88 PEs (80 ocean+8 land) and one with only the ocean domains (80 PEs). Here below a diff file of changes used in the test.
@@ -385,7 +386,7 @@ if [ ${config} -eq 4 ] ; then . ./all_functions.sh . ./prepare_exe_dir.sh JOB_FILE=${EXE_DIR}/run_job.sh - NPROC=16 + NPROC=88 if [ -f ${JOB_FILE} ] ; then \rm ${JOB_FILE} ; fi cd ${EXE_DIR} set_namelist namelist_cfg nn_it000 1 @@ -393,9 +394,9 @@ if [ ${config} -eq 4 ] ; then set_namelist namelist_cfg nn_fwb 0 set_namelist namelist_cfg ln_ctl .false. set_namelist namelist_cfg ln_clobber .true. - set_namelist namelist_cfg jpni 4 - set_namelist namelist_cfg jpnj 4 - set_namelist namelist_cfg jpnij 16 + set_namelist namelist_cfg jpni 8 + set_namelist namelist_cfg jpnj 11 + set_namelist namelist_cfg jpnij 88 set_namelist namelist_cfg nn_solv 2 set_namelist namelist_top_cfg ln_trcdta .false. set_namelist namelist_top_cfg ln_diatrc .false. @@ -425,16 +426,16 @@ if [ ${config} -eq 4 ] ; then export TEST_NAME="REPRO_2_8" . ./prepare_exe_dir.sh JOB_FILE=${EXE_DIR}/run_job.sh - NPROC=16 + NPROC=80 if [ -f ${JOB_FILE} ] ; then \rm ${JOB_FILE} ; fi cd ${EXE_DIR} set_namelist namelist_cfg nn_it000 1 set_namelist namelist_cfg nn_itend 75 set_namelist namelist_cfg ln_clobber .true. set_namelist namelist_cfg nn_fwb 0 - set_namelist namelist_cfg jpni 2 - set_namelist namelist_cfg jpnj 8 - set_namelist namelist_cfg jpnij 16 + set_namelist namelist_cfg jpni 8 + set_namelist namelist_cfg jpnj 11 + set_namelist namelist_cfg jpnij 80 set_namelist namelist_cfg nn_solv 2 set_namelist namelist_top_cfg ln_trcdta .false. set_namelist namelist_top_cfg ln_diatrc .false.
The test for reproducibility failed.
I made the following change in the mppini_2.h subroutine to modify the exclusion criteria, which now consider a domain as land when all points within the region (inner+overlap) are zero (I also modified the offline decomposition tool accordingly):
isurf = 0 - DO jj = 1+jprecj, ilj-jprecj - DO ji = 1+jpreci, ili-jpreci + DO jj = 1, ilj + DO ji = 1, ili IF( imask(ji+iimppt(ii,ij)-1, jj+ijmppt(ii,ij)-1) == 1) isurf = isurf+1 END DO END DO
After these modifications, I rerun SETTE using the configuration 4 test always using 88 PEs, but this time the decomposition was 81 ocean + 7 land. The reproducibility test passed.