= HPC-08_Mueller_OSMOSIS_streamlining Last edition: '''[[Wikinfo(changed_ts)]]''' by '''[[Wikinfo(changed_by)]]''' The PI is responsible to closely follow the progress of the action, and especially to contact NEMO project manager if the delay on preview (or review) are longer than the 2 weeks expected. [[PageOutline(2, , inline)]] == Summary ||=Action || HPC-08_Mueller_OSMOSIS_streamlining || ||=PI(S) || S. Müller || ||=Digest || Optimisation and standardisation of the OSMOSIS OBL scheme implementation || ||=Dependencies || source:/NEMO/branches/NERC/dev_r11078_OSMOSIS_IMMERSE_Nurser_4.0 || ||=Branch || source:/NEMO/branches/2021/dev_r14122_HPC-08_Mueller_OSMOSIS_streamlining/ || ||=Previewer(s) || M. Bell || ||=Reviewer(s) || M. Bell || ||=Ticket || #2353 || === Description This action is the continuation of action wiki:2020WP/ENHANCE-14_smueller_OSMOSIS_streamlining, see wiki:2020WP/ENHANCE-14_smueller_OSMOSIS_streamlining#Description. === Implementation See wiki:2020WP/ENHANCE-14_smueller_OSMOSIS_streamlining#Implementation. === Documentation updates See wiki:2020WP/ENHANCE-14_smueller_OSMOSIS_streamlining#Documentationupdates. == Preview See wiki:2020WP/ENHANCE-14_smueller_OSMOSIS_streamlining#Preview. == Implementation and tests === Previous implementation (2020 work plan) See wiki:2020WP/ENHANCE-14_smueller_OSMOSIS_streamlining#Implementationandtests. === Continuation of implementation (2021 work plan) ||= Changeset =||= Description =||= Original changeset =||= WED025 =||= SETTE WED025 =|| |||||||| ''Recap of changesets listed on wiki:2020WP/ENHANCE-14_smueller_OSMOSIS_streamlining#Implementationandtests'' || || [14126,14136,14149,14265] || Baseline [14126] and temporary adjustments [14136,14149,14265] || || TKE/EVD (r14126), OSMOSIS (r14136,r14149,r14265) || || || [14260,14264,14144,14270] || Bugfixes [14260,14264] and default settings/parameters update [14144,14270] || || OSMOSIS || Reference (r14270) || || [14280,14305,14316] || Computational efficiency improvements [14280,14305,14316] || || OSMOSIS || Unchanged (r14280,r14305,r14316) || |||||||| ''Transition to wiki:2021WP/HPC-08_Mueller_OSMOSIS_streamlining'' || || [14529] || Transition to wiki:2021WP development branch || || OSMOSIS || **OK**, Unchanged || |||||||| ''Synchronisation with source:/NEMO/branches/NERC/dev_r11078_OSMOSIS_IMMERSE_Nurser_4.0@14541'' || || [14531] || Synchronisation with source:/NEMO/branches/NERC/dev_r11078_OSMOSIS_IMMERSE_Nurser_4.0@14541 || || OSMOSIS || **OK**, **DIFFERENCE** w.r.t. r14270 || || [14550] || Synchronisation with source:/NEMO/branches/NERC/dev_r11078_OSMOSIS_IMMERSE_Nurser_4.0@14541 || [14283] || OSMOSIS || **OK**, **DIFFERENCE** w.r.t. r14531 || || [14551] || Synchronisation with source:/NEMO/branches/NERC/dev_r11078_OSMOSIS_IMMERSE_Nurser_4.0@14541 || [14404] || OSMOSIS || **OK**, unchanged w.r.t. r14550 || || [14554] || Synchronisation with source:/NEMO/branches/NERC/dev_r11078_OSMOSIS_IMMERSE_Nurser_4.0@14541 || [14405,14407,14408,14412,14514] || OSMOSIS || **DIFFERENCE** w.r.t. r14550, **REPRO fails** || || [14555] || Synchronisation with source:/NEMO/branches/NERC/dev_r11078_OSMOSIS_IMMERSE_Nurser_4.0@14541 || [14441] || OSMOSIS || **OK**, **DIFFERENCE** w.r.t. r14554, REPRO OK || || [14556,14557] || Synchronisation with source:/NEMO/branches/NERC/dev_r11078_OSMOSIS_IMMERSE_Nurser_4.0@14541 || [14409,14517] || OSMOSIS || **DIFFERENCE** w.r.t. r14555, **REPRO fails** || || [14564] || Synchronisation with source:/NEMO/branches/NERC/dev_r11078_OSMOSIS_IMMERSE_Nurser_4.0@14541 || [14410,14411] || OSMOSIS || Unchanged w.r.t. r14557, **REPRO fails** || || [14565] || Synchronisation with source:/NEMO/branches/NERC/dev_r11078_OSMOSIS_IMMERSE_Nurser_4.0@14541 || [14515,14519,14536], completion of [14405] || OSMOSIS || **DIFFERENCE** w.r.t. r14564, **REPRO fails** || || [14566] || Synchronisation with source:/NEMO/branches/NERC/dev_r11078_OSMOSIS_IMMERSE_Nurser_4.0@14541 || [14516] || OSMOSIS || **DIFFERENCE** w.r.t. r14565, **REPRO fails** || || [14567] || Synchronisation with source:/NEMO/branches/NERC/dev_r11078_OSMOSIS_IMMERSE_Nurser_4.0@14541 || [14520,14541] || OSMOSIS || **DIFFERENCE** w.r.t. r14566, **REPRO fails** || || [14571] || Synchronisation with source:/NEMO/branches/NERC/dev_r11078_OSMOSIS_IMMERSE_Nurser_4.0@14541 || [14406,14518,14521,14534,14539,14540] || OSMOSIS || Unchanged w.r.t. r14567, **REPRO fails** || |||||||| ''Synchronisation with source:/NEMO/branches/NERC/dev_r11078_OSMOSIS_IMMERSE_Nurser_4.0@14705 and bugfixes'' || || [14728] || Bugfix || || OSMOSIS || **DIFFERENCE** w.r.t. r14567, **REPRO fails** || || [14729] || Synchronisation with source:/NEMO/branches/NERC/dev_r11078_OSMOSIS_IMMERSE_Nurser_4.0@14705 || [14645,14646] || OSMOSIS || Unchanged w.r.t. r14728, **REPRO fails** || || [14732] || Synchronisation with source:/NEMO/branches/NERC/dev_r11078_OSMOSIS_IMMERSE_Nurser_4.0@14705 || [14647] || OSMOSIS || Unchanged w.r.t. r14728, **REPRO fails** || || [14734] || Synchronisation with source:/NEMO/branches/NERC/dev_r11078_OSMOSIS_IMMERSE_Nurser_4.0@14705 || [14677,14678,14699,14704,14705] || OSMOSIS || **DIFFERENCE** w.r.t. r14728, **REPRO fails** || || [14743] || Bugfix || || OSMOSIS || **OK**, **DIFFERENCE** w.r.t. r14734 || || [14748] || Bugfix || || OSMOSIS || **OK**, unchanged w.r.t. r14743 || |||||||| ''Synchronisation with source:/NEMO/branches/NERC/dev_r11078_OSMOSIS_IMMERSE_Nurser_4.0@14759, streamlining, and upgrades'' || || [14750] || Streamlining (zdf_osm_velocity_rotation) || || OSMOSIS || **OK**, unchanged w.r.t. r14743 || || [14758] || Streamlining (module arrays) || || OSMOSIS || **OK**, unchanged w.r.t. r14743 || || [14760] || Synchronisation with source:/NEMO/branches/NERC/dev_r11078_OSMOSIS_IMMERSE_Nurser_4.0@14759 || [14755,14756,14759] || OSMOSIS || **OK**, unchanged w.r.t. r14743 || || [14775,14779,14783,14785,14798,14802,14803] || Upgrades of subroutines and arrays, improvement of coding-convention compliance || || OSMOSIS || **OK**, unchanged w.r.t. r14743 || || [14816] || Change of subroutine structure and removal of halo regions from arrays || || OSMOSIS || **OK**, unchanged w.r.t. r14743 || |||||||| ''Synchronisation with source:/NEMO/trunk'' in preparation for NEMO 2021 mid-year merge'' || || [14822] || Synchronisation with source:/NEMO/trunk@14820 || || OSMOSIS || **OK**, **DIFFERENCE** w.r.t. r14816 || || [14824,14828,14829] || Removal of optional debug output and of the extra timing instructions added with changesets [14149,14316]; deactivation of OSMOSIS scheme in WED025 reference configuration || || TKE/EVD || **OK**, **DIFFERENCE** w.r.t. r14822 || [[Image(NEMO_WP2021_HPC-08.png)]] source:/NEMO/branches/2021/dev_r14122_HPC-08_Mueller_OSMOSIS_streamlining@14829 has passed the standard SETTE tests. == Review {{{#!box width=50em info [[Include(wiki:Developers/DevProcess#review)]] }}} ''This change is self contained (to the zdfosm module) and passes SETTE tests. It is ready for merge. Mike Bell (and Andrew Coward)'' == Post-review updates (compatibility with `nn_hls=2` and `ln_tiling=.true.` options) Regression testing has been carried out using the standard `WED025` SETTE test with the modification {{{#!diff Index: cfgs/WED025/EXPREF/namelist_cfg =================================================================== --- cfgs/WED025/EXPREF/namelist_cfg (revision 14901) +++ cfgs/WED025/EXPREF/namelist_cfg (working copy) @@ -444,6 +444,7 @@ !----------------------------------------------------------------------- &namtra_mle ! mixed layer eddy parametrisation (Fox-Kemper) (default: OFF) !----------------------------------------------------------------------- + ln_mle = .true. ! (T) use the Mixed Layer Eddy (MLE) parameterisation / !----------------------------------------------------------------------- &namtra_eiv ! eddy induced velocity param. (default: OFF) @@ -532,13 +533,8 @@ !----------------------------------------------------------------------- ! ! ! type of vertical closure (required) - ln_zdftke = .true. ! Turbulent Kinetic Energy closure (T => fill namzdf_tke) + ln_zdfosm = .true. ! OSMOSIS BL closure (T => fill namzdf_osm) ! - ! ! convection - ln_zdfevd = .true. ! enhanced vertical diffusion - nn_evdm = 0 ! apply on tracer (=0) or on tracer and momentum (=1) - rn_evd = 10.0 ! mixing coefficient [m2/s] - ! ! ! coefficients rn_avm0 = 1.2e-4 ! vertical eddy viscosity [m2/s] (background Kz if ln_zdfcst=F) rn_avt0 = 2e-06 ! vertical eddy diffusivity [m2/s] (background Kz if ln_zdfcst=F) }}} to enable in the reference WED025 configuration both the OSMOSIS boundary-layer scheme (as included in the SETTE WED025 tests listed in the table above) and the OSMOSIS-specific extension of mixed-layer eddy-induced transport (not included in the SETTE WED025 tests listed in the table above), with respect to reference output generated using source:/NEMO/branches/2021/dev_r14122_HPC-08_Mueller_OSMOSIS_streamlining@14822 plus changeset [14858] (to enable compilability on the HPC system used for testing) with the addition of `ln_mle=.true.` in file `cfgs/WED025/EXPREF/namelist_cfg` (`14822+`). Time offsets in parentheses (where reported) indicate the runtime deviation of a test run from the corresponding reference '14822+' of 467.248 s. ||= Changeset =||= Description =||= Standard SETTE =||= **Extended halo** =||= **Tiling** =|| ||= =||= =||= SETTE WED025+zdfosm+tramle (`nn_hls=1`, `nn_tiling=.false.`) =||= SETTE WED025+zdfosm+tramle (`nn_hls=2`, `nn_tiling=.false.`) =||= SETTE WED025+zdfosm+tramle (`nn_hls=2`, `nn_tiling=.true.`) =|| || [14822] (plus [14858]) || Reference (`14822+`) || restart/repro **passed** (+0.000 s) || n/a || n/a || || [14856,14858,14859] || Synchronisation with source:/NEMO/trunk@14854, minor adjustments, and bugfixes related to diagnostic output || restart/repro **passed**, **unchanged** w.r.t. `14822+` || n/a || n/a || || [14863] || Synchronisation with source:/NEMO/trunk@14857 || restart/repro **passed**, **unchanged** w.r.t. `14822+` || n/a || n/a || || [14868] || Removal of redundant halo exchanges || restart/repro **passed**, **unchanged** w.r.t. `14822+` || n/a || n/a || || [14889] || Compatibility with extended halo (`nn_hls=2`) || restart/repro **passed**, **unchanged** w.r.t. `14822+` || restart/repro **passed**, **unchanged** w.r.t. `14822+` || n/a || || [14900,14901] || Compatibliity with tiling option (`ln_tiling=.true.`) || restart/repro **passed**, **unchanged** w.r.t. `14822+` || restart/repro **passed**, **unchanged** w.r.t. `14822+` || restart/repro **passed**, **unchanged** w.r.t. `14822+` || || [14909] || Code tidy-up || restart/repro **passed**, **unchanged** w.r.t. `14822+` (+0.518 s) || restart/repro **passed**, **unchanged** w.r.t. `14822+` (+21.563 s) || restart/repro **passed**, **unchanged** w.r.t. `14822+` (+30.659 s) || The modifications outside of module `zdfosm` to enable the extended halo and tiling options (modifications in source:/NEMO/branches/2021/dev_r14122_HPC-08_Mueller_OSMOSIS_streamlining/src/OCE/TRA/tramle.F90 and source:/NEMO/branches/2021/dev_r14122_HPC-08_Mueller_OSMOSIS_streamlining/src/OCE/ZDF/zdfphy.F90) have been successfully reviewed by D. Calvert. D. Calvert further suggested an efficiency improvement of subroutine `zdf_osm` with activated tiling (avoidance of sub-array copy-in and copy-out), which has been implemented with [14911]. ||= Changeset =||= Description =||= Standard SETTE =||= **Extended halo** =||= **Tiling** =|| || [14911] || Computational efficiency improvement || restart/repro **passed**, **unchanged** w.r.t. `14822+` (-6.411 s) || restart/repro **passed**, **unchanged** w.r.t. `14822+` (+20.121 s) || restart/repro **passed**, **unchanged** w.r.t. `14822+` (+22.505 s) || || [14912,14913] || Code tidy-up and synchronisation with source:/NEMO/trunk@14912 || restart/repro **passed**, **unchanged** w.r.t. `14822+` (-5.578 s) || restart/repro **passed**, **unchanged** w.r.t. `14822+` (+20.344 s) || restart/repro **passed**, **unchanged** w.r.t. `14822+` (-17.832 s) || || [14918] || Removal of problematic nested preprocessor macros || restart/repro **passed**, **unchanged** w.r.t. `14822+` (-0.484 s) || restart/repro **passed**, **unchanged** w.r.t. `14822+` (+27.718 s) || restart/repro **passed**, **unchanged** w.r.t. `14822+` (-17.227 s) || == SETTE testing SETTE (using source:/utils/CI/sette@14844) has been run to test the development branch source:/NEMO/branches/2021/dev_r14122_HPC-08_Mueller_OSMOSIS_streamlining@14918 (without explicit activation of the OSMOSIS boundary-layer scheme) with tiling and extra halo activated as {{{#!sh $ ./sette.sh -T -t -c }}} With the exception of the SETTE tests for reference configuration `ORCA2_OFF_PISCES`, source:/NEMO/branches/2021/dev_r14122_HPC-08_Mueller_OSMOSIS_streamlining@14918 passes all other SETTE tests, including regression tests with respect to corresponding runs carried out using source:/NEMO/trunk@14903. The `ORCA2_OFF_PISCES` test failures are XIOS-related and known to occur on the HPC system used; identical failures also occur when testing source:/NEMO/trunk@14903. As a workaround, the modification {{{#!diff Index: cfgs/ORCA2_OFF_PISCES/EXPREF/namelist_cfg =================================================================== --- cfgs/ORCA2_OFF_PISCES/EXPREF/namelist_cfg (revision 14918) +++ cfgs/ORCA2_OFF_PISCES/EXPREF/namelist_cfg (working copy) @@ -24,7 +24,7 @@ nn_itend = 1460 ! last time step (std 5475) nn_date0 = 19600101 ! date at nit_0000 (format yyyymmdd) used if ln_rstart=F or (ln_rstart=T and nn_rstctl=0 or 1) nn_stock = 1460 ! frequency of creation of a restart file (modulo referenced to 1) - ln_mskland = .true. ! mask land points in NetCDF outputs (costly: + ~15%) + ln_mskland = .false. ! mask land points in NetCDF outputs (costly: + ~15%) ln_cfmeta = .true. ! output additional data to netCDF files required for compliance with the CF metadata standard ln_clobber = .true. ! clobber (overwrite) an existing file / }}} permits to run the `ORCA2_OFF_PISCES` SETTE tests for both the development branch and source:/NEMO/trunk@14903 with matching results: the restartability tests are successful, the reproducibility tests fail with a difference after 28 time steps (similar behaviour has previously been observed on the HPC system used), and the regression test between the two model versions shows no difference. These test outcomes agree with corresponding testing without tiling and extra-halo activation (`./sette.sh -T`).