New URL for NEMO forge!   http://forge.nemo-ocean.eu

Since March 2022 along with NEMO 4.2 release, the code development moved to a self-hosted GitLab.
This present forge is now archived and remained online for history.
2021WP/HPC-08_Mueller_OSMOSIS_streamlining – NEMO
wiki:2021WP/HPC-08_Mueller_OSMOSIS_streamlining

HPC-08_Mueller_OSMOSIS_streamlining

Last edition: Wikinfo(changed_ts)? by Wikinfo(changed_by)?

The PI is responsible to closely follow the progress of the action, and especially to contact NEMO project manager if the delay on preview (or review) are longer than the 2 weeks expected.

  1. Summary
  2. Preview
  3. Implementation and tests
  4. Review
  5. Post-review updates (compatibility with nn_hls=2 and ln_tiling=.true. options)
  6. SETTE testing

Summary

Action HPC-08_Mueller_OSMOSIS_streamlining
PI(S) S. Müller
Digest Optimisation and standardisation of the OSMOSIS OBL scheme implementation
Dependencies source:/NEMO/branches/NERC/dev_r11078_OSMOSIS_IMMERSE_Nurser_4.0
Branch source:/NEMO/branches/2021/dev_r14122_HPC-08_Mueller_OSMOSIS_streamlining/
Previewer(s) M. Bell
Reviewer(s) M. Bell
Ticket #2353

Description

This action is the continuation of action wiki:2020WP/ENHANCE-14_smueller_OSMOSIS_streamlining, see wiki:2020WP/ENHANCE-14_smueller_OSMOSIS_streamlining#Description.

Implementation

See wiki:2020WP/ENHANCE-14_smueller_OSMOSIS_streamlining#Implementation.

Documentation updates

See wiki:2020WP/ENHANCE-14_smueller_OSMOSIS_streamlining#Documentationupdates.

Preview

See wiki:2020WP/ENHANCE-14_smueller_OSMOSIS_streamlining#Preview.

Implementation and tests

Previous implementation (2020 work plan)

See wiki:2020WP/ENHANCE-14_smueller_OSMOSIS_streamlining#Implementationandtests.

Continuation of implementation (2021 work plan)

Changeset Description Original changeset WED025 SETTE WED025
Recap of changesets listed on wiki:2020WP/ENHANCE-14_smueller_OSMOSIS_streamlining#Implementationandtests
[14126,14136,14149,14265] Baseline [14126] and temporary adjustments [14136,14149,14265] TKE/EVD (r14126), OSMOSIS (r14136,r14149,r14265)
[14260,14264,14144,14270] Bugfixes [14260,14264] and default settings/parameters update [14144,14270] OSMOSIS Reference (r14270)
[14280,14305,14316] Computational efficiency improvements [14280,14305,14316] OSMOSIS Unchanged (r14280,r14305,r14316)
Transition to wiki:2021WP/HPC-08_Mueller_OSMOSIS_streamlining
[14529] Transition to wiki:2021WP development branch OSMOSIS OK, Unchanged
Synchronisation with source:/NEMO/branches/NERC/dev_r11078_OSMOSIS_IMMERSE_Nurser_4.0@14541
[14531] Synchronisation with source:/NEMO/branches/NERC/dev_r11078_OSMOSIS_IMMERSE_Nurser_4.0@14541 OSMOSIS OK, DIFFERENCE w.r.t. r14270
[14550] Synchronisation with source:/NEMO/branches/NERC/dev_r11078_OSMOSIS_IMMERSE_Nurser_4.0@14541 [14283] OSMOSIS OK, DIFFERENCE w.r.t. r14531
[14551] Synchronisation with source:/NEMO/branches/NERC/dev_r11078_OSMOSIS_IMMERSE_Nurser_4.0@14541 [14404] OSMOSIS OK, unchanged w.r.t. r14550
[14554] Synchronisation with source:/NEMO/branches/NERC/dev_r11078_OSMOSIS_IMMERSE_Nurser_4.0@14541 [14405,14407,14408,14412,14514] OSMOSIS DIFFERENCE w.r.t. r14550, REPRO fails
[14555] Synchronisation with source:/NEMO/branches/NERC/dev_r11078_OSMOSIS_IMMERSE_Nurser_4.0@14541 [14441] OSMOSIS OK, DIFFERENCE w.r.t. r14554, REPRO OK
[14556,14557] Synchronisation with source:/NEMO/branches/NERC/dev_r11078_OSMOSIS_IMMERSE_Nurser_4.0@14541 [14409,14517] OSMOSIS DIFFERENCE w.r.t. r14555, REPRO fails
[14564] Synchronisation with source:/NEMO/branches/NERC/dev_r11078_OSMOSIS_IMMERSE_Nurser_4.0@14541 [14410,14411] OSMOSIS Unchanged w.r.t. r14557, REPRO fails
[14565] Synchronisation with source:/NEMO/branches/NERC/dev_r11078_OSMOSIS_IMMERSE_Nurser_4.0@14541 [14515,14519,14536], completion of [14405] OSMOSIS DIFFERENCE w.r.t. r14564, REPRO fails
[14566] Synchronisation with source:/NEMO/branches/NERC/dev_r11078_OSMOSIS_IMMERSE_Nurser_4.0@14541 [14516] OSMOSIS DIFFERENCE w.r.t. r14565, REPRO fails
[14567] Synchronisation with source:/NEMO/branches/NERC/dev_r11078_OSMOSIS_IMMERSE_Nurser_4.0@14541 [14520,14541] OSMOSIS DIFFERENCE w.r.t. r14566, REPRO fails
[14571] Synchronisation with source:/NEMO/branches/NERC/dev_r11078_OSMOSIS_IMMERSE_Nurser_4.0@14541 [14406,14518,14521,14534,14539,14540] OSMOSIS Unchanged w.r.t. r14567, REPRO fails
Synchronisation with source:/NEMO/branches/NERC/dev_r11078_OSMOSIS_IMMERSE_Nurser_4.0@14705 and bugfixes
[14728] Bugfix OSMOSIS DIFFERENCE w.r.t. r14567, REPRO fails
[14729] Synchronisation with source:/NEMO/branches/NERC/dev_r11078_OSMOSIS_IMMERSE_Nurser_4.0@14705 [14645,14646] OSMOSIS Unchanged w.r.t. r14728, REPRO fails
[14732] Synchronisation with source:/NEMO/branches/NERC/dev_r11078_OSMOSIS_IMMERSE_Nurser_4.0@14705 [14647] OSMOSIS Unchanged w.r.t. r14728, REPRO fails
[14734] Synchronisation with source:/NEMO/branches/NERC/dev_r11078_OSMOSIS_IMMERSE_Nurser_4.0@14705 [14677,14678,14699,14704,14705] OSMOSIS DIFFERENCE w.r.t. r14728, REPRO fails
[14743] Bugfix OSMOSIS OK, DIFFERENCE w.r.t. r14734
[14748] Bugfix OSMOSIS OK, unchanged w.r.t. r14743
Synchronisation with source:/NEMO/branches/NERC/dev_r11078_OSMOSIS_IMMERSE_Nurser_4.0@14759, streamlining, and upgrades
[14750] Streamlining (zdf_osm_velocity_rotation) OSMOSIS OK, unchanged w.r.t. r14743
[14758] Streamlining (module arrays) OSMOSIS OK, unchanged w.r.t. r14743
[14760] Synchronisation with source:/NEMO/branches/NERC/dev_r11078_OSMOSIS_IMMERSE_Nurser_4.0@14759 [14755,14756,14759] OSMOSIS OK, unchanged w.r.t. r14743
[14775,14779,14783,14785,14798,14802,14803] Upgrades of subroutines and arrays, improvement of coding-convention compliance OSMOSIS OK, unchanged w.r.t. r14743
[14816] Change of subroutine structure and removal of halo regions from arrays OSMOSIS OK, unchanged w.r.t. r14743
Synchronisation with source:/NEMO/trunk in preparation for NEMO 2021 mid-year merge
[14822] Synchronisation with source:/NEMO/trunk@14820 OSMOSIS OK, DIFFERENCE w.r.t. r14816
[14824,14828,14829] Removal of optional debug output and of the extra timing instructions added with changesets [14149,14316]; deactivation of OSMOSIS scheme in WED025 reference configuration TKE/EVD OK, DIFFERENCE w.r.t. r14822

source:/NEMO/branches/2021/dev_r14122_HPC-08_Mueller_OSMOSIS_streamlining@14829 has passed the standard SETTE tests.

Review

Error: Failed to load processor box
No macro or processor named 'box' found

This change is self contained (to the zdfosm module) and passes SETTE tests. It is ready for merge. Mike Bell (and Andrew Coward)

Post-review updates (compatibility with nn_hls=2 and ln_tiling=.true. options)

Regression testing has been carried out using the standard WED025 SETTE test with the modification

  • cfgs/WED025/EXPREF/namelist_cfg

     
    444444!----------------------------------------------------------------------- 
    445445&namtra_mle    !   mixed layer eddy parametrisation (Fox-Kemper)       (default: OFF) 
    446446!----------------------------------------------------------------------- 
     447   ln_mle      = .true.   ! (T) use the Mixed Layer Eddy (MLE) parameterisation 
    447448/ 
    448449!----------------------------------------------------------------------- 
    449450&namtra_eiv    !   eddy induced velocity param.                         (default: OFF) 
    … …  
    532533!----------------------------------------------------------------------- 
    533534   ! 
    534535   !                       ! type of vertical closure (required) 
    535    ln_zdftke   = .true.       !  Turbulent Kinetic Energy closure       (T =>   fill namzdf_tke) 
     536   ln_zdfosm   = .true.      !  OSMOSIS BL closure                     (T =>   fill namzdf_osm) 
    536537   ! 
    537    !                       ! convection 
    538    ln_zdfevd   = .true.       !  enhanced vertical diffusion 
    539       nn_evdm     =    0         ! apply on tracer (=0) or on tracer and momentum (=1) 
    540       rn_evd      =  10.0        ! mixing coefficient [m2/s] 
    541    ! 
    542538   !                       ! coefficients 
    543539   rn_avm0     =   1.2e-4     !  vertical eddy viscosity   [m2/s]       (background Kz if ln_zdfcst=F) 
    544540   rn_avt0     =   2e-06      !  vertical eddy diffusivity [m2/s]       (background Kz if ln_zdfcst=F) 

to enable in the reference WED025 configuration both the OSMOSIS boundary-layer scheme (as included in the SETTE WED025 tests listed in the table above) and the OSMOSIS-specific extension of mixed-layer eddy-induced transport (not included in the SETTE WED025 tests listed in the table above), with respect to reference output generated using source:/NEMO/branches/2021/dev_r14122_HPC-08_Mueller_OSMOSIS_streamlining@14822 plus changeset [14858] (to enable compilability on the HPC system used for testing) with the addition of ln_mle=.true. in file cfgs/WED025/EXPREF/namelist_cfg (14822+). Time offsets in parentheses (where reported) indicate the runtime deviation of a test run from the corresponding reference '14822+' of 467.248 s.

Changeset Description Standard SETTE Extended halo Tiling
SETTE WED025+zdfosm+tramle (nn_hls=1, nn_tiling=.false.) SETTE WED025+zdfosm+tramle (nn_hls=2, nn_tiling=.false.) SETTE WED025+zdfosm+tramle (nn_hls=2, nn_tiling=.true.)
[14822] (plus [14858]) Reference (14822+) restart/repro passed (+0.000 s) n/a n/a
[14856,14858,14859] Synchronisation with source:/NEMO/trunk@14854, minor adjustments, and bugfixes related to diagnostic output restart/repro passed, unchanged w.r.t. 14822+ n/a n/a
[14863] Synchronisation with source:/NEMO/trunk@14857 restart/repro passed, unchanged w.r.t. 14822+ n/a n/a
[14868] Removal of redundant halo exchanges restart/repro passed, unchanged w.r.t. 14822+ n/a n/a
[14889] Compatibility with extended halo (nn_hls=2) restart/repro passed, unchanged w.r.t. 14822+ restart/repro passed, unchanged w.r.t. 14822+ n/a
[14900,14901] Compatibliity with tiling option (ln_tiling=.true.) restart/repro passed, unchanged w.r.t. 14822+ restart/repro passed, unchanged w.r.t. 14822+ restart/repro passed, unchanged w.r.t. 14822+
[14909] Code tidy-up restart/repro passed, unchanged w.r.t. 14822+ (+0.518 s) restart/repro passed, unchanged w.r.t. 14822+ (+21.563 s) restart/repro passed, unchanged w.r.t. 14822+ (+30.659 s)

The modifications outside of module zdfosm to enable the extended halo and tiling options (modifications in source:/NEMO/branches/2021/dev_r14122_HPC-08_Mueller_OSMOSIS_streamlining/src/OCE/TRA/tramle.F90 and source:/NEMO/branches/2021/dev_r14122_HPC-08_Mueller_OSMOSIS_streamlining/src/OCE/ZDF/zdfphy.F90) have been successfully reviewed by D. Calvert. D. Calvert further suggested an efficiency improvement of subroutine zdf_osm with activated tiling (avoidance of sub-array copy-in and copy-out), which has been implemented with [14911].

Changeset Description Standard SETTE Extended halo Tiling
[14911] Computational efficiency improvement restart/repro passed, unchanged w.r.t. 14822+ (-6.411 s) restart/repro passed, unchanged w.r.t. 14822+ (+20.121 s) restart/repro passed, unchanged w.r.t. 14822+ (+22.505 s)
[14912,14913] Code tidy-up and synchronisation with source:/NEMO/trunk@14912 restart/repro passed, unchanged w.r.t. 14822+ (-5.578 s) restart/repro passed, unchanged w.r.t. 14822+ (+20.344 s) restart/repro passed, unchanged w.r.t. 14822+ (-17.832 s)
[14918] Removal of problematic nested preprocessor macros restart/repro passed, unchanged w.r.t. 14822+ (-0.484 s) restart/repro passed, unchanged w.r.t. 14822+ (+27.718 s) restart/repro passed, unchanged w.r.t. 14822+ (-17.227 s)

SETTE testing

SETTE (using source:/utils/CI/sette@14844) has been run to test the development branch source:/NEMO/branches/2021/dev_r14122_HPC-08_Mueller_OSMOSIS_streamlining@14918 (without explicit activation of the OSMOSIS boundary-layer scheme) with tiling and extra halo activated as 

$ ./sette.sh -T -t -c

With the exception of the SETTE tests for reference configuration ORCA2_OFF_PISCES, source:/NEMO/branches/2021/dev_r14122_HPC-08_Mueller_OSMOSIS_streamlining@14918 passes all other SETTE tests, including regression tests with respect to corresponding runs carried out using source:/NEMO/trunk@14903. The ORCA2_OFF_PISCES test failures are XIOS-related and known to occur on the HPC system used; identical failures also occur when testing source:/NEMO/trunk@14903. As a workaround, the modification

  • cfgs/ORCA2_OFF_PISCES/EXPREF/namelist_cfg

     
    2424   nn_itend    =    1460     !  last  time step (std 5475) 
    2525   nn_date0    =  19600101   !  date at nit_0000 (format yyyymmdd) used if ln_rstart=F or (ln_rstart=T and nn_rstctl=0 or 1) 
    2626   nn_stock    =    1460     !  frequency of creation of a restart file (modulo referenced to 1) 
    27    ln_mskland  = .true.      !  mask land points in NetCDF outputs (costly: + ~15%) 
     27   ln_mskland  = .false.      !  mask land points in NetCDF outputs (costly: + ~15%) 
    2828   ln_cfmeta   = .true.      !  output additional data to netCDF files required for compliance with the CF metadata standard 
    2929   ln_clobber  = .true.      !  clobber (overwrite) an existing file 
    3030/ 

permits to run the ORCA2_OFF_PISCES SETTE tests for both the development branch and source:/NEMO/trunk@14903 with matching results: the restartability tests are successful, the reproducibility tests fail with a difference after 28 time steps (similar behaviour has previously been observed on the HPC system used), and the regression test between the two model versions shows no difference. These test outcomes agree with corresponding testing without tiling and extra-halo activation (./sette.sh -T).

Last modified 3 years ago Last modified on 2021-05-28T12:51:02+02:00

Attachments (1)

Download all attachments as: .zip