Opened 4 years ago
Closed 4 years ago
#2572 closed Defect (wontfix)
Issues with trunk when testing beyond SETTE (ORCA2_ICE_PISCES)
Reported by: | acc | Owned by: | acc |
---|---|---|---|
Priority: | low | Milestone: | 2020 WP |
Component: | MULTIPLE | Version: | |
Severity: | minor | Keywords: | |
Cc: | smasson |
Description (last modified by nemo)
Context
Two issues have arisen with the trunk whilst using the SETTE ORCA2_ICE_PISCES configuration for preliminary tests on a new cluster. These both arise when using domain decompositions beyond the standard 32-core, 8x4 or 4x8 used by SETTE.
- Decompositions with land suppression give different run.stat results. This may only be an issue with this new cluster but all successful, fully-populated partitions up to 105 cores give consistent results. Decompositions with land suppression differ. First example is 8x4 with 30 cores; run.stat values differ from the 32 core equivalent after 651 timesteps.
- Some larger decompositions fail to run and are stopped by checks on the validity of the specific humidity input in sbcblk.F90:
ztmp = SUM(tmask(:,:,1)) ! number of ocean points on local proc domain IF( ztmp > 8._wp ) THEN ! test only on proc domains with at least 8 ocean points! ztmp = SUM(sf(jp_humi)%fnow(:,:,1)*tmask(:,:,1))/ztmp ! mean humidity over ocean on proc SELECT CASE( nhumi ) CASE( np_humi_sph ) ! specific humidity => expect: 0. <= something < 0.065 [kg/kg] (0.061 is saturation at 45degC !!!) IF( (ztmp < 0._wp) .OR. (ztmp > 0.065) ) ztmp = -1._wp CASE( np_humi_dpt ) ! dew-point temperature => expect: 110. <= something < 320. [K] IF( (ztmp < 110._wp).OR.(ztmp > 320._wp) ) ztmp = -1._wp CASE( np_humi_rlh ) ! relative humidity => expect: 0. <= something < 100. [%] IF( (ztmp < 0._wp) .OR.(ztmp > 100._wp) ) ztmp = -1._wp END SELECT IF(ztmp < 0._wp) THEN IF (lwp) WRITE(numout,'(" Mean humidity value found on proc #",i6.6," is: ",f10.5)') narea, ztmp CALL ctl_stop( 'STOP', 'Something is wrong with air humidity!!!', & & ' ==> check the unit in your input files' , & & ' ==> check consistence of namelist choice: specific? relative? dew-point?', & & ' ==> ln_humi_sph -> [kg/kg] | ln_humi_rlh -> [%] | ln_humi_dpt -> [K] !!!' ) END IF END IF
This is genuine but arises because the CORE2 input data has small regions of negative specific humidity (mainly arctic). With a 19x14, 223 core decomposition, for example, there is one 36 point domain that has a negative ztmp. With the check suppressed, this decomposition will complete the LONG integration in 74 seconds. A 223 decomposition for ORCA2 may look excessive but the new cluster has 128 core nodes and the charging granularity is per node.
Recommendation
The land suppression tests will be repeated on a different platform to confirm any issue.
The validation of specific humidity should be given an optional override.
...
Commit History (0)
(No commits)
Change History (7)
comment:1 Changed 4 years ago by smasson
- Cc smasson added
comment:2 Changed 4 years ago by nemo
- Description modified (diff)
comment:3 Changed 4 years ago by smasson
I confirm that using X64_IRENE with intel 19, I have different results with jpni = 8, jpnj = 4 running on 30 cores.:
-bash-4.2$ cmp LONG/run.stat LONG30/run.stat
LONG/run.stat LONG30/run.stat differ: byte 95597, line 651
However, the problem disappears when I specifiy ln_icebergs = .False.
So, nothing really new in fact... ;-)
comment:4 Changed 4 years ago by acc
Ah yes, I forgot we knew about the icebergs already in relation to this. On the other point, how do folks feel about making the minimum number of ocean points for the specific humidity test a namelist variable? The 8 that is hard-wired seems arbitrary. At least then the test can be suppressed by setting a large number.
comment:5 Changed 4 years ago by gsamson
Concerning negative humidities in the Arctic, it should be better to set these unphysical values to zero instead.
It can be easily done with cdo for exemple:
cdo mul q_10.15JUNE2009_fill.nc -gec,0. q_10.15JUNE2009_fill.nc q_10.15JUNE2009_ok.nc
If we do so, inputs files also need to be updated on Zenodo.
comment:6 Changed 4 years ago by laurent
Hello,
Hmmm, I'm behind the introduction of the possibility to use these different forms of humidity (for example, ECMWF provides dew-point temperature at 2m), and I also wrote this sanity check on humidity.
I agree with Guillaume, the bug is clearly in the CORE forcing and NEMO is right in this case to complain, negative humidity cannot be considered as okay even with and insignificantly small magnitude. Since I'm NCO-dude, input CORE forcing netcdf files could be cured with a "ncap2", using something like this:
ncap2 -O -s 'where(Q_10 < 0.) Q_10=0.' q_10.15JUNE2009_fill.nc -o q_10.15JUNE2009_fill_ok.nc
I'm about to test SETTE on my branch which is synchronized with the trunk, I may discover the other issue you mention, but cool to know about the "ln_iceberg" workaround!
Cheers, /laurent
Cheers, /laurent
comment:7 Changed 4 years ago by acc
- Resolution set to wontfix
- Status changed from new to closed
Won't fix: The first issue is known and related to icebergs. This will be investigated in a 2021WP entry. The issue with negative humidities should be addressed at source.
I will do some land suppression tests too... I am afraid of what I will find...