Options for improving namelist file access
This page outlines some ideas for reducing the sequential disk access to namelist files at start-up. The current method, where every process repeatedly reads and rewinds a few small text files, is only as scalable as the file-system allows and is unlikely to be fit for purpose when moving to higher core counts.
A simple option, that requires very little code change, is to replace the current unit number variables with character arrays that contain the whole of the corresponding namelist file. I.e. use F90 internal files so that disk access is performed only once in a single sequential read by each process.
If necessary, disk access could be reduced further by having only one process read each namelist and broadcast the array to all others. I believe this is now feasible since Seb's recent updates remove any namelist access before MPI initialisation. This should be considered at a future stage.
Stage 1
- Replace namelist unit variables with allocatable character arrays, e.g. in in_out_manger.F90:
INTEGER :: numnam_ref = -1 !: logical unit for reference namelist INTEGER :: numnam_cfg = -1 !: logical unit for configuration specific namelist
become:
CHARACTER(LEN=:), ALLOCATABLE :: numnam_ref !: character buffer for reference namelist CHARACTER(LEN=:), ALLOCATABLE :: numnam_cfg !: character buffer for configuration namelist
- Replace ctl_opn statements with calls to a new routine that will fill the buffer, e.g.: in nemogcm.F90:
CALL ctl_opn( numnam_ref, 'namelist_ref', 'OLD', 'FORMATTED', 'SEQUENTIAL', -1, -1, .FALSE. )
CALL ctl_opn( numnam_cfg, 'namelist_cfg', 'OLD', 'FORMATTED', 'SEQUENTIAL', -1, -1, .FALSE. )
become:
CALL load_nml( numnam_ref, 'namelist_ref', -1, .FALSE. )
CALL load_nml( numnam_cfg, 'namelist_cfg', -1, .FALSE. )
- Remove all REWIND statements involving the old unit variables (REWIND is not permitted (or necessary) on internal files)
- Remove any CLOSE statements involving the old unit numbers. the load_nml routine will have closed the namelist file after reading.
Job done.
There are however a few caveats:
- To be memory efficient, the character arrays should be allocated with the minimum length required to accommodate the namelist information. This requires each file to be read twice: once to determine and sum up the trimmed length of each line (comments can be stripped at this stage); and once to fill the newly allocated buffer. The example load_nml routine shown below achieves this. As an indication of requirements, this routine compacts the namelist_ref file into about 33KB. Further savings could be made by reducing multiple spaces to single spaces but the need to preserve any character strings within the namelists would complicate the code and would only save around 20%.
- Reads from internal files always start at the beginning of the array (although substrings of the array can be provided to alter the starting point). This means the current practise in BDY of having multiple, identically named namelist blocks depending on the number of boundary segments will not work with internal files. Scanning the array to determine starting points for each block is possible but messy because of the need to support FORTRAN's case-independent attitude. A simpler solution will be to insist on appending a count to second and subsequent occurrences of the same block.
- To be considered: will this work with AGRIF? load_nml still uses ctl_opn to open the namelist file so the correct file will be opened but will each nest get independent copies of the character arrays? What does Agrif_Get_Unit currently do?
The prototype load_nml routine (added to lib_mpp.F90)
SUBROUTINE load_nml( cdnambuff , cdnamfile, kout, ldwp)
CHARACTER(LEN=:), ALLOCATABLE, INTENT(INOUT) :: cdnambuff
CHARACTER(LEN=*), INTENT(IN ) :: cdnamfile
CHARACTER(LEN=256) :: chline
INTEGER, INTENT(IN) :: kout
LOGICAL, INTENT(IN) :: ldwp
INTEGER :: itot, iun, iltc, inl, ios
!
! Check if the namelist buffer has already been allocated. Return if it has.
!
IF ( ALLOCATED( cdnambuff ) ) RETURN
!
! Open namelist file
!
CALL ctl_opn( iun, cdnamfile, 'OLD', 'FORMATTED', 'SEQUENTIAL', -1, kout, ldwp )
!
! First pass: count characters excluding comments and trimable white space
!
itot=0
10 READ(iun,'(A256)',END=20,ERR=20) chline
iltc = LEN_TRIM(chline)
IF ( iltc.GT.0 ) THEN
inl = INDEX(chline, '!')
IF( inl.eq.0 ) THEN
itot = itot + iltc + 1 ! +1 for the newline character
ELSEIF( inl.GT.0 .AND. LEN_TRIM( chline(1:inl-1) ).GT.0 ) THEN
itot = itot + inl ! includes +1 for the newline character
ENDIF
ENDIF
GOTO 10
20 CONTINUE
!
! Allocate text cdnambuff for condensed namelist
!
ALLOCATE( CHARACTER(LEN=itot) :: cdnambuff )
WRITE(*,*) 'ALLOCATED ', itot
!
! Second pass: read and transfer pruned characters into cdnambuff
!
REWIND(iun)
itot=1
30 READ(iun,'(A256)',END=40,ERR=40) chline
iltc = LEN_TRIM(chline)
IF ( iltc.GT.0 ) THEN
inl = INDEX(chline, '!')
IF( inl.eq.0 ) THEN
inl = iltc
ELSE
inl = inl - 1
ENDIF
IF( inl.GT.0 .AND. LEN_TRIM( chline(1:inl) ).GT.0 ) THEN
cdnambuff(itot:itot+inl-1) = chline(1:inl)
WRITE( cdnambuff(itot+inl:itot+inl), '(a)' ) NEW_LINE('A')
itot = itot + inl + 1
ENDIF
ENDIF
GOTO 30
40 CONTINUE
WRITE(*,*) 'ASSIGNED ',itot - 1
!
! Close namelist file
!
CLOSE(iun)
!write(*,'(32A)') cdnambuff
END SUBROUTINE load_nml
Full development branch
This approach has been implemented and tested on a 2019 development branch:
This branch contains only the substantive changes; to compile and run tests all REWIND and CLOSE operations on the (no longer) units have to be removed. These changes affect many more files but can be scripted so are not included the branch in order to make a later merge easier. The scripts used to prepare code for testing are included below. With these additional changes this code passes most SETTE tests but the AGRIF preprocessor does not currently accept the new allocatable character strings. The conversion routine first bails on:
fcm_internal compile:F nemo /home/acc/NEMO/IMMERSE/dev_r11613_ENHANCE-04_namelists_as_internalfiles/tests/VORTEX_ST/NEMOFILES/ppsrc/nemo/in_out_manager.f90 in_out_manager.f90 /home/acc/NEMO/IMMERSE/dev_r11613_ENHANCE-04_namelists_as_internalfiles/mk/agrifpp.sh in_out_manager.f90 /home/acc/NEMO/IMMERSE/dev_r11613_ENHANCE-04_namelists_as_internalfiles/tests/VORTEX_ST/NEMOFILES/inc /home/acc/NEMO/IMMERSE/dev_r11613_ENHANCE-04_namelists_as_internalfiles/tests/VORTEX_ST/NEMOFILES/ppsrc/nemo/in_out_manager.f90 syntax error line 135, file in_out_manager.f90 motclef = |:| fcm_internal compile failed (256) which is the line: CHARACTER(LEN=:), ALLOCATABLE :: numnam_ref !: character buffer for reference namelist
AGRIF will need to be made to understand the (LEN=: ) syntax before this solution can be progressed.
All non-AGRIF SETTE tests are passing but note none of the current tests use multiple BDY namelists; a solution for that issue has yet to be coded.
Lists and scripts To minimise changes that need to be merged later, only the substantive changes have been checked into the development branch. These changes include the new routine in lib_mpp.F90, buffer declarations and replacement of any ctl_opn calls for namelist input. The list of files in this category is held in src/substantive.list and is:
ICE/icestp.F90 OCE/IOM/in_out_manager.F90 OCE/LBC/lib_mpp.F90 OCE/nemogcm.F90 OFF/nemogcm.F90 SAO/nemogcm.F90 SAS/nemogcm.F90 TOP/PISCES/SED/sedini.F90 TOP/PISCES/sms_pisces.F90 TOP/PISCES/trcnam_pisces.F90 TOP/trc.F90 TOP/trcnam.F90
A list of files potentially containing REWIND and CLOSE statements (but excluding files in the substantive list) can be built by running the following script in the src directory:
#!/bin/bash
#
INUNITS=( numnam_ref numnam_cfg numnat_ref numnat_cfg numtrc_ref numtrc_cfg numnam_ice_ref numnam_ice_cfg numnamsed_ref numnamsed_cfg numnatp_cfg numnatp_ref )
RM_CMDS=( REWIND CLOSE )
#
# First build a list of files likely to need alteration
#
if [ -f all_rewfiles.list ] ; then rm all_rewfiles.list; fi
for n in `seq 0 1 $(( ${#INUNITS[*]} - 1 ))`
do
grep -l -i ${INUNITS[$n]} `find ./ -name '*.[Ffh]90'` >> all_rewfiles.list
grep -l -i ${INUNITS[$n]} `find ../tests -name '*.[Ffh]90'` | grep -v WORK | grep -v BLD | grep -v NEMOFILES >> all_rewfiles.list
grep -l -i ${INUNITS[$n]} `find ../cfgs -name '*.[Ffh]90'` | grep -v WORK | grep -v BLD | grep -v NEMOFILES >> all_rewfiles.list
done
sort -u all_rewfiles.list > alluniq_rewfiles.list
for f in `cat substantive.list`
do
ff=`echo $f | sed -e 's:/:\\\/:g'`
echo $ff
ed - alluniq_rewfiles.list << EOF
/$ff/d
w
q
EOF
done
and this list can be used to target an editing script which removes any REWIND or CLOSE statements on converted units (also to be run in src):
#!/bin/bash
#
INUNITS=( numnam_ref numnam_cfg numnat_ref numnat_cfg numtrc_ref numtrc_cfg numnam_ice_ref numnam_ice_cfg numnamsed_ref numnamsed_cfg numnatp_cfg numnatp_ref )
RM_CMDS=( REWIND CLOSE )
#
for f in `cat alluniq_rewfiles.list`
do
n=0
for n in `seq 0 1 $(( ${#INUNITS[*]} - 1 ))`
do
for m in `seq 0 1 $(( ${#RM_CMDS[*]} - 1 ))`
do
perl -ni.bak -e 'print unless m@.*\s*'${RM_CMDS[$m]}'\s*\(\s*'${INUNITS[$n]}'\s*\).*@i' $f
done
done
done
cd ../
rm `find ./ -name '*.bak'`
.
