Porting on RedHat8

After maintenance of April 11 and 12 all the Irene machines of the TGCC will change OS (it will go from redhat7 to redhat8). As an addition of the change in OS old modules will not be available on redhat8 and we need to update compiler versions for our modules.

How to install your environment on redhat8 (skl and rome)

Use the igcmg environment (in bash) with a copy of the IPSL bashrc script shell files in your HOME (and save your old ones to be used if necessary on irene-prev).

ryyy999@irene: cp ~igcmg/MachineEnvironment/irene/bashrc  ~/.bashrc
ryyy999@irene: mv ~/.bashrc_irene ~/.bashrc_irene_prev
ryyy999@irene: mv ~/.bashrc_irene-amd ~/.bashrc_irene-amd_prev
ryyy999@irene: cp ~igcmg/MachineEnvironment/irene/bashrc_irene ~/.bashrc_irene

Note that the same file .bashrc_irene is now used both for skylake and amd(rome). This was not the case before. Additionally, you can complete the example of bashrc_irene file to create your favorite environment (alias, module load ...) by copying what you did in your previous file (now named ~/.bashrc_irene_prev or ~/.bashrc_irene-amd_prev).

We strongly advice you to add the line module switch dfldatadir dfldatadir/genXXXX in your own .bashrc_irene_next.

WARNING : if you have a ~/.profile file, it's better to remove it to avoid any problem during the execution of a simulation with libIGCM

In this environment is specified:

• the path to the compiler tool fcm and to the rebuild tool which recombines output files from a parallel model:

  export PATH=$(ccc_home -u igcmg)/Tools/fcm/bin:$(ccc_home -u igcmg)/Tools/irene/bin:$PATH
  

• the load of modules giving access to analyze data or post processing libraries and tools needed on our platform (done in ccc_home -u igcmg/MachineEnvironment/irene_next/env_atlas_irene).

WARNING : You cannot load in the same time the intel environment for compilation and models computation, AND ferret software. They are not compatible. That's why by default we propose an environment for post processing. The computation environment will be loaded by modipsl and libIGCM during compilation and running.

Modifications need to be done in models

With the new OS redhat8 you will use a new version of svn that will be incompatible with directories extracted on redhat7 - To correct this you need to run the command "svn upgrade" in your directories

LMDZ

Replace %BASE_LD in LMDZ/arch/arch-X64_IRENE.fcm by %BASE_LD -i4 -r8 -auto -L/ccc/products/mkl-20.0.0/system/default/20.0.0/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_core -lmkl_sequential -lpthread

ORCHIDEE

The new version of modules we now use on redhat8 can not handle to many files opened at the same time. When running LMDZOR on hybrid mode (mixed MPI-OpenMP), too many files text output files were opened in ORCHIDEE (out_orchidee_000x.000y). This has been changed in newer versions of ORCHIDEE: in revision 7792 for branch ORHIDEE_2_2 and 7790 in the trunk.

If you work with another version, do the following work-around in the code :

In modeles/ORCHIDEE/src_parallel/mod_orchidee_para.F90, comment the opening of the text output files and change to have numout=6, the output will now be sent to standard output for all cores.

!!!    OPEN(UNIT=numout,FILE=TRIM(fileout),ACTION='write',STATUS='unknown',FORM='formatted',IOSTAT=ierr)
!!!    IF (ierr /= 0) THEN
!!!    #ifdef CPP_PARA
!!!        CALL MPI_FINALIZE(ierr)
!!!    #endif
!!!        WRITE(*,*) "In Set_stdout_file : Erreur can't open file ", filename
!!!        STOP 1
!!!    ENDIF

numout=6

INCA

If you are working with a version older than rev 1050, you need to modify lunout parameter to use standard output file. For this in INCA/src/INCA_MOD/print_inca.F90 comment the opening of the text output files and change to have lunout=6

    !    open(UNIT=lunout,file=fileout, action='write',status='unknown', form='formatted', iostat=ierr2)
    !  IF (ierr2 /= 0) THEN 
    !      stop 'print_inca' 
    !  endif 

     lunout = 6

XIOS

If you are working with a version on XIOS/branchs/xios-2.5, you need to modify XIOS/arch/arch-X64_IRENE.fcm

%BASE_CFLAGS    -std=gnu++98 -diag-disable 1125 -diag-disable 279 -D__XIOS_EXCEPTION

Modifications for V6.2 configurations (only if already extracted)

Compilation and computation

Modify ARCH/arch-X64_IRENE.env (or ARCH/arch-X64_IRENE-AMD.env)

# Compile and running environnement at Irene  

set +e  
module purge
module load intel/20.0.0
module load mpi/openmpi/4.1.4
module load flavor/hdf5/parallel
module load hdf5/1.12.0
module load netcdf-fortran/4.5.3
module load mkl/20.0.0
module load feature/bridge/heterogenous_mpmd
module load c++/gnu/8.3.0
module load c/gnu/8.3.0

post-processing

Modify libIGCM/libIGCM_sys/libIGCM_sys_irene.ksh or libIGCM_sys_irene-amd.ksh to source only env_atlas_irene file. (It will be the same now for irene and irene-amd)

if [ X${TaskType} = Xcomputing ] ; then
  IGCM_debug_Print 1 "Modules will be loaded later in IGCM_sys_activ_variables."
else
  module purge > /dev/null 2>&1
  . $( ccc_home -u igcmg)/MachineEnvironment/irene/env_atlas_irene > /dev/null 2>&1

  export PCMDI_MP=/ccc/work/cont003/igcmg/igcmg/PCMDI-MP
  export UVCDAT_ANONYMOUS_LOG=FALSE
fi

Modification for v6.1 configurations and older ones

compilation

• Create a directory config/.../ARCH/ and create in it the file arch-X64_IRENE.env (or ARCH/arch-X64_IRENE-AMD.env)

  # Compile and running environnemnet at Irene  
  
  set +e  
  module purge
  module load intel/20.0.0
  module load mpi/openmpi/4.1.4
  module load flavor/hdf5/parallel
  module load hdf5/1.12.0
  module load netcdf-fortran/4.5.3
  module load mkl/20.0.0
  module load feature/bridge/heterogenous_mpmd
  module load c++/gnu/8.3.0
  module load c/gnu/8.3.0
  

• Modify AA_make to source this new environment file and indicate to XIOS which environment file it will use

  submitdir=$(shell pwd)
  arch_path=${submitdir}/ARCH/
  
  all :
          if [ -s ./.resol ] ; then $(M_K) `head -1 .resol |cut -c 8-` ; else $(M_K) LMD144142-L79 ; fi
  
  (...)
  
  xios :
        	(cd  ../../modeles/XIOS ; ./make_xios --prod --arch ${FCM_ARCH} --arch_path ${arch_path} --job 4 ; cp bin/xios_server.exe ../../bin/. ; )
  
  
  libioipsl : ../../modeles/IOIPSL/src
  	(. ${arch_path}/arch-X64_IRENE.env; cd ../../modeles/IOIPSL/src ; $(M_K) -f Makefile)
  
  

For older configurations you may have to source the arch.env file for all the models other than XIOS (applies to IPSLCM5A2 but could be needed for other old configurations).

In that case add . ${arch_path}/arch-X64_IRENE.env; before the command that changes the directory in every models rules except for xios where the previous changes are enough.

• IPSLCM5A2 AA_Make

  oasis3-mct:
      (. ${arch_path}/arch-X64_IRENE.env; cd $(OASIS_DIR)/util/make_dir ; cp $(MYSRC_DIR)/SOURCES/OASIS3-MCT/make_${FCM_ARCH} make.inc ; make -f TopMakefileOasis3 ; )
  
  liborchidee : ../../modeles/ORCHIDEE/
      (. ${arch_path}/arch-X64_IRENE.env; cd ../../modeles/ORCHIDEE/ ; ./makeorchidee_fcm -parallel mpi_omp -arch $(FCM_ARCH) -j 8 -xios2)
  
  ...
  
  lmdz: ../../modeles/LMDZ
      (. ${arch_path}/arch-X64_IRENE.env; cd ../../modeles/LMDZ; ./makelmdz_fcm -d $(RESOL_LMDZ) -arch $(FCM_ARCH) -j 8 ce0l ; cp bin/ce0l_$(RESOL_LMDZ)_phylmd_seq.e ../../bin/create_etat0_limit.e ; )
      (. ${arch_path}/arch-X64_IRENE.env; cd ../../modeles/LMDZ ; ./makelmdz_fcm -d $(RESOL_LMDZ) -mem -cosp false -v true -parallel mpi_omp -io xios -c OMCT -job 8 -arch $(FCM_ARCH) gcm  ; mv bin/gcm_$(RESOL_LMDZ)_phylmd_para_mem_orch_couple.e ../../bin/gcm.e ; )
  
  paleolmdz: ../../modeles/LMDZ
      (. ${arch_path}/arch-X64_IRENE.env; cd ../../modeles/LMDZ; ./makelmdz_fcm -d $(RESOL_LMDZ) -arch $(FCM_ARCH) -j 8 ce0l ; cp bin/ce0l_$(RESOL_LMDZ)_phylmd_seq.e ../../bin/create_etat0_limit.e ; )
      (. ${arch_path}/arch-X64_IRENE.env; cd ../../modeles/LMDZ ; ./makelmdz_fcm -d $(RESOL_LMDZ) -mem -cosp false -v true -parallel mpi_omp -io xios -c OMCT -job 8 -arch $(FCM_ARCH) gcm  ; mv bin/gcm_$(RESOL_LMDZ)_phylmd_para_mem_orch_couple.e ../../bin/gcm.e ; )
  
  ...
  
  orca2lim2:  ../../modeles/NEMOGCM/CONFIG
      (. ${arch_path}/arch-X64_IRENE.env; cd ../../modeles/NEMOGCM/CONFIG; cp ../../../config/IPSLCM5A2/SOURCES/NEMO/arch-${FCM_ARCH}.fcm ../ARCH/. ; cp ../../../config/IPSLCM5A2/SOURCES/NEMO/*.cfg ../TOOLS/COMPILE/. ; ./makenemo -m ${FCM_ARCH} -n ORCA2_LIM_PISCES -e "$(MYSRC_DIR)/SOURCES/NEMO/STANDARD" -j16 add_key "key_lim2_vp key_oasis3 key_diahth key_diaar5 key_xios2" del_key "key_nosignedzero " ; cp ORCA2_LIM_PISCES/BLD/bin/nemo.exe ../../../bin/opa ; )
  
  paleorca2lim2:  ../../modeles/NEMOGCM/CONFIG
      (. ${arch_path}/arch-X64_IRENE.env; cd ../../modeles/NEMOGCM/CONFIG; cp ../../../config/IPSLCM5A2/SOURCES/NEMO/arch-${FCM_ARCH}.fcm ../ARCH/. ; cp ../../../config/IPSLCM5A2/SOURCES/NEMO/PALEO/*.cfg ../TOOLS/COMPILE/. ; ./makenemo -m ${FCM_ARCH} -n ORCA2_LIM_PISCES -e "$(MYSRC_DIR)/SOURCES/NEMO/PALEO" -j16 add_key "key_lim2_vp key_oasis3 key_diahth key_diaar5 key_xios2" del_key "key_nosignedzero " ; cp ORCA2_LIM_PISCES/BLD/bin/nemo.exe ../../../bin/opa ; )
  

• create the Makefile as explain ​here
• clean everything on previous compilations :

gmake clean

• compile

Computation

• Extract a new version of libIGCM

  svn co -r 1581 https://forge.ipsl.jussieu.fr/libigcm/svn/trunk/libIGCM libIGCM
  

• Modify config.card file section [UserChoices] to add environment file parameter:

  EnvFile=${SUBMIT_DIR}/../ARCH/arch.env
  

Workflow CMIP6

Not yet available on redhat8