Porting on RedHat8

After maintenance of 11-12th of April 2023 all the Irene machines of the TGCC changed OS (from redhat7 to redhat8). As an addition of the change in OS

How to install your environment on redhat8 (skl and rome)

Use the igcmg environment (in bash) with a copy of the IPSL bashrc script shell files in your HOME (and save your old ones to be used if necessary on irene-prev).

ryyy999@irene: cp ~igcmg/MachineEnvironment/irene/bashrc  ~/.bashrc
ryyy999@irene: mv ~/.bashrc_irene ~/.bashrc_irene_prev
ryyy999@irene: mv ~/.bashrc_irene-amd ~/.bashrc_irene-amd_prev
ryyy999@irene: cp ~igcmg/MachineEnvironment/irene/bashrc_irene ~/.bashrc_irene

Note that the same file .bashrc_irene is now used both for Skylake and amd (Rome). This was not the case before. Additionally, you can complete the example of bashrc_irene file to create your favorite environment (alias, module load ...) by copying what you did in your previous file (now named ~/.bashrc_irene_prev or ~/.bashrc_irene-amd_prev).

We strongly advice you to add the line module switch dfldatadir dfldatadir/genXXXX in your own .bashrc_irene.

WARNING : if you have a ~/.profile file, it's better to remove it to avoid any problem during the execution of a simulation with libIGCM.

In this environment is specified:

• the path to the compiler tool fcm and to the rebuild tool which recombines output files from a parallel model:

  export PATH=$(ccc_home -u igcmg)/Tools/fcm/bin:$(ccc_home -u igcmg)/Tools/irene/bin:$PATH
  

• the load of modules giving access to analyze data or post processing libraries and tools needed on our platform (done in ccc_home -u igcmg/MachineEnvironment/irene/env_atlas_irene).

• Command module purge gives error messages but it is still working (these errors will appaer on connexion and on Script_output_files, and out_compiled files) . The proposed login environment above will therefore give errors while connecting. TGCC is aware of this issue.

  > module purge
  module dfldatadir/gen6328 (Data Directory) cannot be unloaded
  
  Unloading datadir/gen6328
    ERROR: Dependent dfldatadir/gen6328 is loaded
  
  Unloading ccc/1.0
    ERROR: Dependent datadir/gen6328 and dfldatadir/gen6328 are loaded
  

WARNING : You cannot load in the same time the intel environment for compilation and models computation, AND ferret software. They are not compatible. That's why by default we propose an environment for post processing. The computation environment will be loaded by modipsl and libIGCM during compilation and running.

New installation

Main working configurations have been updated to work by default at irene/redhat8, skylake and amd/rome. You need first to make a complete new installation of modipsl and then extract the configuration you want. Applay the modifications listed below before compilation.

The following configurations work by default (after reinstalltaion):

• ORCHIDEE_trunk, ORCHIDEE_4_1, ORCHIDEE_3_head, ORCHIDEE_2_2, ORCHIDEE_2_1 : no modifications are needed

Following configurations need some manual modifications:

• IPSLCM6.2_work, LMDZOR_v6.2_work, LMDZORINCA_v6.2_work, LMDZOR_v6.3_work, LMDZORINCA_v6.3_beta, LMDZOR_v6.4_work, IPSLCM7_work, ICOLMDZOR_v7_work, ICOLMDZORINCA_v7_work, ICOLMDZOR_v7_LAM_work
  • only modification needed is to change before compilation the line BASE_LD in modipsl/modeles/LMDZ/arch/arch-X64_IRENE.fcm, see section LMDZ below
• ORCHIDEE_2_0
  • Modification in XIOS on variable %BASE_CFLAGS as described below
  • As described below in Modification for v6.1 configurations and older ones

Tagged configurations and all other configurations have not been updated. If you need to use one of them, you therefore need to apply all the changes listed in the sections below: modifications in models and in configurations.

Modifications needed to be done in models

With the new OS redhat8 you will use a new version of svn that will be incompatible with directories extracted on redhat7 - To correct this you need to run the command "svn upgrade" in your directories

LMDZ

Replace %BASE_LD in LMDZ/arch/arch-X64_IRENE.fcm by
%BASE_LD -i4 -r8 -auto -L/ccc/products/mkl-20.0.0/system/default/20.0.0/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_core -lmkl_sequential -lpthread

ORCHIDEE

The new version of modules we now use on redhat8 can not handle to many files opened at the same time. When running LMDZOR on hybrid mode (mixed MPI-OpenMP), too many files text output files were opened in ORCHIDEE (out_orchidee_000x.000y). This has been changed in newer versions of ORCHIDEE: in revision 7790 in the trunk, revision 7792 in branch ORHIDEE_2_2 and revision 7967 in the branch ORCHIDEE_3. For these revisions you need to have PRINTLEV=1 (default setting in orchidee.def).

If you work with another version, do the following work-around in the code :

In modeles/ORCHIDEE/src_parallel/mod_orchidee_para.F90, comment the opening of the text output files and change to have numout=6, the output will now be sent to standard output for all cores.

!!!    OPEN(UNIT=numout,FILE=TRIM(fileout),ACTION='write',STATUS='unknown',FORM='formatted',IOSTAT=ierr)
!!!    IF (ierr /= 0) THEN
!!!    #ifdef CPP_PARA
!!!        CALL MPI_FINALIZE(ierr)
!!!    #endif
!!!        WRITE(*,*) "In Set_stdout_file : Erreur can't open file ", filename
!!!        STOP 1
!!!    ENDIF

numout=6

INCA

If you are working with a version older than rev 1050, you need to modify lunout parameter to use standard output file. For this in INCA/src/INCA_MOD/print_inca.F90 comment the opening of the text output files and change to have lunout=6

    !    open(UNIT=lunout,file=fileout, action='write',status='unknown', form='formatted', iostat=ierr2)
    !  IF (ierr2 /= 0) THEN 
    !      stop 'print_inca' 
    !  endif 

     lunout = 6

XIOS

If you are working with a version on XIOS/branchs/xios-2.5, you need to modify XIOS/arch/arch-X64_IRENE.fcm (or XIOS/arch/arch-X64_IRENE-AMD.fcm) modify

%BASE_CFLAGS    -diag-disable 1125 -diag-disable 279 -D BOOST_NO_CXX11_DEFAULTED_FUNCTIONS -D BOOST_NO_CXX11_DELETED_FUNCTIONS

on

%BASE_CFLAGS    -std=gnu++98 -diag-disable 1125 -diag-disable 279 -D__XIOS_EXCEPTION

Modifications for v6.2-v6.5 configurations

Modules for compilation and computation

See here the modules that you should use at irene redhat8. We use the same modules at Irene skylake and Irene amd/Rome.

Modify to the following modipsl/config/xxxx/ARCH/arch-X64_IRENE.env (or ARCH/arch-X64_IRENE-AMD.env):

# Compile and running environement at Irene  

set +e  
module purge
module load intel/20.0.0
module load mpi/openmpi/4.1.4
module load flavor/hdf5/parallel
module load hdf5/1.12.2
module load netcdf-fortran/4.5.3
module load mkl/20.0.0
module load feature/bridge/heterogenous_mpmd
module load c++/gnu/8.3.0
module load c/gnu/8.3.0

libIGCM

Adaptation for the post-processing has been done in libIGCM. You need revision 1582 or later. In the new version, only env_atlas_irene file is now sourced to avoid conflicts.

• Extract a new version of libIGCM

  cd modipsl
  mv libIGCM libIGCM.old
  svn co -r 1582 https://forge.ipsl.fr/libigcm/svn/trunk/libIGCM libIGCM
  

• If this command is asking you a password, you can ask it to platform-users[at]listes.ipsl.fr
• If you already had an experiment folder, delete the old job and create a new one with ins_job as usual. This should always be done if you update your version of libIGCM.

Modification for v6.1 configurations and older ones

Compilation

• Create a directory config/.../ARCH and create in it the file arch-X64_IRENE.env (or ARCH/arch-X64_IRENE-AMD.env)

  # Compile and running environnemnet at Irene  
  
  set +e  
  module purge
  module load intel/20.0.0
  module load mpi/openmpi/4.1.4
  module load flavor/hdf5/parallel
  module load hdf5/1.12.2
  module load netcdf-fortran/4.5.3
  module load mkl/20.0.0
  module load feature/bridge/heterogenous_mpmd
  module load c++/gnu/8.3.0
  module load c/gnu/8.3.0
  

In this directory ARCH, create the symbolic link :

ln -s arch-X64_IRENE-AMD.env arch.env

or

ln -s arch-X64_IRENE.env arch.env

• Modify AA_make to source this new environment file and indicate to XIOS which environment file it will use, for this add these lines just before the line beginning by "all : "

      submitdir=$(shell pwd)
      arch_path=${submitdir}/ARCH/
  
      all :
       (...)
  
  

Then modify the compilation line for xios by adding
̀--arch_path ${arch_path}

for example (in one forced configuration) :

xios :
      	(cd  ../../modeles/XIOS ; ./make_xios --prod --arch ${FCM_ARCH} --job 4 ; cp bin/xios_server.exe ../../bin/. ; )

will become

xios :
      	(cd  ../../modeles/XIOS ; ./make_xios --prod --arch ${FCM_ARCH} --arch_path ${arch_path} --job 4 ; cp bin/xios_server.exe ../../bin/. ; )

Then add `. ${arch_path}/arch-X64_IRENE.env;̀

before ALL models compilation. For example for libioipsl,

libioipsl : ../../modeles/IOIPSL/src
	(cd ../../modeles/IOIPSL/src ; $(M_K) -f Makefile)

will become

libioipsl : ../../modeles/IOIPSL/src
	(. ${arch_path}/arch-X64_IRENE.env; cd ../../modeles/IOIPSL/src ; $(M_K) -f Makefile)

• create the Makefile as explain ​here
• clean everything on previous compilations :

gmake clean

• compile

Computation

• Extract a new version of libIGCM

  cd modipsl
  mv libIGCM libIGCM.old
  svn co -r 1582 https://forge.ipsl.fr/libigcm/svn/trunk/libIGCM libIGCM
  

  • If this command is asking you a password, you can ask it to platform-users[at]listes.ipsl.fr
  • If you already had an experiment folder, delete the old job and create a new one with ins_job as usual. This command will also create post-treatment job in libIGCM directory

• Modify config.card file section [UserChoices] to add environment file parameter:

  EnvFile=${SUBMIT_DIR}/../ARCH/arch.env
  

Workflow CMIP6

Not yet available on redhat8