{{{
#!html
Porting on RedHat8
}}}
[[NoteBox(note, After maintenance of April 11 and 12 all the Irene machines of the TGCC will change OS (it will go from redhat7 to redhat8). As an addition of the change in OS old modules will not be available on redhat8 and we need to update compiler versions for our modules., 600px)]]
[[PageOutline(1-2,Table of contents)]]
= How to install your environment on redhat8 (skl and rome) =
Use the igcmg environment (in bash) with a copy of the IPSL bashrc script shell files in your HOME (and save your old ones to be used if necessary on irene-prev).
{{{
#!sh
ryyy999@irene: cp ~igcmg/MachineEnvironment/irene/bashrc ~/.bashrc
ryyy999@irene: mv ~/.bashrc_irene ~/.bashrc_irene_prev
ryyy999@irene: mv ~/.bashrc_irene-amd ~/.bashrc_irene-amd_prev
ryyy999@irene: cp ~igcmg/MachineEnvironment/irene/bashrc_irene ~/.bashrc_irene
}}}
Note that the same file `.bashrc_irene` is now used both for Skylake and amd (Rome). This was not the case before.
Additionally, you can complete the example of bashrc_irene file to create your favorite environment (`alias`, `module load` ...) by copying what you did in your previous file (now named `~/.bashrc_irene_prev` or `~/.bashrc_irene-amd_prev`).
''We strongly advice you to add the line `module switch dfldatadir dfldatadir/genXXXX` in your own .bashrc_irene.''
WARNING : if you have a `~/.profile` file, it's better to remove it to avoid any problem during the execution of a simulation with libIGCM.
In this environment is specified:
* the path to the compiler tool `fcm` and to the `rebuild` tool which recombines output files from a parallel model:
{{{
export PATH=$(ccc_home -u igcmg)/Tools/fcm/bin:$(ccc_home -u igcmg)/Tools/irene/bin:$PATH
}}}
* the load of modules giving access to analyze data or post processing libraries and tools needed on our platform (done in `ccc_home -u igcmg`/!MachineEnvironment/irene/env_atlas_irene).
Then to allow your `.bashrc_irene[-amd]_prev` files to properly load the env and env_atlas files,[[BR]]
modify the lines containing `MachineEnvironment/irene[-amd]/env[_atlas]_irene` by adding _prev.[[BR]]
They should look like this:
`MachineEnvironment/irene[-amd]_prev/env[_atlas]_irene[-amd]`
[[Image(warning.png, 50px)]] WARNING : You cannot load in the same time the intel environment for compilation and models computation, AND ferret software. They are not compatible. That's why by default we propose an environment for post processing. The computation environment will be loaded by modipsl and libIGCM during compilation and running.
= New installation =
Main working configurations have been updated to work by default at irene/redhat8, skylake and amd/rome. You need first to make a complete new installation of modipsl and then extract the configuration you want. Applay the modifications listed below before compilation.
The following configurations work by default (after reinstalltaion):
* ORCHIDEE_trunk, ORCHIDEE_4_1, ORCHIDEE_3_head, ORCHIDEE_2_2, ORCHIDEE_2_1, ORCHIDEE_2_0 : no modifications are needed
Following configurations need some manual modifications:
* IPSLCM6.2_work, LMDZOR_v6.2_work, LMDZORINCA_v6.2_work, LMDZOR_v6.3_work, LMDZORINCA_v6.3_beta, LMDZOR_v6.4_work, IPSLCM7_work, ICOLMDZOR_v7_work, ICOLMDZORINCA_v7_work, ICOLMDZOR_v7_LAM_work
* only modification needed is to change before compilation the line BASE_LD in modipsl/modeles/LMDZ/arch/arch-X64_IRENE.fcm, see section LMDZ below
Tagged configurations and all other configurations have not been updated. If you need to use one of them, you therefore need to apply all the changes listed in the sections below: modifications in models and in configurations.
= Modifications needed to be done in models =
[[NoteBox(note, With the new OS redhat8 you will use a new version of svn that will be incompatible with directories extracted on redhat7 - To correct this you need to run the command "svn upgrade" in your directories, 600px)]]
== LMDZ ==
Replace `%BASE_LD` in `LMDZ/arch/arch-X64_IRENE.fcm` by [[BR]]
`%BASE_LD -i4 -r8 -auto -L/ccc/products/mkl-20.0.0/system/default/20.0.0/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_core -lmkl_sequential -lpthread`
== ORCHIDEE ==
The new version of modules we now use on redhat8 can not handle to many files opened at the same time. When running LMDZOR on hybrid mode (mixed MPI-OpenMP), too many files text output files were opened in ORCHIDEE (out_orchidee_000x.000y). This has been changed in newer versions of ORCHIDEE: in revision 7790 in the trunk, revision 7792 in branch ORHIDEE_2_2 and revision 7967 in the branch ORCHIDEE_3. For these revisions you need to have PRINTLEV=1 (default setting in orchidee.def).
If you work with another version, do the following work-around in the code : [[BR]]
In '''modeles/ORCHIDEE/src_parallel/mod_orchidee_para.F90''', comment the opening of the text output files and change to have numout=6, the output will now be sent to standard output for all cores.
{{{
!!! OPEN(UNIT=numout,FILE=TRIM(fileout),ACTION='write',STATUS='unknown',FORM='formatted',IOSTAT=ierr)
!!! IF (ierr /= 0) THEN
!!! #ifdef CPP_PARA
!!! CALL MPI_FINALIZE(ierr)
!!! #endif
!!! WRITE(*,*) "In Set_stdout_file : Erreur can't open file ", filename
!!! STOP 1
!!! ENDIF
numout=6
}}}
== INCA ==
If you are working with a version older than rev 1050, you need to modify lunout parameter to use standard output file. For this in INCA/src/INCA_MOD/print_inca.F90 comment the opening of the text output files and change to have lunout=6
{{{
! open(UNIT=lunout,file=fileout, action='write',status='unknown', form='formatted', iostat=ierr2)
! IF (ierr2 /= 0) THEN
! stop 'print_inca'
! endif
lunout = 6
}}}
== XIOS ==
If you are working with a version on XIOS/branchs/xios-2.5, you need to modify `XIOS/arch/arch-X64_IRENE.fcm` (or `XIOS/arch/arch-X64_IRENE-AMD.env`)
{{{
%BASE_CFLAGS -std=gnu++98 -diag-disable 1125 -diag-disable 279 -D__XIOS_EXCEPTION
}}}
= Modifications for v6.2-v6.5 configurations =
== Modules for compilation and computation ==
See here the modules that you should use at irene redhat8. We use the same modules at Irene skylake and Irene amd/Rome.
Modify to the following `modipsl/config/xxxx/ARCH/arch-X64_IRENE.env` (or `ARCH/arch-X64_IRENE-AMD.env`):
{{{
# Compile and running environement at Irene
set +e
module purge
module load intel/20.0.0
module load mpi/openmpi/4.1.4
module load flavor/hdf5/parallel
module load hdf5/1.12.0
module load netcdf-fortran/4.5.3
module load mkl/20.0.0
module load feature/bridge/heterogenous_mpmd
module load c++/gnu/8.3.0
module load c/gnu/8.3.0
}}}
== libIGCM ==
Adaptation for the post-processing has been done in libIGCM. You need revision 1582 or later. In the new version, only env_atlas_irene file is now sourced to avoid conflicts.
* Extract a new version of libIGCM
{{{
cd modipsl
mv libIGCM libIGCM.old
svn co -r 1582 https://forge.ipsl.jussieu.fr/libigcm/svn/trunk/libIGCM libIGCM
}}}
* If you already had an experiment folder, delete the old job and create a new one with ins_job as usual. This should always be done if you update your version of libIGCM.
= Modification for v6.1 configurations and older ones =
== Compilation ==
* Create a directory `config/.../ARCH` and create in it the file `arch-X64_IRENE.env` (or `ARCH/arch-X64_IRENE-AMD.env`)
{{{
# Compile and running environnemnet at Irene
set +e
module purge
module load intel/20.0.0
module load mpi/openmpi/4.1.4
module load flavor/hdf5/parallel
module load hdf5/1.12.0
module load netcdf-fortran/4.5.3
module load mkl/20.0.0
module load feature/bridge/heterogenous_mpmd
module load c++/gnu/8.3.0
module load c/gnu/8.3.0
}}}
In this directory `ARCH`, create the symbolic link :
{{{
ln -s arch-X64_IRENE-AMD.env arch.env
}}}
or
{{{
ln -s arch-X64_IRENE.env arch.env
}}}
* Modify `AA_make` to source this new environment file and indicate to XIOS which environment file it will use
{{{
submitdir=$(shell pwd)
arch_path=${submitdir}/ARCH/
all :
if [ -s ./.resol ] ; then $(M_K) `head -1 .resol |cut -c 8-` ; else $(M_K) LMD144142-L79 ; fi
(...)
## \!/ XIOS compilation for forced models
xios :
(cd ../../modeles/XIOS ; ./make_xios --prod --arch ${FCM_ARCH} --arch_path ${arch_path} --job 4 ; cp bin/xios_server.exe ../../bin/. ; )
## \!/ XIOS compilation for coupled model
xios: ../../modeles/XIOS
(cd ../../modeles/XIOS; ./make_xios --arch $(FCM_ARCH) --arch_path ${arch_path} --prod --use_oasis oasis3_mct --job 8 ; cp bin/xios_server.exe ../../bin/. ; )
libioipsl : ../../modeles/IOIPSL/src
(. ${arch_path}/arch-X64_IRENE.env; cd ../../modeles/IOIPSL/src ; $(M_K) -f Makefile)
}}}
For older configurations you may have to source the arch.env file for all the models other than XIOS (applies to IPSLCM5A2 but could be needed for other old configurations).
In that case add `. ${arch_path}/arch-X64_IRENE.env; ` (or `. ${arch_path}/arch-X64_IRENE-AMD.env; `) before the command that changes the directory in every models rules except for xios where the previous changes are enough.
* IPSLCM5A2 `AA_Make`
{{{
MYSRC_DIR = $(shell /bin/pwd)
arch_path=${MYSRC_DIR}/ARCH/
all :
if [ -s ./.resol ] ; then $(M_K) `head -1 .resol` ; else $(M_K) IPSLCM5A2-VLR ; fi
...
libioipsl : ../../modeles/IOIPSL/src
(. ${arch_path}/arch-X64_IRENE.env; cd ../../modeles/IOIPSL/src ; $(M_K) -f Makefile)
oasis3-mct:
(. ${arch_path}/arch-X64_IRENE.env; cd $(OASIS_DIR)/util/make_dir ; cp $(MYSRC_DIR)/SOURCES/OASIS3-MCT/make_${FCM_ARCH} make.inc ; make -f TopMakefileOasis3 ; )
liborchidee : ../../modeles/ORCHIDEE/
(. ${arch_path}/arch-X64_IRENE.env; cd ../../modeles/ORCHIDEE/ ; ./makeorchidee_fcm -parallel mpi_omp -arch $(FCM_ARCH) -j 8 -xios2)
...
lmdz: ../../modeles/LMDZ
(. ${arch_path}/arch-X64_IRENE.env; cd ../../modeles/LMDZ; ./makelmdz_fcm -d $(RESOL_LMDZ) -arch $(FCM_ARCH) -j 8 ce0l ; cp bin/ce0l_$(RESOL_LMDZ)_phylmd_seq.e ../../bin/create_etat0_limit.e ; )
(. ${arch_path}/arch-X64_IRENE.env; cd ../../modeles/LMDZ ; ./makelmdz_fcm -d $(RESOL_LMDZ) -mem -cosp false -v true -parallel mpi_omp -io xios -c OMCT -job 8 -arch $(FCM_ARCH) gcm ; mv bin/gcm_$(RESOL_LMDZ)_phylmd_para_mem_orch_couple.e ../../bin/gcm.e ; )
paleolmdz: ../../modeles/LMDZ
(. ${arch_path}/arch-X64_IRENE.env; cd ../../modeles/LMDZ; ./makelmdz_fcm -d $(RESOL_LMDZ) -arch $(FCM_ARCH) -j 8 ce0l ; cp bin/ce0l_$(RESOL_LMDZ)_phylmd_seq.e ../../bin/create_etat0_limit.e ; )
(. ${arch_path}/arch-X64_IRENE.env; cd ../../modeles/LMDZ ; ./makelmdz_fcm -d $(RESOL_LMDZ) -mem -cosp false -v true -parallel mpi_omp -io xios -c OMCT -job 8 -arch $(FCM_ARCH) gcm ; mv bin/gcm_$(RESOL_LMDZ)_phylmd_para_mem_orch_couple.e ../../bin/gcm.e ; )
...
xios: ../../modeles/XIOS
(cd ../../modeles/XIOS; ./make_xios --arch $(FCM_ARCH) --arch_path ${arch_path} --prod --use_oasis oasis3_mct --job 8 ; cp bin/xios_server.exe ../../bin/. ; )
...
orca2lim2: ../../modeles/NEMOGCM/CONFIG
(. ${arch_path}/arch-X64_IRENE.env; cd ../../modeles/NEMOGCM/CONFIG; cp ../../../config/IPSLCM5A2/SOURCES/NEMO/arch-${FCM_ARCH}.fcm ../ARCH/. ; cp ../../../config/IPSLCM5A2/SOURCES/NEMO/*.cfg ../TOOLS/COMPILE/. ; ./makenemo -m ${FCM_ARCH} -n ORCA2_LIM_PISCES -e "$(MYSRC_DIR)/SOURCES/NEMO/STANDARD" -j16 add_key "key_lim2_vp key_oasis3 key_diahth key_diaar5 key_xios2" del_key "key_nosignedzero " ; cp ORCA2_LIM_PISCES/BLD/bin/nemo.exe ../../../bin/opa ; )
paleorca2lim2: ../../modeles/NEMOGCM/CONFIG
(. ${arch_path}/arch-X64_IRENE.env; cd ../../modeles/NEMOGCM/CONFIG; cp ../../../config/IPSLCM5A2/SOURCES/NEMO/arch-${FCM_ARCH}.fcm ../ARCH/. ; cp ../../../config/IPSLCM5A2/SOURCES/NEMO/PALEO/*.cfg ../TOOLS/COMPILE/. ; ./makenemo -m ${FCM_ARCH} -n ORCA2_LIM_PISCES -e "$(MYSRC_DIR)/SOURCES/NEMO/PALEO" -j16 add_key "key_lim2_vp key_oasis3 key_diahth key_diaar5 key_xios2" del_key "key_nosignedzero " ; cp ORCA2_LIM_PISCES/BLD/bin/nemo.exe ../../../bin/opa ; )
}}}
* create the Makefile as explain [https://forge.ipsl.jussieu.fr/igcmg_doc/wiki/Doc/Compile#Creatingthemainmakefile here]
* clean everything on previous compilations :
{{{gmake clean }}}
* compile
== Computation ==
* Extract a new version of libIGCM
{{{
svn co -r 1581 https://forge.ipsl.jussieu.fr/libigcm/svn/trunk/libIGCM libIGCM
}}}
* Modify config.card file section `[UserChoices]` to add environment file parameter:
{{{
EnvFile=${SUBMIT_DIR}/../ARCH/arch.env
}}}
= Workflow CMIP6 =
Not yet available on redhat8
= Known errors =
* Command module purge gives error messages but it is still working. The proposed login environment above will therefore give errors while connecting. TGCC is aware of this issue.
{{{
> module purge
module dfldatadir/gen6328 (Data Directory) cannot be unloaded
Unloading datadir/gen6328
ERROR: Dependent dfldatadir/gen6328 is loaded
Unloading ccc/1.0
ERROR: Dependent datadir/gen6328 and dfldatadir/gen6328 are loaded
}}}
* XIOS using more that 1 server and option one_file. Option multiple_file or using just 1 server is working.