After maintenance of April 11 and 12 all the Irene machines of the TGCC will change OS (it will go from redhat7 to redhat8). As an addition of the change in OS old modules will not be available on redhat8 and we need to update compiler versions for our modules.
Use the igcmg environment (in bash) with a copy of the IPSL bashrc script shell files in your HOME (and save your old ones to be used if necessary on irene-prev).
ryyy999@irene: cp ~igcmg/MachineEnvironment/irene/bashrc ~/.bashrc ryyy999@irene: mv ~/.bashrc_irene ~/.bashrc_irene_prev ryyy999@irene: mv ~/.bashrc_irene-amd ~/.bashrc_irene-amd_prev ryyy999@irene: cp ~igcmg/MachineEnvironment/irene/bashrc_irene ~/.bashrc_irene
Note that the same file .bashrc_irene is now used both for Skylake and amd (Rome). This was not the case before. Additionally, you can complete the example of bashrc_irene file to create your favorite environment (alias, module load ...) by copying what you did in your previous file (now named ~/.bashrc_irene_prev or ~/.bashrc_irene-amd_prev).
We strongly advice you to add the line module switch dfldatadir dfldatadir/genXXXX in your own .bashrc_irene_next.
WARNING : if you have a ~/.profile file, it's better to remove it to avoid any problem during the execution of a simulation with libIGCM.
In this environment is specified:
export PATH=$(ccc_home -u igcmg)/Tools/fcm/bin:$(ccc_home -u igcmg)/Tools/irene/bin:$PATH
WARNING : You cannot load in the same time the intel environment for compilation and models computation, AND ferret software. They are not compatible. That's why by default we propose an environment for post processing. The computation environment will be loaded by modipsl and libIGCM during compilation and running.
Main working configurations have been updated to work by default at irene/redhat8, skylake and amd/rome. You need first to make a complete new installation of modipsl and then extract the configuration you want. Applay the modifications listed below before compilation.
The following configurations work by default (after reinstalltaion):
Following configurations need some manual modifications:
Tagged configurations and all other configurations have not been updated. If you need to use one of them, you therefore need to apply all the changes listed in the sections below: modifications in models and in configurations.
With the new OS redhat8 you will use a new version of svn that will be incompatible with directories extracted on redhat7 - To correct this you need to run the command "svn upgrade" in your directories
Replace %BASE_LD in LMDZ/arch/arch-X64_IRENE.fcm by
%BASE_LD -i4 -r8 -auto -L/ccc/products/mkl-20.0.0/system/default/20.0.0/mkl/lib/intel64 -lmkl_intel_lp64 -lmkl_core -lmkl_sequential -lpthread
The new version of modules we now use on redhat8 can not handle to many files opened at the same time. When running LMDZOR on hybrid mode (mixed MPI-OpenMP), too many files text output files were opened in ORCHIDEE (out_orchidee_000x.000y). This has been changed in newer versions of ORCHIDEE: in revision 7792 for branch ORHIDEE_2_2 and 7790 in the trunk. For these revisions you need to have PRINTLEV=1 (default setting in orchidee.def).
If you work with another version, do the following work-around in the code :
In modeles/ORCHIDEE/src_parallel/mod_orchidee_para.F90, comment the opening of the text output files and change to have numout=6, the output will now be sent to standard output for all cores.
!!! OPEN(UNIT=numout,FILE=TRIM(fileout),ACTION='write',STATUS='unknown',FORM='formatted',IOSTAT=ierr) !!! IF (ierr /= 0) THEN !!! #ifdef CPP_PARA !!! CALL MPI_FINALIZE(ierr) !!! #endif !!! WRITE(*,*) "In Set_stdout_file : Erreur can't open file ", filename !!! STOP 1 !!! ENDIF numout=6
If you are working with a version older than rev 1050, you need to modify lunout parameter to use standard output file. For this in INCA/src/INCA_MOD/print_inca.F90 comment the opening of the text output files and change to have lunout=6
! open(UNIT=lunout,file=fileout, action='write',status='unknown', form='formatted', iostat=ierr2) ! IF (ierr2 /= 0) THEN ! stop 'print_inca' ! endif lunout = 6
If you are working with a version on XIOS/branchs/xios-2.5, you need to modify XIOS/arch/arch-X64_IRENE.fcm (or ARCH/arch-X64_IRENE-AMD.env)
%BASE_CFLAGS -std=gnu++98 -diag-disable 1125 -diag-disable 279 -D__XIOS_EXCEPTION
See here the modules that you should use at irene redhat8. We use the same modules at Irene skylake and Irene amd/Rome.
Modify to the following modipsl/config/xxxx/ARCH/arch-X64_IRENE.env (or ARCH/arch-X64_IRENE-AMD.env):
# Compile and running environement at Irene set +e module purge module load intel/20.0.0 module load mpi/openmpi/4.1.4 module load flavor/hdf5/parallel module load hdf5/1.12.0 module load netcdf-fortran/4.5.3 module load mkl/20.0.0 module load feature/bridge/heterogenous_mpmd module load c++/gnu/8.3.0 module load c/gnu/8.3.0
Adaptation for the post-processing has been done in libIGCM. You need revision 1581 or later. In the new version, only env_atlas_irene file is now sourced to avoid conflicts.
cd modipsl mv libIGCM libIGCM.old svn co -r 1581 https://forge.ipsl.jussieu.fr/libigcm/svn/trunk/libIGCM libIGCM
# Compile and running environnemnet at Irene set +e module purge module load intel/20.0.0 module load mpi/openmpi/4.1.4 module load flavor/hdf5/parallel module load hdf5/1.12.0 module load netcdf-fortran/4.5.3 module load mkl/20.0.0 module load feature/bridge/heterogenous_mpmd module load c++/gnu/8.3.0 module load c/gnu/8.3.0
In this diretory ARCH, create the symbolic link :
ln -s arch-X64_IRENE-AMD.env arch.env
or
ln -s arch-X64_IRENE.env arch.env
submitdir=$(shell pwd) arch_path=${submitdir}/ARCH/ all : if [ -s ./.resol ] ; then $(M_K) `head -1 .resol |cut -c 8-` ; else $(M_K) LMD144142-L79 ; fi (...) xios : (cd ../../modeles/XIOS ; ./make_xios --prod --arch ${FCM_ARCH} --arch_path ${arch_path} --job 4 ; cp bin/xios_server.exe ../../bin/. ; ) libioipsl : ../../modeles/IOIPSL/src (. ${arch_path}/arch-X64_IRENE.env; cd ../../modeles/IOIPSL/src ; $(M_K) -f Makefile)
For older configurations you may have to source the arch.env file for all the models other than XIOS (applies to IPSLCM5A2 but could be needed for other old configurations).
In that case add . ${arch_path}/arch-X64_IRENE.env; (or . ${arch_path}/arch-X64_IRENE-AMD.env; ) before the command that changes the directory in every models rules except for xios where the previous changes are enough.
MYSRC_DIR = $(shell /bin/pwd) arch_path=${MYSRC_DIR}/ARCH/ all : if [ -s ./.resol ] ; then $(M_K) `head -1 .resol` ; else $(M_K) IPSLCM5A2-VLR ; fi ... libioipsl : ../../modeles/IOIPSL/src (. ${arch_path}/arch-X64_IRENE.env; cd ../../modeles/IOIPSL/src ; $(M_K) -f Makefile) oasis3-mct: (. ${arch_path}/arch-X64_IRENE.env; cd $(OASIS_DIR)/util/make_dir ; cp $(MYSRC_DIR)/SOURCES/OASIS3-MCT/make_${FCM_ARCH} make.inc ; make -f TopMakefileOasis3 ; ) liborchidee : ../../modeles/ORCHIDEE/ (. ${arch_path}/arch-X64_IRENE.env; cd ../../modeles/ORCHIDEE/ ; ./makeorchidee_fcm -parallel mpi_omp -arch $(FCM_ARCH) -j 8 -xios2) ... lmdz: ../../modeles/LMDZ (. ${arch_path}/arch-X64_IRENE.env; cd ../../modeles/LMDZ; ./makelmdz_fcm -d $(RESOL_LMDZ) -arch $(FCM_ARCH) -j 8 ce0l ; cp bin/ce0l_$(RESOL_LMDZ)_phylmd_seq.e ../../bin/create_etat0_limit.e ; ) (. ${arch_path}/arch-X64_IRENE.env; cd ../../modeles/LMDZ ; ./makelmdz_fcm -d $(RESOL_LMDZ) -mem -cosp false -v true -parallel mpi_omp -io xios -c OMCT -job 8 -arch $(FCM_ARCH) gcm ; mv bin/gcm_$(RESOL_LMDZ)_phylmd_para_mem_orch_couple.e ../../bin/gcm.e ; ) paleolmdz: ../../modeles/LMDZ (. ${arch_path}/arch-X64_IRENE.env; cd ../../modeles/LMDZ; ./makelmdz_fcm -d $(RESOL_LMDZ) -arch $(FCM_ARCH) -j 8 ce0l ; cp bin/ce0l_$(RESOL_LMDZ)_phylmd_seq.e ../../bin/create_etat0_limit.e ; ) (. ${arch_path}/arch-X64_IRENE.env; cd ../../modeles/LMDZ ; ./makelmdz_fcm -d $(RESOL_LMDZ) -mem -cosp false -v true -parallel mpi_omp -io xios -c OMCT -job 8 -arch $(FCM_ARCH) gcm ; mv bin/gcm_$(RESOL_LMDZ)_phylmd_para_mem_orch_couple.e ../../bin/gcm.e ; ) ... xios: ../../modeles/XIOS (cd ../../modeles/XIOS; ./make_xios --arch $(FCM_ARCH) --arch_path ${arch_path} --prod --use_oasis oasis3_mct --job 8 ; cp bin/xios_server.exe ../../bin/. ; ) ... orca2lim2: ../../modeles/NEMOGCM/CONFIG (. ${arch_path}/arch-X64_IRENE.env; cd ../../modeles/NEMOGCM/CONFIG; cp ../../../config/IPSLCM5A2/SOURCES/NEMO/arch-${FCM_ARCH}.fcm ../ARCH/. ; cp ../../../config/IPSLCM5A2/SOURCES/NEMO/*.cfg ../TOOLS/COMPILE/. ; ./makenemo -m ${FCM_ARCH} -n ORCA2_LIM_PISCES -e "$(MYSRC_DIR)/SOURCES/NEMO/STANDARD" -j16 add_key "key_lim2_vp key_oasis3 key_diahth key_diaar5 key_xios2" del_key "key_nosignedzero " ; cp ORCA2_LIM_PISCES/BLD/bin/nemo.exe ../../../bin/opa ; ) paleorca2lim2: ../../modeles/NEMOGCM/CONFIG (. ${arch_path}/arch-X64_IRENE.env; cd ../../modeles/NEMOGCM/CONFIG; cp ../../../config/IPSLCM5A2/SOURCES/NEMO/arch-${FCM_ARCH}.fcm ../ARCH/. ; cp ../../../config/IPSLCM5A2/SOURCES/NEMO/PALEO/*.cfg ../TOOLS/COMPILE/. ; ./makenemo -m ${FCM_ARCH} -n ORCA2_LIM_PISCES -e "$(MYSRC_DIR)/SOURCES/NEMO/PALEO" -j16 add_key "key_lim2_vp key_oasis3 key_diahth key_diaar5 key_xios2" del_key "key_nosignedzero " ; cp ORCA2_LIM_PISCES/BLD/bin/nemo.exe ../../../bin/opa ; )
gmake clean
svn co -r 1581 https://forge.ipsl.jussieu.fr/libigcm/svn/trunk/libIGCM libIGCM
EnvFile=${SUBMIT_DIR}/../ARCH/arch.env
Not yet available on redhat8
> module purge module dfldatadir/gen6328 (Data Directory) cannot be unloaded Unloading datadir/gen6328 ERROR: Dependent dfldatadir/gen6328 is loaded Unloading ccc/1.0 ERROR: Dependent datadir/gen6328 and dfldatadir/gen6328 are loaded