= Xios in ORCHIDEE = The interface for XIOS is implemented in ORCHIDEE/trunk rev [https://forge.ipsl.jussieu.fr/orchidee/changeset/1788 rev 1788] for usage of ORCHIDEE offline.[[BR]] This page is about the on going work with XIOS in ORCHIDEE. The work with XIOS is done directly in the trunk of ORCHIDEE. Here are some notes during implementation and testing phase. Note ! All tests for now have only been done with ORCHIDEE offline mode. The implementation in ORCHIDEE will need to change when adapting to run with LMDZ. = Validation = The implementation of XIOS interface in ORCHIDEE is uder validation work. See details here : [wiki:Branches/Xios/Validation] = How to install ORCHIDEE with XIOS : at obelix = Test done with ORCHIEE rev 2469, LMDZ5 rev 2168 and XIOS branchs/xios-1.0 rev 519. * Install configuration ORCHIDEE_trunk or LMDZOR_v6. * Adapt compile options:[[BR]] 1) Change to have modeles/ORCHIDEE/arch/arch-ifort_LSCE.path as follow: {{{ NETCDF_LIBDIR="/usr/local/install/netcdf-4.3.2p/lib -lnetcdff -lnetcdf -L/usr/local/install/hdf5-1.8.9p/lib -lhdf5_hl -lhdf5 -lhdf5 -lz -lcurl -lstdc++" NETCDF_INCDIR=/usr/local/install/netcdf-4.3.2p/include IOIPSL_INCDIR=${ORCHDIR}/../../lib IOIPSL_LIBDIR=${ORCHDIR}/../../lib XIOS_INCDIR=${ORCHDIR}/../XIOS/inc XIOS_LIBDIR="${ORCHDIR}/../XIOS/lib -lxios" }}} 2) Change in util/AA_make.gdef, the version of netcdf to the following: {{{ #-Q- lxiv8 NCDF_INC = /usr/local/install/netcdf-4.3.2p/include #-Q- lxiv8 NCDF_LIB = -L/usr/local/install/netcdf-4.3.2p/lib -lnetcdff -lnetcdf }}} 3) If coupled mode, change also to have modeles/LMDZ/arch/arch-ifort_LSCE.path as follow: {{{ NETCDF_LIBDIR="-L/usr/local/install/netcdf-4.3.2p/lib -lnetcdff -lnetcdf -L/usr/local/install/hdf5-1.8.9p/lib -lhdf5_hl -lhdf5 -lhdf5 -lz -lcurl" NETCDF_INCDIR=-I/usr/local/install/netcdf-4.3.2p/include IOIPSL_INCDIR=$LMDGCM/../../lib IOIPSL_LIBDIR=$LMDGCM/../../lib ORCH_INCDIR=$LMDGCM/../../lib ORCH_LIBDIR=$LMDGCM/../../lib OASIS_INCDIR= OASIS_LIBDIR= INCA_LIBDIR=$LMDGCM/../INCA/config/lib INCA_INCDIR=$LMDGCM/../INCA/config/inc XIOS_INCDIR=$LMDGCM/../XIOS/inc XIOS_LIBDIR=$LMDGCM/../XIOS/lib }}} * Recreate makefiles with ./ins_make. * Compile for offline configration '''in config/ORCHIDEE_OL with gmake with_xios''' or for coupled configurations as defalut '''in config/LMDZOR_v6 with gmake'''. * '''Launch''' : Tests done for offline with xios in attatched and server mode. * Using libIGCM, add in the main job, before execution : {{{ . /usr/share/Modules/init/ksh module load netcdf/4p }}} = How to install ORCHIDEE with XIOS for offline use : at CURIE = * Extract {{{ > svn co http://forge.ipsl.jussieu.fr/igcmg/svn/modipsl/trunk modipsl > cd modipsl/util Add extraction of XIOS by adding following line in mod.def : (XIOS revision 446, 455, 471 and 480 have been tested successfully with ORCHIDEE, revision 452 and 472 do not work) : #-C- ORCHIDEE_trunk XIOS/trunk 480 12 XIOS modeles > ./model ORCHIDEE_trunk }}} * Adapt compiling with netcdf parallel. Since revision [1875] there is no need to modify makefiles or arch files. Only loading parallel netcdf and hdf5 is needed. At curie, following has to be done(this is the default case using the [http://forge.ipsl.jussieu.fr/igcmg_doc/wiki/DocBenvBtgcc#HowtoinstallyourenvironmentonTGCC proposed IPSL environment]): {{{ module unload netcdf hdf5 module load netcdf/4.2_hdf5_parallel module load hdf5/1.8.9_parallel }}} * Compile using fcm {{{ > cd ../config/ORCHIDEE_OL > gmake with_xios }}} * Lance * The executable can now be used with XIOS or with IOIPSL or with both at the same time. * Activate XIOS by adding XIOS_ORCHIDEE_OK=y * Deactivate IOIPSL by setting WRITE_STEP=0 and STOMATE_HIST_DT=0 in run.def * This executable must be used with netcdf and hdf5 parallel, add in the job just before execution following lines(done in libIGCM since tag v2.2) : {{{ module unload netcdf hdf5 module load netcdf/4.2_hdf5_parallel module load hdf5/1.8.9_parallel }}} * The executable can be launched in * attached mode : this means that the server XIOS is not used. You only launches the orchidee_ol executable. * using the server xios : in this case, one or several cores are dedicated to the server and the reste to orchiee_ol. * Lance using libIGCM * xios files can not have suffix _xios if IOIPSL output is activated as standard. Instead change to use .xios * if using multiple_file, add min_digits="4" on the same line in iodef.xml. This can be kept even for one_file : {{{ }}} * Rebuild * If using '''file_definition type="multiple_file"''' then rebuild must be done. Use this rebuild : /ccc/cont003/home/dsm/p86ipsl/rebuild/src_X64_CURIE/modipsl_v2_2_2_netcdf4.2/bin/rebuild. For exemple : {{{ /ccc/cont003/home/dsm/p86ipsl/rebuild/src_X64_CURIE/modipsl_v2_2_2_netcdf4.2/bin/rebuild -o sechiba_history_xios.nc sechiba_history_xios_* }}} * If using '''file_definition type="one_file"''' then no rebuild is needed. * Visulize netcdf output * ferret : compatible to all netcdf versions, IOIPSL and XIOS output * nco : all versions adapted for IOIPSL output. To vizualize XIOS output, you need to have loaded netcdf and hdf5 parallel. * cdo : all versions adapted for IOIPSL output. To vizualize XIOS output, you need to have loaded netcdf and hdf5 parallel and also a newer version of cdo. * Load following modules to use interactivly these tools : {{{ module unload netcdf hdf5 cdo module load netcdf/4.2_hdf5_parallel module load hdf5/1.8.9_parallel module load cdo/1.6.0 }}} = How to install LMDZ-ORCHIDEE with XIOS : at CURIE = * Extract {{{ > svn co http://forge.ipsl.jussieu.fr/igcmg/svn/modipsl/trunk modipsl > cd modipsl/util }}} * Extract configuration LMDZOR_v6 {{{ > ./model LMDZOR_v6 }}} * Load as for orchidee offline : {{{ module unload netcdf hdf5 module load netcdf/4.2_hdf5_parallel module load hdf5/1.8.9_parallel }}} * Compile (by default with XIOS and for hybrid mpi_omp mode) {{{ cd modipsl/config/LMDZOR_v6 gmake }}} * Lance Use all xml files found in modipsl/modeles/LMDZ/DefLists and modeles/ORCHIDEE/src_xml except iodef.xml. The configuration LMDZOR_v6 is under construction. The experiment LMDZOR/clim is half-adapted.... To be finished. = For old versions of LMDZ : LMDZOR with OpenMP = Do as above but : * take revision 1729 of LMDZ and update some files : {{{ cd modipsl/modeles/LMDZ svn update -r 1905 libf/phylmd/mod_synchro_omp.F90 svn update -r 1905 libf/phylmd/mod_phys_lmdz_omp_transfert.F90 svn update -r 1986 bld.cfg svn update -r 1986 makelmdz_fcm svn update -r 1986 arch/arch-X64_CURIE.path }}} * Add initialization of XIOS in LMDZ/libf/dyn3dpar/mod_const_mpi.F90, change the subroutine Init_MPI to the following : {{{ SUBROUTINE Init_mpi #ifdef CPP_IOIPSL USE IOIPSL #else ! if not using IOIPSL, we still need to use (a local version of) getin USE ioipsl_getincom #endif #ifdef CPP_XIOS USE XIOS #endif IMPLICIT NONE #ifdef CPP_MPI INCLUDE 'mpif.h' #endif INTEGER :: ierr INTEGER :: thread_required INTEGER :: thread_provided LOGICAL :: xios_orchidee_ok CHARACTER(len=*),PARAMETER :: id="client" !! Id for initialization of ORCHIDEE in XIOS xios_orchidee_ok=.FALSE. CALL getin('XIOS_ORCHIDEE_OK ',xios_orchidee_ok) WRITE(*,*)'In LMDZ: xios_orchidee_ok=',xios_orchidee_ok #ifdef CPP_MPI !$OMP MASTER thread_required=MPI_THREAD_SERIALIZED CALL MPI_INIT_THREAD(thread_required,thread_provided,ierr) IF (thread_provided < thread_required) THEN PRINT *,'Warning : The multithreaded level of MPI librairy do not provide the requiered level', & ' in mod_const_mpi::Init_const_mpi' ENDIF IF ( xios_orchidee_ok) THEN #ifdef CPP_XIOS PRINT*,'Before calling xios_initialize in LMDZ' CALL xios_initialize(id,return_comm=COMM_LMDZ) #endif ELSE COMM_LMDZ=MPI_COMM_WORLD END IF MPI_REAL_LMDZ=MPI_REAL8 !$OMP END MASTER #endif END SUBROUTINE Init_mpi }}} * before compiling, add in LMDZ/arch/arch-X64_CURIE.fcm so that {{{ %PROD_FFLAGS -O3 -fp-model precise %OMP_FFLAGS -openmp -openmp-threadprivate compat %OMP_LD -openmp -openmp-threadprivate compat }}} * in ORCHIDEE, add openmp compiling directives in ORCHIDEE/arch/arch-X64_CURIE.fcm {{{ %OMP_FFLAGS -openmp -openmp-threadprivate compat %OMP_LD -openmp -openmp-threadprivate compat }}} = Implementation = In the definition phase using IOIPSL, in the call to histdef, an operator is declared on the field. Using XIOS, some of these operators will be set in the xml file (which contains the definition phase) and some will be applied to the variables in the call to xios_orchidee_send_field.[[BR]] || operation in histdef || terme in xios_orchidee_send_field || operation in xml [[BR]] || flux_op || one_day/dt(*) || - [[BR]] || flux_scinsec || 1/dt (!**) || - [[BR]] || ave || - || average [[BR]] || once || - || once [[BR]] || tmaxcels || - || maximum [[BR]] || tmincels || - || minimum [[BR]] || sumscatter || - || accumulate [[BR]] (*) one_day/dt must be replaced by 48.0 to obtain excactly the same results in double precision between XIOS and IOIPSL output files, for instantaneous output frequency. [[BR]] (!**) 1/dt must be replaced by 0.0005555556 to obtain de same results between XIOS and IOIPSL output files, even in float precision, for instantaneous output frequency. This value is temporary commited in revision [1788] until revision [1877]. [[BR]]