Working on the Irène machine

Update 27/06/18

1. On-line users manual

• ​https://www-tgcc.ccc.cea.fr/docs/irene.info.pdf (you will need a TGCC login and password)

2. Job manager commands

• ccc_msub job -> submit a job
• ccc_mdel ID -> kill the job with the specified ID number
• ccc_mstat -u login -> display all jobs submitted by login
• ccc_mpp -> display all jobs submitted on the machine. ccc_mpp -n to avoid colors.
• ccc_mpp -u $(whoami) ->display your jobs.

3. Irene environment and files system

• Before working on Irène you need to prepare your environment

  cd 
  cp /ccc/cont003/home/igcmg/igcmg/MachineEnvironment/irene/bashrc .bashrc
  cp /ccc/cont003/home/igcmg/igcmg/MachineEnvironment/irene/bashrc_irene .bashrc_irene
  
  after login or source .bashrc_irene: 
  module list 
  Currently Loaded Modulefiles:
   1) ccc                       8) fortran/intel/17.0.4.196                15) feature/openmpi/mpi_compiler/intel(default)  22) netcdf-fortran/4.4.4(default)     
   2) datadir/drf               9) feature/mkl/lp64                        16) feature/openmpi/net/mxm(default)             23) hdf5/1.8.20(default)              
   3) datadir/own(default)     10) feature/mkl/sequential                  17) .tuning/openmpi/2.0(default)                 24) feature/bridge/heterogenous_mpmd  
   4) dfldatadir/own(default)  11) mkl/17.0.4.196                          18) flavor/buildmpi/openmpi/2.0                  25) nco/4.6.0(default)                
   5) licsrv/intel             12) flavor/buildcompiler/intel/17(default)  19) mpi/openmpi/2.0.2                            26) cdo/1.7.2rc6(default)             
   6) c++/intel/17.0.4.196     13) intel/17.0.4.196                        20) flavor/hdf5/parallel                         
   7) c/intel/17.0.4.196       14) hwloc/1.11.3(default)                   21) netcdf-c/4.6.0(default)                      
  
  

  --> Be carefule this environment can be update during next weeks according to TGCC recommendations

• On Irene you have a storedir, a workdir, a scratchdir by project. For example if you are working with project gen2201 and gen2212 you will have all these directories

  /ccc/work/cont003/gen2201/login
  /ccc/work/cont003/gen2212/login
  
  /ccc/store/cont003/gen2201/login
  /ccc/store/cont003/gen2212/login
  
  /ccc/scratch/cont003/gen2201/login
  /ccc/scratch/cont003/gen2212/login
  
  

Before working you need to know with which project you are, and load its environment

module load datadir/gen2201 

you can put this command in your .barshrc_irene file

vi .bashrc_irene
gen2201 ()  { module load datadir/gen2201; }

You will have new environment variables to access working directories

GEN2201_ALL_CCCSCRATCHDIR=/ccc/scratch/cont003/gen2201/gen2201
GEN2201_CCCWORKDIR=/ccc/work/cont003/gen2201/login
GEN2201_ALL_HOME=/ccc/cont003/home/gen2201/gen2201
GEN2201_CCCSTOREDIR=/ccc/store/cont003/gen2201/login
GEN2201_CCCSCRATCHDIR=/ccc/scratch/cont003/gen2201/login
GEN2201_ALL_CCCWORKDIR=/ccc/work/cont003/gen2201/gen2201
GEN2201_HOME=/ccc/cont003/home/gen2201/login
GEN2201_ALL_CCCSTOREDIR=/ccc/store/cont003/gen2201/gen2201

• Default compiler is intel/17.0.6.256 with default librairy MPI mpi/openmpi/2.0.4. WARNING : there is a bug in the compiler intel-17.0.4.196
• Computing nodes: the nodes of partition skylake have 48 cores each, which is 3 times more than the computing nodes from the standard partition of Curie;
• Filesystem accesses MUST be explicit: from the login nodes, you will see the WORK, SCRATCH, STORE spaces as you probably are used to. However, when submitting any job through ccc_msub or ccc_mprun, you must specify -m work, -m scratch, -m store, or combine them like in -m work,scratch; this constraint has the advantage that your jobs won't be supended if a filesystem you don't need becomes unavailable;
• Compute nodes are diskless, meaning that /tmp is not hosted on a local hard drive anymore, but on system memory instead. It offers up to 16 GB (compared to 64 GB on Curie). Please note that any data written to it is reduces the size of the memory that remains available for computations. In our case it change the number of core use for post-treatment like pack_output.
• The default time limit for a job submission is 2hours (7200s) contrary to 24h (86400s) on curie
• The SCRATCH space purge policy has been simplified: any file that stays 60 days without being read or modified will be purged, as well as any directory that remains empty for 30 days; the purge policy does not depend on disk occupation threshold any longer

4. How working with last LMDZOR_v6 or last LMDZORINCA_v6 ?

• install your environment (see below)
• XIOS : After download :update arch XIOS for IRENE

  cd modipsl/modeles/XIOS/arch 
  svn update
  

  • then you can work as on curie

5. Use an old configuration from Curie

• you need to update files AA_make.gdef and w_h_i in util directory

  cd modipsl/util 
  svn update AA_make.gdef
  svn update w_h_i
  

• you need to change your version of libIGCM

  cd modipsl
  mv libIGCM libIGCM_curie 
  svn co -r 1454 svn://forge.ipsl.jussieu.fr/libigcm/svn/trunk/libIGCM 
  

• you need to download arch files for IRENE for LMDZ, ORCHIDEE, INCA

  cd modipsl
  mkdir modele_arch
  cd modele_arch
  svn co svn://forge.ipsl.jussieu.fr/orchidee/trunk/ORCHIDEE/arch/ ORCHIDEE_arch
  svn co svn://forge.ipsl.jussieu.fr/inca/trunk/INCA5/arch/  INCA_arch
  svn co svn://svn.lmd.jussieu.fr/LMDZ/LMDZ6/branches/IPSLCM6.0.15 LMDZ_arch
  svn co svn://forge.ipsl.jussieu.fr/ioserver/svn/XIOS/branchs/xios-2.5 XIOS_arch
  mv ORCHIDEE_arch/*IRENE* ../modeles/ORCHIDEE/arch/.
  mv INCA_arch/*IRENE* ../modeles/INCA/arch/.
  mv LMDZ_arch/*IRENE* ../modeles/LMDZ/arch/.
  mv XIOS_arch/*IRENE* ../modeles/XIOS/arch/.
  

• if you are working with INCA you need a file pre_proc_X64_IRENE.x

  cd modipsl/modeles/INCA
  cp pre_proc_X64_CURIE.x pre_proc_X64_IRENE.x 
  

• create a new makefile

  cd modipsl/config/***
  mv Makefile Makefile_curie 
  ../../util/ins_make 
  gmake clean 
  

• now you can work as on Curie