WikiPrint - from Polar Technologies

Working on the Irene machine

Last update 05/07/2018

1. Introduction

2. Job manager commands

3. Suggested environment

Before working on Irene you need to prepare your environment. This is important to do before compilation to ensure the use of same modules as done by libIGCM running environment. We propose you 2 files which you can copy from the home igcmg. The first one called bashrc will source the second called bashrc_irene. Copy both files to your home, rename them by adding a dot as prefix. You can add personal settings in your .bashrc_irene. Do as follow: 

cp /ccc/cont003/home/igcmg/igcmg/MachineEnvironment/irene/bashrc ~/.bashrc
cp /ccc/cont003/home/igcmg/igcmg/MachineEnvironment/irene/bashrc_irene ~/.bashrc_irene

The .bashrc will source your own .bashrc_irene which must be in your home.

After re-connexion or source of .bashrc_irene, check your loaded modules for intel, netcdf, mpi, hdf5 needed for the compilation: 

module list 
Currently Loaded Modulefiles:
 1) ccc                                10) mkl/17.0.6.256(default)                      19) flavor/hdf5/parallel              
 2) datadir/own(default)               11) flavor/buildcompiler/intel/17(default)       20) netcdf-c/4.6.0(default)           
 3) dfldatadir/own(default)            12) intel/17.0.6.256(default)                    21) netcdf-fortran/4.4.4(default)     
 4) licsrv/intel                       13) hwloc/1.11.3(default)                        22) hdf5/1.8.20(default)              
 5) c++/intel/17.0.6.256(default)      14) feature/openmpi/mpi_compiler/intel(default)  23) feature/bridge/heterogenous_mpmd  
 6) c/intel/17.0.6.256(default)        15) feature/openmpi/net/mxm(default)             24) nco/4.6.0(default)                
 7) fortran/intel/17.0.6.256(default)  16) .tuning/openmpi/2.0(default)                 25) cdo/1.7.2rc6(default)             
 8) feature/mkl/lp64                   17) flavor/buildmpi/openmpi/2.0                  26) ghostscript/9.19(default)         
 9) feature/mkl/sequential             18) mpi/openmpi/2.0.2                            27) ferret/7.2(default)

The modules are specified in the file /ccc/cont003/home/igcmg/igcmg/MachineEnvironment/irene/env_irene which is sourced in bashrc_irene. The same file env_irene is sourced in libIGCM.

--> Be careful this environment can be update during next weeks according to TGCC recommendations

Create ~/.forward file in your main home containing only one line with your email address to receive emails from libIGCM.

4. File system

You have a main home where you arrive when connecting to irene, called "home de connexion" by the TGCC. You also have a home, a storedir, a workdir, a scratchdir by project. For example if you are working with project gen2201 and gen2212 you will have all following directories: 

/ccc/cont003/home/***/login                  # connexion home, where ***=your lab (lsce, ipsl, etc..)

/ccc/cont003/home/gen2201/login     # use it for sources, regular snapshot are in .snapshot
/ccc/cont003/home/gen2212/login

/ccc/store/cont003/gen2201/login
/ccc/store/cont003/gen2212/login

/ccc/work/cont003/gen2201/login      
/ccc/work/cont003/gen2212/login

/ccc/scratch/cont003/gen2201/login
/ccc/scratch/cont003/gen2212/login

IMPORTANT : Check that you have read and write access to above directories (for your projects). Contact TGCC hotline if it is not the case.

On the SCRATCH space any files that stays 60 days without being read or modified will be purged(deleted), as well as any directory that remains empty for 30 days.

After connexion to irene, load your project environment as default using the module dfldatadir. For example if you will work on the project gen2201, do following (we suggest you to add the command into your .bashrc_irene): 

module switch dfldatadir dfldatadir/gen2201

By changing the dfldatadir, the variables $CCCHOME, $CCCWORKDIR, $CCCSTOREDIR and $CCCSCRATCHDIR point to the corresponding project directories. $HOME is always the main connexion home.

You will also have new environment variables to access working directories : 

GEN2201_ALL_CCCSCRATCHDIR=/ccc/scratch/cont003/gen2201/gen2201
GEN2201_CCCWORKDIR=/ccc/work/cont003/gen2201/login
GEN2201_ALL_HOME=/ccc/cont003/home/gen2201/gen2201
GEN2201_CCCSTOREDIR=/ccc/store/cont003/gen2201/login
GEN2201_CCCSCRATCHDIR=/ccc/scratch/cont003/gen2201/login
GEN2201_ALL_CCCWORKDIR=/ccc/work/cont003/gen2201/gen2201
GEN2201_HOME=/ccc/cont003/home/gen2201/login
GEN2201_ALL_CCCSTOREDIR=/ccc/store/cont003/gen2201/gen2201

If you previously worked at curie and your directories were in /cont003/dsm/login you will now find your data in a specific new project file system "dsmipsl". We recommend to move your data in your genci project file system. The TGCC hotline can help you if you want.

5. How the storage project is set by libIGCM

When you use libIGCM it is recommended to dedicate one modipsl/libIGCM to one project allocation. By default, the output folders IGCM_OUT will be created in the directories $CCCSCRATCHDIR, $CCCWORKDIR and $CCCSTOREDIR corresponding to the project used in the main job. It is important that the same project is used in the post-processing jobs in libIGCM.

If you need to use another project for the computing than the storage, it is possible to set the variable DataProject in config.card UserChoices section, for example DataProject=gen6328, read more here. This project will be used for all output directories for the computing job and post-processing jobs even if they have another project for computing in the headers. The variable DataProject can also be used if you work with different project allocations in the same modipsl. Only exception (harmless) is the first RUN_DIR folder which is always created in the $CCCSCRATCHDIR corresponding to the dfldatadir loaded in when submitting main job. When the job resubmits itself, the RUN_DIR will be in the same project space as the rest of the output.

6. A figure to understand file systems used on Irene

No image "TGCC_2018_irene.jpg" attached to DocBenvBtgcc

7. Configurations

Following configurations are technically ready (scientific validation in progress) to be used at Irene. On 2018/8, performances are 1.2 times faster than curie.

8. How to use old configurations

--> if you receive an svn error with this specific command, extract a new modipsl and copy these two files from it