{{{
#!html
<h1>Working on TGCC</h1>
}}}
----
[[PageOutline(1-3,Table of contents,,numbered)]]

# TGCC presentation #

http://www-hpc.cea.fr/en/complexe/tgcc.htm

# TGCC's machines and file systems #

[[Image(wiki:DocBenvBtgcc:TGCC_avril2016.jpg, 360px)]]

# How to install your environment on TGCC #

 * More information on the open-access website: http://www-hpc.cea.fr/en/complexe/tgcc.htm
 * Online access to the machines' users manual (you will need a TGCC login and password): https://www-tgcc.ccc.cea.fr/ . 
   * Online access to technical issues and news : https://www-tgcc.ccc.cea.fr/en/news/index.html
 * The available TGCC's machine is currently '''curie''' (Bull Sandybridge).
 * Note: the '''$HOME/.snapshot''' directory contains hourly, daily, and weekly backups of your $HOME files.

[[NoteBox(note,It is important to take the time to install a comfortable and efficient environment., 600px)]]

We suggest the user to use the p86ipsl login's environment (in bash) as an example (copy the `~p86ipsl/.bashrc_curie` and the `~p86ipsl/.bash_profile` files in your home). You can complete the .bashrc file to create your favorite environment (alias, module load ...) 
{{{
#!sh
ryyy999@curie: cp ~p86ipsl/.bashrc_curie  ~/.
ryyy999@curie: cp ~p86ipsl/.bash_profile ~/.
}}}

WARNING : if you have a ~/.profile file, it's better to remove it to avoid any problem during the execution of a simulation with libIGCM

In this environment is specified:
 * the path to the compiler tool `fcm` and to the `rebuild` tool which recombines output files from a parallel model:
{{{ 
export PATH=~p86ipsl/fcm/bin:~p86ipsl/rebuild/src_X64_CURIE/modipsl_v2_2_2_netcdf4.2/bin/rebuild:$PATH
}}}
 * the load of modules giving access to computing or post processing libraries and tools needed on our platform (done in ~p86ipsl/.atlas_env_netcdf4.3.3.1_hdf5_parallel_curie_ksh). The revision numbers may change, currently (June 2015) following modules are loaded for computing :
{{{
module load ghostscript/9.04
module load ferret/6.9.3
module load gsl/1.14
module load netcdf/4.3.3.1_hdf5_parallel
module load cdo/1.6.7
module load imagemagick/6.7.4
module load nco/4.4.8
}}}

# Project and computing needs #
 * To find out the computing time used by the projects you are involved in (daily update):
{{{
#!sh
ryyy999@curie: ccc_myproject
}}}
 * When you will create a job you need to specify in the header the project from which you will use computing time:
{{{
#!sh
#MSUB -A genxxx
}}}

# About file systems #

## Quotas ##
To check the available and used storage capacities of `HOME`, `SCRATCH`, `CCCWORKDIR` and `CCCSTOREDIR`:
{{{
#!sh
ryyy999@curie: ccc_quota
}}}

On the curie machine this command will also return the space used by scratch (a specificity of the curie machine).

## SCRATCHDIR ##
The `$SCRATCHDIR`directory is often cleaned and only files that are less than 40 days are stored.

## CCCWORKDIR ##
The `$CCCWORKDIR` directory corresponds to the `$WORKDIR` directory on curie. It is large but its content is not backed up.

## CCCSTOREDIR ##
To manipulate the files in /ccc/store a few commands are useful: 
{{{
#!sh
# Demigrate a list of files on CCCSTOREDIR, see also "ccc_hsm -h"
ccc_hsm get $CCCSTOREDIR/FILE1 $CCCSTOREDIR/FILE2 ...

# Demigrate recursively the files from a CCCSTOREDIR directory, see also "ccc_hsm -h"
ccc_hsm get -r $CCCSTOREDIR/DIRECTORY

# Find out the used space on CCCSTOREDIR
cd $CCCSTOREDIR ; find . -printf "%y %s %p \n"  | \
     awk '{ SUM+=$2 } END {print "SUM " SUM/1000000 " Mo " SUM/1000000000 " Go" }'

# or use --apparent-size with du :
du -sh --apparent-size
}}}


## ccc_home command to know directory complete pathname ##

ccc_home could help you to find directory complete pathname for an other user or for you .
{{{
>ccc_home -h
ccc_home: Print the path of a user directory (default: home directory).
usage: ccc_home [ -H | -s | -t | -W | -A | -G | -a] [-u user]
                [-h, --help]

 -H, --home         :  (default) print the home directory path ($HOME)
 -s, -t, --scratch  :  print the scratch directory path   ($SCRATCHDIR)
 -W, --cccwork      :  print the CCC work directory path  ($CCCWORKDIR)
 -A, --cccstore     :  print the CCC store directory path ($CCCSTOREDIR)
 -G, --cccgenostore :  print the CCC genostore directory path ($CCCGENOSTOREDIR)
 -a, --all          :  print all paths
 -u user            :  show paths for the specified user instead of the current user
 -h, --help         :  display this help and exit

> ccc_home -A -u ryyy999   
/ccc/store/cont003/dsm/ryyy999

}}}
## Storage spaces available from DODS ##
To store a file for the first time on dods, you must ask for dods write access by mail to the TGCC hotline access : `hotline.tgcc@cea.fr`. 
{{{
On Curie : 
/ccc/work/cont003/dods/public/login 
/ccc/store/cont003/dods/public/login 

On a server web 
http://dods.extra.cea.fr/work/
http://dods.extra.cea.fr/store/
}}}
 
# Specific file systems for CMIP6 #

For gencmip6 project, and only for it, 3 more file systems and 4 more directories are available. Phase 1 have been installed in april 2016. Phase 2 and Phase 3 will come later in 2017 and 2018.

To use them, in interactive mode, you have to do : {{{module load datadir/gencmip6}}}. 

Since libIGCM_v2.8.1, if you set your project to gencmip6/devcmip6, they are automatically used in place of usual HOME, CCCWORKDIR, CCCSTOREDIR and SCRATCHDIR :  {{{module switch dfldatadir dfldatadir/gencmip6}}} called from libIGCM.

## GENCMIP6_HOME ##
 * 50 TB 
 * gencmip6 group quota
 * dedicated to sources and scripts 

## GENCMIP6_CCCWORKDIR ##
 * 2.5 PB in phase 1, 7 PB in phase 2
 * gencmip6 group quota
 * dedicated to small output files (ATLAS, MONITORING)
 * (soon) available through dods/thredds

## GENCMIP6_CCCSTOREDIR ##
 * 2.5 PB in phase 1, 7 PB in phase 2 and 14 PB on tape in phase 3
 * gencmip6 group quota
 * dedicated to large (more than 1GB) output files (Output, Analyse)
 * (soon) available through dods/thredds
 * linked with HSM (tapes) 

## GENCMIP6_SCRATCHDIR ##
 * same file system as GENCMIP6_CCCWORKDIR
 * used during batch execution (RUN_DIR) and erased at the end of the execution

# End-of-job messages #

To receive the end-of-job messages sent by the job itself: end of simulation, error,... you must specify your address in the $`HOME/.forward` file.

# Simulation outputs #

Final simulation outputs are stored in `$CCCSTOREDIR/IGCM_OUT` and on `$CCCWORKDIR/IGCM_OUT` regarding the `ATLAS` and `MONITORING` directories.

The dods servers on TGCC are available via:  `dods.extra.cea.fr/store`  (files such as `Analyse/TS` and `Analyse/SE`) and `dods.extra.cea.fr/work` for `ATLAS` and `MONITORING`.

Since June 2014, thredds server are also available and will replace dods server early 2015:
  * http://esgf.extra.cea.fr/thredds, click on DODSSTORE, click on your login or directly : http://esgf.extra.cea.fr/thredds/catalog/DODSSTORE/YOURLOGIN/catalog.html and ATM (or an other component)  for `Analyse` files (TS or SE)
  * http://esgf.extra.cea.fr/thredds, click on DODSWORK, click on your login or directly : http://esgf.extra.cea.fr/thredds/catalog/DODSWORK/YOURLOGIN/catalog.html for `ATLAS` and `MONITORING`

# About password #

ccc_password_expiration helps you to know expiration date of your password. Currently password have to be changed one time per year.
{{{
 > ccc_password_expiration
Password for xxxxx@USERS-CCRT.CCC.CEA.FR: PPPPPPPPPP
Your password will expire in 70 days on Fri Nov 22 08:42:59 2013
 > ccc_password_expiration -h
Usage: ccc_password_expiration [username[@realm]]

}}}


# The TGCC's machines # 
 * [wiki:DocBenvBtgccAcurie Curie]