How to reduce the disc space usage

This sites explains how to reduce the disc usage when running simulations with libIGCM on obelix. It concerns primarily simulations which are long and on a large spatial domain.

Background

The libIGCM configuration is designed to ensure high traceability, which means it store lots of information for each simulated model year.

1. Before you run the model: avoid high disc usage by tailoring the XIOS output to your specific needs

ORCHIDEE writes out many variables which you might not need. In addition, ORCHIDEE writes out redundant variables (for example variables aggregated to grid level AND on PFT level).

We recommend to avoid writing out variables you know you are not needed and also variables which are redundant. You can change the default level of output using the keywords in sechiba.card and stomate.card for offline experiences (orchidee.card and stomate.card in coupled experiences). See comments in these files for how to activate/deactivate files, set frequency and level of output for each file.

2. After you ran the model: remove files which are not needed

You can reduce the disc space usage to about 15% (in case of O-CN-P) by removing file you are likely never needing.

Restart files

Restart files allow to rerun the simulation from a certain year. This files are needed in case the model stops before it reached the end date and also allows you to rerun certain periods (for example if you need additional variables).

We recommend to only keep restart files for every 10th year or less frequently.

used_run.def

The run.def for each model years is written out. In case you did not change parameters between years it is the same file for every year. In that case a single used_run.def would be sufficient.

We recommend to only keep used_run.def files for the year a change in parameters was introduced.

Debug files

ORCHIDEE writes out files which help to debug the model. In case you are not debugging these files are useless.

We recommend to remove all debug files.

3. After you ran the model: remove the RUN_DIR

In case you specified a RUN_DIR make sure you remove this directory after the simulation has ended. This folder can be very large.

We recommend to remove immediately after the end of the model simulation the RUN_DIR.

4. Example of simple script to remove not needed files

#!/bin/ksh
# this scripts removes Restart and Debug files (NOT used_run.defs)
# the user has to specify the (1) experiment name, (2) the path to the output, (3) the start and end years of the simulation.
# Optional you can adjust the frequency at which you want to keep files

set -e

# which user is running this script
myself=$(whoami)

# ========================================
# use specification:
igcm_out=/home/surface7/${myself}/IGCM_OUT/OL2/PROD/MODEL1/
expid=simulation01
# start year:
syear=1861
# end year:
eyear=2099
# we keep files for every Xth year at this frequency:
frq_keep=10
# ========================================


# ============================================
# 1. remove debug and restart files
# 1.1 set time counter
yr=$syear
(( yr10th = syear + frq_keep ))

# loop over years:
while [[ $yr -le $eyear ]] ; do
    # check if it is a year we keep ...
    if [[ $yr == $yr10th ]] ; then
        echo 'we keep the files for this year=',$yr
        (( yr10th = yr10th + frq_keep ))
    else
    # ... if not we start removing files:
      for stream in SBG SRF OOL ; do
         if [[ $stream == "SBG" ]]; then
            # in SBG there is only one folder ...
            folders="Restart"
         else
            # ... in the other streams we have two dolfers
            folders="Debug Restart"
         fi

         # now check the folders ... :
         for folder in ${folders} ; do
            # ... for files for the current year:
            r1=${igcm_out}/${expid}/${stream}/${folder}
            files=$( ls ${r1}/*${yr}*)
            for file in $files ;do
                # ... and remove them.
                rm -f $file
            done
         done
      done
      # also remove all files from Out
      r1=${igcm_out}/${expid}/Out
      files=$( ls ${r1}/*${yr}*)
      for file in $files ;do
          rm -f $file
      done
    fi
    (( yr = yr + 1 ))
done