Changes between Version 7 and Version 8 of Documentation/UserGuide/MinimizeDiscUsage


Ignore:
Timestamp:
2020-02-28T14:23:09+01:00 (4 years ago)
Author:
luyssaert
Comment:

--

Legend:

Unmodified
Added
Removed
Modified
  • Documentation/UserGuide/MinimizeDiscUsage

    v7 v8  
    1 = How to reduce the disc space usage = 
    2 This sites explains how to reduce the disc usage when running simulations with libIGCM on obelix. It concerns primarily simulations which are long and on a large spatial domain. 
     1= How to reduce the disk space usage = 
     2Author: D. Goll[[BR]] 
     3Last revision: 2020/02/28, S. Luyssaert[[BR]] 
    34 
    4 == Background == 
    5 The libIGCM configuration is designed to ensure high traceability, which means it store lots of information for each simulated model year.  
     5== Objective == 
     6This item should help you to: (1) correctly use shared disk, (2) avoid excessively large output files, and (3) reduce size of data storage. Although correct disk usage is most pressing on obelix because this is a relatively small server shared by many users, these recommendation can be applied on all computing infrastructure as there is no use in wasting disk space. Although some of these recommendations hardly take any time and could be applied even for very small simulations, they are strongly recommended for long simulations with a large spatial domain and/or a high spatial/temporal resolution of the output files. 
    67 
    7 == 1. Before you run the model: avoid high disc usage by tailoring the XIOS output to your specific needs == 
    8 ORCHIDEE writes out many variables which you might not need. In addition, ORCHIDEE writes out redundant variables (for example variables aggregated to grid level AND on PFT level).  
     8== Correct use of shared disk == 
     9Servers and storage facility are shared among tens (Obelix) to thousands (Irene) of users. The way you use this shared facilities will affect others and thus also the other way around. Most frustration by using these shared facilities comes from your simulation that crashes because others filled the disks with their results.  
     10* Never copy driver files from the repositories to your personal folder use libIGCM to ensure the model uses the correct [https://forge.ipsl.jussieu.fr/orchidee/wiki/Documentation/UserGuide/AddNewInputFiles correct] input files. 
     11* If your simulation experiments require large input files that are not on the repository yet discuss with your supervisor or the ORCHIDEE-team what should be done to move those files into the repository. 
     12* If the latter is not possible, then don't store different versions of the same files on the shared disks. 
     13* Learn how to correctly use the DEVT, TEST and PROD settings in the config.card. This by itself will not really save disk space but it will help to automatically make space if some of the disks fill up. 
     14    
     15  
     16== Avoid excessively large output files == 
     17Better to prevent than to cure: don't tell libIGCM to write output files you will never use. Few users require all the output files for the objectives of their simulation experiments. You can simply tell libIGCM not to produce certain output files in the COMP/stomate.card and the COMP/sechiba.card. Users not interested in stomate output can change the default settings in the COMP/stomate.card to NONE. The NONE ensures the file is not produced. Applying this setting will take you few seconds but may help you saving lots of time in speeding up your simulations. In addition to making better use of the storage facilities this will also speed up the simulations themselves as input and output operations can take up a substantial share (5 to 20%) of the computing time.  
     18{{{ 
     19output_level_stomate_history = NONE 
     20output_level_stomate_ipcc_history = NONE 
     21output_level_stomate_history_4dim = NONE 
     22}}} 
     23[[BR]] 
     24In the same COMP/stomate.card and COMP/sechiba.card, the user has to specify the temporal resolution of the output files. Daily files take 365 times more memory than annual files. Monthly files still take up 12 times the memory of an annual file. Speed up your simulation and save disk space by selecting the temporal resolution you will actually use. Note that the restart files could be wisely used to enable the user to speed up the model while reducing disk usage. For example: during the spin-up write only those variables that you will check (soil C and N pools) at an annual frequency. Following the spin-up the model will be restarted. These experiments will start from the spin-up but could be using a different temporal resolution and could write more variables to the output files. 
    925 
    10 '''We recommend to avoid writing out variables you know you are not needed and also variables which are redundant.''' 
    11 You can change the default level of output using the keywords in sechiba.card and stomate.card for offline experiences (orchidee.card and stomate.card in coupled experiences). See comments in these files for how to activate/deactivate files, set frequency and level of output for each file.  
     26ORCHIDEE makes use of predefined packages of output variables. If in the COMP/stomate.card or COMP/sechiba.card (orchidee.card and stomate.card in coupled experiments) you ask for output level 5, the model will use the ORCHIDEE/src_xml/file_def_orchidee.xml file to learn which output variables belong to which level. Level 5 implies that all variable with an output level smaller or equal to 5 will be written to the output files. If you are about to launch a large experiment, change the output levels in ORCHIDEE/src_xml/file_def_orchidee.xml such that only variables that will be used in post-processing and reporting are written. This will require some careful thinking about the post-processing prior to launching the simulations and it will also take a bit of time to adjust file_def_orchidee.xml but you will easily offset the invested time by speeding up the simulations. 
     27 
     28Pay special attention to the dimensions of the variables, ORCHIDEE writes out redundant variables (for example, variables aggregated to grid level AND on PFT level and variables with a much finer resolution than needed (for example, light absorbed at every canopy layer). If you have no plans to use these variables, don't write them to to the output files.  
     29 
     30 
    1231 
    1332