This was tested for ORCHIDEE-CN-CAN (r5678 of ORCHIDEE and r5673 of ORCHIDEE_OL) on obelix.

First, look at Nicolas's page.

​http://forge.ipsl.jussieu.fr/orchidee/wiki/Scripts/FluxnetValidation

And then look at the README file in config/ORCHIDEE_OL/ENSEMBLE. And then read this whole page before really starting to create a run.

Be sure you have checked out both CN-CAN modeles/ORCHIDEE and config/ORCHIDEE_OL (Documentation/UserGuide/ORCHIDEEDOFOCOInstall).

Be sure that ioipsl_debug=.FALSE. in modeles/IOIPSL/src/errioipsl.f90. Otherwise, the output files become huge because of the high frequency writes combined with the debug information.

Start from a clean SVN ENSEMBLE install. Notice that ENSEMBLE/Job_ENSEMBLE is the main driver, and it should not be deleted! This is what I refer to when I say "Nicolas's FLUXNET scripts". It will create jobs based on SPINUP/SUBJOB/OOL_SEC_STO/.

I have found the following files are used. Care should be made to make sure that conflicting options are not specified in these files (assuming you are running analytical spinup, sechiba, and stomate):

ENSEMBLE/fluxnet.card
ENSEMBLE/PARAM/run.def
SPINUP/COMP/spinup.card
SPINUP/SUBJOBS/OOL_SEC_STO/COMP/sechiba.card
SPINUP/SUBJOBS/OOL_SEC_STO/COMP/stomate.card

I am uncertain what the priority is. All of the .card files add things onto the end of the run.def, and I believe fluxnet.card takes priority over the others in case of conflict.

As of r6358 (much before, actually, but at least this revision), the CAN branch of config/ORCHIDEE_OL/ENSEMBLE contains a series of fluxnet*card files. These different files have different configurations, and different sites. Choose one that best matches what you want.

cp fluxnet_28sp.card fluxnet.card

As of around r6358, the Python script in the config/ORCHIDEE_OL/MAKE_RUN_DEF folder started generating only orchidee_pft.def_* in a few directories: OOL_SEC_STO_FG1trans, OOL_SEC_STO_FG2, SPINUP, and some others. You should make sure that your PARAM directory has all the run.defs it needs, as for a normal run : from the ENSEMBLE folder (or the folder you copied the ENSEMBLE folder to) cp ../OOL_SEC_STO_FG2/PARAM/* PARAM/

I have noticed that the script will complain if a value is specified in fluxnet.card but not the run.def. It will not complain if a value is specified in run.def and not fluxnet.card. Check the [UserChoices?] and [SubJobParams?] sections of fluxnet.card. Many of the UserChoices? are already in SPINUP/COMP/spinup.card, and many of the SubJobParams? are in the run.def. It seems that the scripts make decisions based on what is in fluxnet.card, so this should typically take precedence.

Before we get to some specifics, let's create the jobs.

cd config/ORCHIDEE_OL/ENSEMBLE
vi config.card

Change the following lines (on obelix...on Irene, the ARCHIVE line should be fine):

JobName=FLUXNET
  ARCHIVE=/home/scratch01/$LOGIN

then create the job scripts

../../../libIGCM/ins_job

this creates Job_FLUXNET. Notice that this job will pull from the SPINUP directory as well. ins_job used to create Job files in every directory, but that functionally changed a while ago. Therefore, the following is now necessary (OOL_SEC_STO because we will run a job with sechiba and stomate).

cd ../SPINUP
../../../libIGCM/ins_job
cd SUBJOB/OOL_SEC_STO/
../../../../../libIGCM/ins_job
cd ../../../ENSEMBLE

Now edit the Job file. Notice that this is the Job file that is copied to all the subjobs when they run, so if you want them to run on a different queue (I use the long queue on obelix, as 500 years can take more than 12 hours), you should do that here. I also modify the run directory so I know where the jobs are running and can go to that directory easily if needed.

vi Job_FLUXNET
(change RUN_DIR_PATH=/home/scratch01/mmcgrath/RUN_DIR)
(change JobType=DEV if you are not sure this will work)
mkdir /home/scratch01/mmcgrath/RUN_DIR

Now change the options for the sites to run against.

vi fluxnet.card

Best to run a small test with a single site. If you are running with age classes (not recommended) or any number of PFTs other than the standard 13 in ORCHIDEE, you will need to change the following:

  NbPFTs= 13
  Groups= ( TEST )
  TEST =	   ( BR-Sa3 , BR-Sa3_2000-2003.nc , 2000 , 4 , 0,1,0,0,0,0,0,0,0,0,0,0,0 ) 

If you have 13 PFTs, you can add the above lines just as they are. COMMENT OUT ANY OTHER GROUPS LINES. Else, when you submit the job, you will launch a run over all of the sites in groups, and you have to cancel them one at a time.

The length of the spinup also matters. I use the following for the moment (in fluxnet.card, also in SPINUP/spinup.card)

n_iter=1
duree_inistomate=1
duree_sechiba=500
duree_final=1

All of the other duree values I set to 0. This launches a simulation over one loop of the forcing file, then 500 years (regardless of the length of the forcing file), and then one final loop for analysis.

The section in the fluxnet.card with [SubJobParams?] deserves special mention. As of a recent version of CAN, the run.def has been restructured to include two files: orchidee.def, orchidee_pft.def. This makes the run.def much neater and matches what is done in the coupled simulations. However, the Job_ENSEMBLE script attempts to change some variables in the run.def that fall under the [SubJobParams?] section. To do this, it looks at the actual run.def file, not any included file. If it does not find a line in the run.def corresponding to the lines in [SubJobParams?], it will crash. So make sure all the lines you specific under [SubJobParams?] in fluxnet.card also explicitly appear in the PARAM/run.def file.

Similarly, values found in fluxnet.card [UserChoices?] seem to be required in SPINUP/COMP/spinup.card, else it crashes.

The following directories are used in the runs (from what I can tell):

ENSEMBLE/PARAM/
SPINUP/COMP/
SPINUP/SUBJOBS/OOL_SEC_STO/COMP/

We need to make sure all of the following lines are commented out (or do not exist in the orchidee_pft.def), since the script will change the vegetation for each site by adding lines at the end of the run.def and if these lines are present they will override them:

SECHIBA_VEG__01=0.0769230769231
...
SECHIBA_VEGMAX__01=0.0769230769231
  ...

Make sure the following line is in the orchidee.def.

IMPOSE_VEG=y

Also confirm that IMPOSE_VEG is not set in another of the files above (e.g., SPINUP/COMP/spinup.card,SPINUP/SUBJOB/OOL_SEC_STO/COMP/sechiba.card). Another line that may be needed is:

ATM_CO2 =_AUTO_: DEFAULT = 350.

In some versions of the run.def, no DEFAULT value is given, but the .driver expects a default and will crash if it's not there.

The scripts expect some variables in SPINUP/SUBJOBS/OOL_SEC_STO/COMP/sechiba.card, and will crash if you don't have them. It tries to change them (perhaps based on fluxnet.card) and gives up if it doesn't find them in sechiba.card to change.

[UserChoices]
NEWHYDROL=y
ROUTING=n
LAIMAP=n
IMPOSE_VEG=y
LAND_USE=n
OKCO2=y
CO2varying=n

Something similar in SPINUP/SUBJOBS/OOL_SEC_STO/COMP/orchidee_ol.card

[UserChoices]
NORESTART=n
TIMELENGTH=y

If we turn off XIOS, we have a couple variables undeclared in IOIPSL, so it crashes. Instead, let's leave XIOS on and include the following hack for it to find the iodef.xml file. Note that you will have to change this path!

        (/home/orchidee03/mmcgrath/MYFOLDER/config/ORCHIDEE_OL/SPINUP_ANALYTIC_FG1/PARAM/iodef.xml, .)   ,\

Launch the job (from the README file).

   ./Job_ENSEMBLE fluxnet > out.Job_ENSEMBLE

BE SURE TO CHECK THE USED RUN.DEFs. These can be found by changing to the RUN_DIR when the job is running. The scripts will add flags to the end of the run.def, and sometimes these may conflict with what you want to run.

Speed

Some timing tests were carried out with TAG2.1, TRUNK (r6096), and CAN (r6091) on obelix. This revealed the importance of the NBUFF=0 keyword for running with FLUXNET data for a single site. When running for a single site with forcing that has lower temporal resolution (e.g., CRUNCEP, which has six-hourly resolution instead of the 30 min resolution of FLUXNET), it's much less important. The amount of data output for all runs was adjusted to give approximately the same size of files. The optimized executables were used for all tests (-O3).

I take timings from four locations: CPU Time Global and Real Time Global from out\_orchidee, and then the real and user times reported by time -p ./orhcidee\_ol. For the most part, they are similar. For clairity, I only report Real time Global from out\_orchidee below. Error bars are the standard deviation from 5 independent runs to show the variance.

The TRUNK and TAG21 have 15 PFTs, CAN has 28 PFTs, but they are all set to zero except for NeedleleafEvergreenTemperate? (PFT4...4), Deciduous temperate (PFT6...8), C3Grass (PFT10..23), and C3Crop (PFT12...26), which are all set to 0.25. I wanted to simulate a somewhat realistic pixel with a mix of vegetation.

Using NBUFF=1

FLUXNET forcing, XIOS, half-hour sechiba history, one day stomate history, 5 years, no libIGCM, the total time is [in seconds, with standard deviation from five runs on obelix]

TAG21       1270 $\pm$ 60
TRUNK       4600 $\pm$ 600
CAN         5800 $\pm$ 500

CRUNCEP forcing, XIOS, half-hour sechiba history, one day stomate history, 5 years, no libIGCM

TAG21       1310 $\pm$ 70
TRUNK       1700 $\pm$ 100
CAN         1810 $\pm$ 90

Using NBUFF=0

FLUXNET forcing, XIOS, half-hour sechiba history, one day stomate history, 5 years, no libIGCM, the total time is [in seconds, with standard deviation from five runs on obelix]

TAG21       1250 $\pm$ 140
TRUNK       1480 $\pm$ 90
CAN         1700 $\pm$ 200

CRUNCEP forcing, XIOS, half-hour sechiba history, one day stomate history, 5 years, no libIGCM

TAG21       1310 $\pm$ 50
TRUNK       1440 $\pm$ 110
CAN         1700 $\pm$ 200