If you've never really used LMD before, coupling is a scary event, even if you know ORCHIDEE quite well. Thankfully, there are some tips to make things a bit easier.

Tip 1) If the job crashes immediately (i.e., within a day), it's good to check the coupling between the two models, which variables are passed and what their values are. This can be found in intersurf_gathered, in src_sechiba/intersurf.f90. If you changed the flag check_INPUTS in this file to .TRUE. and recompile, it will create a whole bunch of useful files in the run directory when run, of the format W*.nc (Walb_nir.nc, Wfluxlat.nc, etc., one for all the variables passed to ORCHIDEE and passed from ORCHIDEE). You can look at these with the standard NetCDF tools for each step to see if the values are unusual. In order to determine what is "usual", I do a one day run on the same resolution with the trunk and all the options I'm interested in (11-layer hydrology, STOMATE, new physics, etc.) and enable the check_INPUTS flag. This gives me an identical set of files to compare to.

Question: where is the run directory? It is not the submit directory (config/, where the job is submitted from), and it is not the archive directory (IGCM_OUT/, where the output files are stored). It is a temporary directory where all the files are copied to and execution takes place. If you look in the Script_* output file in the submit directory of a job, you can find a line which looks something like this (on Curie)

IGCM_sys_Cd : /ccc/scratch/cont003/dsm/p529grat/RUN_DIR/1213660_24665/DOFOCO.24665

This is the run directory. It will last for some time after the run has completed (days? weeks?), but eventually it will be deleted by the system.

Tip 2) Turn on debug flags for ORCHIDEE and LMD.

For ORCHIDEE, there are a couple options which involve changing util/AA_make.gdef. On Curie, I generally make my own, but there is a line which already exists in the svn version of the file. All you need to do is change

#-Q- curie  F_O = -DCPP_PARA -O3 $(F_D) $(F_P) -I$(MODDIR) -module $(MODDIR) -fp-model precise
######-Q- curie  F_O = -DCPP_PARA -p -g -traceback -fp-stack-check -ftrapuv -check bounds $(F_D) $(F_P) -I$(MODDIR) -module $(MODDIR)

to

######-Q- curie  F_O = -DCPP_PARA -O3 $(F_D) $(F_P) -I$(MODDIR) -module $(MODDIR) -fp-model precise
#-Q- curie  F_O = -DCPP_PARA -p -g -traceback -fp-stack-check -ftrapuv -check bounds $(F_D) $(F_P) -I$(MODDIR) -module $(MODDIR)

Then rerun util/ins_make, go to modeles/ORCHIDEE, and type "gmake clean && gmake driver". Now ORCHIDEE should be compiled with debug flags.

For LMD, you can change the Makefile directly in config/LMDZOR_v5.2/. If you change the following lines

lmdz: ../../modeles/LMDZ
    (cd ../../modeles/LMDZ; ./makelmdz_fcm -d $(RESOL_LMDZ) -cosp true -v true -parallel mpi -arch $(FCM_ARCH) ce0l ; cp bin/ce0l_$(RESOL_LMDZ)_phylmd_para_orch.e ../../bin/create_etat0_limit.e ; )
    (cd ../../modeles/LMDZ; ./makelmdz_fcm -d $(RESOL_LMDZ) -cosp true -v true -parallel mpi -arch $(FCM_ARCH) gcm ; cp bin/gcm_$(RESOL_LMDZ)_phylmd_para_orch.e ../../bin/gcm.e ; )

to

lmdz: ../../modeles/LMDZ
    (cd ../../modeles/LMDZ; ./makelmdz_fcm -d $(RESOL_LMDZ) -cosp true -v true -parallel mpi -arch $(FCM_ARCH) ce0l ; cp bin/ce0l_$(RESOL_LMDZ)_phylmd_para_orch.e ../../bin/create_etat0_limit.e ; )
    (cd ../../modeles/LMDZ; ./makelmdz_fcm -d $(RESOL_LMDZ) -cosp true -v true -debug -parallel mpi -arch $(FCM_ARCH) gcm ; cp bin/gcm_$(RESOL_LMDZ)_phylmd_para_orch.e ../../bin/gcm.e ; )

and then type "gmake clean && gmake YOUR_RESOLUTION" (where YOUR_RESOLUTION is LMD144142-L39, LMD9695_L39, etc.), LMD should be compiled with debug flags as well. You have to be careful with this since if you run ins_make again, you will lose the addition of the "-debug" flag in the Makefile and you'll have to re-add it. The actual debug flags appear to be stored in modeles/LMDZ/arch/arch-X64_CURIE.fcm.