How to manipulate/compare/examine netcdf files using cdo or nco?

Author: S. Luyssaert and J. Ryder
Last revision: 2020/02/28, B. Guenet

nco

nco is a very powerful tool to print data, modify variable names, concatenate, etc on netcdf file. A detailled documentation can be found here ​http://nco.sourceforge.net/ We will just summarize here some of the most useful nco commands often used in the orchidee context.

• ncdump

This command print all the informations stored in your netcdf file. Some prefix can extract only the information you need.

• To look at header information:

ncdump -h myJob_20060101_20061231_1M_sechiba_history.nc

• To extract one variable (here for 'fluxlat', the latent heat flux)

ncdump -v fluxlat myJob_20060101_20061231_1M_sechiba_history.nc 

• To extract one variable and show them in a tabulated way (here for 'fluxlat', the latent heat flux)

ncdump -v fluxlat -f fortran -n fluxlat myJob_20060101_20061231_1M_sechiba_history.nc 

• ncrename

This command change the name of a given variable/dimension depending on the prefix you are using

• To rename a variable within the file (here for 'fluxlat', the latent heat flux)

  ncrename -v fluxlat,latent_heat_flux myJob_20060101_20061231_1M_sechiba_history.nc 
  

• To rename a variable a creating a new file (here for 'fluxlat', the latent heat flux)

  ncrename -v fluxlat,latent_heat_flux myJob_20060101_20061231_1M_sechiba_history.nc mynewfile.nc
  

• To rename a dimension within the file (here for 'lon', the longitude)

  ncrename -d lon,longitude myJob_20060101_20061231_1M_sechiba_history.nc 
  

• To rename a variable a creating a new file (here for 'lon', the longitude)

  ncrename -d lon,longitude myJob_20060101_20061231_1M_sechiba_history.nc mynewfile.nc
  

• ncks

If you would like to remove a certain time series from a netcdf file (for instance, in this example, all data from 'temp_atmos_pres' in the file jamescdf.nc for which the height dimension is 20) with the output written to jamestest.nc

ncks -H -d height,20 -d temp_atmos_pres,: jamescdf.nc jamestest.nc

• ncap2

The most powerful and quickly evolving tool from the nco set of functions

search what you are looking on this growing list of answers ​forum nco

A simple example showing different capacities (creation of a variable, use of a mask, call to an attribute, count the total and the size of the field (we can restrict this operation to a dimension as shown on the second line)

 ncap2 -O -s 'missing_flag[$y,$x]=0;where(t2m_daily!=t2m_daily@missing_value) missing_flag=1;missing_count=float(missing_flag.total())/missing_flag.size();print(missing_count)' stomate_restart.nc  ~/foo.nc
 ncap2 -O -s 'missing_flag[$y,$x]=0;where(t2m_daily!=t2m_daily@missing_value) missing_flag=1;missing_count=float(missing_flag.total($x))/missing_flag.size($x);print(missing_count)' stomate_restart.nc  ~/foo.nc

XXXXXXXXXXXXXX

Similar to Ferret (see also How To), CDO can also be used to regrid NetCDF files. It is very straight and fast. Contrary to Ferret I [SL] did not find how to increase the memory allocation for CDO. Thus for very large files Ferret seems to be the software of your choice.

The call has the following structure:

cdo -f nc -regriddingMethod,gridTemplate.nc input.nc output.nc

There are three regridding methods: remapbil (bilinear interpolation), remapbic (bicubic interpolation), remapnn (nearest neighbour), remapcon (conservative regridding). Use remapnn and remapcon to avoid weird values if your variable is heterogeneous.

For the template file with your target grid, it can be any netcdf with regular lat/lon. A trick here is to use "full paths" and not relative paths to the files in order to work.

Some examples:

cdo -f nc -remapbil,/gpfs/cru/rst08auu/code/target.0.5degree.gpp.grid.45N.nc esa.ecv.smo.0.25deg.grid.monthly.1979.2010.nc esa.ecv.smo.0.25deg.grid.monthly.1979.2010.cdo.remapped.0.5deg.remapbil.45N.nc
cdo -f nc -remapnn,/gpfs/cru/rst08auu/code/target.0.5degree.gpp.grid.45N.nc esa.ecv.smo.0.25deg.grid.monthly.1979.2010.nc esa.ecv.smo.0.25deg.grid.monthly.1979.2010.cdo.remapped.0.5deg.remapnn.45N.nc

cdo

A command that handle many operations one would like to do when working on outputs of models of the Earth System :

• changing of time axis
• regridding (many grid are already implemented and it is an even more versatile tool)
• complex operations...

See the website of the project {{​https://code.zmaw.de/projects/cdo/wiki}} or the doc : {{:reunions_orchidee:cdo-manual.pdf| CDO Manual}} and the {{:reunions_orchidee:cdo_refcard.pdf | quickref}}

<code>

cdo -v # print list of available operators
cdo -V # version (if very old, think of downloading the source and install a new version, as it is a tool that is evolving quickly
cdo -h remapcon # informations about a command
cdo remapcon,my_grid my_climatology.nc my_regridded_climatology.nc # conservative regridding to an other resolution

</code>

my_grid could be a file like :

<code>

gridtype = lonlat gridsize = 220 xname = longitude xunits = degrees_east yname = latitude yunits = degrees_north xsize = 20 ysize = 11 xfirst = -18.75 xinc = 3.75 yfirst = 20

</code>

<code>

cdo sinfov my_file.nc #most info on the file cdo griddes my_file.nc #grid description (check that the type is not generic (which means unrecognised and thus not useable for mapping

</code>

Sometimes, one just need to add the attribute coordinates (linking to the variables defining the longitude and latitude positions) to enable the recognition of the grid that is used (see ​https://code.zmaw.de/boards/1/topics/55) <code>

ncatted -a coordinates,my_var,o,c,"nav_lon nav_lat" sst_data.nc

</code>

or to correct a wrong spelling of an attribute

<code>

ncatted -a units,latitude,m,c,"degrees_north" force2002.nc

</code>

Operator chaining

It is one of the main features of CDO. Use it as often as possible. But Note: Operatos with a arbitrary list of input files cannot be combined with other operators

# Simple combination: <code>

cdo sub -dayavg ifile2 -timavg ifile1 ofile

</code>

instead of :

<code>

cdo timavg ifile1 tmp1

cdo dayavg ifile2 tmp2

cdo sub tmp2 tmp1 ofile

rm tmp1 tmp2

</code>

To install a newer version than the default one on asterix/obelix

<code>

get files from ​https://code.zmaw.de/projects/cdo/files ./configure --prefix=/home/users/your_ACCOUNT/ --with-netcdf=/usr

make clean make make install

</code>

If you installed proj and hdf5 libraries somewhere, their can also be support for these file formats :

<code>

./configure --prefix=/home/users/your_ACCOUNT/ --with-netcdf=/usr --with-proj=/path/to/install --with-hdf5=/path/where/installed

</code>

ncview: a quick alternative

The quickest way of all to look at NetCDF files and to quickly plot some outputs is to use ncview:

ncview myJob_20060101_20061231_1M_sechiba_history.nc

Unfortunately the ncview application can be sometimes a little unstable.