source: CONFIG_DEVT/IPSLCM6.5_work_ENSEMBLES/oasis3-mct/doc/UGA4_changes.tex

\newpage

\chapter{Changes between the different versions of OASIS3-MCT}
\label{sec_changes}

The evolution between the different versions of OASIS3-MCT can be
followed in real-time by registering on the Redmine project management
site at https://inle.cerfacs.fr/ (see "Register" at the right top of
the page). On this site, registered users can browse the sources and
consult tickets describing bug fixes and developments. To follow day
to day evolution of the OASIS3-MCT sources, it is also possible to
have your e-mail added to the list of addresses to which the log files
of the SVN checkins are automatically sent; contact
oasishelp@cerfacs.fr if you wish to be added to that list.

\section{Changes between current master OASIS3-MCT version and OASIS3-MCT\_4.0}

\begin{itemize}

\item The {\tt oasis\_def\_var} interface was overloaded to support excluding the argument, {\tt id\_var\_shape}, from the argument list.  This argument is no longer used.  For backwards compatibility, it can still be passed.

\item  {\tt DESTARTR}, {\tt FRACARTR} or {\tt FRACNNTR} options : the ``True Area'' (TR) correction

New options for {\tt SCRIPR CONSERV} to have an exact conservation of the field surface-integrated values, based on the "TRue" (TR) area of the cells. Details can be found in section \ref{subsec_interp} and in \cite{valcke19}. Note that to activate these TR options, the true area of the cell must be provided in the auxiliary file {\it areas.nc} in {\bf square radians}.

\item New {\tt GSSPOS} and {\tt BSSPOS} for global CONSERV

These new options operate as {\tt GLBPOS} and {\tt BASPOS} except that the multiplicative term is computed separately for positive
      and negative values of the field. More details can be found in section \ref{subsec_cooking} and in \cite{craig19}.

\item Bugfixes

\begin{enumerate}

\item Argument {\tt id\_var\_nodim} defined as IN in {\tt mod\_oasis\_var.F90} to compile again with NEMO\_4.0 and NEMO trunk (git commit 28f4fe59)

\item Fix the treatment of the periodicity of the grids (git commits 20127fd9, 68021b13, facc08c1)

\item GAUSWGT remapping: exact calculation of average distance (commit 1df003a1) ; see details in https://cerfacs.fr/wp-content/uploads/2019/08/GlobC-TR-maisonnave-gaussian\_interpolation-2\_2019.pdf

 \item Bugfix ``lcoinc''  : ensures that a condition on coincidence of segments is reinitialised to .false. for each cell. A detailed analysis of the bug that lead to this bugfix condition can be found at the end of the page https://inle.cerfacs.fr/projects/oasis3-mct/wiki/Discussion\_about\_specific\_treatments\_of\_polar\_cells\_in\_scrip\_CONSERV\_interpolation.

 \end{enumerate}

\end{itemize}

\section{Changes between OASIS3-MCT\_4.0 and OASIS3-MCT\_3.0}

Different developments were realised to improve the parallel performance of OASIS3-MCT\_4.0. These developments are detailed in \cite{valcke11} .

\begin{itemize}

\item A new communication method, using the remapping weights to define the intermediate mapping decomposition, offers a significant gain at run time, especially for high-resolution cases running on a high number of tasks, thanks to reduced communication. However, as expected, the new method takes longer to initialize, partly due to the fact that the mapping weight file has to be read twice but also due to the extra cost for the initialization of the different MCT routers.  That initialization cost is largely mitigated by an upgrade to MCT 2.10.beta1 which reduces the penalty to few seconds.  Generally, it should be worth the extra initial cost to speed up the run time. See {\tt \$NMAPDEC} in section \ref{subsec_namcouplefirst}.

\item The hybrid MPI+OpenMP parallelisation of the SCRIP library (previously fully sequential) leads to great improvement in the calculation of the remapping weights. The results obtained here show a reduction in the weight calculation time of 2 to 3 orders of magnitude with the new parallel SCRIP library for high-resolution grids. Details are available in \cite{piacentini08}. The {\tt test\_interpolation} environment (see section \ref{subsec_testinterpolation} ) gives a practical example on how to use OASIS3-MCT\_4.0 to pre-calculate (i.e. in a separate job prior to the “real” simulation) the remapping weight and address file.

Thanks to some preliminary work, few bugs were fixed, in particular in the bounding box definition of the grid cells. This solves an important bug observed in the Pacific near the equator for the bilinear and bicubic interpolations for Cartesian grids.

However, given these modifications, one cannot expect to get exactly the same results for the interpolation weight-and-adress remapping files with this new parallel SCRIP version as compared to the previous SCRIP version in OASIS3-MCT\_3.0. We checked in many different cases that the interpolation error is smaller or of the same order than before. We also observed that the parallelisation does not ensure bit reproducible results when varying the number of processes or threads.

\item The new methods introduced in the global CONSERV operation reduce its calculation costs by one order of magnitude while still ensuring an appropriate level of reproducibility. This removes the bottleneck foreseen at high resolution with this important, and in few cases still unavoidable, global operation. These new methods are detailed in section \ref{subsec_cooking} .

\end{itemize}

The other new features offered by OASIS3-MCT\_4.0 are the following:

\begin{itemize}

\item Support for bundled coupling fields.

Bundled fields is now supported in the {\tt oasis\_put} and {\tt oasis\_get} interfaces to allow
easier coupling of multi-level or other related fields via a single
namcouple coupling definition and a single call to {\tt oasis\_put} or {\tt oasis\_get}.
Further details are provided in sec \ref{subsubsec_sendingreceiving}

\item Automatic coupling restart writing

An optional argument {\tt write\_restart} was added to the {\tt oasis\_put} routine. This argument is false by default but if it is explicitly set to true in the code, a coupling restart file will be written for that field only for that coupling timestep, saving the data that exists at the time of the call (see section \ref{prismput}).

\item Exact consistency between the number of weights and fields

Exact consistency is now required between number of weights fields in the coupling restart file and the arrays passed as arguments to the {\tt oasis\_put} routine. For example, for a 2nd order conservative remapping (CONSERV SECOND), 3 weights are needed and 3 fields must be provided as arguments i.e. the value of the field, its gradient with respect to the longitude and its gradient with respect to the latitude. For a first order conservative remapping (CONSERV FIRST), only one weight and one field are needed. Using a weight file with 3 weights for a first order conservative remapping is no longer allowed.

\item Upgrade in the namcouple configuration file

The namcouple reading routine was cleaned up including a refactoring of the gotos and continue statements, addition of few reusable routines including an abort routine, removal of some dead code, addition of support for blank lines (which are now considered comments), removal of requirement that keywords start at character 2 on a line, removal of requirement for {\tt \$END} in the namcouple, and updates to some error messages.

\item Other new functionalities with corresponding new namcouple keywords (see section \ref{subsec_namcouplefirst})

        \begin{itemize}

        \item {\tt \$NUNITNO}: specifies the minimum and maximum unit numbers to be used for input and output files in the coupling layer.

        \item {\tt \$NMATXRD}: indicates the method used to read mapping weights, either {\tt orig} or {\tt ceg}. In both methods, the weights are read in chunks by the model master task. With the {\tt orig} option, the weights are then broadcast to all other tasks and each task then saves the weights that will be applied to its grid points. With the {\tt ceg} option, the master task reads the weights and then identifies to which other task each weight should be sent.  A series of exchanges are then done with each other task involving just the weights needed by that other task. The {\tt orig} method sends much more data but is more parallel, while the {\tt ceg} method does most of the work on the master task but less data is communicated.

        \item {\tt \$NWGTOPT}: indicates how to handle bad interpolation weights.

        \item {\tt \$NNOREST} : if true, OASIS3-MCT will initialise any variable that normally requires a coupling restart file with zeros if that file does not exist.

        \end{itemize}
\end{itemize}

\section{Changes between OASIS3-MCT\_3.0 and OASIS3-MCT\_2.0}
\label{sec_oa2_oa3}

The main evolution of OASIS3-MCT\_3.0 with respect to OASIS3-MCT\_2.0 is the support of coupling exchanges between parallel components deployed in much more diverse configurations than before, for example, within one same executable between components running concurrently on separate sets of tasks or between components running sequentially on overlapping sets of tasks. All details are provided in section \ref{sec_deploy}.

This new version also includes:
\begin{itemize}
\item memory and performance upgrades
\item a new LUCIA tool for load balancing analysis
\item new memory tracking tool (gilt)
\item improved error checking and error messages
\item doxygen documentation
\item expanded test cases and testing automation
\item testing at high resolution (> 1M gridpoints), high processor counts (32k pes), and with large variable counts (> 1k coupling fields)
\item many bug fixes
\end{itemize}

\section{Changes between OASIS3-MCT\_2.0 and OASIS3-MCT\_1.0}
\label{sec_oa1_oa2}

The main changes and bug fixes new in OASIS3-MCT\_2.0 are the
following:
\begin{itemize}

\item Support of {\tt BICUBIC} interpolation, see paragraph {\tt
    BICUBIC} in section \ref{subsec_interp}. If the source grid
    is not a gaussian reduced grid (D), the gradient in the first dimension,
    the gradient in the second dimension, and the cross-gradient
    of the coupling field must be calculated by the model and
    given as arguments to {\tt oasis\_put}, as explained in section \ref{prismput}.
    If the source grid is a gaussian reduced grid (D), OASIS3-MCT\_2.0
    can calculate the interpolated field using only the values of the source
    field points.

\item Support of {\tt CONSERV/SECOND} remapping, see paragraph {\tt
    CONSERV/SECOND} in section \ref{subsec_interp}.

\item Support of components exchanging data on only a subdomain of the
  global grid: a new optional argument, ig\_size was added to
  oasis\_def\_partition, that provides the user with the ability to
  define the total number of grid cells on the grid (see section
  \ref{subsubsec_Partition}).

%\item The CPP key "balance" in mod\_oasis\_advance was added; this
%  option can be used to produce timestamps in OASIS debug file (see
%  section \ref{subsec_cpp}).

\item The variable {\tt TIMER\_Debug} controlling the amount of time
  statistics written out is now an optional argument read in the {\it
    namcouple}; see the {\tt NLOGPRT} line in
  \ref{subsec_namcouplefirst} and all details about time statistics in
  section \ref{timestat}.

\item Specific problems in writing out the time statistics when all
  the processors are not coupling were solved (see Redmine issue
  \#497)

\item The problem with restart files when one coupling field is sent
  to 2 target components was solved (see Redmine ticket \#522)

\item A memory leak in mod\_oasis\_getput\_interface.F90 was fixed
  thanks to R. Hill from the MetOffice (see Redmine ticket \#437)

\item A bug fix was provided to ensure that the nearest neighbour
  option is activated when the option {\tt FRACNNEI} is defined in the
  {\it namcouple} for the conservative interpolation .

\item The behaviour of OASIS3-MCT was changed in the case a
  component model tries to send with {\tt oasis\_put} a field declared
  with a {\tt oasis\_def\_var} but not defined in the configuration
  file {\it namcouple}; this will now lead to an abort. In this case,
  the field ID returned by the {\tt oasis\_def\_var} is equal to -1
  and the corresponding {\tt oasis\_put} should not be called.
  Conversely, all coupling fields appearing in the {\it namcouple}
  must be defined with a call to {\tt oasis\_def\_var}; this constraint
is imposed to avoid that a typo in the namcouple would lead to
coupling exchanges not corresponding to what the user intends to activate.

\item OASIS3-MCT developments are now continuously tested and
  validated on different computers with a test suite under Buildbot,
  which is a software written in Python to automate compile and test
  cycles required in software project (see
  https://inle.cerfacs.fr/projects/oasis3-mct/wiki/Buildbot on the
  Redmine site).

\end{itemize}

\section{Changes between OASIS3-MCT\_1.0 and OASIS3.3}

\subsection{General architecture}
\label{sec_changes_gen}

\begin{itemize}
\item OASIS3-MCT is (only) a coupling library

Much of the underlying implementation of OASIS3 was refactored to
leverage the Model Coupling Toolkit (MCT). OASIS3-MCT is a coupling
library to be linked to the component models and that
carries out the coupling field transformations (e.g. remappings/interpolations)
in parallel on either the source or target processes and that performs
all communication in parallel directly between the component models; there
is no central coupler executable anymore\footnote{As with OASIS3.3,
  the ``put'' calls are non-blocking but the ``get'' calls are
blocking. As before, the user has to take care of implementing a coupling
algorithm that will result in matching ``put'' and ``get'' calls to
avoid deadlocks (see section \ref{subsubsec_sendingreceiving}). The lag (LAG) index works as
before in OASIS3.3 (see section \ref{subsec_lag})}.

\item {\tt MAPPING} transformation to use a pre-defined mapping file

With {\tt MAPPING}, OASIS3-MCT has the ability to read a predefined
set of weights and addresses (mapping file) specified in the {\it
  namcouple} to perform the interpolation/remapping. The user also has
the flexibility to choose the location and the parallelization strategy of the
remapping with specific {\tt MAPPING} options (see section \ref{subsec_interp}).

\item Mono-process mapping file generation with the SCRIP library

But as before, OASIS3-MCT\_1.0 can
also generate the mapping file using the SCRIP library
\citep{Jones99}. When this is the case, the mapping file generation is
done on one process of the model components; all previous {\tt SCRIPR} remapping
schemes available in OASIS3.3 are still supported besides {\tt
  BICUBIC} and {\tt CONSERV/SECOND}. (Note: these remapping schemes, not available in OASIS3-MCT\_1.0
  were reactivated in OASIS3-MCT\_2.0, see \ref{sec_oa1_oa2}.)

\item MPI2 job launching is NOT supported.

  Only MPI1 start mode is allowed. As before with the MPI1 mode, all
  component models must be started by the user in the job script in a
  pseudo-MPMD mode; in this case, they will automatically share the
  same {\tt MPI\_COMM\_WORLD} communicator and an internal
  communicator created by OASIS3-MCT needs to be used for internal
  parallelization (see section \ref{subsec_MPI1}).

\end{itemize}

\subsection{Changes in the coupling interface in the component models}
\label{sec_changes_API}

\begin{itemize}

\item Use statement

The different OASIS3.3 {\tt USE} statements were gathered into one {\tt
  USE mod\_oasis} (or one {\tt USE mod\_prism}), therefore much
  simpler to use.

\item Support of previous {\tt prism\_xxx} and new {\tt oasis\_xxx}
  interfaces

OASIS3-MCT retains prior interface names of OASIS3.3
  (e.g. {\tt prism\_put\_proto}) to ensure full backward
  compatibility. However, new interface names such as {\tt
    oasis\_put} are also available and should be prefered. Both
  routine names are listed in chapter \ref{sec_modelinterfacing}.

\item Auxiliary routines not supported yet

Auxiliary routines {\tt prism\_put\_inquire}, {\tt
  prism\_put\_restart\_proto}, {\tt prism\_get\_freq} are not
  supported yet. (Note: {\tt prism\_put\_inquire} and {\tt prism\_get\_freqs} were  reintroduced in OASIS3-MCT\_3.0 and equivalent of  {\tt
  prism\_put\_restart\_proto} in OASIS3-MCT\_4.0.)

\item Support of components for which only a subset of processes
  participate in the coupling

New routines {\tt oasis\_create\_couplcomm} and {\tt
  oasis\_set\_couplcomm} are now available to create or set a coupling
  communicator in the case only a subset of the component processes
  participate in the coupling. But
  even in this case, all OASIS3-MCT interface routines, besides the
  grid definition (see section \ref{subsubsec_griddef}) and the
  ``put'' and `` get'' call per se (see section
  \ref{subsubsec_sendingreceiving}), are collective and must be called
  by all processes. (Note: this has changed with OASIS3-MCT\_3.0.)

\item New routines {\tt oasis\_get\_debug} and {\tt oasis\_set\_debug}

New routines {\tt oasis\_get\_debug} and {\tt oasis\_set\_debug}
  are now available to respectively retrieve the current OASIS3-MCT
  internal debug level (set by {\tt \$NLOGPRT} in the {\it namcouple}) or to change it (see section
  \ref{subsubsec_auxroutines}).

\end{itemize}

\subsection{Functionality not offered anymore}
\label{sec_changes_old}

\begin{itemize}

\item {\tt SCRIPR/BICUBIC} and {\tt SCRIPR/CONSERV/SECOND} remappings

As in OASIS3.3, the SCRIP library can be used to generate the
  remapping/interpolation weights and addresses and write them to a
  mapping file. All previous {\tt SCRIPR} remapping
schemes available in OASIS3.3 are still supported in OASIS3-MCT\_1.0 besides {\tt
  BICUBIC} and {\tt CONSERV/SECOND} because these remapping involve
at each source grid point the value of the field but also the value of the
gradients of the field (which are not known or calculated). (Note: these remapping schemes,
not available in OASIS3-MCT\_1.0 were reactivated in OASIS3-MCT\_2.0, see \ref{sec_oa1_oa2}.)

\item Vector field remapping

Vector field remapping is not and will not be supported (see ``Support
  of vector fields with the SCRIPR remappings'' in section \ref{subsec_interp}).

\item Automatic calculation of grid mesh corners in {\tt SCRIPR/CONSERV}

For {\tt SCRIPR/CONSERV} remapping, grid mesh corners will not
  be compute automatically if they are needed but not provided.

\item Other transformations not supported

\begin{itemize}

\item The following transformations are not available in OASIS3.3
and will most probably not be implemented as it should be not
too difficult to implement the equivalent operations in the component
models themselves: {\tt CORRECT}, {\tt FILLING}, {\tt SUBGRID}, {\tt MASKP}

\item {\tt LOCTRANS/ONCE} is not explicitely offered as it is equivalent to
defining a coupling period equal to the total runtime.

\item The following transformations are not available as they were already
deprecated in OASIS3.3 : {\tt REDGLO}, {\tt INVERT},
{\tt REVERSE}, {\tt GLORED}

\item {\tt MASK} and {\tt EXTRAP} are not available but the corresponding
linear extrapolation can be replaced by the more efficient option
using the nearest non-masked source neighbour for target points having
their original neighbours all masked. This is now the default option for {\tt SCRIPR/}{\tt DISTWGT}, {\tt GAUSWGT} and {\tt BILINEAR} interpolations. It is
also included in \newline {\tt SCRIPR/CONSERV} if {\tt FRACNNEI}
normalization option is chosen (see section \ref{subsec_interp}).

\item {\tt INTERP} interpolations are not available; {\tt SCRIPR}
  should be used instead.

\item {\tt MOZAIC} is not available as {\tt MAPPING} should be used
  instead.

\item{\tt NOINTERP} does not need to be specified anymore if no
  interpolation is required.

\item Field combination with {\tt BLASOLD} and {\tt BLASNEW}; these
  transformations only support multiplication and addition terms to
the fields (see section \ref{subsec_preproc}).

\end{itemize}

\item Using the coupler in interpolator-only mode

This is not possible anymore as OASIS3-MCT is now only a coupling
library. However, it is planned, in a further release, to provide a
toy coupled model that could be use to check the quality of the
remapping for any specific couple of grids. (Note: this was done in OASIS3-MCT\_2.0.)

\item Coupling field CF standard names

The file cf\_name\_table.txt is not needed or used anymore. The CF
  standard names of the coupling fields are not written to the debug
  files.

\item Binary auxiliay files

All auxiliary files, besides the {\it namcouple} must be NetCDF;
  binary files are not supported anymore.
\end{itemize}

\subsection{New functionality offered}
\label{sec_changes_new}

\begin{itemize}

\item Better support of components for which only a subset of processes
  participate in the coupling

In OASIS3.3, components for which only a subset of processes
  participated in the coupling were supported in a very restricted
  way. In fact, this subset had to be composed of the N first
  processes and N had to be specified in the {\it namcouple}. Now, the
  subset of processes can be composed of any of the component
  processes and does not have to be pre-defined in the {\it
    namcouple}. New routines {\tt oasis\_create\_couplcomm} and {\tt
  oasis\_set\_couplcomm} are now available to create or set a coupling
  communicator gathering only these processes (see section \ref{subsec_MPI1}). (Note: this was further improved in OASIS3-MCT\_3.0.)

\item Exact restart for {\tt LOCTRANS} transformations

If needed, LOCTRANS transformations write partially
  transformed fields  in the coupling restart file at the end of a run
  to ensure an exact restart of the next run (see section
  \ref{subsec_timetrans}). For that
  reason, coupling restart filenames are now required for all {\it
    namcouple} entries that use LOCTRANS (with non INSTANT
  values). This is the reason an optional restart file is now provided
  on the OUTPUT {\it namcouple} input line. If the coupling periods of
  two (or more) coupling fields are different, it is necessary to define
  two (or more) restart files, one for each field.

\item Support to couple multiple fields via a single communication.

 This is supported through colon
delimited field lists in the {\it namcouple}, for example

{\tt ATMTAUX:ATMTAUY:ATMHFLUX  TAUX:TAUY:HEATFLUX 1 3600 3 rstrt.nc EXPORTED}

 in a single {\it namcouple} entry. All fields will use the
{\it namcouple} settings for that entry. In the component model codes,
these fields are still sent (``put'') or received (``get'') one at a
time. Inside OASIS3-MCT, the fields are stored and a single mapping
and send or receive instruction is executed for all fields. This is
useful in cases where multiple fields have the same coupling
transformations and to reduce communication costs by aggregating multiple
fields into a single communication. If a LOCTRAN transformation is needed
for these multiple fields, it is necessary to define a restart file for
these multiple fields. The coupling fields must be sent and received in the same order
as they are defined in the corresponding single entry of the namcouple
(not relevant in further versions of OASIS3-MCT).

\item Matching one source filed with multiple targets

  A coupling field sent by a source component model can be associated
  with more than one target field and model (get). In that case, the
  source model needs to send (``put'') the field only once and the
  corresponding data will arrive at multiple targets as specified in
  the {\it namcouple} configuration file. Different coupling
  frequencies and transformations are allowed for different coupling
  exchanges of the same field. If coupling restart files are required
  (either if a {\tt LAG} or if a {\tt LOCTRANS} transformation is
  specified), it is mandatory to specify different files for the
  different fields.

%In addition, a single
%  coupling field can be sent to the same destination model using
%  multiple transforms as long as the destination field name is
%  unique.
The inverse feature is not allowed, i.e. a single target (get) field
CANNOT be associated with multiple source (put) fields.

\item The debug files

The debug mode was greatly improved as compared to OASIS3.3. The level
of debug information written out to the OASIS3-MCT debug files for
each model process is defined by the \$NLOGPRT value in the {\it
  namcouple}. All details are provided in section
\ref{subsec_namcouplefirst}.

\end{itemize}

\subsection{Changes in the configuration file {\it namcouple}}
\label{sec_changes_namcouple}

\begin{itemize}

\item The {\it namcouple} configuration file of OASIS3-MCT is fully backward
compatible with OASIS3.3. However, several {\it namcouple} keywords
have been deprecated even if they are still
allowed.  These keywords are noted ``UNUSED'' in sections
\ref{subsec_namcouplefirst} and \ref{subsec_namcouplesecond} and are
not fully described. Information below these keywords will not be read
and will not be used: \$SEQMODE , \$CHANNEL, \$JOBNAME, \$INIDATE,
\$MODINFO, \$CALTYPE.

\item Also the following inputs should not appear in the {\it namcouple}
anymore as the related functionality are not supported anymore in
OASIS3-MCT (see above): field status AUXILARY, time transformation
ONCE, REDGLO, INVERT, MASK, EXTRAP, CORRECT, INTERP, MOZAIC, FILLING,
SUBGRID, MASKP, REVERSE, GLORED.

\item To get 2D fields in the debug output NetCDF files, the 2D dimensions of the
  grids must be provided in the {\it namcouple} (except if the field
  has the status OUTPUT); otherwise, the fields in the debug output files will be 1D.

\end{itemize}

\subsection{Other differences}
\label{sec_changes_other}

\begin{itemize}

\item IGNORED and IGNOUT fields are converted to EXPORTED and EXPOUT
  respectively.

\item The file {\tt areas.nc} is not needed anymore to calculate some
  statistics with options CHECKIN and/or CHECKOUT.

\item SEQ index is no longer needed to ensure correct coupling
  sequencing within the coupler. Use of SEQ allows the coupling layer
  to detect potential deadlocks before they happen and to exit
  gracefully (see section \ref{subsec_sec}).

\item The I/O library mpp\_io is no longer used to write the restart and output files.


\end{itemize}