= Name and subject of the action

Last edition: '''[[Wikinfo(changed_ts)]]''' by '''[[Wikinfo(changed_by)]]'''

The PI is responsible to closely follow the progress of the action, 
and especially to contact NEMO project manager if 
the delay on preview (or review) are longer than the 2 weeks expected.

[[PageOutline(2, , inline)]]

== Summary

||=Action       || Loop Fusion                        ||
||=PI(S)        || I.Epicoco, F.Mele                                                 ||
||=Digest       || Adoption of the 'loop fusion' optimization tecnique to reduce the cache misses and to improve the computational performance     ||
||=Dependencies || Extra-halo                                                ||
||=Branch       || source:/NEMO/branches/{YEAR}/dev_r{REV}_{ACTION_NAME} ||
||=Previewer(s) || TBD                                                 ||
||=Reviewer(s)  || TBD                                                 ||
||=Ticket       || #2367                                                 ||

=== Description

{{{#!box width=25em help
Describe the goal of development and the methodology, \\
add reference documents or publications if relevant.
}}}

''
The computational peak performance of the target parallel architecture can be better exploited working on the vectorisation level of the code. Many compilers usually are able to perform automatic vectorisation but the code needs to be written in such a way as to drive the compiler to increase the vectorisation level. A screening of the code will be needed in order to limit the dependency issues. Moreover, directives can also be used to increase the execution of SIMD instructions and to get closer to modern core peak performance.

Single core performance will be enhanced by changing the structure of the DO-loops. Namely, the DO loops will be fused in order to perform as much operations as possibile over the current (j, i, k) grid cell before moving on processing the next one. This approach will ehnance the vectorization level and the cache reuse.

The DO-loops fusion requires also to move the halo exchange before the fused loops, and this implies an extra-halo exchange. 
Part of this action is also focused on moving the communication before a routine/kernel execution extending also the halo region. 

Planned optimisations will be designed taking care to ensure that scientific quality of the code is not compromised. 
''

=== Implementation

{{{#!box width=35em help
Describe flow chart of the changes in the code. \\
List the Fortran modules and subroutines to be created/edited/deleted. \\
Detailed list of new variables to be defined (including namelists), \\
give for each the chosen name and description wrt coding rules.
}}}

''
The DO-loops fusion can be inserted in the NEMO code gradually, but this requires to move the halo exchanges earlier in the code and this is possible thanks to the extended halo=2

a cleanup of the useless communications has been applied
1. the communications before a result writing have been removed because only the inner domanin data are stored into the output files
2. some communications have been removed by changing the DO LOOP ranges
3. most of the communication have been moved earlier in the code but exchanging a wider halo. We still maintain the support for halo=1 in the code
 
Loop fusion implementation plan is as follows
1. we fuse the DO-loops following the strategy defined during the NEMO HPC-WG meeting (please refer to the slides here attached https://forge.ipsl.jussieu.fr/nemo/attachment/wiki/2019WP/HPC-02_Epicoco_SingleCorePerformance/HPC02_SingleCorePerformance_proof_of_concept.pdf). We will start from the advection routines (both from tracers and for ocean dynamics)
1a. the compilation key (key_loop_fusion) to activate or deactivate the loop fusion optiization
2a. the loop fusion has been applied on traadv_fct and traadv_mus (two new files have been added with the loop fused)=
2. proceed with LDF module (next year)
3. we will complete with the remaining routines from the most computing intensive towards the less computing intensive one. (next year)
''

=== Documentation updates

{{{#!box width=55em help
Using previous parts, define the main changes to be done in the NEMO literature 
(manuals, guide, web pages, …).
}}}

''...''

== Preview 

{{{#!box width=50em info
[[Include(wiki:Developers/DevProcess#preview_)]]
}}}

''...''

== Tests

{{{#!box width=50em info
[[Include(wiki:Developers/DevProcess#tests)]]
}}}

''...''

== Review

{{{#!box width=50em info
[[Include(wiki:Developers/DevProcess#review)]]
}}}

''...''