Problems with AMM12 SETTE tests with -O3 optimisation (ifort compiler)

[acc]

My initial attempts to run the AMM12 SETTE tests with a v3.6 trunk (rev 4673) failed when using the ifort compiler (v14) and a -O3 optimisation level. The failure was CFL breaches after 12 time steps. The tests ran successfully at -O2 and below for AMM12 and at -O3 for all other tests (except AGRIF).

Eventually tracked the cause to this block at line 291 in dynspg_ts.F90 ( key_vectopt_loop is not defined ):

      DO jk = 1, jpkm1
#if defined key_vectopt_loop
         DO jj = 1, 1         !Vector opt. => forced unrolling
            DO ji = 1, jpij
#else 
         DO jj = 1, jpj
            DO ji = 1, jpi
#endif                                                                   
               zu_frc(ji,jj) = zu_frc(ji,jj) + fse3u_n(ji,jj,jk) * ua(ji,jj,jk) * umask(ji,jj,jk)
               zv_frc(ji,jj) = zv_frc(ji,jj) + fse3v_n(ji,jj,jk) * va(ji,jj,jk) * vmask(ji,jj,jk)
            END DO
         END DO
      END DO

which looks harmless but adding a compiler directive to suppress loop fusion enables a successful SETTE test at -O3 optimisation. I.e.:

      DO jk = 1, jpkm1
!DIR$ NOFUSION
#if defined key_vectopt_loop
         DO jj = 1, 1         !Vector opt. => forced unrolling
            DO ji = 1, jpij
#else 
         DO jj = 1, jpj
            DO ji = 1, jpi
#endif                                                                   
               zu_frc(ji,jj) = zu_frc(ji,jj) + fse3u_n(ji,jj,jk) * ua(ji,jj,jk) * umask(ji,jj,jk)
               zv_frc(ji,jj) = zv_frc(ji,jj) + fse3v_n(ji,jj,jk) * va(ji,jj,jk) * vmask(ji,jj,jk)
            END DO
         END DO
      END DO

Does anyone have a clue what might be happening here?

[acc]

Alternatively, if I replace the inner loops and reduce the block to:

      DO jk = 1, jpkm1
         zu_frc(:,:) = zu_frc(:,:) + fse3u_n(:,:,jk) * ua(:,:,jk) * umask(:,:,jk)
         zv_frc(:,:) = zv_frc(:,:) + fse3v_n(:,:,jk) * va(:,:,jk) * vmask(:,:,jk)
      END DO

Then the tests are successful at -O3 without any compiler directives. Can anyone see any reason not to make this change permanent?

[jchanut]

We found exactly the same solution: That's however weird.
It may depend on which ifort version you use.
No reason not to do this change.

[acc]

Change submitted at revision #4687

[smasson]

do you use the compilation option:

-fp-model precise

[acc]

Yes. Although on this machine '-fp-model source' makes more sense otherwise you get a lot of warnings:

ifort: command line warning #10212: -fp-model precise evaluates in source precision with Fortran.

Both give the same error after 12 time steps with the original code.