Changes between Version 2 and Version 3 of SourceCode/TestRev2425


Ignore:
Timestamp:
2014-12-04T19:37:41+01:00 (10 years ago)
Author:
jgipsl
Comment:

--

Legend:

Unmodified
Added
Removed
Modified

  • SourceCode/TestRev2425

    v2 v3  
    11= ORCHIDEE trunk rev 2425 = 
    22 
    3 == Offline choisnel curie == 
    4 Debug=dev=prod different number of cores mpi give the same results. 
     3All coupled simulations are done with LMDZ5 testing rev 2163 using the old physics. 
     4 
     5== Parallelism and compile mode == 
     6 
     7 * ORCHIDEE gives the identical results for simulations with different number of cores MPI and/or OMP. It does not matter if the executable is compiled for pur MPI mode or for mixt MPI/OMP mode.  
     8  * At curie, different number of cores MPI changes the results due to LMDZ. In offline mode it's ok. 
     9 * Simulations coupled to LMDZ can now run in debug mode. The correction in LMDZ trunk was done : [http://web.lmd.jussieu.fr/trac/changeset/2158].  
     10 * The executable compiled in pur MPI and in mixt MPI_OMP gives the same results if the same optimization mode is used (prod/dev/debug) 
     11 * Sequential mode is tested with gfortran :  
     12 
     13==== ADA/IDRIS ==== 
     14 * Compilation en mode prod, dev and debug are successful for pur mpi and mixt mpi_omp.  
     15 * Changing the number of cores MPI and/or OMP does not change the results. For example running on 23MPI=32MPI=16MPI.2OMP 
     16 * Using optimization mode prod does not give the same results as optimization mode dev or debug 
     17 * LMDZORChoi and LMDZORCWRR fails in debug mode for 18mpi8omp. Maybe problem with memory. 
     18 
     19 
     20==== CURIE/TGCC ==== 
     21 * Compilation en mode prod, dev and debug are successful for pur mpi and mixt mpi_omp.  
     22 * Offline mode: all simulations are identical : compile mode, different number of cores MPI, pur mpi or mixt mpi_omp (but only running on 1 omp) 
     23 * Coupled mode:  
     24   * '''Simulations are not identical while changing the number of cores MPI also using LMDZ bucket without ORCHIDEE''' 
     25   * '''Simulations are identical while changing the number of cores MPI and OMP in hybrid mode ''' 
     26   * Results are different between debug and dev optimisation mode and between debug and prod.  
     27 
     28==== local PC using gfortran ==== 
     29 * LMDZ do not compile with cosp 
     30 
     31== Restartablilty == 
     32 * Same simulation set up is reproductible 
     33 * '''1+1 not = 2''' in coupled and in offline mode 
     34 
     35 
     36 
     37= Details of simulations = 
     38== Offline == 
     39{{{ 
     40ORChoi at curie : ALL OK : debug=dev=prod  
     41                            Different number of cores mpi or mpi+1omp give the same results.  
     42                            All simulations finished 
     43******************************************************************************* 
     44 
     45results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi_omp/32mpi_1omp 
     46results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = dev_mpi/23mpi 
     47results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = dev_mpi_omp/32mpi_1omp 
     48results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = prod_mpi/23mpi 
     49results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = prod_mpi_omp/32mpi_1omp 
     50 
     51results.out.debug_mpi:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi/32mpi 
     52results.out.dev_mpi:DIRECTORIES OK : dev_mpi/23mpi = dev_mpi/32mpi 
     53results.out.prod_mpi:DIRECTORIES OK : prod_mpi/23mpi = prod_mpi/32mpi 
     54 
     55 
     56 
     57ORChoi at ada : debug = dev not= prod   
     58                 Different number of cores mpi or mpi+1omp give the same results.  
     59                 All simulations finished 
     60******************************************************************************* 
     61results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi_omp/32mpi_1omp 
     62results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = dev_mpi/23mpi 
     63results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = dev_mpi_omp/32mpi_1omp 
     64results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = prod_mpi/23mpi 
     65results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = prod_mpi_omp/32mpi_1omp 
     66results.out.cross:DIRECTORIES OK : prod_mpi/23mpi = prod_mpi_omp/32mpi_1omp 
     67results.out.cross:DIRECTORIES OK : dev_mpi/23mpi = dev_mpi_omp/32mpi_1omp 
     68results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi_omp/32mpi_1omp 
     69 
     70results.out.debug_mpi:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi/32mpi 
     71results.out.dev_mpi:DIRECTORIES OK : dev_mpi/23mpi = dev_mpi/32mpi 
     72results.out.prod_mpi:DIRECTORIES OK : prod_mpi/23mpi = prod_mpi/32mpi 
     73 
     74 
     75 
     76ORCWRR sur curie : ALL simulations ok and equal 
     77                   Debug simulations did not finish : 30min not enough, 1hour ok 
     78******************************************************************************* 
     79results.out.cross:DIRECTORIES OK : debug_mpi/23mpin = debug_mpi_omp/32mpi_1ompn 
     80results.out.cross:DIRECTORIES OK : debug_mpi/23mpin = dev_mpi/23mpi 
     81results.out.cross:DIRECTORIES OK : debug_mpi/23mpin = dev_mpi_omp/32mpi_1omp 
     82results.out.cross:DIRECTORIES OK : debug_mpi/23mpin = prod_mpi/23mpi 
     83results.out.cross:DIRECTORIES OK : debug_mpi/23mpin = prod_mpi_omp/32mpi_1omp 
     84 
     85results.out.debug_mpi:DIRECTORIES OK : debug_mpi/23mpin = debug_mpi/32mpin 
     86results.out.dev_mpi:DIRECTORIES OK : dev_mpi/23mpi = dev_mpi/32mpi 
     87results.out.prod_mpi:DIRECTORIES OK : prod_mpi/23mpi = prod_mpi/32mpi 
     88 
     89 
     90 
     91ORCWRR sur ada : debug = dev not= prod   
     92                 Different number of cores mpi or mpi+1omp give the same results.  
     93                 All simulations finished 
     94******************************************************************************* 
     95results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi_omp/32mpi_1omp 
     96results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = dev_mpi/23mpi 
     97results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = dev_mpi_omp/32mpi_1omp 
     98results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = prod_mpi/23mpi 
     99results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = prod_mpi_omp/32mpi_1omp 
     100results.out.cross:DIRECTORIES OK : prod_mpi/23mpi = prod_mpi_omp/32mpi_1omp 
     101results.out.cross:DIRECTORIES OK : dev_mpi/23mpi = dev_mpi_omp/32mpi_1omp 
     102results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi_omp/32mpi_1omp 
     103 
     104results.out.debug_mpi:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi/32mpi 
     105results.out.dev_mpi:DIRECTORIES OK : dev_mpi/23mpi = dev_mpi/32mpi 
     106results.out.prod_mpi:DIRECTORIES OK : prod_mpi/23mpi = prod_mpi/32mpi 
     107}}} 
     108== LMDZ without ORCHIDEE == 
     109{{{ 
     110LMDZ only(without ORCHIDEE) at curie: All simulations finished 
     111                    Different number of cores mpi DO NOT give the same results with mpi mode. 
     112                    Different number of cores mpi or omp in hybrid mode give the same results 
     113******************************************************************************* 
     114results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi_omp/16mpi_8omp 
     115results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = dev_mpi/23mpi 
     116results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = dev_mpi_omp/16mpi_8omp 
     117results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = prod_mpi/23mpi 
     118results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = prod_mpi_omp/16mpi_8omp 
     119results.out.cross:DIRECTORIES DIFFER : prod_mpi/23mpi NOT = prod_mpi_omp/16mpi_8omp 
     120results.out.cross:DIRECTORIES DIFFER : dev_mpi/23mpi NOT = dev_mpi_omp/16mpi_8omp 
     121results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi_omp/16mpi_8omp 
     122 
     123results.out.debug_mpi:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi/32mpi 
     124results.out.debug_mpi_omp:DIRECTORIES OK : debug_mpi_omp/16mpi_8omp = debug_mpi_omp/32mpi_1omp 
     125results.out.debug_mpi_omp:DIRECTORIES OK : debug_mpi_omp/16mpi_8omp = debug_mpi_omp/32mpi_2omp 
     126results.out.dev_mpi:DIRECTORIES DIFFER : dev_mpi/23mpi NOT = dev_mpi/32mpi 
     127results.out.dev_mpi_omp:DIRECTORIES OK : dev_mpi_omp/16mpi_8omp = dev_mpi_omp/32mpi_1omp 
     128results.out.dev_mpi_omp:DIRECTORIES OK : dev_mpi_omp/16mpi_8omp = dev_mpi_omp/32mpi_2omp 
     129results.out.prod_mpi:DIRECTORIES DIFFER : prod_mpi/23mpi NOT = prod_mpi/32mpi 
     130results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/16mpi_8omp = prod_mpi_omp/32mpi_1omp 
     131results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/16mpi_8omp = prod_mpi_omp/32mpi_2omp 
     132 
     133 
     134LMDZ only(without ORCHIDEE) at ada: All simulations finished 
     135                  Different number of cores mpi or omp give the same results.  
     136******************************************************************************* 
     137results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi_omp/16mpi_8omp 
     138results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = dev_mpi/23mpi 
     139results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = dev_mpi_omp/16mpi_8omp 
     140results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = prod_mpi/23mpi 
     141results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = prod_mpi_omp/16mpi_8omp 
     142results.out.cross:DIRECTORIES OK : prod_mpi/23mpi = prod_mpi_omp/16mpi_8omp 
     143results.out.cross:DIRECTORIES DIFFER : dev_mpi/23mpi NOT = dev_mpi_omp/16mpi_8omp 
     144results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi_omp/16mpi_8omp 
     145 
     146results.out.debug_mpi:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi/32mpi 
     147results.out.debug_mpi_omp:DIRECTORIES OK : debug_mpi_omp/16mpi_8omp = debug_mpi_omp/32mpi_1omp 
     148results.out.debug_mpi_omp:DIRECTORIES OK : debug_mpi_omp/16mpi_8omp = debug_mpi_omp/32mpi_2omp 
     149results.out.dev_mpi:DIRECTORIES OK : dev_mpi/23mpi = dev_mpi/32mpi 
     150results.out.dev_mpi_omp:DIRECTORIES OK : dev_mpi_omp/16mpi_8omp = dev_mpi_omp/32mpi_1omp 
     151results.out.dev_mpi_omp:DIRECTORIES OK : dev_mpi_omp/16mpi_8omp = dev_mpi_omp/32mpi_2omp 
     152results.out.prod_mpi:DIRECTORIES OK : prod_mpi/23mpi = prod_mpi/32mpi 
     153results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/16mpi_8omp = prod_mpi_omp/32mpi_1omp 
     154results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/16mpi_8omp = prod_mpi_omp/32mpi_2omp 
     155}}} 
     156 
     157== LMDZOR == 
     158{{{ 
     159LMDZORChoi at curie: All simulations finished 
     160                    Different number of cores mpi DO NOT give the same results with mpi mode. For debug compiling it's ok.  
     161                    Different number of cores mpi or omp in hybrid mode give the same results 
     162******************************************************************************* 
     163results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi_omp/16mpi_8omp 
     164results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = dev_mpi/23mpi 
     165results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = dev_mpi_omp/16mpi_8omp 
     166results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = prod_mpi/23mpi 
     167results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = prod_mpi_omp/16mpi_8omp 
     168results.out.cross:DIRECTORIES DIFFER : prod_mpi/23mpi NOT = prod_mpi_omp/16mpi_8omp 
     169results.out.cross:DIRECTORIES DIFFER : dev_mpi/23mpi NOT = dev_mpi_omp/16mpi_8omp 
     170results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi_omp/16mpi_8omp 
     171 
     172results.out.debug_mpi:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi/32mpi 
     173results.out.debug_mpi_omp:DIRECTORIES OK : debug_mpi_omp/16mpi_8omp = debug_mpi_omp/32mpi_1omp 
     174results.out.debug_mpi_omp:DIRECTORIES OK : debug_mpi_omp/16mpi_8omp = debug_mpi_omp/32mpi_2omp 
     175results.out.dev_mpi:DIRECTORIES DIFFER : dev_mpi/23mpi NOT = dev_mpi/32mpi 
     176results.out.dev_mpi_omp:DIRECTORIES OK : dev_mpi_omp/16mpi_8omp = dev_mpi_omp/32mpi_1omp 
     177results.out.dev_mpi_omp:DIRECTORIES OK : dev_mpi_omp/16mpi_8omp = dev_mpi_omp/32mpi_2omp 
     178results.out.prod_mpi:DIRECTORIES DIFFER : prod_mpi/23mpi NOT = prod_mpi/32mpi 
     179results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/16mpi_8omp = prod_mpi_omp/32mpi_1omp 
     180results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/16mpi_8omp = prod_mpi_omp/32mpi_2omp 
     181 
     182 
     183 
     184LMDZORChoi at ada : 1 simulation not finished debug : 16mpi_8omp => after relaunch still not ok! 
     185                  Different number of cores mpi or omp give the same results.  
     186******************************************************************************* 
     187 
     188results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi_omp/32mpi_1omp 
     189results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = dev_mpi/23mpi 
     190results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = dev_mpi_omp/16mpi_8omp 
     191results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = prod_mpi/23mpi 
     192results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = prod_mpi_omp/16mpi_8omp 
     193results.out.cross:DIRECTORIES OK : prod_mpi/23mpi = prod_mpi_omp/16mpi_8omp 
     194results.out.cross:DIRECTORIES DIFFER : dev_mpi/23mpi NOT = dev_mpi_omp/16mpi_8omp 
     195results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi_omp/32mpi_1omp 
     196 
     197results.out.debug_mpi:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi/32mpi 
     198results.out.debug_mpi_omp:DIRECTORIES OK : debug_mpi_omp/32mpi_1omp = debug_mpi_omp/32mpi_2omp 
     199results.out.dev_mpi:DIRECTORIES OK : dev_mpi/23mpi = dev_mpi/32mpi 
     200results.out.dev_mpi_omp:DIRECTORIES OK : dev_mpi_omp/16mpi_8omp = dev_mpi_omp/32mpi_1omp 
     201results.out.dev_mpi_omp:DIRECTORIES OK : dev_mpi_omp/16mpi_8omp = dev_mpi_omp/32mpi_2omp 
     202results.out.prod_mpi:DIRECTORIES OK : prod_mpi/23mpi = prod_mpi/32mpi 
     203results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/16mpi_8omp = prod_mpi_omp/32mpi_1omp 
     204results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/16mpi_8omp = prod_mpi_omp/32mpi_2omp 
     205 
     206 
     207LMDZORCWRR at curie: All simulations finished 
     208                    Different number of cores mpi DO NOT give the same results with mpi mode. For debug compiling it's ok.  
     209                    Different number of cores mpi or omp in hybrid mode give the same results 
     210******************************************************************************* 
     211results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi_omp/16mpi_8omp 
     212results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = dev_mpi/23mpi 
     213results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = dev_mpi_omp/16mpi_8omp 
     214results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = prod_mpi/23mpi 
     215results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = prod_mpi_omp/16mpi_8omp 
     216results.out.cross:DIRECTORIES DIFFER : prod_mpi/23mpi NOT = prod_mpi_omp/16mpi_8omp 
     217results.out.cross:DIRECTORIES DIFFER : dev_mpi/23mpi NOT = dev_mpi_omp/16mpi_8omp 
     218results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi_omp/16mpi_8omp 
     219 
     220results.out.debug_mpi:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi/32mpi 
     221results.out.debug_mpi_omp:DIRECTORIES OK : debug_mpi_omp/16mpi_8omp = debug_mpi_omp/32mpi_1omp 
     222results.out.debug_mpi_omp:DIRECTORIES OK : debug_mpi_omp/16mpi_8omp = debug_mpi_omp/32mpi_2omp 
     223results.out.dev_mpi:DIRECTORIES DIFFER : dev_mpi/23mpi NOT = dev_mpi/32mpi 
     224results.out.dev_mpi_omp:DIRECTORIES OK : dev_mpi_omp/16mpi_8omp = dev_mpi_omp/32mpi_1omp 
     225results.out.dev_mpi_omp:DIRECTORIES OK : dev_mpi_omp/16mpi_8omp = dev_mpi_omp/32mpi_2omp 
     226results.out.prod_mpi:DIRECTORIES DIFFER : prod_mpi/23mpi NOT = prod_mpi/32mpi 
     227results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/16mpi_8omp = prod_mpi_omp/32mpi_1omp 
     228results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/16mpi_8omp = prod_mpi_omp/32mpi_2omp 
     229 
     230 
     231 
     232 
     233LMDZORCWRR at ada : 1 simulation not finish: debug 16mpi_8omp => after relanch still not ok. Pb memoire ? 
     234                  Different number of cores mpi or omp give the same results.  
     235******************************************************************************* 
     236results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi_omp/32mpi_1omp 
     237results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = dev_mpi/23mpi 
     238results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = dev_mpi_omp/16mpi_8omp 
     239results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = prod_mpi/23mpi 
     240results.out.cross:DIRECTORIES DIFFER : debug_mpi/23mpi NOT = prod_mpi_omp/16mpi_8omp 
     241results.out.cross:DIRECTORIES OK : prod_mpi/23mpi = prod_mpi_omp/16mpi_8omp 
     242results.out.cross:DIRECTORIES DIFFER : dev_mpi/23mpi NOT = dev_mpi_omp/16mpi_8omp 
     243results.out.cross:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi_omp/32mpi_1omp 
     244 
     245results.out.debug_mpi:DIRECTORIES OK : debug_mpi/23mpi = debug_mpi/32mpi 
     246results.out.debug_mpi_omp:DIRECTORIES OK : debug_mpi_omp/32mpi_1omp = debug_mpi_omp/32mpi_2omp 
     247results.out.dev_mpi:DIRECTORIES OK : dev_mpi/23mpi = dev_mpi/32mpi 
     248results.out.dev_mpi_omp:DIRECTORIES OK : dev_mpi_omp/16mpi_8omp = dev_mpi_omp/32mpi_1omp 
     249results.out.dev_mpi_omp:DIRECTORIES OK : dev_mpi_omp/16mpi_8omp = dev_mpi_omp/32mpi_2omp 
     250results.out.prod_mpi:DIRECTORIES OK : prod_mpi/23mpi = prod_mpi/32mpi 
     251results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/16mpi_8omp = prod_mpi_omp/32mpi_1omp 
     252results.out.prod_mpi_omp:DIRECTORIES OK : prod_mpi_omp/16mpi_8omp = prod_mpi_omp/32mpi_2omp 
     253}}}