Changeset 2364 for branches/nemo_v3_3_beta/DOC
- Timestamp:
- 2010-11-05T16:22:12+01:00 (14 years ago)
- Location:
- branches/nemo_v3_3_beta/DOC/TexFiles
- Files:
-
- 1 added
- 1 edited
Legend:
- Unmodified
- Added
- Removed
-
branches/nemo_v3_3_beta/DOC/TexFiles/Chapters/Chap_MISC.tex
r2349 r2364 508 508 % Standard Model Output 509 509 % ------------------------------------------------------------------------------------------------------------- 510 \subsection{Model Output (default or \key{iomput} or \key{dimgout} )}510 \subsection{Model Output (default or \key{iomput} or \key{dimgout} or \key{netcdf4})} 511 511 \label{MISC_iom} 512 512 … … 536 536 an IEEE output format. 537 537 538 Since version 3.3, support for NetCDF4 chunking and (loss-less) compression has 539 been included. These options build on the standard NetCDF output and allow 540 the user control over the size of the chunks via namelist settings. Chunking 541 and compression can lead to significant reductions in file sizes for a small 542 runtime overhead. For a fuller discussion on chunking and other performance 543 issues the reader is referred to the NetCDF4 documentation: 544 http://www.unidata.ucar.edu/software/netcdf/docs/netcdf.html\#Chunking 545 546 The new features are only available when the code has been linked with a 547 NetCDF4 library (version 4.1 onwards, recommended) which has been built 548 with HDF5 support (version 1.8.4 onwards, recommended). Datasets created 549 with chunking and compression are not backwards compatible with NetCDF3 550 "classic" format but most analysis codes can be relinked simply with the 551 new libraries and will then read both NetCDF3 and NetCDF4 files. NEMO 552 executables linked with NetCDF4 libraries can be made to produce NetCDF3 553 files by setting the \np{ln\_nc4zip} logical to false in the \np{namnc4} 554 namelist: 555 556 %------------------------------------------namnc4---------------------------------------------------- 557 \namdisplay{namnc4} 558 %------------------------------------------------------------------------------------------------------------- 559 560 If \key{netcdf4} has not been defined, these namelist parameters are not read. 561 In this case, \np{ln\_nc4zip} is set false and dummy routines for a few 562 NetCDF4-specific functions are defined. These functions will not be used but 563 need to be included so that compilation is possible with NetCDF3 libraries. 564 565 When using NetCDF4 libraries, \key{netcdf4} should be defined even if the 566 intention is to create only NetCDF3-compatible files. This is necessary to 567 avoid duplication between the dummy routines and the actual routines present 568 in the library. Most compilers will fail at compile time when faced with 569 such duplication. Thus when linking with NetCDF4 libraries the user must 570 define \key{netcdf4} and control the type of NetCDF file produced via the 571 namelist parameter. 572 573 Chunking and compression is applied only to 4D fields and there is no 574 advantage in chunking across more than one time dimension since previously 575 written chunks would have to be read back and decompressed before being 576 added to. Therefore, user control over chunk sizes is provided only for the 577 three space dimensions. The user sets an approximate number of chunks along 578 each spatial axis. The actual size of the chunks will depend on global domain 579 size for mono-processors or, more likely, the local processor domain size for 580 distributed processing. The derived values are subject to practical minimum 581 values (to avoid wastefully small chunk sizes) and cannot be greater than the 582 domain size in any dimension. The algorithm used is: 583 584 \begin{alltt} {{\tiny 585 \begin{verbatim} 586 ichunksz(1) = MIN( idomain_size,MAX( (idomain_size-1)/nn_nchunks_i + 1 ,16 ) ) 587 ichunksz(2) = MIN( jdomain_size,MAX( (jdomain_size-1)/nn_nchunks_j + 1 ,16 ) ) 588 ichunksz(3) = MIN( kdomain_size,MAX( (kdomain_size-1)/nn_nchunks_k + 1 , 1 ) ) 589 ichunksz(4) = 1 590 \end{verbatim} 591 }}\end{alltt} 592 593 \noindent As an example, setting: 594 \vspace{-20pt} 595 \begin{alltt} {{\tiny 596 \begin{verbatim} 597 nn_nchunks_i=4, nn_nchunks_j=4 and nn_nchunks_k=31 598 \end{verbatim} 599 }}\end{alltt} \vspace{-10pt} 600 601 \noindent for a standard ORCA2\_LIM configuration gives chunksizes of {\small\tt 46x38x1} 602 respectively in the mono-processor case (i.e. global domain of {\small\tt 182x149x31}). 603 An illustration of the potential space savings that NetCDF4 chunking and compression 604 provides is given in table \ref{Tab_NC4} which compares the results of two short 605 runs of the ORCA2\_LIM reference configuration with a 4x2 mpi partitioning. Note 606 the variation in the compression ratio achieved which reflects chiefly the dry to wet 607 volume ratio of each processing region. 608 609 \begin{table} 610 \begin{tabular}{lrrr} 611 Filename & NetCDF3 & NetCDF4 & Reduction\\ 612 &filesize & filesize & \% \\ 613 &(KB) & (KB) & \\ 614 ORCA2\_restart\_0000.nc & 16420 & 8860 & 47\%\\ 615 ORCA2\_restart\_0001.nc & 16064 & 11456 & 29\%\\ 616 ORCA2\_restart\_0002.nc & 16064 & 9744 & 40\%\\ 617 ORCA2\_restart\_0003.nc & 16420 & 9404 & 43\%\\ 618 ORCA2\_restart\_0004.nc & 16200 & 5844 & 64\%\\ 619 ORCA2\_restart\_0005.nc & 15848 & 8172 & 49\%\\ 620 ORCA2\_restart\_0006.nc & 15848 & 8012 & 50\%\\ 621 ORCA2\_restart\_0007.nc & 16200 & 5148 & 69\%\\ 622 ORCA2\_2d\_grid\_T\_0000.nc & 2200 & 1504 & 32\%\\ 623 ORCA2\_2d\_grid\_T\_0001.nc & 2200 & 1748 & 21\%\\ 624 ORCA2\_2d\_grid\_T\_0002.nc & 2200 & 1592 & 28\%\\ 625 ORCA2\_2d\_grid\_T\_0003.nc & 2200 & 1540 & 30\%\\ 626 ORCA2\_2d\_grid\_T\_0004.nc & 2200 & 1204 & 46\%\\ 627 ORCA2\_2d\_grid\_T\_0005.nc & 2200 & 1444 & 35\%\\ 628 ORCA2\_2d\_grid\_T\_0006.nc & 2200 & 1428 & 36\%\\ 629 ORCA2\_2d\_grid\_T\_0007.nc & 2200 & 1148 & 48\%\\ 630 ... & ... & ... & .. \\ 631 ORCA2\_2d\_grid\_W\_0000.nc & 4416 & 2240 & 50\%\\ 632 ORCA2\_2d\_grid\_W\_0001.nc & 4416 & 2924 & 34\%\\ 633 ORCA2\_2d\_grid\_W\_0002.nc & 4416 & 2512 & 44\%\\ 634 ORCA2\_2d\_grid\_W\_0003.nc & 4416 & 2368 & 47\%\\ 635 ORCA2\_2d\_grid\_W\_0004.nc & 4416 & 1432 & 68\%\\ 636 ORCA2\_2d\_grid\_W\_0005.nc & 4416 & 1972 & 56\%\\ 637 ORCA2\_2d\_grid\_W\_0006.nc & 4416 & 2028 & 55\%\\ 638 ORCA2\_2d\_grid\_W\_0007.nc & 4416 & 1368 & 70\%\\ 639 \end{tabular} 640 \caption{\label{Tab_NC4} Filesize comparison between NetCDF3 and NetCDF4 641 with chunking and compression} 642 \end{table} 643 538 644 Since version 3.2, an I/O server has been added which provides more 539 645 flexibility in the choice of the fields to be output as well as how the 540 646 writing work is distributed over the processors in massively parallel 541 computing. It is activated when \key{iomput} is defined. 647 computing. It is activated when \key{iomput} is defined. 648 649 When \key{iomput} is activated with \key{netcdf4} chunking and 650 compression parameters for fields produced via \np{iom\_put} calls are 651 set via an equivalent and identically named namelist to \np{namnc4} in 652 \np{xmlio\_server.def}. Typically this namelist serves the mean files 653 whilst the \np{ namnc4} in the main namelist file continues to serve the 654 restart files. This duplication is unfortunate but appropriate since, if 655 using io\_servers, the domain sizes of the individual files produced by the 656 io\_server processes may be different to those produced by the invidual 657 processing regions and different chunking choices may be desired. 658 { 542 659 543 660 % -------------------------------------------------------------------------------------------------------------
Note: See TracChangeset
for help on using the changeset viewer.