NCO is the result of software needs that arose while I worked
on projects funded by NCAR, NASA, and ARM.
Thinking they might prove useful as tools or templates to others,
it is my pleasure to provide them freely to the scientific community.
Many users (most of whom I have never met) have encouraged the
development of NCO.
Thanks espcially to Jan Polcher, Keith Lindsay, Arlindo da Silva, John
Sheldon, and William Weibel for stimulating suggestions and
correspondence.
Your encouragment motivated me to complete the NCO User's Guide.
So if you like NCO, send me a note!
I should mention that NCO is not connected to or
officially endorsed by Unidata, ACD, ASP,
CGD, or Nike.
Charlie Zender
Major feature improvements entitle me to write another Foreword. In the last five years a lot of work has been done refining NCO. NCO is now an open source project and appears to be much healthier for it. The list of illustrious institutions which do not endorse NCO continues to grow, and now includes UCI.
Charlie Zender
The most remarkable advances in NCO capabilities in the last few years are due to contributions from the Open Source community. Especially noteworthy are the contributions of Henry Butowsky and Rorik Peterson.
Charlie Zender
This manual describes NCO, which stands for netCDF Operators. NCO is a suite of programs known as operators. Each operator is a standalone, command line program executed at the shell-level like, e.g., ls or mkdir. The operators take netCDF file(s) (or HDF4 files) as input, perform an operation (e.g., averaging or hyperslabbing), and produce a netCDF file as output. The operators are primarily designed to aid manipulation and analysis of data. The examples in this documentation are typical applications of the operators for processing climate model output. This reflects their origin, but the operators are as general as netCDF itself.
The complete NCO source distribution is currently distributed
as a compressed tarfile from
http://sf.net/projects/nco
and from
http://dust.ess.uci.edu/nco/nco.tar.gz.
The compressed tarfile must be uncompressed and untarred before building
NCO.
Uncompress the file with gunzip nco.tar.gz.
Extract the source files from the resulting tarfile with tar -xvf
nco.tar.
GNU tar lets you perform both operations in one step
with tar -xvzf nco.tar.gz.
The documentation for NCO is called the NCO User's Guide. The User's Guide is available in Postscript, HTML, DVI, TeXinfo, and Info formats. These formats are included in the source distribution in the files nco.ps, nco.html, nco.dvi, nco.texi, and nco.info*, respectively. All the documentation descends from a single source file, nco.texi 1. Hence the documentation in every format is very similar. However, some of the complex mathematical expressions needed to describe ncwa can only be displayed in the Postscript and DVI formats.
If you want to quickly see what the latest improvements in NCO are (without downloading the entire source distribution), visit the NCO homepage at http://nco.sf.net. The HTML version of the User's Guide is also available online through the World Wide Web at URL http://nco.sf.net/nco.html. To build and use NCO, you must have netCDF installed. The netCDF homepage is http://www.unidata.ucar.edu/packages/netcdf.
New NCO releases are announced on the netCDF list and on the
nco-announce mailing list
http://lists.sf.net/mailman/listinfo/nco-announce.
NCO has been successfully ported and tested and is known to work on the following 32 and 64 bit platforms: IBM AIX 4.x, 5.x, FreeBSD 4.x, GNU/Linux 2.x, LinuxPPC, LinuxAlpha, LinuxSparc64, SGI IRIX 5.x and 6.x, MacOS X 10.x, NEC Super-UX 10.x, DEC OSF, Sun SunOS 4.1.x, Solaris 2.x, Cray UNICOS 8.x–10.x, all MS Windows. If you port the code to a new operating system, please send me a note and any patches you required.
The major prerequisite for installing NCO on a particular platform is the successful, prior installation of the netCDF library (and, as of 2003, the UDUnits library). Unidata has shown a commitment to maintaining netCDF and UDUnits on all popular UNIX platforms, and is moving towards full support for the Microsoft Windows operating system (OS). Given this, the only difficulty in implementing NCO on a particular platform is standardization of various C and Fortran interface and system calls. NCO code is tested for ANSI compliance by compiling with C compilers including those from GNU (gcc -std=c99 -pedantic -D_BSD_SOURCE -Wall) 2, Comeau Computing (como --c99), Cray (cc), HP/Compaq/DEC (cc), IBM (xlc -c -qlanglvl=extended), Intel (icc), NEC (cc), SGI (cc -LANG:std), and Sun (cc). NCO (all commands and the libnco library) and the C++ interface to netCDF (called libnco_c++) comply with the ISO C++ standards as implemented by Comeau Computing (como), Cray (CC), GNU (g++ -Wall), HP/Compaq/DEC (cxx), IBM (xlC), Intel (icc), NEC (c++), SGI (CC -LANG:std), and Sun (CC -LANG:std). See nco/bld/Makefile and nco/src/nco_c++/Makefile.old for more details and exact settings.
Until recently (and not even yet), ANSI-compliant has meant
compliance with the 1989 ISO C-standard, usually called C89 (with
minor revisions made in 1994 and 1995).
C89 lacks variable-size arrays, restricted pointers, and some useful
printf formats.
These are valuable features of C99, the 1999 ISO C-standard.
NCO is C99-compliant where possible and C89-compliant where
necessary.
Certain branches in the code are required to satisfy the native
SGI and SunOS C compilers, which are strictly ANSI
C89 compliant, and cannot benefit from C99 features.
However, C99 features are fully supported by modern AIX,
GNU, Intel, NEC, Solaris, and UNICOS
compilers.
NCO requires a C99-compliant compiler as of NCO
version 2.9.8, released in August, 2004.
The most time-intensive portion of NCO execution is spent in
arithmetic operations, e.g., multiplication, averaging, subtraction.
These operations were performed in Fortran by default until August,
1999.
This was a design decision based on the relative speed of Fortran-based
object code vs. C-based object code in late 1994.
C compilers have dramatically improved their vectorization
capabilities since 1994.
We have accordingly replaced all Fortran subroutines with C functions.
This greatly simplifies the task of building NCO on nominally
unsupported platforms.
As of August 1999, NCO built entirely in C by default.
This allowed NCO to compile on any machine with an
ANSI C compiler.
In August 2004, the first C99 feature, the restrict type
qualifier, entered NCO in version 2.9.8.
C compilers can obtain better performance with C99 restricted
pointers since they inform the compiler when it may make Fortran-like
assumptions regarding pointer contents alteration.
Subsequently, NCO requires a C99 compiler to build correctly
3.
Furthermore, NCO automagically takes advantage of extensions
to ANSI C when compiled with the GNU compiler
collection, GCC.
As of July 2000 and NCO version 1.2, NCO no
longer supports performing arithmetic operations in Fortran.
We decided to sacrifice executable speed for code maintainability
Since no objective statistics were ever performed to quantify
the difference in speed between the Fortran and C code,
the performance penalty incurred by this decision is unknown.
Supporting Fortran involves maintaining two sets of routines for every
arithmetic operation.
The USE_FORTRAN_ARITHMETIC flag is still retained in the
Makefile.
The file containing the Fortran code, nco_fortran.F, has been
deprecated but can be resurrected if a volunteer comes forward.
If you would like to volunteer to maintain nco_fortran.F please
contact me.
NCO has been successfully ported and tested on the Microsoft
Windows (95/98/NT/2000/XP) operating systems.
The switches necessary to accomplish this are included in the standard
distribution of NCO.
Using the freely available Cygwin (formerly gnu-win32) development
environment
4, the compilation process is very similar to
installing NCO on a UNIX system.
Set the PVM_ARCH preprocessor token to WIN32.
Note that defining WIN32 has the side effect of disabling
Internet features of NCO (see below).
Unless you have a Fortran compiler (like g77 or f90)
available, no other tokens are required.
Users with fast Fortran compilers may wish to activate the Fortran
arithmetic routines.
To do this, define the preprocessor token USE_FORTRAN_ARITHMETIC
in the makefile which comes with NCO, Makefile, or in
the compilation shell.
The least portable section of the code is the use of standard
UNIX and Internet protocols (e.g., ftp, rcp,
scp, getuid, gethostname, and header files
<arpa/nameser.h> and
<resolv.h>).
Fortunately, these UNIX-y calls are only invoked by the single
NCO subroutine which is responsible for retrieving files
stored on remote systems (see Remote storage).
In order to support NCO on the Microsoft Windows platforms,
this single feature was disabled (on Windows OS only).
This was required by Cygwin 18.x—newer versions of Cygwin may
support these protocols (let me know if this is the case).
The NCO operators should behave identically on Windows and
UNIX platforms in all other respects.
Like all executables, the NCO operators can be built using dynamic linking. This reduces the size of the executable and can result in significant performance enhancements on multiuser systems. Unfortunately, if your library search path (usually the LD_LIBRARY_PATH environment variable) is not set correctly, or if the system libraries have been moved, renamed, or deleted since NCO was installed, it is possible NCO operators will fail with a message that they cannot find a dynamically loaded (aka shared object or .so) library. This will produce a distinctive error message, such as ld.so.1: /usr/local/bin/ncea: fatal: libsunmath.so.1: can't open file: errno=2. If you received an error message like this, ask your system administrator to diagnose whether the library is truly missing 5, or whether you simply need to alter your library search path. As a final remedy, you may re-compile and install NCO with all operators statically linked.
netCDF version 2.x was released in 1993.
NCO (specifically ncks) began soon after this in 1994.
netCDF 3.0 was released in 1996, and we were eager to reap the
performance advantages of the newer netCDF implementation.
One netCDF 3.x interface call (nc_inq_libvers) was added to
NCO in January, 1998, to aid in maintainance and debugging.
In March, 2001, the final conversion of NCO to netCDF 3.x
was completed (coincidentally on the same day netCDF 3.5 was
released).
NCO versions 2.0 and higher are built with the
-DNO_NETCDF_2 flag to ensure no netCDF 2.x interface calls
are used.
However, the ability to compile NCO with only netCDF 2.x
calls is worth maintaining because HDF version 4
6
(available from HDF)
supports only the netCDF 2.x library calls
(see http://hdf.ncsa.uiuc.edu/UG41r3_html/SDS_SD.fm12.html#47784).
Note that there are multiple versions of HDF.
Currently HDF version 4.x supports netCDF 2.x and thus
NCO version 1.2.x.
If NCO version 1.2.x (or earlier) is built with only
netCDF 2.x calls then all NCO operators should work with
HDF4 files as well as netCDF files
7.
The preprocessor token NETCDF2_ONLY exists
in NCO version 1.2.x to eliminate all netCDF 3.x
calls.
Only versions of NCO numbered 1.2.x and earlier have this
capability.
The NCO 1.2.x branch will be maintained with bugfixes only
(no new features) until HDF begins to fully support the
netCDF 3.x interface (which is employed by NCO 2.x).
If, at compilation time, NETCDF2_ONLY is defined, then
NCO version 1.2.x will not use any netCDF 3.x calls
and, if linked properly, the resulting NCO operators will work
with HDF4 files.
The Makefile supplied with NCO 1.2.x is written
to simplify building in this HDF capability.
When NCO is built with make HDF4=Y, the Makefile
sets all required preprocessor flags and library links to build
with the HDF4 libraries (which are assumed to reside under
/usr/local/hdf4, edit the Makefile to suit your
installation).
HDF version 5.x became available in 1999, but did not support netCDF (or, for that matter, Fortran) as of December 1999. By early 2001, HDF version 5.x did support Fortran90. However, support for netCDF 3.x in HDF 5.x is incomplete. Much of the HDF5-netCDF3 interface is complete, however, and it may be separately downloaded from the HDF5-netCDF website. Now that NCO uses only netCDF 3.x system calls we are eager for HDF5 to add complete netCDF 3.x support. This is scheduled to occur sometime in 2005, with the release of netCDF version 4, a collaboration between Unidata and NCSA.
We generally receive three categories of mail from users: help requests, bug reports, and feature requests. Notes saying the equivalent of "Hey, NCO continues to work great and it saves me more time everyday than it took to write this note" are a distant fourth.
There is a different protocol for each type of request. The common procedural etiquette is to communicate with us via NCO Project Forums. Unless your request comes with money or you have damaging information about our personal lives, use the Forums. They preserve a record of the questions and answers so that others might learn from our exchange. Also, should NCO ever be funded (as is our hope), this record will provide program officers with information they need to evaluate us.
Before posting to the NCO forums described below, you might first register your name and email address with SourceForge.net or else all of your postings will be attributed to "nobody". Once registered you may choose to "monitor" any forum and to receive (or not) email when there are any postings including responses to your questions. We usually reply to the forum message, not to the original poster.
If you want us to include a new feature in NCO, check first to see if that feature is already on the TODO list. If it is, why not implement that feature yourself and send us the patch? If the feature is not yet on the list, then send a note to the NCO Discussion forum.
Read the manual before reporting a bug or posting a help request. Sending questions whose answers are not in the manual is the best way to motivate us to write more documentation. We would also like to accentuate the contrapositive of this statement. If you think you have found a real bug the most helpful thing you can do is simplify the problem to a manageable size and then report it. The first thing to do is to make sure you are running the latest publicly released version of NCO.
Once you have read the manual, if you are still unable to get NCO to perform a documented function, write help request. Follow the same procedure as described below for reporting bugs (after all, it might be a bug). That is, describe what you are trying to do, and include the complete commands (run with -D 5), error messages, and version of NCO (with -r). Post your help request to the NCO Help forum.
If you think you are using the right command, but NCO is misbehaving, then you might have found a bug. Incorrect numerical answers are the highest priority. We usually fix those within one or two days. Core dumps and sementation violations receive lower priority. They are always fixed.
How do you simplify a problem that may be revealing a bug? Cut out extraneous variables, dimensions, and metadata from the offending files and re-run the command until it no longer breaks. Then back up one step and report the problem. Usually the file(s) will be very small, i.e., one variable with one or two small dimensions ought to suffice. Run the operator with -r and then run the command with -D 5 to increase the verbosity of the debugging output. It is very important that your report contain the exact error messages and compile-time environment. Include a copy of your sample input file, or place one on a publically accessible location, of the file(s). Post the full bug report to the NCO Project buglist.
Build failures count as bugs.
Our limited machine access means we cannot fix all build failures.
The information we need to diagnose, and often fix, build failures
are the three files output by GNU build tools,
nco.config.log.${GNU_TRP}.foo,
nco.configure.${GNU_TRP}.foo,
and nco.make.${GNU_TRP}.foo.
The file configure.eg shows how to produce these files.
Here ${GNU_TRP} is the "GNU architecture triplet",
the chip-vendor-OS string returned by config.guess.
Please send us your improvements to the examples supplied in
configure.eg.
The regressions archive at http://dust.ess.uci.edu/nco/rgr
contains the build output from our standard test systems.
You may find you can solve the build problem yourself by examining the
differences between these files and your own.
The main design goal has been to produce operators that can be invoked from the command line to perform useful operations on netCDF files. Many scientists work with models and observations which produce too much data to analyze in tabular format. Thus, it is often natural to reduce and massage this raw or primary level data into summary, or second level data, e.g., temporal or spatial averages. These second level data may become the inputs to graphical and statistical packages, and are often more suitable for archival and dissemination to the scientific community. NCO performs a suite of operations useful in manipulating data from the primary to the second level state. Higher level interpretive languages (e.g., IDL, Yorick, Matlab, NCL, Perl, Python), and lower level compiled languages (e.g., C, Fortran) can always perform any task performed by NCO, but often with more overhead. NCO, on the other hand, is limited to a much smaller set of arithmetic and metadata operations than these full blown languages.
Another goal has been to implement enough command line switches so that frequently used sequences of these operators can be executed from a shell script or batch file. Finally, NCO was written to consume the absolute minimum amount of system memory required to perform a given job. The arithmetic operators are extremely efficient; their exact memory usage is detailed in Memory usage.
NCO was developed at NCAR to aid analysis and manipulation of datasets produced by General Circulation Models (GCMs). Datasets produced by GCMs share many features with all gridded scientific datasets and so provide a useful paradigm for the explication of the NCO operator set. Examples in this manual use a GCM paradigm because latitude, longitude, time, temperature and other fields related to our natural environment are as easy to visualize for the layman as the expert.
NCO operators are designed to be reasonably fault tolerant, so
that if there is a system failure or the user aborts the operation (e.g.,
with C-c), then no data are lost.
The user-specified output-file is only created upon successful
completion of the operation
8.
This is accomplished by performing all operations in a temporary copy
of output-file.
The name of the temporary output file is constructed by appending
.pid<process ID>.<operator name>.tmp to the
user-specified output-file name.
When the operator completes its task with no fatal errors, the temporary
output file is moved to the user-specified output-file.
Note the construction of a temporary output file uses more disk space
than just overwriting existing files “in place” (because there may be
two copies of the same file on disk until the NCO operation
successfully concludes and the temporary output file overwrites the
existing output-file).
Also, note this feature increases the execution time of the operator
by approximately the time it takes to copy the output-file.
Finally, note this feature allows the output-file to be the same
as the input-file without any danger of “overlap”.
Other safeguards exist to protect the user from inadvertently overwriting data. If the output-file specified for a command is a pre-existing file, then the operator will prompt the user whether to overwrite (erase) the existing output-file, attempt to append to it, or abort the operation. However, in processing large amounts of data, too many interactive questions slows productivity. Therefore NCO also implements two ways to override its own safety features, the -O and -A switches. Specifying -O tells the operator to overwrite any existing output-file without prompting the user interactively. Specifying -A tells the operator to attempt to append to any existing output-file without prompting the user interactively. These switches are useful in batch environments because they suppress interactive keyboard input.
A frequently useful operation is adding variables from one file to another. This is referred to as appending, although some prefer the terminology merging 9 or pasting. Appending is often confused with what NCO calls concatenation. In NCO, concatenation refers to splicing a variable along the record dimension. Appending, on the other hand, refers to adding variables from one file to another 10. In this sense, ncks can append variables from one file to another file. This capability is invoked by naming two files on the command line, input-file and output-file. When output-file already exists, the user is prompted whether to overwrite, append/replace, or exit from the command. Selecting overwrite tells the operator to erase the existing output-file and replace it with the results of the operation. Selecting exit causes the operator to exit—the output-file will not be touched in this case. Selecting append/replace causes the operator to attempt to place the results of the operation in the existing output-file, See ncks netCDF Kitchen Sink.
The simplest way to create the union of two files is
ncks -A fl_1.nc fl_2.nc
This puts the contents of fl_1.nc into fl_2.nc. The -A is optional. On output, fl_2.nc is the union of the input files, regardless of whether they share dimensions and variables, or are completely disjoint. The append fails if the input files have differently named record dimensions (since netCDF supports only one), or have dimensions of the same name but different sizes.
Users comfortable with NCO semantics may find it easier to perform some simple mathematical operations in NCO rather than higher level languages. ncbo (see ncbo netCDF Binary Operator) does file addition, subtraction, multiplication, division, and broadcasting. ncflint (see ncflint netCDF File Interpolator) does file addition, subtraction, multiplication and interpolation. Sequences of these commands can accomplish simple but powerful operations from the command line.
The most frequently used operators of NCO are probably the averagers and concatenators. Because there are so many permutations of averaging (e.g., across files, within a file, over the record dimension, over other dimensions, with or without weights and masks) and of concatenating (across files, along the record dimension, along other dimensions), there are currently no fewer than five operators which tackle these two purposes: ncra, ncea, ncwa, ncrcat, and ncecat. These operators do share many capabilities 11, but each has its unique specialty. Two of these operators, ncrcat and ncecat, are for concatenating hyperslabs across files. The other two operators, ncra and ncea, are for averaging hyperslabs across files 12. First, let's describe the concatenators, then the averagers.
Joining independent files together along a record dimension is called
concatenation.
ncrcat is designed for concatenating record variables, while
ncecat is designed for concatenating fixed length variables.
Consider five files, 85.nc, 86.nc,
... 89.nc each containing a year's worth of data.
Say you wish to create from them a single file, 8589.nc
containing all the data, i.e., spanning all five years.
If the annual files make use of the same record variable, then
ncrcat will do the job nicely with, e.g.,
ncrcat 8?.nc 8589.nc.
The number of records in the input files is arbitrary and can vary from
file to file.
See ncrcat netCDF Record Concatenator, for a complete description of
ncrcat.
However, suppose the annual files have no record variable, and thus
their data are all fixed length.
For example, the files may not be conceptually sequential, but rather
members of the same group, or ensemble.
Members of an ensemble may have no reason to contain a record dimension.
ncecat will create a new record dimension (named record
by default) with which to glue together the individual files into the
single ensemble file.
If ncecat is used on files which contain an existing record
dimension, that record dimension is converted to a fixed-length
dimension of the same name and a new record dimension (named
record) is created.
Consider five realizations, 85a.nc, 85b.nc,
... 85e.nc of 1985 predictions from the same climate
model.
Then ncecat 85?.nc 85_ens.nc glues the individual realizations
together into the single file, 85_ens.nc.
If an input variable was dimensioned [lat,lon], it will
have dimensions [record,lat,lon] in the output file.
A restriction of ncecat is that the hyperslabs of the
processed variables must be the same from file to file.
Normally this means all the input files are the same size, and contain
data on different realizations of the same variables.
See ncecat netCDF Ensemble Concatenator, for a complete description
of ncecat.
ncpdq makes it possible to concatenate files along any
dimension, not just the record dimension.
First, use ncpdq to convert the dimension to be concatenated
(i.e., extended with data from other files) into the record dimension.
Second, use ncrcat to concatenate these files.
Finally, if desirable, use ncpdq to revert to the original
dimensionality.
As a concrete example, say that files x_01.nc, x_02.nc,
... x_10.nc contain time-evolving datasets from spatially
adjacent regions.
The time and spatial coordinates are time and x, respectively.
Initially the record dimension is time.
Our goal is to create a single file that contains joins all the
spatially adjacent regions into one single time-evolving dataset.
for idx in 01 02 03 04 05 06 07 08 09 10; do # Bourne Shell
ncpdq -a x,time x_${idx}.nc foo_${idx}.nc # Make x record dimension
done
ncrcat foo_??.nc out.nc # Concatenate along x
ncpdq -a time,x out.nc out.nc # Revert to time as record dimension
Note that ncrcat will not concatenate fixed-length variables, whereas ncecat concatenates both fixed-length and record variables along a new record variable. To conserve system memory, use ncrcat where possible.
The differences between the averagers ncra and ncea are analogous to the differences between the concatenators. ncra is designed for averaging record variables from at least one file, while ncea is designed for averaging fixed length variables from multiple files. ncra performs a simple arithmetic average over the record dimension of all the input files, with each record having an equal weight in the average. ncea performs a simple arithmetic average of all the input files, with each file having an equal weight in the average. Note that ncra cannot average fixed-length variables, but ncea can average both fixed-length and record variables. To conserve system memory, use ncra rather than ncea where possible (e.g., if each input-file is one record long). The file output from ncea will have the same dimensions (meaning dimension names as well as sizes) as the input hyperslabs (see ncea netCDF Ensemble Averager, for a complete description of ncea). The file output from ncra will have the same dimensions as the input hyperslabs except for the record dimension, which will have a size of 1 (see ncra netCDF Record Averager, for a complete description of ncra).
ncflint can interpolate data between or two files. Since no other operators have this ability, the description of interpolation is given fully on the ncflint reference page (see ncflint netCDF File Interpolator). Note that this capability also allows ncflint to linearly rescale any data in a netCDF file, e.g., to convert between differing units.
Occasionally one desires to digest (i.e., concatenate or average)
hundreds or thousands of input files.
Unfortunately, data archives (e.g., NASA EOSDIS) may not
name netCDF files in a format understood by the -n loop
switch (see Specifying input files) that automagically generates
arbitrary numbers of input filenames.
The -n loop switch has the virtue of being concise,
and of minimizing the command line.
This helps keeps output file small since the command line is stored
as metadata in the history attribute
(see History attribute).
However, the -n loop switch is useless when there is no
simple, arithmetic pattern to the input filenames (e.g.,
h00001.nc, h00002.nc, ... h90210.nc).
Moreover, filename globbing does not work when the input files are too
numerous or their names are too lengthy (when strung together as a
single argument) to be passed by the calling shell to the NCO
operator
13.
When this occurs, the ANSI C-standard argc-argv
method of passing arguments from the calling shell to a C-program (i.e.,
an NCO operator) breaks down.
There are (at least) three alternative methods of specifying the input
filenames to NCO in environment-limited situations.
The recommended method for sending very large numbers (hundreds or
more, typically) of input filenames to the multi-file operators is
to use the standard input, aka stdin, feature of
UNIX.
# Pipe large numbers of filenames to stdin
/bin/ls | grep ${CASEID}_'......'.nc | ncecat -O -o foo.nc
This method avoids all constraints on command line size imposed by
the operating system.
A drawback to this method is that the history attribute
(see History attribute) does not record the name of any input
files since the names were not passed on the command line.
This makes determining the data provenance at a later date difficult.
To remedy this situation, multi-file operators store the number of
input files in the nco_input_file_number global attribute and the
input file list itself in the nco_input_file_list global attribute
(see Input file list attributes).
Although this does not preserve the exact command used to generate the
file, it does retains all the information required to reconstruct the
command and determine the data provenance.
A second option is to use the UNIX xargs command. This simple example selects as input to xargs all the filenames in the current directory that match a given pattern. For illustration, consider a user trying to average millions of files which each have a six character filename. If the shell buffer can not hold the results of the corresponding globbing operator, ??????.nc, then the filename globbing technique will fail. Instead we express the filename pattern as an extended regular expression, ......\.nc (see Variable subsetting). We use grep to filter the directory listing for this pattern and to pipe the results to xargs which, in turn, passes the matching filenames to an NCO multi-file operator, e.g., ncecat.
# Use xargs to transfer filenames on the command line
/bin/ls | grep ${CASEID}_'......'.nc | xargs -x ncecat -o foo.nc
The single quotes protect the only sensitive parts of the extended
regular expression (the grep argument), and allow shell
interpolation (the ${CASEID} variable substitution) to
proceed unhindered on the rest of the command.
xargs uses the UNIX pipe feature to append the
suitably filtered input file list to the end of the ncecat
command options.
The -o foo.nc switch ensures that the input files supplied by
xargs are not confused with the output file name.
xargs does, unfortunately, have its own limit (usually about
20,000 characters) on the size of command lines it can pass.
Give xargs the -x switch to ensure it dies if it
reaches this internal limit.
When this occurs, use either the stdin method above, or the
symbolic link presented next.
Even when its internal limits have not been reached, the xargs technique may not be sophisticated enough to handle all situations. A full scripting language like Perl can handle any level of complexity of filtering input filenames, and any number of filenames. The technique of last resort is write a script that creates symbolic links between the irregular input filenames and a set of regular, arithmetic filenames that -n loop switch understands. For example, the following Perl script a monotonically enumerated symbolic link to up to one million .nc files in a directory. If there are 999,999 netCDF files present, the links are named 000001.nc to 999999.nc:
# Create enumerated symbolic links
/bin/ls | grep \.nc | perl -e \
'$idx=1;while(<STDIN>){chop;symlink $_,sprintf("%06d.nc",$idx++);}'
ncecat -O -n 999999,6,1 000001.nc foo.nc
# Remove symbolic links when finished
/bin/rm ??????.nc
The -n loop option tells the NCO operator to
automatically generate the filnames of the symbolic links.
This circumvents any OS and shell limits on command line size.
The symbolic links are easily removed once NCO is finished.
One drawback to this method is that the history attribute
(see History attribute) retains the filename list of the symbolic
links, rather than the data files themselves.
This makes it difficult to determine the data provenance at a later date.
Large datasets are those files that are comparable in size to the amount of random access memory (RAM) in your computer. Many users of NCO work with files larger than 100 MB. Files this large not only push the current edge of storage technology, they present special problems for programs which attempt to access the entire file at once, such as ncea and ncecat. If you work with a 300 MB files on a machine with only 32 MB of memory then you will need large amounts of swap space (virtual memory on disk) and NCO will work slowly, or even fail. There is no easy solution for this. The best strategy is to work on a machine with sufficient amounts of memory and swap space. Since about 2004, many users have begun to produce or analyze files exceeding 2 GB in size. These users should familiarize themselves with NCO's Large File Support (LFS) capabilities (see Large File Support). The next section will increase your familiarity with NCO's memory requirements. With this knowledge you may re-design your data reduction approach to divide the problem into pieces solvable in memory-limited situations.
If your local machine has problems working with large files, try running
NCO from a more powerful machine, such as a network server.
Certain machine architectures, e.g., Cray UNICOS, have special
commands which allow one to increase the amount of interactive memory.
On Cray systems, try to increase the available memory with the
ilimit command.
If you get a memory-related core dump
(e.g., Error exit (core dumped)) on a GNU/Linux system,
try increasing the process-available memory with ulimit.
The speed of the NCO operators also depends on file size.
When processing large files the operators may appear to hang, or do
nothing, for large periods of time.
In order to see what the operator is actually doing, it is useful to
activate a more verbose output mode.
This is accomplished by supplying a number greater than 0 to the
-D debug-level (or --debug-level, or
--dbg_lvl) switch.
When the debug-level is nonzero, the operators report their
current status to the terminal through the stderr facility.
Using -D does not slow the operators down.
Choose a debug-level between 1 and 3 for most situations,
e.g., ncea -D 2 85.nc 86.nc 8586.nc.
A full description of how to estimate the actual amount of memory the
multi-file NCO operators consume is given in
Memory usage.
Many people use NCO on gargantuan files which dwarf the memory available (free RAM plus swap space) even on today's powerful machines. These users will want NCO to consume the absolute minimum peak memory possible so that their scripts do not have to tediously cut files into smaller pieces that fit into memory. We commend these greedy users for pushing NCO to its limits!
This section describes the memory NCO requires during operation. The required memory is based on the underlying algorithms. The description below is the memory usage per thread. Users with shared memory machines may use the threaded NCO operators (see OpenMP threading). The peak and sustained memory usage will scale accordingly, i.e., by the number of threads. Memory consumption patterns of all operators are similar, with the exception of ncap.
The multi-file operators currently comprise the record operators, ncra and ncrcat, and the ensemble operators, ncea and ncecat. The record operators require much less memory than the ensemble operators. This is because the record operators operate on one single record (i.e., time-slice) at a time, wherease the ensemble operators retrieve the entire variable into memory. Let MS be the peak sustained memory demand of an operator, FT be the memory required to store the entire contents of all the variables to be processed in an input file, FR be the memory required to store the entire contents of a single record of each of the variables to be processed in an input file, VR be the memory required to store a single record of the largest record variable to be processed in an input file, VT be the memory required to store the largest variable to be processed in an input file, VI be the memory required to store the largest variable which is not processed, but is copied from the initial file to the output file. All operators require MI = VI during the initial copying of variables from the first input file to the output file. This is the initial (and transient) memory demand. The sustained memory demand is that memory required by the operators during the processing (i.e., averaging, concatenation) phase which lasts until all the input files have been processed. The operators have the following memory requirements: ncrcat requires MS <= VR. ncecat requires MS <= VT. ncra requires MS = 2FR + VR. ncea requires MS = 2FT + VT. ncbo requires MS <= 2VT. ncpdq requires MS <= 2VT. ncflint requires MS <= 2VT. Note that only variables which are processed, i.e., averaged or concatenated, contribute to MS. Memory is never allocated to hold variables which do not appear in the output file (see Variable subsetting).
ncap has unique memory requirements due its ability to process arbitrarily long scripts of any complexity. All script acceptable to ncap are ultimately processed as a sequence of binary or unary operations. ncap requires MS <= 2VT under most conditions. An exception to this is when left hand casting (see Left hand casting) is used to stretch the size of derived variables beyond the size of any input variables. Let VC be the memory required to store the largest variable defined by left hand casting. In this case, MS <= 2VC.
ncap scripts are complete dynamic and may be of arbitrary length. A script that contains many thousands of operations, may uncover a slow memory leak even though each single operation consumes little additional memory. Memory leaks are usually identifiable by their memory usage signature. Leaks cause peak memory usage to increase monotonically with time regardless of script complexity. Slow leaks are very difficult to find. Sometimes a malloc() failure is the only noticeable clue to their existance. If you have good reasons to believe that a malloc() failure is ultimately due to an NCO memory leak (rather than inadequate RAM on your system), then we would be very interested in receiving a detailed bug report.
Many features have been implemented in more than one operator and are described here for brevity. The description of each feature is preceded by a box listing the operators for which the feature is implemented. Command line switches for a given feature are consistent across all operators wherever possible. If no “key switches” are listed for a feature, then that particular feature is automatic and cannot be controlled by the user.
|
Availability: All operators Short options: All Long options: All |
|
Availability: ncbo, ncea, ncpdq,
ncra, ncrcat, ncwa Short options: -t Long options: --thr_nbr, --threads, --omp_num_threads |
OMP_NUM_THREADS environment variable, if present, or from the
OS, if not.
NCO may modify thr_nbr according to its own internal
settings before it requests any threads from the system.
Certain operators contain hard-code limits to the number of threads they
request.
We base these limits on our experience and common sense, and to reduce
potentially wasteful system usage by inexperienced users.
For example, ncrcat is extremely I/O-intensive.
No one has demonstrated that threading improves ncrcat
performance so we restrict thr_nbr <= 1 for ncrcat.
Compute-intensive operators (ncwa and ncpdq)
are expected to benefit the most from threading.
The greatest increases in throughput due to threading will occur on
large dataset where each thread performs millions or more floating
point operations.
Otherwise, the system overhead of setting up threads may outweigh
the theoretical speed enhancements due to SMP parallelism.
However, no one has ever demonstrated that the SMP parallelism
scales well beyone four threads for these operators.
Hence we restrict thr_nbr <= 4 for all operators.
We encourage users to play with these limits (edit file
nco_omp.c) and send us their feedback.
Once the initial thr_nbr has been modified for any operator-specific limits, NCO requests the system to allocate a team of thr_nbr threads for the body of the code. The operating system then decides how many threads to allocate based on this request. Users may keep track of this information by running the operator with dbg_lvl > 0.
By default, operators with thread attach one global attribute to any
file they create or modify.
The nco_openmp_thread_number global attribute contains the
number of threads the operator used to process the input files.
This information helps to verify that the answers with threaded and
non-threaded operators are equal to within machine precision.
This information is also useful for benchmarking.
|
Availability: All operators Short options: All Long options: All |
|
Availability: All operators Short options: All Long options: All |
Extended options, also called long options, are implemented using the system-supplied getopt.h header file, if possible. This provides the getopt_long function to NCO 14.
The syntax of short options (single letter options) is -key value (dash-key-space-value). Here, key is the single letter option name, e.g., -D 2.
The syntax of long options (multi-letter options) is --long_name value (dash-dash-key-space-value), e.g., --dbg_lvl 2 or --long_name=value (dash-dash-key-equal-value), e.g., --dbg_lvl=2. Thus the following are all valid for the -D (short version) or --dbg_lvl (long version) command line option.
ncks -D 3 in.nc # Short option
ncks --dbg_lvl=3 in.nc # Long option, preferred form
ncks --dbg_lvl 3 in.nc # Long option, alternate form
The last example is preferred for two reasons. First, --dbg_lvl is more specific and less ambiguous than -D. The long option form makes scripts more self documenting and less error prone. Often long options are named after the source code variable whose value they carry. Second, the equals sign = joins the key (i.e., long_name) to the value in an uninterruptible text block. Experience shows that users are less likely to mis-parse commands when restricted to this form.
GNU implements a superset of the POSIX standard which allows any unambiguous truncation of a valid option to be used.
ncks -D 3 in.nc # Short option
ncks --dbg_lvl=3 in.nc # Long option, full form
ncks --dbg=3 in.nc # Long option, unambiguous truncation
ncks --db=3 in.nc # Long option, unambiguous truncation
ncks --d=3 in.nc # Long option, ambiguous truncation
The first four examples are equivalent and will work as expected. The final example will exit with an error since ncks cannot disambiguate whether --d is intended as a truncation of --dbg_lvl, of --dimension, or of some other long option.
NCO provides many long options for common switches. For example, the debugging level may be set in all operators with any of the switches -D, --debug-level, or --dbg_lvl. This flexibility allows users to choose their favorite mnemonic. For some, it will be --debug (an unambiguous truncation of --debug-level, and other will prefer --dbg. Interactive users usually prefer the minimal amount of typing, i.e., -D. We recommend that scripts which are re-usable employ some form of the long options for future maintainability.
This manual generally uses the short option syntax. This is for historical reasons and to conserve space. The remainder of this manual specifies the full long_name of each option. Users are expected to pick the unambiguous truncation of each option name that most suits their taste.
Availability (-n): ncea, ncecat, ncra, ncrcatAvailability ( -p): All operatorsShort options: -n, -p Long options: --nintap, --pth, --path |
ncra 85.nc 86.nc 87.nc 88.nc 89.nc 8589.nc
ncra 8[56789].nc 8589.nc
ncra -p input-path 85.nc 86.nc 87.nc 88.nc 89.nc 8589.nc
ncra -n 5,2,1 85.nc 8589.nc
The first method (explicitly specifying all filenames) works by brute
force.
The second method relies on the operating system shell to glob
(expand) the regular expression 8[56789].nc.
The shell passes valid filenames which match the expansion to
ncra.
The third method uses the -p input-path argument to specify
the directory where all the input files reside.
NCO prepends input-path (e.g.,
/data/usrname/model) to all input-files (but not to
output-file).
Thus, using -p, the path to any number of input files need only
be specified once.
Note input-path need not end with /; the / is
automatically generated if necessary.
The last method passes (with -n) syntax concisely describing the entire set of filenames 15. This option is only available with the multi-file operators: ncra, ncrcat, ncea, and ncecat. By definition, multi-file operators are able to process an arbitrary number of input-files. This option is very useful for abbreviating lists of filenames representable as alphanumeric_prefix+numeric_suffix+.+filetype where alphanumeric_prefix is a string of arbitrary length and composition, numeric_suffix is a fixed width field of digits, and filetype is a standard filetype indicator. For example, in the file ccm3_h0001.nc, we have alphanumeric_prefix = ccm3_h, numeric_suffix = 0001, and filetype = nc.
NCO is able to decode lists of such filenames encoded using the
-n option.
The simpler (3-argument) -n usage takes the form
-n file_number,digit_number,numeric_increment
where file_number is the number of files, digit_number is
the fixed number of numeric digits comprising the numeric_suffix,
and numeric_increment is the constant, integer-valued difference
between the numeric_suffix of any two consecutive files.
The value of alphanumeric_prefix is taken from the input file,
which serves as a template for decoding the filenames.
In the example above, the encoding -n 5,2,1 along with the input
file name 85.nc tells NCO to
construct five (5) filenames identical to the template 85.nc
except that the final two (2) digits are a numeric suffix to be
incremented by one (1) for each successive file.
Currently filetype may be either be empty, nc,
cdf, hdf, or hd5.
If present, these filetype suffixes (and the preceding .)
are ignored by NCO as it uses the -n arguments to
locate, evaluate, and compute the numeric_suffix component of
filenames.
Recently the -n option has been extended to allow convenient
specification of filenames with “circular” characteristics.
This means it is now possible for NCO to automatically
generate filenames which increment regularly until a specified maximum
value, and then wrap back to begin again at a specified minimum value.
The corresponding -n usage becomes more complex, taking one or
two additional arguments for a total of four or five, respectively:
-n
file_number,digit_number,numeric_increment[,numeric_max[,numeric_min]]
where numeric_max, if present, is the maximum integer-value of
numeric_suffix and numeric_min, if present, is the minimum
integer-value of numeric_suffix.
Consider, for example, the problem of specifying non-consecutive input
files where the filename suffixes end with the month index.
In climate modeling it is common to create summertime and wintertime
averages which contain the averages of the months June–July–August,
and December–January–February, respectively:
ncra -n 3,2,1 85_06.nc 85_0608.nc
ncra -n 3,2,1,12 85_12.nc 85_1202.nc
ncra -n 3,2,1,12,1 85_12.nc 85_1202.nc
The first example shows that three arguments to the -n option
suffice to specify consecutive months (06, 07, 08) which do not
“wrap” back to a minimum value.
The second example shows how to use the optional fourth and fifth
elements of the -n option to specify a wrap value to NCO.
The fourth argument to -n, if present, specifies the maximum
integer value of numeric_suffix.
In this case the maximum value is 12, and will be formatted as
12 in the filename string.
The fifth argument to -n, if present, specifies the minimum
integer value of numeric_suffix.
The default minimum filename suffix is 1, which is formatted as
01 in this case.
Thus the second and third examples have the same effect, that is, they
automatically generate, in order, the filenames 85_12.nc,
85_01.nc, and 85_02.nc as input to NCO.
|
Availability: All operators Short options: -o Long options: --fl_out, --output |
Specifying fl_out with a switch, rather than as a positional argument, allows fl_out to precede input files in the argument list. This is particularly useful with multi-file operators for three reasons. Multi-file operators may be invoked with hundreds (or more) filenames. Visual or automatic location of fl_out in such a list is difficult when the only syntactic distinction between input and output files is their position. Second, specification of a long list of input files may be difficult (see Large numbers of input files). Making the input file list the final argument to an operator facilitates using xargs for this purpose. Some alternatives to xargs are very ugly and undesirable. Finally, many users are more comfortable specifying output files with -o fl_out near the beginning of an argument list. Compilers and linkers are usually invoked this way.
|
Availability: All operators Short options: -p, -l Long options: --pth, --path, --lcl, --local |
To access a file on NCAR's MSS, specify the full MSS pathname of the
remote file.
NCO will attempt to detect whether the local machine has direct
(synchronous) MSS access.
In this case, NCO attempts to use the NCAR
msrcp command
16, or, failing that, /usr/local/bin/msread.
Otherwise NCO attempts to retrieve the MSS file
through the (asynchronous) Masnet Interface Gateway System
(MIGS) using the nrnet command.
The following examples show how one might analyze files stored on remote systems.
ncks -l ./ ftp://dust.ess.uci.edu/pub/zender/nco/in.nc
ncks -l ./ dust.ess.uci.edu:/home/zender/nco/in.nc
ncks -l ./ /ZENDER/nco/in.nc
ncks -l ./ mss:/ZENDER/nco/in.nc
ncks -l ./ -p http://www.cdc.noaa.gov/cgi-bin/nph-nc/Datasets/\
ncep.reanalysis.dailyavgs/surface air.sig995.1975.nc
The first example will work verbatim on your system if your system is
connected to the Internet and is not behind a firewall.
The second example will work on your system if you have rcp
or scp access to the machine dust.ess.uci.edu.
The third example will work from NCAR computers with local
access to the msrcp, msread, or nrnet
commands.
The fourth command will work if your local version of NCO was
built with DODS/OPeNDAP capability (see DODS/OPeNDAP).
The above commands can be rewritten using the -p input-path
option as follows:
ncks -p ftp://dust.ess.uci.edu/pub/zender/nco -l ./ in.nc
ncks -p dust.ess.uci.edu:/home/zender/nco -l ./ in.nc
ncks -p /ZENDER/nco -l ./ in.nc
ncks -p mss:/ZENDER/nco -l ./ in.nc
Using -p is recommended because it clearly separates the input-path from the filename itself, sometimes called the stub. When input-path is not explicitly specified using -p, NCO internally generates an input-path from the first input filename. The automatically generated input-path is constructed by stripping the input filename of everything following the final / character (i.e., removing the stub). The -l output-path option tells NCO where to store the remotely retrieved file and the output file. Often the path to a remotely retrieved file is quite different than the path on the local machine where you would like to store the file. If -l is not specified then NCO internally generates an output-path by simply setting output-path equal to input-path stripped of any machine names. If -l is not specified and the remote file resides on the NCAR MSS system, then the leading character of input-path, /, is also stripped from output-path. Specifying output-path as -l ./ tells NCO to store the remotely retrieved file and the output file in the current directory. Note that -l . is equivalent to -l ./ though the latter is recommended as it is syntactically more clear.
The Distributed Oceanographic Data System (DODS) provides useful replacements for common data interface libraries like netCDF. The DODS versions of these libraries implement network transparent access to data via a client-server data access protocol that uses the HTTP protocol for communication. Although DODS-technology originated with oceanography data, it applyies to virtually all scientific data. In recognition of this, the data access protocol underlying DODS (which is what NCO cares about) has been renamed the Open-source Project for a Network Data Access Protocol, OPeNDAP. We use the terms DODS and OPeNDAP interchangeably, and often write DODS/OPeNDAP for now. In the future we will deprecate DODS in favor of OPeNDAP 17.
NCO may be DODS/OPeNDAP-enabled by linking
NCO to the DODS/OPeNDAP libraries.
This is described in the DODS/OPeNDAP documentation and
automagically implemented in NCO build mechanisms
18.
The ./configure mechanism automatically enables NCO as
DODS/OPeNDAP clients if it can find the required
DODS/OPeNDAP libraries
19.
in the usual locations.
The $DODS_ROOT environment variable may be used to override the
default DODS/OPeNDAP library location at NCO
compile-time.
Building NCO with bld/Makefile and the command
make DODS=Y adds the (non-intuitive) commands to link to the
DODS/OPeNDAP libraries installed in the $DODS_ROOT
directory.
The file doc/dods.sh contains a generic script intended to help
users install DODS/OPeNDAP before building NCO.
The documentation at the
DODS Homepage
is voluminous.
Check there and on the
DODS mail lists.
to learn more about the extensive capabilities of DODS/OPeNDAP
20.
Once NCO is DODS/OPeNDAP-enabled the operators are DODS/OPeNDAP clients. All DODS/OPeNDAP clients have network transparent access to any files controlled by a DODS/OPeNDAP server. Simply specify the input file path(s) in URL notation and all NCO operations may be performed on remote files made accessible by a DODS/OPeNDAP server. For example, this extracts (remotely) an equatorial hyperslab, from NCEP daily reanalyses data of the year 1969 made freely available by a NOAA DODS/OPeNDAP server. Then it sends the hyperslab to the local ncwa client which computes and stores (locally) the regional mean surface pressure (in Pa).
ncwa -O -C -a lat,lon,time -d lon,-10.,10. -d lat,-10.,10. -l ./ -p \
http://www.cdc.noaa.gov/cgi-bin/nph-nc/Datasets/ncep.reanalysis.dailyavgs/surface \
pres.sfc.1969.nc foo.nc
All with one command! The data in this particular input file also happen to be packed (see Intrinsic functions), although this is completely transparent to the user since NCO automatically unpacks them before attempting arithmetic.
NCO obtains remote files from the DODS/OPeNDAP server (e.g., www.cdc.noaa.gov) rather than the local machine. Input files are first copied to the local machine, then processed. The DODS/OPeNDAP server performs data access, hyperslabbing, and transfer to the local machine. This allows the I/O to appear to NCO as if the input files were local. The local machine performs all arithmetic operations. Only the hyperslabbed output data are transferred over the network (to the local machine) for the number-crunching to begin. The advantages of this are obvious if you are examining small parts of large files stored at remote locations.
|
Availability: All operators Short options: -R Long options: --rtn, --retain |
Note that the remote retrieval features of NCO can always be used to retrieve any file, including non-netCDF files, via SSH, anonymous FTP, or msrcp. Often this method is quicker than using a browser, or running an FTP session from a shell window yourself. For example, say you want to obtain a JPEG file from a weather server.
ncks -R -p ftp://weather.edu/pub/pix/jpeg -l ./ storm.jpg
In this example, ncks automatically performs an anonymous FTP login to the remote machine and retrieves the specified file. When ncks attempts to read the local copy of storm.jpg as a netCDF file, it fails and exits, leaving storm.jpg in the current directory.
If your NCO is DODS-enabled (see DODS/OPeNDAP), then you may use NCO to retrieve any files (including netCDF, HDF, etc.) served by a DODS server to your local machine. For example,
ncks -R -l ./ -p \
http://www.cdc.noaa.gov/cgi-bin/nph-nc/Datasets/ncep.reanalysis.dailyavgs/surface \
pres.sfc.1969.nc
Note that NCO is never the preffered way to transport files from remote machines. For large jobs, that is best handled by FTP, SSH, or wget. It may occasionally be useful to use NCO to transfer files when your other preferred methods are not available locally.
|
Availability: (ncap), ncbo, ncea,
ncecat, ncflint, ncks, ncpdq,
ncra, ncrcat, ncwa Short options: -v, -x Long options: --variable, --exclude or --xcl |
As of NCO 2.8.1 (August, 2003), variable name arguments of the -v switch may contain extended regular expressions. For example, -v '^DST' selects all variables beginning with the string DST. Extended regular expressions are defined by the GNU egrep command. The meta-characters used to express pattern matching operations are ^$+?.*[]{}|. If the regular expression pattern matches any part of a variable name then that variable is selected. This capability is called wildcarding, and is very useful for sub-setting large data files.
Because of its wide availability, NCO uses the POSIX
regular expression library regex.
Regular expressions of arbitary complexity may be used.
Since netCDF variable names are relatively simple constructs, only a
few varieties of variable wildcards are likely to be useful.
For convenience, we define the most useful pattern matching operators
here:
Q, Q01–Q99,
Q100, QAA–QZZ, Q_H2O, X_H2O,
Q_CO2, X_CO2.
ncks -v 'Q+' in.nc # Select variables that start with Q
ncks -v '^Q+.?.' in.nc # Select Q, Q01--Q99, QAA--QZZ, etc.
ncks -v '^Q..' in.nc # Select Q01--Q99, QAA--QZZ, etc.
ncks -v '^Q[0-9][0-9]' in.nc # Select Q01--Q99
ncks -v '^Q[[:digit:]]{2}' in.nc # Select Q01--Q99
ncks -v 'H2O$' in.nc # Select Q_H2O, X_H2O
ncks -v 'H2O$|CO2$' in.nc # Select Q_H2O, X_H2O, Q_CO2, X_CO2
ncks -v '^Q[0-9][0-9]' in.nc # Select Q01--Q99, Q100
ncks -v '^Q[0-9][0-9]$' in.nc # Select Q01--Q99
ncks -v '^[a-z]_[a-z]{3}$' in.nc # Select Q_H2O, X_H2O, Q_CO2, X_CO2
Beware—two of the most frequently used repetition pattern matching operators, * and ?, are also valid pattern matching operators for filename expansion (globbing) at the shell-level. Confusingly, they have different meanings in extended regular expressions than in shell-level filename expansion. In an extended regular expression, * matches zero or more occurences of the preceding regular expression. Thus Q* selects all variables, and Q+.* selects all variables containing Q (the + ensures the preceding item matches at least once). To match zero or one occurence of the preceding regular expression, use ?. Thus Q? selects all variables, Q+.? selects all Documentation for the UNIX egrep command details the extended regular expressions that NCO supports.
One must be careful to protect any special characters in the regular expression specification from being interpreted (globbed) by the shell. This is accomplish by enclosing special characters within single or double quotes
ncra -v Q?? in.nc out.nc # Error: Shell attempts to glob wildcards
ncra -v '^Q+..' in.nc out.nc # Correct: NCO interprets wildcards
ncra -v '^Q+..' in*.nc out.nc
The final example shows that commands may use a combination of variable wildcarding and shell filename expansion (globbing). For globbing, * and ? have nothing to do with the preceding regular expression! In shell-level filename expansion, * matches any string, including the null string and ? matches any single character. Documentation for bash and csh describe the rules of filename expansion (globbing).
|
Availability: ncap, ncbo, ncea,
ncecat, ncflint, ncks, ncpdq,
ncra, ncrcat, ncwa Short options: -C, -c Long options: --no-coords, --no-crd, --crd, --coords |
lat always carry the
values of lat with them into the output-file.
This feature can be disabled with -C, which causes NCO
to not automatically add coordinates to the variables appearing in the
output-file.
However, using -C does not preclude the user from including some
coordinates in the output files simply by explicitly selecting the
coordinates with the -v option.
The -c option, on the other hand, is a shorthand way of
automatically specifying that all coordinate variables in the
input-files should appear in the output-file.
Thus -c allows the user to select all the coordinate variables
without having to know their names.
|
Availability: ncbo, ncea, ncecat,
ncflint, ncks, ncpdq, ncra,
ncrcat, ncwa Short options: -F Long options: --fortran |
Consider a file 85.nc containing 12 months of data in the
record dimension time.
The following hyperslab operations produce identical results, a
June-July-August average of the data:
ncra -d time,5,7 85.nc 85_JJA.nc
ncra -F -d time,6,8 85.nc 85_JJA.nc
Printing variable three_dmn_var in file in.nc first with C indexing conventions, then with Fortran indexing conventions results in the following output formats:
% ncks -v three_dmn_var in.nc
lat[0]=-90 lev[0]=1000 lon[0]=-180 three_dmn_var[0]=0
...
% ncks -F -v three_dmn_var in.nc
lon(1)=0 lev(1)=100 lat(1)=-90 three_dmn_var(1)=0
...
|
Availability: ncbo, ncea, ncecat,
ncflint, ncks, ncpdq, ncra,
ncrcat, ncwa Short options: -d Long options: --dimension, --dmn |
-d
dim,[min][,[max]] short option (or with the
--dimension or --dmn long options).
The bounds of the hyperslab to be extracted are specified by the
associated min and max values.
A half-open range is specified by omitting either the min or
max parameter but including the separating comma.
The unspecified limit is interpreted as the maximum or minimum value in
the unspecified direction.
A cross-section at a specific coordinate is extracted by specifying only
the min limit and omitting a trailing comma.
Dimensions not mentioned are passed with no reduction in range.
The dimensionality of variables is not reduced (in the case of a
cross-section, the size of the constant dimension will be one).
If values of a coordinate-variable are used to specify a range or
cross-section, then the coordinate variable must be monotonic (values
either increasing or decreasing).
In this case, command-line values need not exactly match coordinate
values for the specified dimension.
Ranges are determined by seeking the first coordinate value to occur in
the closed range [min,max] and including all subsequent
values until one falls outside the range.
The coordinate value for a cross-section is the coordinate-variable
value closest to the specified value and must lie within the range or
coordinate-variable values.
Coordinate values should be specified using real notation with a decimal point required in the value, whereas dimension indices are specified using integer notation without a decimal point. This convention serves only to differentiate coordinate values from dimension indices. It is independent of the type of any netCDF coordinate variables. For a given dimension, the specified limits must both be coordinate values (with decimal points) or dimension indices (no decimal points).
User-specified coordinate limits are promoted to double precision values
while searching for the indices which bracket the range.
Thus, hyperslabs on coordinates of type NC_BYTE and
NC_CHAR are computed numerically rather than lexically, so the
results are unpredictable.
The relative magnitude of min and max indicate to the operator whether to expect a wrapped coordinate (see Wrapped coordinates), such as longitude. If min > max, the NCO expects the coordinate to be wrapped, and a warning message will be printed. When this occurs, NCO selects all values outside the domain [max < min], i.e., all the values exclusive of the values which would have been selected if min and max were swapped. If this seems confusing, test your command on just the coordinate variables with ncks, and then examine the output to ensure NCO selected the hyperslab you expected (coordinate wrapping is currently only supported by ncks).
Because of the way wrapped coordinates are interpreted, it is very
important to make sure you always specify hyperslabs in the
monotonically increasing sense, i.e., min < max
(even if the underlying coordinate variable is monotonically
decreasing).
The only exception to this is when you are indeed specifying a wrapped
coordinate.
The distinction is crucial to understand because the points selected by,
e.g., -d longitude,50.,340., are exactly the complement of the
points selected by -d longitude,340.,50..
Not specifying any hyperslab option is equivalent to specifying full
ranges of all dimensions.
This option may be specified more than once in a single command
(each hyperslabed dimension requires its own -d option).
|
Availability: ncks Short options: -d Long options: --dimension, --dmn |
A single -d option can only specify a contiguous and/or regularly spaced multi-dimensional array of data. Multislabs are constructed from multiple -d options and may therefore have non-regularly spaced arrays. For example, suppose it is desired to operate on all longitudes from 10.0 to 20.0 and from 80.0 to 90.0 degrees. The combined range of longitudes is not selectable in a single hyperslab specfication of the form -d lon,min,max or -d lon,min,max,stride because its elements are irregularly spaced in coordinate space (and presumably in index space too). The multislab specification for obtaining these values is simply the union of the hyperslabs specifications that comprise the multislab, i.e.,
ncks -d lon,10.,20. -d lon,80.,90. in.nc out.nc
ncks -d lon,10.,15. -d lon,15.,20. -d lon,80.,90. in.nc out.nc
Any number of hyperslabs specifications may be chained together to specify the multislab. Multislabs are more efficient than the alternative of sequentially performing hyperslab operations and concatenating the results. This is because NCO employs a novel multislab algorithm to minimize the number of I/O operations when retrieving irregularly spaced data from disk.
Users may specify redundant ranges of indices in a multislab, e.g.,
ncks -d lon,0,4 -d lon,2,9,2 in.nc out.nc
This command retrieves the first five longitudes, and then every other longitude value up to the tenth. Elements 0, 2, and 4 are specified by both hyperslab arguments (hence this is redundant) but will count only once if an arithmetic operation is being performed. The NCO multislab algorithm retrieves each element from disk once and only once. Thus users may take some shortcuts in specifying multislabs and the algorithm will obtain the intended values. Specifying redundant ranges is not encouraged, but may be useful on occasion and will not result in unintended consequences.
A final example shows the real power of multislabs. Suppose the Q variable contains three dimensional arrays of distinct chemical constituents in no particular order. We are interested in the NOy species in a certain geographic range. Say that NO, NO2, and N2O5 are elements 0, 1, and 5 of the species dimension of Q. The multislab specification might look something like
ncks -d species,0,1 -d species,5 -d lon,0,4 -d lon,2,9,2 in.nc out.nc
Multislabs are powerful because they may be specified for every dimension at the same time. Thus multislabs obsolete the need to execute multiple ncks commands to gather the desired range of data. We envision adding multislab support to all arithmetic operators in the future.
|
Availability: ncbo, ncea, ncecat,
ncflint, ncks, ncpdq, ncra,
ncrcat, ncwa Short options: -d Long options: --dimension, --dmn |
Two examples suffice to demonstrate the power and convenience of UDUnits support. First, consider extraction of a variable containing non-record coordinates with physical dimensions stored in MKS units. In the following example, the user extracts all wavelengths in the visible portion of the spectrum in terms of the units very frequently used in visible spectroscopy, microns:
% ncks -O -C -u -v wvl -d wvl,"0.4 micron","0.7 micron" in.nc
wvl[0]=5e-07 meter
The hyperslab returns the correct values because the wvl variable
is stored on disk with a length dimension that UDUnits recognizes in the
units attribute.
The automagical algorithm that implements this functionality is worth
describing since understanding it helps one avoid some potential
pitfalls.
First, the user includes the physical units of the hyperslab dimensions
she supplies, separated by a simple space from the numerical values of
the hyperslab limits.
She encloses each coordinate specifications in quotes so that the shell
does not break the value-space-unit string into separate
arguments before passing them to NCO.
Double quotes ("foo") or single quotes ('foo') are equally
valid for this purpose.
Second, NCO recognizes that units translation is requested
because each hyperslab argument contains text characters and non-initial
spaces.
Third, NCO determines whether the wvl is dimensioned
with a coordinate variable that has a units attribute.
In this case, wvl itself is a coordinate variable.
The value of its units attribute is meter.
Thus wvl passes this test so UDUnits conversion is attempted.
If the coordinate associated with the variable does not contain a
units attribute, then NCO aborts.
Fourth, NCO passes the specified and desired dimension strings
(microns are specified by the user, meters are required by
NCO) to the UDUnits library.
Fifth, the UDUnits library that these dimension are commensurate
and it returns the appropriate linear scaling factors to convert from
microns to meters to NCO.
If the units are incommensurate (i.e., not expressible in the same
fundamental MKS units), or are not listed in the UDUnits database, then
NCO aborts since it cannot determine the user's intent.
Finally, NCO uses the scaling information to convert the
user-specified hyperslab limits into the same physical dimensions as
those of the corresponding cooridinate variable on disk.
At this point, NCO can perform a coordinate hyperslab using
the same algorithm as if the user had specified the hyperslab without
requesting units conversion.
The translation and dimennterpretation of time coordinates shows a more powerful, and probably more common, application of the UDUnits feature. In this example, the user prints all data between the eighth and ninth of December, 1999, from a variable whose time dimension is hours since the year 1900:
% ncks -O -C -u -v time_udunits -d time_udunits,"1999-12-08 \
12:00:0.0","1999-12-09 00:00:0.0",2 in.nc foo2.nc
% time_udunits[1]=876018 hours since 1900-01-01 00:00:0.0
Here, the user invokes the stride (see Stride) capability to obtain every other timeslice. This is possible because the UDUnits feature is additive, not exclusive—it works in conjunction with all other hyperslabbing (see Hyperslabs) options and in all operators which support hyperslabbing. The following example shows how one might average data in a time period spread across multiple input files
ncra -O -d time,"1939-09-09 12:00:0.0","1945-05-08 00:00:0.0" \
in1.nc in2.nc in3.nc out.nc
Note that there is no excess whitespace before or after the individual
elements of the -d argument.
This is important since, as far as the shell knows, -d takes
only one command-line argument.
Parsing this argument into its component
dim,[min],[max],stride elements
(see Hyperslabs) is the job of NCO.
When unquoted whitespace is present between these elements, the shell
passes NCO arugment fragments which will not parse as
intended.
The UDUnits package documentation describes the supported formats of time dimensions. Among the metadata conventions which adhere to these formats are the Climate and Forecast (CF) Conventions and the Cooperative Ocean/Atmosphere Research Data Service (COARDS) Conventions. The following -d arguments extract the same data using commonly encountered time dimension formats:
-d time,"1918-11-11 11:00:0.0","1939-09-09 00:00:0.0"
All of these formats include at least one dash - in a non-leading character position (a dash in a leading character position is a negative sign). NCO assumes that a non-leading dash in a limit string indicates that a UDUnits date conversion is requested.
netCDF variables should always be stored with MKS (i.e., God's) units, so that application programs may assume MKS dimensions apply to all input variables. The UDUnits feature is intended to alleviate some of the NCO user's pain when handling MKS units. It connects users who think in human-friendly units (e.g., miles, millibars, days) to extract data which are always stored in God's units, MKS (e.g., meters, Pascals, seconds). The feature is not intended to encourage writers to store data in esoteric units (e.g., furlongs, pounds per square inch, fortnights).
|
Availability: ncks Short options: -d Long options: --dimension, --dmn |
Assume the domain of the monotonically increasing longitude coordinate
lon is 0 < lon < 360.
ncks will extract a hyperslab which crosses the Greenwich
meridian simply by specifying the westernmost longitude as min and
the easternmost longitude as max.
The following commands extract a hyperslab containing the Saharan desert:
ncks -d lon,340.,50. in.nc out.nc
ncks -d lon,340.,50. -d lat,10.,35. in.nc out.nc
The first example selects data in the same longitude range as the Sahara.
The second example further constrains the data to having the same
latitude as the Sahara.
The coordinate lon in the output-file, out.nc, will
no longer be monotonic!
The values of lon will be, e.g., 340, 350, 0, 10, 20, 30,
40, 50.
This can have serious implications should you run out.nc through
another operation which expects the lon coordinate to be
monotonically increasing.
Fortunately, the chances of this happening are slim, since lon
has already been hyperslabbed, there should be no reason to hyperslab
lon again.
Should you need to hyperslab lon again, be sure to give
dimensional indices as the hyperslab arguments, rather than coordinate
values (see Hyperslabs).
|
Availability: ncks, ncra, ncrcat Short options: -d Long options: --dimension, --dmn |
The stride is specified as the optional fourth argument to the
-d hyperslab specification:
-d dim,[min][,[max]][,[stride]].
Specify stride as an integer (i.e., no decimal point) following
the third comma in the -d argument.
There is no default value for stride.
Thus using -d time,,,2 is valid but -d time,,,2.0 and
-d time,,, are not.
When stride is specified but min is not, there is an
ambiguity as to whether the extracted hyperslab should begin with (using
C-style, 0-based indexes) element 0 or element stride-1.
NCO must resolve this ambiguity and it chooses element 0
as the first element of the hyperslab when min is not specified.
Thus -d time,,,stride is syntactically equivalent to
-d time,0,,stride.
This means, for example, that specifying the operation
-d time,,,2 on the array 1,2,3,4,5 selects the hyperslab
1,3,5.
To obtain the hyperslab 2,4 instead, simply explicitly specify
the starting index as 1, i.e., -d time,1,,2.
For example, consider a file 8501_8912.nc which contains 60 consecutive months of data. Say you wish to obtain just the March data from this file. Using 0-based subscripts (see Fortran indexing) these data are stored in records 2, 14, ... 50 so the desired stride is 12. Without the stride option, the procedure is very awkward. One could use ncks five times and then use ncrcat to concatenate the resulting files together:
for idx in 02 14 26 38 50; do # Bourne Shell
ncks -d time,${idx} 8501_8912.nc foo.${idx}
done
foreach idx (02 14 26 38 50) # C Shell
ncks -d time,${idx} 8501_8912.nc foo.${idx}
end
ncrcat foo.?? 8589_03.nc
rm foo.??
With the stride option, ncks performs this hyperslab extraction in one operation:
ncks -d time,2,,12 8501_8912.nc 8589_03.nc
See ncks netCDF Kitchen Sink, for more information on ncks.
The stride option is supported by ncra and ncrcat for the record dimension only. This makes it possible, for instance, to average or concatenate regular intervals across multi-file input data sets.
ncra -F -d time,3,,12 85.nc 86.nc 87.nc 88.nc 89.nc 8589_03.nc
ncrcat -F -d time,3,,12 85.nc 86.nc 87.nc 88.nc 89.nc 8503_8903.nc
|
Availability: ncap, ncbo, ncea,
ncflint, ncpdq, ncra, ncwa Short options: None |
The phrase missing data refers to data points that are missing, invalid, or for any reason not intended to be arithmetically processed in the same fashion as valid data. The NCO arithmetic operators attempt to handle missing data in an intelligent fashion. There are four steps in the NCO treatment of missing data:
NCO follows the convention that missing data should be stored
with the missing_value specified in the variable's
missing_value attribute.
The only way NCO recognizes that a variable may
contain missing data is if the variable has a missing_value
attribute.
In this case, any elements of the variable which are numerically equal
to the missing_value are treated as missing data.
Consider a variable var of type var_type with a
missing_value attribute of type att_type containing the
value missing_value.
As a guideline, the type of the missing_value attribute should be
the same as the type of the variable it is attached to.
If var_type equals att_type then NCO
straightforwardly compares each value of var to
missing_value to determine which elements of var are to be
treated as missing data.
If not, then NCO converts missing_value from
att_type to var_type by using the implicit conversion rules
of C, or, if att_type is NC_CHAR
21, by typecasting the results of the C function
strtod(missing_value).
You may use the NCO operator ncatted to change the
missing_value attribute and all data whose data is
missing_value to a new value
(see ncatted netCDF Attribute Editor).
When an NCO arithmetic operator processes a variable var
with a missing_value attribute, it compares each value of
var to missing_value before performing an operation.
Note the missing_value comparison inflicts a performance penalty
on the operator.
Arithmetic processing of variables which contain the
missing_value attribute always incurs this penalty, even when
none of the data are missing.
Conversely, arithmetic processing of variables which do not contain the
missing_value attribute never incurs this penalty.
In other words, do not attach a missing_value attribute to a
variable which does not contain missing data.
This exhortation can usually be obeyed for model generated data, but it
may be harder to know in advance whether all observational data will be
valid or not.
NCO averagers (ncra, ncea, ncwa) do not count any element with the value missing_value towards the average. ncbo and ncflint define a missing_value result when either of the input values is a missing_value. Sometimes the missing_value may change from file to file in a multi-file operator, e.g., ncra. NCO is written to account for this (it always compares a variable to the missing_value assigned to that variable in the current file). Suffice it to say that, in all known cases, NCO does “the right thing”.
It is impossible to determine and store the correct result of a binary operation in a single variable. One such corner case occurs when both operands have differing missing_value attributes, i.e., attributes with different numerical values. Since the output (result) of the operation can only have one missing_value, some information may be lost. In this case, NCO always defines the output variable to have the same missing_value as the first input variable. Prior to performing the arithmetic operation, all values of the second operand equal to the second missing_value are replaced with the first missing_value. Then the arithmetic operation proceeds as normal, comparing each element of each operand to a single missing_value. Comparing each element to two distinct missing_value's would be much slower and would be no likelier to yield a more satisfactory answer. In practice, judicious choice of missing_value values prevents any important information from being lost.
|
Availability: ncra, ncea, ncwa Short options: -y Long options: --operation, --op_typ |
avgsqravgavgsqrmaxminrmsrmssdnsqrtttl|
Note for HTML users:
|
The definitions of some of these operations are not universally useful. Mostly they were chosen to facilitate standard statistical computations within the NCO framework. We are open to redefining and or adding to the above. If you are interested in having other statistical quantities defined in NCO please contact the NCO project (see Help and Bug reports).
EXAMPLES
Suppose you wish to examine the variable prs_sfc(time,lat,lon)
which contains a time series of the surface pressure as a function of
latitude and longitude.
Find the minimium value of prs_sfc over all dimensions:
ncwa -y min -v prs_sfc in.nc foo.nc
Find the maximum value of prs_sfc at each time interval for each
latitude:
ncwa -y max -v prs_sfc -a lon in.nc foo.nc
Find the root-mean-square value of the time-series of prs_sfc at
every gridpoint:
ncra -y rms -v prs_sfc in.nc foo.nc
ncwa -y rms -v prs_sfc -a time in.nc foo.nc
The previous two commands give the same answer but ncra is
preferred because it has a smaller memory footprint.
Also, ncra leaves the (degenerate) time dimension in the
output file (which is usually useful) whereas ncwa removes the
time dimension.
These operations work as expected in multi-file operators.
Suppose that prs_sfc is stored in multiple timesteps per file
across multiple files, say jan.nc, feb.nc,
march.nc.
We can now find the three month maximium surface pressure at every point.
ncea -y max -v prs_sfc jan.nc feb.nc march.nc out.nc
It is possible to use a combination of these operations to compute the variance and standard deviation of a field stored in a single file or across multiple files. The procedure to compute the temporal standard deviation of the surface pressure at all points in a single file in.nc involves three steps.
ncwa -O -v prs_sfc -a time in.nc out.nc
ncbo -O --op_typ=sub -v prs_sfc in.nc out.nc out.nc
ncra -O -y rmssdn out.nc out.nc
First the output file out.nc is contructed containing the
temporal mean of prs_sfc.
Next out.nc is overwritten with the deviation from the mean.
Finally out.nc is overwritten with the root-mean-square of
itself.
Note the use of -y rmssdn (rather than -y rms) in the
final step.
This ensures the standard deviation is correctly normalized by one fewer
than the number of time samples.
The procedure to compute the variance is identical except for the use of
-y var instead of -y rmssdn in the final step.
The procedure to compute the spatial standard deviation of a field in a single file in.nc involves three steps.
ncwa -O -v prs_sfc,gw -a lat,lon -w gw in.nc out.nc
ncbo -O --op_typ=sub -v prs_sfc,gw in.nc out.nc out.nc
ncwa -O -y rmssdn -v prs_sfc -a lat,lon -w gw out.nc out.nc
First the appropriately weighted (with -w gw) spatial mean values are written to the output file. This example includes the use of a weighted variable specified with -w gw. When using weights to compute standard deviations one must remember to include the weights in the initial output files so that they may be used again in the final step. The initial output file is then overwritten with the gridpoint deviations from the spatial mean. Finally the root-mean-square of the appropriately weighted spatial deviations is taken.
The procedure to compute the standard deviation of a time-series across multiple files involves one extra step since all the input must first be collected into one file.
ncrcat -O -v tpt in.nc in.nc foo1.nc
ncwa -O -a time foo1.nc foo2.nc
ncbo -O --op_typ=sub -v tpt foo1.nc foo2.nc foo2.nc
ncra -O -y rmssdn foo2.nc out.nc
The first step assembles all the data into a single file. This may require a lot of temporary disk space, but is more or less required by the ncbo operation in the third step.
|
Availability: ncap, ncbo, ncea,
ncra, ncwa Short options: None |
NC_SHORT (two bytes) to NC_DOUBLE (eight bytes).
Type conversion is automatic when the language carries out this
promotion according to an internal set of rules without explicit user
intervention.
In contrast, manual type conversion refers to explicit user commands to
change the type of a variable or attribute.
Most type conversion happens automatically, yet there are situations in
which manual type conversion is advantageous.
As a general rule, automatic type conversions should be avoided for at
least two reasons.
First, type conversions are expensive since they require creating
(temporary) buffers and casting each element of a variable from
the type it was stored at to some other type.
Second, the dataset's creator probably had a good reason
for storing data as, say, NC_FLOAT rather than NC_DOUBLE.
In a scientific framework there is no reason to store data with more
precision than the observations were made.
Thus NCO tries to avoid performing automatic type conversions
when performing arithmetic.
Automatic type conversion during arithmetic in the languages C and
Fortran is performed only when necessary.
All operands in an operation are converted to the most precise type
before the operation takes place.
However, following this parsimonious conversion rule dogmatically
results in numerous headaches.
For example, the average of the two NC_SHORTs 17000s and
17000s results in garbage since the intermediate value which
holds their sum is also of type NC_SHORT and thus cannot
represent values greater than 32,767
22.
There are valid reasons for expecting this operation to succeed and
the NCO philosophy is to make operators do what you want, not
what is most pure.
Thus, unlike C and Fortran, but like many other higher level interpreted
languages, NCO arithmetic operators will perform automatic type
conversion when all the following conditions are met
23:
NC_BYTE, NC_CHAR,
NC_SHORT, or NC_INT.
Type NC_DOUBLE is not type converted because there is no type of
higher precision to convert to.
Type NC_FLOAT is not type converted because, in our judgement,
the performance penalty of always doing so would outweigh the (extremely
rare) potential benefits.
When these criteria are all met, the operator promotes the variable in
question to type NC_DOUBLE, performs all the arithmetic
operations, casts the NC_DOUBLE type back to the original type,
and finally writes the result to disk.
The result written to disk may not be what you expect, because of
incommensurate ranges represented by different types, and because of
(lack of) rounding.
First, continuing the above example, the average (e.g., -y avg)
of 17000s and 17000s is written to disk as 17000s.
The type conversion feature of NCO makes this possible since
the arithmetic and intermediate values are stored as NC_DOUBLEs,
i.e., 34000.0d and only the final result must be represented
as an NC_SHORT.
Without the type conversion feature of NCO, the average would
have been garbage (albeit predictable garbage near -15768s).
Similarly, the total (e.g., -y ttl) of 17000s and
17000s written to disk is garbage (actually -31536s) since
the final result (the true total) of 34000 is outside the range
of type NC_SHORT.
Type conversions use the floor function to convert floating point
number to integers.
Type conversions do not attempt to round floating point numbers to the
nearest integer.
Thus the average of 1s and 2s is computed in double
precisions arithmetic as
(1.0d + 1.5d)/2) = 1.5d.
This result is converted to NC_SHORT and stored on disk as
floor(1.5d) = 1s
24.
Thus no "rounding up" is performed.
The type conversion rules of C can be stated as follows:
If n is an integer then any floating point value x
satisfying
n <= x < n+1
will have the value n when converted to an integer.
ncap provides intrinsic functions for performing manual type
conversions.
This, for example, converts variable tpt to external type
NC_SHORT (a C-type short), and variable prs to
external type NC_DOUBLE (a C-type double).
ncap -O -s "tpt=short(tpt);prs=double(prs);" in.nc out.nc
See ncap netCDF Arithmetic Processor, for more details.
|
Availability: All operators Short options: -O, -A Long options: --ovr, --overwrite, --apn, --append |
|
Availability: All operators Short options: -h Long options: --hst, --history |
history global attribute to
any file they create or modify.
The history attribute consists of a timestamp and the full string
of the invocation command to the operator, e.g., Mon May 26 20:10:24
1997: ncks in.nc foo.nc.
The full contents of an existing history attribute are copied
from the first input-file to the output-file.
The timestamps appear in reverse chronological order, with the most
recent timestamp appearing first in the history attribute.
Since NCO and many other netCDF operators adhere to the
history convention, the entire data processing path of a given
netCDF file may often be deduced from examination of its history
attribute.
As of May, 2002, NCO is case-insensitive to the spelling
of the history attribute name.
Thus attributes named History or HISTORY (which are
non-standard and not recommended) will be treated as valid history
attributes.
When more than one global attribute fits the case-insensitive search
for "history", the first one found will be used.
history attribute
To avoid information overkill, all operators have an optional switch
(-h, --hst, or --history) to override
automatically appending the history attribute
(see ncatted netCDF Attribute Editor).
Note that the -h switch also turns off writing the
nco_input_file_list attribute for multi-file operators
(see Input file list attributes).
|
Availability: ncea, ncecat, ncra, ncrcat Short options: -H Long options: --fl_lst_in, --file_list |
history
attribute no longer contains the exact command by which the file
was created.
NCO solves this dilemma by archiving input file list
attributes.
When the input file list to a multi-file operator is specified
via stdin, the operator, by default, attaches two global
attributes to any file they create or modify.
The nco_input_file_number global attribute contains the number of
input files, and nco_input_file_list contains the file names,
specified as standard input to the multi-file operator.
This information helps to verify that all input files the user thinks
were piped through stdin actually arrived.
Without the nco_input_file_list attribute, the information is lost
forever and the “chain of evidence” would be broken.
The -H switch overrides (turns off) the default behavior of
writing the input file list global attributes when input is from
stdin.
The -h switch does this too, and turns off the history
attribute as well (see History attribute).
Hence both switches allows space-conscious users to avoid storing what
may amount to many thousands of filenames in a metadata attribute.
|
Availability: ncbo, ncea, ncecat,
ncflint, ncra, ncwa Short options: None |
Currently, NCO determines whether a datafile is a
CCSM output datafile simply by checking whether value of the
global attribute convention (if it exists) equals
NCAR-CSM.
Should convention equal NCAR-CSM in the (first)
input-file, NCO will attempt to treat certain variables
specially, because of their meaning in CCSM files.
NCO will not average the following variables often found in
CCSM files:
ntrm, ntrn, ntrk, ndbase, nsbase,
nbdate, nbsec, mdt, mhisf.
These variables contain scalar metadata such as the resolution of the
host CCSM model and it makes no sense to change their values.
Furthermore, the ncbo operator does not operate on (i.e., add,
subtract, etc.) the following variables:
gw, ORO, date, datesec, hyam,
hybm, hyai, hybi.
These variables represent the Gaussian weights, the orography field,
time fields, and hybrid pressure coefficients.
These are fields which you virtually always want to remain unaltered in
the output file.
If you decide you would like any of the above CCSM fields
processed, you must spoof NCO.
For example rename the variables first with ncrename,
or alter the convention attribute.
|
Availability: ncrcat Short options: None |
base_time, and a record variable, time_offset.
Subtle but serious problems can arise when these type of files are
just blindly concatenated.
Therefore ncrcat has been specially programmed to be able to
chain together consecutive ARM input-files and produce
and an output-file which contains the correct time information.
Currently, ncrcat determines whether a datafile is an
ARM datafile simply by testing for the existence of the
variables base_time, time_offset, and the dimension
time.
If these are found in the input-file then ncrcat will
automatically perform two non-standard, but hopefully useful,
procedures.
First, ncrcat will ensure that values of time_offset
appearing in the output-file are relative to the base_time
appearing in the first input-file (and presumably, though not
necessarily, also appearing in the output-file).
Second, if a coordinate variable named time is not found in the
input-files, then ncrcat automatically creates the
time coordinate in the output-file.
The values of time are defined by the ARM convention
time = base_time + time_offset.
Thus, if output-file contains the time_offset
variable, it will also contain the time coordinate.
A short message is added to the history global attribute
whenever these ARM-specific procedures are executed.
|
Availability: All operators Short options: -r Long options: --revision, --version, or --vrs |
1.2.
However using -r on, say, ncks, will produce something
like NCO netCDF Operators version 1.2
Copyright (C) 1995--2000 Charlie Zender
ncks version 1.30 (2000/07/31) "Bolivia".
This tells you ncks contains all patches up to version
1.30, which dates from July 31, 2000.
This chapter presents reference pages for each of the operators individually. The operators are presented in alphabetical order. All valid command line switches are included in the syntax statement. Recall that descriptions of many of these command line switches are provided only in Common features, to avoid redundancy. Only options specific to, or most useful with, a particular operator are described in any detail in the sections below.
ncap [-A] [-C] [-c] [-D dbg]
[-d dim,[min][,[max]][,[stride]]] [-F] [-f]
[-l path] [-O] [-o output-file] [-p path] [-R] [-r]
[-s algebra] [-S fl.nco] [-v]
input-file [output-file]
DESCRIPTION
ncap arithmetically processes a netCDF file. The processing instructions are contained either in the NCO script file fl.nco or in a sequence of command line arguments. The options -s (or long options --spt or --script) are used for in-line scripts and -S (or long options --fl_spt or --script-file) are used to provide the filename where (usually multiple) scripting commands are pre-stored. ncap was written to perform arbitrary albebraic transformations of data and archive the results as easily as possible. See Missing values, for treatment of missing values. The results of the algebraic manipulations are called derived fields.
Unlike the other operators, ncap does not accept a list of variables to be operated on as an argument to -v (see Variable subsetting). Rather, the -v switch takes no arguments and indicates that ncap should output only user-defined variables. ncap does not accept or understand the -x switch.
The following examples demonstrate the utility of the left hand casting ability of ncap. Consider first this simple, artificial, example. If lat and lon are one dimensional coordinates of dimensions lat and lon, respectively, then addition of these two one-dimensional arrays is intrinsically ill-defined because whether lat_lon should be dimensioned lat by lon or lon by lat is ambiguous (assuming that addition is to remain a commutative procedure, i.e., one that does not depend on the order of its arguments). Differing dimensions are said to be orthogonal to one another, and sets of dimensions which are mutually exclusive are orthogonal as a set and any arithmetic operation between variables in orthogonal dimensional spaces is ambiguous without further information.
The ambiguity may be resolved by enumerating the desired dimension ordering of the output expression inside square brackets on the left hand side (LHS) of the equals sign. This is called left hand casting because the user resolves the dimensional ordering of the RHS of the expression by specifying the desired ordering on the LHS.
ncap -O -s "lat_lon[lat,lon]=lat+lon" in.nc out.nc
ncap -O -s "lon_lat[lon,lat]=lat+lon" in.nc out.nc
The explicit list of dimensions on the LHS, [lat,lon]
resolves the otherwise ambiguous ordering of dimensions in
lat_lon.
In effect, the LHS casts its rank properties onto the
RHS.
Without LHS casting, the dimensional ordering of lat_lon
would be undefined and, hopefully, ncap would print an error
message.
Consider now a slightly more complex example.
In geophysical models, a coordinate system based on
a blend of terrain-following and density-following surfaces is
called a hybrid coordinate system.
In this coordinate system, four variables must be manipulated to
obtain the pressure of the vertical coordinate:
PO is the domain-mean surface pressure offset (a scalar),
PS is the local (time-varying) surface pressure (usually two
horizontal spatial dimensions, i.e, latitude by longitude), hyam
is the weight given to surfaces of constant density (one spatial
dimension, pressure, which is orthogonal to the horizontal
dimensions), and hybm is the weight given to surfaces of
constant elevation (also one spatial dimension).
This command constructs a four-dimensional pressure prs_mdp
from the four input variables of mixed rank and orthogonality:
ncap -O -s "prs_mdp[time,lat,lon,lev]=P0*hyam+PS*hybm" in.nc out.nc
Manipulating the four fields which define the pressure in a hybrid coordinate system is easy with left hand casting.
Mastering ncap is relatively simple. Each valid statement statement consists of standard forward algebraic expression. The fl.nco, if present, is simply a list of such statements, whitespace, and comments. The syntax of statements is most like the computer language C. The following characteristics of C are preserved:
[] characters;
/* */ characters.
Single line comments are preceded by // characters.
#include script.
Note that the #include command is not followed by a semi-colon
because it is a pre-processor directive, not an assignment statement.
The filename script is interpreted relative to the run directory.
@ is used to delineate an attribute name from a
variable name.
ncap contains a small (and growing) library of intrinsic functions. In addition to the standard mathematical functions (see Intrinsic mathematical functions), ncap currently supports packing and unpacking.
pack(x)NC_SHORT
with the two attributes required to unpack the variable,
scale_factor and add_offset, stored at the original
(unpacked) precision of the variable
27.
Let min and max be the minimum and maximum values
of x.
scale_factor = (max-min)/ndrv
where ndrv is the number of discrete representable values for
given type of packed variable.
The theoretical maximum value for ndrv is two raised to the
number of bits used to store the packed variable.
Thus if the variable is packed into type NC_SHORT, a two-byte
datatype, then there are at most 2^16 = 65536 distinct values
representible.
In practice, the number of discretely representible values is taken
to be one less than the theoretical maximum.
This leaves extra space and solves potential problems with rounding
which can occur during the unpacking of the variable.
Thus for NC_SHORT, ndrv = 65536 - 1 = 65535.
Less often, the variable may be packed into type NC_CHAR,
where ndrv = 256 - 1 = 255, or type NC_INT where
where ndrv = 4294967295 - 1 = 4294967294.
unpack(x)scale_factor and add_offset.
If scale_factor is present for a variable, the data are
multiplied by the value scale_factor after the data are read.
If add_offset is present for a variable, then the
add_offset value is added to the data after the data are read.
If both scale_factor and add_offset attributes are
present, the data are first scaled by scale_factor before the
offset add_offset is added.
upk = scale_factor*pck + add_offset = (max-min)*pck/ndrv + 0.5*(min+max)
When scale_factor and add_offset are used for packing, the
associated variable (containing the packed data) is typically of type
byte or short, whereas the unpacked values are intended to
be of type int, float, or double.
An attribute's scale_factor and add_offset and
missing_value, if any, should all be of the type intended for the
unpacked data, i.e., int, float or double.
These intrinsic functions allow ncap to convert variables on disk among the available types supported by netCDF.
byte(x)NC_BYTE
Converts x to external type NC_BYTE, a C-type signed char.
char(x)NC_CHAR
Converts x to external type NC_CHAR, a C-type unsigned char.
double(x)NC_DOUBLE
Converts x to external type NC_DOUBLE, a C-type double.
float(x)NC_FLOAT
Converts x to external type NC_FLOAT, a C-type float.
int(x)NC_INT
Converts x to external type NC_INT, a C-type int.
short(x)NC_SHORT
Converts x to external type NC_SHORT, a C-type short.
ncap supports the standard mathematical functions supplied with most operating systems. Standard calculator notation is used for addition +, subtraction -, multiplication *, division /, exponentiation ^, and modulus %. The available elementary mathematical functions are:
abs(x)acos(x)acosh(x)asin(x)asinh(x)atan(x)atanh(x)ceil(x)cos(x)cosh(x)erf(x)erfc(x)exp(x)floor(x)gamma(x)ln(x)log(x)log10(x)nearbyint(x)pow(x,y)pow function,
integer arguments are promoted (see Type conversion) to type
NC_FLOAT before evaluation.
Example: pow(2,3) = 8
rint(x)round(x)sin(x)sinh(x)sqrt(x)tan(x)tanh(x)trunc(x)ncap contains a small (and growing) library of intrinsic functions. In addition to the standard mathematical functions (see Intrinsic mathematical functions), ncap currently supports packing and unpacking.
pack(x)NC_SHORT
with the two attributes required to unpack the variable,
scale_factor and add_offset, stored at the original
(unpacked) precision of the variable
27.
Let min and max be the minimum and maximum values
of x.
scale_factor = (max-min)/ndrv
where ndrv is the number of discrete representable values for
given type of packed variable.
The theoretical maximum value for ndrv is two raised to the
number of bits used to store the packed variable.
Thus if the variable is packed into type NC_SHORT, a two-byte
datatype, then there are at most 2^16 = 65536 distinct values
representible.
In practice, the number of discretely representible values is taken
to be one less than the theoretical maximum.
This leaves extra space and solves potential problems with rounding
which can occur during the unpacking of the variable.
Thus for NC_SHORT, ndrv = 65536 - 1 = 65535.
Less often, the variable may be packed into type NC_CHAR,
where ndrv = 256 - 1 = 255, or type NC_INT where
where ndrv = 4294967295 - 1 = 4294967294.
unpack(x)scale_factor and add_offset.
If scale_factor is present for a variable, the data are
multiplied by the value scale_factor after the data are read.
If add_offset is present for a variable, then the
add_offset value is added to the data after the data are read.
If both scale_factor and add_offset attributes are
present, the data are first scaled by scale_factor before the
offset add_offset is added.
upk = scale_factor*pck + add_offset = (max-min)*pck/ndrv + 0.5*(min+max)
When scale_factor and add_offset are used for packing, the
associated variable (containing the packed data) is typically of type
byte or short, whereas the unpacked values are intended to
be of type int, float, or double.
An attribute's scale_factor and add_offset and
missing_value, if any, should all be of the type intended for the
unpacked data, i.e., int, float or double.
These intrinsic functions allow ncap to convert variables on disk among the available types supported by netCDF.
byte(x)NC_BYTE
Converts x to external type NC_BYTE, a C-type signed char.
char(x)NC_CHAR
Converts x to external type NC_CHAR, a C-type unsigned char.
double(x)NC_DOUBLE
Converts x to external type NC_DOUBLE, a C-type double.
float(x)NC_FLOAT
Converts x to external type NC_FLOAT, a C-type float.
int(x)NC_INT
Converts x to external type NC_INT, a C-type int.
short(x)NC_SHORT
Converts x to external type NC_SHORT, a C-type short.
ncap supports the standard mathematical functions supplied with most operating systems. Standard calculator notation is used for addition +, subtraction -, multiplication *, division /, exponentiation ^, and modulus %. The available elementary mathematical functions are:
abs(x)acos(x)acosh(x)asin(x)asinh(x)atan(x)atanh(x)ceil(x)cos(x)cosh(x)erf(x)erfc(x)exp(x)floor(x)gamma(x)ln(x)log(x)log10(x)nearbyint(x)pow(x,y)pow function,
integer arguments are promoted (see Type conversion) to type
NC_FLOAT before evaluation.
Example: pow(2,3) = 8
rint(x)round(x)sin(x)sinh(x)sqrt(x)tan(x)tanh(x)trunc(x)ncap contains a small (and growing) library of intrinsic functions. In addition to the standard mathematical functions (see Intrinsic mathematical functions), ncap currently supports packing and unpacking.
pack(x)NC_SHORT
with the two attributes required to unpack the variable,
scale_factor and add_offset, stored at the original
(unpacked) precision of the variable
27.
Let min and max be the minimum and maximum values
of x.
scale_factor = (max-min)/ndrv
where ndrv is the number of discrete representable values for
given type of packed variable.
The theoretical maximum value for ndrv is two raised to the
number of bits used to store the packed variable.
Thus if the variable is packed into type NC_SHORT, a two-byte
datatype, then there are at most 2^16 = 65536 distinct values
representible.
In practice, the number of discretely representible values is taken
to be one less than the theoretical maximum.
This leaves extra space and solves potential problems with rounding
which can occur during the unpacking of the variable.
Thus for NC_SHORT, ndrv = 65536 - 1 = 65535.
Less often, the variable may be packed into type NC_CHAR,
where ndrv = 256 - 1 = 255, or type NC_INT where
where ndrv = 4294967295 - 1 = 4294967294.
unpack(x)scale_factor and add_offset.
If scale_factor is present for a variable, the data are
multiplied by the value scale_factor after the data are read.
If add_offset is present for a variable, then the
add_offset value is added to the data after the data are read.
If both scale_factor and add_offset attributes are
present, the data are first scaled by scale_factor before the
offset add_offset is added.
upk = scale_factor*pck + add_offset = (max-min)*pck/ndrv + 0.5*(min+max)
When scale_factor and add_offset are used for packing, the
associated variable (containing the packed data) is typically of type
byte or short, whereas the unpacked values are intended to
be of type int, float, or double.
An attribute's scale_factor and add_offset and
missing_value, if any, should all be of the type intended for the
unpacked data, i.e., int, float or double.
These intrinsic functions allow ncap to convert variables on disk among the available types supported by netCDF.
byte(x)NC_BYTE
Converts x to external type NC_BYTE, a C-type signed char.
char(x)NC_CHAR
Converts x to external type NC_CHAR, a C-type unsigned char.
double(x)NC_DOUBLE
Converts x to external type NC_DOUBLE, a C-type double.
float(x)NC_FLOAT
Converts x to external type NC_FLOAT, a C-type float.
int(x)NC_INT
Converts x to external type NC_INT, a C-type int.
short(x)NC_SHORT
Converts x to external type NC_SHORT, a C-type short.
ncap supports the standard mathematical functions supplied with most operating systems. Standard calculator notation is used for addition +, subtraction -, multiplication *, division /, exponentiation ^, and modulus %. The available elementary mathematical functions are:
abs(x)acos(x)acosh(x)asin(x)asinh(x)atan(x)atanh(x)ceil(x)cos(x)cosh(x)erf(x)erfc(x)exp(x)floor(x)gamma(x)ln(x)log(x)log10(x)nearbyint(x)pow(x,y)pow function,
integer arguments are promoted (see Type conversion) to type
NC_FLOAT before evaluation.
Example: pow(2,3) = 8
rint(x)round(x)sin(x)sinh(x)sqrt(x)tan(x)tanh(x)trunc(x)ncap contains a small (and growing) library of intrinsic functions. In addition to the standard mathematical functions (see Intrinsic mathematical functions), ncap currently supports packing and unpacking.
pack(x)NC_SHORT
with the two attributes required to unpack the variable,
scale_factor and add_offset, stored at the original
(unpacked) precision of the variable
27.
Let min and max be the minimum and maximum values
of x.
scale_factor = (max-min)/ndrv
where ndrv is the number of discrete representable values for
given type of packed variable.
The theoretical maximum value for ndrv is two raised to the
number of bits used to store the packed variable.
Thus if the variable is packed into type NC_SHORT, a two-byte
datatype, then there are at most 2^16 = 65536 distinct values
representible.
In practice, the number of discretely representible values is taken
to be one less than the theoretical maximum.
This leaves extra space and solves potential problems with rounding
which can occur during the unpacking of the variable.
Thus for NC_SHORT, ndrv = 65536 - 1 = 65535.
Less often, the variable may be packed into type NC_CHAR,
where ndrv = 256 - 1 = 255, or type NC_INT where
where ndrv = 4294967295 - 1 = 4294967294.
unpack(x)scale_factor and add_offset.
If scale_factor is present for a variable, the data are
multiplied by the value scale_factor after the data are read.
If add_offset is present for a variable, then the
add_offset value is added to the data after the data are read.
If both scale_factor and add_offset attributes are
present, the data are first scaled by scale_factor before the
offset add_offset is added.
upk = scale_factor*pck + add_offset = (max-min)*pck/ndrv + 0.5*(min+max)
When scale_factor and add_offset are used for packing, the
associated variable (containing the packed data) is typically of type
byte or short, whereas the unpacked values are intended to
be of type int, float, or double.
An attribute's scale_factor and add_offset and
missing_value, if any, should all be of the type intended for the
unpacked data, i.e., int, float or double.
These intrinsic functions allow ncap to convert variables on disk among the available types supported by netCDF.
byte(x)NC_BYTE
Converts x to external type NC_BYTE, a C-type signed char.
char(x)NC_CHAR
Converts x to external type NC_CHAR, a C-type unsigned char.
double(x)NC_DOUBLE
Converts x to external type NC_DOUBLE, a C-type double.
float(x)NC_FLOAT
Converts x to external type NC_FLOAT, a C-type float.
int(x)NC_INT
Converts x to external type NC_INT, a C-type int.
short(x)NC_SHORT
Converts x to external type NC_SHORT, a C-type short.
ncap supports the standard mathematical functions supplied with most operating systems. Standard calculator notation is used for addition +, subtraction -, multiplication *, division /, exponentiation ^, and modulus %. The available elementary mathematical functions are:
abs(x)acos(x)acosh(x)asin(x)asinh(x)atan(x)atanh(x)ceil(x)cos(x)cosh(x)erf(x)erfc(x)exp(x)floor(x)gamma(x)ln(x)log(x)log10(x)nearbyint(x)pow(x,y)pow function,
integer arguments are promoted (see Type conversion) to type
NC_FLOAT before evaluation.
Example: pow(2,3) = 8
rint(x)round(x)sin(x)sinh(x)sqrt(x)tan(x)tanh(x)trunc(x)ncap contains a small (and growing) library of intrinsic functions. In addition to the standard mathematical functions (see Intrinsic mathematical functions), ncap currently supports packing and unpacking.
pack(x)NC_SHORT
with the two attributes required to unpack the variable,
scale_factor and add_offset, stored at the original
(unpacked) precision of the variable
27.
Let min and max be the minimum and maximum values
of x.
scale_factor = (max-min)/ndrv
where ndrv is the number of discrete representable values for
given type of packed variable.
The theoretical maximum value for ndrv is two raised to the
number of bits used to store the packed variable.
Thus if the variable is packed into type NC_SHORT, a two-byte
datatype, then there are at most 2^16 = 65536 distinct values
representible.
In practice, the number of discretely representible values is taken
to be one less than the theoretical maximum.
This leaves extra space and solves potential problems with rounding
which can occur during the unpacking of the variable.
Thus for NC_SHORT, ndrv = 65536 - 1 = 65535.
Less often, the variable may be packed into type NC_CHAR,
where ndrv = 256 - 1 = 255, or type NC_INT where
where ndrv = 4294967295 - 1 = 4294967294.
unpack(x)scale_factor and add_offset.
If scale_factor is present for a variable, the data are
multiplied by the value scale_factor after the data are read.
If add_offset is present for a variable, then the
add_offset value is added to the data after the data are read.
If both scale_factor and add_offset attributes are
present, the data are first scaled by scale_factor before the
offset add_offset is added.
upk = scale_factor*pck + add_offset = (max-min)*pck/ndrv + 0.5*(min+max)
When scale_factor and add_offset are used for packing, the
associated variable (containing the packed data) is typically of type
byte or short, whereas the unpacked values are intended to
be of type int, float, or double.
An attribute's scale_factor and add_offset and
missing_value, if any, should all be of the type intended for the
unpacked data, i.e., int, float or double.
These intrinsic functions allow ncap to convert variables on disk among the available types supported by netCDF.
byte(x)NC_BYTE
Converts x to external type NC_BYTE, a C-type signed char.
char(x)NC_CHAR
Converts x to external type NC_CHAR, a C-type unsigned char.
double(x)NC_DOUBLE
Converts x to external type NC_DOUBLE, a C-type double.
float(x)NC_FLOAT
Converts x to external type NC_FLOAT, a C-type float.
int(x)NC_INT
Converts x to external type NC_INT, a C-type int.
short(x)NC_SHORT
Converts x to external type NC_SHORT, a C-type short.
ncap supports the standard mathematical functions supplied with most operating systems. Standard calculator notation is used for addition +, subtraction -, multiplication *, division /, exponentiation ^, and modulus %. The available elementary mathematical functions are:
abs(x)acos(x)acosh(x)asin(x)asinh(x)atan(x)atanh(x)ceil(x)cos(x)cosh(x)erf(x)erfc(x)exp(x)floor(x)gamma(x)ln(x)log(x)log10(x)nearbyint(x)pow(x,y)pow function,
integer arguments are promoted (see Type conversion) to type
NC_FLOAT before evaluation.
Example: pow(2,3) = 8
rint(x)round(x)sin(x)sinh(x)sqrt(x)tan(x)tanh(x)trunc(x)ncap contains a small (and growing) library of intrinsic functions. In addition to the standard mathematical functions (see Intrinsic mathematical functions), ncap currently supports packing and unpacking.
pack(x)NC_SHORT
with the two attributes required to unpack the variable,
scale_factor and add_offset, stored at the original
(unpacked) precision of the variable
27.
Let min and max be the minimum and maximum values
of x.
scale_factor = (max-min)/ndrv
where ndrv is the number of discrete representable values for
given type of packed variable.
The theoretical maximum value for ndrv is two raised to the
number of bits used to store the packed variable.
Thus if the variable is packed into type NC_SHORT, a two-byte
datatype, then there are at most 2^16 = 65536 distinct values
representible.
In practice, the number of discretely representible values is taken
to be one less than the theoretical maximum.
This leaves extra space and solves potential problems with rounding
which can occur during the unpacking of the variable.
Thus for NC_SHORT, ndrv = 65536 - 1 = 65535.
Less often, the variable may be packed into type NC_CHAR,
where ndrv = 256 - 1 = 255, or type NC_INT where
where ndrv = 4294967295 - 1 = 4294967294.
unpack(x)scale_factor and add_offset.
If scale_factor is present for a variable, the data are
multiplied by the value scale_factor after the data are read.
If add_offset is present for a variable, then the
add_offset value is added to the data after the data are read.
If both scale_factor and add_offset attributes are
present, the data are first scaled by scale_factor before the
offset add_offset is added.
upk = scale_factor*pck + add_offset = (max-min)*pck/ndrv + 0.5*(min+max)
When scale_factor and add_offset are used for packing, the
associated variable (containing the packed data) is typically of type
byte or short, whereas the unpacked values are intended to
be of type int, float, or double.
An attribute's scale_factor and add_offset and
missing_value, if any, should all be of the type intended for the
unpacked data, i.e., int, float or double.
These intrinsic functions allow ncap to convert variables on disk among the available types supported by netCDF.
byte(x)NC_BYTE
Converts x to external type NC_BYTE, a C-type signed char.
char(x)NC_CHAR
Converts x to external type NC_CHAR, a C-type unsigned char.
double(x)NC_DOUBLE
Converts x to external type NC_DOUBLE, a C-type double.
float(x)NC_FLOAT
Converts x to external type NC_FLOAT, a C-type float.
int(x)NC_INT
Converts x to external type NC_INT, a C-type int.
short(x)NC_SHORT
Converts x to external type NC_SHORT, a C-type short.
ncap supports the standard mathematical functions supplied with most operating systems. Standard calculator notation is used for addition +, subtraction -, multiplication *, division /, exponentiation ^, and modulus %. The available elementary mathematical functions are:
abs(x)acos(x)acosh(x)asin(x)asinh(x)atan(x)atanh(x)ceil(x)cos(x)cosh(x)erf(x)erfc(x)exp(x)floor(x)gamma(x)ln(x)log(x)log10(x)nearbyint(x)pow(x,y)pow function,
integer arguments are promoted (see Type conversion) to type
NC_FLOAT before evaluation.
Example: pow(2,3) = 8
rint(x)round(x)sin(x)sinh(x)sqrt(x)tan(x)tanh(x)trunc(x)ncap contains a small (and growing) library of intrinsic functions. In addition to the standard mathematical functions (see Intrinsic mathematical functions), ncap currently supports packing and unpacking.
pack(x)NC_SHORT
with the two attributes required to unpack the variable,
scale_factor and add_offset, stored at the original
(unpacked) precision of the variable
27.
Let min and max be the minimum and maximum values
of x.
scale_factor = (max-min)/ndrv
where ndrv is the number of discrete representable values for
given type of packed variable.
The theoretical maximum value for ndrv is two raised to the
number of bits used to store the packed variable.
Thus if the variable is packed into type NC_SHORT, a two-byte
datatype, then there are at most 2^16 = 65536 distinct values
representible.
In practice, the number of discretely representible values is taken
to be one less than the theoretical maximum.
This leaves extra space and solves potential problems with rounding
which can occur during the unpacking of the variable.
Thus for NC_SHORT, ndrv = 65536 - 1 = 65535.
Less often, the variable may be packed into type NC_CHAR,
where ndrv = 256 - 1 = 255, or type NC_INT where
where ndrv = 4294967295 - 1 = 4294967294.
unpack(x)scale_factor and add_offset.
If scale_factor is present for a variable, the data are
multiplied by the value scale_factor after the data are read.
If add_offset is present for a variable, then the
add_offset value is added to the data after the data are read.
If both scale_factor and add_offset attributes are
present, the data are first scaled by scale_factor before the
offset add_offset is added.
upk = scale_factor*pck + add_offset = (max-min)*pck/ndrv + 0.5*(min+max)
When scale_factor and add_offset are used for packing, the
associated variable (containing the packed data) is typically of type
byte or short, whereas the unpacked values are intended to
be of type int, float, or double.
An attribute's scale_factor and add_offset and
missing_value, if any, should all be of the type intended for the
unpacked data, i.e., int, float or double.
These intrinsic functions allow ncap to convert variables on disk among the available types supported by netCDF.
byte(x)NC_BYTE
Converts x to external type NC_BYTE, a C-type signed char.
char(x)NC_CHAR
Converts x to external type NC_CHAR, a C-type unsigned char.
double(x)NC_DOUBLE
Converts x to external type NC_DOUBLE, a C-type double.
float(x)NC_FLOAT
Converts x to external type NC_FLOAT, a C-type float.
int(x)NC_INT
Converts x to external type NC_INT, a C-type int.
short(x)NC_SHORT
Converts x to external type NC_SHORT, a C-type short.
ncap supports the standard mathematical functions supplied with most operating systems. Standard calculator notation is used for addition +, subtraction -, multiplication *, division /, exponentiation ^, and modulus %. The available elementary mathematical functions are:
abs(x)acos(x)acosh(x)asin(x)asinh(x)atan(x)atanh(x)ceil(x)cos(x)cosh(x)erf(x)erfc(x)exp(x)floor(x)gamma(x)ln(x)log(x)log10(x)nearbyint(x)pow(x,y)pow function,
integer arguments are promoted (see Type conversion) to type
NC_FLOAT before evaluation.
Example: pow(2,3) = 8
rint(x)round(x)sin(x)sinh(x)sqrt(x)tan(x)tanh(x)trunc(x)ncap contains a small (and growing) library of intrinsic functions. In addition to the standard mathematical functions (see Intrinsic mathematical functions), ncap currently supports packing and unpacking.
pack(x)NC_SHORT
with the two attributes required to unpack the variable,
scale_factor and add_offset, stored at the original
(unpacked) precision of the variable
27.
Let min and max be the minimum and maximum values
of x.
scale_factor = (max-min)/ndrv
where ndrv is the number of discrete representable values for
given type of packed variable.
The theoretical maximum value for ndrv is two raised to the
number of bits used to store the packed variable.
Thus if the variable is packed into type NC_SHORT, a two-byte
datatype, then there are at most 2^16 = 65536 distinct values
representible.
In practice, the number of discretely representible values is taken
to be one less than the theoretical maximum.
This leaves extra space and solves potential problems with rounding
which can occur during the unpacking of the variable.
Thus for NC_SHORT, ndrv = 65536 - 1 = 65535.
Less often, the variable may be packed into type NC_CHAR,
where ndrv = 256 - 1 = 255, or type NC_INT where
where ndrv = 4294967295 - 1 = 4294967294.
unpack(x)scale_factor and add_offset.
If scale_factor is present for a variable, the data are
multiplied by the value scale_factor after the data are read.
If add_offset is present for a variable, then the
add_offset value is added to the data after the data are read.
If both scale_factor and add_offset attributes are
present, the data are first scaled by scale_factor before the
offset add_offset is added.
upk = scale_factor*pck + add_offset = (max-min)*pck/ndrv + 0.5*(min+max)
When scale_factor and add_offset are used for packing, the
associated variable (containing the packed data) is typically of type
byte or short, whereas the unpacked values are intended to
be of type int, float, or double.
An attribute's scale_factor and add_offset and
missing_value, if any, should all be of the type intended for the
unpacked data, i.e., int, float or double.
These intrinsic functions allow ncap to convert variables on disk among the available types supported by netCDF.
byte(x)NC_BYTE
Converts x to external type NC_BYTE, a C-type signed char.
char(x)NC_CHAR
Converts x to external type NC_CHAR, a C-type unsigned char.
double(x)NC_DOUBLE
Converts x to external type NC_DOUBLE, a C-type double.
float(x)NC_FLOAT
Converts x to external type NC_FLOAT, a C-type float.
int(x)NC_INT
Converts x to external type NC_INT, a C-type int.
short(x)NC_SHORT
Converts x to external type NC_SHORT, a C-type short.
ncap supports the standard mathematical functions supplied with most operating systems. Standard calculator notation is used for addition +, subtraction -, multiplication *, division /, exponentiation ^, and modulus %. The available elementary mathematical functions are:
abs(x)acos(x)acosh(x)asin(x)asinh(x)atan(x)atanh(x)ceil(x)cos(x)cosh(x)erf(x)erfc(x)exp(x)floor(x)gamma(x)ln(x)log(x)log10(x)nearbyint(x)pow(x,y)pow function,
integer arguments are promoted (see Type conversion) to type
NC_FLOAT before evaluation.
Example: pow(2,3) = 8
rint(x)round(x)sin(x)sinh(x)sqrt(x)tan(x)tanh(x)trunc(x)ncap contains a small (and growing) library of intrinsic functions. In addition to the standard mathematical functions (see Intrinsic mathematical functions), ncap currently supports packing and unpacking.
pack(x)NC_SHORT
with the two attributes required to unpack the variable,
scale_factor and add_offset, stored at the original
(unpacked) precision of the variable
27.
Let min and max be the minimum and maximum values
of x.
scale_factor = (max-min)/ndrv
where ndrv is the number of discrete representable values for
given type of packed variable.
The theoretical maximum value for ndrv is two raised to the
number of bits used to store the packed variable.
Thus if the variable is packed into type NC_SHORT, a two-byte
datatype, then there are at most 2^16 = 65536 distinct values
representible.
In practice, the number of discretely representible values is taken
to be one less than the theoretical maximum.
This leaves extra space and solves potential problems with rounding
which can occur during the unpacking of the variable.
Thus for NC_SHORT, ndrv = 65536 - 1 = 65535.
Less often, the variable may be packed into type NC_CHAR,
where ndrv = 256 - 1 = 255, or type NC_INT where
where ndrv = 4294967295 - 1 = 4294967294.
unpack(x)scale_factor and add_offset.
If scale_factor is present for a variable, the data are
multiplied by the value scale_factor after the data are read.
If add_offset is present for a variable, then the
add_offset value is added to the data after the data are read.
If both scale_factor and add_offset attributes are
present, the data are first scaled by scale_factor before the
offset add_offset is added.
upk = scale_factor*pck + add_offset = (max-min)*pck/ndrv + 0.5*(min+max)
When scale_factor and add_offset are used for packing, the
associated variable (containing the packed data) is typically of type
byte or short, whereas the unpacked values are intended to
be of type int, float, or double.
An attribute's scale_factor and add_offset and
missing_value, if any, should all be of the type intended for the
unpacked data, i.e., int, float or double.
These intrinsic functions allow ncap to convert variables on disk among the available types supported by netCDF.
byte(x)NC_BYTE
Converts x to external type NC_BYTE, a C-type signed char.
char(x)NC_CHAR
Converts x to external type NC_CHAR, a C-type unsigned char.
double(x)NC_DOUBLE
Converts x to external type NC_DOUBLE, a C-type double.
float(x)NC_FLOAT
Converts x to external type NC_FLOAT, a C-type float.
int(x)NC_INT
Converts x to external type NC_INT, a C-type int.
short(x)NC_SHORT
Converts x to external type NC_SHORT, a C-type short.
ncap supports the standard mathematical functions supplied with most operating systems. Standard calculator notation is used for addition +, subtraction -, multiplication *, division /, exponentiation ^, and modulus %. The available elementary mathematical functions are:
abs(x)acos(x)acosh(x)asin(x)asinh(x)atan(x)atanh(x)ceil(x)cos(x)cosh(x)erf(x)erfc(x)exp(x)floor(x)gamma(x)ln(x)log(x)log10(x)nearbyint(x)pow(x,y)pow function,
integer arguments are promoted (see Type conversion) to type
NC_FLOAT before evaluation.
Example: pow(2,3) = 8
rint(x)round(x)sin(x)sinh(x)sqrt(x)tan(x)tanh(x)trunc(x)ncap contains a small (and growing) library of intrinsic functions. In addition to the standard mathematical functions (see Intrinsic mathematical functions), ncap currently supports packing and unpacking.
pack(x)NC_SHORT
with the two attributes required to unpack the variable,
scale_factor and add_offset, stored at the original
(unpacked) precision of the variable
27.
Let min and max be the minimum and maximum values
of x.
scale_factor = (max-min)/ndrv
where ndrv is the number of discrete representable values for
given type of packed variable.
The theoretical maximum value for ndrv is two raised to the
number of bits used to store the packed variable.
Thus if the variable is packed into type NC_SHORT, a two-byte
datatype, then there are at most 2^16 = 65536 distinct values
representible.
In practice, the number of discretely representible values is taken
to be one less than the theoretical maximum.
This leaves extra space and solves potential problems with rounding
which can occur during the unpacking of the variable.
Thus for NC_SHORT, ndrv = 65536 - 1 = 65535.
Less often, the variable may be packed into type NC_CHAR,
where ndrv = 256 - 1 = 255, or type NC_INT where
where ndrv = 4294967295 - 1 = 4294967294.
unpack(x)scale_factor and add_offset.
If scale_factor is present for a variable, the data are
multiplied by the value scale_factor after the data are read.
If add_offset is present for a variable, then the
add_offset value is added to the data after the data are read.
If both scale_factor and add_offset attributes are
present, the data are first scaled by scale_factor before the
offset add_offset is added.
upk = scale_factor*pck + add_offset = (max-min)*pck/ndrv + 0.5*(min+max)
When scale_factor and add_offset are used for packing, the
associated variable (containing the packed data) is typically of type
byte or short, whereas the unpacked values are intended to
be of type int, float, or double.
An attribute's scale_factor and add_offset and
missing_value, if any, should all be of the type intended for the
unpacked data, i.e., int, float or double.
These intrinsic functions allow ncap to convert variables on disk among the available types supported by netCDF.
byte(x)NC_BYTE
Converts x to external type NC_BYTE, a C-type signed char.
char(x)NC_CHAR
Converts x to external type NC_CHAR, a C-type unsigned char.
double(x)NC_DOUBLE
Converts x to external type NC_DOUBLE, a C-type double.
float(x)NC_FLOAT
Converts x to external type NC_FLOAT, a C-type float.
int(x)NC_INT
Converts x to external type NC_INT, a C-type int.
short(x)NC_SHORT
Converts x to external type NC_SHORT, a C-type short.
ncap supports the standard mathematical functions supplied with most operating systems. Standard calculator notation is used for addition +, subtraction -, multiplication *, division /, exponentiation ^, and modulus %. The available elementary mathematical functions are:
abs(x)acos(x)acosh(x)asin(x)asinh(x)atan(x)atanh(x)ceil(x)cos(x)cosh(x)erf(x)erfc(x)exp(x)floor(x)gamma(x)ln(x)log(x)log10(x)nearbyint(x)pow(x,y)pow function,
integer arguments are promoted (see Type conversion) to type
NC_FLOAT before evaluation.
Example: pow(2,3) = 8
rint(x)round(x)sin(x)sinh(x)sqrt(x)tan(x)tanh(x)trunc(x)ncap contains a small (and growing) library of intrinsic functions. In addition to the standard mathematical functions (see Intrinsic mathematical functions), ncap currently supports packing and unpacking.
pack(x)NC_SHORT
with the two attributes required to unpack the variable,
scale_factor and add_offset, stored at the original
(unpacked) precision of the variable
27.
Let min and max be the minimum and maximum values
of x.
scale_factor = (max-min)/ndrv
where ndrv is the number of discrete representable values for
given type of packed variable.
The theoretical maximum value for ndrv is two raised to the
number of bits used to store the packed variable.
Thus if the variable is packed into type NC_SHORT, a two-byte
datatype, then there are at most 2^16 = 65536 distinct values
representible.
In practice, the number of discretely representible values is taken
to be one less than the theoretical maximum.
This leaves extra space and solves potential problems with rounding
which can occur during the unpacking of the variable.
Thus for NC_SHORT, ndrv = 65536 - 1 = 65535.
Less often, the variable may be packed into type NC_CHAR,
where ndrv = 256 - 1 = 255, or type NC_INT where
where ndrv = 4294967295 - 1 = 4294967294.
unpack(x)scale_factor and add_offset.
If scale_factor is present for a variable, the data are
multiplied by the value scale_factor after the data are read.
If add_offset is present for a variable, then the
add_offset value is added to the data after the data are read.
If both scale_factor and add_offset attributes are
present, the data are first scaled by scale_factor before the
offset add_offset is added.
upk = scale_factor*pck + add_offset = (max-min)*pck/ndrv + 0.5*(min+max)
When scale_factor and add_offset are used for packing, the
associated variable (containing the packed data) is typically of type
byte or short, whereas the unpacked values are intended to
be of type int, float, or double.
An attribute's scale_factor and add_offset and
missing_value, if any, should all be of the type intended for the
unpacked data, i.e., int, float or double.
These intrinsic functions allow ncap to convert variables on disk among the available types supported by netCDF.
byte(x)NC_BYTE
Converts x to external type NC_BYTE, a C-type signed char.
char(x)NC_CHAR
Converts x to external type NC_CHAR, a C-type unsigned char.
double(x)NC_DOUBLE
Converts x to external type NC_DOUBLE, a C-type double.
float(x)NC_FLOAT
Converts x to external type NC_FLOAT, a C-type float.
int(x)NC_INT
Converts x to external type NC_INT, a C-type int.
short(x)NC_SHORT
Converts x to external type NC_SHORT, a C-type short.
ncap supports the standard mathematical functions supplied with most operating systems. Standard calculator notation is used for addition +, subtraction -, multiplication *, division /, exponentiation ^, and modulus %. The available elementary mathematical functions are:
abs(x)acos(x)acosh(x)asin(x)asinh(x)atan(x)atanh(x)ceil(x)cos(x)cosh(x)erf(x)erfc(x)exp(x)floor(x)gamma(x)ln(x)log(x)log10(x)nearbyint(x)pow(x,y)pow function,
integer arguments are promoted (see Type conversion) to type
NC_FLOAT before evaluation.
Example: pow(2,3) = 8
rint(x)round(x)sin(x)sinh(x)sqrt(x)tan(x)tanh(x)trunc(x)ncap contains a small (and growing) library of intrinsic functions. In addition to the standard mathematical functions (see Intrinsic mathematical functions), ncap currently supports packing and unpacking.
pack(x)NC_SHORT
with the two attributes required to unpack the variable,
scale_factor and add_offset, stored at the original
(unpacked) precision of the variable
27.
Let min and max be the minimum and maximum values
of x.
scale_factor = (max-min)/ndrv
where ndrv is the number of discrete representable values for
given type of packed variable.
The theoretical maximum value for ndrv is two raised to the
number of bits used to store the packed variable.
Thus if the variable is packed into type NC_SHORT, a two-byte
datatype, then there are at most 2^16 = 65536 distinct values
representible.
In practice, the number of discretely representible values is taken
to be one less than the theoretical maximum.
This leaves extra space and solves potential problems with rounding
which can occur during the unpacking of the variable.
Thus for NC_SHORT, ndrv = 65536 - 1 = 65535.
Less often, the variable may be packed into type NC_CHAR,
where ndrv = 256 - 1 = 255, or type NC_INT where
where ndrv = 4294967295 - 1 = 4294967294.
unpack(x)scale_factor and add_offset.
If scale_factor is present for a variable, the data are
multiplied by the value scale_factor after the data are read.
If add_offset is present for a variable, then the
add_offset value is added to the data after the data are read.
If both scale_factor and add_offset attributes are
present, the data are first scaled by scale_factor before the
offset add_offset is added.
upk = scale_factor*pck + add_offset = (max-min)*pck/ndrv + 0.5*(min+max)
When scale_factor and add_offset are used for packing, the
associated variable (containing the packed data) is typically of type
byte or short, whereas the unpacked values are intended to
be of type int, float, or double.
An attribute's scale_factor and add_offset and
missing_value, if any, should all be of the type intended for the
unpacked data, i.e., int, float or double.
These intrinsic functions allow ncap to convert variables on disk among the available types supported by netCDF.
byte(x)NC_BYTE
Converts x to external type NC_BYTE, a C-type signed char.
char(x)NC_CHAR
Converts x to external type NC_CHAR, a C-type unsigned char.
double(x)NC_DOUBLE
Converts x to external type NC_DOUBLE, a C-type double.
float(x)NC_FLOAT
Converts x to external type NC_FLOAT, a C-type float.
int(x)NC_INT
Converts x to external type NC_INT, a C-type int.
short(x)NC_SHORT
Converts x to external type NC_SHORT, a C-type short.
ncap supports the standard mathematical functions supplied with most operating systems. Standard calculator notation is used for addition +, subtraction -, multiplication *, division /, exponentiation ^, and modulus %. The available elementary mathematical functions are:
abs(x)acos(x)acosh(x)asin(x)asinh(x)atan(x)atanh(x)ceil(x)cos(x)cosh(x)erf(x)erfc(x)exp(x)floor(x)gamma(x)ln(x)log(x)log10(x)nearbyint(x)pow(x,y)pow function,
integer arguments are promoted (see Type conversion) to type
NC_FLOAT before evaluation.
Example: pow(2,3) = 8
rint(x)round(x)sin(x)sinh(x)sqrt(x)tan(x)tanh(x)trunc(x)ncap contains a small (and growing) library of intrinsic functions. In addition to the standard mathematical functions (see Intrinsic mathematical functions), ncap currently supports packing and unpacking.
pack(x)NC_SHORT
with the two attributes required to unpack the variable,
scale_factor and add_offset, stored at the original
(unpacked) precision of the variable
27.
Let min and max be the minimum and maximum values
of x.
scale_factor = (max-min)/ndrv
where ndrv is the number of discrete representable values for
given type of packed variable.
The theoretical maximum value for ndrv is two raised to the
number of bits used to store the packed variable.
Thus if the variable is packed into type NC_SHORT, a two-byte
datatype, then there are at most 2^16 = 65536 distinct values
representible.
In practice, the number of discretely representible values is taken
to be one less than the theoretical maximum.
This leaves extra space and solves potential problems with rounding
which can occur during the unpacking of the variable.
Thus for NC_SHORT, ndrv = 65536 - 1 = 65535.
Less often, the variable may be packed into type NC_CHAR,
where ndrv = 256 - 1 = 255, or type NC_INT where
where ndrv = 4294967295 - 1 = 4294967294.
unpack(x)scale_factor and add_offset.
If scale_factor is present for a variable, the data are
multiplied by the value scale_factor after the data are read.
If add_offset is present for a variable, then the
add_offset value is added to the data after the data are read.
If both scale_factor and add_offset attributes are
present, the data are first scaled by scale_factor before the
offset add_offset is added.
upk = scale_factor*pck + add_offset = (max-min)*pck/ndrv + 0.5*(min+max)
When scale_factor and add_offset are used for packing, the
associated variable (containing the packed data) is typically of type
byte or short, whereas the unpacked values are intended to
be of type int, float, or double.
An attribute's scale_factor and add_offset and
missing_value, if any, should all be of the type intended for the
unpacked data, i.e., int, float or double.
These intrinsic functions allow ncap to convert variables on disk among the available types supported by netCDF.
byte(x)NC_BYTE
Converts x to external type NC_BYTE, a C-type signed char.
char(x)NC_CHAR
Converts x to external type NC_CHAR, a C-type unsigned char.
double(x)NC_DOUBLE
Converts x to external type NC_DOUBLE, a C-type double.
float(x)NC_FLOAT
Converts x to external type NC_FLOAT, a C-type float.
int(x)NC_INT
Converts x to external type NC_INT, a C-type int.
short(x)NC_SHORT
Converts x to external type NC_SHORT, a C-type short.
ncap supports the standard mathematical functions supplied with most operating systems. Standard calculator notation is used for addition +, subtraction -, multiplication *, division /, exponentiation ^, and modulus %. The available elementary mathematical functions are:
abs(x)acos(x)acosh(x)asin(x)asinh(x)atan(x)atanh(x)ceil(x)cos(x)cosh(x)erf(x)erfc(x)exp(x)floor(x)gamma(x)ln(x)log(x)log10(x)nearbyint(x)pow(x,y)pow function,
integer arguments are promoted (see Type conversion) to type
NC_FLOAT before evaluation.
Example: pow(2,3) = 8
rint(x)round(x)sin(x)sinh(x)sqrt(x)tan(x)tanh(x)trunc(x)ncap contains a small (and growing) library of intrinsic functions. In addition to the standard mathematical functions (see Intrinsic mathematical functions), ncap currently supports packing and unpacking.
pack(x)NC_SHORT
with the two attributes required to unpack the variable,
scale_factor and add_offset, stored at the original
(unpacked) precision of the variable
27.
Let min and max be the minimum and maximum values
of x.
scale_factor = (max-min)/ndrv
where ndrv is the number of discrete representable values for
given type of packed variable.
The theoretical maximum value for ndrv is two raised to the
number of bits used to store the packed variable.
Thus if the variable is packed into type NC_SHORT, a two-byte
datatype, then there are at most 2^16 = 65536 distinct values
representible.
In practice, the number of discretely representible values is taken
to be one less than the theoretical maximum.
This leaves extra space and solves potential problems with rounding
which can occur during the unpacking of the variable.
Thus for NC_SHORT, ndrv = 65536 - 1 = 65535.
Less often, the variable may be packed into type NC_CHAR,
where ndrv = 256 - 1 = 255, or type NC_INT where
where ndrv = 4294967295 - 1 = 4294967294.
unpack(x)scale_factor and add_offset.
If scale_factor is present for a variable, the data are
multiplied by the value scale_factor after the data are read.
If add_offset is present for a variable, then the
add_offset value is added to the data after the data are read.
If both scale_factor and add_offset attributes are
present, the data are first scaled by scale_factor before the
offset add_offset is added.
upk = scale_factor*pck + add_offset = (max-min)*pck/ndrv + 0.5*(min+max)
When scale_factor and add_offset are used for packing, the
associated variable (containing the packed data) is typically of type
byte or short, whereas the unpacked values are intended to
be of type int, float, or double.
An attribute's scale_factor and add_offset and
missing_value, if any, should all be of the type intended for the
unpacked data, i.e., int, float or double.
These intrinsic functions allow ncap to convert variables on disk among the available types supported by netCDF.
byte(x)NC_BYTE
Converts x to external type NC_BYTE, a C-type signed char.
char(x)NC_CHAR
Converts x to external type NC_CHAR, a C-type unsigned char.
double(x)NC_DOUBLE
Converts x to external type NC_DOUBLE, a C-type double.
float(x)NC_FLOAT
Converts x to external type NC_FLOAT, a C-type float.
int(x)NC_INT
Converts x to external type NC_INT, a C-type int.
short(x)NC_SHORT
Converts x to external type NC_SHORT, a C-type short.
ncap supports the standard mathematical functions supplied with most operating systems. Standard calculator notation is used for addition +, subtraction -, multiplication *, division /, exponentiation ^, and modulus %. The available elementary mathematical functions are:
abs(x)acos(x)acosh(x)asin(x)asinh(x)atan(x)atanh(x)ceil(x)cos(x)cosh(x)erf(x)erfc(x)exp(x)floor(x)gamma(x)ln(x)log(x)log10(x)nearbyint(x)pow(x,y)pow function,
integer arguments are promoted (see Type conversion) to type
NC_FLOAT before evaluation.
Example: pow(2,3) = 8
rint(x)round(x)sin(x)sinh(x)sqrt(x)tan(x)tanh(x)trunc(x)ncap contains a small (and growing) library of intrinsic functions. In addition to the standard mathematical functions (see Intrinsic mathematical functions), ncap currently supports packing and unpacking.
pack(x)NC_SHORT
with the two attributes required to unpack the variable,
scale_factor and add_offset, stored at the original
(unpacked) precision of the variable
27.
Let min and max be the minimum and maximum values
of x.
scale_factor = (max-min)/ndrv
where ndrv is the number of discrete representable values for
given type of packed variable.
The theoretical maximum value for ndrv is two raised to the
number of bits used to store the packed variable.
Thus if the variable is packed into type NC_SHORT, a two-byte
datatype, then there are at most 2^16 = 65536 distinct values
representible.
In practice, the number of discretely representible values is taken
to be one less than the theoretical maximum.
This leaves extra space and solves potential problems with rounding
which can occur during the unpacking of the variable.
Thus for NC_SHORT, ndrv = 65536 - 1 = 65535.
Less often, the variable may be packed into type NC_CHAR,
where ndrv = 256 - 1 = 255, or type NC_INT where
where ndrv = 4294967295 - 1 = 4294967294.
unpack(x)scale_factor and add_offset.
If scale_factor is present for a variable, the data are
multiplied by the value scale_factor after the data are read.
If add_offset is present for a variable, then the
add_offset value is added to the data after the data are read.
If both scale_factor and add_offset attributes are
present, the data are first scaled by scale_factor before the
offset add_offset is added.
upk = scale_factor*pck + add_offset = (max-min)*pck/ndrv + 0.5*(min+max)
When scale_factor and add_offset are used for packing, the
associated variable (containing the packed data) is typically of type
byte or short, whereas the unpacked values are intended to
be of type int, float, or double.
An attribute's scale_factor and add_offset and
missing_value, if any, should all be of the type intended for the
unpacked data, i.e., int, float or double.
These intrinsic functions allow ncap to convert variables on disk among the available types supported by netCDF.
byte(x)NC_BYTE
Converts x to external type NC_BYTE, a C-type signed char.
char(x)NC_CHAR
Converts x to external type NC_CHAR, a C-type unsigned char.
double(x)NC_DOUBLE
Converts x to external type NC_DOUBLE, a C-type double.
float(x)NC_FLOAT
Converts x to external type NC_FLOAT, a C-type float.
int(x)NC_INT
Converts x to external type NC_INT, a C-type int.
short(x)NC_SHORT
Converts x to external type NC_SHORT, a C-type short.
ncap supports the standard mathematical functions supplied with most operating systems. Standard calculator notation is used for addition +, subtraction -, multiplication *, division /, exponentiation ^, and modulus %. The available elementary mathematical functions are:
abs(x)acos(x)acosh(x)asin(x)asinh(x)atan(x)atanh(x)ceil(x)cos(x)cosh(x)erf(x)erfc(x)exp(x)floor(x)gamma(x)ln(x)log(x)log10(x)nearbyint(x)pow(x,y)pow function,
integer arguments are promoted (see Type conversion) to type
NC_FLOAT before evaluation.
Example: pow(2,3) = 8
rint(x)round(x)sin(x)sinh(x)sqrt(x)tan(x)tanh(x)trunc(x)ncap contains a small (and growing) library of intrinsic functions. In addition to the standard mathematical functions (see Intrinsic mathematical functions), ncap currently supports packing and unpacking.
pack(x)NC_SHORT
with the two attributes required to unpack the variable,
scale_factor and add_offset, stored at the original
(unpacked) precision of the variable
27.
Let min and max be the minimum and maximum values
of x.
scale_factor = (max-min)/ndrv
where ndrv is the number of discrete representable values for
given type of packed variable.
The theoretical maximum value for ndrv is two raised to the
number of bits used to store the packed variable.
Thus if the variable is packed into type NC_SHORT, a two-byte
datatype, then there are at most 2^16 = 65536 distinct values
representible.
In practice, the number of discretely representible values is taken
to be one less than the theoretical maximum.
This leaves extra space and solves potential problems with rounding
which can occur during the unpacking of the variable.
Thus for NC_SHORT, ndrv = 65536 - 1 = 65535.
Less often, the variable may be packed into type NC_CHAR,
where ndrv = 256 - 1 = 255, or type NC_INT where
where ndrv = 4294967295 - 1 = 4294967294.
unpack(x)scale_factor and add_offset.
If scale_factor is present for a variable, the data are
multiplied by the value scale_factor after the data are read.
If add_offset is present for a variable, then the
add_offset value is added to the data after the data are read.
If both scale_factor and add_offset attributes are
present, the data are first scaled by scale_factor before the
offset add_offset is added.
upk = scale_factor*pck + add_offset = (max-min)*pck/ndrv + 0.5*(min+max)
When scale_factor and add_offset are used for packing, the
associated variable (containing the packed data) is typically of type
byte or short, whereas the unpacked values are intended to
be of type int, float, or double.
An attribute's scale_factor and add_offset and
missing_value, if any, should all be of the type intended for the
unpacked data, i.e., int, float or double.
These intrinsic functions allow ncap to convert variables on disk among the available types supported by netCDF.
byte(x)NC_BYTE
Converts x to external type NC_BYTE, a C-type signed char.
char(x)NC_CHAR
Converts x to external type NC_CHAR, a C-type unsigned char.
double(x)NC_DOUBLE
Converts x to external type NC_DOUBLE, a C-type double.
float(x)NC_FLOAT
Converts x to external type NC_FLOAT, a C-type float.
int(x)NC_INT
Converts x to external type NC_INT, a C-type int.
short(x)NC_SHORT
Converts x to external type NC_SHORT, a C-type short.
ncap supports the standard mathematical functions supplied with most operating systems. Standard calculator notation is used for addition +, subtraction -, multiplication *, division /, exponentiation ^, and modulus %. The available elementary mathematical functions are:
abs(x)acos(x)acosh(x)asin(x)asinh(x)atan(x)atanh(x)ceil(x)cos(x)cosh(x)erf(x)erfc(x)exp(x)floor(x)gamma(x)ln(x)log(x)log10(x)nearbyint(x)pow(x,y)pow function,
integer arguments are promoted (see Type conversion) to type
NC_FLOAT before evaluation.
Example: pow(2,3) = 8
rint(x)round(x)sin(x)sinh(x)sqrt(x)tan(x)tanh(x)trunc(x)ncap contains a small (and growing) library of intrinsic functions. In addition to the standard mathematical functions (see Intrinsic mathematical functions), ncap currently supports packing and unpacking.
pack(x)NC_SHORT
with the two attributes required to unpack the variable,
scale_factor and add_offset, stored at the original
(unpacked) precision of the variable
27.
Let min and max be the minimum and maximum values
of x.
scale_factor = (max-min)/ndrv
where ndrv is the number of discrete representable values for
given type of packed variable.
The theoretical maximum value for ndrv is two raised to the
number of bits used to store the packed variable.
Thus if the variable is packed into type NC_SHORT, a two-byte
datatype, then there are at most 2^16 = 65536 distinct values
representible.
In practice, the number of discretely representible values is taken
to be one less than the theoretical maximum.
This leaves extra space and solves potential problems with rounding
which can occur during the unpacking of the variable.
Thus for NC_SHORT, ndrv = 65536 - 1 = 65535.
Less often, the variable may be packed into type NC_CHAR,
where ndrv = 256 - 1 = 255, or type NC_INT where
where ndrv = 4294967295 - 1 = 4294967294.
unpack(x)scale_factor and add_offset.
If scale_factor is present for a variable, the data are
multiplied by the value scale_factor after the data are read.
If add_offset is present for a variable, then the
add_offset value is added to the data after the data are read.
If both scale_factor and add_offset attributes are
present, the data are first scaled by scale_factor before the
offset add_offset is added.
upk = scale_factor*pck + add_offset = (max-min)*pck/ndrv + 0.5*(min+max)
When scale_factor and add_offset are used for packing, the
associated variable (containing the packed data) is typically of type
byte or short, whereas the unpacked values are intended to
be of type int, float, or double.
An attribute's scale_factor and add_offset and
missing_value, if any, should all be of the type intended for the
unpacked data, i.e., int, float or double.
These intrinsic functions allow ncap to convert variables on disk among the available types supported by netCDF.
byte(x)NC_BYTE
Converts x to external type NC_BYTE, a C-type signed char.
char(x)NC_CHAR
Converts x to external type NC_CHAR, a C-type unsigned char.
double(x)NC_DOUBLE
Converts x to external type NC_DOUBLE, a C-type double.
float(x)NC_FLOAT
Converts x to external type NC_FLOAT, a C-type float.
int(x)NC_INT
Converts x to external type NC_INT, a C-type int.
short(x)NC_SHORT
Converts x to external type NC_SHORT, a C-type short.
ncap supports the standard mathematical functions supplied with most operating systems. Standard calculator notation is used for addition +, subtraction -, multiplication *, division /, exponentiation ^, and modulus %. The available elementary mathematical functions are:
abs(x)acos(x)acosh(x)asin(x)asinh(x)atan(x)atanh(x)ceil(x)cos(x)cosh(x)erf(x)erfc(x)exp(x)floor(x)gamma(x)ln(x)log(x)log10(x)nearbyint(x)pow(x,y)pow function,
integer arguments are promoted (see Type conversion) to type
NC_FLOAT before evaluation.
Example: pow(2,3) = 8
rint(x)round(x)sin(x)sinh(x)sqrt(x)tan(x)tanh(x)trunc(x)ncap contains a small (and growing) library of intrinsic functions. In addition to the standard mathematical functions (see Intrinsic mathematical functions), ncap currently supports packing and unpacking.
pack(x)NC_SHORT
with the two attributes required to unpack the variable,
scale_factor and add_offset, stored at the original
(unpacked) precision of the variable
27.
Let min and max be the minimum and maximum values
of x.
scale_factor = (max-min)/ndrv
where ndrv is the number of discrete representable values for
given type of packed variable.
The theoretical maximum value for ndrv is two raised to the
number of bits used to store the packed variable.
Thus if the variable is packed into type NC_SHORT, a two-byte
datatype, then there are at most 2^16 = 65536 distinct values
representible.
In practice, the number of discretely representible values is taken
to be one less than the theoretical maximum.
This leaves extra space and solves potential problems with rounding
which can occur during the unpacking of the variable.
Thus for NC_SHORT, ndrv = 65536 - 1 = 65535.
Less often, the variable may be packed into type NC_CHAR,
where ndrv = 256 - 1 = 255, or type NC_INT where
where ndrv = 4294967295 - 1 = 4294967294.
unpack(x)scale_factor and add_offset.
If scale_factor is present for a variable, the data are
multiplied by the value scale_factor after the data are read.
If add_offset is present for a variable, then the
add_offset value is added to the data after the data are read.
If both scale_factor and add_offset attributes are
present, the data are first scaled by scale_factor before the
offset add_offset is added.
upk = scale_factor*pck + add_offset = (max-min)*pck/ndrv + 0.5*(min+max)
When scale_factor and add_offset are used for packing, the
associated variable (containing the packed data) is typically of type
byte or short, whereas the unpacked values are intended to
be of type int, float, or double.
An attribute's scale_factor and add_offset and
missing_value, if any, should all be of the type intended for the
unpacked data, i.e., int, float or double.
These intrinsic functions allow ncap to convert variables on disk among the available types supported by netCDF.
byte(x)NC_BYTE
Converts x to external type NC_BYTE, a C-type signed char.
char(x)NC_CHAR
Converts x to external type NC_CHAR, a C-type unsigned char.
double(x)NC_DOUBLE
Converts x to external type NC_DOUBLE, a C-type double.
float(x)NC_FLOAT
Converts x to external type NC_FLOAT, a C-type float.
int(x)NC_INT
Converts x to external type NC_INT, a C-type int.
short(x)NC_SHORT
Converts x to external type NC_SHORT, a C-type short.
ncap supports the standard mathematical functions supplied with most operating systems. Standard calculator notation is used for addition +, subtraction -, multiplication *, division /, exponentiation ^, and modulus %. The available elementary mathematical functions are:
abs(x)acos(x)acosh(x)asin(x)asinh(x)atan(x)atanh(x)ceil(x)cos(x)cosh(x)erf(x)erfc(x)exp(x)floor(x)gamma(x)ln(x)log(x)log10(x)nearbyint(x)pow(x,y)pow function,
integer arguments are promoted (see Type conversion) to type
NC_FLOAT before evaluation.
Example: pow(2,3) = 8
rint(x)round(x)sin(x)sinh(x)sqrt(x)tan(x)tanh(x)trunc(x)ncap contains a small (and growing) library of intrinsic functions. In addition to the standard mathematical functions (see Intrinsic mathematical functions), ncap currently supports packing and unpacking.
pack(x)NC_SHORT
with the two attributes required to unpack the variable,
scale_factor and add_offset, stored at the original
(unpacked) precision of the variable
27.
Let min and max be the minimum and maximum values
of x.
scale_factor = (max-min)/ndrv
where ndrv is the number of discrete representable values for
given type of packed variable.
The theoretical maximum value for ndrv is two raised to the
number of bits used to store the packed variable.
Thus if the variable is packed into type NC_SHORT, a two-byte
datatype, then there are at most 2^16 = 65536 distinct values
representible.
In practice, the number of discretely representible values is taken
to be one less than the theoretical maximum.
This leaves extra space and solves potential problems with rounding
which can occur during the unpacking of the variable.
Thus for NC_SHORT, ndrv = 65536 - 1 = 65535.
Less often, the variable may be packed into type NC_CHAR,
where ndrv = 256 - 1 = 255, or type NC_INT where
where ndrv = 4294967295 - 1 = 4294967294.
unpack(x)scale_factor and add_offset.
If scale_factor is present for a variable, the data are
multiplied by the value scale_factor after the data are read.
If add_offset is present for a variable, then the
add_offset value is added to the data after the data are read.
If both scale_factor and add_offset attributes are
present, the data are first scaled by scale_factor before the
offset add_offset is added.
upk = scale_factor*pck + add_offset = (max-min)*pck/ndrv + 0.5*(min+max)
When scale_factor and add_offset are used for packing, the
associated variable (containing the packed data) is typically of type
byte or short, whereas the unpacked values are intended to
be of type int, float, or double.
An attribute's scale_factor and add_offset and
missing_value, if any, should all be of the type intended for the
unpacked data, i.e., int, float or double.
These intrinsic functions allow ncap to convert variables on disk among the available types supported by netCDF.
byte(x)NC_BYTE
Converts x to external type NC_BYTE, a C-type signed char.
char(x)NC_CHAR
Converts x to external type NC_CHAR, a C-type unsigned char.
double(x)NC_DOUBLE
Converts x to external type NC_DOUBLE, a C-type double.
float(x)NC_FLOAT
Converts x to external type NC_FLOAT, a C-type float.
int(x)NC_INT
Converts x to external type NC_INT, a C-type int.
short(x)NC_SHORT
Converts x to external type NC_SHORT, a C-type short.
ncap supports the standard mathematical functions supplied with most operating systems. Standard calculator notation is used for addition +, subtraction -, multiplication *, division /, exponentiation ^, and modulus %. The available elementary mathematical functions are:
abs(x)acos(x)acosh(x)asin(x)asinh(x)atan(x)atanh(x)ceil(x)cos(x)cosh(x)erf(x)erfc(x)exp(x)floor(x)gamma(x)ln(x)log(x)log10(x)nearbyint(x)pow(x,y)pow function,
integer arguments are promoted (see Type conversion) to type
NC_FLOAT before evaluation.
Example: pow(2,3) = 8
rint(x)round(x)sin(x)sinh(x)sqrt(x)tan(x)tanh(x)trunc(x)ncap contains a small (and growing) library of intrinsic functions. In addition to the standard mathematical functions (see Intrinsic mathematical functions), ncap currently supports packing and unpacking.
pack(x)NC_SHORT
with the two attributes required to unpack the variable,
scale_factor and add_offset, stored at the original
(unpacked) precision of the variable
27.
Let min and max be the minimum and maximum values
of x.
scale_factor = (max-min)/ndrv
where ndrv is the number of discrete representable values for
given type of packed variable.
The theoretical maximum value for ndrv is two raised to the
number of bits used to store the packed variable.
Thus if the variable is packed into type NC_SHORT, a two-byte
datatype, then there are at most 2^16 = 65536 distinct values
representible.
In practice, the number of discretely representible values is taken
to be one less than the theoretical maximum.
This leaves extra space and solves potential problems with rounding
which can occur during the unpacking of the variable.
Thus for NC_SHORT, ndrv = 65536 - 1 = 65535.
Less often, the variable may be packed into type NC_CHAR,
where ndrv = 256 - 1 = 255, or type NC_INT where
where ndrv = 4294967295 - 1 = 4294967294.
unpack(x)scale_factor and add_offset.
If scale_factor is present for a variable, the data are
multiplied by the value scale_factor after the data are read.
If add_offset is present for a variable, then the
add_offset value is added to the data after the data are read.
If both scale_factor and add_offset attributes are
present, the data are first scaled by scale_factor before the
offset add_offset is added.
upk = scale_factor*pck + add_offset = (max-min)*pck/ndrv + 0.5*(min+max)
When scale_factor and add_offset are used for packing, the
associated variable (containing the packed data) is typically of type
byte or short, whereas the unpacked values are intended to
be of type int, float, or double.
An attribute's scale_factor and add_offset and
missing_value, if any, should all be of the type intended for the
unpacked data, i.e., int, float or double.
These intrinsic functions allow ncap to convert variables on disk among the available types supported by netCDF.
byte(x)NC_BYTE
Converts x to external type NC_BYTE, a C-type signed char.
char(x)NC_CHAR
Converts x to external type NC_CHAR, a C-type unsigned char.
double(x)NC