Next: , Previous: ncflint netCDF File Interpolator, Up: Operator Reference Manual

4.7 ncks netCDF Kitchen Sink

SYNTAX 

     ncks [-4] [-A] [-a] [-B] [-b binary-file] [-C] [-c] [-D dbg]
     [-d dim,[min][,[max][,[stride]]]
     [-F] [-H] [-h] [--hdr_pad nbr] [-l path] [-M] [-m]
     [-O] [-o output-file]
     [-P] [-p path] [-Q] [-q] [-R] [-r] [-s format] [-u]
     [-v var[,...]] [-x] input-file [[output-file]]

DESCRIPTION

ncks combines selected features of ncdump, ncextr, and the nccut and ncpaste specifications into one versatile utility. ncks extracts a subset of the data from input-file and prints it as ASCII text to stdout, writes it in flat binary format to binary-file, and writes (or pastes) it in netCDF format to output-file.

ncks will print netCDF data in ASCII format to stdout, like ncdump, but with these differences: ncks prints data in a tabular format intended to be easy to search for the data you want, one datum per screen line, with all dimension subscripts and coordinate values (if any) preceding the datum. Option ‘-s’ (or long options ‘--sng_fmt’ and ‘--string’) lets the user format the data using C-style format strings.

Options ‘-a’, ‘-F’ , ‘-H’, ‘-M’, ‘-m’, ‘-P’, ‘-Q’, ‘-q’, ‘-s’, and ‘-u’ (and their long option counterparts) control the formatted appearance of the data.

ncks extracts (and optionally creates a new netCDF file comprised of) only selected variables from the input file (similar to the old ncextr specification). Only variables and coordinates may be specifically included or excluded—all global attributes and any attribute associated with an extracted variable are copied to the screen and/or output netCDF file. Options ‘-c’, ‘-C’, ‘-v’, and ‘-x’ (and their long option synonyms) control which variables are extracted.

ncks extracts hyperslabs from the specified variables (ncks implements the original nccut specification). Option ‘-d’ controls the hyperslab specification. Input dimensions that are not associated with any output variable do not appear in the output netCDF. This feature removes superfluous dimensions from netCDF files.

ncks will append variables and attributes from the input-file to output-file if output-file is a pre-existing netCDF file whose relevant dimensions conform to dimension sizes of input-file. The append features of ncks are intended to provide a rudimentary means of adding data from one netCDF file to another, conforming, netCDF file. If naming conflicts exist between the two files, data in output-file is usually overwritten by the corresponding data from input-file. Thus, when appending, the user should backup output-file in case valuable data are inadvertantly overwritten.

If output-file exists, the user will be queried whether to overwrite, append, or exit the ncks call completely. Choosing overwrite destroys the existing output-file and create an entirely new one from the output of the ncks call. Append has differing effects depending on the uniqueness of the variables and attributes output by ncks: If a variable or attribute extracted from input-file does not have a name conflict with the members of output-file then it will be added to output-file without overwriting any of the existing contents of output-file. In this case the relevant dimensions must agree (conform) between the two files; new dimensions are created in output-file as required. When a name conflict occurs, a global attribute from input-file will overwrite the corresponding global attribute from output-file. If the name conflict occurs for a non-record variable, then the dimensions and type of the variable (and of its coordinate dimensions, if any) must agree (conform) in both files. Then the variable values (and any coordinate dimension values) from input-file will overwrite the corresponding variable values (and coordinate dimension values, if any) in output-file 1.

Since there can only be one record dimension in a file, the record dimension must have the same name (but not necessarily the same size) in both files if a record dimension variable is to be appended. If the record dimensions are of differing sizes, the record dimension of output-file will become the greater of the two record dimension sizes, the record variable from input-file will overwrite any counterpart in output-file and fill values will be written to any gaps left in the rest of the record variables (I think). In all cases variable attributes in output-file are superseded by attributes of the same name from input-file, and left alone if there is no name conflict.

Some users may wish to avoid interactive ncks queries about whether to overwrite existing data. For example, batch scripts will fail if ncks does not receive responses to its queries. Options ‘-O’ and ‘-A’ are available to force overwriting existing files and variables, respectively.

Options specific to ncks

The following list provides a short summary of the features unique to ncks. Features common to many operators are described in Common features.

‘-a’
Do not alphabetize extracted fields. By default, the specified output variables are extracted, printed, and written to disk in alphabetical order. This tends to make long output lists easier to search for particular variables. Specifying -a results in the variables being extracted, printed, and written to disk in the order in which they were saved in the input file. Thus -a retains the original ordering of the variables. Also ‘--abc’ and ‘--alphabetize’.


‘-B file’
Activate native machine binary output writing to the default binary file, ncks.bnr. The -B switch is redundant when the -b file option is specified, and native binary output will be directed to the binary file file. Also ‘--bnr’ and ‘--binary’. Writing packed variables in binary format is not supported.


‘-b file’
Activate native machine binary output writing to binary file file. Also ‘--fl_bnr’ and ‘--binary-file’. Writing packed variables in binary format is not supported.


‘-d dim,[min][,[max][,[stride]]]’
Add stride argument to hyperslabber. For a complete description of the stride argument, See Stride.


‘-H’
Print data to screen. Also activated using ‘--print’ or ‘--prn’. By default ncks prints all metadata and data to screen if no netCDF output file is specified. Use ‘-H’ to print data to screen if a netCDF output is specified, or to restrict printing to data (no metadata) when no netCDF output is specified. Unless otherwise specified (with -s), each element of the data hyperslab prints on a separate line containing the names, indices, and, values, if any, of all of the variables dimensions. The dimension and variable indices refer to the location of the corresponding data element with respect to the variable as stored on disk (i.e., not the hyperslab). 
          % ncks -C -v three_dmn_var in.nc
          lat[0]=-90 lev[0]=100 lon[0]=0 three_dmn_var[0]=0
          lat[0]=-90 lev[0]=100 lon[1]=90 three_dmn_var[1]=1
          lat[0]=-90 lev[0]=100 lon[2]=180 three_dmn_var[2]=2
          ...
          lat[1]=90 lev[2]=1000 lon[1]=90 three_dmn_var[21]=21
          lat[1]=90 lev[2]=1000 lon[2]=180 three_dmn_var[22]=22
          lat[1]=90 lev[2]=1000 lon[3]=270 three_dmn_var[23]=23

Printing the same variable with the ‘-F’ option shows the same variable indexed with Fortran conventions 

          % ncks -F -C -v three_dmn_var in.nc
          lon(1)=0 lev(1)=100 lat(1)=-90 three_dmn_var(1)=0
          lon(2)=90 lev(1)=100 lat(1)=-90 three_dmn_var(2)=1
          lon(3)=180 lev(1)=100 lat(1)=-90 three_dmn_var(3)=2
          ...

Printing a hyperslab does not affect the variable or dimension indices since these indices are relative to the full variable (as stored in the input file), and the input file has not changed. However, if the hypserslab is saved to an output file and those values are printed, the indices will change: 

          % ncks -O -H -d lat,90.0 -d lev,1000.0 -v three_dmn_var in.nc out.nc
          ...
          lat[1]=90 lev[2]=1000 lon[0]=0 three_dmn_var[20]=20
          lat[1]=90 lev[2]=1000 lon[1]=90 three_dmn_var[21]=21
          lat[1]=90 lev[2]=1000 lon[2]=180 three_dmn_var[22]=22
          lat[1]=90 lev[2]=1000 lon[3]=270 three_dmn_var[23]=23
          % ncks -C -v three_dmn_var out.nc
          lat[0]=90 lev[0]=1000 lon[0]=0 three_dmn_var[0]=20
          lat[0]=90 lev[0]=1000 lon[1]=90 three_dmn_var[1]=21
          lat[0]=90 lev[0]=1000 lon[2]=180 three_dmn_var[2]=22
          lat[0]=90 lev[0]=1000 lon[3]=270 three_dmn_var[3]=23


‘-M’
Print to screen the global metadata describing the file. This includes file summary information and global attributes. Also ‘--Mtd’ and ‘--Metadata’. By default ncks prints global metadata to screen if no netCDF output file and no variable extraction list is specified (with ‘-v’). Use ‘-M’ to print global metadata to screen if a netCDF output is specified, or if a variable extraction list is specified (with ‘-v’).

The various combinations of printing switches can be confusing. In an attempt to anticipate what most users want to do, ncks uses context-sensitive defaults for printing. Our goal is to minimize the use of switches required to accomplish the common operations. We assume that users creating a new file or overwriting (e.g., with ‘-O’) an existing file usually wish to copy all global and variable-specific attributes to the new file. In contrast, we assume that users appending (e.g., with ‘-A’ an explicit variable list from one file to another usually wish to copy only the variable-specific attributes to the output file. The switches ‘-H’, ‘-M’, and ‘-m’ switches are implemented as toggles which reverse the default behavior. The most confusing aspect of this is that ‘-M’ inhibits copying global metadata in overwrite mode and causes copying of global metadata in append mode. 

          ncks -O              in.nc out.nc # Copy   all variable and global attributes
          ncks -O       -v one in.nc out.nc # Copy   all variable and global attributes
          ncks -O -M    -v one in.nc out.nc # Copy   variable but not global attributes
          ncks -O    -m -v one in.nc out.nc # Copy   global but not variable attributes
          ncks -O -M -m -v one in.nc out.nc # Copy   only data (no attributes)
          ncks -A              in.nc out.nc # Append all variable and global attributes
          ncks -A       -v one in.nc out.nc # Append variable but not global attributes
          ncks -A -M    -v one in.nc out.nc # Append all variable and global attributes
          ncks -A    -m -v one in.nc out.nc # Append only data (no attributes)
          ncks -A -M -m -v one in.nc out.nc # Append global but not variable attributes


‘-m’
Print variable metadata to screen (similar to ncdump -h). This displays all metadata pertaining to each variable, one variable at a time. Also ‘--mtd’ and ‘--metadata’. The ncks default behavior is to print variable metadata to screen if no netCDF output file is specified. Use ‘-m’ to print variable metadata to screen if a netCDF output is specified.


‘-P’
Print data, metadata, and units to screen. The ‘-P’ switch is a convenience abbreviation for ‘-C -H -M -m -u’. Also activated using ‘--print’ or ‘--prn’. This set of switches is useful for exploring file contents.

-Q’, ‘-q’, ‘-s’, and ‘-u’ (and their long option counterparts) control the formatted appearance of the data.

ncks extracts (and optionally creates a new netCDF file comprised of) only selected variables from the input file (similar to the old ncextr specification). Only variables and coordinates may be specifically included or excluded—all global attributes and any attribute associated with an extracted variable are copied to the screen and/or output netCDF file. Options ‘-c’, ‘-C’, ‘-v’, and ‘-x’ (and their long option synonyms) control which variables are extracted.

ncks extracts hyperslabs from the specified variables (ncks implements the original nccut specification). Option ‘-d’ controls the hyperslab specification. Input dimensions that are not associated with any output variable do not appear in the output netCDF. This feature removes superfluous dimensions from netCDF files.

ncks will append variables and attributes from the input-file to output-file if output-file is a pre-existing netCDF file whose relevant dimensions conform to dimension sizes of input-file. The append features of ncks are intended to provide a rudimentary means of adding data from one netCDF file to another, conforming, netCDF file. If naming conflicts exist between the two files, data in output-file is usually overwritten by the corresponding data from input-file. Thus, when appending, the user should backup output-file in case valuable data are inadvertantly overwritten.

If output-file exists, the user will be queried whether to overwrite, append, or exit the ncks call completely. Choosing overwrite destroys the existing output-file and create an entirely new one from the output of the ncks call. Append has differing effects depending on the uniqueness of the variables and attributes output by ncks: If a variable or attribute extracted from input-file does not have a name conflict with the members of output-file then it will be added to output-file without overwriting any of the existing contents of output-file. In this case the relevant dimensions must agree (conform) between the two files; new dimensions are created in output-file as required. When a name conflict occurs, a global attribute from input-file will overwrite the corresponding global attribute from output-file. If the name conflict occurs for a non-record variable, then the dimensions and type of the variable (and of its coordinate dimensions, if any) must agree (conform) in both files. Then the variable values (and any coordinate dimension values) from input-file will overwrite the corresponding variable values (and coordinate dimension values, if any) in output-file 1.

Since there can only be one record dimension in a file, the record dimension must have the same name (but not necessarily the same size) in both files if a record dimension variable is to be appended. If the record dimensions are of differing sizes, the record dimension of output-file will become the greater of the two record dimension sizes, the record variable from input-file will overwrite any counterpart in output-file and fill values will be written to any gaps left in the rest of the record variables (I think). In all cases variable attributes in output-file are superseded by attributes of the same name from input-file, and left alone if there is no name conflict.

Some users may wish to avoid interactive ncks queries about whether to overwrite existing data. For example, batch scripts will fail if ncks does not receive responses to its queries. Options ‘-O’ and ‘-A’ are available to force overwriting existing files and variables, respectively.

Options specific to ncks

The following list provides a short summary of the features unique to ncks. Features common to many operators are described in Common features.

‘-a’
Do not alphabetize extracted fields. By default, the specified output variables are extracted, printed, and written to disk in alphabetical order. This tends to make long output lists easier to search for particular variables. Specifying -a results in the variables being extracted, printed, and written to disk in the order in which they were saved in the input file. Thus -a retains the original ordering of the variables. Also ‘--abc’ and ‘--alphabetize’.


‘-B file’
Activate native machine binary output writing to the default binary file, ncks.bnr. The -B switch is redundant when the -b file option is specified, and native binary output will be directed to the binary file file. Also ‘--bnr’ and ‘--binary’. Writing packed variables in binary format is not supported.


‘-b file’
Activate native machine binary output writing to binary file file. Also ‘--fl_bnr’ and ‘--binary-file’. Writing packed variables in binary format is not supported.


‘-d dim,[min][,[max][,[stride]]]’
Add stride argument to hyperslabber. For a complete description of the stride argument, See Stride.


‘-H’
Print data to screen. Also activated using ‘--print’ or ‘--prn’. By default ncks prints all metadata and data to screen if no netCDF output file is specified. Use ‘-H’ to print data to screen if a netCDF output is specified, or to restrict printing to data (no metadata) when no netCDF output is specified. Unless otherwise specified (with -s), each element of the data hyperslab prints on a separate line containing the names, indices, and, values, if any, of all of the variables dimensions. The dimension and variable indices refer to the location of the corresponding data element with respect to the variable as stored on disk (i.e., not the hyperslab). 
          % ncks -C -v three_dmn_var in.nc
          lat[0]=-90 lev[0]=100 lon[0]=0 three_dmn_var[0]=0
          lat[0]=-90 lev[0]=100 lon[1]=90 three_dmn_var[1]=1
          lat[0]=-90 lev[0]=100 lon[2]=180 three_dmn_var[2]=2
          ...
          lat[1]=90 lev[2]=1000 lon[1]=90 three_dmn_var[21]=21
          lat[1]=90 lev[2]=1000 lon[2]=180 three_dmn_var[22]=22
          lat[1]=90 lev[2]=1000 lon[3]=270 three_dmn_var[23]=23

Printing the same variable with the ‘-F’ option shows the same variable indexed with Fortran conventions 

          % ncks -F -C -v three_dmn_var in.nc
          lon(1)=0 lev(1)=100 lat(1)=-90 three_dmn_var(1)=0
          lon(2)=90 lev(1)=100 lat(1)=-90 three_dmn_var(2)=1
          lon(3)=180 lev(1)=100 lat(1)=-90 three_dmn_var(3)=2
          ...

Printing a hyperslab does not affect the variable or dimension indices since these indices are relative to the full variable (as stored in the input file), and the input file has not changed. However, if the hypserslab is saved to an output file and those values are printed, the indices will change: 

          % ncks -O -H -d lat,90.0 -d lev,1000.0 -v three_dmn_var in.nc out.nc
          ...