NOUSERMACRO(natlog protocol)

includefile(../../release.yo)

htmlbodyopt(text)(#27408B)
htmlbodyopt(bgcolor)(#FFFAF0)
whenhtml(mailto(Frank B. Brokken: f.b.brokken@rug.nl))

DEFINEMACRO(lsoption)(2)(\
    bf(--ARG1) (bf(-ARG2))\
)
DEFINEMACRO(lsaoption)(3)(\
    bf(--ARG1)=tt(ARG3) (bf(-ARG2))\
)
DEFINEMACRO(laoption)(2)(\
    bf(--ARG1)=tt(ARG2)\
)
DEFINEMACRO(loption)(1)(\
    bf(--ARG1)\
)
DEFINEMACRO(soption)(1)(\
    bf(-ARG1)\
)

DEFINEMACRO(itx)(0)()
DEFINEMACRO(itemlist)(1)(ARG1)

DELETEMACRO(tt)
DEFINEMACRO(tt)(1)(em(ARG1))

COMMENT( man-request, section, date, distribution file, general name)
manpage(natlog)(1)(_CurYrs_)(natlog._CurVers_.tar.gz)
        (natlog)

COMMENT( man-request, larger title )
manpagename(natlog)(source-nat logging tool)

COMMENT( all other: add after () )
manpagesynopsis()
       bf(natlog) [OPTIONS] tt(command)

manpagesection(DESCRIPTION)

    Firewalls like bf(iptables)(1) may offer tt(POSTROUTING) (source network
address translation, snat) facilities changing the source address of a host
behind the firewall to the address of the host connected to the outer
world. With tt(snat) the following combinations of IP addresses and port
numbers are encountered:
    itemization(
    it() the IP address and port number used by the host protected by (i.e.,
behind) the firewall initiates a connection to the outer world (the source
host, in this manual page referred to as tt(IPsrc, sport));
    it() the IP address and port number of the host outside (i.e., before) the
firewall that tt(IPsrc) connects to (the destination host, in this manual page
referred to as tt(IPdst, dport));
    it() the IP address and port number of the host where the firewall has
been installed. This host performs the source natting, and its IP-address and
the port it uses when forwarding tt(IPsrc, sport)'s requests to tt(IPdst,
dport) are in this manual page referred to as tt(IPfw, fwport)).
    )

    Source natting usually uses tt(sport) for tt(fwport), but tt(fwport) may
already be in use, in which case the firewalling host must use another,
available port to forward communication from tt(IPsrc, sport) to tt(IPdst,
dport). 

    The general scheme that applies to source natting, therefore, looks like
this:
        verb(
    IPsrc:sport is translated by the firewall to IPfw:fwport;
    IPfw:fwport is used when communicating with IPdst:dport.
        )
    From the perspective of the destination host the communication originates
at tt(IPfw::fwport) and consequently all communication (e.g., incident
reports) sent by the systems administrator maintaining tt(IPdst) to tt(IPfw)'s
systems administrator will refer to tt(IPfw:fwport), rather than to
tt(IPsrc::sport).

    
Relating tt(IPfw:fwport) to tt(IPsrc:sport) is difficult when merely using the
standard log facilities provided by tt(iptables) and tt(natlog) was developed
to fill in that particular niche.

    tt(Natlog) provides data about source natting in various forms. The
standard logging mode consists of messages sent to the syslog daemon (cf.,
bf(rsyslogd)(1)) and/or to the standard output stream showing the essential
characteristics of connections using source natting. Here is an example of a
logged message (log-entries occupy single lines; the line-breaks below are to
enhance readability):
        verb(
    NATLOG: from 1338990672:55588 thru 1338990747:807100 (UTC): tcp
        192.168.19.72:4467 (via: 129.125.90.132:4467) to 
        to 200.49.219.180:443; sent: 802, received: 7669
        )
    The values tt(1338990672:55588) and tt(1338990747:807100) are time stamps
showing the begin- and end-times in seconds:microseconds of a tcp connection
since the beginning of the epoch (Jan 1, 1970, 0:00 UTC). tt(Natlog) offers
the tt(--time) option for requesting human-readable time specifications like
tt(Nov 2 13:29:11) rather than time representations using seconds and micro
seconds.

    The next value (tt(192.168.19.72:4467)) represents tt(IPsrc::sport). This
is followed by tt(129.125.90.132:4467), representing tt(IPfw:fwport). The
third pair of values (tt(200.49.219.180:443)) represents tt(IPdst:dport). 

    In this example, host tt(192.168.19.72), using port tt(4467), connected to
host tt(200.49.219.180), port tt(443). To this latter host the connection
appears to have originated from tt(129.125.90.132) port tt(4467). The log
message allows us to associate this with the `real' host and port from which
the connection originated: tt(192.168.19.72:4467).

    The final entries show the number of bytes that were sent by the
source-host (tt(IPsrc)) and received from the destination-host (tt(IPdst)).

    When tt(natlog) is terminated it can no longer track connections that are
still open. If tt(natlog) was terminated (by a tt(SIGINT) or tt(SIGTERM)
signal), then it logs a `terminating' line, followed by an overview of all
(potentially) still open connections. Those connections are flagged with a
trailing '(EOP)' (end of program) log-element, and their end-times show
tt(natlog's) termination time. Incomplete connections show tt((EXPIRED)).

    In addition to the standard logs the option tt(--log-data) is
available. This option requires the path to a file where information is logged
in tabular form, which can easily be processed by statistical software like
bf(R)(1). When specifying this option information will be appended to an
existing file. When the log file does not yet exist it is created. The first
line of the thus written log files names the columns of the table. The column
names are (all on one line):
    verb(
    type, srcNr, srcIP, srcPort, dstNr, dstIP, dstPort, 
        sent, recvd, begin,  end, beginTime, endTime, status 
    )
    Most column labels will be self-explanatory. tt(Type) indicates the
connection type, logged as tt(icmp, tcp,) or tt(udp); tt(srcNr) and tt(dstNr)
are the 32 bit numeric values of, respectively, the source host's IP address
and the destination host's IP address (decimal representations); tt(begin) and
tt(end) are the times in seconds since the beginning of the epoch,
corresponding to the times displayed at, respectively, tt(beginTime) and
tt(endTime); tt(status) indicates the status of the logged connection
information: tt(ok) indicates a connection that was normally completed;
tt(expired) indicates that the connection was recognized, but was not
normally completed; tt(eop) is used for connections that were still active by
the time tt(natlog) terminates. When the status equals tt(expired), the
time entries show the times of receiving the first and last packets of that
connection; when tt(eop), then the tt(end) and tt(endTime) entries show
tt(natlog's) termination time.

Log entries look like this (each entry occupies one line, header line and
logged data lines are right-aligned):
    verb(
    tcp, 101820608,    192.168.17.6,        48886, 
        4012145084,  188.121.36.239,           80,       
               430,            2266,   1517387644,    1517387644, 
        Jan 31 08:34:04:318340, Jan 31 08:34:04:383170,  ok
    )

manpagesection(MODES AND COMMANDS)

    itemization(
    it() tt(conntrack): the `conntrack'-mode. This command can only be used on
platforms using bf(iptables)(1) where bf(conntrack)(1) has also been
installed. Information about source-natted connections is obtained from
bf(conntrack)(1)'s output. In this mode all, or one of the (by default) tcp,
udp, and icmp layer four protocols can be monitored.

    By default tt(conntrack) does not report byte counts. To have conntrack
report byte counts the command
    verb(
     $ echo 1 > /proc/sys/net/netfilter/nf_conntrack_acct
    )
    must have been issued before starting tt(conntrack). 

tt(Conntrack) includes the sizes of the IP headers (usually 20 bytes) in
reported byte counts. Thus, tt(icmp) packets are usually reported as having
size 84, even though bf(ping)(1) reports a payload of 64 bytes. Since the
actual sizes of IP headers cannot be determined from conntrack's output, the
sizes reported when using tt(natlog's) conntrack mode are as reported by
tt(conntrack), and are therefore not corrected for IP header lengths. The
option tt(--conntrack-ip-header-size) can be used to correct for the (assumed)
IP header sizes.

See also the tt(conntrack-command) option.
    
    it() tt(indevice outdevice): the `devices'-mode. Here, tt(indevice) is the
name of the device behind the firewall: addresses living behind the
tt(indevice) are source-natted to the firewall host's IP address when passed
on to the tt(outdevice);

tt(outdevice) is the name of the device where source-natted packets are
forwarded to, and from where replies for source-natted hosts living behind the
tt(indevice) are received. With this command all, or any combination of the
(by default) tcp, udp, and icmp layer four protocols can be monitored.

    it() tt(infile in-address in-mask outfile out-address out-mask): the
`tcpdump'-mode. This command can be used to process bf(tcpdump)(1) generated
binary files, generated on the source-natting host. If a source natting host
uses interface tt(eth1) behind the firewall and tt(eth0) to connect to the
outside world, then the follow tt(tcpdump) commands produce the required
binary files (these commands will normally be run in the background, hence the
trailing tt(&)):
        verb(
    tcpdump -wi eth0 /tmp/eth0 &
    tcpdump -wi eth1 /tmp/eth1 &
        )
    To have tt(natlog) process these files, terminate the tt(tcpdump)
commands, and transfer the files tt(/tmp/eth0) and tt(/tmp/eth1) to the host
where  tt(natlog) has been installed. The required addresses and masks are
shown by the bf(ifconfig)(1) command. E.g.,
        verb(
eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
      inet 129.125.1.123  netmask 255.255.0.0  
                          broadcast 129.125.255.255

eth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.1.1  netmask 255.255.255.0  
                          broadcast 192.168.1255
        )
    The relevant info is shown in the lines following the interface's name:
the value following tt(inet) is the interface's IP address, and the value
following tt(netmask) is the network's mask. 

    Combining files and addresses, tt(natlog) is run as follows (all on one
line):
        verb(
    natlog  /tmp/eth0 129.125.1.123  255.255.0.0 
            /tmp/eth1 192.168.1.1 255.255.255.0
        )
    Instead of fully specifying the netmask, netmaks specifications like /24
are also accepted. In that case the number following the slash indicates the
number of non-zero bits of the netmask. In practice, each value of the netmask
is either 255 (8 bits are set) or 0 (0 bits are set), and so 255.255.0.0 can
also be specified as /16, while 255.255.255.0 can be specified like /24.
    )

manpagesection(OPTIONS)

    See also section tt(SYSTEMD).

    itemization(
    it() lsaoption(config)(c)(config-path)nl()
       The argument tt(config-path) defines the path to tt(natlog's)
        configuration file. By default it is tt(/etc/natlog.conf). All
        configuration options have defaults, which are used when no
        configuration file and no command-line options were provided.

       All options, except for tt(config, help, verbose) and tt(version) can
        also be specified in the configuration file. The configuration file
        ignores empty lines and all information on lines beginning with a
        hash-mark (tt(#)). In the configuration file initial hyphens should be
        omitted, and option names may immediately be followed by a
        colon. Multi-word arguments should not be surrounded by
        quotes. Examples:
       verb(
    stdout
    syslog-facility: LOCAL0
        )
       Command-line options override configuration file options.

    it() laoption(conntrack-command)(path [options])nl()
       The path and options to the bf(conntrack)(1) program. By default this
        is 
       verb(
    /usr/sbin/conntrack -p tcp -E -n -o timestamp -e NEW,DESTROY
        )
        resulting in:nl()

       - Monitoring the tcp layer four protocol;nl()
       - Displaying real-time event logs (tt(-E));nl()
       - Displaying time stamps (tt(-o timestamp));nl()
       - Logging all new and destroyed (ended) events (tt(-e
            NEW,DESTROY));nl()

       By default tt(tcp) is monitored. Other protocols can be configured
        using the tt(--protocol) option.

       The tt(conntrack) program must be available when requesting
        tt(natlog)'s tt(conntrack) command. Layer four protocols other than
        tcp, udp and icmp are currently not supported. A subset of the
        supported protocols may be requested using tt(conntrack's -p tcp, -p
        udp) or tt(-p icmp) options.

    it() laoption(conntrack-device)(dev)nl()
       By default tt(conntrack) monitors the information made available at the
        tt(/proc/net/nf_conntrack) device. When another device is used, it can
        be specified using this option.

    it() laoption(conntrack-ip-header-size)(size)nl()
       This option can be used to correct for the IP header sizes. By default,
        tt(conntrack) includes these sizes in reported byte counts. By
        specifying this option packet sizes reported by tt(conntrack) are
        reduced by tt(size). Commonly IP headers consist of 20 bytes (so, to
        correct for this specify tt(--conntrack-ip-header-size 20)).

    it() laoption(conntrack-restart)(max)nl()
       If the conntrack process prematurely ends it is restarted at most
        tt(max) times (these are pure em(restarts): conntrack's initial
        startup is not counted for this option). By default 10 restarts are
        allowed.

    it() loption(debug)nl()
       Write additional info to the log file. Suppressed when tt(--log off)
        has been specified. Currently, tt(--debug) writes information about
        memory consumption to the log file.

    it() loption(help) (soption(h))nl()
       Write basic usage information to the standard output stream and
        terminate.

    it() laoption(log)(argument)nl()
       By default tt(natlog) forwards log messages about tt(natlog) and
        connection information to the syslog daemon using the tt(DAEMON)
        facility with priority tt(NOTICE) (see below at the tt(syslog*)
        options). This is identical to specifying the argument
        tt(syslog). Alternatively, specify the argument tt(off) to suppress
        writing log messages. Any other argument is interpreted as a
        path-specification to a file to receive the log messages: log-messages
        are appended to existing files. If the log file does not yet exist it
        is first created.

    it() laoption(log-data)(path)nl()
       tt(Path) specifies the pathname of the file where information about
        observed connections is written in tabular form. Information is
        appended to existing files. If tt(path) does not yet exist it is first
        created. Refer to the tt(DESCRIPTION) section for information about
        the format of the generated table. Specify tt("") as command-line
        option if the configuration file specifies a log data file but you
        don't want the data to be logged in a particular run of tt(natlog).

    it() laoption(log-rotate)(spec)nl()
       This option is used to specify the frequency and the number of
        log-files that are rotated. By default log-files are not rotated.nl()
       Specify tt(time[mhd]) or tt(time[mhd]nFiles). A time unit specification
        must follow tt(time) and is tt(m) for minutes, tt(h) for hours, and
        tt(d) for days. tt(nFiles) specifies the max. number of rotated
        files. If only tt(time[mhd]) is specified, then tt(nFiles) is set to
        1.  By default (or if tt(time) or tt(nfiles) are specified as zero)
        log files are not rotated.

    it() loption(no-daemon)nl()
       By default, tt(natlog) runs in the background (a daemon). tt(Natlog)
        runs as an ordinary program (i.e., in the foreground when the option
        tt(no-daemon) is provided). When running as a daemon, tt(--stdout)
        (see below) is suppressed, and tt(--verbose) messages (see below) are
        sent to the syslog daemon, unless tt(--no-syslog) was specified. When
        using the tcpdump-mode tt(natlog) does not run in the background. In
        this case, if tt(no-daemon) is omitted a warning message is logged,
        and tt(natlog) continues as an ordinary program.

    it() lsaoption(pid-file)(p)(path)nl()
       When tt(natlog) runs in the background, then tt(path) is the name of
        the path of the file holding the daemon's process-id. By default this
        is tt(/run/natlog.pid). To end the daemon, send a tt(SIGINT) or
        tt(SIGTERM) signal to the process id mentioned in the
        tt(pid-file). tt(Natlog) ignores tt(SIGHUP) signals (but writes a log
        message when receiving a tt(SIGHUP) interrupt).
        
    it() lsaoption(protocol)(P)(specification)nl()
       The protocol(s) to monitor. By default the tcp layer four protocol is
        monitored. Currently tt(natlog's conntrack) command can monitor the
        tcp, udp, and icmp layer four protocols.  Using the tt(protocol)
        option (note: singular!)  any subset of these protocols can be
        selected by specifying a colon-separated subset of tcp, udp, and icmp
        (e.g., tt(--protocol udp:tcp)). The specification tt(all) can be used
        to monitor all three protocols (tcp, udp, and icmp). When using the
        conntrack-mode, only a single protocol (including tt(all)) can be
        specified.


    it() soption(S)nl()
       Use this option as first option, immediately following the program
        name, when starting tt(natlog) from a bf(systemd)(1)
        tt(natlog.service) file. See also section bf(SYSTEMD) below.

    it() loption(stdout) (soption(s))nl() 
       Syslog-equivalent messages are sent to the standard output.
        This option is implied by tt(--verbose), but is suppressed once
        tt(natlog) runs as a daemon.

    it() laoption(syslog-facility)(facility)nl()
       The facility that is used to write the syslog messages to. By default
        this is tt(DAEMON). For an overview of facilities and their meanings,
        see, e.g., bf(syslog)(3). With tt(natlog) the facilities tt(DAEMON,
        LOCAL0, LOCAL1, LOCAL2, LOCAL3, LOCAL4, LOCAL5, LOCAL6, LOCAL7), and
        tt(USER) can be used.

    it() laoption(syslog-priority)(priority)nl()
       The priority that is used to write the syslog messages to. By default
        this is tt(NOTICE). For an overview of priorities and their meanings,
        see, e.g., bf(syslog)(3). With tt(natlog) all defined priorities can
        be used. E.g., tt( EMERG, ALERT, CRIT, ERR, WARNING, NOTICE, INFO) and
        tt(DEBUG).

    it() laoption(syslog-tag)(tag)nl()
       When syslog messages are generated they can be provided with a
        em(tag), which can be used to filter tt(natlog)'s syslog messages from
        the log-files. By default the tag tt(NATLOG) is used. See also section
        tt(RSYSLOG FILTERING) below.

    it() loption(terminate)nl() 
       When tt(natlog) is running as a daemon, a separate command tt(natlog
        --terminate) can be issued to terminate the daemon. This assumes that
        the default pid-file (tt(/run/natlog.pid)) was used or that a
        non-default pid-file was specified in the default configuration
        file. If not, then the location of the pid-file or the location of the
        non-default configuration file must also be specified. When the daemon
        could be terminated 0 is returned. Otherwise, an error message is
        displayed and 1 is returned,

    it() lsaoption(time)(t)(spec)nl()
        By default time stamps written by tt(natlog) are in raw, numeric
        form. E.g.,
        verb(
NATLOG: From 1338990672:55588 thru 1338990747:807100
        )
       These time stamps indicate times in seconds:microseconds since the
        beginning of the epoch, January 1, 1970, 0:00 UTC. This option can be
        used to change the seconds part of the time stamps to more
        conventional representations.nl()
       Specify tt(raw) (the default) for the default representation in seconds
        since the epoch; nl()
       specify tt(utc) for a representation like tt(Jun 6 13:29:11), using
        Universal Time Coordinated;nl()
       specify tt(local) for a representation like tt(Jun 6 13:29:11), using
        the local time zone defined by the computer running tt(natlog).

    it() lsaoption(ttl)(T)(secs[ui])nl()
       time-to-live for received connections. At most two time-to-live
        specifications can be provided: for udp/icmp connections a letter
        tt(u) must be appended to the specified seconds. By default tt(60u) is
        used. For tcp connections a letter tt(t) must be appended to the
        specified seconds. By default tt(3000t) is used. Both time-to-live
        specifications may be combined: tt(--ttl 120u1800t) specifies a
        time-to-live of two minutes for udp/icmp connections and a
        time-to-live of half an hour for tcp connections. Time-to-live is not
        used in conntrack-mode.

    it() lsoption(verbose)(V)nl() 
       Additional messages about tt(natlog)'s mode of operation are sent to
        the standard output stream. When tt(natlog) runs as a daemon these
        messages are sent to the syslog daemon, unless tt(--no-syslog) was
        specified. 

        When tt(--verbose) is specified twice then all actual configuration
        parameters are shown just before tt(natlog) starts.

        When tt(--verbose) is specified more often then tt(natlog) ends after
        reporting the configuration parameters.

    it() loption(version) (soption(v))nl() 
       Write tt(natlog)'s version number to the standard output stream and
        terminate.

COMMENT(
    it() loption(warn) (soption(w))nl() 
       Warn about terminating connections not yet registered in tt(natlog)'s
        database. This normally only happens during a short period after
        starting tt(natlog), when existing connections haven't yet been
        noticed. 
END)
    )
                    
manpagesection(SYSTEMD)

    An annoying characteristic of bf(systemd)(1) is that environment variables
containing blanks are passed as single arguments to the program being called
by their tt(.service) files. As a consequence, it is very hard to provide an
environment variable in, e.g., tt(/etc/default/natlog) specifying tt(natlog's)
arguments: in practice the number of arguments varies, and so even 
constructions like tt(ARG1=value1, ARG2=value2), etc. are awkward at best.

    As a stopgap for this unwelcome characteristic of tt(systemd) the option
tt(-S) is provided. When used it em(must) be specified as tt(natlog's) first
argument. tt(Natlog) will then inspect all remaining arguments, splitting
arguments containing blanks into separate arguments, which are then processed
by tt(natlog) as intended. Be aware that, to limit the complexity of the
splitting-procedure, it is not full-proof: double- or single-quote delimited 
string-arguments will also be split into separate arguments. Unless filenames
themselves containing blanks are passed as arguments to tt(natlog) this
limitation is probably not very serious. 

As an example, here is an example of tt(systemd's) tt(ExecStart)
specification:
        verb(
    ExecStart=/usr/bin/natlog -S -p ${PIDFILE} ${DAEMON_ARGS}
        )
    where tt(DAEMON_ARGS) might have been specified in tt(/etc/default/natlog)
as
        verb(
    DAEMON_ARGS=--log /tmp/natlog.log --log-data /dev/null conntrack
        )

manpagesection(RSYSLOG FILTERING)

    When using bf(rsyslogd)(1) property based filters may be used to filter
syslog messages and write them to a file of your choice. E.g., to filter
messages starting with the syslog message tag (e.g., tt(NATLOG)) use
        verb(
:syslogtag, isequal, "NATLOG:"   /var/log/natlog.log
:syslogtag, isequal, "NATLOG:"   stop
        )
    Note that the colon is part of the tag, but is not specified with the 
tt(syslog-tag) option.

    This causes all messages having the tt(NATLOG:) tag to be written on
tt(/var/log/natlog.log) after which they are discarded. More extensive
filtering is also supported, see, e.g.,
tt(http://www.rsyslog.com/doc/rsyslog_conf_filter.html) and
tt(http://www.rsyslog.com/doc/property_replacer.html)

manpagesection(EXAMPLES)

    Examples of tt(natlog) activations:
    itemization(
    it() tt(natlog --no-daemon --no-syslog -s br0 eth0)nl()
        tt(Natlog) remains active as a foreground process, no syslog messages
are written, syslog-equivalent message are written to the standard
output. tt(Natlog) uses the pcap library to capture packets from the tt(br0)
device, which is active behind the firewall, and to capture packets from the
tt(eth0) device, which is the device to where source-natted packages are sent.

    it() tt(natlog conntrack)nl()
        Depending on the options specified in tt(/etc/natlog.conf) (or, if not
available, tt(natlog)'s default options) source-natted connections are
obtained from bf(conntrack)(1). By default tt(natlog) continues as a daemon
process, generating syslog messages using syslog tags tt(NATLOG:), and
containing information about source-natted connections.
    )

    Here is tt(natlog)'s default configuration file. Empty lines and lines
starting with hash-marks (#) are ignored. Options adhere to the following
syntax: 
    verb(
option  value 
    )
    Option and value are separated by white space, a colon may be appended to
option names:

    verbinclude(../../etc/natlog.conf)
        
manpagesection(FILES)
    itemization(
    it() tt(/etc/natlog.conf): default configuration file location;
    it() tt(/etc/default/natlog): arguments for startup scripts;
    it() tt(/etc/init.d/natlog): SysV startup script;
    it() tt(/etc/systemd/system/natlog.service): systemd startup script
        (calling tt(/etc/init.d/natlog)).
    )

manpagesection(SEE ALSO)

    bf(conntrack)(1), bf(ifconfig)(1), bf(iptables)(1), bf(pcap-filter)(7), 
    bf(ping)(1), bf(R)(1), bf(rsyslogd)(1), bf(syslog)(3), bf(systemd)(1),
    bf(tcpdump)(1)

manpagesection(BUGS)

    tt(Natlog) currently may process tcp, udp and icmp layer four protocols.

manpageauthor()

Frank B. Brokken (f.b.brokken@rug.nl).

