\title Apt-Cacher-NG User Manual

\cfg{xhtml-leaf-level}{1}
\cfg{xhtml-leaf-smallest-contents}{2}
\cfg{xhtml-leaf-contains-contents}{true}
\cfg{xhtml-address-start}{Comments to <a href='mailto:blade@debian.org'>blade@debian.org</a>}

\cfg{xhtml-contents-filename}{index.html}
\cfg{xhtml-template-filename}{%k.html}
\cfg{xhtml-template-fragment}{%k}

\cfg{man-identity}{apt-cacher-ng}{8}{November 2007}{apt-cacher-ng}{Eduard Bloch}
\cfg{man-filename}{apt-cacher-ng.8}

Apt-Cacher NG is a caching proxy for software packages which are downloaded by
Unix/Linux system distribution mechanisms from mirror servers accessible via
HTTP.

This manual provides an overview of Apt-Cacher-NG's features and a walk through
the required configuration steps for server administrators and users of the
proxy.

\C{Intro} Introduction

apt-cacher-ng attempts to achieve the same goals as related proxies
- it acts as a proxy which is used by clients in the local network
to share the data that has been downloaded. It monitors the state of
packages and is capable of merging downloads of the same packages
from different locations (real or simulated).

The program reuses many ideas behind the other famous proxy, its
predecessor apt-cacher 1.x (which has been written in Perl). In
contrast to apt-cacher, different aspects have been declared as primary
targets during the development of apt-cacher-ng:

\b lightweight implementation - allow use on systems
   with low memory and processing resources

\b internal (native) threading - avoiding process fork'ing
   wherever possible, avoiding kludges for pseudo-thread
   synchronization, avoiding relying on special file system features for
   internal operations where possible

\b real (effective) support of HTTP pipelining, using an internal client with
   native stream control (having the nice side effect: reduction of resource
   overhead and minimization of possible points of failure)

\b avoiding featuritis where they cause too much bloat and the
   functionality can be provided by native OS features

\b reliable but efficient content merging in the local package
   pool, avoiding delivering of wrong data.

As with apt-cacher, explicit tracking of dynamically changed and
unchanged files is established, and the use in non-Debian
environment is supported.

Long story:
Not all goals have been achieved. The initial plan of using
background databases to merge any download from any arbitrary
location has been dropped because of complexity and performance
considerations, reliable heuristics could not be found either.
Instead, a semi-automated solution has been created which used
machine-parsable files with mirror information, like the one
available for Debian mirrors in \W{http://anonscm.debian.org/viewvc/webwml/webwml/english/mirror/Mirrors.masterlist?view=co}{Debian's CVS repository}.

\C{usage} Running apt-cacher-ng

Run "build/apt-cacher-ng -c conf" when configured where conf is the
configuration directory. See \k{cfgfiles} for details on possible and required
contents of this directory.

Most options from the configuration file can also be passed through command
line parameters. Just append them with the same format as in the configuration
file but without separating spaces inside, e.g.

\c Port:4855 ForeGround=1

For convenience, the colon can also be replaced with the equals sign and letter case does not matter, so this is also possible:

\c port=4855 foreground:1.

\C{config-servquick} Basic Configuration

\H{server-config} Server Configuration

Unlike some rumors on the internet claim, there should be no need for
exhausting configuration work to just test apt-cacher-ng and run it with default
parameters. It's actually designed to bootstrap most of its working environment
without additional help.

The package setup scripts used by distributions should already prepare working
initial settings for apt-cacher-ng. Check the file
\c{/etc/apt-cacher-ng/acng.conf}
file where most settings are explained. For the beginning they should not be
changed, the only interesting setting present there is the TCP port. See
Advanced Server Configuration for details.

There is also a daily cron job which executes some maintenance work. Additional
automated control commands can be added by administrator.

\H{config-client} Client Configuration

From the client side, apt-cacher-ng can be used as a drop-in
replacement for apt-cacher. The same rules apply, e.g. Debian/Ubuntu 
users should EITHER:

\b   Specify the caching machine as HTTP Proxy for your download client. This
can be usually done (temporarily) with the environment variable \c{http_proxy}
or (permanently) in some configuration file or dialog. For APT, this means
putting a line like the following into a file like /etc/apt/apt.conf.d/02proxy:

\lcont{
\c Acquire::http::proxy "http://CacheServerIp:3142";
}

OR:

\b  Replace all mirror hostnames with cachinghost/hostname in
   sources.list, so 

\lcont{
\c deb http://ftp.uni-kl.de/debian etch main
   
   should now become:

\c deb http://192.168.0.17:3142/ftp.uni-kl.de/debian etch main
}

\e{(assuming that CacheServerIp is 192.168.0.17 and the service port is 3142)}.

These both methods have their own pros and cons. The last method can be used with
clients which cannot configure an additional http proxy for some reason. The
disadvantages are the worse progress visibility (all downloads seem to come
from the same machine) and some resource usage limits might be hit (i.e.
maximum number of simultaneous downloads from the "same" machine). It might
also require to modify many different URLs and some of those URLs might be
hardcoded in software and not be accessible to the user.

The former method is more convenient since it usually means less configuration
work; however, it implies that all relevant client programs respect the proxy
setting and use it from this central location.

Mixing the configuration methods is usually possible but not within the same
client program instance. Doing that (going with proxy mode AND use rewritten
URLs) will probably confuse the server: in best-case, the connection will
be slower because of a little transport loop on the server side, and in the
worst-case, the target URL will eventually become not resolvable and the
downloads will just fail.

Using SSL/TLS transport (i.e. https urls) is also possible with some
restrictions, see \k{ssluse} for details.

\C{config-serv} Advanced Server Configuration

\H{Vocabulary}Vocabulary

This chapter introduces some terminology which is needed to understand the
functionality of apt-cacher-ng; it's recommended to understand it before
continuing with the advanced configuration.

\b "Backend":
a text file consisting of a list of mirror URLs, one per line (a
more complex RFC822-like format is also supported). Used for URL remapping;
see \k{repmap}.

\b "Volatile files":
files with changing contents on the server while keeping the same URL (i.e. the
same filename and location). For example, metadata pertaining to package files
stored in a remote archive is classified as 'volatile'. They are usually 'index
files' known as Packages, Sources, Release, Pdiff and the like, and maybe also
associated extra data (like translations and thumbnail icons).

\b "Package files":
files that contain software packages and other "solid" data: DEBs, source files
for their creation (.tar.gz, .diff, .dsc), various metadata which is not
subject to change after first appearance on the server.

\b "Configuration line":
one single line in the configuration file. Some examples in this chapter may
be displayed as wrapped lines but should be stored as a single line in the configuration.
Some variables (like \c{RequestAppendix}) support multiline data which needs to be separated with \\n delimiter.

\H{cfgfiles} Configuration file types

By default, the /etc/apt-cacher-ng directory (or the one specified with program
options) contains all config files, HTML page templates, the stylesheet and
other text-based support files used by apt-cacher-ng. The contents may vary
depending on the installation of apt-cacher-ng, refer to the package
documentation for Linux Distribution packages.

There are a few certain file types distinguished by apt-cacher-ng:

\n Main configuration files:

\lcont{
*.conf files 
are assumed to contain configuration directives in the form of
"key: value" pairs. The package comes with a commented example configuration
file. apt-cacher-ng reads all files matching *.conf in alphabetical order
and merges the contents. For options documentation, see commented example file
shipped with apt-cacher-ng (conf/ directory in original source).

For security reasons, the files can be made readable only to the daemon and
administorator accounts, e.g. when they contain passwords or other sensitive
data.
}

\n URL lists and remote repository list files. The file names are arbitrary, no
special suffix is required. They are read and included during processing of
configuration files and can contain data in one of the following formats:

\lcont{

\b simple text files with one URL per line (the URL should point to the base
  directory of the repository, e.g. "http://ftp.de.debian.org/debian/"). A URL
  must start with http:// and should end with a slash

\b an RFC822-like format, with lines like 'Site: <hostname>' and
  'Archive-http: /base/directory/of/repository/'. Optional fields are also used
  in this remapping descriptions to add more possible variants (Alias, Aliases,
  X-Archive-http:) of the URLs to the lookup list

}

\n Various support files used for the configuration web interface, named like *.css and *.html.

\n *.default files are used in some rare cases as replacement for list files having the same name without .default suffix.

\n *.hooks files specify custom actions which can be executed upon connection/disconnection (see \k{remap-trickz} for details).

Except from .conf files, most files listed above can be moved to another
"support" directory and the daemon will look for them therein if they are not
present in the primary configuration directory. This feature is intended to
help keeping pure configuration data and mostly static data in different
locations. The directory path is specified at build time and can be overriden
with the \e{SupportDir} directive (and if used, this should be set as early as
possible).

\H{repmap} Repositories and URL mapping

With the most simple configuration, apt-cacher-ng will act almost like an
ordinary HTTP proxy with improved caching behaviour. When files are requested,
they are downloaded from a remote location specified in client's request and
are stored in a unique way.

However, for some use cases it can be beneficial to specify additional rules to
achieve further improvements, e.g. in order to detect and prevent avoidable
downloads, to reduce space requirements for the cache directory or simply hide
real download locations from the APT clients.

These modifications are generally achieved by two strategies, \c{Merging} and
\c{Redirection}, which are configured in a context of a specified cache
\c{Repository}. The configuration for them is created using one or multiple
Remap-... configuration directives (see below).

\e{Merging:}

"Merging" of incoming requests can be done if some subdirectories 
of different remote servers are considered equal where the last part
of the remote file path leads to the same file content. When
specified, the internal cache content is shared and the live download
stream is shared. The configuration work consists of setting an
"equality list" containing a set of URLs representing the base
directories (like \c{http://ftp.debian.org/debian}
and \c{http://ftp.uni-kl.de/pub/linux/debian}).

\e{Redirection:}

With redirection, client requests cause a download from a remote location which
is different from what clients requested and believe to receive from.
Redirection is an optional feature; if used, it's configured by one or multiple
URL(s) pointing to target servers. The URL(s) must include a directory spec
which matches the directory level of the URLs in the \c{Merging} URL(s), for
example all ending with /ubuntu/ for usual Ubuntu mirror URLs. If redirection
is not used (i.e. the target URL list is empty) the original URL from client's
request is used to get the data.

\e{Repository:}

A (cache) repository is the internal identifier which declares the scope in
which \c{Merging/Redirection} specs are applied. It also represents the name of an
internal cache subdirectory.

\S{basic-map}Writing Remap-... configuration


When use cases for merging/redirection are identified and a repository name is
chosen, these components are written into configuration directives starting
with Remap- which follow the simple syntax:

Remap-\c{RepositoryName}: \c{MergingURLs} ; \c{TargetURLs} ; \c{OptionalFlags}

The repository name is a symbolic name which should be chosen carefully and
should not be changed afterwards, otherwise the data might become inaccessible
for clients until the files are extracted and reimported semi-manually.
Internally, this string shares the namespace with host names and/or top
directory names of other URLs. Name collisions can cause nasty side effects and
should be avoided. Recommended names are made up from alphanumeric or URL-friendly
characters. Also, a repository name should not be associated to a real hostname. Examples for
good names: \c{archlinux-repo}, \c{debianlocal}. Examples for bad names:
\c{fedora.example.com}, \c{_very&weird}.

The \c{TargetURLs} part is optional (see \c{Redirection} description above). If
multiple targets are specified, the order of servers here defines their order of preference (see also the \c{NetworkTimeout} option and additional
notes below).

Both URL lists simply contain URLs separated by spaces. The strings must be
properly URL-encoded. Since all URLs are assumed to belong to http:// protocol
and point to a remote directory, the \c{http://} protocol prefix and trailing
slashes are optional. There is no hard limit to the number of URLs. However,
for readability reasons it's recommended to put them into separate list files
(see \k{cfgfiles}) and specify the particular list files with tags like
\c{file:urlsDebian.list} instead of writing them into a single line. Raw URLs
and \c{file:...} lists can be mixed.

Fully configured Remap lines can look like:

\e{Example I:}

\cw{Remap-debrep: ftp.de.debian.org/debian http://ftp.at.debian.org/debian}

for the use case: small home network, clients have de... or at... servers in
their sources.list files and use acng as HTTP proxy. Now the files are still
downloaded from at... or de... mirrors depending on the user request, but
already cached data is served to both, at... and de... users.

\e{Example II:}

\cw{Remap-ubuntu: file:ubumir.lst ; 192.168.17.23/pu ca.archive.ubuntu.com/ubuntu}

for the use case: small home network, clients have various Ubuntu mirrors
(which are listed in ubumir.lst) in their sources.list files and use acng as
HTTP proxy. All requests are redirected to a mirror in the /pu directory of
some local machine. When that machine is down, Canadian public server is used
instead.

\S{remap-trickz}Special tricks and additional notes

There are some implementation details (partially explained above) and some
configuration options related to repository settings which should be mentioned
explicitly.

The internal cache directory tree follows the URL requests from the clients
unless modified by Remapping rules. For proxy-style configuration on the user
side, it is always the hostname of the requested URL. But if clients access the
apt-cacher-ng server like a regular mirror (not using APT's proxy config) then
it's just passed as regular directory name. And at this point, it's possible to
use Remapping constructs to access random remote locations while the client
assumes to download from a subdirectory of apt-cacher-ng (as http server). This
is configured by simply using /some/directory/string/ instead of URLs in the
\c{Merging} list to let your clients download from
http://acngserver/some/directory/string/... paths.

If multiple Remap- lines for the same \c{Repository} are specified, the
contents of both URL lists are merged.

On some restricted networks, it may be needed to enforce the use of predefined
mirrors. If the \c{ForceManaged} option is set, only requests to URL matched in
some Remap-... config is allowed.

Sometimes, it may be needed to execute a system command before connection to
certain machines is established. This is possible by associating commands with a
repository declaration, i.e. by storing a file named like
\c{repositoryname}.hooks in the main configuration directory.  It can contain
PreUp, Down and DownTimeout settings. PreUp/Down are executed by the system
shell and it's up to the administrator to make sure that no malicious code is
contained there and that the execution of these commands does not cause
significant delays for other apt-cacher-ng users. See package documentation for
an exemplary hooks file.

If the Redirection part contains multiple URLs, the server prefers to use them
in the order of appearance. On success, the first target is used all the time,
and so this should be the preferred mirror (note: "success" means getting a
started download or a non-critical failure in this context. A "404 File not
found" status is not considered critical since client's apt can expect and use
it to check the existence of remote files and then change its own behaviour
accordingly).

And finally, there is an optional third field in the \c{Remap}
directives which can contain extra flags to modify downloading behavior in the
scope of that particular cache repository.

\b \c{keyfile=...} The meaning of this setting is: if any real download
error (status code 400 and higher) happens on a file which path ends with the
specified string then the target server is blacklisted (considered faulty)
immediately and this download (and subsequent ones requested by this client
connection) are retried from other servers (see \c{TargetURLs} description
above). Can be used multiple times to define a list. See below for documented
example.

\b \c{deltasrc=URL} Configures the base URL used to download .debdelta files.
The path hierarchy below this URL should correspond to the source URLs and file
paths in the cache. Only one URL can be specified at the moment. It is used for
explicit mirroring operations, see \k{mirroring} for details.

\b \c{proxy=proxyspec} Configures an alternative proxy URL which overrides the
global proxy setting in the context of this repository. Can be set empty to
disable proxy usage.

Config example:

\c Remap-debrep: file:deb_mirror*.gz ; file:backends_debian ;
\c    keyfile=Release keyfile=.deb

If the first mirror from backends_debian goes wild and returns 404 responses
for everything then the next candidate will be used. However, while this
feature can improve redundancy for certain installations it needs to be used
with care! Some file types are allowed to be missing and apt interprets their
absence to change its behavior as needed. keyfile= should only match files
which have an essential role and which disappearance is undoubtful indication
of a broken server.

\C{secure} Security

Like many data storing daemons with predictable filenames, apt-cacher-ng is
vulnerable to symlink attacks and similar malicious actions. Therefore, the
user must make sure that the cache and log directories are writable only to the
user account under which apt-cacher-ng is running.

General network access control can be established with operating system's
facilities in kernel space (see below) or in user space (e.g. with inetd, see
\k{acl-inetd}).  There is also experimental built-in access filter feature
which uses the TCP Wrapper library for filtering (i.e. uses the same mechanism
as tcpd). See \k{acl-libwrap} for details.

As to the program's internal security, apt-cacher-ng has been developed with
concern about a certain grade of attacks from internal users as well as from
malicious external hosts. However, no guarantees can be made about the security
of the program. It is recommended to run apt-cacher-ng under a dedicated system
account which has no write access to any files outside of the cache and log
directories. Refer to the manuals of the administration utilities of the system
distribution (like start-stop-daemon) to create the required configuration.

If relaxed permissions are required, e.g. to make files group-writeable, this
can be established through the appropriate use of umask command in the startup scripts
of apt-cacher-ng (see \c{/etc/default/apt-cacher-ng}, for example) and the sticky
bit on the cache directories (see \c{chmod(1)} manpage for details). However,
write permissions should be assigned very carefully because that may make the
server vulnerable to symlink attacks and like-minded activities.

The management web interface which allows execution of certain operations can
be protected by HTTP credentials (username/password). The trust levels for most
actions depend on their purpose. Those which should never be able to destroy
important local files can be triggered by anyone if no password is set.  And on
the other hand, some operations are considered dangerous and can only be
accessed when the admin password is configured and was entered correctly.

\H{acl-bind-iface} Access control by IP interface

A simple control method for incoming connections is listening only to
network interfaces which are inside a secure perimeter, e.g. belong to the
local network. See \k{howto-interfaces} for details on this configuration
parameter. The default setting is listening to all interfaces.

\H{acl-libwrap} Access control with libwrap

If the access control with operating system's ip filters is not possible, the
embedded filtering might be using instead. It is turned on if the feature is
included at build time and the \c{UseWrap} option is set to non-zero. If
\c{UseWrap} is not set at all, it might be enabled implicitly if the
hosts.allow or hosts.deny files have rules for apt-cacher-ng.

For more information about host access list creation, refer to hosts_access(5)
manual page (or manual pages for hosts.allow and hosts.deny).

The libwrap filtering method has a little drawback compared to alternatives.
The host checks are called in single-threaded context and can add some lag for
other incoming connections.

\H{acl-inetd} Access control with inetd

In some situations, access filtering by client IP/hostname might be not
supported directly or there are other reasons to use inetd to wrap access to
apt-cacher inetd. For this case, an inetd daemon is shipped with the package
which makes the use of tcpd possible. Installation is done in following steps:

1. compile the inetd bridge tool "in.acng", if not already done (check 
\c{/usr/lib/apt-cacher-ng}).

2. Edit apt-cacher-ng's configuration (acng.conf, for example), and set a path
for a new file in a writable directory, like this:

\c{SocketPath:/var/run/apt-cacher-ng/socket}

3. Edit /etc/inetd.conf and add following line with appropriate path names and
TCP port:

\c 3143  stream  tcp nowait  user /usr/sbin/tcpd 
\c         /usr/local/sbin/in.acng /var/run/apt-cacher-ng/socket

4. Edit hosts.allow and other files to configure ACLs for port 3143. See
tcpd(8) and related manpages for further details.

5. Configure clients to use the alternative port (3143 in the example above).

\H{acl-iptables} Access control with iptables

Looking at performance, the most efficient way to estables access control by
source IP is the use of system's native mechanisms. For the Linux kernel, it
can be created with iptables rules like in the following example, usually
stored in system's firewall setup script (or any init script like
\c{/etc/rc.local}):

\c iptables -A INPUT -p tcp --dport 3142 --source 127.0.0.0/8 -j ACCEPT
\c iptables -A INPUT -p tcp --dport 3142 --source 192.168.0.0.0/16 -j ACCEPT
\c iptables -A INPUT -p tcp --dport 3142  -j REJECT --reject-with tcp-reset

\H{acl-tgtports} Target port filter

In the default configuration, \e{apt-cacher-ng} restricts the choice of the
target port on the remote mirror to the default HTTP port (80). This is
intended to prevent creation of HTTP tunnels and similar malicious user
activities.

Cache administrator can define an own list of allowed ports by setting them as
(space separated) list with the AllowUserPorts configuration directive.

Alternatively, access to custom TCP ports can be established through custom
remapping specification.

\C{distinstructions} Distribution specific instructions

\H{hin-debuntu} Debian and Ubuntu

Use as is. Report bugs using reportbug (Debian) or to Launchpad (Ubuntu).

\H{hint-suse}OpenSUSE

Server can be used as is with limited expiration (see below, and see
\c{INSTALL} file for compilation hints). The merging mode (multiple servers
mapped into the same repository) is not preconfigured in the example
configuration. This is object to research, competent support is required.

Clients can configure apt-cacher-ng as central proxy in Yast ("Network
devices"/"Proxy" tab). If this is not desirable then each sofware source can be
edited to be redirected through the server. This can be done in the Software
Installer view of Yast2, see menu Configuration/Repositories. To edit them
quickly, switch to URL editing mode and insert \c{host:port/} (of the
apt-cacher-ng server) between \c{http://} and the source server name.

\H{hints-fccore}Fedora Core

Attempts to add apt-cacher-ng support ended up in pain and the author lost any
motivation in further research on this subject.

\H{hint-alx}Arch Linux

Mostly usable. The mirror structure design has been identified by
trial-and-error and the regular expressions might need some tuning by experts.

The installer seems to have no way to specify a dedicated proxy but it's
possible to edit the source URL and insert the \c{host:port/} part into it. The
pacman mirror list can be modified the same way.

Expiration code should work in the basic mode (index data is extracted from
*.db.tar.gz files). File checksum checking mode might also work (untested). The
example configuration contains a preconfigured list of mirrors which can be
rebuilt with the Makefile if needed. The preferred backend server can be
specified like with other distributions (see above for details).

\H{hints-sfnet}Sourceforge mirror network

Not a Linux distro but commonly used by those to download certain files.
Therefore most mirrors can get unified access cache sharing the
files in the same cache repository. Some .exe files there are never expired.

\H{hint-cygwin}Cygwin mirrors

While not being a pure Linux distro it's mostly GNU and has a nice mirror
setup. Proxy server can be used as is with limited expiration (see below).
Apt-cacher-ng can also be compiled and used on Windows machines in Cygwin
environment (see \c{INSTALL} file for details).

Clients need to specify the server as HTTP proxy in the setup.exe wizard, only
HTTP mirrors should be selected in the mirror list.

\H{hints-limexpire}Limited expiration

The expiration code for Non-Debian/Ubuntu repositories is quite limited due to
lack of man-power or know-how. For some distros, the support is limited to
checks of the filename and no further validation is supported. Therefore, the
extra validation of path location or file contents should NOT be turned on when
running expiration with data from that distros in the cache, because good data
may be deleted in this case.

\C{maint} Maintenance

There are few optional tasks that need to be executed by the administrator from
time to time or during the initial configuration.

\H{chapter-cleanup} Cache cleanup

If a package is no longer downloadable by APT clients then its files are also
not referenced in any index file and can be removed. This rule also applies to
most volatile files from the distribution metadata. For example, Debian's
Release file references some Packages and Sources files or Diff-Index file, and
those do reference most other non-volatile files (binary packages, source
packages, index diffs, ...).

\S{cleanup-manual} Manual expiration

To run the cleanup action manually visit the report page in a browser and
trigger the \c{Expiration} operation there.

There are different flags configuring the parameters of this tracking described
below. Usually just the filename is sufficient to consider a file in the cache
as a valid (downloadable) file. This is ok in most cases but sometimes leads to
false positives, i.e. when another repository in the cache refers to a file with
the same name but the reference to the original location is gone. On the other
hand there can be cases where the assignment to different repositories happened
by mistake and administrator would like to merge repositories later on.

For most files the checksum values are also provided in the index files and so
the file contents can be validated as well. This requires reading of the whole
cache archive to generate local checksums. It should also not be done when
apt-cacher-ng is being used (file locking is not used here).

Usually it's necessary to bring various index files
(Release,Sources,Packages,Index) in sync with the repository. This is necessary
because apt works around the whole file download by fetching small patches for
the original file, and this mode of operation is not supported yet by
apt-cacher-ng (and might still be unreliable).  When this synchronization fails,
the index files might be incomplete or obsolete or damaged, and they might no
longer contain references to some files in the cache. Abortion of the cleanup
process is advisable in this case.

There is also a precaution mechanism designed to prevent the destruction of
cache contents when some volatile index files have been lost temporarily. The
results of cache examination are stored in a list with the date when the
particular files became orphaned. The removals are only executed after few days
(configurable, see configuration file) unless they are removed from this list in
the meantime.

Parameters of \c{Expiration}:

\dt \e{Skip header checks}

\dd
By default, header description file of every package is opened and checked for
bad data and for obvious inconsistencies (like local file being larger than
specified by server). Which means opening reading a few kilobytes from disk for
almost every file in the cache, and slightly degrades performance of the
process. This option skips that basic checks.

\dt \e{Stop cleanup on errors during index update step}

\dd
Index files update is done first, on errors the expiration will be interrupted.

\dt \e{Validate by file name AND file directory}

\dd 
This option can be used to remove distribution stages. Example: to remove
"oldstable" one just needs to delete the "Release" files in the cache and run
\e{Expiration} with this option two times. There are some issues with this mode
operation, see above for details.

\dt \e{Validate by file name AND file contents (through checksum)}

\dd
Checking file size and contents where possible against the metadata in the
index files. Note: future calls of Expiration process without this option will
discard the results of this check and forget about corrupted files.
Therefore, an action on this files needs to be done ASAP, like
truncating them (see below) or removing via the removal operation (using the
checkbox and the Delete button, see process output) or via the "Delete all
unreferenced files" operation on the main control page.

\dt \e{Force the download of index files}

\dd 
Sometimes it may be needed to redownload all index files, explicitly replacing
the cached versions. This flag enables this behaviour.

\dt \e{Purge unreferenced files after scan}

\dd
Avoid the use of the orphan list and delete files instead. This option is
dangerous and should not be used unless when absolutely no mistakes/problems
can happen. Instead, it's possible to view the orphan list later and delete
then (see control web interface).

\dt \e{Truncate damaged files immediately}

\dd
If a file has been identified as damaged, it will be truncated (file size reset
to 0). Setting this option is a good compromise for debugging purposes compared
to the simple deletion since it will keep the header files on the disk, for
further analysis of the problem's cause.

\dt \e{More verbosity}

\dd
Shows more information, e.g. each scanned file when used with some of the other
options. This might result in a very large HTML page, making the watching HTML
browser very slow.

In additional to the default scan run, there are some "Direct Action" buttons
in the Web frontend. It's possible to see the temporary list of files that have
been identified as orphaned (unreferenced), and it's possible to delete all
files from that list immediately. To be used carefully!

\S{auto-cleanup} Automated cache cleanup

A program called \c{acngtool}
is shipped with the package. Among other functionality, this tool effectively
implements a HTTP client
which operates like a human who runs the expiration manually (see
above). If needed, it would read the admin credentials and unix socket file path
from the local configuration files. On Debian installations it is called by the file 
\c{/etc/cron.daily/apt-cacher-ng} so it should run automatically as a daily
\c{cron}
task. The results are usually not reported unless an error occurs, in which
case some hints are written to the standard error output (i.e. sent in cron
mails). 

The operator script can take some options from the environment, see below.  The
default operation mode is calling the expiration operation with default
parameters and with credentials from local system's apt-cacher-ng installation.
However, this can be changed with \c{ACNGREQ} variable.

\dt \c{ACNGIP=10.0.1.3}

\dd
The network address for remote connection may be guessed incorrectly by the
operator script. This variable can specify an explicit target to connect to,
e.g. the same IP as the one used by the clients (unless this network connection
is somehow restricted in the local setup).

\dt \c{HOSTNAME=localOrPublicName}

\dd
When an error occurs, the operator script most likely adds an URL to be opened
for further investigation. The host name of in this URL can be customized, i.e.
can be set to a public domain name representing the server as accessible from
the administrator's machine.

\dt \c{ACNGREQ=cgiparameters}

\dd
Override the auto-detected command parameters with a custom query string. This
is the part of a command URL from the management interface after the ? sign.

\S{extrakeep} Keeping latest versions of expired package files

Sometimes it makes sense to keep a couple of versions of (Debian) packages even
after they have been removed from remote source. It is possible to set an
exceptional rule for package files which follow the naming and versioning
scheme of .deb-packages. This extra handling is configured by the
\c{KeepExtraVersions} options which tells how many of the top-latest versions
shall be kept. The cache system needs the dpkg program and sufficient CPU power
(depending on the option value).

\H{distkill}Removal of distribution releases

Sometimes it's needed to remove all files from a distribution, i.e. when a new
release became Stable and older package files are still lying around. In
perfect conditions the reference tracking described above should take care of
it and remove them soon.

However, this solution will fail if the release files are still available on
the server AND apt-cacher-ng learned their real location (i.e. the code name
instead of not the release state name) and so they are refreshed during regular
expiration. 

After all, if the old release is no longer used by local cache users then the
extra disk usage becomes a problem.  This problem will go away after many
months when the old release files are finally deleted on the servers, then the
package expiration will start complaining for some days (the expiration delay)
and only then the finally unreferenced files will be removed.

To speed up this process, the local administrator can remove the traces of the
old distribution release from the archive. Either the top-level "Release"
files, or even the whole index file trees relevant for certain releases.

To make this task easier, a "brutal" script called distkill.pl is shipped with
apt-cacher-ng. It runs interactively, it scans the package directory and
presents an overview of index file trees assumed to represent distro releases.
Then it provides a command promt to remove some immediately. The script should
be used with extreme care! See
\k{cache-overview} for example of its output.

\C{howtos} HOWTOs and FAQ

\H{imp} Package import

Already existing packages can be imported into apt-cacher-ng's cache pool
instead of downloading them. There are some restrictions:

\n Don't try to import incomplete files. They will be refused since their
  contents cannot be checked against the archive metadata.

\n If possible, don't import symbolic links. Even if doing so, they should not
point to other files inside of the cache and especially not to other files
under the \c{_import} directory.

HOWTO:

\n{howto-update-ifiles} Make sure that apt-cacher-ng has valid index files in the cache. 
This is the tricky part. To get them right, a client needs to download them
through apt-cacher-ng once. Therefore:

\lcont{ 
\n Configure the server and one client before doing the import.
See above for instructions.

\n Run "apt-get update" on client(s) once to teach ACNG about remote locations of
(volatile) index files. In some cases this is not sufficient. See the
note on APT below for a workaround.
}


\n Store copies of your .debs, .orig.tar.gz, ... somewhere in the \c{_import}
subdirectory in the cache, ie. in \c{/var/cache/apt-cacher-ng/_import/}. The files may
be links or symlinks, it does not matter. When done, apt-cacher will move those
files to its own internal locations. Example:

\lcont{
\c cd /var/cache
\c mkdir apt-cacher-ng/_import
\c cp -laf apt-proxy apt-cacher /var/cache/apt-cacher-ng/_import
\c chown -R apt-cacher-ng:apt-cacher-ng apt-cacher-ng/_import
}

\n Visit the report page and trigger the import action there.
   Check the results, look for (red) error messages.
   
\n Check the \c{_import} directory again. All files that could be identified as
referenced by archive metadata should no longer be there if they have been
successfully moved. If some files have been left behind, check whether the
client can use them, i.e. with "apt-cache policy ..." and/or checking checksums
with md5sum/sha1sum tools. Probably they are no longer needed by anyone and therefore apt-cacher-ng just left them behind. If no, follow the instructions in \k{howto-update-ifiles} or do similar things for your distribution and retry the import operation. Setting the verbosity flag (see checkbox on the command-and-control page) can also help to discover the reason for the refusal to import the particular files.

NOTE: APT is pretty efficient on avoiding unneccessary downloads which can
make a proxy blind to some relevant files.
ACNG makes some attempts to guess the remote locations of missed (not
downloaded) files but these heuristics may fail, especially on non-Debian
systems.
When some files are permanently ignored, check the process output for
messages about the update of Packages/Sources files. When some relevant
package sources are missing there, there is a brute-force method for
Debian/Ubuntu users to force their download to the client side. To do that,
run:

\c rm /var/cache/apt/*cache.bin
\c rm /var/lib/apt/lists/*Packages 
\c rm /var/lib/apt/lists/*Sources

on the client to purge APT's internal cache, and then rerun "apt-get update" there.

\H{cache-overview}Cache overview

To get a basic overview of the cache contents, the distkill.pl script may be used. See \k{distkill} for details and warnings.

\c # /usr/lib/apt-cacher-ng/distkill.pl
\c Scanning /var/cache/apt-cacher-ng, please wait...
\c Found distributions:
\c 	1. testing (6 index files)
\c 	2. sid (63 index files)
\c 	3. etch-unikl (30 index files)
\c 	4. etch (30 index files)
\c 	5. experimental (505 index files)
\c 	6. lenny (57 index files)
\c 	7. unstable (918 index files)
\c 	8. stable (10 index files)
\c 
\c WARNING: The removal action would wipe out whole directories containing
\c          index files. Select d to see detailed list.
\c 
\c Which distribution to remove? (Number, 0 to exit, d for details): d
\c 
\c Directories to remove:
\c  1. testing: 
\c   /var/cache/apt-cacher-ng/debrep/dists/testing 
\c  2. sid: 
\c   /var/cache/apt-cacher-ng/localstuff/dists/sid 
\c   /var/cache/apt-cacher-ng/debrep/dists/sid 
\c  4. etch: 
\c   /var/cache/apt-cacher-ng/ftp.debian-unofficial.org/debian/dists/etch 
\c  5. experimental: 
\c   /var/cache/apt-cacher-ng/debrep/dists/experimental 
\c  6. lenny: 
\c   /var/cache/apt-cacher-ng/security.debian.org/dists/lenny 
\c   /var/cache/apt-cacher-ng/debrep/dists/lenny 
\c  7. unstable: 
\c   /var/cache/apt-cacher-ng/debrep/dists/unstable 
\c   /var/cache/apt-cacher-ng/localstuff/debian/dists/unstable 
\c  8. stable: 
\c   /var/cache/apt-cacher-ng/debrep/dists/stable 
\c Found distributions:
\c 
\c WARNING: The removal action would wipe out whole directories containing
\c          index files. Select d to see detailed list.
\c

\H{ssluse}Access to SSL/TLS remotes (HTTPS)

It is possible to have encrypted access to remote sites via HTTPS protocol with
recent versions of apt-cacher-ng if the OpenSSL support was enabled at compile
time. However this leads certain side effects and complications; due to the
nature of the HTTPS connection model, it is not possible to act as an
intermediate server (e.g. caching proxy) by the same rules as with HTTP:

\b SSL ensures strict verification of remote host name. Therefore, you cannot
add an URL to sources.list and make them point to "hostname of acng server": the download
client would expect the remote site to offer the certificate of "hostname of acng server" but the certificate of the remote server would be used.

\b This restriction also applies if the LAN admin somehow manages to trick the
client that it connects to the remote site but the real connection is silently
rerouted to the local server: since the proxy in the middle cannot fake the
real certificate it would need to create a custom one but that manipulation
usually cannot go unnoticed by the client unless some local hack disables or
manipulates the certificate verification. All parts of this should be
considered a crude hack and won't be described here in detail. In fact,
apt-cacher-ng does not implement local SSL server functionality (as of version
0.8.1), partly because of these considerations.

\b If the client assumes that it needs to access a HTTPS remote server through
a HTTP proxy, it usually goes for the HTTP tunneling (see the
\W{http://en.wikipedia.org/wiki/HTTP_tunnel}{Wikipedia article} for details). This is a
well-known and widely-used method but it comes with some disadvantages: the
proxy has no control of the data flowing through it so the proxy cannot cache
the data in any useful way; in environments with additional security
restrictions, it's also hard to identify malicious users that abuse the service
for illegitimate purposes.

Considering these difficulties, there are three (and a half) methods to use SSL.

\b First, the "half method" - not using the proxy at all, configuring each
client to not use the HTTP proxy for HTTPS urls. This will obviously disable
central caching and requires the client has separated configuration options to
set this. For Debian based distros, this can be done by adding a configuration
like this:
\c{Acquire::https::proxy "DIRECT";}
to apt.conf or one of the apt.conf.d files. See \k{prob-proxy} for further information.

\b The "backend configuration method": if the clients access the remote sites through URLs 
remapped on the server, the cacher admin can add https URLs to backend lists
instead of http urls. Data will be cached just like usual.

\b The "laissez-faire method": in acng.conf (or related) configure the
\c{PassThroughPattern} option to contain a regex like \c{.*} and configure the clients to
use apt-cacher-ng as HTTP proxy and let the clients connect to https URLs "as
usual". Some limited access control can be achieved through adjustment of the regular expression (.* permits access to any host and any port, including 443 for https). Data is not cached on the server.

\b The "tell-me-what-you-need method": on the client side, modify the access
URLs and change https to http and put the string "HTTPS///" between http:// and the host name. With this trick, the user client will access the proxy like going for a usual HTTP download and the proxy will access the remote URL with the https protocol. Caching (and file merging to repositories) will work and there is still enough flexibility for the users. The disadvantages of this method are basically the same as with the access URLs rewriting (see \K{config-client}) but the method is still the preferred one by the apt-cacher-ng author. For apt's sources.list, the modification may look like the following example.

\c deb http://HTTPS///get.docker.com/ubuntu docker main
\c # If apt-cacher-ng is configured as proxy in APT, this makes it
\c # switch internally to https://get.docker.com/ubuntu
\c deb http://acnghost:3142/HTTPS///get.docker.com/ubuntu docker main
\c # Basically the same just with access to apt-cacher-ng through
\c # URL rewritting instead of setting http proxy.

\H{jigdo}JIGDO usage

It's possible to use apt-cacher-ng source with the jigdo-lite utility. There are some limitations, though:

\b since many mirrors do not distribute the jigdo files (or even nothing from
cdimage.debian.org at all), there is a high chance to be redirected to a such
mirror when using the backend-mapped configuration. I.e. when user follows the
official documentation and edits wgetOpts in the jigdo configuration, it will
fail in many cases.

\b apt-cacher-ng does not support .template files properly. They might be
cached but will be expired (removed from cache), sooner or later.

But it's possible to feed jigdo-lite with the package contents from your
mirror. To do that, first start jigdo-lite as usual, something like:

\c{jigdo-lite http://cdimage.debian.org/.../...-DVD-1.jigdo}

When asked about Debian mirror, enter something like:

\c{http://proxy.host:3142/ftp.de.debian.org/debian/}

i.e. construct the same URL as present in usual apt-cacher-ng's user's sources.list.

That's all, jigdo-lite will fetch the package files using apt-cacher-ng proxy.

\H{prob-proxy} Avoid use of apt-cacher-ng for certain hosts

Sometimes clients might need to access some remote side directly to do some
non-file-transfer oriented work but still passing the data through configured
apt-cacher-ng proxy. Such remote hosts can be marked for direct access in apt
configuration, e.g. in \c{/etc/apt/apt.conf}:

\c Acquire::HTTP::Proxy::archive.example.org "DIRECT";
\c //or Acquire::HTTP::Proxy::archive.example.org  "other.proxy:port"

\H{howto-dontcache1} Avoid caching for certain domains or certain file types

Sometimes clients to download through apt-cacher-ng but the data shall not be stored on the harddisk of the server. To get it, use the DontCache directive (see examples for details) to define such files.

\H{howto-faster} How to make big download series faster

Symptom: A common situation is a periodic download of hundreds of files through
apt-cacher-ng where just a half is present in the cache. Although caching works
fine, there are visible delays on some files during the download.

Possible cause and relief: the download from the real mirror gets interrupted
while apt-cacher-ng delivers a set of files from the internal cache. While the
connection is suspended, it times out and needs to be recreated when a miss
occurs, i.e. apt-cacher-ng has to fetch more from the remote mirror. A
workaround to this behaviour is simple, provided that the remote mirror can
handle long request queues: set the pipelining depth to a very high value in
apt.conf file or one of its replacement files in /etc/apt/apt.conf.d/. With
something like:

\c{Acquire::http \{ Pipeline-Depth "200"; \} }

there is a higher chance to get the server connection "preheated" before a
stall occurs.

\H{howto-importiso} How to import DVDs or ISO images

First, it should be clear what is needed to be done. In order to integrate the
packages from a DVD or ISO image, read on in \k{howto-importdisk}.

The situation with ISO files import is complicated. They are not supported by
the cache and there is also no expiration mode for them. The feature might be
considered for addition in some future release of apt-cacher-ng.

What is possible now is publishing a directory with ISO files using its web
server mode, see \c{LocalDirs} config option for details.

\H{howto-importdisk} How to integrate DVDs or ISO image data

Integrating package files from DVD or ISO images is not much different to the
usual import operation, see above for instructions.

One possible way to get files into the \c{_import} directory is simply mounting
it there:

\c  mount -o loop /dev/cdrom /var/cache/apt-cacher-ng/_import

After running the import operation, the disk can be umounted and removed.

A possible variation is import via symlinks. This can make sense where the space
consumption must be reduced and the ISO image should stay on the server for a
long time. To achive this, the image should be mounted at some mount point
outside of the \c{_import} directory; the mounted state should be made permanent, maybe via
an /etc/fstab entry with the loop option; then a symbolic link tree
pointing to the mountpoint location should be created in the \c{_import}
directory (something like \c{cp -as /mnt/image_jessie_01/pool
/var/cache/apt-cacher-ng/_import/}). The subsequent "import" operation should
pick up the symlinks and continue using them as links istead of file copies.

\H{howto-hooks} How to execute commands before and after going online?

It is possible to configure custom commands which are executed before the
internet connection attempt and after a certain period after closing the
connection. The commands are bound to a remapping configuration and the config
file is named after the name of that remapping config, like \c{debrep.hooks} for
\c{Remap-debrep}. See \k{remap-trickz}, \c{conf/*.hooks} and 
\c{/usr/share/doc/apt-cacher-ng/examples/*.hooks} files for details.

\H{howto-interfaces} Listen to only specific interfaces or IP protocols 

Unless configured explicitely, the server listens to any interface with IPv4 or
IPv6 protocol. To disable some of this, use the \c{BindAddress} option. It
should contain a list of IP adresseses associated with particular network
interfaces, separated by space. When option is set then the server won't listen
to addresses or protocols not included there.

To limit to specific IP protocol, the address should only be present in the
protocol specific syntax (like 192.0.43.10) will limit the use to the specific
protocol.

The usual wildcard addresses can also be used to match all interfaces
configured for the specific protocol, like 0.0.0.0 for IPv4.

\H{howto-outproto} How to avoid use of IPv4 (or IPv6) where possible?

Usually, outgoing hosts are accessed by the protocol and with the target IP
reported as the first candidate by operating system facilities (getaddrinfo).
It is possible to change this behavior, i.e. to skip IPv6 or IPv4 versions or
try IPv6 connection first and then use IPv4 as alternative (or vice versa). See
option \e{ConnectProto} in configuration examples.

\H{howto-acngfs} Use the proxy without storing all data twice

There is a general use case where the data storing behavior of APT is not so
fortunate. Imagine an old laptop with a slow and small harddisk but a modern
network connection (i.e. Cardbus-attached WLAN card). But there is not enough
space for APT to store the downloaded packages on the local disk, or not enough
to perform the upgrade afterwards.

A plausible workaround in this case are moving contents of
/var/cache/apt/archives directory to a mounted NFS share and replacing the
original directory with a symlink (or bind-mount to the mentioned share).
However, this solution would transfer all data at least three times over
network. Another plausible workaround might be the use of curlftpfs which would
embedd a remote FTP share which then can be specified as file:// URL in
sources.list. However, this solution won't work with a local HTTP proxy like
apt-cacher-ng (and httpfs http://sourceforge.net/projects/httpfs/ is not an
alternative because it works only with a single file per mount).

As real alternative, apt-cacher-ng comes with an own implementation of a http
file system called \c{acngfs}. It makes some assumptions of proxy's behaviour
in order to emulate a real directory structure. Directories can be entered but
not browsed (i.e. content listing is disallowed because of HTTP protocol
limitations). Anyhow, this solution is good enough for APT. When it's checking
the contents of the data source located on acngfs share, it reads the file
contents of just the files required for the update which makes the
apt-cacher-ng server download them on-the-fly.

And finally, angfs usage can be optimized for local access. This works best if
the proxy daemons runs on the same machine as acngfs and there are hundreds of
packages to update while filesystem access costs are negligible. Here the cache
directory can be specified in acngfs parameters, and then it gets files
directly from the cache if they are completely downloaded and don't have
volatile contents.

\H{optproxy} Optional semi-automatic use of proxy

Apt-Cacher NG daemon has some optional operations modes regading the use of
external proxy (configured with "Proxy" setting). The default mode means use of
that proxy for all remote connections.

However, in some environments like when running on a portable system, the
"upstream" proxy server may not reachable when running outside of home LAN
area. For this situation, there are automated ways to detect it and to switch
to direct internet access without additional configuration work.

One way is to use a (short) timeout value (OptProxyTimeout setting) which
simply makes a failed connection attempt after the timeout being interpret as
broken proxy. Then the proxy is not used in context of this connection until a
new connection is to be established. If OptProxyCheckInterval is set (see
below) then this change is effective for the configured time span. 

Another way is through a custom (shell) command - when it returns successfully,
the proxy is used, otherwise not and the command will be rerun only after a
specified period. The command may check IP routes or a specific router address
in the MAC pool or similar traces of the current environment. This setting
(OptProxyCheckCommand) is also affected by another option
(OptProxyCheckInterval) that specifies how often this check command shall be
rerun. In the meantime, the configured proxy is considered faulty.

Hint: this special modes can be combined with non-caching behavior (see
above) and another ACNG proxy in the home LAN to get smart caching for laptop
users.

\H{mirroring} Partial Mirroring

It is possible to create a partial local mirror of a remote package repository.
The method to do this is usually known as pre-caching. A such mirror would
contain all files available to apt through \c{apt-cacher-ng}, making the cache
server suitable for pure off-line use.

The config uses index files in the local cache in order to declare which
remote files shall be mirrored. Choice of relevant files decides which branch,
which architecture or which source tree is to be mirrored. For convenience,
it's possible to use glob expressions to create semi-dynamic list. The format
is shell-like and relative to cache directory, a shell running in the cache
directory can be helpful to verify the correctness.

\e{Example:}

\cw{PrecacheFor: debrep/dists/unstable/*/binary-amd64/Packages*} 

\cw{PrecacheFor: emacs.naquadah.org/unstable/*}

Assuming that debrep repository is configured with proper remapping setup (see
above), this would download all Debian packages listed for amd64 architecture in the unstable branch.

There is also support for faster file update using deltas, see
\W{http://debdelta.debian.net/}{Debdelta} for details. The delta_uri URL
mentioned there needs to be added as deltasrc option, see \k{remap-trickz} for
details.

The operation is triggered using the web interface, various options or
estimation mode can also be configured there. The CGI URL generated by the
browser can be called with other clients to repeat this job, for example in a
daily executed script. Another possible command line client can be the
\c{expire-caller.pl} script shipped with this package (replacing the CGI
parameters through environment, see \k{auto-cleanup}). For regular tools like
wget or curl, remember the need of quotation and secrecy of user/password data
- command calls might expose them to local users.

\C{troublefaq} Troubleshooting

\H{debug}Debugging

Preliminary meanings of Debug option settings are:

\b 0: No debug printing

\b 1: Log file buffers are flushed faster

\b 2: Some additional information appears within usual transfer/error logs

\b 4: extra debug information is written to apt-cacher.err (also enables lots of additional trace points when apt-cacher-ng binary is built with debug configuration, see \k{prob-freeze} for details)

To combine that settings, add them (i.e. 7 enables all messages and log flushing)

Getting HTTP headers from apt-get works like this:

\c apt-get update -o Debug::Acquire::Http=true

\H{cleanup-damaged} Problem: keeps delivering damaged files

Even in this millennium, sometimes damaged files are downloaded from the server
and are stored in the cache. Sometimes lazy maintainers of 3rd party archives
replace package files with the same name but different contents. Sometimes the
server's file system gets corrupted without detection by the OS.

Anyhow, there might be cases where cached data becomes invalid. Volatile files
might be replaced by fixed version on some future download but static package
files are never changed upon completion and even incomplete downloads are
resumed and keep bad data downloaded before.

Usually the damage is only discovered by the client later. The particular file
can be located in the cache and replaced manually. And if there are many of
them, a mass file check might be needed to clean the mess. Fortunately, there
are helpers in cache maintenance interface to automate this process.

To start, visit the web control interface and check the options of
\e{Expiration} task. Enable the check for explicit paths and the check of data
contents, then start the expiration. With this parameters, complete files with
incorrect checksum are detected. The default action for such files is adding
them to a list of damage files. After that, the "Delete damaged files" button
in the main web page can be used to remove them (or the Show button to display
them first). Alternatively, the checkboxes appearing aside of each damage
detection can be used together with the control buttons which appear at the end
of the report. And another way of dealing with them is truncating (setting to
zero size). This can be done on-the-fly and is enabled by the expiration
parameters, or with the appropriate command button in the web interface.

NOTE: several index files and related support files can create false positives,
i.e. as incomplete or bad files. This usually happens because their volatile
contents has changed but the file was not downloaded for a while and another
version of it was used instead (like bzip2-compressed instead of
gzip-compressed or uncompressed). The default code attempts to detect files
with good reasons to stay in the cache and does not mark them as damaged.

\H{cleanup-prob} Problem: regular expiration action reproducibly aborts

A quick investigation of action logs should help identifying the problem. A
typical one is a mirror listed somewhere which is not reachable when expiration runs.

Unfortunately there is no simple and safe way to solve this. One method is
setting the ExAbortOnProblems configuration variable, but this can destroy the
whole cache if a bigger problem with index file occurs and this state remains
unnoticed for many days until ExTreshold period (see configuration) is over.

Another way is listing the index files of the faulty mirrors to a special file.
It needs to be stored as "ignore_list" in the configuration directory and store
one path name per line with paths relative to the cache directory, as seen in
the error messages.

\H{sigbus-prob} Problem: cacher suddenly terminates, log reports IO errors

For simplicity and memory saving reasons, apt-cacher-ng assumes that some files
can be opened exclusively for reading and they don't suddenly become
unreadable. Unfortunatelly, in some conditions and in case of IO errors, this
doesn't work and the file systems sends a fatal signal and eventually
terminates the program.

To track this down to the likely reasons, it's possible to execute a custom script in the moment of the reception of the fatal signal. To do that, add something like this to the configuration:

\c BusAction = ls -l /proc/$PPID/ | mail -s SIGBUS! root

This command would send a list of opened files with paths to the "root" mail
user, and the root shall check the state of those files (maybe run them through
md5sum or similar).

NOTE: Beware of the security implications of this
configuration option. It runs regular shell code in context of the daemon user
in a blocking way, so it may be vulnerable to symlink attacks and it will delay the
automatic restart of the daemon through systemd.

\H{503-prob} Problem: download fails with 503 ... status message

Code 503 usually represents an internal failure which could not be described
correctly by other HTTP status codes. In the most cases it's caused by file
system errors or incorrect cache directory setup, like files or directories
with incorrect owner, missing write/read permissions for the effective user
account or other system related exceptions like running out of disk space.

The log file \c{apt-cacher.err} located in the \c{LogDir} directory should
document more details. In case it doesn't, setting the \c{Debug} config option to a higher value 
might reveal more information.

Fixing permission problems shouldn't be a real challenge for system
administrators. Usually, a command set like this should do the trick on
Debian/Ubuntu systems, assuming that all group users should receive write
access to the cache files:

\c  chown -R apt-cacher-ng:apt-cacher-ng /var/cache/apt-cacher-ng
\c  chmod -R a+rX,g+rw,u+rw /var/cache/apt-cacher-ng

\H{prob-freeze} Problem: \c{apt-get} freezes when downloading files

Solution: First, check:

\b Free disk space and inode usage (\c{df}, \c{df -i})

\b Internet connection to the remote sites (browse them via HTTP, e.g. visiting http://ftp.your.mirror)

If nothing helps then you may have hit a spooky problem which is hard to track
down. If you like, help the author on problem identification. To do that, do:

\c   su -
\c   # enter root password
\c   cd /tmp
\c   apt-get source apt-cacher-ng
\c   apt-get build-dep apt-cacher-ng
\c   cd apt-cacher-ng-*
\c   ./build.sh DEBUG
\c   /etc/init.d/apt-cacher-ng stop
\c   builddir/apt-cacher-ng -c /etc/apt-cacher-ng logdir=/tmp foreground=1 debug=7
\c   # (let apt-get run now, on timeouts just wait >> 20 seconds)
\c   # stop the daemon with Ctrl-C
\c   /etc/init.d/apt-cacher-ng start
\c   # compress /tmp/apt-cacher.err and send it to author
\c   chown -R apt-cacher-ng:apt-cacher-ng /var/cache/apt-cacher-ng

The value of debug can be varied to have different verbosity (see \k{debug} for
more information about Debug levels).

\H{prob-bzip2} \c{apt-get} reports corrupted bzip2 data

Symptoms: apt-get fails to run through "update" no matter what you do. And you
may have get a message like this one.

\c 99% [6 Packages bzip2 0] [Waiting for headers] [Waiting for headers]
\c bzip2: Data integrity error when decompressing.
\c         Input file = (stdin), output file = (stdout)
\c
\c It is possible that the compressed file(s) have become corrupted.
\c You can use the -tvv option to test integrity of such files.
\c 
\c You can use the `bzip2recover' program to attempt to recover
\c data from undamaged sections of corrupted files.
\c 
\c Err http://debian.netcologne.de unstable/main Packages              
\c   Sub-process bzip2 returned an error code (2)

\b This might be one of Apt's problem with insufficient handling of errors, i.e. passing 
incomplete files to bzip2 on premature connection termination. Retry the update and it might work.

\b Another issue is more severe: old versions of apt-cacher-ng had a bug which
could cause data corruption while resuming downloads however this problem
appears only in unusual conditions. To make sure there are no broken files in
your repository, run the Expiration task with content verification enabled, and
also "immediate deletion" (or delete later after checking the list). See
\k{cleanup-manual} for details.

\H{prob-eaddrinuse} Problem: \c{apt-cacher-ng} refuses to start with "Address already in use"

Another service is already listening on the port which apt-cacher-ng is
configured to use. This might be the apt-cacher daemon which used the same port
number by default. To identify the daemon behind that process, use the fuser
utility, executing it as root for IPv4 and IPv6 protocol versions. Example:

\c fuser -4 -v -n tcp 3142
\c fuser -6 -v -n tcp 3142
\c                     USER        PID ACCESS COMMAND
\c 3142/tcp:           xwwwfsd   17914 F....  xwwwfsd

(where 3142 is the port number from the apt-cacher-ng configuration file). To
resolve the collision, reconfigure the other daemon or apt-cacher-ng to use
another free port (and reconfigure the clients to use the new apt-cacher-ng
port accordingly).


\C{bugs} Known Bugs and Limitations

\b Only the "Basic" type of HTTP proxy authentication is supported at the moment

\b Only HTTP HEAD and GET commands are supported properly.

\b CONNECT (for https/SSL/TLS tunnels) is supported but needs to be enabled by local administrators, see \k{ssluse} for details

\b POST support is restricted to apt-listbugs actions and is actually implemented as simple data forwarding with minor header manipulation. If additional proxy is configured, the client will be redirected to the HTTPS version of bugs.debian.org service, i.e. switching to the CONNECT method (see above)

\b Transparent proxy mode not implemented yet

\b The metadata update in maintenance jobs (like expiration task) attempts to guess the best source of update based on original source information and current backend configuration. In case of repeating failures, please run a "forced" download of index files through the web interface

\b See TODO file in apt-cacher-ng source for more related information

\C{cont} Contact

The planned features are listed in the file TODO. Don't hesitate to
contact the author if you really need something not found there and you can
explain the severity of your request.

There are also a public
\W{https://alioth.debian.org/mail/?group_id=100566}{mailing list} and
\W{https://alioth.debian.org/tracker/?group_id=100566}{request trackers}
available on the \W{http://alioth.debian.org/projects/apt-cacher-ng/}{Alioth project page}.

And last, but not least: feel free to express your gratitude by donating a
small amount of money via \W{https://www.paypal.com}{PayPal} (to edi@gmx.de) or just
\W{https://flattr.com/thing/51105/Apt-Cacher-NG}{Flattr it!}

\versionid Eduard Bloch, Sun, 19 Apr 2015 10:25:49 +0200
