Cyrus IMAP Server: Mailbox File Formats
***************************************


Intro
=====

This documentation refers to the "version 12" cyrus index format and
associated mailbox files.

No external tools should make use of this information. The only
supported method of access to the mail store is through the standard
interfaces: IMAP, POP, NNTP, LMTP, etc.

A cyrus mailbox is a directory in the filesystem. It contains the
following files:

* zero or more message files

* the "cyrus.header" metadata file

* the "cyrus.index" metadata file

* the "cyrus.cache" metadata file

* zero or one "cyrus.squat" search indexes

* zero or more subdirectories

With "split metadata" configuration, the mailbox may actually be split
between multiple disks, with the files being in the same relative
directory on the meta disk. See the "imapd.conf" option
"metapartition_files" for more information


Message Files
=============

The message files are named by their UID, followed by a ".", so UID
423 would be named ""423."". They are stored in wire-format: lines are
terminated by CRLF and binary data is not allowed.


"cyrus.header"
==============

This file contains mailbox-wide information that does not change
often. Its format:

   <Mailbox Header Magic String>
   <Quota Root>\t<Mailbox Unique ID String>\n
   <Space-separated list of user flags>\n
   <Mailbox ACL>\n

The Mailbox Unique ID String is used for non-owner per-user \Seen
flags so they remain with the mailbox during renames, and also by the
replication subsystem to detect mailbox renames.

The ACL is a copy of the value stored in mailboxes.db, and isn't
actually used.


Locking Considerations
----------------------

The "cyrus.index" file must be locked in exclusive mode while making
changes to the "cyrus.header" file to ensure consistency. All changes
are made by rewriting the entire file and renaming the new version
into place.


"cyrus.cache"
=============

The "cyrus.cache" file is a pure cache of information that's also
present in the message files. It exists to make ENVELOPE and specific
header fetches more efficient, as well as to assist with searches and
sorts.

If a "cyrus.cache" file is missing or corrupted, it can be re-
generated by running a "reconstruct" on the mailbox.

The format is 10 individual records each prefixed with a 32 bit length
value in network byte order. The offset of each message's cache record
is stored in the "cyrus.index" file (documented below). The records in
a cyrus.cache file are of variable length, depending on the contents
of the associated message.

The first 4 bytes of the cyrus.cache file are a "generation number"
which must match the first 4 bytes of the associated cyrus.index file.
In the past this was used to track consistency between the files, but
the name locking scheme and per-record CRC check in cyrus 2.4 and
above means this is just a backup consistency check rather than an
essential format feature.

   +------------------------------------------------------------------------+
   |Gen # (32bits)|Size 1 (32bits)|Data 1                                   |
   +------------------------------------------------------------------------+
   |           |Size 2 (32bits)|Data 2            |Size 3 (32bits)| Data 3  |
   +------------------------------------------------------------------------+
   | .....                                                                  |
   +------------------------------------------------------------------------+

While there are occasional changes to the cache format, this
information is NOT stored in the cyrus.cache file. Instead, there is a
"cache_version" field in the cyrus.index record, so multiple different
versions of cache data may exist in the same cache file.

The order of fields per record in the cache file is as follows: (keep
in mind that they are all preceded by a 4 byte network byte order
size).

Envelope Response
   Raw IMAP response for a request for the envelope.

Bodystructure Response
   Raw IMAP response for a request for the bodystructure.

Body Response
   Raw IMAP response for an (old style) request for the body.

Binary Bodystructure
   Offsets into the message file to pull out various body parts.
   Because of the nature of MIME parts, this is somewhat recursive.

   This looks like the following (starting the octet following the
   cache field size). All of the fields are bit32s.

      [
       [Number of message parts+1 for the rfc822 header if present]
       [
        [Offset in the message file of the header of this part]
        [Size (octets) of the header of this part]
        [Offset in the message file of the content of this part]
        [Size (octets) of the content of this part]
        [Encoding Type of this part]
       ]
          (repeat for each part as well as once for the headers)
       [zero *or* number of sub-parts in the case of a multipart.
        if nonzero, this is a recursion into the top structure]
          (repeat for each part)
      ]

   Note if this is not a message/rfc822, than the values for the sizes
   of the part 0 are -1 (to indicate that it doesn't exist). Sub-parts
   are not possible for a part 0, so they aren't included when finding
   recursive entries.

   The offset and size info for both the mime header and content part
   are useful in order to do fast indexing on the appropriate parts of
   the message file when a client does a FETCH request for
   BODY[HEADER], or BODY[2.MIME].

   Note that the top level RFC822 headers are a treated as a separate
   part from their body text ("0" or "HEADER").

   In the case of a multipart/alternative, the content size & offset
   refers to the size of the entire mime part.

   A very simple message (with a single text/plain part) would
   therefore look like:

      [[2][rfc822 header][text/plain body part info][0]]

   A simple multipart/alternative message might look like:

      [[3][rfc822 header][text/plain message part info]
          [second message part info][0][0]]

   A message with an attachment that has two subparts:

      [[3][rfc822 header info][rfc822 first body part info][attachment info][0][
            [3][NIL header info][sub part 1 info][sub part 2 info][0][0]]]

   A message with an attached message/rfc822 message with the
   following total structure:

      message/rfc822
        0 headers; content-type: multipart/mixed
        1 text/plain
        2 message/rfc822
          0 headers; content-type: multipart/alternative
          1 text/plain
          2 text/html

      [[3][rfc822 header part 0][text/plain part 1][overall attachment info][0][
           [3][rfc822 header part 2.0][text/plain part 2.1][text/html part 2.2]
              [0][0]]]

Cache Header
   Any cached header fields. The exact set of fields here depends on
   the cache record version - there is a function in "imap/mailbox.c"
   to determine if a named header would be cached based on the
   version. These are in the same format they would appear in the
   message file:

      HeaderName: headerdata\r\n

   Examples include: References, In-Reply-To, etc.

From
   The from header.

To
   The to header.

Cc
   The CC header.

Bcc
   The BCC header.

Subject
   The Subject header.


Locking Considerations
----------------------

The "cyrus.index" file must be locked in exclusive mode while making
changes to the "cyrus.cache" file to ensure consistency. All new cache
records are created by reading the current end-of-file offset,
appending the new cache record, and storing that start offset into the
associated cyrus.index record.


"cyrus.index"
=============

The cyrus.index file is NOT just a cache - it stores information not
present in the message file!

The cyrus.index file consists of a fixed width header, followed by
fixed width records. In the past, it would be rewritten on every
expunge, but since Cyrus 2.4 the expunged records remain in the
cyrus.index file for a configurable time to support QRESYNC and more
efficient delayed expunge.

The cyrus.index file is the "heart" of the mailbox format - containing
checksums (CRC32) of everything else, and the most frequently updated
fields. All fields are stored in network byte order and aligned on 4
byte boundaries. Due to some 64 bit values being stored, the header
and individual records are aligned on 8 byte boundaries.

The overall format looks sort of like this:

   cyrus.index:
   +----------------+
   | Mailbox Header |
   +----------------+
   | Msg: Num 1     |
   +----------------+
   | Msg: Num 2     |
   +----------------+
   |     ...        |
   +----------------+

The basic idea being that there is one header, and then all the
message records are evenly spaced throughout the file. All of the
message records are at well-known offsets, making any part of the file
accessable at roughly equal speed.


Locking Considerations
----------------------

"cyrus.index" files can not be repacked (i.e. records can not change
UID for a particular offset, and the file can't be rewritten or
deleted) unless there's an exclusive namelock held for the mailbox
name. This is to avoid race conditions and simplify the use of
mailboxes. Whenever a mailbox is opened, the caller holds a shared
namelock on the mailbox name for the duration of the "mailbox
object"'s existence.

All reads of a "cyrus.index" file must be done with a lock held, and
all writes must be done with an exclusive lock held. This ensures
CRC32 checksums of individual headers and records are always
consistent. There are no direct "offset" reads done any more, instead
the mailbox API provides a way to read an entire cyrus.index header or
cyrus.index record into a struct, performing consistency checks.
Writes are also done with a complete record struct.


Detail of "cyrus.index" header
------------------------------

The index header contains the following information, in order:

Generation Number (4 bytes)
   A number that is basically the "revision number" of the mailbox. It
   must match between the cache and index files. This is to ensure
   that if we fail to sync both the cache and index files and a crash
   happens (so that only one is synced), we do not provide bad data to
   the user. This is also backed by having individual cache checksums
   on each record.

Format (4 bytes)
   Basically obsolete (indicates netnews or regular).

Minor Version (4 bytes)
   Indicates the version number of the index file. This can be used
   for on-the-fly upgrades of the index and cache files.

Start Offset (4 bytes)
   Size of index header.

Record Size (4 bytes)
   Size of an index record.

Num Records (4 bytes)
   How many records are in this index (including records for expunged
   records. See below for "Exists" which has moved from pre-version 12
   files.

Last Appenddate (4 bytes)
   (time_t) of the last time a message was appended

Last UID (4 bytes)
   Highest UID of all messages in the mailbox (UIDNEXT - 1).

Quota Mailbox Used (8 bytes)
   Total amount of storage used by all of the messages in the mailbox.
   Platforms that don't support 64-bit integers only use the last 4
   bytes.

POP3 Last Login (4 bytes)
   (time_t) of the last pop3 login to this INBOX, used to enforce the
   "poptimeout" "imapd.conf" option.

UIDvalidity (4 bytes)
   The UID validitiy of this mailbox. Cyrus currently uses the
   "time()" when this mailbox was created.

Deleted, Answered, and Flagged (4 bytes each)
   Counts of how many messages have each flag.

Mailbox Options (4 bytes)
   Bitmask of mailbox options, consisting of any combination of the
   following:

   POP3_NEW_UIDL
      Flag signalling that we're using "*uidvalidity*.*uid*" instead
      of just "*uid*" for the output of the POP3 UIDL command.

   IMAP_SHAREDSEEN
      Flag signalling that we're supporting a shared \Seen flag on the
      mailbox.

   IMAP_DUPDELIVER
      Flag signalling that we're allowing duplicate delivery of
      messages to the mailbox, overriding system-wide duplicate
      suppression.

   MAILBOX_NEEDS_REPACK
      Flag signalling that the mailbox is due to be repacked. During
      mailbox_close() every process will attempt to take an exclusive
      namelock on the mailbox and repack.

   MAILBOX_DELETED
      Flag signalling that the mailbox is deleted. This can be set
      with a shared namelock, and indicates to all other users of the
      mailbox that they need to close it and attempt cleanup. The last
      process to close the mailbox will perform the final cleanup
      under an exclusive namelock, giving the other processes a chance
      to finish their current operation first without files
      disappearing from under them!

Leaked Cache (4 bytes)
   Number of leaked records in the cache file.

Highest ModSeq (8 bytes)
   Highest Modification Sequence of all the messages in the mailbox
   (CONDSTORE).

Deleted ModSeq (8 bytes)
   Lowest Modification Sequence before which expunged message data may
   have been purged from the mailbox and forgotten (CONDSTORE/QRESYNC
   support).

Exists (4 bytes)
   See NumRecords above. This is the count of non-expunged records in
   the mailbox and corresponds to the IMAP status item "EXISTS".

First Expunged (4 bytes)
   lowest modified time of an expunged message in this mailbox (or
   zero if there are no expunged messages) - used to determine if the
   mailbox needs repacking.

Last Repack Time (4 bytes)
   a timestamp for the last repack, to ensure repacks aren't done too
   close together if expunges were closely spaced

Header File CRC (4 bytes)
   CRC32 value of the bytes in the "cyrus.header" file for this
   mailbox. Must be rewritten whenever the cyrus.header file is
   changed (see locking considerations above - this is why the
   cyrus.index must be exclusively locked!)

Sync CRC (4 bytes)
   An XOR of the CRC32 of a specially generated value for each of the
   non-expunged records in this mailbox. This is a cached value which
   allows the replication subsystem to quickly determine that all non-
   expunged records in a mailbox are in sync and detect possible
   "split brain" scenarios with low bandwidth use.

Recent UID (4 bytes)
   The highest UID last time an IMAP client logged in as the mailbox
   owner (or anybody if SHAREDSEEN is enabled) selected this mailbox.
   Used to generate the \Recent flags in IMAP

Recent Time (4 bytes)
   Used for consistency with the seen_db code, but probably not
   actually necessary. Oh well

Header CRC (4 bytes)
   Must always be the LAST record of the header. This is the CRC32 of
   the actual bytes on disk (network order format) for the rest of the
   cyrus.index. By keeping it last, it can be easily calculated with
   the following snippet of code: "crc = crc32_map(buf,
   OFFSET_HEADER_CRC);" - i.e. crc32 from the start of the buffer to
   just before this field.

There are also spare fields in the index header, to allow for future
expansion without forcing an upgrade of the file, and to round up to
be divisible by 8 bytes.


Detail of "cyrus.index" records
-------------------------------

These records start immediately following the "cyrus.index" header,
and are all fixed size. They are in-order by uid of the message.

UID (4 bytes)
   UID of the message

INTERNALDATE (4 bytes)
   INTERNALDATE of the message (where possible, this matches the
   creation and modification times of the file on disk to help
   reconstruct in the event of data loss)

SENTDATE (4 bytes)
   Contents of the Date: header chomped to day resolution with
   timezone stripped.

SIZE (4 bytes)
   Size of the whole message (in octets)

HEADER SIZE (4 bytes)
   Size of the message header (in octets)

GMTIME (4 bytes)
   Contents of the Date: header at 1 second resolution and converted
   to GMT (for sort)

CACHE_OFFSET (4 bytes)
   Offset into the "cyrus.cache" file for the beginning of this
   message's cache entry.

LAST UPDATED (4 bytes)
   (time_t) of the last time this record was changed

SYSTEM FLAGS (4 bytes)
   Bitmask showing which system flags are set/unset

USER FLAGS (MAX_USER_FLAGS / 32 bytes)
   Bitmask showing which user flags are set/unset (bits correspond to
   positions in the cyrus.header flag list, i.e. (1<<0) == the flag
   name

CONTENT_LINES (4 bytes)
   Number of text lines contained in the message content (body).

CACHE_VERSION (4 bytes)
   Indicates the version number of the cache record for the message
   (determines which headers are cached, see list in mailbox.c).

GUID (MESSAGE_GUID_SIZE bytes)
   Globally Unique IDentifier of the message (used by replication
   engine). This is the sha1 value of the bytes as stored on disk.

MODSEQ (8 bytes)
   Modification Sequence of the message (CONDSTORE).

CACHE_CRC (4 bytes)
   This is the CRC32 of all the bytes of the cache record (all 10
   fields) as stored on disk. Again, calculated over the exact bytes
   stored in the "cyrus.cache" file.

RECORD_CRC (4 bytes)
   Like the header CRC - this is the CRC32 of all the bytes in on-disk
   order that exist in this record. Records are always rewritten as
   the entire record, including the updated CRC, so it's always
   consistent if you have a lock on the "cyrus.index" file, because
   writers will wait until they get an exclusive lock to make
   modifications.


Notes
=====

* Expunge is super quick now - it's just a flag update!

* Append is relatively fast (it only adds to the end of both the cache
  and index files and modifies the index header)

* Message unlinks always happen during the "close" phase - which may
  be noticed when you select another mailbox, but otherwise are
  delayed from the actual action. With delayed expunge, the unlinks
  are pushed off to cyr_expire which is a background task, and will
  never be noticed by the user.

* Message delivery is something like this:

  1. write/sync message file

  2. write/sync new "cyrus.cache" record

  3. write/sync new "cyrus.index" record

  4. calculate, write, sync new "cyrus.index" header

  5. acknowledge message delivery

  The message isn't delivered until the new index header is written.
  In case of a crash before the new index header is written, any
  previous writes will be overwritten on the next delivery (and will
  not be noticed by the readers).

  Note that certain power failure situations (power failure in the
  middle of a disk sector write) could cause a mailbox to need
  reconstruction (possibly even losing some flag state). These failure
  modes are not possible in the "Hardware RAID disk model" (which we
  will describe somewhere else when we get around to it).


Future considerations
=====================

* Cache all header fields? (or all up to Xk?) This could greatly
  improve speeds of clients that just ask for everything, but also
  increases the expense of rewriting the cache file (as well as the
  size it takes on disk).

* Reformat cache file to use a
  (size)(size)(size)(size)(data)(data)(data) format. This makes
  accesses anywhere in the cache file equally fast, as opposed to
  having to iterate through all the entires for a given message to get
  to the last one. Note that either way is still O(1) so maybe it
  doesn't matter much.

* It would be useful to store a uniqueid -> mailbox name index, so
  that we could fix arbitron again.
