Unit Tests
**********


Table of Contents
=================

* 1. Introduction

* 2. What Is A Unit Test?

* 3. Running The Tests

  * 3.1. Setting Up The Machine

  * 3.2 Configure Script

  * 3.3 Make

  * 3.4 Using Valgrind

  * 3.5 The Tests Are Failing

  * 3.6 Debugging A Test

* 4. Adding Your Own Tests

  * 4.1 Where To Put Your Tests

  * 4.1 Adding A New Suite

  * 4.2 Adding A Test To A Suite

  * 4.3 Suite Init And Cleanup


1. Introduction
===============

Recently, a set of regression unit tests has been added to Cyrus. This
document explains the purpose implementation of those unit tests, and
gives an example of how to add more unit tests (because there are
never enough unit tests!).


2. What Is A Unit Test?
=======================

The definition on Wikipedia sheds some light:

   ...**unit testing** is a method by which individual units of source
   code are tested to determine if they are fit for use. A unit is the
   smallest testable part of an application.

In other words, unit testing is about verifying that small pieces of
code, like individual functions, modules, or classes, work in
isolation. It is **not** about testing the system as a whole.

The tests implemented here are also **regression tests**, which in
Wikipedia's words means:

   **Regression testing** is any type of software testing that seeks
   to uncover software errors after changes to the program (e.g.
   bugfixes or new functionality) have been made, by retesting the
   program. The intent of regression testing is to assure that a
   change, such as a bugfix, did not introduce new bugs.

In other words, the tests are designed to be easy to run and to work
out fully automatically whether they have passed or failed, so that
they can be run usefully by people who didn't write them.


3. Running The Tests
====================

This section takes you through the process of running Cyrus' unit
tests.


3.1. Setting Up The Machine
---------------------------

Cyrus' unit tests are all located in a new directory, "cyrus-
imapd/cunit/". They're written in C, like the remainder of Cyrus, and
use the CUnit library from SourceForge, with some home grown wrappers
and other improvements to make our lives easier.

Your first step is step is to ensure that the CUnit library (including
the headers) is installed. Some modern operating systems already have
CUnit, for example on Ubuntu you can just do:

   me@ubuntu> sudo apt-get install libcunit1-dev

Alternately, you can download the CUnit source, build it and install
it. It's not a complicated or difficult library, this shouldn't take
long. When you've done, install it in "/usr/include" and "/usr/lib".


3.2 Configure Script
--------------------

Because of the dependency on the CUnit library, the tests are disabled
by default; this means you need enable them with an option to the
"configure" script:

   me@mybox> ./configure --enable-unit-tests
   ...
   checking for CU\_initialize\_registry in -lcunit... yes
   checking CUnit/CUnit.h usability... yes
   checking CUnit/CUnit.h presence... yes
   checking for CUnit/CUnit.h... yes
   ...


3.3 Make
--------

First you need to build Cyrus itself, using the traditional "all:"
target.

   me@mybox> make all
   ...

Then, use the new "check:" target to build and run the unit tests.

   me@mybox> make check
    cd . && /bin/bash /home/me/cyrus-imapd/missing --run automake-1.11 --foreign Makefile
    cd . && /bin/bash ./config.status Makefile depfiles (a)
   config.status: creating Makefile
   config.status: executing depfiles commands
   ...
   make[3]: Entering directory \`/home/me/cyrus-imapd'
   make[3]: \`sieve/test' is up to date.
   cunit/cunit.pl --project cunit/default.cunit --generate-wrapper cunit/mboxname.testc (b)
   gcc -DHAVE\_CONFIG\_H ... -c -o cunit/mboxname.o cunit/mboxname.testc-cunit.c
   rm -f cunit/mboxname.testc-cunit.c
   /bin/bash ./libtool --tag=CC --mode=link gcc -fPIC -g -O2 -o
   cunit/unit cunit/unit.o ... lib/libcyrus\_min.la ... (c)
   ...
   Running unit tests (d)

       CUnit - A Unit testing framework for C - Version 2.1-0
       http://cunit.sourceforge.net/

   ...
   Suite: mboxname (e)
     Test: dir\_hash\_c ... passed
     Test: to\_parts ... passed
     Test: to\_userid ... passed
     Test: to\_usermbox ... passed
   ...
   --Run Summary: Type      Total     Ran  Passed  Failed (f)
                  suites       34      34     n/a       0
                  tests       323     323     323       0
                  asserts 1079745 1079745 1079745       0
   make[1]: Leaving directory `/home/me/cyrus-imapd/cunit'

Let's take a closer look at what's happening here.

1. The "check:" target causes automake to re-run itself. This is
   normal automake behaviour. Note that the older build system used to
   run make recursively in sub-directories, the newer automake-based
   system builds everything from the top directory.

2. The "cunit/" directory contains a number of C source files (called,
   for reasons too complicated to explain here, *whatever*.testc) with
   test code in them. For each of those, a small wrapper C source file
   is generated and then compiled into an object file.

3. Finally, all the compiled object files are linked into an
   executable, with a "main()" routine from "unit.c", and a number of
   libraries and object files from other parts of the Cyrus tree.

4. The resulting executable is then run.

5. The test executable runs all the built tests one by one, telling us
   which ones passed and which ones failed as it runs them. You can
   also run it manually with the name of a test as an argument, and it
   will run only the named test.

6. At the end, the text executable prints a summary of how many tests
   it ran and how many passed and failed. The key thing to look at
   here is the rightmost column, it should be all zero.


3.4 Using Valgrind
------------------

Some failure modes are subtle, and cannot be detected in the C code
itself; this is where the Valgrind program comes in very handy. It
detects buffer overruns and memory leaks and various other kinds of
subtle errors.

To run the unit tests with Valgrind, use the new "valgrind:" target.

   me@mybox> make valgrind
   ...
   valgrind --tool=memcheck --leak-check=full ./unit -v (a)
   ==2999== Memcheck, a memory error detector
   ==2999== Copyright (C) 2002-2010, and GNU GPL'd, by Julian Seward et al.
   ==2999== Using Valgrind-3.6.0.SVN-Debian and LibVEX; [...]
   ==2999== Command: ./unit -v
   ==2999==
   ...
   --Run Summary: Type      Total     Ran  Passed  Failed   (b)
                  suites        9       9     n/a       0
                  tests        51      51      50       1
                  asserts     474     474     473       1
   ...
   ==2999== HEAP SUMMARY:   (c)
   ==2999==     in use at exit: 4,489 bytes in 134 blocks
   ==2999==   total heap usage: 715 allocs, 581 frees, 352,763 bytes allocated
   ==2999==
   ==2999== 4 bytes in 1 blocks are definitely lost in loss record 3 of 50
   ==2999==    at 0x4C2815C: malloc (vg_replace_malloc.c:236)
   ==2999==    by 0x44A0CA: xmalloc (xmalloc.c:57)
   ==2999==    by 0x4399D8: strconcat (util.c:631)
   ==2999==    by 0x40C059: test_uncast_null (strconcat.c:51)
   ==2999==    by 0x61B32A9: ??? (in /usr/lib/libcunit.so.1.0.1)
   ==2999==    by 0x61B36ED: ??? (in /usr/lib/libcunit.so.1.0.1)
   ==2999==    by 0x61B3827: CU_run_all_tests (in /usr/lib/libcunit.so.1.0.1)
   ==2999==    by 0x4066CC: run_tests (unit.c:144)
   ==2999==    by 0x406806: main (unit.c:283)
   ==2999==
   ...

Here's an explanation of what's happening in the example.

1. The test executable is run as before, but using the "valgrind"
   program. The first thing we see is Valgrind's banner message.

2. The test executable proceeds as normal and eventually emits it's
   run summary, then exits.

3. After the test executable exits, Valgrind checks for memory leaks
   and prints both a summary of all leaks and a stack trace showing
   where each block of leaked memory was allocated.

I'd just like to say that I love Valgrind and I think it's immensely
useful. I would have made running the tests under Valgrind the only
option for the "check:" target, except that Valgrind is not available
on all of Cyrus' supported platforms.


3.5 The Tests Are Failing
-------------------------

So you've noticed that some of the tests are failing. Let me make the
guiding principle of unit testing as clear as possible: **THE UNIT
TESTS SHOULD NOT FAIL**. All of the tests are designed to pass all the
time, in everyone's environment. The unit tests are run automatically
every twelve hours on the Cyrus Continuous Integration server, and a
failing test fails the whole build and makes people unhappy.

There are a few rules which you should follow to help us all get the
most benefit out of unit testing

* If you see a test failing, investigate it.

* If you can't investigate, complain on the mailing list or raise a
  bug so that somebody else can investigate.

* When writing tests, write them to work in all environments and all
  combinations of "configure" script options. It's ok to have a test
  which is empty in some circumstances; it's not ok to have a test
  that fails.

* When adding code, write new tests for the new code.

* When modifying code, write new tests for the new behaviour.

* When looking at old code, also take a look at the coverage report
  and consider writing tests for the existing code.


3.6 Debugging A Test
====================

With the new Cyrus build system, the file "cunit/unit" is no longer an
executable, it's a shell script which sets up some environment
variables before running the real executable which is hidden away.
This makes debugging a failing test somewhat challenging. The solution
is:

   me@mybox> ( cd cunit ; libtool --mode=execute gdb --args unit -t crc32 )
   ...
   Reading symbols from /home/me/cyrus-imapd/cunit/.libs/lt-unit...done.
   (gdb) list crc32.testc:1
   1       /* Unit test for lib/crc32.c */
   2       #include "cunit/cyrunit.h"
   3       #include "crc32.h"
   ...
   (gdb) break test_map
   Breakpoint 1 at 0x44a2f8: file ./cunit/crc32.testc, line 11.
   (gdb) run
   Starting program: /home/me/cyrus-imapd/cunit/.libs/lt-unit -t -v crc32
   [Thread debugging using libthread_db enabled]

       CUnit - A Unit testing framework for C - Version 2.1-0
       http://cunit.sourceforge.net/

   Suite: crc32
     Test: map ...
   Breakpoint 1, test_map () at ./cunit/crc32.testc:11
   11          c = crc32_map(TEXT, sizeof(TEXT)-1);
   (gdb)

Note the **-t** option. This turns off test timeouts, which is very
useful for manual debugging.


4. Adding Your Own Tests
========================

Adding your own tests is quite simple. Here's how.


4.1 Where To Put Your Tests
===========================

The unit test code in Cyrus is contained in a set of C source files in
the "cunit" directory. For reasons too complex to go into here, these
are named *whatever*.testc instead of the more usual *whatever*.c. If
you look closely, you will see that each of those C source files maps
to a "Suite" in CUnit parlance. For example, "cunit/glob.testc" is
listed as the Suite "glob" in CUnit's runtime output.

Typically, each Suite tests a single module or a related set of
functions; for example, "cunit/glob.testc" contains tests for the glob
module in "lib/glob.c".

So, if you want to add a new test for a module which already has some
existing tests, the sensible thing to do is to add a new test to the
existing suite. Otherwise, you'll need to add a new Suite.


4.1 Adding A New Suite
======================

Each Suite is a single C source file in the "cunit/" directory. Your
first step is to create a new C source file. For this example, you'll
create a new Suite to test the CRC32 routines which live in
"lib/crc32.c".

   me@mybox> vi cunit/crc32.testc
   ...

The file should contain something like this.

   /* Unit test for lib/crc32.c */
   #include "cunit/cyrunit.h"  (a)
   #include "crc32.h"  (b)

   static void test_map(void)  (c)
   {
       static const char TEXT[] = "lorem ipsum";  (d)
       static uint32_t CRC32 = 0x0;
       uint32_t c;  (e)

       c = crc32_map(TEXT, sizeof(TEXT)-1);  (f)
       CU_ASSERT_EQUAL(c, CRC32);  (g)
   }

Here's an explanation of what all these bits are for.

1. You need to include the header ""cunit/cyrunit.h"", which is a thin
   Cyrus wrapper around the CUnit's library's header,
   "<CUnit/CUnit.h>" with some extra conveniences.

2. You should also include any headers you need for declarations of
   the functions which you'll be testing. Note that the Cyrus "lib/"
   and "imap/" directories are already in the include path, so any
   header in there can be included without the directory prefix, e.g.
   ""crc32.h"" for "lib/crc32.h".

3. You need to have at least one function which looks like this: it
   takes no arguments, returns void, and is named "test_whatever". It
   may be "static" or "extern", but I recommend "static". Functions
   with this signature are automatically discovered in the source code
   by the Cyrus unit test infrastructure, so all you have to do is
   write the function. Later, a CUnit test named "whatever" will be
   created automatically for your "test_whatever" function.

4. Here's a good place to define the test inputs and expected outputs.
   Note that for this example you have no idea of the actual correct
   output. The right thing to do there is to manually calculate the
   expected result from first principles, or to use a different piece
   of software which you believe to be working. For this example,
   let's just use a known incorrect value and see what happens.

5. Here's a good place for local variables you need during the test.

6. Call the function under test ("crc32_map()" in this example) with
   known inputs, and capture the results in a local variable "c".

7. Compare the actual result in "c" with the expected result in
   "CRC32". The "CU_ASSERT_EQUAL()" macro checks that it's two
   arguments are equal (using an integer comparison), and if they're
   different it prints a message and records a failure. Note that
   unlike the libc "assert()" macro, control will continue even if the
   assert fails. The CUnit library provides a whole family of similar
   macros, see the online CUnit documentation for more details.

Now you need to tell the Cyrus build system about your new Suite.

   me@mybox> vi Makefile.am
   ...

You need to add the filename of your new test to the definition of the
"cunit_TESTS" variable.

   cunit_TESTS = \
       cunit/aaa-db.testc \
       cunit/annotate.testc \
       cunit/backend.testc \
       cunit/binhex.testc \
       cunit/bitvector.testc \
       cunit/buf.testc \
       cunit/byteorder.testc \
       cunit/charset.testc \
       cunit/crc32.testc \
       cunit/dlist.testc \
       cunit/duplicate.testc \

At this point you should be able to just rebuild and rerun using
**make check**. You can also just rebuild without rerunning by using
the command **make cunit/unit**.

Note that sometimes this doesn't quite work right, and you may be able
to work around this problem using the command **rm
cunit/default.cunit**.

   me@mybox> make check
   ...
   ../cunit/cunit.pl [...] --add-sources [...] crc32.testc
   ...
   ../cunit/cunit.pl [...] --generate-wrapper crc32.testc
   gcc -c [...] -g -O2 .cunit-crc32.c
   gcc [...] -o unit [...] .cunit-crc32.o ...
   Running unit tests

       CUnit - A Unit testing framework for C - Version 2.1-0
       http://cunit.sourceforge.net/

   ...
   Suite: crc32
     Test: map ... FAILED
       1. crc32.testc:12  - CU_ASSERT_EQUAL(c=1926722702,CRC32=0)

Note how the test failure told us which in source file and at what
line number the failure occurred, and what the actual and expected
values were. Let's go and fix that up now.

   static const char TEXT[] = "lorem ipsum";
   static uint32\_t CRC32 = 0x72d7748e;

Re-run "make check" and you'll see your test being rebuilt and rerun,
and this time passing.

   me@mybox> make check
   ...
   ../cunit/cunit.pl [...] --generate-wrapper crc32.testc
   gcc -c [...] -g -O2 .cunit-crc32.c
   gcc [...] -o unit [...] .cunit-crc32.o
   ...
   Running unit tests

       CUnit - A Unit testing framework for C - Version 2.1-0
       http://cunit.sourceforge.net/

   ...
   Suite: crc32
     Test: map ... passed


4.2 Adding A Test To A Suite
============================

Adding a new test to an existing test is easy: all you have to do is
add a new function to an existing C source file in the "cunit/"
directory. As an example, let's add a test for the "crc_iovec()"
function.

   me@mybox> vi cunit/crc32.testc
   ...

   static void test_iovec(void)  (a)
   {
       static const char TEXT1[] = "lorem";  (b)
       static const char TEXT2[] = " ipsum";
       static uint32_t CRC32 = 0x72d7748e;
       uint32_t c;  (c)
       struct iovec iov[2];

       memset(iov, 0, sizeof(iov));  (d)
       iov[0].iov_base = TEXT1;
       iov[0].iov_len = sizeof(TEXT1)-1;
       iov[1].iov_base = TEXT2;
       iov[1].iov_len = sizeof(TEXT2)-1;

       c = crc32_iovec(iov, 2);  (e)
       CU_ASSERT_EQUAL(c, CRC32);  (f)
   }

Here's an explanation of what all these bits are for.

1. Your new test function should look like this: it takes no
   arguments, returns void, and is named "test_whatever". It may be
   "static" or "extern", but I recommend "static". Functions with this
   signature are automatically discovered in the source code by the
   Cyrus unit test infrastructure, so all you have to do is write the
   function. Later, a CUnit test named "whatever" will be created
   automatically for your "test_whatever" function. Note that the
   opening curly brace must be on the next line or the unit test
   infrastructure will not find the function.

2. Here's a good place to define the test inputs and expected outputs.

3. Here's a good place for local variables you need during the test.

4. Here you set up the input conditions for the function under test.

5. Call the function under test with your known inputs, and capture
   the results in a local variable, here "c".

6. Compare the actual result in "c" with the expected result in
   "CRC32". The "CU_ASSERT_EQUAL()" macro checks that it's two
   arguments are equal (using an integer comparison), and if they're
   different it prints a message and records a failure. Note that
   unlike the libc "assert()" macro, control will continue even if the
   assert fails. The CUnit library provides a whole family of similar
   macros, see the online CUnit documentation for more details.

Now run "make check" and you'll see your test being built and run.

   me@mybox> make check
   ...
   ../cunit/cunit.pl [...] --generate-wrapper crc32.testc
   gcc -c [...] -g -O2 .cunit-crc32.c
   gcc [...] -o unit [...] .cunit-crc32.o
   ...
   Running unit tests


        CUnit - A Unit testing framework for C - Version 2.1-0
        http://cunit.sourceforge.net/

   ...
   Suite: crc32
     Test: map ... passed
     Test: iovec ... passed


4.3 Suite Setup And Teardown
============================

Sometimes the behaviour of the functions under test depend on external
influences such as environment variables, global variables, or the
presence of certain files.

These kinds of functions need special treatment to ensure that their
behaviour is locked down during the running of your tests. Otherwise,
all sorts of strange behaviour may confuse the results of the tests.
For example, a test might succeed the first time it's run in a given
directory and fail the next time. Or a test might succeed when run by
the author of the test but fail when run by another user.

CUnit provides a special arrangement which helps you in such cases:
the suite initialisation and cleanup functions. These are two
functions that you write and which live in the suite source. They are
called from CUnit respectively before any of the tests in the suite is
run, and after all tests from that suite are run.

Here's how to use them. The suite setup function should set up any
global state that the functions under test rely on, in such a way that
their state is predictable and always the same no matter who runs the
test or when or how many times. Similarly the suite teardown function
should clean up any state which might possibly interfere with other
test suites. Note that some suites will need a setup function but not
necessarily a teardown function.

Adding these functions is very easy: you just write functions of the
appropriate signature (names, arguments and return type) and the Cyrus
unit test infrastructure will automatically discover them and arrange
for them to be called. The functions should look like (actual example
taken from "cunit/mboxname.testc") this:

   static enum enum_value old_config_virtdomains;

   static int set_up(void)
   {
       old_config_virtdomains = config_virtdomains;
       config_virtdomains = IMAP_ENUM_VIRTDOMAINS_ON;
       return 0;
   }

   static int tear_down(void)
   {
       config_virtdomains = old_config_virtdomains;
       return 0;
   }

The functions should return 0 on success, and non-zero on error. They
must not call and "CU_*" functions or macros.

Good luck and good testing!
