:mod:`test_actions` - Testing Actions
================================================================================

.. module:: test_actions
  :synopsis: Testing Actions

.. currentmodule:: test_actions

This module is **not** part of the |project| binary distribution. That is the
productive bit running to produce models. It is only part of the source
distribution intended to help developing |project|. Basically it supports you
creating new actions along immediate tests, which will be stored as unit tests
and stay available to monitor later changes.

.. note::

   A couple of different paths will be mentioned in the following. To make
   things easier to tell apart, a prefix :file:`<SOURCE>` refers to the code
   repository, :file:`<BUILD>` to the build directory tree.

Inside the development environment, the module is only available to unit tests
in the :file:`<SOURCE>/actions/tests` directory. There is one special thing
about using it in your tests for an action, emerging from the way ``make`` runs
unit tests as set up via |cmake|. |python| modules are imported from the source
directory, here this is :file:`<SOURCE>/actions/tests`, while the tests run
inside :file:`<BUILD>/tests`, here this is :file:`<BUILD>/tests/actions`. When
|python| imports a module, its usually compiled into bytecode. This new file
would clutter up the source repository, it would always show up as untracked
file on ``git status``. To prevent this, tell |python| to stop producing
bytecode right at the beginning of your test-script:

.. literalinclude:: ../../../tests/doc/scripts/action_test.py
  :lines: 1-5
  :emphasize-lines: 5
  :linenos:

Line 5 does the trick. This needs to be set by you in every action unit test
file since |python| only recognises it **before** the module is imported.
Otherwise a module could disable bytecoding for all other modules loaded.

Testing actions, basically those are commands run in a shell, is very similar
across various actions. Additionally, there are some things that should be
tested for all actions like exit codes. That is why this module exists.

When developing an action, you will try it in the shell during the process. You
have to check that its doing what you intend, that it delivers the right
output, that it just behaves right on various kinds of input. This module
supports you by providing functionality to run scripts out of |python|. The
goal is to not trigger test runs manually in a shell but have a script that
does it for you. From there, you do not need to remember all the calls you
punched into the command line a year ago, when you come back to change
something, add new functionality, etc..

--------------------------------------------------------------------------------
Creating an Action Unit Test Script
--------------------------------------------------------------------------------
In the next couple of paragraphs, we will walk through setting up a new unit
test script for an imaginary action. We will continuously extend the file
started above, so keep an eye on line numbers. Lets just assume your action is
called ``do-awesome`` for the rest of this section.

The Test Script
--------------------------------------------------------------------------------
The script to supervise your action needs to be placed in
:file:`<SOURCE>/actions/tests` and follow the naming convention
:file:`test_action_<NAME>.py`, where :file:`<NAME>` is the name for your
action. So here we create a file :file:`test_action_do_awesome.py` (recognise
the underscore between ``do`` and ``awesome`` instead of a hyphen, that's
|pep8|_).

.. code-block:: console

   $ touch <SOURCE>/actions/tests/test_action_do_awesome.py
   $

As a starter, we disable bytecode compilation in the script:

.. literalinclude:: ../../../tests/doc/scripts/action_test.py
  :lines: 1-5
  :linenos:

|cmake| Integration
--------------------------------------------------------------------------------
As always, when introducing new material to |project|, it has to be announced
to the |cmake| build system. For action unit tests, fire up
:file:`<SOURCE>/actions/tests/CMakeLists.txt` in your favourite text editor and
add your new script:

.. code-block:: cmake
  :emphasize-lines: 3
  :linenos:

  set(ACTION_UNIT_TESTS
    test_action_help.py
    test_action_do_awesome.py
    test_actions.py # leave this as last item so it will be executed first!
  )

  promod3_unittest(MODULE actions SOURCES "${ACTION_UNIT_TESTS}" TARGET actions)

The important thing is to leave :file:`test_actions.py` as last item in the
list. This script contains the tests around the
:class:`test_actions.ActionTestCase` class, which is the foundation of the
tests for your action. If this class is broken, we are lost. Putting it as the
last element in the list, |cmake| will execute this script first, before any
other action test script is run.

Creating a Test Subclass
--------------------------------------------------------------------------------
:class:`test_actions.ActionTestCase` is sort of a template class for your
tests. By spawning off from this you inherit a bunch of useful methods for your
testing. To make it work, the childclass needs to be set up properly. But
first, :file:`test_actions.py` has to be loaded as a module:

.. literalinclude:: ../../../tests/doc/scripts/action_test.py
  :lines: 7
  :lineno-start: 6
  :linenos:

To showcase, the test cases, we explain how one would (and does) test the
``help`` action of ``pm``.
First, we create the childclass for the action.
Go for :class:`<NAME>ActionTests` as a naming scheme:

.. literalinclude:: ../../../tests/doc/scripts/action_test.py
  :lines: 9-12
  :lineno-start: 7
  :linenos:

Pay attention that in your own class, you must set :attr:`pm_action` to make
everything work. Also :meth:`__init__` needs certain parameters, as everything
is derived from the :class:`unittest.TestCase` class.

Must Have Tests
--------------------------------------------------------------------------------
What needs testing without exclusion are the exit codes of actions. Those
states will be placed in the userlevel documentation. This topic is already
covered in :class:`test_actions.ActionTestCase` by :meth:`~ActionTestCase.RunExitStatusTest`.
As an example, testing for ``$?=0`` could work like this:

.. literalinclude:: ../../../tests/doc/scripts/action_test.py
  :lines: 14-15
  :lineno-start: 11
  :linenos:

That will call the action stored in :attr:`pm_action` with the provided list of
parameters and check that ``0`` is returned on the command line.

In a more general way, you need to test that your action is working as
intended. Do not forget some negative testing, with the idea in mind what
happens if a user throws dirty input data in.

Making the Script Executable
--------------------------------------------------------------------------------
In |project|, unit tests are run via |ost_s|_'s :mod:`ost.testutils` and |python|'s
:class:`unittest.TestCase`. Those are called when the test module is executed
as a script:

.. literalinclude:: ../../../tests/doc/scripts/action_test.py
  :lines: 17-19
  :lineno-start: 13
  :linenos:

These three lines should be the same for all unit tests.

Running the Test Script
--------------------------------------------------------------------------------
Unit tests are executed via ``make check`` and so are |project| action tests.
But for every test script, we also provide a private ``make`` target, ending
with :file:`_run`. To solely run the tests for the awesome action, hit

.. code-block:: console

  $ make test_action_do_awesome.py_run

Output Of :class:`test_actions.ActionTestCase`
--------------------------------------------------------------------------------
When running the test script you will notice that its not really talkative.
Basically you do not see output to :file:`stdout`/ :file:`stderr` of your
action, while the test script fires it a couple of times. That is by design.
When running the full unit test suite, usually nobody wants to see the output
of **everything** tested and working. The interesting bits are where we fail.
But for developing a new application you certainly need all the output you can
get. For this, some functions in :class:`test_actions.ActionTestCase` have a
parameter :attr:`verbose`. That triggers specific functions to flush captured
output onto the command line. The idea is to turn it on for development, but
once done, disable it to keep output of unit tests low.

To get the test for exit code ``0`` talking to you, just do

.. literalinclude:: ../../../tests/doc/scripts/action_test_verbose.py
  :lines: 14-15
  :lineno-start: 11
  :linenos:

and

.. literalinclude:: ../../../tests/doc/scripts/action_test.py
  :lines: 14-15
  :lineno-start: 11
  :linenos:

keeps it silent (:attr:`verbose` is set to ``False`` by default). If enabled,
output will be separated into :file:`stdout` and :file:`stderr`:

.. code-block:: console

  $ make test_action_do_awesome.py_run
  <Lots of output from the build process>
  running checks test_action_do_awesome.py
  stdout of '<BUILD>/stage/bin/pm do-awesome'
  ------
  <Output to stdout>
  ------
  stderr of '<BUILD>/stage/bin/pm do-awesome'
  ------
  <Output to stderr>
  ------
  <More output from unit test runner>

--------------------------------------------------------------------------------
Unit Test Actions API
--------------------------------------------------------------------------------

.. autoclass:: test_actions.ActionTestCase
  :members:

..  LocalWords:  ActionTestCase currentmodule cmake bytecode emphasize sys py
..  LocalWords:  linenos pyc dont promod unittest childclass lineno init args
..  LocalWords:  ActionTests DoAwesomeActionTests kwargs attr meth TestCase
..  LocalWords:  userlevel RunExitStatusTest testExit ost testutils RunTests
..  LocalWords:  stdout stderr nobytecode testsetup actiontest os
..  LocalWords:  getcwd pardir builtin testoutput NORMALIZE WHITESPACE API
..  LocalWords:  autoclass
