There are two ways to execute a testsuite. The most common way is when there is existing support in the Makefile. This support consists of a check target. The other way is to execute the runtest program directly. To run runtest directcly from the command line requires either all the correct options, or the Local Config File must be setup correctly.
To run tests from an existing collection, first use configure as usual to set up the build directory. Then try typing:
make check
If the check target exists, it usually saves you some trouble. For instance, it can set up any auxiliary programs or other files needed by the tests. The most common file the check builds is the site.exp. The site.exp file contains various variables that DejaGnu used to dertermine the configuration of the program being tested. This is mostly for supporting remote testing.
The check target is supported by GNU Automake. To have DejaGnu support added to your generated Makefile.in, just add the keyword dejagnu to the AUTOMAKE_OPTIONS variable in your Makefile.am file.
Once you have run make check to build any auxiliary files, you can invoke the test driver runtest directly to repeat the tests. You will also have to execute runtest directly for test collections with no check target in the Makefile.
runtest is the executable test driver for DejaGnu. You can specify two kinds of things on the runtest command line: command line options, and Tcl variables for the test scripts. The options are listed alphabetically below.
runtest returns an exit code of 1 if any test has an unexpected result; otherwise (if all tests pass or fail as expected) it returns 0 as the exit code.
runtest flags the outcome of each test as one of these cases. A POSIX Conforming Test Framework for a discussion of how POSIX specifies the meanings of these cases.
The most desirable outcome: the test succeeded, and was expected to succeed.
A pleasant kind of failure: a test was expected to fail, but succeeded. This may indicate progress; inspect the test case to determine whether you should amend it to stop expecting failure.
A test failed, although it was expected to succeed. This may indicate regress; inspect the test case and the failing software to ocate the bug.
A test failed, but it was expected to fail. This result indicates no change in a known bug. If a test fails because the operating system where the test runs lacks some facility required by the test, the outcome is UNSUPPORTED instead.
Output from a test requires manual inspection; the testsuite could not automatically determine the outcome. For example, your tests can report this outcome is when a test does not complete as expected.
A test case is not yet complete, and in particular cannot yet produce a PASS or FAIL. You can also use this outcome in dummy ``tests'' that note explicitly the absence of a real test case for a particular property.
A test depends on a conditionally available feature that does not exist (in the configured testing environment). For example, you can use this outcome to report on a test case that does not work on a particular target because its operating system support does not include a required subroutine.
runtest may also display the following messages:
Indicates a major problem (detected by the test case itself) in running the test. This is usually an unrecoverable error, such as a missing file or loss of communication to the target. (POSIX testsuites should not emit this message; use UNSUPPORTED, UNTESTED, or UNRESOLVED instead, as appropriate.)
Indicates a possible problem in running the test. Usually warnings correspond to recoverable errors, or display an important message about the following tests.
An informational message about the test case.
This is the full set of command line options that runtest recognizes. Arguments may be abbreviated to the shortest unique string.
--all (-a)Display all test output. By default, runtest shows only the output of tests that produce unexpected results; that is, tests with status FAIL (unexpected failure), XPASS (unexpected success), or ERROR (a severe error in the test case itself). Specify --all to see output for tests with status PASS (success, as expected) XFAIL (failure, as expected), or WARNING (minor error in the test case itself).
--build [string]string is a full configuration ``triple'' name as used by configure. This is the type of machine DejaGnu and the tools to be tested are built on. For a normal cross this is the same as the host, but for a canadian cross, they are seperate.
--host [string]string is a full configuration
``triple'' name as used by configure. Use this
option to override the default string recorded by your
configuration's choice of host. This choice does not change how
anything is actually configured unless --build is also specified; it
affects only DejaGnu procedures that compare the
host string with particular values. The procedures
ishost, istarget,
isnative, and setupxfail}
are affected by --host. In this usage,
host refers to the machine that the tests are to
be run on, which may not be the same as the
build machine. If --build
is also specified, then --host refers to the
machine that the tests wil, be run on, not the machine DejaGnu is run
on.
--host_board [name]The host board to use.
--target [string]Use this option to override the default setting (running native tests). string is a full configuration ``triple'' name of the form cpu-vendor-os as used by configure. This option changes the configuration runtest uses for the default tool names, and other setup information.
--debug (-de)Turns on the expect internal debugging output. Debugging output is displayed as part of the runtest output, and logg CLASS="sect2" >4.2. Runtest
runtest is the executable test driver for DejaGnu. You can specify two kinds of things on the runtest command line: command line options, and Tcl variables for the test scripts. The options are listed alphabetically below.
runtest returns an exit code of 1 if any test has an unexpected result; otherwise (if all tests pass or fail as expected) it returns 0 as the exit code.
runtest flags the outcome of each test as one of these cases. A POSIX Conforming Test Framework for a discussion of how POSIX specifies the meanings of these cases.
The most desirable outcome: the test succeeded, and was expected to succeed.
A pleasant kind of failure: a test was expected to fail, but succeeded. This may indicate progress; inspect the test case to determine whether you should amend it to stop expecting failure.
A test failed, although it was expected to succeed. This may indicate regress; inspect the test case and the failing software to ocate the bug.
A test failed, but it was expected to fail. This result indicates no change in a known bug. If a test fails because the operating system where the test runs lacks some facility required by the test, the outcome is UNSUPPORTED instead.
Output from a test requires manual inspection; the testsuite could not automatically determine the outcome. For example, your tests can report this outcome is when a test does not complete as expected.
A test case is not yet complete, and in particular cannot yet produce a PASS or FAIL. You can also use this outcome in dummy ``tests'' that note explicitly the absence of a real test case for a particular property.
A test depends on a conditionally available feature that does not exist (in the configured testing environment). For example, you can use this outcome to report on a test case that does not work on a particular target because its operating system support does not include a required subroutine.
runtest may also display the following messages:
Indicates a major problem (detected by the test case itself) in running the test. This is usually an unrecoverable error, such as a missing file or loss of communication to the target. (POSIX testsuites should not emit this message; use UNSUPPORTED, UNTESTED, or UNRESOLVED instead, as appropriate.)
Indicates a possible problem in running the test. Usually warnings correspond to recoverable errors, or display an important message about the following tests.
An informational message about the test case.
This is the full set of command line options that runtest recognizes. Arguments may be abbreviated to the shortest unique string.
--all (-a)Display all test output. By default, runtest shows only the output of tests that produce unexpected results; that is, tests with status FAIL (unexpected failure), XPASS (unexpected success), or ERROR (a severe error in the test case itself). Specify --all to see output for tests with status PASS (success, as expected) XFAIL (failure, as expected), or WARNING (minor error in the test case itself).
--build [string]string is a full configuration ``triple'' name as used by configure. This is the type of machine DejaGnu and the tools to be tested are built on. For a normal cross this is the same as the host, but for a canadian cross, they are seperate.
--host [string]string is a full configuration
``triple'' name as used by configure. Use this
option to override the default string recorded by your
configuration's choice of host. This choice does not change how
anything is actually configured unless --build is also specified; it
affects only DejaGnu procedures that compare the
host string with particular values. The procedures
ishost, istarget,
isnative, and setupxfail}
are affected by --host. In this usage,
host refers to the machine that the tests are to
be run on, which may not be the same as the
build machine. If --build
is also specified, then --host refers to the
machine that the tests wil, be run on, not the machine DejaGnu is run
on.
--host_board [name]The host board to use.
--target [string]Use this option to override the default setting (running native tests). string is a full configuration ``triple'' name of the form cpu-vendor-os as used by configure. This option changes the configuration runtest uses for the default tool names, and other setup information.
--debug (-de)Turns on the expect internal debugging output. Debugging output is displayed as part of the runtest output, and logg CLASS="sect2" >4.2. Runtest
runtest is the executable test driver for DejaGnu. You can specify two kinds of things on the runtest command line: command line options, and Tcl variables for the test scripts. The options are listed alphabetically below.
runtest returns an exit code of 1 if any test has an unexpected result; otherwise (if all tests pass or fail as expected) it returns 0 as the exit code.
runtest flags the outcome of each test as one of these cases. A POSIX Conforming Test Framework for a discussion of how POSIX specifies the meanings of these cases.
The most desirable outcome: the test succeeded, and was expected to succeed.
A pleasant kind of failure: a test was expected to fail, but succeeded. This may indicate progress; inspect the test case to determine whether you should amend it to stop expecting failure.
A test failed, although it was expected to succeed. This may indicate regress; inspect the test case and the failing software to ocate the bug.
A test failed, but it was expected to fail. This result indicates no change in a known bug. If a test fails because the operating system where the test runs lacks some facility required by the test, the outcome is UNSUPPORTED instead.
Output from a test requires manual inspection; the testsuite could not automatically determine the outcome. For example, your tests can report this outcome is when a test does not complete as expected.
A test case is not yet complete, and in particular cannot yet produce a PASS or FAIL. You can also use this outcome in dummy ``tests'' that note explicitly the absence of a real test case for a particular property.
A test depends on a conditionally available feature that does not exist (in the configured testing environment). For example, you can use this outcome to report on a test case that does not work on a particular target because its operating system support does not include a required subroutine.
runtest may also display the following messages:
Indicates a major problem (detected by the test case itself) in running the test. This is usually an unrecoverable error, such as a missing file or loss of communication to the target. (POSIX testsuites should not emit this message; use UNSUPPORTED, UNTESTED, or UNRESOLVED instead, as appropriate.)
Indicates a possible problem in running the test. Usually warnings correspond to recoverable errors, or display an important message about the following tests.
An informational message about the test case.
This is the full set of command line options that runtest recognizes. Arguments may be abbreviated to the shortest unique string.
--all (-a)Display all test output. By default, runtest shows only the output of tests that produce unexpected results; that is, tests with status FAIL (unexpected failure), XPASS (unexpected success), or ERROR (a severe error in the test case itself). Specify --all to see output for tests with status PASS (success, as expected) XFAIL (failure, as expected), or WARNING (minor error in the test case itself).
--build [string]string is a full configuration ``triple'' name as used by configure. This is the type of machine DejaGnu and the tools to be tested are built on. For a normal cross this is the same as the host, but for a canadian cross, they are seperate.
--host [string]string is a full configuration
``triple'' name as used by configure. Use this
option to override the default string recorded by your
configuration's choice of host. This choice does not change how
anything is actually configured unless --build is also specified; it
affects only DejaGnu procedures that compare the
host string with particular values. The procedures
ishost, istarget,
isnative, and setupxfail}
are affected by --host. In this usage,
host refers to the machine that the tests are to
be run on, which may not be the same as the
build machine. If --build
is also specified, then --host refers to the
machine that the tests wil, be run on, not the machine DejaGnu is run
on.
--host_board [name]The host board to use.
--target [string]Use this option to override the default setting (running native tests). string is a full configuration ``triple'' name of the form cpu-vendor-os as used by configure. This option changes the configuration runtest uses for the default tool names, and other setup information.
--debug (-de)Turns on the expect internal debugging output. Debugging output is displayed as part of the runtest output, and logg CLASS="sect2" >4.2. Runtest
runtest is the executable test driver for DejaGnu. You can specify two kinds of things on the runtest command line: command line options, and Tcl variables for the test scripts. The options are listed alphabetically below.
runtest returns an exit code of 1 if any test has an unexpected result; otherwise (if all tests pass or fail as expected) it returns 0 as the exit code.
runtest flags the outcome of each test as one of these cases. A POSIX Conforming Test Framework for a discussion of how POSIX specifies the meanings of these cases.
The most desirable outcome: the test succeeded, and was expected to succeed.
A pleasant kind of failure: a test was expected to fail, but succeeded. This may indicate progress; inspect the test case to determine whether you should amend it to stop expecting failure.
A test failed, although it was expected to succeed. This may indicate regress; inspect the test case and the failing software to ocate the bug.
A test failed, but it was expected to fail. This result indicates no change in a known bug. If a test fails because the operating system where the test runs lacks some facility required by the test, the outcome is UNSUPPORTED instead.
Output from a test requires manual inspection; the testsuite could not automatically determine the outcome. For example, your tests can report this outcome is when a test does not complete as expected.
A test case is not yet complete, and in particular cannot yet produce a PASS or FAIL. You can also use this outcome in dummy ``tests'' that note explicitly the absence of a real test case for a particular property.
A test depends on a conditionally available feature that does not exist (in the configured testing environment). For example, you can use this outcome to report on a test case that does not work on a particular target because its operating system support does not include a required subroutine.
runtest may also display the following messages:
Indicates a major problem (detected by the test case itself) in running the test. This is usually an unrecoverable error, such as a missing file or loss of communication to the target. (POSIX testsuites should not emit this message; use UNSUPPORTED, UNTESTED, or UNRESOLVED instead, as appropriate.)
Indicates a possible problem in running the test. Usually warnings correspond to recoverable errors, or display an important message about the following tests.
An informational message about the test case.
This is the full set of command line options that runtest recognizes. Arguments may be abbreviated to the shortest unique string.
--all (-a)Display all test output. By default, runtest shows only the output of tests that produce unexpected results; that is, tests with status FAIL (unexpected failure), XPASS (unexpected success), or ERROR (a severe error in the test case itself). Specify --all to see output for tests with status PASS (success, as expected) XFAIL (failure, as expected), or WARNING (minor error in the test case itself).
--build [string]string is a full configuration ``triple'' name as used by configure. This is the type of machine DejaGnu and the tools to be tested are built on. For a normal cross this is the same as the host, but for a canadian cross, they are seperate.
--host [string]string is a full configuration
``triple'' name as used by configure. Use this
option to override the default string recorded by your
configuration's choice of host. This choice does not change how
anything is actually configured unless --build is also specified; it
affects only DejaGnu procedures that compare the
host string with particular values. The procedures
ishost, istarget,
isnative, and setupxfail}
are affected by --host. In this usage,
host refers to the machine that the tests are to
be run on, which may not be the same as the
build machine. If --build
is also specified, then --host refers to the
machine that the tests wil, be run on, not the machine DejaGnu is run
on.
--host_board [name]The host board to use.
--target [string]Use this option to override the default setting (running native tests). string is a full configuration ``triple'' name of the form cpu-vendor-os as used by configure. This option changes the configuration runtest uses for the default tool names, and other setup information.
--debug (-de)Turns on the expect internal debugging output. Debugging output is displayed as part of the