87 lines
4.4 KiB
Markdown
87 lines
4.4 KiB
Markdown
### A brütally simple test harness
|
|
|
|
Contents
|
|
========
|
|
- [Rationale](#rationale)
|
|
- [Requirements](#requirements)
|
|
- [Usage](#usage)
|
|
- [Writing Tests](#writing_tests)
|
|
|
|
## Rationale
|
|
The purpose of this package is both practical and as a demonstration for just how
|
|
simple a test harness can be, in order to encourage both more testing in C code
|
|
as well as the use of simple tools. As such, the **Roadrunner** test runner consists
|
|
of a test runner written in a single file of highly portable C, totaling 43 lines,
|
|
and a simple, POSIX compatible Makefile which works with both GNU and BSD make.
|
|
|
|
Most "modern" programming languages have a built in testing framework and this is
|
|
often a selling point, as **C** just leaves it to the programmer to pick from the
|
|
thousands of different ways to not only build but also test their code, or roll
|
|
their own. In this guy's opinion, 99.99% of all modern build systems are overkill
|
|
and the vast majority actually introduce corner cases and bugs into the process,
|
|
making things like cross compilation a *serious* PITA. If you stick with **make**,
|
|
without any wrappers around it, you can pretty easily build even complex software
|
|
in a way that doesn't require 15k line configure scripts, keeping up to date with
|
|
changes in Python (WTF, why use *Python* to build *C*?) and that will be friendly
|
|
to cross compilation. This simplistic test harness is a natural outgrowth of that
|
|
mental model of building software (at least for software written in C or Fortran).
|
|
|
|
The original version of this project was written entirely in POSIX make, but due to
|
|
the variations in POSIX shell behavior, especially in regards to the variety of ways
|
|
that both the built in *echo* command and */bin/echo* screw up sending terminal escape
|
|
sequences, something more robust and portable was desirable. There were simply too
|
|
many corner cases to be bothered to find them all.
|
|
|
|
## Requirements
|
|
- a functional C compiler
|
|
- a **POSIX** compatible *make* program (GNU and BSD make are both fine)
|
|
- your code built as a static library
|
|
|
|
## Usage
|
|
The best way to use this is to stuff it into a `test` subdirectory and call it via
|
|
`make -C test`, or invoke it from your project's `Makefile` as a sub-make. If you
|
|
are using something other than plain *make* to build your project then you probably
|
|
don't want this, but also, why? That said, let's not get sidetracked.
|
|
|
|
The example Makefile also goes into the `test` subdirectory, and you will want to add
|
|
the name of the library that you are testing on the appropriate line as well as the
|
|
names of all of the tests that you wish to run. Add those to the `tests` array.
|
|
```Makefile
|
|
tests += my_test
|
|
```
|
|
For each member of `tests` there should be a corresponding file `<testname>.c` which
|
|
will be compiled when you run `make` in this directory, along with the small helper
|
|
program `runner.c`. Make will then invoke `./runner` with the list of tests as it's
|
|
arguments. The *runner* program will keep track of successes and failures, as well
|
|
as skipped tests, giving colored output and a nice summary at the end. If there are
|
|
no failures the program exits with a value of **0**, otherwise the number of failures
|
|
is returned.
|
|
|
|
Your library will have to be built as a static library and statically linked to each
|
|
test program. All you have to do (after building the static library) is add it to
|
|
the `$(LDLIBS)` array in the Makefile.
|
|
|
|
### Cross compilation
|
|
It would be entirely *possible* to extend this harness to work for cross compilation,
|
|
substituting `$(HOSTCC)` for `$(CC)` when building *runner.c* and calling qemu in
|
|
user mode to run the binary tests. However, this is out of scope for **Roadrunner**.
|
|
|
|
## Writing Tests
|
|
### Return values
|
|
The return values for each test shouuld follow these conventions.
|
|
- 0 for success
|
|
- 255 or -1 indicates this test was skipped
|
|
- any other value indicates failure
|
|
|
|
### Skips
|
|
If your test requires elevated privileges, you can simply test for uid == 0 and then
|
|
bail with code 255 if the runner is run by a different user. Similarly, if your test
|
|
needs the network then you can test programmatically for a working connection and bail
|
|
with 255 if it isn't available.
|
|
|
|
### Don't use *assert*
|
|
The `assert` function from `assert.h` will not cause your test to fail, because it
|
|
simply aborts the program after printing a message to stderr. Instead, check for
|
|
proper results and return an exit code if the test fails. You can also print error
|
|
messages in your tests, which *runner* will capture, assume it means failure, and
|
|
print to stderr in the terminal session.
|