| Makefile | ||
| README.md | ||
| runner.c | ||
A brütally simple test harness
Contents
Rationale
The purpose of this package is both practical and as a demonstration for just how simple a test harness can be, in order to encourage both more testing in C code as well as the use of simple tools. As such, the Roadrunner test runner consists of a test runner written in a single file of highly portable C, totaling 43 lines, and a simple, POSIX compatible Makefile which works with both GNU and BSD make.
Most "modern" programming languages have a built in testing framework and this is often a selling point, as C just leaves it to the programmer to pick from the thousands of different ways to not only build but also test their code, or roll their own. In this guy's opinion, 99.99% of all modern build systems are overkill and the vast majority actually introduce corner cases and bugs into the process, making things like cross compilation a serious PITA. If you stick with make, without any wrappers around it, you can pretty easily build even complex software in a way that doesn't require 15k line configure scripts, keeping up to date with changes in Python (WTF, why use Python to build C?) and that will be friendly to cross compilation. This simplistic test harness is a natural outgrowth of that mental model of building software (at least for software written in C or Fortran).
The original version of this project was written entirely in POSIX make, but due to the variations in POSIX shell behavior, especially in regards to the variety of ways that both the built in echo command and /bin/echo screw up sending terminal escape sequences, something more robust and portable was desirable. There were simply too many corner cases to be bothered to find them all.
Requirements
- a functional C compiler
- a POSIX compatible make program (GNU and BSD make are both fine)
- your code built as a static library
Usage
The best way to use this is to stuff it into a test subdirectory and call it via
make -C test, or invoke it from your project's Makefile as a sub-make. If you
are using something other than plain make to build your project then you probably
don't want this, but also, why? That said, let's not get sidetracked.
The example Makefile also goes into the test subdirectory, and you will want to add
the name of the library that you are testing on the appropriate line as well as the
names of all of the tests that you wish to run. Add those to the tests array.
tests += my_test
For each member of tests there should be a corresponding file <testname>.c which
will be compiled when you run make in this directory, along with the small helper
program runner.c. Make will then invoke ./runner with the list of tests as it's
arguments. The runner program will keep track of successes and failures, as well
as skipped tests, giving colored output and a nice summary at the end. If there are
no failures the program exits with a value of 0, otherwise the number of failures
is returned.
Your library will have to be built as a static library and statically linked to each
test program. All you have to do (after building the static library) is add it to
the $(LDLIBS) array in the Makefile.
Cross compilation
It would be entirely possible to extend this harness to work for cross compilation,
substituting $(HOSTCC) for $(CC) when building runner.c and calling qemu in
user mode to run the binary tests. However, this is out of scope for Roadrunner.
Writing Tests
Return values
The return values for each test shouuld follow these conventions.
- 0 for success
- 255 or -1 indicates this test was skipped
- any other value indicates failure
Skips
If your test requires elevated privileges, you can simply test for uid == 0 and then bail with code 255 if the runner is run by a different user. Similarly, if your test needs the network then you can test programmatically for a working connection and bail with 255 if it isn't available.
Don't use assert
The assert function from assert.h will not cause your test to fail, because it
simply aborts the program after printing a message to stderr. Instead, check for
proper results and return an exit code if the test fails. You can also print error
messages in your tests, which runner will capture, assume it means failure, and
print to stderr in the terminal session.