Frogatto & Friends

Frogatto sprite

Frogatto & Friends is an action-adventure platformer game, starring a certain quixotic frog.
We're an open-source community project, and welcome contributions!
We also have a very flexible editor and engine you can use to make your own creations.

Unit Tests in Frogatto

November 5th, 2009 by Sirp

People who are familiar with any of my projects probably know that I am not always fond of using existing tools and libraries, often preferring to implement things myself, for one reason or another. This is particularly so if I feel a certain library imposes a significant intellectual burden in understanding its workings, while some hand-written code tailored to the purposes of the project could have much less intellectual burden.

I’ve used a few popular unit testing frameworks, such as cppunit, and found them generally cumbersome. I hate having to write an entire class to write a unit test, with a bunch of boilerplate code for each test. I must admit to not being entirely aware of all the benefits of using a unit test framework. I know that some frameworks give support for mocking and stubbing objects out and so forth, but, well, that all sounds like far more burden than it’s worth.

So, in Frogatto I have implemented a super-simple unit testing framework. If you want to write a unit test, then in any Frogatto source file, just include unit_test.hpp and add some code like this:

UNIT_TEST(simple_arithmetic) {
    int x = 4;
    x += 8;
    EXPECT_EQ(x, 12);

That’s all a test takes! Just write UNIT_TEST(name_of_test) and then a code block with your testing code. When you’re ready to see if your test ran as expected, you can use the macros provided, EXPECT_EQ/NE/GT/GE, etc etc to see if values came out as you expected them too.

Then, when you start Frogatto, all tests are run automatically, every time! The program will die with an error message if any tests fail. This forces and guarantees that people will not check in code with broken tests, because if they do, it’ll break the game horribly.

This is what unit tests should be like, in my view, very simple, easy to write, and easy to run. I’ve worked at too many places where some Agile fanatic tells everyone that they should write unit tests to test every single bit of code and are willing to discuss the joys of testing for hours on end with all the nuances of how many different code paths you should test, and then want you to learn some framework that they can understand very easily because they are interested in unit testing, but which everyone else who just wants to write code finds to be a burden to learn.

Sure, unit testing is a good idea, but to be workable it has to have minimum overhead. That is the aim of Frogatto’s framework.

The second part of Frogatto’s framework involves performance testing. If you’re writing a piece of code and you think its performance is going to be important, or if you have a piece of code and you know it’s taking up a significant amount of time, it’s nice to have an isolated test to verify that. So, the unit testing framework provides a benchmarking system.

Let’s see an example of how to add a benchmark. I wanted to benchmark how long it takes to query and convert a WML attribute into an integer, so here’s my benchmark:

BENCHMARK(wml_get_int) {
    wml::node_ptr node(new wml::node("a"));
    node->set_attr("abc", "47");
        wml::get_int(node, "abc");

This declares the benchmark with its name, and sets up some test data. Then the code inside BENCHMARK_LOOP is the part we actually want to benchmark. The benchmarking framework will run this part an appropriate number of times. So if the code takes a second to run, it’ll only be run a few times, and that averaged. If it takes a few nanoseconds to run it’ll be run billions of times, and that averaged.

How does a benchmark get run? Unlike unit tests, we don’t want to run benchmarks every time we run Frogatto, because that’d slow down startup and most people would just ignore the results. Instead, you run Frogatto with the –benchmarks argument. Frogatto will run all benchmarks with a report, and exit. You can also provide a comma-separated list to –benchmarks of the names of the benchmarks you want run. e.g. –benchmarks=formula_load to only run the ‘formula_load’ benchmark.

Output of the benchmarks framework looks like this:

BENCH wml_get_int: 1000000 iterations, 341ns/iteration; total, 341000000ns

This shows us that our wml_get_int benchmark ran for a million iterations, and each iteration took 341ns. So, it takes 341ns for us to get an integer from a WML node. Being able to work with and quickly intuit time periods is important in performance analysis. It takes 341ns to extract and parse an integer from a WML node. How significant is this? For one thing, we could find the wml::get_int() function and put a counter in to see how many times it is executed in a typical load of Frogatto. However, my guess is that even in a long game we’d parse less than 100,000 integers. This would make this function take up only around 34ms per run, which really isn’t very much.

On the other hand, 341ns seems like a lot to me to parse an integer. Of course, this code also extracts the integer, which requires a map lookup, and string construction, and so forth. If it turned out to be important to optimize this code, we probably could.


2 responses to “Unit Tests in Frogatto”

  1. Jetrel says:

    A fundamentally important point here, which gets glossed over, is that these tests get run on every launch, by default. This is crucial. Not doing this voids the whole point of unit tests.

    I’ve had people seriously suggest making unit tests as a separate program on some of my other projects, and this is a terrible idea because 1] it adds another chore you have to remember to do, and 2] because people will end up skipping it occasionally (if only by accident); it turns a 100% guarantee into a .. “maybe?”. Not only is it more work, but it’s less reliable.

    The one-and-only downside to having them compiled-in is a slight speed penalty at launch time. It’s a non-issue because it’s a tiny fraction of an already tiny launch time (3 seconds on my older hardware). And even if they were an issue, they’re easily turned off with a code tweak.

    Furthermore, having them in shipping code can help us track down issues on user’s computers. If you need unit tests to diagnose a problem (such as an unexpected hardware conflict – perhaps an endianness issue between PPC Macs and x86 machines, or a difference between ATI and NVIDIA cards), it’s the only feasible way to run them on a user’s computer. There’s no way you’re going to set a user up with a separate unit testing framework. Not a chance.

  2. Sirp says:

    Yes, the unit tests being run every time the program runs as part of the program is a critical part of our unit testing practice. This ensures that if you break something, you will know straight away.

    Having unit tests as separate programs (typically one program per test), is pretty common practice with most unit testing frameworks. However, it means that you have to take time to run these tests. You can have them run by an automated script, however then actually compiling and running them all tends to take a substantial amount of time

    Another cool thing about having Frogatto’s unit testing as part of the program is that we run unit tests after main program initialization. This has the small disadvantage that if there is a bug in initialization it won’t be caught in the unit test and that the program being in existence could ‘corrupt’ the unit tests.

    It has the huge advantage that the unit tests can then touch/prod/poke any aspect of the game. Want to load a level object in for your test and run some tests on it? No problem, just make the normal calls to load a level and start poking away.

    With a typical unit testing framework, if you want to load a level in you’ll have to work out which source files to link to your level unit test, maintain your makefile, which is going to be a pain, make sure if other people use different build systems they maintain the unit test in their build systems. Then you’re going to find level depends on a gazillion other classes, so you’re going to want to link them all in and now your unit test takes forever to compile, and then you discover that you don’t have all this initialization code that is needed so you either copy and paste it from the game or start making up mock objects and then you realize that this whole thing is almost as painful as this sentence is long and nobody in their right mind would do such a thing on a project they’re doing for fun…

    Having unit tests as part of the main executable and running on startup has been working really, really nicely for us so far. Easy adding of tests. Quick reporting of errors.