Unit testing revelations
The other day I experienced an unexpected light bulb moment concerning unit testing. Maybe this one is obvious to most of you, but I wish someone would have told me earlier. So here goes.
My biggest gripes with unit testing has been that I couldn’t get any satisfactory answers to these two questions:
- Why should I practice Test-First?
- How do you test the tests?
Concerning the first issue, well we discussed some papers that where trying to answer that question when I attended a software quality course at university. The gist of the results were: There is no statistically significant difference in code quality between Test-First and and Test-Later. (Sorry can’t find links to the papers atm. Holler if you want me to find them and I’ll do some more digging.)
The second issue is discussed often as well: If tests are code and code should be tested, doesn’t that lead to more and more tests? This is sometimes referred to as the stack overflow of unit testing.
The revelation I had was this: These two questions answer each other! What’s the easiest way to test a test? A broken implementation. Where do you get a broken implementation? Just use an incomplete implementation. If you practice Test-First all your implementations start off incomplete by definition. This means that each assertion in your test is guaranteed to fail at least once, giving you the confidence that the assertions actually perform significant work.
I am often surprised that assertions that I expect to fail actually go through just fine. This usually means one of two things: Either the functionality is already there by some fluke (cf. accidental correctness), or my test is incorrect. In either case I can then fine-tune and adapt the assertions to make sure that they fail and thus test some missing functionality.
Two conundrums solved.