I was listening to the Stackoverflow podcast number 41 today. One of the issues discussed was unit testing and when is it beneficial to do unit testing. I agreed with some of the points but not all of them. The point I especially found interesting was the “eating your own dogfood” concept where writing tests forces you to use the interfaces you created. I also never considered the perspective on tests serving as documentation or a spec on how the system should work.
There are lots of good reasons to do unit testing but there are a few issues I have run across with it that go beyond the obvious ones of time and effort.
- Bugs in the tests – Let’s say a hypothetical programmer creates a bug in every 100 lines of code he writes. Logically, that would mean he would be creating a bug in every 100 lines of test code also. Unit tests do provide a double check to validate your code but a bug in the unit test could give you a false positive that your code works. A programmer who writes bad code will also write bad unit tests so even additional unit test coverage will result in poor code quality.
- Tests are Bias – Unit tests are written by the programmer who wrote the code. If the programmer misunderstands how the code works or should be used (even if they wrote it), then it is likely that the test will also be built around this misunderstanding.
- Test Failures due to design changes – Most times when I see unit tests fail it is because the code that was being tested was intentionally changed. Often the test is the last piece of code to be updated (right or wrong) to the new functionality of the rest of the system.
- Code that Must be Maintained – Tests are additional code that developers need to maintain. I try to write as little code as possible to keep it clear and simple which makes it hard to justify writing a lot of extra code.
Writing tests requires effort. They take time to develop and maintain but they do have a payoff. I try to find a balance between the “100% test coverage” people and the “we don’t need any tests” people. The following is a list of criteria I use as a general guide to what I feel is most important to test. I choose the stuff that is most important areas to test.
- Libraries – I always make sure I write tests for common libraries or utilities. These are generally easiest to test and are designed to be used in many places. Testing these has the biggest payoff.
- APIs – APIs are a contract between systems. These are logical choices for tests as they should not be changing often and have strict rules.
- Business Logic/Rules – Anytime there are business rules or logic that perform some defined function where there are clear inputs and outputs, these should be tested.
Libraries are usually number one on my testing priority. I usually aim for 90% or more coverage. GUI testing I have found to be of little benefit since the effort is so high and you end up with very little reward. Glue code or anything that ties different components together is hard to test and are more easily tested with basic smoke tests on the complete product.