Update note
Based on the interesting feedback I got (can be seen on Tom’s ramblings), I realized this post probably needed some tweaking and scope precision. I did put them at the end.
What is the adequate objective for test coverage?
60%?
80%?
99%?
100%?
Like many others, I have often pondered this question, like many before me, and many after I suppose. Why aiming for 100%? The 80/20 law clearly applies to test coverage: to try to cover every corner cases that lie in the code is going to require a significant investment in time and brain cells. Plus integration point can never really be properly covered.
On the other hand, having 100% coverage provides huge benefits:
- Every single line of code is constantly tested
- Trust in code is high
- Any uncovered line is a regression
What happens if the target is, 80%:
- Significant part of the code is never tested
- Trust in code is moderate and can degrade
- 20% uncovered line codes is significant, 2000 lines for a 10K code base. That means full namespaces can hide in there.
For me, there is no question 100% coverage is the only worthy objective, do not settle for less.
Yes there are exceptions, usually at integration points. Mocks are not a real solution either, they can help you increase your coverage but not by that much. The pragmatic solution is to wrap them into isolated modules (jars/assemblies). Think hexagonal architecture there. You will have specific coverage target for those, you also need to make sure that no other code creeps in and finally, understands those are weak points in your design.
While I am working on nFluent, I constantly make sure unit tests exert every single line of code of the library. It also means that I help contributors reach the target. It is not that difficult, especially if you do TDD!
There is one simple golden rule: to reach and maintain 100% coverage, you do not need to add tests, you have to remove not covered lines!
This is important, so let me restate that: a line of code that is not covered is not maintainable, must be seen as not working and must be removed!
Think about it:
- The fact that no automated test exists means that the behavior can be silently changed, not just the implementation!
- Any newcomer, including your proverbial future self, will have to guess the expected behavior !
- What will happen if the code get executed some day in production?
- If you are doing TDD you can safely assume the code is useless!
So, when you discover not covered lines, refactor to remove them or to make sure they are exerted. But do not add tests for the sake of coverage.
Having 100% coverage does not mean the code is bug free
Tom’s comments implied that I was somehow trying to promote this idea. 100% coverage is no bug free proof at all, and do not imply this at all. Quality and relevance of your tests are essential attributes; that is exactly why I promote removing non tested lines. Any specially crafted test will not be driven by an actual need and would be artificial. The net result would be a less agile code base.
On the other hand, if you have 100% coverage and you discover some reproducible bug, either by manual testing or in production, you should be able to add an automated test to prevent any re occurrence.
When coverage is insufficient, there is a high probability that you will not be able to add this test, keeping the door open for future regression!
If you want to build trust based on coverage metrics, you need to look into branch coverage and property based testing at the very least. But I do not think this is a smart objective.
Note
- This post focuses on new code! For legacy code, the approach should be to add tests before anything else, and never remove working code 🙂
Many Cores 
This is a troll, right?
LikeLike
Of course not! But I understand this is troll bait 🙂
LikeLike
Well, IRL working code providing business value beats coverage/good design by miles… Also, there is a lot of added business value in legacy code, so don’t delete it, but refactor it (see fowler’s book).
Another thing people sometimes do is throw away test code, as it is holding them back (in fact, lots of tests are considered legacy), so I wouldn’t approach this black & white.
While I love good, well written code (preferably developed with BDD or similar), that is not always the case. Just deleting it would be …
To quote someone replying on twitter: “rm -rf /”; 100% coverage done, now let’s grab a beer…
LikeLike
Thanks for your feedback. Working software over anything else, yes :-), but working software without adequate automated tests getting legacy by the minute..
Definitely, I did not have legacy code in mind, so I will edit the post to clarify that.
Adding test before refactoring is a good approach, but 100 % coverage is not only overkill, but counterproductive.
When tests are thrown away, someone has failed: either the test was not relevant (probably too close to the implementation) or the maintainer was lazy…
My 2 cts
LikeLike
There’s different concepts of testing; TDD usually tests the implementation, not the behavior AFAIK… So when you change your implementation, you need to change your tests. BDD removes some of those issues.
Using TDD for problem exploration is usually a major PITA; Peter Norvig’s Sudoku [1] is a commonly known example…
When you think about the problem first, manage to understand it and keep the solution simple, a few integration tests are all you need…
Test-driven design can help you to create an implementation, not to find a solution. Proper modelling is necessary and way too often people forget that.
Also, if you want to verify whether your code works: 100% test coverage actually proves nothing; take the simple example if forgetting to check for a division by zero somewhere.
If you really want to verify how good your solution matches the problem space, I would suggest you to use property based testing; that will discover way more edge- and side-cases then you could ever imagine….
[1] http://devgrind.com/2007/04/25/how-to-not-solve-a-sudoku/
LikeLike
FYI, I blogged about this here: http://tojans.me/blog/2014/02/24/trolling-the-100-percent-tdd-coverage-approach-or-not/
LikeLike
Hey Tom, I completely disagree with the statement: “TDD usually tests the implementation, not the behavior AFAIK”. On that topic, and before we re-read all Kent Beck’s “Test-Driven Development”, I strongly recommend the viewing of the excellent session by Ian Cooper: “TDD, where did it all go wrong” (http://vimeo.com/68375232) .
Because TDD is really about testing behaviors, not testing classes or implementation details.
LikeLike
Yep, I agree.
Naïve TDD is no magical wand that can guide to build up adequate algorithm (try anything requiring randomization for example).
For me, Test Driven Development encompasses any Test first approach, and it encompasses BDD, property based testing, …: all those methods/approaches work by defining (relevant) expectations and making sure the business logic matches them.
It definitely does not limit to: here is my class contract, lets secure it with some tests.
Yes, you can say that 100 % coverage proves nothing. Beyond problem related to edge cases, such as null div, you may not have relevant assertions (or none at all for that matter).
Yes, making sure your tests coverage the full scope of input value is important. I would also love to talk about path/branch coverage, instead of the simple line coverage.
But on the other hand: 0% coverage proves a lot! Even 80% coverage proves a lot.
Let’s say my post aims at the less mature/experienced devevelopper that uses statement as yours to justify insufficient test base.
LikeLike
I really do think coverage percentage gives you a false idea about what works in your system, because you only test what you designed, not what it will be used for, but OTOH we probably have some common ground as well, so let’s call it a day here.
Thank you for your post & comments!
LikeLike