7 Comments How do you know your tests are good? - 01/26/09
I keep asking myself, how do you know your test suite is healthy? I mean:
- How do you know your tests are testing what they should be testing?
- How do you know your tests aren’t testing what they shouldn’t be testing?
You could use code coverage as a metric. But that’s not going to get you all the way. Code coverage is especially handy for people that are doing the testing, but not the TDD-way. Just run TestDriven.net with NCover, and you’ll immediately know what you forgot to test.
But still, code coverage isn’t smart enough to know if what you’re testing is actually meaningfull. Let me simplify this with an example.
We have a very simple method, that calculates the age of a person, based on his/her birthday, and today’s date (I’m not going to involve correct calculation, nor validation, since it’s not the scope of this example):
1: public class AgeCalculator
2: {
3: /// <summary>
4: /// Calculates the age based on a given birthday and the current date
5: /// </summary>
6: /// <param name="birthday">Birthday to base age-calculation on</param>
7: /// <returns></returns>
8: public int CalculateAge(DateTime birthday)
9: {
10: return DateTime.Now.Year - birthday.Year;
11: }
12: }
Then we have useless test number one:
1: [Test]
2: public void TestThatCoversAgeCalculationButIsNotTestingIt()
3: {
4: AgeCalculator calculator = new AgeCalculator();
5: int age = calculator.CalculateAge(DateTime.Now.AddYears(-5));
6: CanDriveSpecification canDriveSpecification = new CanDriveSpecification();
7: Assert.IsFalse(canDriveSpecification.IsSatisfiedBy(age));
8: }
Here’s it’s coverage:
NCover is telling me that this test (I used TestDriven.Net to run it with coverage) has 100% coverage. But taking a second look at the test, you’ll see (if you haven’t already) that this test is really testing something else, it’s testing if someone with age 5 may drive or not, which doesn’t ensure that our age calculation is correct. If that’s the only test you have covering your age calculation, your tests havn’t functionally covered it.
The second useless test:
1: [Test]
2: public void TestThatCoversAgeCalculationsButTestsItWrong()
3: {
4: AgeCalculator calculator = new AgeCalculator();
5: DateTime birthDay = new DateTime(1984, 10, 31);
6: Assert.AreEqual(24, calculator.CalculateAge(birthDay));
7: }
This is a test dedicated only to the age calculation, so we’re getting closer. Still, this test is just wrong! This will run today, tomorrow, even next month, but forget about it in november! And I’m still getting my 100% coverage.
The two tests above will run. They will even provide you with a 100% code coverage for the CalculateAge method, but they are not at all representative. So code coverage is not even close to measuring the quality of our test suite.
How can you ensure the tests are good?
You will always need to keep a human eye on the effectiveness of your tests. It’s the developer’s responsibility to be testing something usefull.
That brings me to the following conclusion: we can distinguish two different kind of developers that write tests:
1) developers that write tests as part of their duty (because management says so)
2) developers that write tests to ensure quality of their work, even without management asking
You’ll be thinking, that the code I showed you above is just plain stupidity, and can’t be written by any developer. Well, I’m sorry to dissapoint you. Maybe in this example I exagerated a bit, but I’m sorry to say, that this type of test-code is actually very common. Developers that don’t care about testing, write this type of code, and not because they’re too dumb to see what’s wrong with it, but just because they don’t care about the tests.
That’s what leads me to test-driven development.
Test-driven development is a technique that requires developers to first write their tests, and then write code to make the tests pass. This all is done is an iteration known as Red-Green-Refactor.
1) Think => how should you write your test?
2) Red => write your test, and see it fail (since there is no implementation code)
3) Green => write production code to make your test run
4) Refactor => refactor both test- and production code
5) Repeat => Do it all again
James Shore does a great job covering this topic in more depth, so don’t forget to take a look!
Revisiting the AgeCalculator example
How could I best write my test?
I need to calculate the age of a person, based on his/her birthday, and the current date. So it would be nice to call a method CalculateAge on a calculation class named AgeCalculator.
Red
I took a screenshot, just to show you that this won’t even build (since this class and method don’t even exist)!
Green
Create an interface IAgeCalculator with a CalculateAge method. This method should accept a DateTime parameter and return an int. Create a class AgeCalculator that implements the interface.
1: public interface IAgeCalculator
2: {
3: /// <summary>
4: /// Calculates the age based on a given birthday and the current date
5: /// </summary>
6: /// <param name="birthday">Birthday to base age-calculation on</param>
7: /// <returns></returns>
8: int CalculateAge(DateTime birthday);
9: }
10:
11: public class AgeCalculator : IAgeCalculator
12: {
13: /// <summary>
14: /// Calculates the age based on a given birthday and the current date
15: /// </summary>
16: /// <param name="birthday">Birthday to base age-calculation on</param>
17: /// <returns></returns>
18: public int CalculateAge(DateTime birthday)
19: {
20: return DateTime.Now.Year - birthday.Year;
21: }
22: }
Refactor
Maybe in this simple case, you wouldn’t want to refactor your code. But imagine, your AgeCalculator should be able to block invalid birthdays. An invalid birthday could be a date in the future. First you’d write a test that should throw an exception when you’re passing a future date. Then adjust the AgeCalculator class. We’re instantiating the AgeCalculator two times now. You could now choose to instantiate your AgeCalculator class in your test-setup. Then, we’re asked to add a new feature, in which you should be able to pass a date to calculate the age against (in stead of DateTime.Now).
We implement it the RGR-way and it looks as follows:
1: /// <summary>
2: /// Calculates the age given a birthday and a date to calculate the actual age against
3: /// </summary>
4: /// <param name="birthDay">Birthday to base age-calculation on</param>
5: /// <param name="compareDate">Date to calculate age against</param>
6: /// <returns></returns>
7: public int CalculateAge(DateTime birthDay, DateTime compareDate)
8: {
9: return compareDate.Year - birthDay.Year;
10: }
Then you’ll notice you could adjust the CalculateAge method that only accepts the birthdate. You could have that method call the CalculateAge(DateTime birthDate, DateTime compareDate) and pass it DateTime.Now, to avoid code duplication (remember the DRY principle?). So, as you can already see, the refactoring step never ends. We’ve already added 2 new features, and we’re still refactoring feature number one 
.
Roundup
I’m not at all a TDD-expert (I wish!), this post only reflects how I got interested in TDD. Right now, I try to practice it whenever I can, but I have a very long way to go. I’m currently also reading the TDD-starter book: Test-driven development by example.
I think that in this little example, I proved that TDD helps to keep your tests focused, clear and meaningfull. It also follows the YAGNI principle, since you’ll never be writing code you havn’t written a test for, thus you won’t be writing code you don’t need, unless you’re writing tests you don’t need, and then you’re doing TDD all wrong .
TDD can only be done by developers that actually have interest in building qualitative software (if you don’t care, I don’t think you can find the discipline to do it). Just imagine the first useless test I wrote above failing because the calculation was changed and the developer introduced a bug. That will keep the team looking for bugs in the specification, while it’s the calculation that went wrong. And what about the second useless test? That one just won’t run in a few months. And you can start looking for the bug again. In such a case, it’s better not to have the tests at all.
Last but not least
James Shore already linked to the rules to follow in TDD in his Red-Green-Refactor post, but if you havn’t read it yet, then do it now. I’m just repeating the rules here because I think they are very valuable.
Quoting Michael Feathers:
A test is not a unit test if:
1) It talks to the database
2) It communicates across the network
3) It touches the file system
4) It can’t run correctly at the same time as any of your other unit tests
5) You have to do special things to your environment (such as editing config files) to run it.
[...] How do you know your tests are good? – Laila Bougria looks at how Test Driven Development can give you better certainty that your tests are good. [...]
[...] How Do You Know Your Tests Are Good? (Laila Bougria) – Link of the Day [...]
Nice post.
I would change this:
“1) Think => how should you write your test?”
to:
1) Think => how do I know if I got it right / if it is working.
The thing is it isn’t revealing the purpose of that step. It is not a about looking for a test without a reason. It is about looking at how we can verify the new functionality. It is just easier to be guided that way:
How do I know if the age calculation is right:
- it works in a normal case such as x.
- it rejects future dates
- it considers the month of the year
- it doesn’t confuse the month of different years.
…
I am not saying write all at once, just that in each TDD iteration that would be the best first question to ask.
Of course thinking on the question up-front does the reveal what you are about to code wasn’t that simple at the end and you want to use already tested stuff for the date calculations – so you don’t end with a huge amount of tests
Nocturn vision » How do you know your tests are good?…
Thank you for submitting this cool story – Trackback from DotNetShoutout…
@Freddy: Thx for reading!
If I should cut my thinking phase into little pieces, I’d ask these questions:
- What functionality do I have to create (calculate age)
- How do I write it so it’s easily readable by clients (write what you want your client to be writing)
- How do I keep it extensible (applying SOLID principles)
If later I notice I have to reject future dates, consider the month of the year, or whatever other constraint or feature, I just repeat the whole iteration.
I think in TDD even for such a small thingie as the AgeCalculator, you use different RGR-iterations.
About the multiple tests: for adding features such as fe a date to calculate against, I start with a new test.
When starting with -extensions- (such as reject future dates), I have two options: Add a new test, or refactor the old test, depending on what you need to do.
The amount of tests I create doesn’t really scare me, what scares me is having useless tests
[...] How do you know your tests are good? [...]
[...] got developers on your team that don’t do TDD, you’re doomed to have passing tests that don’t test anything useful or are empty. That’s why, in my opinion, all tests should fail after generation. That way the [...]