As I’ve mentioned lately, I’ve been involved with some code generation lately. The software factory generates, on one side, production code, and on the other side, tests to verify the generated code. So that’s all good and well, but…
The software factory also generates a set of tests for each layer, that are there to give you a starting point. I’m having trouble with this, because of several reasons. Since the generated code is fully data-driven (based on stored procedures), the datalayer and its test are ok. They don’t need much adjustment. The tests are testing the way they should, and they are passing, and thus verifying the generation went well.
But the data layer is not the only one with generated tests. The Business layer and presentation layer also have generated tests. As I’ve already said, the generated tests provide developers with a starting point, but all the tests pass when they are generated (while they don’t test any logic). And that bothers me a lot. They should all fail! This may be a good system for developers doing TDD, but if you’ve got developers on your team that don’t do TDD, you’re doomed to have passing tests that don’t test anything useful or are empty. That’s why, in my opinion, all tests should fail after generation. That way the developer will be forced to fix them. And if I see an Assert.IsTrue(true); then I’ll just have a valid reason to kill .
It you have your SF generating failing tests (I mean only the tests that aren’t complete), it has it downsides too. For starters, it will take you a while to be completly green again. And as Beck says, the time you’re red should be minimal, for motivation reasons. Still, I prefer failing generated tests, since I’ve seen a lot of misuse (useless tests) when they’re passing. And if you can’t live with that, then don’t generate tests that are not finished. You could generate parts, for example: generate the test class itsself, containing a setup and a teardown method. If it’s obvious this code will need some mocking or stubbing, you can initialize a mockrepository in the setup method. But avoid tests that are just half-tests or no tests at all. They should be written by the developers. At least, you’ll have usefull tests.
Having passing useless tests just leads to a maintenance problem after a while, and you’re completely missing the safety net that your tests should provide. The test-safety-net can contain wholes, which you’ll find in bugs, but these are fixed, because you fix a bug by writing a new test. But the net should still be a net, and not some spider web that isn’t going to hold your weight when you fall.
What do you think about test generation? Let me know!