6 Comments Test-driven development with code generation - 03/5/09
As I’ve mentioned lately, I’ve been involved with some code generation lately. The software factory generates, on one side, production code, and on the other side, tests to verify the generated code. So that’s all good and well, but…
The software factory also generates a set of tests for each layer, that are there to give you a starting point. I’m having trouble with this, because of several reasons. Since the generated code is fully data-driven (based on stored procedures), the datalayer and its test are ok. They don’t need much adjustment. The tests are testing the way they should, and they are passing, and thus verifying the generation went well.
But the data layer is not the only one with generated tests. The Business layer and presentation layer also have generated tests. As I’ve already said, the generated tests provide developers with a starting point, but all the tests pass when they are generated (while they don’t test any logic). And that bothers me a lot. They should all fail! This may be a good system for developers doing TDD, but if you’ve got developers on your team that don’t do TDD, you’re doomed to have passing tests that don’t test anything useful or are empty. That’s why, in my opinion, all tests should fail after generation. That way the developer will be forced to fix them. And if I see an Assert.IsTrue(true); then I’ll just have a valid reason to kill 
.
It you have your SF generating failing tests (I mean only the tests that aren’t complete), it has it downsides too. For starters, it will take you a while to be completly green again. And as Beck says, the time you’re red should be minimal, for motivation reasons. Still, I prefer failing generated tests, since I’ve seen a lot of misuse (useless tests) when they’re passing. And if you can’t live with that, then don’t generate tests that are not finished. You could generate parts, for example: generate the test class itsself, containing a setup and a teardown method. If it’s obvious this code will need some mocking or stubbing, you can initialize a mockrepository in the setup method. But avoid tests that are just half-tests or no tests at all. They should be written by the developers. At least, you’ll have usefull tests.
Having passing useless tests just leads to a maintenance problem after a while, and you’re completely missing the safety net that your tests should provide. The test-safety-net can contain wholes, which you’ll find in bugs, but these are fixed, because you fix a bug by writing a new test. But the net should still be a net, and not some spider web that isn’t going to hold your weight when you fall.
What do you think about test generation? Let me know!
[...] has a post about a code generation process which generates tests: Since the generated code is fully [...]
Test-driven development with code generation – Nocturn vision…
Thank you for submitting this cool story – Trackback from DotNetShoutout…
I saw a presentation on Microsoft “Pex” this weekend at Roanoke Code Camp and I found myself thinking something very similar. All the generated tests do is validate your code, bugs and all. This will lead to a generation of “programmers” who’s only exposure to unit tests is relying on tools to auto-generate their tests to validate their buggy code. 100% code coverate FTW.
I think there’s a big difference between Pex and the testsuite described above. Pex actually generates smart tests (property-based checking). The tests described in this post are just stubs, and in my opinion useless noise. The judges aren’t in yet on the usefullness of Pex ( it only works out of the box for VS2008 and my current project uses VS2005, too many toys, too litle time
), but I have been using QuickCheck in Haskell (along with HUnit) with extremely satisfactory results.
Test generation (like any form kind of code generation) completely misses the point. This has nothing to do with TDD or BDD, but everything with just testing your code. TDD and certainly BDD is writing a valuable test driven by requirements before any production code is written to satisfy that test. Its not that using NUnit, xUnit, etc. is the same as TDD ord BDD. These are design activities. That’s what Kent Beck is referring to.
Generating production code with some half baked ‘integration tests’ has nothing to do with designing soluble code. Code generation makes you instantly productive but brings a lot of suffering as the project evolves and the code needs maintenance. Its like shooting yourself in the foot while completely drugged by morphine. You won’t feel the pain immediately, but it will get there.
I’ve been ranting about code generation in the past (http://vanryswyckjan.blogspot.com/2008/02/on-software-factories.html), although I think I didn’t make myself entirely clear. Generating test code is even worse than generating production code in my book.
Sorry for the long rant, but code generation keeps pissing me off.
@Jan
I think the point about code generation being evil was clearly made in one of Davy Brion’s posts and I fully agree with everything that was said there. Most of the time code generation is a hack.
However there is another way (using machine learning techniques). Google QuickCheck, or read a John Koza paper. Tricky, but not impossible, just a lot of work, to get it right in an imperative language. You really need stuff like an abstract syntax tree, and things like a state monad are very usefull indeed. Genetic algorithms f.i. can generate solutions much better than any deterministic solution for problems like diophantine equations, or the travelling salesman problem (the canonnical example of GA’s).
Even Ward Cunningham used code generation : http://www.neocoretechs.com/vision.html. Something I have implemented in dotnet myself with less than satisfactory results but still an interesting idea. Seeing how mr Ward didn’t follow up on this I assume he also hit a wall.
A working and very impressive example : QuickCheck.
It allows you to define invariants of your program as properties and it runs thousands of randomly generated tests checking these properties. Upon failure it reduces this to the simplest possible failure case and reports it.
see : http://video.google.com/videoplay?docid=4655369445141008672.
Code generation is evil. That’s the rule. But as the first xp-rule says : There are no rules.
On a side-note : Pex (although helpfull) doesn’t seem to do the trick unfortunately. It’s still just unit testing.