Comments on: Test-driven development with code generation http://www.noctovis.net/blog/index.php/2009/03/05/test-driven-development-with-code-generation/ A night's vision on .NET Sat, 17 Oct 2009 11:00:07 +0200 http://wordpress.org/?v=2.8.4 hourly 1 By: kilfour http://www.noctovis.net/blog/index.php/2009/03/05/test-driven-development-with-code-generation/comment-page-1/#comment-324 kilfour Wed, 11 Mar 2009 20:56:51 +0000 http://www.noctovis.net/blog/index.php/2009/03/05/test-driven-development-with-code-generation/#comment-324 @Jan I think the point about code generation being <i>evil</i> was clearly made in one of Davy Brion's posts and I fully agree with everything that was said there. Most of the time code generation is a hack. However there is another way (using machine learning techniques). Google QuickCheck, or read a John Koza paper. Tricky, but not impossible, just a lot of work, to get it right in an imperative language. You really need stuff like an abstract syntax tree, and things like a state monad are very usefull indeed. Genetic algorithms f.i. can generate solutions much better than any deterministic solution for problems like diophantine equations, or the travelling salesman problem (the canonnical example of GA's). Even Ward Cunningham used code generation : http://www.neocoretechs.com/vision.html. Something I have implemented in dotnet myself with less than satisfactory results but still an interesting idea. Seeing how mr Ward didn't follow up on this I assume he also hit a wall. A working and very impressive example : QuickCheck. It allows you to define invariants of your program as properties and it runs thousands of randomly generated tests checking these properties. Upon failure it reduces this to the simplest possible failure case and reports it. see : http://video.google.com/videoplay?docid=4655369445141008672. Code generation is evil. That's the rule. But as the first xp-rule says : There are no rules. On a side-note : Pex (although helpfull) doesn't seem to do the trick unfortunately. It's still just unit testing. @Jan
I think the point about code generation being evil was clearly made in one of Davy Brion’s posts and I fully agree with everything that was said there. Most of the time code generation is a hack.
However there is another way (using machine learning techniques). Google QuickCheck, or read a John Koza paper. Tricky, but not impossible, just a lot of work, to get it right in an imperative language. You really need stuff like an abstract syntax tree, and things like a state monad are very usefull indeed. Genetic algorithms f.i. can generate solutions much better than any deterministic solution for problems like diophantine equations, or the travelling salesman problem (the canonnical example of GA’s).
Even Ward Cunningham used code generation : http://www.neocoretechs.com/vision.html. Something I have implemented in dotnet myself with less than satisfactory results but still an interesting idea. Seeing how mr Ward didn’t follow up on this I assume he also hit a wall.
A working and very impressive example : QuickCheck.
It allows you to define invariants of your program as properties and it runs thousands of randomly generated tests checking these properties. Upon failure it reduces this to the simplest possible failure case and reports it.
see : http://video.google.com/videoplay?docid=4655369445141008672.
Code generation is evil. That’s the rule. But as the first xp-rule says : There are no rules.
On a side-note : Pex (although helpfull) doesn’t seem to do the trick unfortunately. It’s still just unit testing.

]]> By: Jan Van Ryswyck http://www.noctovis.net/blog/index.php/2009/03/05/test-driven-development-with-code-generation/comment-page-1/#comment-323 Jan Van Ryswyck Wed, 11 Mar 2009 20:03:48 +0000 http://www.noctovis.net/blog/index.php/2009/03/05/test-driven-development-with-code-generation/#comment-323 Test generation (like any form kind of code generation) completely misses the point. This has nothing to do with TDD or BDD, but everything with just testing your code. TDD and certainly BDD is writing a valuable test driven by requirements before any production code is written to satisfy that test. Its not that using NUnit, xUnit, etc. is the same as TDD ord BDD. These are design activities. That's what Kent Beck is referring to. Generating production code with some half baked 'integration tests' has nothing to do with designing soluble code. Code generation makes you instantly productive but brings a lot of suffering as the project evolves and the code needs maintenance. Its like shooting yourself in the foot while completely drugged by morphine. You won't feel the pain immediately, but it will get there. I've been ranting about code generation in the past (http://vanryswyckjan.blogspot.com/2008/02/on-software-factories.html), although I think I didn't make myself entirely clear. Generating test code is even worse than generating production code in my book. Sorry for the long rant, but code generation keeps pissing me off. Test generation (like any form kind of code generation) completely misses the point. This has nothing to do with TDD or BDD, but everything with just testing your code. TDD and certainly BDD is writing a valuable test driven by requirements before any production code is written to satisfy that test. Its not that using NUnit, xUnit, etc. is the same as TDD ord BDD. These are design activities. That’s what Kent Beck is referring to.

Generating production code with some half baked ‘integration tests’ has nothing to do with designing soluble code. Code generation makes you instantly productive but brings a lot of suffering as the project evolves and the code needs maintenance. Its like shooting yourself in the foot while completely drugged by morphine. You won’t feel the pain immediately, but it will get there.

I’ve been ranting about code generation in the past (http://vanryswyckjan.blogspot.com/2008/02/on-software-factories.html), although I think I didn’t make myself entirely clear. Generating test code is even worse than generating production code in my book.

Sorry for the long rant, but code generation keeps pissing me off.

]]> By: kilfour http://www.noctovis.net/blog/index.php/2009/03/05/test-driven-development-with-code-generation/comment-page-1/#comment-319 kilfour Wed, 11 Mar 2009 10:36:25 +0000 http://www.noctovis.net/blog/index.php/2009/03/05/test-driven-development-with-code-generation/#comment-319 I think there's a big difference between Pex and the testsuite described above. Pex actually generates <i>smart tests</i> (property-based checking). The tests described in this post are just stubs, and in my opinion useless noise. The judges aren't in yet on the usefullness of Pex ( it only works out of the box for VS2008 and my current project uses VS2005, too many toys, too litle time ;-) ), but I have been using QuickCheck in Haskell (along with HUnit) with extremely satisfactory results. I think there’s a big difference between Pex and the testsuite described above. Pex actually generates smart tests (property-based checking). The tests described in this post are just stubs, and in my opinion useless noise. The judges aren’t in yet on the usefullness of Pex ( it only works out of the box for VS2008 and my current project uses VS2005, too many toys, too litle time ;-) ), but I have been using QuickCheck in Haskell (along with HUnit) with extremely satisfactory results.

]]> By: Nathan http://www.noctovis.net/blog/index.php/2009/03/05/test-driven-development-with-code-generation/comment-page-1/#comment-301 Nathan Mon, 09 Mar 2009 12:40:28 +0000 http://www.noctovis.net/blog/index.php/2009/03/05/test-driven-development-with-code-generation/#comment-301 I saw a presentation on Microsoft "Pex" this weekend at Roanoke Code Camp and I found myself thinking something very similar. All the generated tests do is validate your code, bugs and all. This will lead to a generation of "programmers" who's only exposure to unit tests is relying on tools to auto-generate their tests to validate their buggy code. 100% code coverate FTW. I saw a presentation on Microsoft “Pex” this weekend at Roanoke Code Camp and I found myself thinking something very similar. All the generated tests do is validate your code, bugs and all. This will lead to a generation of “programmers” who’s only exposure to unit tests is relying on tools to auto-generate their tests to validate their buggy code. 100% code coverate FTW.

]]> By: DotNetShoutout http://www.noctovis.net/blog/index.php/2009/03/05/test-driven-development-with-code-generation/comment-page-1/#comment-277 DotNetShoutout Sat, 07 Mar 2009 16:30:49 +0000 http://www.noctovis.net/blog/index.php/2009/03/05/test-driven-development-with-code-generation/#comment-277 <strong>Test-driven development with code generation - Nocturn vision...</strong> Thank you for submitting this cool story - Trackback from DotNetShoutout... Test-driven development with code generation – Nocturn vision…

Thank you for submitting this cool story – Trackback from DotNetShoutout…

]]> By: The Inquisitive Coder - Davy Brion’s Blog » Blog Archive » Smarten Up Instead Of Generating Dumb Tests http://www.noctovis.net/blog/index.php/2009/03/05/test-driven-development-with-code-generation/comment-page-1/#comment-253 The Inquisitive Coder - Davy Brion’s Blog » Blog Archive » Smarten Up Instead Of Generating Dumb Tests Thu, 05 Mar 2009 22:58:59 +0000 http://www.noctovis.net/blog/index.php/2009/03/05/test-driven-development-with-code-generation/#comment-253 [...] has a post about a code generation process which generates tests: Since the generated code is fully [...] [...] has a post about a code generation process which generates tests: Since the generated code is fully [...]

]]>