I found some of the discussion very interesting. It seems like a lot of developers still don't believe in unit testing their code. In fact, many made arguments that questioned, or even outright dismissed the value of unit testing (for more such comments, see the reddit and ycombinator threads, or the thread of comments on the article itself). What surprised me most, though, was the number of misconceptions people have about what testing actually is, why we test, and how long it takes. Many, if not most, of the anti-testing arguments are based on entirely false premises.
In this on-going series, I'll put those misconceptions to the test (pun intended), and provide my take on what the truth is.
Testing Myth #1: I can't test first, because I don't have an overall picture of my program.
BTUF (Big Test Up Front) incurrs [sp] many of the same risks as BDUF (Big Design Up Front). It assumes you are creating artifacts now that will last and not change drastically in the future.
Yes, TDD implies that there is a more or less exact specification. Otherwise, if you're just experimenting, you would have to write the test and your code, and that's going to make you less inclined to throw it away and test out something else (see "Planning is highly overrated").
When I really have latitude in my goals, my code is just about impossible to pin down until it's 95% implemented.
How can you test something if you don't even know how or if it works? You need to hack on it and see if you can get things going before you nail it down, no?According to this group, testing first is impossible because they're not sure exactly what they're writing. Some of them go so far as to equate testing first with big up front designs. The assumption, in both cases, is that writing your tests first means writing all of your tests first, or at least enough to require a general overview of your program. Nothing could be further from the truth.
It seems likely to me that this group's misconception stems from mixing up unit testing with acceptance testing. Acceptance testing, whether automated or manual, would require an overall specification for how (at least some major portion of) the system should function. Nobody is suggesting that you write your acceptance tests first.
Unit tests verify components of your program in isolation. They should be as small as possible. And, in fact, if your unit tests know enough about your program that they're starting to look like acceptance tests, their effectiveness is going to be diminished considerably. That is, you don't want your unit tests to have an overall picture of what you're building. They should have as little of that picture as possible.
Separating Concerns
Writing tests first doesn't mean you can't explore. It means that the exploration process happens in your tests, instead of your code - which is great! In your tests is where the exploration process belongs.When you explore in your implementation code, you're trying to answer two questions at once: "What should my code do?" and "What's the best way to implement my code's functionality?". Instead of trying to juggle both concerns at once, testing first divides your exploration in to two stages. It creates a separation of concerns. You might even say that TDD is like MVC for your coding process.
You begin your exploration, of course, by putting together some preliminary tests for the first bit of functionality you're going to write. By considering the output before the implementation, you gain several advantages. The classic example, here, is that you get the experience of using your interface, before you've invested any time in bad API design ideas you may have had. But wait, there's more!
You also get the opportunity to focus on what your code will do. Before I began practicing TDD, I would regularly be almost all the way through writing a block of code before I realized that the idea just wasn't going to work. The thing is, when you're exploring, and you're focused on one or two implementation lines at a time, the result of the code becomes an afterthought. By spending that minute or two up front thinking about what should come out of your code, you'll save yourself a ton of backtracking, and rethinking later on.
Even in the case that was not what the "the truth is" it does sound pretty good ;)
Nice writeup James, I hope your serie will help a couple of people better understand the value of testing and how to do it properly!
But about separation of concerns, your unit test should test one unit at the time, seperating each one individually, so I agree on this one, but your functional, integration and system test won't respect that rule of separation of concerns the more you move up the lader to user centric test. But in a way your system tests shouldn't be aware of the inner working of the unit it's testing (black box testing) so you're right. It's the comparison to MVC that is not 100% right, it's not a one-to-one mapping but still layer type architecture.
Actually, Marc, what I meant by the MVC analogy was that testing separates your expectations, and implementation. You only have to think about one at a time. And, since they're separate, you can make changes to your implementation (refactoring), without having to change your verification process. It's a separation of concerns for your process, too, not strictly your code.
Well putting things together into OOP/classes is also splitting concerns, and you can hardwire tests into this all as well. I still believe the biggest factor is the human factor.
The problem with many programming methodology blogs like this one is that the author usually does not specify which kind of programming is the given idea is applicable for.
I propose a measurement: the cost of programmer hours per user: expected revenue / (expected number of users / expected number of hours)
I'm customizing the production planning facilities of an ERP system. Having two planners at the client and an hourly charge of GBP 110, every programmer hour costs USD 110 peruser_! Of course it means the only kind of testing feasible in such conditions is having the users play around with it. They happily do that as their cost per hour is, of course, a lot lower and the cost gain of an automated test is offset by the fact that they do know the fringe cases, I don't. (Because, specifing them would cost USD 110 per hour per user too.) (Of course, the trick is that everything I do is overridable manually and there are reports which help the users point out if something is wrong, therefore no bug can incur costs for other than the cost of manually entering data for that short period until I fix the bug, which is usually hours.)
Bloggers, please, specify a from-to range of the cost of programmer hours per user you ideas are applicable for! There is no such thing as a right way of doing something, every decision must be based on comparing costs, including opportunity costs.
That much on the general side. On the practical side, sometimes I'd doing stuff other than ERP customization as it's a bit boring after five years. The other day I was playing around with genetic algorithms in Python. Now, how do I write tests until I know what classes will I have, what methods will they have, what are the parameters and the expected results? Of course I have to write them first, but this means, that it cannot be exploratory programming as in exploratory programming you have no friggin' idea about these ones. Exploratory programming usually means, at leat for me, copying a tutorial for a library, then modifying it with hardcoded data close to what my real data will be, then breaking them up into functions and at the end, breaking them up into classes. So I only know the classes at the point when the exploration is done and all I need to do is add handling the fringe cases and throwing the exceptions.
Also, unit testing might be a great idea if you are stuck with imperative programming but the way I program in Python is functional style: having many one-liner functions that do nothing else but return a computed expression, and then other, still one-liner functions will map, filter, reduce data based on these functions and so on. These side-effect-less one-liners simply cannot be wrong, as they can easily be reasoned about. Like, def reverseDictionary(dict): return dict( (dict[key], key) for key in _dict ) - there is no bloody way it can be wrong.
Errors lurk in the later stages, in forgetting to throw exceptions and to check fringe cases. I can test these, but it means I can only write test when about half of the code is already written.
Hmmm, unit tests aren't set in stone. Maybe I'm jumping ahead, but TDD is a cycle, a short feedback loop between unit tests and code. If whatever behaviour changes, the test needs to be re-evaluated. Surely TDD can be used in exploratory programming.
Writing the tests (doesn't have to be many or complicated) down first allows us to unload our brain of the assumptions, so that we can make room in order to elaborate even further and build on that. The cycle helps us gain confidence in our code. We can even be bold and try something risky, as existing behaviour will either pass or fail: it helps us know if we broke something and if yes, where.
Maybe James can do a screencast with autotest.
Good article. A small nitpick however:
"Nobody is suggesting that you write your acceptance tests first."
Why not? I'd argue that acceptance tests are easier to write first as they are at a much higher level. While you may start with some exploration at the unit level for your design, you should already have an idea of what your customers expect from the application before hand.
There is also one important gain when you use tests: they document your application. As much as comments in the code show how the methods work, the tests help other coders to understand how the class is supposed to be used and the mindset of the original coder.
It also speed up your coding since you dont always have to press the refresh button to see the result of the change.