About

James Golick

James Golick is an engineer, entrepreneur, speaker, and above all else, a grinder.

As CTO (or something?) of BitLove, he scaled FetLife.com's traffic by more than an order of magnitude (and counting).

James spends most of his time writing ruby and scala, building infrastructure, and extinguishing fires.

He speaks regularly at conferences and blogs periodically, but James values shipping code over just about anything else.

Latest Tweets

follow me on Twitter

James on the Web

Crazy, Heretical, and Awesome: The Way I Write Rails Apps

Mar 21 2010

Note: This is going to sound crazy at first, but bear with me.

The current best-practice for writing rails code dictates that your business logic belongs in your model objects. Before that, it wasn't uncommon to see business logic scattered all over controller actions and even view code. Pushing business logic in to models makes apps easier to understand and test.

I used this technique rather successfully for quite some time. With plugins like resource_controller, building an app became simply a matter of implementing the views and the persistence layer. With so little in the controller, the focus was mainly on unit tests, which are easier to write and maintain than their functional counterparts. But it wasn't all roses.

As applications grew, test suites would get slow — like minutes slow. When you're depending on your persistence objects to do all of the work, your unit tests absolutely must hit the database, and hitting the database is slow. It's a given in the rails world: big app == slow tests.

But slow tests are bad. Developers are less likely to run them. And when they do, it takes forever, which often turns in to checking twitter, reading reddit, or a coffee break, harming productivity.

Also, coupling all of your business logic to your persistence objects can have weird side-effects. In our application, when something is created, an after_create callback generates an entry in the logs, which are used to produce the activity feed. What if I want to create an object without logging — say, in the console? I can't. Saving and logging are married forever and for all eternity.

When we deploy new features to production, we roll them out selectively. To achieve this, both versions of the code have to co-exist in the application. At some level, there's a conditional that sends the user down one code path or the other. Since both versions of the code typically use the same tables in the database, the persistence objects have to be flexible enough to work in either situation.

If calling #save triggers version 1 of the business logic, then you're basically out of luck. The idea of creating a database record is inseparable from all the actions that come before and after it.

Here Comes the Crazy Part

The solution is actually pretty simple. A simplified explanation of the problem is that we violated the Single Responsibility Principle. So, we're going to use standard object oriented techniques to separate the concerns of our model logic.

Let's look at the first example I mentioned: logging the creation of a user. Here's the tightly coupled version:

class User < ActiveRecord::Base
  after_create :log_creation

  protected
    def log_creation
      Log.new_user(self)
    end
end

To decouple the logging from the creation of the database record, we're going to use something called a service object. A service object is typically used to coordinate two or more objects; usually, the service object doesn't have any logic of its own (simplified definition). We're also going to use Dependency Injection so that we can mock everything out and make our tests awesomely fast (seconds not minutes). The implementation is simple:

class UserCreationService
  def initialize(user_klass = User, log_klass = Log)
    @user_klass = user_klass
    @log_klass  = log_klass
  end

  def create(params)
    @user_klass.create(params).tap do |u|
      @log_klass.new_user(u)
    end
  end
end

The specs:

describe UserCreationService do
  before do
    @user       = stub("User")
    @user_klass = stub("Class:User", :create   => @user)
    @log_klass  = stub("Class:Log",  :new_user => nil)
    @service    = UserCreationService.new(@user_klass, @log_klass)
    @params     = {:name => "Matz", :hobby => "Being Nice"}
    @service.create(@params)
  end

  it "creates the user with the supplied parameters" do
    @user_klass.should have_received(:create).with(@params)
  end

  it "logs the creation of the user" do
    @log_klass.should have_received(:new_user).with(@user)
  end
end

Aside from being able to create a user record in the console without triggering a log item, there are a few other advantages to this approach. The specs will run at lightning speed because no work is actually being done. We know that Fast specs make happier and more productive programmers.

Also, debugging the actions that occur after save becomes much simpler with this approach. Have you ever been in a situation where a model wouldn't save because a callback was mistakenly returning nil? Debugging (necessarily) opaque callback mechanisms is hard.

But then I'll have all these extra classes in my app!

Yeah, it's true. You might write a few more "class X; end"s with this approach. You might even write a few percent more lines of actual code. But you'll wind up with more maintainability for it (not to mention faster tests, code that's easier to understand, etc).

The truth is that in a simple application, obese persistence objects might never hurt. It's when things get a little more complicated than CRUD operations that these things start to pile up and become pain points. That's why so many rails plugins seem to get you 80% of the way there, like immediately, but then wind up taking forever to get that extra 20%.

Ever wondered why it seems impossible to write a really good state machine plugin — or why file uploads always seem to hurt eventually, even with something like paperclip? It's because these things don't belong coupled to persistence. The kinds of functionality that are typically jammed in to active record callbacks simply do not belong there.

Something like a file upload handler belongs in its own object (at least one!). An object that is properly encapsulated and thus isolated from the other things happening around it. A file upload handler shouldn't have to worry about how the name of the file gets stored to the database, let alone where it is in the persistence lifecycle and what that means. Are we in a transaction? Is it before or after save? Can we safely raise an error?

In the tightly coupled version of the example above, the interaction between the User object and the Log object are implicit. They're unstated side-effects of their respective implementations. In the UserCreationService version, they are completely explicit, stated nicely for any reader of our code to see. If we wanted to log conditionally (say, if the User object is valid), a plain old if statement would communicate our intent far better than simply returning false in a callback.

These kinds of interactions are hard enough to get right as it is. Properly separating concerns and responsibilities is a tried, tested, and true method for simplifying software development and maintenance. I'm not just pulling this stuff out of my ass.


On Mocks and Mockist Testing

Mar 10 2010

Every so often, somebody blogs about getting bit by what they usually call "over-mocking". That is, they mocked some object, its interface changed, but the tests that were using mocks didn't fail because they were using mocks. The conclusion is: "mocks are bad".

Martin Fowler outlines two kinds of unit testers: stateist and mockist. To simplify things for a minute, a stateist tester asserts that a method returns a particular value. A mockist tester asserts that a method triggers a specific set of interactions with the object's dependencies. The "mocks are bad" crowd is arguing for a wholly stateist approach to unit testing.

On the surface, stateist testing seems certainly more convenient. A mockist is burdened with maintaining both the implementation of an object and its various test doubles. So why mocks? It seems like a lot of extra work for nothing.

Why Mocks?

A better place to start might be: what are the goals of unit testing?

For a stateist tester, unit tests serve primarily as a safety net. They catch regressions, and thus facilitate confident refactoring. If the tests are written in advance of the implementation (whether Test Driven or simply test-first), a stateist tester will derive some design benefit from their tests by virtue of designing an object's interface from the perspective of its user.

A mockist draws a thick line between unit tests and functional or integration tests. For a mockist, a unit test must only test a single unit. Test doubles replace any and all dependencies, ensuring that only an error in the object under test will cause a failure. A few design patterns facilitate this style of testing.

Dependency Injection is at the top of the list. In order to properly isolate the object under test, its dependencies must be replaced with doubles. In order to replace an object's dependencies with doubles, they must be supplied to its constructor (injected) rather than referred to explicitly in the class definition.

class VideoUploader
  def initialize(persister = Persister.new)
    @persister = persister
  end

  def create(parameters)
    @persister.save(parameters[:temp_file_name])
  end
end

When we're unit testing the above VideoUploader (ruby code, by the way), it's easy to see how we'd replace the concrete Persister implementation with a fake persister for test purposes. Rather than test that the file was actually saved to the file system (the stateist test), the mockist tester would simply assert that the persister mock was invoked correctly.

This design has the benefit of easily supporting alternate persister implementations. Instead of persisting to the filesystem, we may wish to persist videos to Amazon's S3. With this design, it's as simple as implementing an S3Persister that conforms to the persister's interface, and injecting an instance of it.

This is possible because the VideoUploader is decoupled from the Persister. If the Persister class was referred to explicitly in the VideoUploader, it would be far more difficult to replace it with a different implementation. For more on decoupled code, you must read Nick Kallen's excellent article that goes in to far more detail on these patterns and their benefits.

To be sure, we're really talking more about Dependency Injection here than anything else, and stateist testers can and do make use of DI. But the mockist test paradigm prods us towards this sort of design.

We're forced to look at the system we're building in terms of objects' interactions and boundaries. This is because it tends to be quite painful (impossible in many languages) and verbose to unit test tightly coupled code in a mockist style.

So the primary goal of a mockist's unit tests is to guide design of their object model. Making it difficult to couple objects tightly is one such guiding force.

Mockist tests also tend to highlight objects that violate the Single Responsibility Principle since their tests become a jungle of test double setup code. We can think of mockist testing like a kind of shock therapy that pushes you towards a certain kind of design. You can ignore it, but it'll hurt.

Failure isolation is probably the other big advantage of mockist tests. If your unit tests are correctly isolated, you can be sure exactly which object is responsible for a test failure. With stateist tests, a given unit test could fail if the unit or any of its dependencies are broken.

But is it worth it?

Mockist or Stateist?

The burden of maintaining mocks is by far the most common argument against mockist tests. You have to write both the implementation and at least one test double. When one changes, the other has to change too.

Perhaps most troubling, if an object's interface changes, its dependencies' unit tests will continue to pass because the mock objects will function as always — arguably a hinderance to refactoring. Since you need to test for that scenario, mockists also write integration tests. Integration tests are probably a good idea anyway, but as a mockist, you don't really have a choice.

Also, the refactoring problem only applies to dynamic languages. In a statically typed language, the program will simply fail to compile.

I find this burden troubling. More code to write makes the “we don't have time” argument come out in pressure situations. For a design exercise, the cost of mockist tests seems quite high.

On my last open source project (friendly), I decided to give mockist testing a try. Most of the code turned out beautifully. And the mistakes I did make could have been avoided had I listened to the pain I felt while testing them.

Since that project worked out well, I've been applying mockist techniques to other work. I've written mockist tests in everything from my scala projects to my rails apps. So far, so good.

In theory, I hate the idea of mockist tests. They just seem like too much work. I don't want to like them and remain reluctant to admit that I do. But in practice, I'm writing better code, and it's hard to hate that.


One Line Tests Without the Smells

Jan 16 2009

Everybody loves one line tests. That's one of the great things about shoulda. It allows you to test a lot of simple functionality using one liners, called "macros". Unfortunately, though, there are a lot of things about shoulda macros that make them less than ideal as a testing mechanism.

Firstly, they get called at class scope, so the variables from your setup blocks isn't available. A quick read through the archives of the shoulda mailing list proves that this confuses people. The workaround for this issue is a major code smell: string programming.

For example...

should_redirect_to 'post_url(@post)'

Here, 'post_url(@post)' is evaluated at the scope of the controller. A parse error in your string wouldn't point you to your should_redirect_to declaration. It would point you to a line inside of shoulda, where the instance_eval actually happens. Not good. There are lots of other reasons string programming sucks, too, but I won't get in to them here.

The second issue with macros is that failures often happen deep inside of shoulda. Backtraces often become completely useless, forcing you to open up shoulda and wade through it to figure out why your code is failing.

One More Problem

context "A blog post" do
  setup do
    @author = create_person
    @post   = create_post :author => @author
  end

  should "be editable by its author" do
    assert @post.editable_by?(@author)
  end
end

Don't see the problem?

Why bother writing self-documenting test code if you always have to explain it to the reader? Test names are essentially glorified comments and comments are frequently code smells. Furthermore, all the extra code required to create a test (should "" do ... end) almost certainly discourages one assertion per test. If the assertion is one line and the code can explain itself, why bother with all the other crap?

The Solution: Zebra

context "With a blog post" do
  setup do
    @author        = create_person
    @somebody_else = create_person
    @post          = create_post :author => @author
  end

  expect { @post.to be_editable_by(@author) }
  expect { @post.not_to be_editable_by(@author) }
end

But, what about the test name?

I'm glad you asked. This is where zebra gets really cool. The above code will create tests with the following names:

"test: With a blog post expect @post.to(be_editable_by(@author))"
"test: With a blog post expect @post.not_to(be_editable_by(@somebody_else))"

Now, that is self-documenting code.

The right tool for the job

The cool thing about zebra is that it's an extension to context or shoulda and matchy (shoulda support coming very soon). If you have a test that belongs in an it or should block, with a big, old-fashioned test name, you can have it. Just use should or it. When you have a short, self-documenting test, use expect. Best of both worlds.

Get It!

`sudo gem install giraffesoft-zebra`

http://github.com/giraffesoft/zebra


Gems I've Known and Loved #1: Expectations

Jun 08 2008

Recently, I realized that something was wrong with all of the conventional testing and specing libraries.

One of the things I loved about RSpec, when I read thin's specs, for the first time, was how easy it was to read the assertion code:

some_var.should == 5

But, when I tried using RSpec on a project, it continually frustrated me that I had to describe each test twice (by naming it, and writing pseudo-english code):

it "should augment the count by one" do
  some_var += 1
  some_var.should == 5
end

So, I went back to the familiar Shoulda. But, then, a couple of weeks ago, I came to a realization: Shoulda has exactly the same problem — it's just hidden under the awesome set of macros. Hell, even test/unit has this problem. Test names are comments. And, frankly, many, if not most of the tests that I write (like the example above) just don't need commenting — they're simple enough that a comment is unnecessary verbosity. A comment may even be distracting the reader from actual test code. Then, a couple of days later, Jay Fields blogged about exactly that. That's what made me take a look at his testing framework: expectations.

Naming Tests with Code

When I was working with RSpec, I started to wish that I could do something like this, to avoid the naming penalty:

before do
  some_var += 1
end

some_var.should == 5
some_var.should be_an_instance_of?(Fixnum)

Shoulda can do something like that, but it doesn't work in all situations, because it gets called at the class scope, so you have to go guessing about what the name of the instance variable will be. Besides, since we're sticking to one assertion per test, couldn't the assertion be the name of the test?

some_var.should == 5 do
  some_var += 1
end

Oops, we just derived the syntax for expectations.

Assertions are King

Except that expectations takes it a few steps further. It provides a ton of niceties for describing your tests in sensible language, and great mocking facilities. Our last example would look like this:

expect 5 do
  some_var  = 4
  some_var += 1
end

If we wanted to test for Fixnum, like in our second assertion, above, there's a nice shorthand for that.

expect Fixnum do
  some_var  = 4
  some_var += 1
end

As I write this, I can hear somebody out there screaming: "That's not very DRY!!". Even though I prefer the duplication in this particular example for readability reasons, let's DRY it up, just for fun.

setup = lambda do 
  some_var  = 4
  some_var += 1
end

expect(Fixnum, &setup)
expect(5, &setup)

Looks a lot like what I was wishing for with RSpec, doesn't it?

Next, let's look at a mocking example. One of the things that makes mocking a little bit weird with conventional testing and specing frameworks is that assertions come last, but mocks come first. So, when you're trying to follow the one assertion per test pattern, you end up with two different flows: setup, assert, for assertions, and: mock, setup, with mocks. Since assertions always come first with expectations, there's only one possible flow, making tests more readable.

expect mock.to.receive(:some_method).with(1) do |my_mock|
  my_mock.some_method(1)
end

Finally, spec junkies can even write expectations using BDD-style language.

expect klass.new.to.have.finished do |process|
  process.finished = true
end

Caveats

Expectations is pretty new, so I haven't yet seen any niceties for testing rails. It would be great to be able to do something like:

expect controller.to_render('index') do |c|
  get :index
end

As a big fan of Shoulda, I'd love to see some of the same types of macros for expectations, too (not in the core framework, but as an add-on).

Get It

$ sudo gem install expectations

Check out the RDoc for more examples, and a full set of documentation.


Noisy Backtraces Got You Down?

Dec 01 2007

Bothered by all the noise in my tests' backtraces, I was thrilled when I first saw a thread on the shoulda mailing list with some discussion around making them a little bit easier on the eyes. Assuming that creating such a filter would be a long and tedious process of monkey-patching test/unit, I forgot about the idea, assuming the job better left for somebody with more time to spare than myself.

When Dan Croak revived the thread with some sample code, cooked up at a Boston.rb hackfest, it occurred to me that the job was far more manageable than I had originally conceived. I quickly fired Dan an email asking whether he'd be interested in a pluginization of their concept. With a resounding yes! from Dan, we set off to create quiet_stacktracebacktrace.

The rest of this post is cross-posted on GIANT ROBOTS

90% of this typical backtrace will not help you hunt and kill bad code:

1) Failure:
test: logged in on get to index should only show projects for the user's account. (ProjectsControllerTest)
[/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/test/unit/assertions.rb:48:in `assert_block'
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/test/unit/assertions.rb:500:in `_wrap_assertion'
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/test/unit/assertions.rb:46:in `assert_block'
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/test/unit/assertions.rb:63:in `assert'
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/test/unit/assertions.rb:495:in `_wrap_assertion'
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/test/unit/assertions.rb:61:in `assert'
test/functional/projects_controller_test.rb:31:in `__bind_1196527660_342195'
/Users/james/Documents/railsApps/projects/vendor/plugins/shoulda/lib/shoulda/context.rb:98:in `call'
/Users/james/Documents/railsApps/projects/vendor/plugins/shoulda/lib/shoulda/context.rb:98:in `test: logged in on get to index should only show projects for the user's account. '
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/test/unit/testcase.rb:78:in `__send__'
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/test/unit/testcase.rb:78:in `run'
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/test/unit/testsuite.rb:34:in `run'
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/test/unit/testsuite.rb:33:in `each'
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/test/unit/testsuite.rb:33:in `run'
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/test/unit/testsuite.rb:34:in `run'
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/test/unit/testsuite.rb:33:in `each'
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/test/unit/testsuite.rb:33:in `run'
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/test/unit/ui/testrunnermediator.rb:46:in `run_suite'
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/test/unit/ui/console/testrunner.rb:67:in `start_mediator'
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/test/unit/ui/console/testrunner.rb:41:in `start'
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/test/unit/ui/testrunnerutilities.rb:29:in `run'
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/test/unit/autorunner.rb:216:in `run'
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/test/unit/autorunner.rb:12:in `run'
/System/Library/Frameworks/Ruby.framework/Versions/1.8/usr/lib/ruby/1.8/test/unit.rb:278
test/functional/projects_controller_test.rb:36]:
one or more projects shown does not belong to the current user's account.
&lt;false&gt; is not true.

Noisy backtraces must be ruthlessly silenced like political dissidents in Stalinist Russia. This much is clear.

Quiet Backtrace

Install the gem:

  sudo gem install quietbacktrace

Require quietbacktrace:

## test_helper.rb
require 'quietbacktrace'

Run your Test::Unit tests:

1) Failure:
test: logged in on get to index should only show projects for the user's account. (ProjectsControllerTest)
[test/functional/projects_controller_test.rb:31
vendor/plugins/shoulda/lib/shoulda/context.rb:98
vendor/plugins/shoulda/lib/shoulda/context.rb:98
test/functional/projects_controller_test.rb:36]:
one or more projects shown does not belong to the current user's account.
&lt;false&gt; is not true.

Ooh la la! Now we’re cooking with gas. However, those shoulda-related lines are cluttering an otherwise perfect backtrace. Luckily, Quiet Backtrace is designed to be extended by calling two types of blocks that yield one line of the backtrace at a time.

Silencers and filters

  1. Silencers let you specify conditions that, if true, will remove the line from the backtrace.
  2. Filters let you use Ruby’s succulent set of String methods to modify a line by slicing and stripping and chomping away at anything you deem ugly and unnecessary. (such as the :in `__bind_1196527660_342195’ in the original example)

Say you want to remove Shoulda-related lines… you create a new silencer and add it the Array of backtrace_silencers:

class Test::Unit::TestCase
  self.new_backtrace_silencer :shoulda do |line| 
    line.include? 'vendor/plugins/shoulda'
  end
  
  self.backtrace_silencers << :shoulda
end

Re-run your tests and bask in the sweet sounds of silence:

1) Failure:
test: logged in on get to index should only show projects for the user's account. (ProjectsControllerTest)
[test/functional/projects_controller_test.rb:31
test/functional/projects_controller_test.rb:36]:
one or more projects shown does not belong to the current user's account.
&lt;false&gt; is not true.

Exquisitely sparse. Quiet Backtrace clears distractions from the “getting to green” TDD process like a Buddhist monk keeping his mind clear during meditation.

Getting noisy again

On occasion, you’ll want to see the noisy backtrace. Easy:

class Test::Unit::TestCase
  self.quiet_backtrace = false
end

You can set Test::Unit::TestCase.quiet_backtrace to true or false at any level in your Test::Unit code. Stick it in your test_helper.rb file or get noisy in an individual file or test. More flex than a rubber band.

Using Quiet Bactrace with Rails

After you have installed the gem on your local machine, I recommend using the excellent gemsonrails plugin to freeze it to your vendor/gems directory and automatically add it to your load path.

Install gemsonrails and add it your Rails app if you don’t already have it:

gem install gemsonrails
cd rails-app-folder
gemsonrails

Then freeze quietbacktrace:

rake gems:freeze GEM=quietbacktrace

Quiet Backtrace will now work with your tests, but because this gem is meant to work on any Ruby project with Test::Unit, it does turn any Rails-specific silencers or filters on by default. However, there is one of each, ready to be switched on, that remove the most dastardly lines.

Add these lines to your /test/test_helper.rb file to get perfectly clean Rails backtraces:

class Test::Unit::TestCase
  self.backtrace_silencers << :rails_vendor
  self.backtrace_filters   << :rails_root
end

Ongoing development

Bug reports and patches are welcome at RubyForge. Talk about it on the mailing list. It’s is a safe place. A place where we can feel free sharing our feelings. A nest in a tree of trust and understanding.


Testing Misconceptions #1: Exploratory Programming

Oct 05 2007

So, my last essay on testing was ycombinatored, and then reddited the next day. Cool! I'm honored to have so many people reading, and discussing my article. It's a pleasure.

I found some of the discussion very interesting. It seems like a lot of developers still don't believe in unit testing their code. In fact, many made arguments that questioned, or even outright dismissed the value of unit testing (for more such comments, see the reddit and ycombinator threads, or the thread of comments on the article itself). What surprised me most, though, was the number of misconceptions people have about what testing actually is, why we test, and how long it takes. Many, if not most, of the anti-testing arguments are based on entirely false premises.

In this on-going series, I'll put those misconceptions to the test (pun intended), and provide my take on what the truth is.

Testing Myth #1: I can't test first, because I don't have an overall picture of my program.

BTUF (Big Test Up Front) incurrs [sp] many of the same risks as BDUF (Big Design Up Front). It assumes you are creating artifacts now that will last and not change drastically in the future.
Yes, TDD implies that there is a more or less exact specification. Otherwise, if you're just experimenting, you would have to write the test and your code, and that's going to make you less inclined to throw it away and test out something else (see "Planning is highly overrated").
When I really have latitude in my goals, my code is just about impossible to pin down until it's 95% implemented.
How can you test something if you don't even know how or if it works? You need to hack on it and see if you can get things going before you nail it down, no?
According to this group, testing first is impossible because they're not sure exactly what they're writing. Some of them go so far as to equate testing first with big up front designs. The assumption, in both cases, is that writing your tests first means writing all of your tests first, or at least enough to require a general overview of your program. Nothing could be further from the truth.

It seems likely to me that this group's misconception stems from mixing up unit testing with acceptance testing. Acceptance testing, whether automated or manual, would require an overall specification for how (at least some major portion of) the system should function. Nobody is suggesting that you write your acceptance tests first.

Unit tests verify components of your program in isolation. They should be as small as possible. And, in fact, if your unit tests know enough about your program that they're starting to look like acceptance tests, their effectiveness is going to be diminished considerably. That is, you don't want your unit tests to have an overall picture of what you're building. They should have as little of that picture as possible.

Separating Concerns

Writing tests first doesn't mean you can't explore. It means that the exploration process happens in your tests, instead of your code - which is great! In your tests is where the exploration process belongs.

When you explore in your implementation code, you're trying to answer two questions at once: "What should my code do?" and "What's the best way to implement my code's functionality?". Instead of trying to juggle both concerns at once, testing first divides your exploration in to two stages. It creates a separation of concerns. You might even say that TDD is like MVC for your coding process.

You begin your exploration, of course, by putting together some preliminary tests for the first bit of functionality you're going to write. By considering the output before the implementation, you gain several advantages. The classic example, here, is that you get the experience of using your interface, before you've invested any time in bad API design ideas you may have had. But wait, there's more!

You also get the opportunity to focus on what your code will do. Before I began practicing TDD, I would regularly be almost all the way through writing a block of code before I realized that the idea just wasn't going to work. The thing is, when you're exploring, and you're focused on one or two implementation lines at a time, the result of the code becomes an afterthought. By spending that minute or two up front thinking about what should come out of your code, you'll save yourself a ton of backtracking, and rethinking later on.

That's all for today

I hope you enjoyed the first installment of Testing Misconceptions. I'd love to hear your feedback, or ideas for topics. Please feel free to leave them in the comments, or shoot me an email. Please check back for more episodes.


We don't write tests. There just isn't time for luxuries.

Aug 22 2007

Over the last few weeks and months, I've often had the displeasure of being reminded of the common opinion that writing automated tests slows down the development process. "We just didn't have time to write tests" or "Nope, we don't have any test coverage... we just didn't have time." are common expressions of this problematic trend in software development. In order to disregard something widely accepted as being a necessary practise, writing automated tests must really take forever, right? Wrong. I call this the testing-slows-us-down argument, and, frankly, I don't buy it at all. Here's why.

Everybody Tests

Everybody has to test their code. It's just that those of us who write automated tests write code to do our testing, where non-testers use humans (themselves, usually) to manually verify correct behavior. So, we can be sure that the testing-slows-us-down argument rests on the premise that manually verifying behavior is faster than writing automated tests.

Investing Time

The two methods of testing distribute your time investment differently. Since automated tests run very quickly, the time investment is made in writing them. Running the tests is nearly instantaneous. With manual testing, it is the opposite. Designing the tests takes nearly zero time, whereas actually running the tests takes a measurable amount of time each time you need the test. So, with automated testing, you get most of your time investment out of the way at the outset. Once it's written, it's written. With manual tests, your time investment grows each time you test for something. In order for the testing-slows-us-down argument to remain valid, the total time investment made on manual testing must be less than the time investment required to write the automated tests.

In these terms, the testing-slows-us-down argument can be expressed as follows: The time it takes to write the automated tests for a feature is greater than the total time that will be spent manually testing that feature throughout the lifetime of the project.

Since one of my goals for this article is proving the "no time to test, we need to launch our product" people wrong, I want to show that this (short-sighted) argument is as, if not more, flawed than the testing-slows-us-down argument. I'm going to phrase the product-launch argument as follows: The time it takes to write the tests for a feature is greater than the total time that will be spent manually testing that feature until product launch.

Breaking Even

If it takes twenty minutes to write your automated tests, and one minute to test your feature manually, the break even point for an automated tester is when they've run their tests twenty times. After that point, automated tests become cheaper and cheaper (per test). Manual tests continue to become increasingly expensive over their lifetime. So, the testing-slows-us-down argument can be rephrased once more, making use of our new terminology: The automated testing break even point will not be reached before product launch.

Developers Developers Developers

Because the cost of manual testing grows considerably each time you run your tests, bringing on extra developers adds a multiplier to your (growing) test cost. Anytime any developer wants to merge their branch back in to trunk, they're going to have to take a look over the whole source tree to make sure they didn't break anything. This means, for the reasonably attentive developer, testing code that they didn't write - going over all of the features. It's the manual testing equivalent of running the whole test suite. Not only is this a time-consuming process, but it is incredibly error prone.

Debugging Time!

No developer can be reasonably expected to remember, and meticulously verify all of expected functionality from an application, at each check-in. That's a superhuman expectation. You might even say it's a job better suited to a machine? Seriously, though, code is interdependent. Changes in one area can have impacts all over the application. When relying on humans to verify application behavior, a lot of bugs are going to slip through the cracks.

More bugs means more time spent debugging, which, incidentally, means more time spent testing, for the manual tester. Moreover, it's not uncommon for the same bug to surface repeatedly. We've all seen it. It's called regression, and it's why us automated testers have something wonderful called regression tests.

With every bug, automated testers (like always), make a one time investment. Since automated testing is far more likely to prevent regressions than manual testing, the time benefits here are two-fold. First, once the regression test is written, it's written, and the behavior doesn't need to be repeatedly verified by hand. Second, the bug is far less likely to resurface. Manual testers may argue, here, that they are capable of adding the regression tests to their regular passes over their code, keeping the bugs out, just the same. While this may be the case for the most superhuman of individuals, the liklihood that an entire team may be capable of such incredible manual testing is very low.

So, as a general rule, I think it's safe to say that automated tests significantly reduce debugging time, by providing a much higher degree of accuracy, and acting as a powerful weapon for preventing regression.

Release Already

It seems pretty clear to me, after a thorough analysis of the testing-slows-us-down argument that its proponents are, at the least, misguided. Automated testing is at least as fast as manual testing. In writing this article, I thought a lot about why so many people have this common misconception. I think it mostly stems from one of the following.

The most common cause of this misconception is likely naivete. Many of the challenges that really bring out the best in automated testing (and, consequently, the worst in manual testing) are far more evident with bigger projects. While I maintain that automated testing is at least as fast as manual testing on all projects, it's likely that bigger projects will see much bigger benefits. The problem, here, is that big projects often start out as small ones. And, unfortunately, growing pains can cause some of the most difficult problems with keeping software working properly.

My assumption is that the ones who aren't naive are just lazy. Learning how, what, and when to write automated tests can be a difficult undertaking, but it's well worth it. Like writing the tests themselves, a little bit of up-front investment in your skillset will save you loads of time, and headache later on. So, do yourself a favor, and learn to test. You'll thank yourself for it.

Update: See my first response in a new series to some of the discussion surrounding this article.


3 ways to improve your bullshit methodology

Aug 10 2007

Marc Cournoyer writes a great post, detailing 5 ways to know whether your methodology is working (or whether its bullshit), specifically re: TDD/XP.

I have been trying to improve my TDD practice for some time now. I am slowly getting better at writing tests first (and just writing tests, of course), but it does represent quite a significant shift in thinking. And, when you're used to writing the code first, as Marc says, that's where you're naturally going to go when the pressure is on. So, how do we stop this behavior? How do we get in to the test-first zone?

Here are 3 things that have started working for me:

1. When you're stuck on what to test, make a list of possible inputs and selected outputs


One of the biggest challenges for me has been overcoming my tendency towards doing something like exploratory testing of my own code, as I write it. This was the bad habit of not knowing what my code was going to do, before I wrote it. I'd spend some time fiddling around with a few lines that I thought might accomplish what I wanted, and looking at output, until it looked right (sound familiar?). With TDD, you have to start by thinking about what your code will output.
Take a second before you write any tests, and make a list of input parameters, and output expectations. Once you have this list, you'll see that it is much easier to know what you need to test for, and it will even help you write your code afterwards, too. This is an easy one, but it illustrates the point:
# PostsController#show
#
# Inputs:
#   params[:id]
# Outputs:
#   @post <-- contains the Post which corresponds to the params[:id] input parameter
#   OR
#   throws ActiveRecord::RecordNotFound if Post w/id == params[:id] does not exist

2. Make it an exercise and practice, practice, practice


Take 2 hours at home, in your spare time, and give yourself too much to do. Outline more features than you can realistically implement in that timeframe, and go for it. Racing the clock helps, because that's what you'll be facing on a real project. It sounds cheesy, but it has really worked well for me.

3. Use autotest

(the direct link is here, but it seems to be down right now)

The easier, and more comfortable testing is, the more likely you are to do it. Autotest watches all of your files, and when one changes, it runs the appropriate tests. All you have to do is save the relevant file, and look over at your terminal window to see the results. No more hitting refresh in your browser, or even running tests manually.
Marc also told me about CruiseControl.rb. I haven't had a chance to play with it yet, but it looks very cool. The idea is that if somebody checks something in to source control that breaks any tests, they are alerted immediately. Anything that makes testing easier is probably better.

What methodology-improvement tips do you have?


When to test

Aug 10 2007

Always

A lot of people seem to struggle with knowing when to test their code. The simple answer is always, and everything.

Always test every line of code that you write.

Know that your code works (and interacts) as expected


First and foremost, writing tests provides you with a safety net. Making changes and refactoring code becomes a calculated maneuver, instead of a guessing game. Stop asking yourself: Did I break something? And know that if the tests didn't break, you're safe.

When you first write a block of code, manual testing might be sufficient to ensure that it works correctly, because its purpose is fresh in your mind. But, what about all the code it interacts with? Can you possibly remember test for all of the things you might have affected? Can the next user of your code remember to test for all of the things they might have affected? Of course not, and that's exactly why it's up to you to build those tests for them.

Living proof


Take a look at this snippet that got checked in to svn today:
belongs_to :user, :company
Had the author of this code written tests for his code, or even run the generated test suite for his model, he might have realized that he'd made a typo, and that the belongs_to method doesn't accept multiple models as its first argument, and that the correct code would have looked like this:
belongs_to :user
belongs_to :company
Test first, test always.