James on Software

I am working on integrating a third party service in to a web app for a client — a task I can always count on to be at least somewhat painful. This particular service's API, however, is worse than painful. Their API is a failure in nearly every possible way — and some new and surprising ways I could have never imagined. After a couple of weeks of working with this API, I have compiled a list of DON'Ts, FAILs, and OMGs for your enjoyment.

How to FAIL at API

Sample Code

On their libraries page, they list an "unfinished" ruby library. I thought: no problem, even if the implementation isn't complete, at least this is something I can work with — a starting point. Boy, was I wrong.

FAIL: The library doesn't work, isn't tested, and is impressively unreadable. Is it a gem? There's no gemspec. Is it a rails plugin? There's no init.rb. Out of the box, it's not even clear how to test it in a console. Once I did manage to get it running in a console, it was pretty obvious that nothing worked. Worst of all, the way the error handling was set up, it kicked me out of irb every time a call failed (which was every time, because the code was FUBARed).

Lesson Learned: DON'T provide untested client code that doesn't even come close to working. It's better to provide nothing at all than to waste your client's time working with an unreadable mess of garbage that leads them nowhere. If you want to be in the business of distributing code, learn the language, and make sure you're distributing something that works.

Error Messages

There is absolutely no mention of error codes in the documentation for this particular API. So, I assumed that their data validation was pretty loose, and that as long as I satisfied the required parameters for a call, it would succeed. So, when I re-implemented their API client in ruby, I didn't build in any exception handling beyond the http communications layer. I was pretty confused when nothing was working.

OMG: There are error codes; they're just not documented. I dumped the messages out to inspect them. Maybe I had goofed something up on my end? Apparently the authors of this API decided that their error messages are so good that documenting the error codes was an unnecessary task. Unfortunately, my code doesn't understand english particularly well.

Lesson Learned: Document your error codes! From an implementer's point of view, missing error codes almost definitely means that your API is going to be unreliable, and thus, useless for any important task.

Security

I'm no guru, but I know enough about security to know that asymmetric encryption (or public-key cryptography) is secure as hell; that's why https, ssh, pgp/gpg, etc, choose it as their supporting technology. Using a public / private key pair, asymmetric encryption allows two parties to create a secure connection over an otherwise insecure channel without worrying about interception.

Data encrypted with a given public key can only be decrypted with the corresponding private key. So, unless the private key has been compromised, it is impossible for the communication to be decrypted by a third party. The sender of data can also use their private key to sign the message, verifying its identity; the signature can then be decrypted with their corresponding public key. It's quite an elegant system, actually.

It has some traction, too: it's secure enough for your bank, operating system vendors of all kinds, secure email vendors, and more, but apparently, the implementers of this API didn't feel public-key cryptography was adequate.

OMG: Instead, they chose to use symmetric encryption. That means, there is one single key that both parties posses, and use to encrypt, and decrypt their communications. Naturally, the flaw in this approach is that somebody in possession of the key can intercept communications with ease. So, when making use of symmetric encryption, the key has to be transferred across a highly secure channel.

FAIL: They sent me the key in an unencrypted email.

Lesson Learned: If you're going to diverge from the industry standard somewhere, especially when it comes to security, make sure to have a good reason for doing it. It wouldn't have taken much research to determine that https is an extremely secure protocol.

It's supported on nearly every system in the world. So, piggy backing on https would have meant that clients wouldn't have had to re-invent the wheel just to use this particular API.

The fact that they weren't aware of that in advance suggests that they don't understand the technologies that they're using, which is going to throw up a gigantic red flag for anybody working on integrating with their service.

EPIC FAIL: The funniest part about the whole encryption thing? HTTPS only uses public-key crypto to create a secure channel, in which to exchange the symmetric key. For performance reasons, the rest of the data is exchanged using a symmetric algorithm. The result is effectively the same, except that https is actually secure, whereas this symmetric key may have been compromised by somebody sniffing my network traffic when I grabbed my email that day.

 

Remarkably, there is more to this story, but it'll have to wait for another day. For now, what are some of your integration horror stories?

Pretty soon after starting to use has_finder, I needed to write a browse interface. That is, given a set of parameters, return all the models that match. With has_finder, it is possible to compartmentalize those parameters in to small chunks, each with names that make sense in your model's domain. You can then chain them together to get their combined scope. The only problem iswas calling them when you aren't aware, in advance, of which finders need to be called.

Having had this problem a few times, and being sick of solving it over and over again, I decided to extract a general solution from a project I'm working on.

has_browser

It's a simple plugin, with a simple syntax. Using the canonical blog example, let's imagine we want to create a browse interface to posts. We'd want the user to be able to browse by category, author, or tags, but not to be able to access any of the other finders on the Post model for obvious security reasons. To set up has_browser, we'd do something like this:

has_browser :category, :tags, :author

Then, assuming the has_finders are already written, the posts can be browsed as follows:

Post.browse(:category => 'Testing', :tags => 'activerecord', :author => 'james')

In that example, each of the finders requires an argument; has_browser also supports finders that don't. As long as the argumentless finder is present in the browse hash, it will be called:

has_browser :category, :tags, :author, :without_args => [:order_by_date, :order_by_number_of_comments]

Post.browse(:category => 'Testing', :tags => 'activerecord', :author => 'james', :order_by_number_of_comments => 'true')

Browse can also be called from association_proxies. For a multi-blog platform, we could easily restrict browsing of posts to the current blog:

@blog.posts.browse(:category => 'Testing', :tags => 'activerecord', :author => 'james', :order_by_number_of_comments => 'true')

Since has_browser returns the same proxy as has_finder, it is possible to further restrict the results of a browse by chaining finders after the browse call. With our blog, for example, we'd probably want to restrict browsing to published posts.

@blog.posts.browse(:category => 'Testing', :tags => 'activerecord', :author => 'james', :order_by_number_of_comments => 'true').published

Note: It is not possible to chain finders before the browse call.

Finally, like has_finder, has_browser is compatible with will_paginate out of the box.

Get It

Releases of has_browser will be available as a gem, which you can freeze in to your rails app (init.rb included):

$ sudo gem install has_browser

Development, of course, is at github.

A few weeks ago, Nate Wiger emailed me to ask whether I was interested in a patch for r_c. Evidently, it is possible to determine which controller actions need to be created by examining the routes that are pointing to it. Nate had also noticed that most controller actions follow a pattern:

1. Load or build an object or a collection (i.e. Post.find(1) or Post.build(params[:post])).
2. Optionally do something with the object(s) (i.e. @post.save).
3. Optionally set the flash.
4. Render a response.

Put those pieces together, and it's possible to create a controller abstraction that is aware of what's routed to it, and sets up a set of sensible defaults, even for custom actions. As you might have guessed, Nate and I have done exactly that.

For the 7 default RESTful actions, things haven't changed much. The only real difference is that you can now change step 2 (refer to list above), by passing a block to action.action.

create.action do
  @post.my_custom_save_method
end

The action block's return value determines whether the success or failure callbacks (i.e. success.flash vs failure.wants) are triggered.

For custom actions, you'll now be able to use the same DSL you're used to for customizing default actions. For example, if you had a custom action called update_collection, you might put something like this in your routes:

map.connect '/posts', :controller => 'posts', :action => 'update_collection', :conditions => {:method => :put}

Based on that route, r_c will automatically be aware of your action, and create a basic skeleton for it, following the pattern of the list above. For a collection update method, you might want to customize your action like this:

update_collection do
  before { @posts.each { |p| p.update_attributes params[:posts] } }
  update_collection.action { @posts.save }
end

That's all you'd have to do. r_c would automatically load the objects (including any parent, respecting polymorphism), and render using its internal helpers.

Note: It is also possible to modify loading by setting a block for the build accessor, which corresponds to step 1.

Highly Experimental

This is highly experimental software. I'm not sure how much I love it, since it's a bit on the magical side for my tastes. However, it is kinda neat to have your controller be aware of which actions to create, based on what is routed to it. I have been having a bit of fun playing around with this.

So, please give it a shot, and let us know how you like it on the mailing list.

Get it from github (checkout the automatic_route_discovery branch). If you aren't using git, you can get a tarball here.