Automated Test Distinctions: The Food Pyramid

This is something I came up with a while ago when I was working on a large integration/orchestration project for a now-defunct MVNO (mobile virtual network operator). It was a pretty complex project, which had many external collaborators and end-points into the system. The project was having some issues with being able to reliably deliver incremental functionality on time. I had a directive to use my judgment to essentially turn the project around.

I had just come off a project which had excellent pure unit test coverage, but not very good overall test coverage with actual collaborators. The integration points were much more difficult and painful than they needed to be. On the other extreme, I inherited a bunch of existing non-unit tests as part of the MVNO project. These were brittle, took a long-time to run, and failed for a bunch of different reasons. As a result, they weren’t run very often, became neglected, and eventually worthless.  There was no continuous integration in place, so the feedback loop between breaking something and finding what you broke was very loose.

To make matters just a little more dicey, I was working with another test-focused developer who had a strong preference for writing epic end-to-end tests, while I was trying to make the code more able to be tested in isolation. I resented the fact that “his” tests were keeping me from running the tests as part of the cruise control build.

I came up with what I called the “Food Pyramid” as a way of balancing these different forces and points of view. I broke the one massive test project into three distinct assemblies with their own goals.  The differences between the projects are best explained in a series of pictures:

So, did it work? Yes and no. We eventually had much smoother and more confident deployments thanks to the end-to-end tests. I also managed to refactor and add new code more quickly with fewer bugs thanks to my unit-test safety net. But, just as I was starting to get a handle on the project, the client filed for bankruptcy and stopped paying their vendors (which unfortunately included me). It seems that having a viable business model trumps all.

I have, however, used these and similar distinctions successfully on subsequent projects, and I know that at least one development organization I’ve explained the idea to has adopted the pyramid as part of their overall testing strategy.

One thought on “Automated Test Distinctions: The Food Pyramid

  1. I have nothing to add to this post… I just wanted to give you a head’s up that I’m going to pillage/plunder/borrow from this post to coach/train people around me (I’ll credit/cite you, don’t worry). I really like how you visualized this.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: