Category Archives: Lean Software

Continuous Deployment for Existing Software – Do it Now

I recently wrote about doing continuous deployment from day one with a software project that had pretty good test automation and (if I do say so myself) a somewhat modern and decent architecture.

I’ve recently transitioned to working on a fairly ambitious overhaul of an existing project that has been around for a few years. The software has already reached a (mostly) working steady state and is used every day by real customers.

The very first thing I did, ahead of making any functional changes, was automate the deployment system using essentially the same kind of test gauntlet and approach for zero-downtime deployments I was using at Victors United. I added some very basic test automation, starting from the top of the test automation food pyramid with a simple “is the web server able to execute code, connect to the database server, and return an ‘OK’ rsponse?” test. From there, I’ve working my way down, just getting around to writing my first “pure” unit test just today.

Only after I was confident that I could do zero-effort and zero-downtime deployments that

  1. Wouldn’t completely destroy the system (I had a test for that) and
  2. Could be rolled back very easily if something went screwy

did I make my first functional change to the software.

And then, I made the smallest functional changes that could work. I tested them locally, added some test automation around them, and then let the automatic deployment system do its magic.

A few years ago, I was interested in doing continuous deployment, but I felt that the level of test automation wasn’t good enough, I wanted to be at least at around 90%.  Now I think that the less test automation you have, the more important it is to start doing continuous deployment of tiny incremental changes right away.

If you are working on software that’s actually in use and are interested in doing continuous deployment someday, there’s no better someday than today. Seriously. Do it now. It will start making your life better almost immediately.


Always Be Shipping – Real-World Continuous Deployment

Have you ever edited code directly on a production server? I’ll admit that I have, years ago, before I knew better. It’s easy and fast and gets emergency fixes out there as quickly as possible. You get to know what works and what doesn’t because you make tiny changes. If you’re working with a system that anyone cares about, it’s also dangerous and stupid.

I’ve also worked with development organizations that take the opposite extreme. Even the most trivial server updates needed to be scheduled weeks in advance. In order to push the updated code, you had to notify everyone, take all servers down in the middle of the night, run through a long series of manual steps, and then do a lot of manual testing. When things in production aren’t quite how they are in the test environment, this is followed by a hasty rollback or panic-induced middle-of-the-night attempts at bug fixing.

Both of these extreme approaches are (maybe) appropriate in some environments. But neither are appropriate for a new web startup that’s trying to move quickly and still provide a reliable trustworthy experience to their customers.

At the previous company I worked for, we often talked about IMVU-Style Continuous Deployment as our ideal process, but it was always something “in the future”. We were hesitant (some of us more than others) to do automatic deployment without at least a little manual intervention. We always wanted to have more test automation, or a smoother deployment system, or whatever.

Since it seemed to be hard (for me anyway) to move to an existing development organization to a continuous deployment system, I started to wonder what would happen if you do it that way from day one? I got a chance to answer that question when I co-founded a startup last year. One of the very first things I did, before we had anyone using the site, was to create an solid automated test & deployment system that was as fast and easy as possible without being dangerous and stupid.

Here’s the basic workflow that happens in our office multiple times every day.

Step 0. We make changes on our local dev envirnments, with a bias toward making the smallest possible change that adds value. That could be a bug fix, correcting a typo, a stylistic tweak, a stubbed-out new feature, whatever. Once I’m confident in my local (manual and automated) testing that the change is good (not perfect, not feature-complete, but just better), I push that to my github repository.

From there, the continuous integration server pulls down the new code and does the following:

Step 1. Does the code still compile. If not, the build fails and everything stops.

Step 2. The build agent runs the unit tests (where “unit tests” are defined as tests that run with no external dependencies, these take just a few seconds). For anything that does require external (generally slow) dependencies (network API, databases, filesystem, whatever) we use test doubles (fakes, mocks, stubs, whatever).

This first feedback loop is about catching and preventing errors in core business logic and is generally measured in seconds, not minutes.

Step 3. The build agent runs a set of tests that rebuild the database from a reference schema and exercises all of the repository layer code.

Step 4. The build agent runs another set of tests that test our dependencies on external APIs (twitter, geolocation services, etc.)

These two sets of tests run in a few minutes, so the feedback loop isn’t quite as tight, but it’s still pretty darn fast. Basically, they make sure that the assumptions that we make with our test doubles in our unit tests aren’t totally wrong.

I’ve written about these sorts of automated test distinction a couple of years ago, in a post about the Automated Testing Food Pyramid.

Step 5. Provided that the entire gauntlet of tests has passed so far, the code gets automatically deployed to a staging server.

Step 6. There’s an additional set of tests that run against the staging web server. These tests can find configuration problems and code that just does the wrong thing in a web context. These tests are pretty shallow. They hit all of the user-facing pages/JSON endpoints and fail if anything is totally broken.

Step 7. The build artifacts are copied from TeamCity to a new folder on our productionvserver, and then the web server is reconfigured to serve from that folder instead of the folder it had been serving from.

At this point, we’ve verified that the core business (game, in this case) is OK, verified that the persistence stack works as expected, that our integration with external APIs works as expected, and that the code doesn’t completely break in a web context. We’ve done a zero-downtime deploy to the production web server.

That’s cool, but we’re not quite done yet. There’s two more steps.

Step 8. Run a set of tests against the production web site to make sure that all of the pages that worked a few moments ago still work.

Step 9.  Have external monitoring systems in place, so if your changes make things slow or unresponsive. You’ll know.  We use pingdom.

Yikes! There’s a bunch of distinct steps here, and it seems really complicated (because it is). But it’s all totally automated. All I need to do ?

git push origin master

Because there’s zero deployment effort on my part, I do this all the time.  I find it very energizing to know that I can just do stuff in minutes instead of hours or days or (heaven forbid) months.

If (when) something goes wrong, I’ll know immediately. If a bad bit of code manages to roll through the test gauntlet, I can roll back easily (just reconfiguring the web server to use the last known good set of code). I’ve only had to roll back a couple of times over the course of several months and 326 deployments.

Just like when the folks at IMVU wrote about this process, I’m sure that some people in the audience convinced that I’m a crazy person advocating a crazy way of working. Here are the objections I’ve heard before.

Yeah, but this doesn’t give your QA team any time to test the code before it goes out!
We’re a small startup. We don’t have a QA team. Problem solved.

Yeah, but isn’t that incredibly dangerous?
No. The safest change you can make to a stable production system is the smallest change possible. Also, we design the individual parts of the system to be as encapsulated as possible, so we don’t tend to have crazy side-effects that ripple through and create unintended bugs.

When we make a change or add a new feature, we can manually test the hell out of that one thing in isolation (before checking in) instead of feeling like we need to spend a lot of time and effort manually testing everything in order to ship anything.

Yeah, but what about schema changes?
For schema changes that are backwards compatible with the code that’s out there (e.g. new tables, whatever). We have a simple system that executes the appropriate DML on application startup.

For non-compatible schema changes and things like server migrations, we have to take down the site and do everything manually. Fortunately, we’ve only had to do that twice now.

Yeah, but you have to spend all of that time writing tests. What a waste!
The time we spend writing tests is like the time a surgeon spends washing their hands before they cut you open. It’s a proven way to prevent bugs that can kill you.

Also, we get that time back, and then some, by not having to spend nearly as much time with manual testing and manual deployments.

Yeah, but what about big new features you can’t implement in just one sitting?
Traditional software development models (both waterfall and agile methods like Scrum) are organized around the idea of “multiple features per release (or iteration)“. Continuous deployment is organized around the idea of “multiple releases (or iterations) per feature“. As a result, we end up pushing a lot of code for features that aren’t done yet. For the most part, these simply unavailable through the UI or only exposed to users who have particular half-built features associated with their accounts. I credit Flickr with this general approach.

Yeah, but that might work for a solo developer, but it can’t work for a team.
There are actually three developers on the team.

Yeah, but I’m sure this only works with very experienced developers
One of the guys on the team has only been programming for the last year or so and hasn’t ever worked on a web project before. Tight feedback loops help everyone.

Yeah, but what about code you want to share with other that you don’t want released?
We use github, so creating additional branches and sharing with them is trivial. We also have a dedicated “preview” branch that triggers a parallel test/deploy gauntlet that sends code to a staging server instead of the production servers.

Yeah, but this will never work at my organization because…
OK. That’s cool. Don’t try it if you feel that it won’t work for you. You’re probably right. You’re not going to hurt my feelings either way. I found something that’s working really well for me, and I want share my experience to show other people that it’s possible.

What this really means

Half of what makes this process work is that we’re honest with ourselves that we’re human and will make mistakes. If we have multiple tight feedback loops between when we’ve broken something and when we know we’ve broken it, it’s faster and easier and cheaper to fix those mistakes and prevent similar mistakes from happening again.

The other half is the idea that if you design, implement, test, and release exactly one thing at a time, you know with certainty which change introduced a problem instead of having to ask the question “which of the dozen or so changes that we rolled out this month/sprint/whatever are causing this problem”.

About the site

Victors United is an online turn-based strategic conquest game. You can play asynchronously or in real time. You can play against robots or humans. If you’re playing against humans, you can play against your friends or against strangers. Unlike some other popular web based social games that I don’t like to mention, this is a real competitive game where strategy and gameplay matter.

About the tech 

The tech here is kind of beside the point. This general approach would work just as well with different technology stacks.

The front end is HTML5 + JavaScript + jQuery. The backend is IIS/ASP.NET MVC2/SQL Server/Entity Framework 4. Our servers are hosted in the SoftLayer cloud. External monitoring is provided by pingdom.

The test gauntlet is a series of distinct nUnit assemblies, executed by TeamCity when we push new code to GitHub. There’s a single custom PowerShell script that pulls down the build artifacts and tells IIS to change what directory it serves code from.

Not Sure the World Needs Another Named Prescriptive Methodology

This, on the very same day that some tries to create a definitive list of methodologies, Net Objectives (the one-time sister company and training provider for the company I contract through) has just announced their new flavor/version/whatever of Scrum, which they are calling Scrum# (pronounced “sharp”, like C-sharp). It is (as far as I can tell) essentially Scrum + Lean Thinking + Emergent Design + Focus on scalability. Many things that a lot of us are doing already.

To be fair, the approach they are taking is exactly the one that I would advocate, focusing on and being mindful of the principles/values of what works and what doesn’t work. I’m just not sure I can get behind the name.

Respect for People, and Mean-Spirited Trash Talk

I was at lunch with a few developer friends a few weeks ago when a story of rather horrible code experience came up. Someone mentioned that they should send it to The Daily WTF.

“I don’t really like The Daily WTF.” I said, off-handedly.

Everyone looked at me as if I had said that I didn’t like kittens, or apple pie, or democracy.

“What?” one of them asked.

“I think it’s too mean-spirited” I said, lamely.

I didn’t always feel this way. My position on code-quality-improvement-via-harsh-mocking changed after working with a great developer (the kind that reminds me that I am merely a good developer). He was very focused on code quality and would say things like “I should hop on a plane and fly to where this guy who wrote this is and punch him in the nose for writing this method this way” or just cry out “Jesus Christ!” when he found something that he didn’t like.

Even though we always got along, it made me nervous. If this is what he’s saying about other people’s code, what’s he saying about my code? It created a sort of climate of fear. Not only that, it didn’t do any good to actually improve the codebase as a whole. We still had a team member who insisted on writing things in his own way. He brushed off valid concerns as “well, that guy just hates everything”.

This culture of harsh criticism is what leads people to be generally closed and insecure. Personal example: I thought about releasing the source code to my little Survival-Horror Asteroids game, but I haven’t yet, for fear of having some sharp-tongued rock star developer come along and tear me to pieces. 

None of this is to say that you shouldn’t be discerning. I’m as concerned about the distinctions between (and definitions of) good code and bad code as much as anyone else. I’ve literally had nightmares about having to work with a particularly nasty procedure. I’ve just learned to not be so insecure that I have to insult and belittle people over it.

This isn’t just a problem with developers, either. I enjoy reading about design and usability, and I found a great article about a particularly mean-spirited Flickr thread in response to a screenshot from an iPhone app. The app itself may not look great, but the developer had done nothing to warrant the vitriol of the responses. 

How to make things better:

1. Be kind and respectful, even to inferior coders. When I first read about the Lean Software Development concept of “Respect for People” I read it as “Respect the developers, especially Martin”. But it’s actually omnidirectional. Respect the other developers. Respect the folks in marketing, respect the pointy-haired-bosses (the first step, of course, is to stop calling them pointy-haired-bosses). 

2. Think incrementally. If you’re working with a developer who needs to improve, select a few distinct values-based things that need to be changed when you provide feedback.  If you try to fix absolutely everything all at once, you’ll fail.

3. Be constructive. Comments like “my eyes bleed” or “you, sir, are too stupid for words” don’t help do anything besides inflate the ego of the insult hurler. Comments like “if you do an extract method refactoring instead of copy-paste, it will be easier to understand and maintain” or “some stronger lines of alignment would make this easier to understand” are helpful.  

3a. (update, July-19-08) Realize that reasonable people can have different opinions and pick your battles. I’ve stopped getting hung up on stylistic things such as formatting or casing. I’ve learned to adapt to the consensus of the project. It’s a useful skill.

Also, realize that there’s generally no one right way, most things are tradeoffs. In this de-normalization example, the author is looking into trading one kind of pain (arguably slow joins and more complex queries) for another kind of pain (risk of data anomalies). It’s generally not the tradeoff that I would make, but it surely  doesn’t warrant the “you have no business designing databases” hate that he got in the comments

4. Realize that mistakes are OK and a necessary part of progress. Some developers (including me sometimes) get defensive or insecure when people find bugs in their work. The best developers I’ve worked with never are. It’s not that they never make bugs, but they don’t take their bugs personally.

Agreement on the Lean-Agile Connection

I’ve been arguing (mostly unsuccessfully) for some time that Lean Software Development and Agile are two different intellectual frameworks for understanding the same underlying concepts. The artificial distinctions that people draw are often based on connotations of the words themselves, such as “agile is about fast while lean is about cheap”.

Now, in a shameless appeal to authority, I find that Martin Fowler sees things the same way. From a recent post:

You can’t really talk about them being alternatives, if you are doing agile you are doing lean and vice-versa. Agile was always meant as a very broad concept, a core set of values and principles that was shared by processes that look superficially different. You don’t do agile or lean you do agile and lean. The only question is how explicitly you use ideas that draw directly from lean manufacturing.

Right on!

One thought on drawing ideas directly from lean manufacturing: The ideas that work in both manufacturing and software (such as empowerment, incrementalism, tight feedback loops) don’t necessarily work in software just because they work in manufacturing. That’s a naïve mindset that can lead you to all sorts of bad conclusions.  Some ideas work in both due to the nature of the people involved in both. 

It’s easy to get ideas from manufacturing as it’s a human endeavor that’s more mature and has been very well studied and documented.  An observant and mindful person could surely draw lessons from all sorts of places.  It’s possible that someday we’ll all understand software development well enough that we won’t need to borrow concepts from other disciplines. It’s clear to me that we aren’t there yet, though.

Finally, I get to talk about Lean Software…

…even if it’s only for a few seconds.

I occasionally do training seminars on software development technologies. Some are internal to my company, some are external.  Most are on the more technical aspects of Agile software (using the definition of Agile as “any modern software development practices that work”) such as TDD, design patterns,  and refactoring. What I really care about and want to talk about are Lean Software Development and other more overarching values-based approaches to making software, but there doesn’t seem to be the same level of interest. Until today.

This morning, while I was in the elevator at work, I overheard one passenger (total stranger) ask another “so, what does “kanban” stand for anyway?” the other passenger said she didn’t know.

So I pounced!

I managed to cram a pretty good history of how kanban came from the Toyota Production System into a few floors of elevator travel. And I get the sense of satisfaction that comes from making someone’s day a little more strange.