Concept: the Developer Blind Spot

Concept #1. Knowledge Work

Software developers are largely “knowledge workers”, which (broadly) means that the job they do day-to-day is something that their manager can’t do, at least not as readily.

Note: this isn’t the typical tribalist anti-business developer rant. I’m not talking about “pointy haired bosses”. I’m talking about the fact that much of the time, developers are managed by people who either aren’t currently or haven’t ever been, software developers themselves.  I think that’s fine, but it leads to something that I’m calling the “developer blind spot”

Concept #2. Signal Attenuation


Even the best transmission medium has some signal attenuation.

As an idea goes through multiple people, it can change subtly, so an idea such as “for my service types, I like to encapsulate construction instead of using ‘new foo()’ to get an instance, so my consuming code isn’t tightly coupled to a specific concrete implementation” gets mangled to become “never ever use ‘new foo()‘” which a developer will hear and say “WTF? That doesn’t even make any sense!“. Because it doesn’t make any sense, not once the signal has degraded that far.

Concept #3. Fashion and Skepticism

Many ideas come and go, and it’s time consuming to try to differentiate solid ideas from fads, snake oil, and deep-sounding fluff that some executive got from a breathlessly enthusiastic book he read on an airplane.

In a perfect world, executives would limit their airplane reading to trashy fiction.  Feel free to contact me for specific suggestions as I read a lot of really trashy fiction that I don’t exactly feel comfortable blogging about.


Show. Don’t tell.

If you’re managing or leading a software team and want your team to do things differently, it’s possible that your (brilliant) advice will hit your team right in their blind spot and be ignored.  If you really want to plant an idea into a developer’s head, you need to obliquely inject it, Inception-style. Demonstrate new ideas in a proper real-world context, without appeals to top-down authority, blog posts written by random dudes (especially mine), or light airplane reading. Connect real-world developers with their peers and let them have meaningful conversations without latency and signal attenuation.  Give people enough creative space and room to experiment that they feel invested in these ideas. Anything new must actually make sense with the dozens of micro-scale cost-benefit decisions that good developers are making all day every day. Otherwise, they’re just checking mysterious magic checkboxes on some mysterious magic checklist because some manager says so for mysterious magic reasons.

Thinking of starting an Innovation Lab? Here’s what not to do.

I recently got to experience the exquisite joy and excruciating pain of helping to build a new software team for a large, well-established company. We had the mandate to do things differently and to help other teams modernize. While a lot of things went very well, I think the most helpful, or at least entertaining, things I can write about are what didn’t go well and why. So, at taking the risk of being a prescriptive hypocrite, here’s a list of what I think you should not do.

1. Do not muddle the message.

It’s about Agile! It’s about Lean Startup! It’s about DevOps! It’s about a green-field project! It’s about The Cloud! It’s about Open Source! It’s about automation! It’s about whatever personal pet perspective you care about.  Yes, when your mandate is to innovate, you’re going to work with different tools and techniques, but If you’re known as “that cloud team” or “that agile team” you’ve lost control of your own identity. You should be about innovation, experimentation, and letting go of the way things have always been done. Everything else is just implementation details.

2. Do not change the experiment halfway through.

A few months into the project, a new executive informed us of the new reality. “You guys have been operating all on your own. That ends now.” Looking back on it now, that was the moment when all value in the experiment was lost. We started working with the existing processes (which nearly everyone agreed were terrible).

Between the radical process change and the lack of clear messaging, any person could take any result and use that to validate their own existing opinions.  If the project were a huge success, it was because of technology X. If it were a failure, that was obviously because of process Y.  

I don’t expect a lot of scientific rigor, but seriously….

3. Do not keep things a secret.

I fear that I’m being a little unkind to the anonymous executive in the point above. He was reacting to the fact that we were completely opaque.  Apparently (I didn’t know this at the time) other groups in other offices were either not informed that we existed or specifically told to not interact with us.  A more sustainable approach would have been to have full transparency, along with broad autonomy, from day one. Other groups should know what’s going on, and even to give advice and suggestions, but not to swoop in and micro-manage.

4. Do not piss all over other people.

One of the first things that our team did was to look at an existing project underway from an existing team and to throw it away entirely. From a purely technological perspective, that may have been a good decision, but it was, if I may speak plainly, a total dick move. That, coupled with our opacity, created the perception that we were a bunch of elitists. I always wanted to be elite, but not elitist.  We never wanted to be a group of better technologists. Just technologists with a different mandate and a different toolkit.

The boastful and tone deaf claims of “look at this team getting done in six months what that other team spent several years on” didn’t help matters. If anyone on that other team reads this. You have my sincere apologies.

5. Don’t conflate being an “innovator” with being an “agent of change”.

If your organization is large enough that an innovation lab makes sense, you’ll also need change agents who can evangelize and support changes. These are different things. Your change agents can point at what the innovation lab is doing, which will help work around the developer blind spot, but being a change agent requires more patience and political savvy than being an innovator.

Change agents are there to dismantle the dysfunctional engine, innovators are there to create fuel for new engines.

Software Architect as Hands on… Something

One of the good things about writing is that it forces you to organize your thinking. While organizing my thinking around the role of dedicated “software architects” I actually changed my opinion. Slightly.

My original extremely persuasive argument was that the main goal of a software architect should be to minimize the day-to-day pain of the developers on the team and should therefore have hands-on involvement with the code so that the pain is felt and understood.

While reading that over, I realized it was a bit too myopic and dev-centric (sorry, I’m only human) so I broadened the goal to minimize the day-to-day pain of the cross-functional team that is responsible for developing and testing and deploying and supporting the product. After all, Architectural decisions that make things easier for those developing the product but harder for those supporting the product are most likely bad decisions. You know, the whole “optimize the whole” concept that goes all the way back to the Toyota Production System.

So then, I guess a good software architect would be someone who still feels the day-to-day pain of their architecture decisions as an actual hands-on-contributor to the cross functional team, regardless of discipline. Ideally it should be someone who has gone through at least one cross-disciplinary crop-rotation.

Crop Rotation as a metaphor for interdisciplinary software work

As long as you don’t abuse them (mostly by implicitly imposing constraints from the source concept), metaphors can be useful shorthand to talk about and explore ideas. I think that if you’re thoughtful, you can use metaphors from just about every source. I go back to biological metaphors a lot: thinking about ecosystems, monocultures, germ theory, immune response, etc. One of my very favorites is the agricultural practice of crop rotation.

If you didn’t grow up on a farm (as I sort of did… long story) the concept may be foreign. The basic idea is to use the same piece of land to grow different plants over sequential seasons. You know, rotate the crops. There are a few reasons to do this. One is that different plants interact with the soil differently. Some pull nitrogen out, some put nitrogen in. It also helps with pest control, as bugs who like to eat crop X might not like to eat crop Y. Wikipedia has a good writeup.

I started thinking about this metaphor many years ago after I was on a project where the dedicated tester left and I had to step in and be entirely test-focused for several months. It was a good experience for me. For one, I learned that even though I had a QA background (software testing was my first real job) I wasn’t half as good at being a test-focused developer (aka SDET) as the guy I was trying to replace.  I had to learn some different tools and techniques and it felt great.

The surprising part was, after this project was over and I was doing lead development on a different project, I was a better developer as a result of it. Like the nitrogen left behind in the soil after growing legumes, validation strategies were left behind in my brain after letting myself focus on the test discipline.

I like to imagine a healthy-sized cross-functional team where the different team members deliberately rotate through which particular discipline/kind of effort that they “own” or pay attention to. The rotation could be along the frontend/middle/backend axis, or the dev/test/support axis, or whatever distinct kinds of efforts the team requires.  Even though there is a meaningful cost to letting your people get up to speed in multiple related disciplines (just as there is a meaningful cost in planting different crops from year to year) the benefits in terms of big-picture vision, creativity, preventing burnout, and team empathy could be huge.

Let’s look inside this complex opaque thing, shall we?

A few months ago, Mrs. Cron insisted that I go to the Emergency Room. She insisted that writhing around in pain on the bathroom floor isn’t a great way to spend an evening. She eventually brought me around to her way of thinking. Persuasive woman, Mrs. Cron.

I noticed that after giving me some industrial-strength painkillers, most of the effort was spent trying to look inside my generally opaque body without cutting me open (thanks!). There were blood tests, urine tests, ultrasounds, x-rays, MRIs, CT scans. Probably other tests as well, I don’t remember too clearly thanks to the aforementioned industrial-strength painkillers.

It turns out that I had gall stones, and they needed to remove my gallbladder. When it was time to cut me open, the surgeon already knew exactly what he would find inside and how to take care of it. The incisions were small and I recovered quickly. I’ll never be an underwear model, but that was never a reasonable goal of mine anyway.

The parallel to professional software development is pretty obvious. Many of the tools that we use are about looking inside a complex opaque thing without having to cut it open. We use debuggers, profilers, log analyzers, static analyzers, code coverage tools, and the like. Just like how the ER staff had a diverse set of tools for looking into me, it’s good for coders to have a diverse toolkit for looking into an application.

I just started using the free tier of New Relic for keeping track of how a very complex (and yes, very opaque) legacy application is behaving and it is my new favorite thing in the universe.

What exactly is culture and why am I wearing a suit today?


Alas, Preston just had to one-up me with the bow tie.


Like I am on every Friday, I’m dressed up today. Today I’m wearing a vintage two-button pinstripe suit with a paisley tie. It’s Fancy Friday, which emerged years ago when I started working at the Cheezburger network. It’s a bit of the culture that I’ve carried with me since then.

Why? There are a few reasons.

1. It’s an oblique “screw you” to the entire idea of corporate dress codes and “Casual Friday”.  I once worked for a small startup that was bought up by a larger more established company. One of our new corporate overlords came in and saw that I was wearing an aloha shirt and said “You would LOVE our office in California! We do CASUAL FRIDAYS!” I left soon after that.

Every day is casual day when you look this good.

Every day is casual day when you look this good.


2. Collecting formal wear (specifically ties and cuff links) makes it easy for your loved ones to get you gifts. They don’t have to worry about if I have one already (because I do) or if it will fit (because it will). Today I’m wearing 3D printed cuff links modeled after the classic anglepoise lamps that were a gift from Mrs. Cron.

3. I like how mirrors look when I’m dressed up.

4. (and this is the most important). I have come to the understanding that culture is the set of choices you get to make without having to explain yourself. When I first show up at a technology job wearing a suit and tie, I have to explain myself. Always. It just isn’t part of the culture.  After a few weeks go by, Fancy Friday becomes part of the culture. Sometimes people join me in dressing up, as my old friend Preston did today, but mostly I stop having to explain myself because the culture absorbs this persistent choice.

The public choices you make over and over will influence the culture of your organization. This concept also applies to choices more important than what you wear. If you make a deliberate choice to take the time to help someone instead of narrowly focusing on your own tasks, that informs culture too.




Tagged , ,

Update on Continuous Deployment a few months in

On average, I’m doing just under four totally automated zero-downtime production deployments per day since I started doing continuous deployment on this project in early August 2011.

Obviously, that’s an average, so some days there are more, and some days there are fewer.

How many times during that span has improperly tested code broken things in a way that impacted users? Just once, and it was fixed in just a few minutes by adding back the code that I over-zealously removed.

Also in that time, we went from 0% automated test coverage to 18% automated test coverage, refactored a lot of duplicate code, and added new features.