The Developer Landgrab – Another Way to Look at DevOps

Developers are Insourcing

Developers have been in-sourcing tasks they’d previously jettisoned from their core functionality for a few years now and cloud computing has brought one more land-grab: operations. Agile tricked developers into caring about QA and testing, but also requirements and product management. Now, devops is “tricking” developers to care about operations. I put tricking in quotes because the developers actually want this – the good ones at least. The benefits, or at least goals are clear: delivering software that users like with a more frequent cadence. Those are the two aspects of software development that most interest me now: frequent functionality and using rapid feedback-loops to improve the user experience and overall usefulness of the software.

A Brief, Hand-wavy History

At the very beginning, development teams did everything, or at the very least were intimately involved: gathered requirements, writing the code, testing the code, running the code. IT being expensive, organizations sought to divide up those tasks into shared resources, often under different management chains. Developers, of course, also just wanted to write code: not “talk with customers about what they needed” (requirements and product management) or keep their applications humming along nicely in production (operations).

Throw in mainframe MIPs accounting and high costs, and you can see how separating out those rascally developers from expensive mainframe toys makes business sense, at least in a spreadsheet. (Side-note: with the metered pricing of cloud computing, thus far there’s no reason to think that in 10 or so years, cloud computing resources will be any less iron-gripped controlled than mainframe resources – that is, use and consumption of them will be slowed down in favor of controlling costs. We’ll see.)

And writing and running tests? What developer wants to do that?

Agile Starts to In-source: QA

One of the emergent principals of Agile (perhaps an anti-pattern) is that once the core team who wrote the software gives control of that software to another group, overall quality tends to go down. This isn’t across the board, but you tend to see that. As an individual, if you don’t somehow “own” the software, you won’t give it the same love and care that an “owner” will. This applies to QA and operations, usually not so much to product management.

Agile is always trying to get developers to do more process. To own more of the software life-cycle. It never says this outright, but cynical developers will spot it right away: “wait, my job is to write code, not write-up use cases and rank them. Let alone help take care of my software in production, in the hands of dirty users. I mean: are you going to pay me more?”

Somehow, Agile got developers to care about QA. It started small with unit testing by promising, after a huge amount of initial work, to make them more productive. Then functional tests came in, and now if you’ve got yourself wired up correctly, you can test whole use cases (or “stories” to use the term Agile hid the dread “use case” behind).

The bigger win was getting QA to be part of the development team. In reality, good QA people are often the foremost experts on the product – they’re the ones that spend hours, weeks poking and prodding it. Getting their input, starting at the beginning, is a great way to improve software. As with developers who are writing and running tests, the QA person ceases to be “just QA”: they’re one of the many owners of the software.

“I take the requirements from the customer to the developers.”

Seeking feedback and rich interaction with users hasn’t always been a strong-point of developers. They haven’t wanted to do product management (in this context: figuring out what users/customers want and then prioritizing which ones get into which release). “That’s someone else’s job” is what you’d hear. As with QA, that role has seeped a bit into developers hands as well. There’s simply not enough time in the day to write code, test, and also do product management, so it’s probably a good idea to have a whole person filling that role: but, having that person be the sole conduit between developers and users is not entirely helpful.

Thanks to the ability to run software as a service (SaaS) and mature cloud offering, a lot of the feedback teams need to do product management can now be automated. If you’re running your application as a SaaS, you can see what every single user is doing all the time. It’s like you have infinite one-way mirror usability tests going on. Public web apps (Amazon, Facebook, etc.) have known this for a long time, getting into advanced practices like A/B feature testing: let’s release two different ways of implementing this feature to sub-sets of user group and see which one results in more book sales, and then we’ll switch everyone over the better one. What you’re driving at is feeding aggregated user behavior data into the product management process.

There’s a quote I use all the time from Alterity’s Brian Sweat that I use all the time to summarize this point:

I can actually look at [a feature] and say, ‘nobody uses feature X.’ It’s not even being looked at. And it really helps us shape the future of the app which, on a desktop product, we don’t have a lot of data like that.

There’s two concepts running around here that I speak a lot about: frequent functionality and rapid feedback loops. Getting features in small chunks in production sooner and then collecting a huge amount of usage feedback from users. You’re first delighting your users by keeping your software up-to-date and more functional (a lesson learned from the consumer space where new frequent functionality and integrations with other services is key for long-term success) and also giving yourself the chance to see what works best (or worst!) with users. You can empirically improve the quality of your software. And here, by “quality” I don’t mean “bug free” (which is a less helpful definition in this context) but something more like “the ability to help users do their job better, faster, and more profitably.”

I go over these concepts a lot in talks now-a-days, most recently on requirements management and back at devoxx 2010.

Operations

Ask any developer if they’d like to take a pager and be woken up at 3am to reboot a server. You can guess what the answer is. A blank stare that says “no” in a string a four-letter words. These guys are coders, not pager monkeys.

And yet, developers are increasingly taking on that task, perhaps not seeing those bat-belt ops guys clip a pager onto the developer’s t-shirt. The speed at which cloud computing technologies and practices allows developers to get their core job done (spinning up virtual labs, getting access to resources without having to wait 6 weeks for the DBA to create a new column in a table, etc.) has sold developers on cloud computing. Throw in SaaSified parts of the development tool-chain like GitHub, and you start to see why developers like cloud-based technology so much: it speeds up their work, gives them more power (they don’t have to ask IT permission for a server, and then wait for it), and overall improves their ability to produce good software.

As developers using cloud computing technologies get closer and closer to production, you can see them starting to in-source even operations. At the bleeding edge, those using Platform-as-a-Services (PaaS) are almost forced into doing this. Indeed, as its name implies, much of what devops is wants to do it bring the operations function back to the core team.

This doesn’t suggest that you get rid of the operations people at all – the good ones at least. Rather, it means that as with QA and product management, their role moves from “keeping the lights green” to “delivering good, productive experiences.” Operations becomes one of the product owners, not just the “monkeys” who hook up wires to servers and increase disk-space.

As some point, IT became a cost-center, a provider of “services” to the business. That’s terrible for them. No one wants to be the manager of a “shared resource.” Just go ask your janitorial staff or the guy who keeps your office supply closet stocked how secure and well paid they are. Before the current crop of IT Management technologies finally ripened (Agile development, virtualization, open source, and now cloud), perhaps, taking this “services” approach was cost effective. But, now it means that IT has one way to show value to the rest of the company: budget cuts. For IT, the promise is to become top-line revenue: part of the way a company makes money, not hidden somewhere in the expenses.

(Hey, I’m a big IT Management guy, here, so realize that I’m being tongue-in-check with this whole “monkey” thing. If you want equal servings the other way: “Developers”? Those are the guys who write bugs, right?)

Automation

All of this in-sourcing relies on automating parts of what’s being brought back to the core development team: automating testing with the write testing frameworks and continuous integration tools, automating understanding how users are interacting with your software with cloud-driven feedback, and automating IT management in production.

Evaluating Offerings & Programs

To me, this understanding that automation is key is critical because it means what I’ve been seeing here and there is driven by actual, new technology: not just vapor-ware and Unicorn-meat. It means something else great: if someone comes peddling devops or some other whacky cloud-based way to improve your software delivery process, you can ask them to show you the tools – “where’s the beef?” and all that.

Avoiding Outsourcing

[I]f you worked for AT&T in my day, it was a great bureaucracy.  Who in the hell was really thinking about the shareholder or anything else?  And in a bureaucracy, you think the work is done when it goes out of your in-basket into somebody else’s in-basket.  But, of course, it isn’t.  It’s not done until AT&T delivers what it’s supposed to deliver.  So you get big, fat, dumb, unmotivated bureaucracies.

—Charles Munger

There’s a tremendous amount of “cultural change” (read: getting employees to do things differently and like it) needed, but the hope is that this trend of developers in-sourcing tasks means that cultural change will be possible. It doesn’t happen often, but many parts of the IT department (developers, QA, and a bit of ops) are actually looking for new ways of doing things. The best part is that the promise – that some folks have been realizing – is that IT can become part of the business, not just a cut-to-the-bone cost center that keeps email up and running and AntiVirus software updated.

Or, to put it all more simply: Conway’s Law.

Disclosure: GitHub is a client.