Microservices and the migrating Unix philosophy

A core Unix tenet pioneered by Ken Thompson was its philosophy of one tool, one job. As described by Wikipedia:

The Unix philosophy emphasizes building short, simple, clear, modular, and extendable code that can be easily maintained and repurposed by developers other than its creators. The philosophy is based on composable (as opposed to contextual) design.

This philosophy was most clearly visible through the existence of a substantial set of small tools designed to accept input and output such that they could be chained together in a series using pipes (|) a la `cat file | sed | tail`. Other instantiations include the “everything is a file” mentality and the near-universal use of plain text as a communication format. Both of these encouraged the sharing of a common toolset for accessing and processing data of all types, regardless of its source.

Following up on Steve’s writeup on microservices last week, I figured I’d better get this post out the door. I’ve had the ideas on the back burner for a year or so, but the burgeoning interest in microservices means now is the right time to tell this story.

The “new” composable world, same as the old one

Composability has made a resurgence in the past couple of years, inspired in part by the now-infamous 2011 post by Steve Yegge. It described Amazon’s move to a service-oriented organization where all data between teams must be transferred via API rather than emailing around Excel spreadsheets.

We’ve seen this pervade through to the design of AWS itself, and the ability of Amazon to keep up an astonishing pace of feature releases in AWS. More recently, the PaaS community, incited by the cries of Warner’s Jonathan Murray for a composable enterprise, has begun talking specifically about the virtues of composability (although it’s enabled it implicitly for much longer).

Another area where composability’s had a huge impact is IT monitoring. The ELK stack of Elasticsearch, Logstash, and Kibana as well as the #monitoringsucks/#monitoringlove movements (see Jason Dixon’s Monitorama conferences) serve to define the new composable monitoring infrastructure. It exists as a refutation of the old-style monolithic approach best embodied by the Big Four of HP, BMC, IBM, and CA. This movement further refutes the last revolution led by the still-dominant open-source alternative, Nagios, and the kingmakers-style bottom-up approach that enabled Splunk’s success.

Composability embodies the Unix philosophy that I began this piece by describing, and we’re now seeing it move up the software stack from its advent in Unix 40+ years ago at Bell Labs.

Granularity collapses when unneeded

The key point I want to make in this piece, however, is that composability does not and can not exist everywhere simultaneously. It just won’t scale. Although the flexibility that a composable infrastructure provides is vital during times of rapid innovation, such that pieces can be mixed and matched as desired, it also sticks users with a heavy burden when it’s unneeded.

As developer momentum and interest continues to concentrate up the stack toward cloud, containers like Docker, and PaaS, and away from concerns about the underlying system, that lower-level system tends to re-congeal into a monolith rather than remaining composable.

We’ve seen this happen in a number of instances. One prime example is in the Linux base system today, where systemd is gradually taking over an increasing level of responsibility across jobs formerly owned by init systems, device managers, cron daemons, and loggers. Only the first of those has seen significant reinvention in the last decade or so, with alternatives to the old-school SysV init system cropping up including Upstart, OpenRC, and systemd. But with systemd’s gradual integration both horizontally and vertically into the kernel and the GNOME desktop environment, it’s quickly becoming mandatory if you want one option that works everywhere.

Even beyond that, the advent of container technologies and distributions like CoreOS mean that users care increasingly less about the underlying system and just want it served to them as a working blob they can ignore. This is a similar driver to what Red Hat’s doing with CentOS, by attempting to provide a stable underlying firmament that you treat essentially as a large blob to build applications upon.

Another example is in X.Org, the primary Unix window system. Ten years ago, it was undergoing a period of rapid innovation driven in part by its recent fork from XFree86. The entire monolithic codebase was modularized into hundreds of separate applications, libraries, drivers, and the server. But now that community has realized it’s difficult to maintain so many stable APIs and the cost is no longer worth the benefit, so it’s considering bringing drivers and server back together into a mini-monolith of sorts.

Think of it as an accordion. Parts of it expand when there’s rapid innovation underway, often driven by external forces like the advent of the cloud, and then contract again when consensus is generally reached on the right solution and the market’s settled down. Then another part of the accordion expands, and so ad infinitum.

Disclosure: Amazon, IBM, Pivotal, Splunk, and Red Hat are clients. Microsoft, HP, and CA have been clients. Docker, Elasticsearch, CoreOS, Nagios, BMC, and Warner Music Group are not clients.