Blogs

RedMonk

Skip to content

Go: the emerging language of cloud infrastructure

Over the past year in particular, an increasing number of open-source projects in Go have emerged or gained significant adoption. Some of them:

  • Docker
  • Packer
  • Serf
  • InfluxDB
  • Cloud Foundry’s gorouter and CLI
  • CoreOS’s etcd and fleet
  • Vitess, YouTube’s tooling for MySQL scaling
  • Canonical’s Juju (rewritten in Go)
  • Mozilla’s Heka
  • A Go interface to OpenStack Swift
  • Heroku’s Force.com and hk CLIs
  • Apcera’s NATS and gnatsd

Although this seemingly shows the importance of Go, I have a background as a scientist so I hate to be influenced by random anecdotes. It therefore raised the question to me of whether this was a real trend or just observation bias.  To answer this question, I went to Ohloh’s huge data set of more than 600,000 free and open-source software (FOSS) projects. Below, I plotted a number of different ways to look at Go adoption over time:

ohloh_20131216_go

Data from Ohloh tracks new code added during each month. This plot is from a dataset as of 20131216.

As you can see, Go’s rapidly closing in on 1% of total commits and half a percent of projects and contributors. While the trend is obviously interesting, at first glance numbers well under one percent look inconsequential relative to overall adoption. To provide some context, however, each of the most popular languages on Ohloh (C, C++, Java, JavaScript) only constitutes ~10% of commits and ~5% of projects and contributors. That means Go, a seemingly very minor player, is already used nearly one tenth as much in FOSS as the most popular languages in existence.

One of the aspects I found most interesting about the marquee projects I mentioned earlier is how many of them are cloud-centric or otherwise made for dealing with distributed systems or transient environments. Go’s big selling point is concurrency according to one of its designers, Rob Pike (the same one who coauthored the famous “The Unix Programming Environment“). That make it particularly gratifying that people writing projects in Go seem to see it the same way.

Cloud infrastructure is famously complex and requires a great deal of effort to build a truly reliable, scale-out architecture because everything needs redundancy and coordination at the software level rather than the hardware level. Thus tools like Netflix’s Simian Army, one component of their increasingly full-featured platform that’s still waiting to be packaged up, have emerged to provide acid tests of cloud software. On the other side, an underappreciated aspect of PaaS (Platform as a Service) is its improvements not just to developer productivity but also to operator complexity by providing similar benefits at a higher level as Go does at the code level. There’s a lot of value to a packaged solution that handles the complexity of concurrency, coordination, and reliability in a gray-box fashion that enables transparency without requiring manual composition of an entire infrastructure.

Tooling that can ease the complexity for both new entrants and existing users of the cloud will continue to gain prominence at all levels of the stack, whether it’s languages like Go, middle ground like the Simian Army, or higher-level options like PaaS.

Update (2014/03/19): Added Heroku, Apcera

Disclosures: Pivotal, Black Duck, Heroku, and a number of OpenStack vendors are clients. Canonical has been a client. Docker, Hashicorp, InfluxDB, CoreOS, Google, Mozilla, Apcera, and the OpenStack Foundation are not.

by-sa

Categories: adoption, cloud, devops, go, open-source, packaging, programming-languages.