Donnie Berkholz's Story of Data

Go: the emerging language of cloud infrastructure

Share via Twitter Share via Facebook Share via Linkedin Share via Reddit

Over the past year in particular, an increasing number of open-source projects in Go have emerged or gained significant adoption. Some of them:

  • Docker
  • Packer
  • Serf
  • InfluxDB
  • Cloud Foundry’s gorouter and CLI
  • CoreOS’s etcd and fleet
  • Vitess, YouTube’s tooling for MySQL scaling
  • Canonical’s Juju (rewritten in Go)
  • Mozilla’s Heka
  • A Go interface to OpenStack Swift
  • Heroku’s Force.com and hk CLIs
  • Apcera’s NATS and gnatsd

Although this seemingly shows the importance of Go, I have a background as a scientist so I hate to be influenced by random anecdotes. It therefore raised the question to me of whether this was a real trend or just observation bias.  To answer this question, I went to Ohloh’s huge data set of more than 600,000 free and open-source software (FOSS) projects. Below, I plotted a number of different ways to look at Go adoption over time:

ohloh_20131216_go
Data from Ohloh tracks new code added during each month. This plot is from a dataset as of 20131216.

As you can see, Go’s rapidly closing in on 1% of total commits and half a percent of projects and contributors. While the trend is obviously interesting, at first glance numbers well under one percent look inconsequential relative to overall adoption. To provide some context, however, each of the most popular languages on Ohloh (C, C++, Java, JavaScript) only constitutes ~10% of commits and ~5% of projects and contributors. That means Go, a seemingly very minor player, is already used nearly one tenth as much in FOSS as the most popular languages in existence.

One of the aspects I found most interesting about the marquee projects I mentioned earlier is how many of them are cloud-centric or otherwise made for dealing with distributed systems or transient environments. Go’s big selling point is concurrency according to one of its designers, Rob Pike (the same one who coauthored the famous “The Unix Programming Environment“). That make it particularly gratifying that people writing projects in Go seem to see it the same way.

Cloud infrastructure is famously complex and requires a great deal of effort to build a truly reliable, scale-out architecture because everything needs redundancy and coordination at the software level rather than the hardware level. Thus tools like Netflix’s Simian Army, one component of their increasingly full-featured platform that’s still waiting to be packaged up, have emerged to provide acid tests of cloud software. On the other side, an underappreciated aspect of PaaS (Platform as a Service) is its improvements not just to developer productivity but also to operator complexity by providing similar benefits at a higher level as Go does at the code level. There’s a lot of value to a packaged solution that handles the complexity of concurrency, coordination, and reliability in a gray-box fashion that enables transparency without requiring manual composition of an entire infrastructure.

Tooling that can ease the complexity for both new entrants and existing users of the cloud will continue to gain prominence at all levels of the stack, whether it’s languages like Go, middle ground like the Simian Army, or higher-level options like PaaS.

Update (2014/03/19): Added Heroku, Apcera

Disclosures: Pivotal, Black Duck, Heroku, and a number of OpenStack vendors are clients. Canonical has been a client. Docker, Hashicorp, InfluxDB, CoreOS, Google, Mozilla, Apcera, and the OpenStack Foundation are not.

by-sa

17 comments

  1. Other Cloud Foundry components, in addition to gorouter and the go cli, are being developed in Go. For instance loggregator and garden (a replacement for the warden component). The increasing popularity of Go is well deserved, quite apart from its excellent support for concurrency.

  2. Heroku seems to be shifting development from ruby to go. The Force.com CLI was written in go last fall and their new Heroku “hk” CLI is also a go client.

    1. Thanks for pointing that out, Jeff! I’d forgotten about that example.

  3. Apcera is written mostly in Go as well.

    1. Cheers, added that to the list as well.

  4. ClearBlade’s entire BaaS technology stack is written in Go.

  5. On the closed side, SendGrid and DigitalOcean are a couple of notable companies moving to Go.

  6. We’re adding Go bindings to GlusterFS / GFAPI – https://forge.gluster.org/gogfapi

  7. Flynn.io, a new docker powered PaaS is also written mostly in Go.

  8. We’ve been using Go in production at Iron.io for well over two years now and it’s been great for us. I wrote a post about our experience after 2 years in production: http://blog.iron.io/2013/08/go-after-2-years-in-production.html

  9. […] mentions. Everything else is, on a volume basis, irrelevant today, even if it’s showing fantastic growth like Go and will likely be relevant in these rankings within the next year or […]

  10. Hi, how did you generate the graph? can you post the script?

  11. The growth of cloud market has been just a matter of time! I feel quite happy that even smaller RMD companies (such as https://www.apponfly.com/en/) have worked through and now are equal competitors to the strongest players like Google, Hostgator, or Mohoro!

Leave a Reply

Your email address will not be published. Required fields are marked *