Over the past year in particular, an increasing number of open-source projects in Go have emerged or gained significant adoption. Some of them:
- Docker
- Packer
- Serf
- InfluxDB
- Cloud Foundry’s gorouter and CLI
- CoreOS’s etcd and fleet
- Vitess, YouTube’s tooling for MySQL scaling
- Canonical’s Juju (rewritten in Go)
- Mozilla’s Heka
- A Go interface to OpenStack Swift
- Heroku’s Force.com and hk CLIs
- Apcera’s NATS and gnatsd
Although this seemingly shows the importance of Go, I have a background as a scientist so I hate to be influenced by random anecdotes. It therefore raised the question to me of whether this was a real trend or just observation bias. To answer this question, I went to Ohloh’s huge data set of more than 600,000 free and open-source software (FOSS) projects. Below, I plotted a number of different ways to look at Go adoption over time:
As you can see, Go’s rapidly closing in on 1% of total commits and half a percent of projects and contributors. While the trend is obviously interesting, at first glance numbers well under one percent look inconsequential relative to overall adoption. To provide some context, however, each of the most popular languages on Ohloh (C, C++, Java, JavaScript) only constitutes ~10% of commits and ~5% of projects and contributors. That means Go, a seemingly very minor player, is already used nearly one tenth as much in FOSS as the most popular languages in existence.
One of the aspects I found most interesting about the marquee projects I mentioned earlier is how many of them are cloud-centric or otherwise made for dealing with distributed systems or transient environments. Go’s big selling point is concurrency according to one of its designers, Rob Pike (the same one who coauthored the famous “The Unix Programming Environment“). That make it particularly gratifying that people writing projects in Go seem to see it the same way.
Cloud infrastructure is famously complex and requires a great deal of effort to build a truly reliable, scale-out architecture because everything needs redundancy and coordination at the software level rather than the hardware level. Thus tools like Netflix’s Simian Army, one component of their increasingly full-featured platform that’s still waiting to be packaged up, have emerged to provide acid tests of cloud software. On the other side, an underappreciated aspect of PaaS (Platform as a Service) is its improvements not just to developer productivity but also to operator complexity by providing similar benefits at a higher level as Go does at the code level. There’s a lot of value to a packaged solution that handles the complexity of concurrency, coordination, and reliability in a gray-box fashion that enables transparency without requiring manual composition of an entire infrastructure.
Tooling that can ease the complexity for both new entrants and existing users of the cloud will continue to gain prominence at all levels of the stack, whether it’s languages like Go, middle ground like the Simian Army, or higher-level options like PaaS.
Update (2014/03/19): Added Heroku, Apcera
Disclosures: Pivotal, Black Duck, Heroku, and a number of OpenStack vendors are clients. Canonical has been a client. Docker, Hashicorp, InfluxDB, CoreOS, Google, Mozilla, Apcera, and the OpenStack Foundation are not.
Glyn Normington says:
March 19, 2014 at 4:51 am
Other Cloud Foundry components, in addition to gorouter and the go cli, are being developed in Go. For instance loggregator and garden (a replacement for the warden component). The increasing popularity of Go is well deserved, quite apart from its excellent support for concurrency.
Jeff Douglas says:
March 19, 2014 at 11:55 am
Heroku seems to be shifting development from ruby to go. The Force.com CLI was written in go last fall and their new Heroku “hk” CLI is also a go client.
Donnie Berkholz says:
March 19, 2014 at 3:59 pm
Thanks for pointing that out, Jeff! I’d forgotten about that example.
adrian cockcroft says:
March 19, 2014 at 4:05 pm
Apcera is written mostly in Go as well.
Donnie Berkholz says:
March 19, 2014 at 4:12 pm
Cheers, added that to the list as well.
esimone928 says:
March 20, 2014 at 8:05 am
ClearBlade’s entire BaaS technology stack is written in Go.
Donnie Berkholz says:
March 20, 2014 at 11:56 am
On the closed side, SendGrid and DigitalOcean are a couple of notable companies moving to Go.
Donnie Berkholz says:
April 28, 2014 at 8:33 pm
Twitch’s stats pipeline is another prominent example: http://ossareh.posthaven.com/the-twitch-statistics-pipeline
Donnie Berkholz says:
May 7, 2014 at 1:18 pm
And here’s Disqus: http://highscalability.com/blog/2014/5/7/update-on-disqus-its-still-about-realtime-but-go-demolishes.html
Donnie Berkholz says:
July 7, 2014 at 7:55 pm
Dropbox: https://tech.dropbox.com/2014/07/open-sourcing-our-go-libraries/
Donnie Berkholz says:
July 24, 2014 at 6:33 pm
Runscope: http://blog.runscope.com/posts/go-at-runscope
John Mark says:
March 22, 2014 at 9:51 am
We’re adding Go bindings to GlusterFS / GFAPI – https://forge.gluster.org/gogfapi
Wayne Walls says:
March 23, 2014 at 10:50 am
Flynn.io, a new docker powered PaaS is also written mostly in Go.
treeder says:
March 24, 2014 at 10:08 am
We’ve been using Go in production at Iron.io for well over two years now and it’s been great for us. I wrote a post about our experience after 2 years in production: http://blog.iron.io/2013/08/go-after-2-years-in-production.html
GitHub language trends and the fragmenting landscape – Donnie Berkholz's Story of Data says:
May 2, 2014 at 3:55 pm
[…] mentions. Everything else is, on a volume basis, irrelevant today, even if it’s showing fantastic growth like Go and will likely be relevant in these rankings within the next year or […]
Gerald says:
August 28, 2014 at 8:32 am
Hi, how did you generate the graph? can you post the script?
Tyler says:
January 7, 2015 at 10:45 am
The growth of cloud market has been just a matter of time! I feel quite happy that even smaller RMD companies (such as https://www.apponfly.com/en/) have worked through and now are equal competitors to the strongest players like Google, Hostgator, or Mohoro!