Today, Netflix released yet another component of its infrastructure as open-source software — this time the culprit was Hystrix, a library to aid in building resilient software, while just last week Netflix opened Blitz4j, a framework for scalable logging. They join a growing array of the core components of Netflix’s well-known cloud-based architecture.
The opportunity
Building software with a true cloud-based architecture, resilient to arbitrary (sometimes large) chunks of the infrastructure instantly disappearing from under the feet of the running application, is notoriously difficult. Netflix has spent years building and hosting on Amazon’s AWS cloud. The rest of the world benefits greatly from the knowledge and code shared by Netflix about how to keep a large-scale service up and running when it resembles the Biblical house built on sand.
As usual, packaging is a great opportunity — in this case, to be the vendor who provides the lowest barrier to entry to building complex cloud software. While Netflix has released its software in a collection on GitHub, what will make it truly approachable to developers and companies who are generally unfamiliar with the cloud is aggregating and integrating all those bits together into an easy-to-install, easy-to-use bundle. When I brought this up on Twitter earlier today, Jeff Sussna argued against it:
Their focus is on large complex apps so not sure it’s for the “idiot-proof” market.
I would suggest that the reason this kind of thing hasn’t been for the beginning user, or the smaller application is not because it shouldn’t but instead because it’s too difficult and not worth the tradeoff. If it were relatively easy to have a redundant architecture where you could essentially drop in your app, people would do it far more often. Think convention over configuration [writeup/video].
adrian cockcroft says:
November 26, 2012 at 7:49 pm
The @NetflixOSS platform isn’t hard to deploy, however there are key pieces that aren’t released yet like the Jenkins based continuous integration bakery and a base AMI. The main problem is that it doesn’t scale down for a trivial deployment. You need at least three of each type of service and there are lots of component services, so the minimum installation worth considering is probably getting on for a hundred instances. That’s small for a lot of enterprises, but overkill for a typical Cloud Foundry or Heroku app. However if you want to scale up from hundreds to thousands to tens of thousands of instances distributed globally and stay up when bits of the cloud go down, then @NetflixOSS is in it’s sweet spot.
Donnie Berkholz says:
November 26, 2012 at 8:12 pm
Thanks for your comment, Adrian. Just wanted to add that Neil Levine made the useful point on Twitter that implementing my final point as a PaaS could be useful. When combined with your extra details, that means the PaaS, whether public — or private at the company or department levels — could make the 100-instance minimum more palatable.
Jeff Sussna says:
November 27, 2012 at 11:00 am
Didn’t mean to imply packaging up the Netflix tools wouldn’t be useful. But to Adrian’s point they are targeted at large, complex applications. I don’t think you’d end up with something that would make it trivial to scale trivial apps with trivial in-house expertise. It could on the other hand help sophisticated shops with sophisticated needs get things done more quickly and easily. Obama 2012 as a prime example.
Adron says:
November 27, 2012 at 7:48 pm
Interesting write up. I’m in the middle of trying to figure out some full stack distributed applications in Ruby on Rails, Node.js + JavaScript and prospectively something like PHP. I have however looked at Java – primarily because of the great work the team at Netflix has done with Adrian.
Will have to send you the info on my progress in a few months, I’m sure I’ll have more than a few write ups in the coming weeks too.
What’s in Store for 2014: A Few Predictions – tecosystems says:
February 13, 2014 at 9:42 am
[…] Before Them My colleague has been arguing for the packaging of Netflix’s cloud assets since November of 2012, and to some extent this is already occurring – we spoke to a French ISV in the wake of […]
Go: the emerging language of cloud infrastructure – Donnie Berkholz's Story of Data says:
March 18, 2014 at 1:41 pm
[…] Simian Army, one component of their increasingly full-featured platform that’s still waiting to be packaged up, have emerged to provide acid tests of cloud software. On the other side, an […]