HashiCorp: Guardians of the Enterprise?

TL; DR: HashiCorp continues to focus on composable solutions to difficult problems. Their approach to policy, and by extension compliance, as code has the potential to be a game changer.

We had the opportunity to attend HashiConf in Austin, Texas recently and spend some time with the team from HashiCorp. As we have noted previously HashiCorp has consistently demonstrated strong growth across all its open source products, as well as an ever-growing footprint. The question from many quarters has often been how will HashiCorp make money. The answer is very straight forward – by providing enterprise collaboration and compliance tooling for infrastructure.

Closing keynote at #hashiconf is fascinating, @phinze is walking through the rise & demise of Atlas. Listen to customers & iterate

— Fintan Ryan (@fintanr) September 20, 2017

HashiCorp have been quite open about their product evolution, indeed they went through it in some detail during the closing keynote. The fully integrated platform that was Atlas has gone, and given way to an approach where you pick and choose the set of products you require. As we will return to later, this approach has a strong customer appeal.

This also marks out a clear “wedge” approach. If you want to go ahead and run one or all the open source versions yourself, feel free too. When you reach the point of needing enterprise level collaboration and governance, HashiCorp have a commercial offering for you.

This, to my mind, is a very clever strategy. Unix has always been successful by focusing on composable tools. Simple tools that do things well are easy to grasp at a conceptual level, and easy to combine when engineered properly.

The Boring Bits

There is a common theme across all the cloud native projects and companies to “make infrastructure boring”. We couldn’t agree more. However, the approaches vary immensely. For quite a few of the projects the approach is to package everything up and obscure the details. For one set of users this is a more than acceptable approach, but it comes at the cost of giving up control (logically both PaaS and serverless are just further extensions of this approach, but that is a somewhat esoteric debate for another time).

A certain class of more advanced users look for both the control points and the composability of tools. They like to understand what everything is doing, and have the technical know how to do so. They have also been trying to make infrastructure boring, with some success, for quite a long time already. Any tooling that helps them further along this journey, but without losing sight of the inner workings of their environment, is likely to be adopted.

Caius Howcroft of Citadel summed it up best when he spoke about the process they had gone through to evaluate other tools in the cloud native space. Paraphrasing slightly, he said I can’t see under the hood – and that is a problem.

Customer Conversations

As is always the case at any event, conversations with customers and end users are by far the most interesting aspect from an analyst perspective. In HashiCorp’s case their customer base is cutting across pretty much every industry vertical you can think of. We had the opportunity to speak to several of them over the course of the conference. The key point that we took away from our conversations was that none of these customers are merely kicking the tires – they are using the products at scale in extremely large environments.

By far the most interesting conversation topic was the entry point to using HashiCorp’s products, and where their journey had taken them. The process starts with one or two tools, generally one of Vagrant, Terraform, Vault or Consul, in one or two teams. In general, there will be one specific need across the customer infrastructure that is being addressed – the solution is scaled out, the need is addressed and the team moves onto their next problem. However, the problems tend to be interconnected, and people tend to come back and look at tooling that solved another problem as a starting point and then look at related problems.

Interestingly for a tool that is often dismissed, Vagrant came up frequently, albeit in conjunction with Packer, as a basis for modernizing and organisation. Bringing consistency of toolchains to an organisation with twenty or thirty years of legacy and a rigid IT department is never easy – and a workflow based on using VMs as a basis point is far easier to sell into a more traditional organisation than a wholesale move to containers.

For those that were using Nomad one theme came up consistently – simplicity. Several of our conversations involved companies running a huge amount of legacy infrastructure and a lot of “big bets” placed in the past. The debate about container platforms is raging on, and the people we spoke with were not interested in going all in in any one direction. They viewed Nomad as meeting their needs in several cases, and are very happy to use it alongside other approaches which are generally vendor supplied.

Attending to Learn

We at RedMonk attend a lot of events, and in my opinion the behaviour of the attendees tends to fall into a couple of distinct patterns. You see people who are there because they must be, those that are there for a break from work and to attend some sessions, and finally those that are there explicitly to learn. Not many conferences have the vast majority of their attendees in this final category, AWS re:Invent is one, HashiConf is another. A variety of sessions, especially on Vault, were at standing room only. More notably people were actively taking notes (yes on paper) in a manner I have rarely observed.

This bodes well for HashiCorp. This is not people vaguely playing around with the various tools to see what they do, this is people learning from an existing customer use case and seeing what comes next.

Sentinel: Policy as Code

We have written in the past about the concept of compliance as code, and have noted a multitude of different attempts across many enterprises to implement such programs.  These compliance efforts fall into two inter-related buckets – regulatory compliance and policy enforcement. Simply put this is a problem space that pretty much every industry must address on multiple levels, but in almost every case there is a set of technological core building blocks upon which everything else is built.

In our conversations with organisations everyone is trying to solve the problem, particularly on the policy enforcement side, but no one enjoys this work. Its time consuming, error prone and completely inconsistent across and within organisations.

The approach HashiCorp have taken to this problem is to embed a policy layer, Sentinel, across their enterprise offering. This is sensible on two fronts – firstly a lot of people just aren’t interested in this kind of functionality up front. It is something they realise they need later. Secondly enterprises fully understand the need for policy enforcement – they are wrestling with the problem – and they also understand the need for collaboration tools, which the other enterprise offerings address. Policy and compliance is an issue on every CIOs desk, and they will spend money on solutions.

The actual raw implementation is a simple language in which you can define rules across Consul, Vault, Nomad and perhaps most importantly for initial adoption Terraform. By making the rules programmable users can version precisely what they are doing, in the same way they can version their infrastructure.

As with other Hashicorp tools this builds on the composability aspects, but allows you to introduce complex policies which build upon each other. An obvious scenario is using terraform to layout an infrastructure for Kubernetes, another tool such as Chef or Ansible for some of the setup, and then using consul for service discovery and vault for secrets management.  Sentinel policies could be built into each area, covering infrastructure and application requirements from the same tooling.

Our conversations about Sentinel were relatively limited, but we did speak with some beta customers. They love the approach. It solves a problem they all have.

Terraform Module Registry

Terraform is a gateway drug for HashiCorp, and we continue to see increased adoption. As more enterprises are moving towards a multicloud world, the range of providers for terraform, ranging from Amazon, Microsoft and Google through to Aliababa, IBM and VMWare, continues to grow and provides a welcome abstraction layer.

Hitherto we have seen a lot of repetition of essentially the same steps by end users. The Terraform Module Registry addresses this problem by providing a set of reusable templates, several which are verified by HashiCorp.  Overall this is a welcome step, and will ease adoption.

Concerns?

We have three major concerns for HashiCorp. First and foremost is the risk of a monopoly or duopoly versus multi-cloud world. As of now that risk seems to be abating, but the industry would be foolish to rule such a scenario out. With scale comes benefits, and utilities have a natural tendency to move towards being monopolies. The question for everyone is will some of the cloud players eventually throw their hands up and say enough, lets just focus on software.

The next is serverless computing. The most important utility in serverless computing is the ability to use adjacent services such as storage, analytics and so forth. But this also quickly leads to an all-in on a specific provider model. In this case, the utility of tools such as Terraform is greatly diminished, and a popular gateway disappears.

Our final concern is the ability to look towards the future, while reflecting on the past. HashiCorp have been successful in both their choice of problems to solve and their approach. However, with such success comes an inherent risk of not listening to others. For now, HashiCorp are very focused on listening to their customers – this is key, but carefully listening to what others in the industry can offer is equally as important. This takes humility, and in this many companies stumble.

Disclaimers: Microsoft, Amazon, IBM and Google are current RedMonk clients. Hashicorp paid my T&E.

Credits: Cogs Image – Copyright Arthur John Picton and licensed for reuse under this Creative Commons Licence