Kubernetes has firmly established itself as the industry standard for container orchestration. There is basically no other game in town at this point. But when it comes to user experience there is plenty to play for – we’re still early in a secular shift. So far a lot of the Kubernetes build-out has been very much full stack, with the expectation that the developer is also an operator. This needs to change. Specialisation will be required for Kubernetes to continue to flourish. Operator Experience and Developer Experience are not the same thing – they concern two different personae.
At which point I want to mention a couple of clients that recently asked for quotes for press releases about their wholehearted adoption of Kubernetes. Both Docker and Pivotal began as conflicted about Kubernetes, given their own architecture choices and legacies, but have little if any choice but to focus on encouraging and enabling Kubernetes adoption today. One reason I find both companies’ story arcs so interesting is that Docker essentially invented developer experience for containers, while Pivotal created a market around operator customers with Cloud Foundry, inspired by Heroku, for organisations to run their own PaaS platforms as internal service providers.
The key design point for Docker Enterprise 3.0 is improving the Kubernetes experience. It makes a lot of sense for Docker to take this approach. While Docker’s own Swarm orchestration platform offered a simpler experience that some early adopters still prefer, the Kubernetes market is substantially bigger. The opportunity for Docker is to bridge Docker desktop and Kubernetes experiences with a managed experience that makes sense to enterprise customers and buyers. Here’s my quote in the Docker press release.
“Increasing application development velocity and digital agility are a strategic imperative for companies in all sectors today. Developer experience is the killer app,” said RedMonk co-founder, James Governor. “Docker Kubernetes Service and Docker Application aim to package and simplify developer and operator experience, making modern container based workflows more accessible to developers and operators alike.”
So what about Pivotal Software? From the press release:
“Kubernetes is a runaway train right now, and organizations are struggling to keep deployments and management on track. They want to standardize processes, tools, and people, but face a fragmented, complex, and rapidly changing environment. Pivotal has a history of standardizing and documenting cloud-native tools and methods, and is now applying that experience to Kubernetes and associated technology—like Istio. It is finessing the industry shift to Kubernetes by taking management features from the Pivotal Application Service, and re-applying them.”
Kubernetes is extremely sophisticated, but with great sophistication comes great complexity. As the broader stack is becoming clearer, we’re also seeing some dependencies emerge (for example in the Istio stack, with its Mixer, Pilot, Galley and Citadel components), which is kind of ironic for a platform intended to run microservices. Some folks were surprised when a Google Kubernetes Engine (GKE) upgrade in Google Cloud Platform also upgraded Istio.
I've upgraded GKE to 1.13 and boom 💥 Istio went from 1.0 to 1.1. Then policy and mixer went into crash loop backoff, galley responded with TLS handshake timeouts and same with the gateway. Like all distributed systems, restarting things in a **specific** order fixed it 😜
— Stefan Prodan (@stefanprodan) June 11, 2019
A bit ironic given that Stefan is building Flagger, an automated tool for canarying and blue/green deployments, designed to prevent rollouts going awry by making an application or service runs in the environment before cutting over to it. But there you go.
When I spoke to the Pivotal Service Mesh (PAS) team recently about Istio, and the danger of breaking a customer app when upgrading Kubernetes, they said they’d made some design choices to mitigate that kind of risk, based on their experience running operations at scale with customers. The Operator Experience I mentioned before – you can’t break one thing when you upgrade another at a customer like Bank of America, Comcast or Orange. Pivotal has focused engineering on isolating functions such as telemetry and logging aggregation, with a more prescriptive view on exposing services, rather than exposing every low level API.
Shannon Coen, Product Manager at Pivotal Software, said that while customers asked for more access to lower level APIs, there was a trade off involved. Pivotal’s strength is in offering velocity, and a feedback loop, with an opinionated stack. The primary users of Pivotal Software are the Platform Engineering Team, providing services to an Application Development Team. It’s a different design point.
By acquiring Heptio, and with contributions from Pivotal, VMware is now the second main contributor to Kubernetes. Quite the commitment.
As if we needed any further evidence that Kubernetes user experience is the industry focus now, Mesosphere has just rebranded as D2IQ, to underline the fact Mesos is no longer its core engineering focus. The D2 stands for Day Two, when an environment is running in production, rather than just being under development. Again, this reflects an operator focus.
As Kubernetes matures we’re going to see a great focus on operator roles. Industry history can tell us a lot here. VMware began as a platform for dev/test and QA, only latterly becoming the production environment of choice for enterprise workloads. I call this the VMware pattern, and it’s a common pattern for platform maturation. Specialisation and certification emerges in more mature markets as markets become commoditised. A Kubernetes admin is not going to be the same person as someone that deploys an application on Kubernetes.
See for example Google Cloud Run, which is designed to abstract the back end. It’s a tool for developers to target and use.
VMware/Pivotal, IBM/RedHat, Google, are pulling their assets and go to market plans together for this market maturation. Microsoft has a Kubernetes play but is not packaging as effectively for Kubernetes consolidation as yet. That’s the anyone-but-Amazon club. Meanwhile Amazon Web Services is pragmatic and tactical about containers – customers want Kubernetes support and so they’ll provide it. Operations and Productions are generally where the money is, which is where the industry is coalescing now.
Disclosure: Google Cloud, IBM, Red Hat, Microsoft, Pivotal, and VMware are all clients, but this research was published independently of client relationships.
Related posts:
Pivotal, Transactions and Taking an Axe to Long Term Support
Thomas Kurian and Google Cloud: Convenience is the Killer App
On AWS and Pivotal, opinions and overlaps
The Kubernetes World: VMware Acquires Heptio
No Comments