On AWS and Pivotal, opinions and overlaps

After a couple of back to back weeks in which I attended AWS re:Invent and SpringOne Platform, with a little time since to think about what I had learned, I got to thinking about the nature of Opinion as Code. Frankly we shouldn’t expect AWS to be too opinionated. Just consider Amazon’s heritage. It began by selling some books, then all books, then more and more products in every category. Amazon as retailer is happy to help with customer choices, but it’s not going to make them for you. AWS thrives on choice and redundancy.

One of the narratives people picked up in 2017 post re:Invent was the sheer breadth of AWS services being offered made it harder to navigate. Different AWS technologies can overlap with each other- but what to choose? We talked about this in 2016. We had already this noticed the issue emerge at re:Invent 2016 – customers were beginning to have issues navigating the complexity and richness of the AWS catalog. I quote our Slackchat from last year at length here:

monkchips: But essentially it’s kind of hard to stay on track. That was a meta narrative of re:Invent. think about it. 1000 significant service or feature releases in 2016, say 3 a day. How on earth can anyone keep track of that? You could now be a full time AWS industry analyst, and still not keep up. Customers are feeling the difficulty, and it’s only going to get worse. 1500 services next year, perhaps?

sogrady: Do you get any sense that AWS has a problem there?

monkchips: We met super smart switched on people from startups, and they were like: “wait, AWS has that already”, or oh, “we know AWS now has that feature, but we just haven’t re-architected to take advantage of it yet.”

sogrady: We talk a lot about accelerating complexity and fragmentation at RedMonk, and it has clear negative implications outside of AWS. Could it be a real problem for them?

fintanr: I feel AWS may have a problem, particularly where things are interlinked. And yes complexity and fragmentation could definitely be a problem @sogrady.

monkchips: We can end up in a situation like at IBM, where the customer ends up making architectural decisions based on what the Technical Sales Engineer recommends – you get very different answers based on whether they’re more oriented to Series i, p, x or z.

fintanr: I mentioned the compute options during the keynote as an area where choice was good, but further up, it is becoming very complex

monkchips: During the industry analyst summit you could almost feel the audible sighs of relief. Thank God! Complexity! We still have jobs…

sogrady: I think the IBM comparison is an interesting one, for the simple reason of accessibility. Do we think AWS has an advantage here over that traditional IBM acquisition process because developers are – frequently, anyway – the ones making the selection rather than having to rely on a salesperson?

A year later and the issue – which AWS service should we use to solve a problem – is arguably becoming more acute. But when, exactly, is AWS supposed to stop delivering new functions and services and focus on consolidation? Look at data stores, for example, a fast moving market marked by fragmentation and open source innovation. Postgres, MongoDB, MySQL, MariaDB, Cassandra, Hadoop, Spark, with all the associated stack software coming down the pike- Beam, Parquet, etc. AWS is just not going to make that choice on behalf of the customer. Of course AWS offers Aurora, DynamoDB, Glacier, a mix of managed and unmanaged services. There will be more open source databases and NoSQL stores and streaming engines. We can safely expect AWS to support all of them on Infrastructure, and as managed services, as well as cherry-picking the best features for its managed platforms. The heritage of AWS is at the Infrastructure level, rather than higher level services, though undoubtedly the mix has changed now.

Of course life was rather easier from a choice architecture perspective when the choice was just S3. Then EC2. But we have moved on a fair bit since 2009 when simplicity, a “weakness” was actually a strength. At its core Amazon likes to offer customers choice.

There is a school of thought – pushed by consultants – that enterprises should only use lower level lowest common denominator AWS services in order to enable portability. But enterprises should not shut themselves off from innovation any more than AWS should. Customers should choose the right tool for the job, because higher value services are potentially the key to competitive advantage. It makes no sense to compete with Amazon in building infrastructure or management tools, which is exactly what you end up doing if you only use lower level primitives. As Stephen O’Grady has argued, if Salesforce.com doesn’t feel it can compete on infrastructure, then you probably shouldn’t either. Most of us after all are not Lyft, Netflix, Salesforce or Uber.

Last week all the cool kids were at KubeCon, building out the Kubernetes (k8s) ecosystem, but I think it’s important to look at the “Why” not just the “how”? What will k8s enable? Is it about underpinning new CI/CD processes? Standing up big data stacks? Improving operational management, enabling new app architectures and approaches such as blue/green deployments and “canarying“? Making it easier to create platforms? A platform of platforms, fostering an era of domain specific platforms? An adjunct to Envoy, Istio and service mesh approaches? All of the above, obviously.

Kelsey Hightower recently said k8s is boring, but it has definitely not reached boring status yet. Where were the paying customers during KubeCon? SpringOne. What are these customers buying? Software development velocity, not Infrastructure. As Stephen has eloquently argued k8s and Cloud Foundry are different things.

At this point you might be asking – but wait! Pivotal is opinionated. Cloud Foundry is an opinionated Platform as a Service. Spring Boot is an opinionated framework for building apps and services in Java. Pivotal is helping its customers with a How and a Why. Isn’t this a better approach than the AWS services free for all? Kind of, but really it’s about the jobs to be done. Pivotal is about an opinionated digital transformation. AWS is about a better cheaper more flexible infrastructure and services model. Got a workload? Move it to AWS.

Pivotal has a far higher cost of sale than AWS, while AWS has far higher revenues –  $200m a year vs $18bn. Pointing out revenue differences is not to denigrate Pivotal, but rather to say that the two different notions of architecture freedom – both positive and negative (absence of restraints) have value.

Pivotal is becoming less opinionated over time, and so opening up the funnel for customer engagement. Embracing k8s is likely to accelerate the tendency. Pivotal’s new function as a service (FaaS) platform, Riff, announced at SpringOne is built on k8s and initially targets Node.js as a programming language. Quite a departure for Pivotal.

Pivotal has also embraced k8s to give it a stronger Infrastructure story with its new Pivotal Container Services (PKS). For gnarly workloads where persistent storage and more low level flexibility is required PKS will be the offering of choice. We should over time see Pivotal begin to use PKS as a backplane for data services like Elastic and Spark. The next wave will be data and Pivotal will need to compete with k8s packagers here.

I am not trying to make a point positioning AWS and Pivotal as competitors, though it is notable that both Microsoft Azure and Google Cloud Platform (GCP) last year named Pivotal as partner of the year. Pivotal drives workloads. Hosting however is a zero sum game.

I am pointing to a difference in philosophy. Opinionated PaaS has only see two major successes in the market – Heroku and Pivotal. Heroku hit the Rails development and adoption wave perfectly, with an offering that seemed magical at the time. Heroku is still very very good at developer experience.

Pivotal is not the only vendor becoming less opinionated. Azure and GCP both began life with a mission to leapfrog AWS away by offering higher level services. Azure would revolutionise the data backplane with a new model for scaling data and building apps. The model failed. Microsoft learned the lesson and leaned heavily into Infrastructure, making overtures to open source projects and commercial open source projects to cement the deal.

GCP began life with managed services such as GCP App Engine, designed to compete with AWS Elastic Beanstalk, but it too has been forced over time to also lead with an Infrastructure story. Opinionated infrastructure may make developers and organisations more productive, but not everyone buys it. AWS success is based on customer choice and it would be very strange indeed if it tried to start limiting that choice.

Does offering 1300+ new services a year create management and cognitive overheads? Certainly. Do functional overlaps present some challenges? Sure. But the benefits of these services far outweighs that cost. It’s pretty simple maths.

K8s and Istio may usher in a new golden age of workload portability but it seems unlikely it will make much of a dent in AWS. In the end customers find choice, flexibility and convenience more important than portability. they want software and infrastructure to have an opinion until they don’t. They want to take advantage of higher level and managed services.

 

full disclosure: AWS and Pivotal are both clients.

photo by J D Hancock Photography.