The Return of the Pendulum: Consolidation is Here

From its birth through the first few years following the turn of the century, technology was most properly considered as a centralized industry. Dominated by a smaller number of large players and their monolithic, relatively homogenous platforms, it was predictable in its progression and characterized by a steady, stately pace. There were disruptions and revolutions, of course – the personal computer era being perhaps the most notable of these – but overall, the landscape was largely coalesced around a small number of vendors and their respective technologies.

Around the turn of the new millennium, however, a variety of macro market pressures were building. What would come to be called open source, for one. The internet, for another. Twelve years after one of the original internet pioneers was born, meanwhile, it gave birth to yet one more market moving factor: the cloud. These and a myriad of other pressures, trends and developments combined to start history’s pendulum on a reverse trajectory.

Seemingly overnight, the wider technology market developed not only a tolerance of but appetite for specialized technologies – specialized technologies that could only come from a more decentralized, distributed technology industry. So it was that the small stable of big tech companies gave way to a new, far more crowded landscape of players. As but one example, what had been staid, heavily centralized markets like databases blew up into the fragmentation that was NoSQL. One software category populated by a small number of relevant players became a half dozen or more categories each of which had four or more relevant vendors of its own.

As Ashlee Vance quoted Andrew Feldman, then CEO of SeaMicro (acquired by AMD in 2012) in 2010:

“There is a foment happening. It’s a bubbling of ideas and technology.”

None of the above should be controversial, or particular open to dispute. No one can credibly question the assertion that the market, on the whole, swung towards specialization. The only real question is when the pendulum would swing back in the other direction. When, the question was, would the market again advantage general purpose offerings, with consolidation as its direct, inevitable consequence?

Based on the evidence at hand, the answer appears to be now.

Two and a half years ago, it was noted in this space that the database market – the same market that was aggressively and forcefully decentralized beginning around 2009 – was beginning to consolidate. In principles consistent with convergent evolution, specialized database players began to add general purpose, adjacent features to expand their addressable market and to satisfy customers seeking to reduce their vendor overhead. General purpose relational databases, meanwhile, increased their ability to compete with their more specialized counterparts by adding specific areas of capability – the ability to ingest and operate on JSON, for example.

That was an interesting development for the database market, certainly, but if it was an isolated development its significance would be limited. Instead, however, this type of functional consolidation is rippling outwards and impacting category after category. The APM, logging and observability markets, for example, have been aggressively merging for years. Developer toolchains, once made up of independent, best of breed components from version control to issue tracking to CI/CD to vulnerability scanning, are driving towards native, pre-integrated experiences offered by a single party. Developers individually, meanwhile, have been clamoring for Heroku-like platforms that abstract all of the underlying plumbing away, leaving them with only the simple task of pushing code to it. CDNs become more general purpose, clouds become better CDNs. One time areas of high visibility and focus like operating systems, virtual machines or container orchestration platforms are gradually fading from sight, obscured beneath multiple layers of abstraction and consolidation. Where RedMonk’s language rankings were once somewhat dynamic, characterized by growth, sometimes rapid, of late they have been more static and reflective of minimal movement.

From individuals to markets, the inevitable outcome of having too many choices is a desire to make fewer choices. Consolidation and centralization are real, and they are here, observable in most markets today with one notable exception which we’ll come back to.

Some will likely argue that, like every other aspect of life today, that fragmentation was a zero interest rate phenomenon. And inarguably the availability of free money contributed to a world with more startups. But to point to ZIRP as the sole or even primary driver of fragmentation and decentralization is reductive, and ignores the reality which is that this trend was always inevitable independent of interest rates, because history demonstrates conclusively that pendulums that swing in one direction will inevitably, at some point, swing in the other.

The question facing us, then, is not whether the pendulum has started to reverse its course, but what that swing means. The implications are wide, but a few conclusions seem obvious:

1. General purpose, more broadly capable platforms will increasingly have an advantage. Specialized players must not only be better than their general purpose competitors to survive, they must be significantly better.
2. For specialized players who are unable or unwilling to broaden their functional surface areas, in addition to being technically superior they need to significantly increase their ability to partner. Multiple complementary and specialized third party offerings are going to be facing customers preferring single vendor, general purpose systems and they must mitigate their fragmented nature through superior technical integration, yes, but also non-technical, operational excellence via partnered pricing, joint go to markets, packaging, co-marketing and so on.
3. One of general purpose plaforms’ advantages, in theory if not always in practice, is in experience. No need for multiple logins, for example. No need for the cognitive overhead of multiple user interfaces, and so on. Combined with the wider markets current existing preference for and emphasis on developer experience, this implies that specialized players must heavily emphasize experience in their own offerings – and again, potentially mitigate some of the weaknesses there via partnerships.
4. Specialized vendors should be prepared to answer customer questions that have been uncommon in these recent years of fragmentation: why should I choose a specialized offering? Why shouldn’t I just use this easy general purpose platform even if it’s not as capable? Why should my procurement team have to deal with multiple vendors when I can just manage one?
5. Merger and acquisition work is ahead – probably a great deal of it. While the wider US economy, at least, is performing above expectations and interest rates are beginning to come down, it’s a challenging time for financing for everyone except AI companies. The combination of larger players seeking to broaden their capability footprint and smaller players finding it challenging to compete with larger, more broadly capable platforms is likely to lead to some of the former acquiring some of the latter.

Speaking of AI, however, that is the notable exception to this trend mentioned above. If anything, fragmentation in the AI world is expanding, not contracting. New models arrive by the day, as do new use cases and the vendors that would target them. This is likely to continue in the near term, but there are already signs that AI is poised to be impacted by precisely the same trends currently being experienced by other software categories. AI is new, so its period of frothy experimentation is tolerated, but there will come a time when customers, and even most developers, tire of having to continually making choices about models and otherwise, and default to whatever general purpose platform they’ve imprinted on.

Fourteen years ago, this was the observation regarding the arrival and onset of specialization.

As to when the pendulum will shift back towards general purpose, all I can tell you is that it will at some point. The inevitable result of an explosion of choice is a reactionary market shift away from it. All this has happened before, and all of this will happen again.

Today, in 2024, that same question remains. When will the pendulum shift back to specialization? The answer is that we don’t know, but we can confidently state that it will. It always does.