Skip to content

DevOps: All Development, No Database

Since the last time I touched working code in a production environment, it’s no exaggeration to say that no part of the development process remains untouched. Over the last decade plus, effectively every aspect of the application development process has been scrutinized, rethought and in many cases reinvented. From version control to build systems to configuration and deployment to monitoring, modern development’s toolchain is multi-part and sophisticated.

As it must be. Processes that work for code released in cycles measured in months cannot be expected to handle workflows measured in days or minutes.

For all that the process of developing software has evolved, however, the database remains curiously overlooked. Consider the example of Cloud Native. Describing a modern, typically legacy-free approach to building applications appropriate for cloud environments, the term Cloud Native has gone from informal descriptor to accepted industry shorthand in short order – to the extent that it has its own technical foundation.

If we look at the membership of that foundation, the CNCF, it would appear that the roster includes no database vendors at the Platinum or Gold membership levels, at least if you assume Google’s involvement is around Kubernetes and not tools such as BigQuery. Of the 41 silver members, meanwhile, two can be considered database vendors: Crunchy and Treasure Data.

For its part the Cloud Native architecture diagram from Pivotal, the company that first got behind the term, does not explicitly call out databases or database administrators.

Nor does Red Hat’s Open Hybrid Cloud architecture diagram.

If you query the website for the term “database,” meanwhile, you get about 20 results back, none of which are database-centric and many of which are translations.

From a market standpoint, it’s clear that databases are anything but an afterthought. The market valuations are substantial, and customers choices are expanding as are commercial investments in the space. But if you were judging simply by visibility of databases within the developmental toolchain and associated discussion, it is clear that databases do not occupy a position of prominence.

The question is why.

Some of the vendors in the space make an implicit argument that it’s a technology problem. But while it certainly cannot be argued that database tooling has undergone as rapid or thorough of an evolution as their application development counterparts, there are many operationally focused database vendors with technology that could certainly be useful in a DevOps context. Some market to this – see, for example, DBmaestro’s tagline of “DevOps for Database.” Others with applicable functional coverage such as Datos, Delphix or Redgate do not, with their messaging clearly aimed at traditional database administration buyers. Even developer-accessible offerings such as Citus Data or Heroku Postgres don’t explicitly try and make the connection from the operational data layer back to DevOps, or on a detailed level the task of application lifecycle management.

To some degree, this is to be expected as an artifact of a divide that continues to persist within most enterprises. While barriers between developers and operators have been actively targeted for elimination, the boundaries between developers and database administrators have remained well off the radar. Operators were once a distant, disconnected constituency, but are increasingly being integrated into development teams and vice versa. A similar thawing between the developer and DBA has yet to occur.

Which is why we still have application development models that don’t mention the database, and development-oriented database tooling that doesn’t mention DevOps. Neither of which, it should be noted, makes business sense.

Or is likely sustainable. Sooner or later, someone is going to realize that the continuing divide between the people developing the applications and the people that manage the data behind those applications is an obvious inefficiency, and like all market inefficiencies it will be targeted.

The question is not if, but when and who.

Disclosure: Citus, Pivotal, Red Hat, Salesforce (Heroku) and Treasure Data are RedMonk customers. Crunchy, Datos, DBmaestro, Delphix and Redgate are not currently customers.

Categories: Cloud, Databases.

On Digital Transformation

Lokomobile 3

The importance of technology, from an organizational perspective, is cyclical. Misunderstood and feared in its infancy, in a manner that will be familiar to those who’ve studied the history of the industrial revolution, technology was subsequently embraced first in specialized roles but went on steadily annex every unoccupied area of an organization. Having conquered the office, it then spilled over into the home with the introduction and rise of the Personal Computer in the early 1980’s.

But something happened along the way to everyone having a computer in their pocket and thirty in their car: people stopped caring. When everyone had technology, technology no longer mattered, at least according to the core thesis from the influential title by Nicholas Carr. A book that was indirectly responsible for unintended side effects from booms in outsourcing to downward price pressures on technology vendors. If technology was no more differentiating for an organization than its plumbing, the thinking went, why pay premiums for superior technology, or indeed own the capability at all? Many organizations, in an effort to focus on a core set of business concerns which did not include technology, chose to effectively divest themselves of responsibility via large outsourcing contracts.

This trend has begun to reverse itself in recent years, however. Seven years after the publication of “Does IT Matter,” many CIOs read Marc Andreessen’s Wall Street Journal Op-Ed describing the evisceration of traditional markets by digital players such as Amazon, Netflix, Pandora, Pixar and Spotify (for the record, no, Andreessen mentioned neither Airbnb nor Uber). When they asked themselves whether IT mattered in 2011, a number of them now answered the question differently.

Which is why and how the term “Digital Transformation” has come into fashion. It is no longer enough for technology providers to be able to solve complex technical problems. Customers today take the infrastructure for granted, and want instead to know how they can end up being Amazon or Netflix rather than Borders or Blockbuster.

While understandable, there are three basic problems with this request.

  1. Most Technology Providers Aren’t Equipped to Have That Conversation
    Independent of industry, most companies do a limited number of things well. The technology industry is no exception. Apple, for example, is rightly regarded as being the best in the world at user experience. In the context of delivering online services, however, Apple’s UX prowess has not translated particularly well because services are not a core competency for Apple – however much users wish otherwise. When enterprises turn to technology vendors for their digital transformation needs, in turn, what they are effectively asking for is a combination of deep technical experience and product design skills with very high end management consultancy or venture capital skills on top of that. It is difficult to envision a single company providing both at a high level of quality. One, certainly, but not both.

  2. Even if the Technology Providers Could, The Organization’s DNA Will Fight It
    There is a famous and possibly apocryphal story told in the research community about a call that an analyst had with one of the major US telcos in the early days of the cloud. According to the account, the analyst got on a call with sixty or so telco team members. The goal, the analyst was told, was to compete head to head with Amazon Web Services. The analyst then asks which fifty people on this call the telco is planning on firing, saying that you don’t tend build the next generation infrastructure with the people who built what you’re replacing. True or not, the implication is clear: organizational change, and in particular the type of organizational change that is required to respond to true disruption, is exceptionally difficult. There’s a reason most companies don’t survive the experience. Even if technology providers are equipped to provide not just the required technical expertise but the business innovation consulting that is more typically the province of a Bain or a McKinsey, many and perhaps most organizations will find it difficult to execute the necessary changes.

  3. The People Whose Job It Is to Predict the Next Airbnb, Uber, et al Can’t
    In June of 2008, a friend of the Airbnb CEO and co-founder introduced the company to seven prominent Silicon Valley investors. The company was attempting to raise $150,000 on a $1.5 million valuation. Five investors sent notes back declining the opportunity to purchase 10% of the company for $150,000, two didn’t bother to reply at all. In June of 2016, eight years later, the company was raising money at a $30 billion valuation. The point here isn’t to laugh at the unnamed investors, the point here is that predicting outcomes generally, and digital transformation specifically, is hard. Hard enough that very highly paid and educated professionals who do nothing but evaluate businesses that intend to transform industries digitally fail more often than they don’t. Expecting that your technology supplier, then, can be responsible for shepherding you through this process is probably not reasonable.

None of which is to say that technology providers cannot and should not be good partners as businesses in industries roiled by change seek to retake control of their technology stack and transform themselves amidst highly dynamic markets. Indeed, technology providers that can provide more than technology, in the form of education, guidance and, yes, good tech – are seeing tangible rewards in deal size and customer goodwill. More to the point, as one vendor put it this week, having the conversation is not optional for vendors any longer.

But enterprises must have realistic expectations, both of themselves and their technology providers. If they expect to be Uber tomorrow based on purchasing technology from a vendor with a deck that could have come from McKinsey, they’re likely to be very disappointed.

It is nice, however, to see people coming back around to the idea that yes, IT matters.

Categories: Business Models.

Lighting Out For the Territories

Apart from the fact that working at RedMonk affords me a legitimate opportunity to make a difference in the lives of developers, perhaps the best part of this job is the people that I work with – both at RedMonk and the clients and contacts we talk with daily. For those wondering, I am fully aware of how lucky I am to like both what I do and the people I work with.

Still, given the travel schedule analysts sustain for months on end, with long hours and a lot of time spent on planes and in hotels away from home, it’s nice to have some downtime – not in a metal tube – to take a step back. Every summer, as things in our industry slow down, I try and take a few consecutive weeks away from the job. While there are some in the industry who would make a virtue of never taking vacation, from my perspective, the time away is essential, not just to recharge the batteries but also to gain some necessary perspective. If you spend all of your time within the industry, it’s easy to lose that.

Fortunately, here in Maine, as much as Portland is attracting more and more industry talent, it’s not hard to get away from tech. As of tomorrow morning, I will be dividing my time between building furniture and swimming under waterfalls, painting the house and meeting friends for an afternoon beer at Allagash, ripping up shrubs from our front yard and, well, heading back to Allagash. My last week of vacation, with any luck, will be spent waking up to the sound of AC/DC and Led Zeppelin cranking from lobster boats a hundred yards away on the water.

Hopefully, it’ll be a few quiet weeks while I’m out. More likely, all hell will break loose, markets will roil and major acquisitions will come fast and furious. If you’re interested to hear about how and how badly I injure myself working on home improvement projects, Twitter, as always, is your friend. If you’re a RedMonk client and need help while I’m out, Juliane (jleary @ will be able to direct you appropriately.

Otherwise, I’ll see you on the other side, and until then, enjoy your time.

Categories: Personal.

Where the Database Market Goes From Here

At the crossroads

It’s hard to remember now, but a decade ago the idea of non-relational databases was a foreign one. Outside of successful and widely adopted alternatives such as Berkeley DB, generally the word database could reasonably be assumed to mean relational database. When we wrote about the possibility of non-relational alternatives then eleven years ago last March, the general reaction was a shrug, consternation or both.

As developers increasingly took control of the decision making processes around technology selection, however, they looked outside the enterprise to the likes of Google for architectural inspiration, and non-relational databases first emerged and then exploded. From a consolidated handful of enterprise-oriented relational databases which are still the backbone for millions of existing applications, the database market added a wide variety of new specialized database types: columnar, distributed storage and process, document, graph, in-memory, key-value and more.

Each of these categories began with the creation of specialized engines that excelled at a particular task, but that also involved tradeoffs traditional database buyers were unfamiliar with. Hadoop’s Map Reduce, for example, was less accessible to traditional DBAs (at least until companies such as Facebook wrote SQL-like interfaces such as Hive), but it could attack larger scale datasets than was practical with traditional relational databases, and it could do so far more efficiently.

The database market today, then, looks very different than the database market of a decade ago. The traditional relational databases are all still around, but they are increasingly one of many databases employed in a given business rather than the database employed.

Just as it was clear a decade ago that the market would be expanded, however, it is equally apparent today that the database market is poised for change. Functionally, we will continue to see a steady, even accelerating evolution of new approaches – fueled in large part by the release or replication of technologies developed at companies occupying the bleeding edge of web scale. Strategically, however, the available evidence suggests we should look for two major shifts in market.

A Return to General Purpose Datastores

By necessity, most of the major emergent non-relational database platforms of the last decade – projects such as Cassandra, Hadoop, MongoDB or Redis – were specialized in their design. In order to compete with the incumbent general purpose relational database platforms, their focus was asymmetric. Of all of the technology categories, database buyers have perhaps the least tolerance for risk. Which means that to justify using something other than the tried and true relational database technologies that had evolved and been improved over decades, alternatives couldn’t just be a little bit faster or a little bit more accessible: they had to be an order of magnitude improvement or more.

This, plus their built-from-scratch nature, inevitably produced a host of new database software that was highly differentiated from the traditional relational databases in approach, scale and function. Which is how we ended up with a database market composed of half a dozen or more relatively distinct categories.

Inevitably, however, these specialized platforms will seek to become less specialized over time. Much as lightweight, developer–friendly MySQL steadily added features such as stored procedures and triggers due to enterprise demand, many of today’s vertical non-relational stores will trend back towards their general purpose, relational ancestors.

In many ways, this transition has been underway for some time.

  • The category, for example, that once defined itself by its universal lack of a structured query language has been, in many cases, working to add that functionality. Software once born out of frustration with databases shackled to SQL, in other words, have been compelled over time to add query languages that look very like SQL.
  • Or look back to MongoDB’s acquisition of WiredTiger, a storage engine built by some of the same architects of Berkeley DB once upon a time. In many respects, the WiredTiger acquisition was a crucial step for MongoDB, because it allowed the database once prized for its ease of use and accessibility to add features such as better write performance, compression and document-level locking. The kinds of features, again, which enterprises look for.
  • Cloudera, for its part, began as a company with a single mission: applying Map Reduce and HDFS to data problems of unusual size and complexity. Today, its message is much broader, encompassing the traditional batch workloads, but search, streaming and SQL on top of those. Much more general purpose.

This trend will continue. As enterprises have acclimated to more complex functional markets, their willingness to purchase commercial solutions or support in specialized categories has ticked up. This is incentive both for players in market, but also players in nearby or adjacent markets. Importantly, open source has also lowered barriers to entry between these markets, as some of the core developmental costs can be offset and because in some cases necessary integration work between projects has already been performed.

The key question facing the market around this development, however, concerns developers. There is little question, historically, that enterprise buyers have preferred to consolidate purchases amongst the fewest possible number of suppliers. Which means that all-in-one, general purpose offerings will be welcome to CIOs and other purchasing agents. Developers, however, have historically advantaged more specialized, single purpose offerings over general purpose alternatives.

Which means that while the trend towards general purpose commercial datastores is seemingly inevitable, its outcome is not. It will be important for commercial vendors making such a transition to ensure that their developer engagement and narrative is at least as strong as the one-throat-to-choke message they can present buyers. Because otherwise, as the market performance of general purpose relational databases suggests, even perfect buyer messaging cannot make up for a lack of developer interest and adoption.

The Rise of as-a-Service Offerings

If the developer appetite for general purpose non-relational on premise solutions is uncertain, however, the interest in Database-as-a-Service offerings is not. This has been evident for some time, as the two fastest growing services in the history of Amazon’s fast growing Web Services business are Redshift (datacenter-as-a-service) and Aurora (MySQL-compatible proprietary database). Merger and acquisition activity in the space has likewise been steady: IBM, after previously acquiring Cloudant, bought Compose. Elastic purchased Found. Both of which follow acquisitions such as CenturyLink/ and Rackspace/ObjectRocket. And of course there is organic development. MongoDB’s Atlas service [coverage], for example, is very likely the shape of things to come. As are updates such as Cloudera’s 5.8 drop which added Impala support for AWS’s S3.

As-a-service offerings have limitations relative to on premises alternatives: they have less of a track record, latency between the database and compute tiers can be an issue, and it can be difficult to migrate large scale datastores to the cloud if only because of network limitations. But the advantages of instant-on, pay-as-you-go services that allow developers to make the database and everything that comes with it someone else’s problem have proven to be more than attractive enough to offset those and other concerns. Convenience, as ever, will trump just about everything more often than it won’t. Faced with the prospect of a fraught selection of the appropriate database, scaling it as required, protecting it and keeping it backed up, many developers are opting out.

Further, while as-a-service databases are compelling enough as stand-alone options, they also stand to benefit from general market adoption of cloud services. If your workloads are on premises, DB-as-a-service offerings are significantly disadvantaged, but with so much growth coming from cloud at the expense of on premises alternatives, the growth opportunities for hosted databases are substantial. This is true for base IaaS, but even more true for cloud services operating at higher levels of abstraction such as PaaS or serverless. If you’re content to outsource the infrastructure for your application, you’re more likely to do so for your database as well.

The Net

Given that the database market is subject to the same market forces as other enterprise categories, on premises software both specialized and general purpose is likely to be a tightening market over time. There is a great deal of revenue to be had in the category, without question, but it will be more difficult to obtain as there is more competition generally, more competition from open source specifically and on premises alternatives increasingly compete directly with service based alternatives. These price pressures are one reason vendors are increasingly moving back towards general purpose datastores from specialized roots: the broader the functional capabilities, the wider the addressable market, at least in theory.

Even long time database incumbents, however, are scrambling to develop or acquire their way into service-based businesses because that is where much of the growth will occur. Whether adopted as more convenient stand-alone alternatives to on premises databases or deployed in conjunction with other cloud infrastructure, DBaaS offerings are attractive to both developers and their employers, if for very different reasons. Importantly, this is the case in spite of the fact that the DBaaS market is in its infancy; many popular databases are not yet available as services, and those that are don’t yet have the provider choice that they will ultimately. Which implies that the DBaaS market has been successful, to a degree, in spite of itself.

From a provider perspective, then, a choice is implied. The existing spend on on premises relational solutions is measured in tens of billions of dollars, which is why many database providers today still regard their primary market competition as Oracle, even if they’re selling non-relational solutions. Vendors focused on trajectory, however, tend to see Amazon as the more important target, given that the most common report when talking to purveyors of on premises software is that a significant percentage of their existing customers are already in the cloud and most of those are on Amazon.

Implied choice or not, however, a legitimate market approach is also not to choose. On premises providers in most cases will need to follow the lead of players like MongoDB, because competing with DBaaS players without an as-a-service option is a non-starter. Worse, competitors that operate as-a-service businesses have an enormous intelligence advantage over purely on premises competitors, thanks to the difference in available operational telemetry. But neither should on premises vendors deprecate their existing business; instead, they can differentiate from pure play as-a-service options by offering customers their choice of running in an existing datacenter or in the cloud.

What is clear, however, is that a status quo approach in this market is one that will lead to diminishing returns over time. Choose your path carefully.

Disclosure: Amazon, CenturyLink, Cloudera, MongoDB and Oracle are RedMonk customers. Rackspace is not.

Categories: Cloud, Databases.

What Amazon Learned From Microsoft

As the company most clearly associated with the rise of the commercial software industry, it is no surprise that Microsoft’s business model historically reflected a profound, even emotional, attachment to software as an asset. Microsoft didn’t make money with software, it made money from software. At rates that were unprecedented in the technology industry at the time.

In practical terms, Microsoft was a platform business – and arguably even still is as it moves aggressively into services. Microsoft owned two of the most important platforms in the business and productivity software and operating system markets, and it studiously built on top of and alongside these platforms to deliver not just base platforms to their customers, but complete end-to-end stacks. Need a database? Microsoft had one. Ditto for web servers, app servers and integrated development environments. The value of these individual pieces was intended to be more than the sum of the parts, thanks to a practice known as integrated innovation. This describes an approach in which individual components are improved when used in conjunction with one another.

The byproduct of both the integrated innovation approach and Microsoft’s fixation on software as an asset was that company felt compelled to own the stack. If software was an asset, and higher switching costs meant higher profits, then logically Microsoft needed to sell more and more software that would raise the switching costs which would result in higher profits. While the company pointed to integrated innovation as an opportunity for partners as well as Microsoft, the Redmond software giant left little available oxygen amidst the core infrastructure – and later, segments of the packaged applications market – for partners. The end to end stack that Microsoft took to market was, by and large, built and sold by Microsoft.

Which in turn meant tradeoffs for customers. On the one hand, the stack offered the promise of lower integration costs, the proverbial one throat to choke and Microsoft’s characteristically strong developer experience. On the other hand, the stack was in many respects an all or nothing proposition. At the time, developing in Microsoft languages or leveraging Microsoft infrastructure meant that you used Windows. And while there was certainly a wealth of non-Microsoft server software that ran very capably and could be mixed and matched on Windows, it was difficult to do this with Microsoft server software – SQL Server being a notable exception.

Microsoft’s vision for customers, in other words, was complete, end-to-end and full service. Which, again, was not a surprise given that the company had internalized – was probably the best example of, in fact – Shapiro and Varian’s maxim that “the profits you can earn from a customer – on a going forward, present-value basis – exactly equal the total switching costs.” Customers that used less Microsoft software could switch platforms more easily. If customers that used one Microsoft product could be incented to use another Microsoft product, however, they would be less likely to make a jump from the platform. Each additional Microsoft product consumed further raised the switching costs, which according to Shapiro and Varian meant that Microsoft’s profits would rise in direct proportion.

This strategy worked very well, and for a very long time. While it’s impossible to know for sure, it’s not impossible that it could have survived the transition to mobile had the company executed more effectively in that market.

In the cloud, however, the player that appears to have learned the most from Microsoft’s platform success ironically has not been Microsoft, but Amazon. What the companies have in common – besides their geographical location, respective market dominance and some shared history amongst its executives – is that they are both platform businesses. Amazon’s Web Services business, it can be argued, is the new Microsoft, in the cloud. Which would imply that the AWS Management Console, as mentioned above, is the new Visual Studio.

(click to embiggen)

Like Microsoft, AWS is intent on delivering a complete, end-to-end experience for its customers. Whether the market need is a database, a CDN, directory service, streaming data support, logging, storage, messaging, data warehousing capabilities or something else, AWS has an offering. Or if they don’t today, they are likely to eventually. Like Microsoft with Office or SQL Server, AWS is comfortable augmenting its internal development with inorganic acquisition or OEMing where need be. AWS’ relentless drive to deliver the most complete platform in the cloud, at a rate that is impressive bordering on intimidating, has garnered the company a great deal of attention over the past few years. But it is not unprecedented. Those who followed Microsoft before it was temporarily stalled by the emergence of cloud and mobile will recognize a great deal of Amazon’s behavior and ambition.

Amazon has broken from the Microsoft playbook in one very important way, however: they have provided more and more accessible on ramps.

Microsoft woke to this strategy late, as evidenced by decisions such as the one to explicitly court the PHP community. As popular and ubiquitous as Microsoft technologies were, the theory went, there were open source alternatives that were far more popular that might be growth opportunities for IIS, and by default, the only operating system it ran on.

Amazon, by contrast, has had this essentially baked in from day one. The Elastic Compute Cloud, for example, may be accessible via proprietary APIs, but what runs on an EC2 LAMP stack will run equally well on premises or in a competitor’s cloud. Likewise, when Amazon’s Relational Database Service launched three years after EC2 and S3 debuted, it was an implementation of an open source project – MySQL. Unlike SQL Server, which could only be obtained from Microsoft, Amazon could plausibly tell its users that they if they were unsatisfied, they could get MySQL support from any number of alternative providers. Buying into Amazon, in other words, doesn’t mean a commitment to only buying from Amazon.

Which is not to say, of course, that Amazon has pursued a purely open strategy where Microsoft’s was proprietary. Quite the contrary; the majority of Amazon’s services are proprietary to Amazon, and in cases such as Aurora even open source-based offerings have a proprietary flavor to them. Amazon, and its sizable body of employees from Microsoft, have not forgotten the lessons of the software giant. But by including open or open-ish entrypoints such as Elasticsearch, MariaDB, MySQL, PostgreSQL – not to mention the core support for distributions such as RHEL or Ubuntu, or the Docker support within ECS – Amazon is not requiring customers to make the same tradeoffs that Microsoft did once upon a time. If customers choose to leverage standard services such as compute or open source databases, Amazon can monetize them. If they expand from there into proprietary services such as Kinesis or Redshift, AWS profit potential goes up, but the company will happily pursue volume and margin markets simultaneously. Amazon is like Microsoft, in other words, if Microsoft had been able to offer more, better and wider support for open source platforms years earlier.

Whether this is because Amazon understands that its cloud services’ instant availability and the inertia that comes from its usage is a powerful staying factor, the fact that they can monetize even non-AWS software services by selling the hardware to run it on, or some combination of the two is ultimately irrelevant. The net result is a provider that appears to have understood the most important lessons learned by one of the most dominant technology companies on the planet, and has even managed to improve upon them. It also is suggestive of the best way to compete with Amazon, but that’s a post for another day.

Disclosure: Amazon and Microsoft are both RedMonk customers.

Categories: Economics, Platforms.

The RedMonk Programming Language Rankings: June 2016

With the spring and summer travel schedule drawing to a close, we finally have had time to sit down and run the numbers collected back in June. As always, aside from the fact that we run our own GitHub rankings now, the process used for our bi-annual programming language rankings remains the same as when Drew Conway and John Myles White first looked at the question late in 2010. We have continued this analysis, comparing the performance of programming languages relative to one another on GitHub and Stack Overflow twice a year. The idea is not to offer a statistically valid representation of current usage, but rather to correlate language discussion (Stack Overflow) and usage (GitHub) in an effort to extract insights into potential future adoption trends.

With the exception of GitHub’s decision to no longer provide language rankings on its Explore page – they are now calculated from the GitHub archive – the rankings are performed in the same manner, meaning that we can compare rankings from run to run, and year to year, with confidence.

Historically, the correlation between how a language ranks on GitHub versus its ranking on Stack Overflow has been strong, but this had been weakening in recent years. From its highs of .78, the correlation was down to .73 this time last year – the lowest recorded. For this run, however, the correlation between the properties is once again robust. As with last quarter’s ranking, the correlation between the properties was .77, just shy of its all time mark. This is arguably noise, but we believe the correlation is worth noting at a minimum.

Before we continue, please keep in mind the usual caveats.

  • To be included in this analysis, a language must be observable within both GitHub and Stack Overflow.
  • No claims are made here that these rankings are representative of general usage more broadly. They are nothing more or less than an examination of the correlation between two populations we believe to be predictive of future use, hence their value.
  • There are many potential communities that could be surveyed for this analysis. GitHub and Stack Overflow are used here first because of their size and second because of their public exposure of the data necessary for the analysis. We encourage, however, interested parties to perform their own analyses using other sources.
  • All numerical rankings should be taken with a grain of salt. We rank by numbers here strictly for the sake of interest. In general, the numerical ranking is substantially less relevant than the language’s tier or grouping. In many cases, one spot on the list is not distinguishable from the next. The separation between language tiers on the plot, however, is generally representative of substantial differences in relative popularity.
  • GitHub language rankings are based on raw lines of code, which means that repositories written in a given language that include a greater amount of code in a second language (e.g. JavaScript) will be read as the latter rather than the former.
  • In addition, the further down the rankings one goes, the less data available to rank languages by. Beyond the top tiers of languages, depending on the snapshot, the amount of data to assess is minute, and the actual placement of languages becomes less reliable the further down the list one proceeds.

(click to embiggen the chart)

Besides the above plot, which can be difficult to parse even at full size, we offer the following numerical rankings. As will be observed, this run produced several ties which are reflected below (they are listed out here alphabetically rather than consolidated as ties because the latter approach led to misunderstandings). Note that this is actually a list of the Top 21 languages, not Top 20, because of said ties.

1 JavaScript
2 Java
4 Python
5 C#
5 C++
5 Ruby
9 C
10 Objective-C
11 Shell
12 R
13 Perl
14 Scala
15 Go
16 Haskell
17 Swift
18 Matlab
19 Visual Basic
20 Clojure
20 Groovy

JavaScript retains its position atop the rankings for yet another quarter, as do Java and PHP in their second and third positions respectively. There is no movement, in fact, among languages ranked within our Top 10. The positions have solidified, and it’s becoming apparent that it will take a serious push – or crisis – to significantly alter the dynamics of the top tier absent minor and statistically irrelevant drifts from quarter to quarter. It may or may not suggest that fragmentation is beginning to slow, but that’s an analysis outside the scope of these rankings.

We do have movement outside of the Top 10, however. Here they are in no alphabetical order.

  • Elixir: Elixir jumped again this quarter, but to a smaller degree (2 spots) than last (6) run. Its trajectory and functional appeal make it a language to watch, but whether or not Elixir can sustain this momentum is the important question. As even very popular languages like Swift have proven, the difficulty of growth is proportional to the rankings themselves – as one rises, so does the other. It’s also worth noting that Erlang has not seen a bounce from Elixir; it was static this period, holding at 26.

  • Julia: Julia’s growth has always been slow, but this is the first period in a number of quarters where Julia actually slid. Having moved up to #51 last quarter, it slid back to #52 for this run. This is not particularly surprising, as the language is not currently demonstrating the traction, visibility and enthusiasm characteristic of faster adoption rates. We’ll watch over the next quarter or two to see whether Julia can resume its climb, or whether it has stalled in a manner similar to CoffeeScript.

  • R: Out of all the back half of the Top 20 languages, R has shown the most consistent upwards movement over time. From its position of 17 back in 2012, it has made steady gains over time, but had seemed to stall at 13 having stuck there for three consecutive quarters. This time around, however, R took over #12 from Perl which in turn dropped to #13. There’s still an enormous amount of Perl in circulation, but the fact that the more specialized R has unseated the language once considered the glue of the web says as much about Perl as it does about R. Which is irrelevant to R advocates, of course. Whatever the cause, R’s relatively unique Top 20 path is one for fans of the language to cheer.

  • Rust: Interestingly, given that the past two quarters have anecdotally seen an uptick in Rust discussion, the language actually followed Julia’s lead and gave up one spot in the rankings this quarter. From a big picture standpoint, this is not particularly problematic, given that individual ranks should be taken with a grain of salt always, particularly so the further down the rankings a given language sits. That said, upward trajectories are preferable to the opposite, even if the actual rankings themselves are not to be obsessed over. Like Julia, it will be interesting to see whether or not Rust will gain next quarter or if it has instead plateaued.

  • Swift: Swift at this point has become the canonical example for the inertia of incumbent languages. Have followed an unprecendented growth trajectory since its introduction, this run is the first in which Swift has not gained but merely held its position of #17. In Swift’s defense, it at least performed better than the language directly ahead of it, Haskell, which fell out of a tie with Go for 15th place into #16. But it’s clear that further gains for Swift will not come easily, and will instead be the product of widespread usage across an array of communities. As discussed in the last iteration of these rankings, Swift has opened up new avenues for growth beyond iOS development via its release as open source software and the embrace of third parties like IBM or Perfect, but these have yet to yield gains in new discussion or code sufficient to propel it forward in these rankings. We’ll be watching for signs of this type of new growth closely.

  • TypeScript: Outside of Go or Swift, the fastest growing language we’ve observed in recent years is TypeScript. The Microsoft-backed JavaScript superset and Angular 2 foundation has made significant gains for the second consecutive quarter, jumping from #31 to #26. That was the biggest single change in any Top 30 language, and the second largest jump overall (Standard ML, 7 spots). At #26, in fact, TypeScript is now tied with Erlang, one spot behind Powershell and four behind CoffeeScript, which is just outside the Top 20. The question facing the language isn’t whether it can grow, but whether it has the momentum to crack the Top 20 in the next two to three quarters, leapfrogging the likes of CoffeeScript and Lua in the process.

The Historical Rankings

As we did last quarter, this visualization will allow you to dynamically select or deselect languages at will, tracing their individual rankings back to the first runs of this exercise.

A few notes:

  • This is not a complete ranking of all the languages we survey. It includes only languages that are currently or have been at one time in the Top 20.
  • This graphic is interactive, and allows you to select as many or as few languages as you prefer. Just click on the language in the legend to toggle them on or off. This is helpful because Swift fundamentally breaks any visual depiction of growth: de-select it and the chart becomes much more readable.
  • The visualization here, courtesy of Ramnath Vaidyanathan’s rCharts package, is brand new and hasn’t been extensively tested. Mobile may or may not work, and given the hoops we had to jump through to host a D3-based visualization on a self-hosted WordPress instance, it’s likely that some browsers won’t support the visualization, HTTPS will break it, etc. We’ll work on all of that, and do let us know if you have problems, but we wanted to share at least the preliminary working copy as soon as we were able.

With that, we hope you enjoy this visual depiction of the Historical Programming Language Rankings.

The Net

In general, these rankings are experiencing less volatility over time. The Top 10 in particular is fairly static, and even within the Top 20 movement is becoming more limited. It may be, as mentioned above, that we’re at or near peak fragmentation, and that the Cambrian explosion of programming languages is leading to a predictable period of consolidation. We’ll look at this question in more detail shortly.

Even if that is the case, however, it is not true that there are no interesting trends to watch in the programming language landscape. R’s continued ascent is interesting, particularly following the acquisition of Revolution Analytics by Microsoft. Another language with ties to Microsoft, TypeScript’s ascent is as notable as it is surprising, and whether Swift can pass Haskell and potentially even Go will be fascinating to watch. In the last few spots in the Top 20, meanwhile, there are suggestions that change is coming: Clojure, Groovy and Haskell all fell back this quarter.

We’ll be back with you in two quarters to assess these and other questions.

Categories: Programming Languages.

Hark Episode 3, “Getting Medieval on You”: Guest, KellyAnn Fitzpatrick

As newsletter subscribers are aware, Episode 3 of my podcast Hark, “Getting Medieval on You,” dropped the first of July. In this month’s episode, KellyAnn Fitzpatrick dropped by to get medieval on everyone. We discussed everything from Dungeons and Dragons to Zelda, Game of Thrones to Tolkien, the Arts and Craft Movement to Shrek…wait, what? Yes, that Arts and Crafts movement. Victorian England. Stephenson’s the Diamond Age. Ready Player One. The History of Rome podcast. And more. If you like references, this episode’s for you.

For those of you who don’t do podcasts, however, we offer a transcription of the episode below. Enjoy.

Stephen: Maybe it’s the location. Maybe it’s the great people who attend every year or maybe it’s just the beer. But we get a lot of proposals to talk at the Monktoberfest every year. Depending on sort of which year we’re talking about, there’ll be somewhere between 10 and 20 proposals per open slot. Competition to speak, in other words, is fierce.

Every so often, however, there is a talk that is so unique that it’s accepted even before I get around to reading the abstract. KellyAnn Fitzpatrick’s talk was one of these. I got a note one day from one of her coworkers that read, “I’ve got a proposal for you from our technical writer who’s writing her English doctoral thesis on MMORPGs.” To which my reply was a polite but non-committal “Interesting, have her send it along to the link for the CFP.”

Then the talk proposal came in and it was titled “Dungeons and Towers: Medievalism, Gaming, and the Academy.” Now let me ask you. If you were running a conference for developers and geeks, would or could you turn down a talk with that title? The answer is of course you can’t. And that’s how KellyAnn came to join the ranks of Monktoberfest speakers. She was kind enough to stop by to revisit the subject with us and to talk more broadly about medievalism, its role and its importance in media, gaming, and society today. Welcome to Hark Episode 3, “Getting Medieval on You.”

Stephen: So excellent. So, welcome to the show, KellyAnn. As we like to do to start the show, can you tell me who you are and what you do?

KellyAnn: My name is KellyAnn Fitzpatrick and I am a writer and probably, more importantly, a reader. And at present, I kind of have this like split path going on in terms of what I do. I try to be an academic and in that, I teach at the University at Albany. I teach classes and I work with the writing center. And I present at conferences for kind of academic English and medieval-related stuff and I attempt to publish. But I’m also a technical writer at a company called Apprenda and we do enterprise platform as a service.

Stephen: Indeed. Yeah, so this conversation, at least from my end, it’s, you know, originally a calling back from your Monktoberfest presentation, which touched on medievalism and gaming and so on. Which is I think for probably a not insignificant portion of our listeners, a couple of topics I think that are potentially of interest. So, you know, I guess my first question is, how did you get into what you do? Like what career path lad you down the “Hey, I teach at the University of Albany. I’m an expert on medievalism,” you know? So, how did you end up where you are?

KellyAnn: I, like many people who were at Monktoberfest, had as a child like a strong interest in the works of J.R.R. Tolkien and the videogames that kind of come out of his work. So I’m from the generation of the first Nintendo so I played “Zelda.” Before that, I played “Gauntlet” not in the arcade but on my cousin’s Tandy.

Stephen: Oh, nice.

KellyAnn: And if anyone actually remembers that, you can kind of date me there. So, between that type of kind of games that I was exposed to as a child and the things I would read, I became interested in the Middle Ages. You know I read up on it. I had some type of history class in high school that world history and we kind of covered the Middle Ages. This was kind of before the internet had this wealth of information that it does today. So I would spend a lot of time at a library looking at actual books. And, when I ended up going to college, I thought I was gonna go and be a doctor. But I got to college and I did my…

Stephen: Didn’t we all? Yeah.

KellyAnn: Yes, we all thought we were going to be a doctor or something else. I went to… I did my undergraduate at the University of Notre Dame and they have an excellent Medieval Institute there. The entire seventh floor of their library is kind of dedicated to the Medieval Institute. And they have it not only as a space but also as this kind of like cross-disciplinary program in the university.

So I thought I would be…go. You know, I would come out and go to medical school. But by the time I got through college, I decided I wanted to actually pursue Medieval Studies and kind of English jointly. So that’s kind of what set me on the track of learning more about the Middle Ages.

At the time, I didn’t entirely understand that the actual study of the Middle Ages, like looking at the history, you know, culture, literature necessarily tied in to the games that I played as a kid. Until I stumbled upon a couple… there were a couple of different books, a couple of different scholars that were very important for me for this. One is Tom Shippey who wrote a book called “J.R.R. Tolkien: Author of the Century” in which he talks about J.R.R. Tolkien as an author, as a product of the 20th century not necessarily the author of the 20th century. But that really, you know, kind of turned my interest to Tolkien in a more academic way. And then Tolkien himself, of course, was an Anglo-Saxon scholar. And so there are these ties to the types of work I was doing with like “Beowulf” and actual medieval literature.

And the second scholar that really kind of turned me on this path is… her name is Jane Chance and she also did a lot of work with kind of like Tolkien’s work and the images you come out of Tolkien. But she also was an Anglo-Saxonist so she’s done kind of a lot of “Beowulf.”

So you know somewhere between the end of my undergraduate and the beginning of even my graduate studies, I kind of had this in my head that this is what I wanted to do. And I’m glad I ended up this way because it ties into a lot of things that I work with now. I mean I’m surrounded by kind of technical people all the time. So the fact that part of what I study is the Middle Ages in videogames, I actually have a lot more kind of understanding of where those videogames come from and how.

Stephen: No, I can certainly see that. So that actually sort of begs the question. As you mentioned, just dating back to “Zelda” and “Gauntlet” and obviously going through to board games, you know the “Dungeons & Dragons” up to a lot of today’s present day games, “World of Warcraft” and so on. There’s obviously a lot of ties between medievalism and gaming which was again a big part of the presentation you gave at Monktoberfest. Where does that date to do, do you know? In other words where did that all start? How did these ties originally form?

KellyAnn: And I think I covered this in my Monktoberfest presentation. But even before, I think it was necessarily a videogame tie, it was like a game tie.

Stephen: Right.

KellyAnn: So “Dungeons & Dragons” is for me the big space where you have medievalism and gaming coming together in a way that becomes just very, very popular. There are like manifestations of, you know, medievalism that you can kind of see maybe hopping up in like smaller spaces. But that becomes the basis for taking this idea, not just of the Middle Ages, but of like medievalism itself. So the type of like high fantasy world of Tolkien that Tolkien kind of puts together and translating that into this game system to the point where almost anytime you see a game that has medieval components, especially the earlier ones like something like “Zelda,” there’s a dungeon in it.

Stephen: Indeed. Yeah, I remember the screenshot from the Mocktoberfest presentation. So you know that’s sort of some of the roots. Where do you think the appeal comes from? Because to be perfectly honest, and sort of I guess by way of full disclosure, when I was a kid I was very into medievalism as well. You know, loved Tolkien, read tons and tons of medieval history, took clearly not as much medieval history at undergrad as you did. But certainly, it was an area of interest.

And, I couldn’t honestly tell you sort of where that came from or what the appeal was. Right? I mean, do you have any sort of better sense of that having studied it and obviously embraced it on the game side as well? In terms of you know culturally, we look at the phenomenon like “Game of Thrones,” do you have any idea sort of where that universal appeal for the time period comes from?

KellyAnn: There are a number of theories about this. So any information I kind of you know give you on this has been talked about and debated and probably comes from people who are much smarter than myself. But one of the bigger ideas is that the Middle Ages, and even especially in the versions of the Middle Ages, in the medieval that we get kind of passed down and modified and appropriated in these types of games. It’s a break from what we’re dealing with today. It’s a break from what we consider the modern or post-modern world. So that it’s so different from what we do. That most of us get up and go to work every day.

Stephen: Right.

KellyAnn: You know, some of us get to get up and work in our pajamas which is always kind of fun.

Stephen: Guilty.

KellyAnn: But yeah. But the idea that this a break from that and that it’s a space where you get to control the position that you imagine yourself in. So, if you think about something like medieval Times, the restaurant where you go and you watch jousting. Right? Every customer who walks into medieval Times gets to wear a crown. And everyone is kind of the king, even though there is a king and a princess and things like that. But everyone is treated as if they are of this kind of like noble class within that. And they get to cheer for their knight and they get to rip apart a chicken with their hands because they won’t give you a fork.

So it’s this very fun way of imagining the Middle Ages. There’s really no space where you go and you’re like, “I’m gonna be the peasant who’s cleaning up after the horse” at medieval Times. Right? Thank goodness!

Stephen: In spite of the fact that, you know, according to the statistical probability if you look at the actual distributions of populations of the time period, if you’re gonna be anyone, you’re probably gonna be a serf. But we’ll sort of let that go.

Actually, on a facts bent, as long as we’re sort of talking about that. As you’ve brought up, and I think you used the example of dungeons being equated with medievalism in spite of the fact that they pre-date and post-date the period. As somebody who is academically an expert in the field, you know has studied it a great deal, does the sort of factual departures, and I’m not even talking about things like dragons which don’t exist. But, in other words the liberties, I guess, that some of the games or sort of other media take with the period. I mean does that bug you at all? Is that something that you’re fine with as a fictional liberty? Like from an academic standpoint, how does that strike you?

KellyAnn: It strikes me as good that we can take something like the Middle Ages and change it and alter it. There are a number of different ways to kind of approach that. For instance, if I were teaching a course on medieval history, I would probably be horrified by it in some ways. In that, if I walked in the first day of classes, say it was like an undergrad class on medieval history, some of the kind of preconceptions that I would expect from my students about what the Middle Ages are could be difficult to deal with. Right?

So the idea that they’re always associating the Middle Ages with like princesses and dragons that don’t exist and dungeons.

Stephen: Right, right.

KellyAnn: And there is like a school of thought that looks at that and is just kind of banging their heads against the wall. Like “No, this is not what the Middle Ages are. This is exactly what it is.” But there’s also the argument that the term medieval itself is a construct. It’s something that even within the realms of academia that we’ve kind of put together. Not that we can’t point to kind of what we would think of as a period in history and maybe point to like this happened on this date. But, there’s no event where it was announced to the world that the Middle Ages have started.

Stephen: Right.

KellyAnn: Or the Middle Ages have ended so, you know, therefore everyone go out and be modern.

Stephen: Yeah, yeah, yeah.

KellyAnn: So from that point of view, it suddenly becomes less disturbing. Even if you’re teaching something like history to think of the Middle Ages and the medieval as something that is kind of taken off and changed by people. And to me, the interesting thing is what we do with it.

Stephen: Right.

KellyAnn: For…

Stephen: Go ahead.

KellyAnn: For instance, in the 19th century, the idea of Medieval Studies itself like as a contract, construct and as a discipline. I mean that’s very much in a response to like this 19th century need to kind of categorize things and understand them and give this kind of empirical view of the world. And videogames are very much a 20th and 21st century way of taking the Middle Ages and using it as a response, I think.

Stephen: Yeah, so that actually so let’s… We’ll come back to the modern incorporation or modern, I guess, iteration of medievalism. But let’s go back to Tolkien for just a second. So, from a history standpoint, you know how did he sort of arrive on that? Obviously, he was a sort of linguist by trade. But, you know, was he one of the first to pick up on a lot of the history and incorporate it? Or is this… You know, was he drawing on traditions of other storytellers? In other words, how did that first get going that medievalism became sort of a “thing” so to speak, from a fictional narrative standpoint?

KellyAnn: Yeah. So Tolkien is… He’s born at the end of the 19th century and he’s very much a product of the 19th century. Both in that he becomes a linguist, he becomes a philologist. And philology has its own kind of like history. And it’s something that was still going on and have been a profession that he could enter into when he came of age.

But it also means that, you know, when young Tolkien decided to start reading, he could read things like William Morris especially. So he inherits this kind of like 19th-century medievalism and there is a very big, what we call the “Medieval Revival” in Britain in the 19th century. And William Morris is one of the author’s that he like cites directly as someone he had read. That he actually tries to… when he’s maybe like in his late teens, he tries to write a story after the style of William Morris. And for those of you who… Like I don’t know if you’re familiar with William Morris?

Stephen: I am not.

KellyAnn: He was this 19th-century kind of like polymath. He did everything. Like he wrote, he painted. He started a publishing company, the Kelmscott Press. Morris & Company, the kind of if you’ve ever heard of that, the type of design firm, Morris & Company, that’s what he started as well.

Stephen: Interesting. Okay.

KellyAnn: He’s part of what is known as the Arts and Crafts Movement.

Stephen: Okay.

KellyAnn: Which…

Stephen: I was gonna say Kate’s a big fan of that so.

KellyAnn: Well it’s great. And in the context of kind of like 19th century Britain, if you can imagine what we often think of when we think of like Victorian décor. And you can think of maybe of a lot of like frills and lace. And things kind of being produced at a time when the Industrial Revolution is in full swing. So a lot of the decor that people would have in their homes like furniture would be like mass produced, the result of factories, like coming out of factories. And the decor itself often being very cluttered.

Stephen: Mm-hmm.

KellyAnn: Morris and other people kind of involved in this Arts and Crafts Movement look at that and they’re like, you know, “This is terrible. We’re getting all this horrible, cheap furniture and everything is cluttered.”

And for Morris, he actually turned to the idea of the Middle Ages in terms of design. So he saw the Middle Ages as this space where you could kind of pull out these more like clean designs. And the way things were produced were not on this mass scale. That they were done by hand through these like kind of artisanal craftsmen. And that the labor involved…and this was very important for him because he was also a socialist. The labor involved in a craftsman creating a chair is very different than the number of people in a factory who are maybe putting different pieces of the chair together.

Stephen: Yeah, yeah.

KellyAnn: So for him, he has this idea, very, very polymathic understanding of the world and what he did with it. But one of the things he also did was he wrote. And he wrote travel narratives of his own travels. He did translations. And then towards the end of his life, he started writing what he called these kind of romances that were these part fantasy, part semi-historical fiction narratives, and these influenced J.R.R. Tolkien a lot.

Stephen: That’s actually, that’s… Particularly the angle that resonates with me, well, there are several, but the idea of the Arts and Crafts movement. And essentially almost a rejection of industrial production actually reminds me of I wanna say it’s “The Diamond Age” by Neal Stephenson. You know, it’s essentially a world where they have, sort of I guess the term we would use today would be 3D printing. So you can 3D print basically anything. And yet, you still have craftsmen which they call “Vickies,” obviously harkening back to Victorian England who manufacture things by hand. And there is a value attached to that. You know, just as being distinct from things that are sort of mass produced by 3D printing.

But that’s obviously an aside and not necessarily on the topic of medievalism. But going back to sort of the Tolkien topic. So Tolkien fed into, I assume and maybe this is a direct link or maybe it’s indirect, Gary Gygax and the foundation of “Dungeons & Dragons.” Do you know how to trace that? What the direct influence was or lack thereof?

KellyAnn: There… I mean and I’m not an expert on kind of the exact places where “Dungeons & Dragons” is pulling from Tolkien. But to look at the different types of what we would consider like races involved like elves and dwarves. And then the one that is probably the biggest giveaway would be what in “Dungeons & Dragons” would be a halfling.

Stephen: Mm-hmm. Yeah, yeah, yeah.

KellyAnn: Because using the word hobbit would be probably something actionable by law.

Stephen: Yeah. So similar but legally distinct, shall we say?

KellyAnn: Yes, very legally, legally distinct.

Stephen: Yeah. So, yeah. It’s always interesting to me because it’s one of those things that are obviously trends and fads that come up from time to time. And, you know, each generation has its time period that they may sort of focus on. But sort of the interesting part of medievalism for me is that it really is pretty enduring. Right?

It seems like… you know, I can remember reading, oh, god. They had to be published I guess in the late 1800s. It was books that were in my grandmother’s house growing up, crusader tales and things like that. All the way through “Dungeons & Dragons” and obviously we have the sort of mass-produced media today like “Game of Thrones.” And it’s all sort of very, very medieval. So, it’s had a longevity that I don’t know that I would have expected sort of given the time period.

KellyAnn: It does and there’s the argument that medievalism existed before the middle ages ended.

Stephen: Gosh, that’s excellent. How so?

KellyAnn: Yeah. Well, a great example of this would be the Tudors and that’s another topic that has kind of been taken up and re-imagined in popular culture like with the series “The Tudors” about Henry VIII of England. But Henry VIII’s father, Henry II, and this is probably like late 15th century, early 16th century we’re talking about then. He actually ends the War of the Roses in England, which you’ve had different factions fighting over who is gonna be king. The Lancaster and York are the two names that kind of get thrown out. And interestingly enough, another aside, that is one of the basis for “Game of Thrones,” right?

Stephen: Yeah. Yeah. Right. Yeah.

KellyAnn: The War of the Roses in England. And you can see the name was translated. Like “Lannister” for Lancaster and “Stark” for York and things like that. That’s an entire other study, right?

Stephen: Of course, yeah.

KellyAnn: But when Henry VII, he claims the crown by conquest. Basically, he kind of has a claim on the throne through blood. There are a whole bunch of people that have a claim to the crown ahead of him but he wins. His armies are triumphant. So, he gets to be, you know, King of England. And he does a very smart thing in that he marries the York princess, Elizabeth of York. So he kind of like ties the two houses together in some ways.

But the other thing he does is he really plays up on Arthurian legend. In that he, he names his first son Arthur. And you know as much as possible goes… you know, relies on the legends of like King Arthur and claiming descendency from King Arthur which is always the thing to do in England, if you can.

And his son, Henry VIII takes this up in that he… There is this giant round table in Winchester and you can kind of go see. It’s hanging up on the wall. I think it dates from one of the Edwards, like Edward II or III or whatnot in terms of when it was made. But he has it repainted in Tudor colors with a giant Tudor Rose in the middle of it. So that the Tudors themselves in terms of establishing their own legitimacy to the English throne use this older idea of King Arthur which is like going back to even a perceived earlier time in what we would consider to be the Middle Ages.

So, even then you have this idea of medievalism, this looking to the past. This idea that you can find this knightly or courtly or chivalric tradition. And use it as a basis for anything is happening even then.

Stephen: So would you say that’s one of the reasons I guess that would keep medievalism and sort of medievalism-themed media, be that games or TV or movies you know, competitive an environment? Like looking at games is just one example. Games like “World of Warcraft” more than hold their own, at least from what I can tell as a casual observer, against the “Calls of Duty” things like “Grand Theft Auto” which are certainly big titles in and of themselves. But medievalism is again basically very competitive in that environment. So do you think it’s basically that sort of look to the past, that escapism again that keeps it on…well, yeah, that keeps it sort of a first-tier option as far as media goes?

KellyAnn: I think so. And, the other element that kind of works into that, especially in other…even outside of gaming and definitely within gaming is the possibility of magic.

Stephen: Okay.

KellyAnn: Right? So, one thing that you can do in “World of Warcraft” and other types of games like “Dark Souls,” there’s always some type of sorcerer…

Stephen: Right, right, right, right.

KellyAnn: …or mage. You know, there are these elements of magic that you don’t necessarily find something like in a first person modern day shooter.

Stephen: Yeah, yeah, actually it’s funny you bring that up because I just read “Ready Player One.” In “Ready Player One,” it’s this sort of large virtual universe is one of the settings. And they have to sort of explicitly define in any given region, “Does magic work here or not? Does technology work here or not?”

So you can have Tolkien-like worlds where no technology is allowed but magic works. You can have “Blade Runner” worlds where technology works and magic doesn’t. You can have other areas where they both work at the same time. So yeah, it’s… I don’t know, the interplay is, again not being a gamer. It’s always one of those things that’s sort of more academically interesting to me. But yeah, like I said, I think it’s just very curious to see. Like I said the longevity of the material because it’s… You know, sort of on paper, you’d think, “Hey, kids are typically obsessed with things that are new.” Right? In other words, well for a lot of good reasons, political correctness among them, you don’t see things like cowboys and Indians anymore. There’s not too many kids who are interested in being cowboys. But, they are still interested in being knights and going on quests, which, yeah, I find that really interesting.

KellyAnn: Yeah and it’s something especially for kids that gets tied into the whole idea of fairy tales as well. So, like Disney culture. There’s an excellent article that I read while I was completing my dissertation. And it talks about how in the kind of Disney realm, in just, you know, talking about the films and specifically the princess movies, which there are many of. That if you ask the typical person on the street, how to describe all these different things, the term medieval comes up to describe almost all the fairy tales and all of the princesses. Even if it’s one that’s not even necessarily in a medieval context or a medieval setting.

Stephen: Interesting. Okay.

KellyAnn: Yes. So if you think back just kind of like the new classic and new Disney movies, something like “Sleeping Beauty”is it’s set very firmly something in the 14th century. Right?

Stephen: Yeah. I was gonna say you don’t think of it that way but that’s absolutely true.

KellyAnn: Yeah. In that, you can make the argument that this is a medieval fairytale. But if you look at something like “Snow White” or “Cinderella,” which is “Cinderella” is very much more 19th century in context. Even that gets branded as medieval or like associated with the Middle Ages for a lot of people. To the point where something like the princess or the idea of the princess being rescued and all the other kind of like tropes that kind of go with it are… they become these medieval tropes. So you can’t really go through much of the fairytale culture without getting some idea or glimmer or redirection to the medieval.

Stephen: So what do you think… You know, sort of when we think about those medieval tropes, right? In other words obviously, one of the things about tropes is they are to some degree narratively helpful in that they are constructs that a reader or a listener or a viewer will understand, sort of instinctively. They have some frame of reference to deal with. But on the flip side, they’re also at risk of course of becoming clichés. Right? “Okay, yes, it’s been done a thousand times. I’ve seen this. Like give me something new.” Are there aspects to medievalism that you think are sort of undermined, not undermined but sort of under, I don’t know, underappreciated, not yet discovered that may yet emerge?

In other words, when you’ve been sort of doing… well when you were doing your undergrad research, were there aspects to it that have yet to be embraced? I’m thinking of things like “Game of Thrones.” Right? “Game of Thrones” obviously incorporates a lot of the standard tropes, you know, dragons, sort of magic and things of that nature. But also is for example, exploring currency issues in sort of one of the plot lines and so on. So are there things like that you’ve studied that you think have yet to emerge but will begin to pop up in medieval media moving forward?

KellyAnn: I don’t and I don’t know if I can think of something that I have studied that I haven’t seen pop up somewhere. But one kind of image of the Middle Ages that I think I would like to see more would be more of what we get through “Monty Python and the Holy Grail” which is this very almost like disjointed version of the medieval narrative. Right?

And if you can kind of look back at it, it’s like it kind of has a story. Arthur is going to find the Holy Grail. But all these other things just kind of like pop up and thrown in and very clearly are being used to talk about other things. For instance, there’s the… Arthur shows up at this kind of farm. Right?

Stephen: Mm-hmm.

KellyAnn: This is the one where he’s being questioned as, “Well, what do you mean you’re the king?” So you’re very much talking about these kind of leader politics being re-represented in this. What is allegedly this movie about King Arthur and the Middle Ages. So, that where it’s just very clearly pointing out this is how we’re using the Middle Ages kind of forward our own agendas or our own ideas. And doing it in this very just disjointed way. I think I’d like to see more that or I think we’re hopefully going to see a little bit more of that where it’s not necessarily the simplistic, relying on of tropes but actually turning these tropes on their heads.

Stephen: Yeah, which actually is a perfect segue to the next question, which is, why should we today continue to study medievalism, right? And obviously, I have my own sort of answer to this as somebody who is certainly not an expert in medieval history but somebody who is a fan of history generally. But I’m curious, for somebody who is very well-versed in this particular time period, what are the things you think it offers obviously beyond the entertainment value that make it a subject that is worth us studying and really understanding in a fundamental way today?

KellyAnn: I’m very curious to hear what your answer is.

Stephen: Well, we can… I can go first if you like.

KellyAnn: Yeah, please.

Stephen: Yeah. I mean I think from my perspective, always having some distance from a historical standpoint is useful. One of the really interesting things, I think… I don’t know if you’ve listened to the “History of Rome” podcast but one of the things I thought was fascinating about that, aside from the fact that I thought I was somewhat versed in Roman history before I started listening to that. And as it turned out, I was not at all. I’d forgotten – I’d either forgotten or not known most of it. So, for anybody listening to this if you haven’t listened to that podcast, I would highly recommend it.

But anyhow, one of the interesting things that essentially Mike Duncan sets up in that podcast is how essentially the decline of Rome leads to essentially the formation of feudal states. Which obviously later give rise to what we would consider the medieval power structure. Right? You know, feudal lords and serfs and so on. And that transition from basically the large, centralized Roman Republic to feudal states is one that, as I said, I frankly had not understood very well.

But also, I think it offers some very interesting and important lessons for us today. Because you know it’s… Comparisons between the decline of the Roman Empire and sot of the United States are in my opinion and I would… Actually, I know this for a fact. He wrote about it. Mike Duncan’s as well I think are overblown. But that doesn’t mean there aren’t parallels and there aren’t a lot of lessons to be learned in terms of okay, this is what happened to a republic that made certain decisions. This is where we ended up.

And studying the history of the medieval period, these are some of the long-term ramifications. Right? In terms of disruptions to commerce, disruptions to learning. Frankly higher rates of death really across the board from infant mortality all the way to adult mortality. So I think it is, from my perspective, just again certainly not being an expert but just knowing what I know of the medieval period. I think that there are a lot of important, illustrative lessons in terms of where things end up, you know, if you make certain decisions along the way. So, that’s certainly one reason, I think. You know, I’m sure there are hundreds but that’s certainly one reason I would imagine the medieval period is still very much worth studying. But yeah, like I said I’m curious to hear what you have to say.

KellyAnn: Yeah and that’s well said. I’m glad I made you go first. I think you said it better than I would have. And then just to kind of add to that train of thinking. Not only like the fall of Rome and you can kind of see the rise of these feudal states. But within the medieval period, we actually see the decline of feudalism and this nascent capitalism, the beginning of like our economic system as we have it.

Stephen: Right.

KellyAnn: So like that in terms of going back to history and seeing what we have in terms of record, there’s definitely some stuff worth there. But also, in terms of just kind of like history of nations and how nations rise and fall, the Middle Ages has that as well. Towards the end of the Middle Ages and early Renaissance, of course, we have in Europe especially religious wars. Right?

Stephen: Mm-hmm. Yup.

KellyAnn: So you have Protestants and Catholics. Not that many people that even I know who aren’t necessarily academics go back to that and understand how much fighting within Christianity there actually was.

Stephen: Oh, yeah, absolutely.

KellyAnn: They do know there is a drink called “Bloody Mary” but not necessarily that it’s named after a Queen of England who was a Catholic and put a number of Protestants to death. So there’s a lot in history for us to kind of go back and learn and we can kind of draw parallels to as well.

Stephen: Yeah. Yeah, I think it’s, it’s… Unfortunately, it’s always a sad commentary. But the Santayana quote that “Those who can’t learn from history are doomed to repeat it” really does seem to be true. You know, I don’t know. I guess the older I get, or whatever it is maybe I’m just getting old and curmudgeonly, but it really is.

You see these patterns sort of play out. And when you have a period of transition between one time period and another, right? So in other words, we go from sort of history of Rome and that goes back, that transitions out to the medieval period where we have a much more fragmented, much more sort of nation state. You know in many cases, sort of warlord type of distribution of power, distribution of economic systems and the consequences of that. Like I said, I think there’s a lot to be learned there. But we have to pay attention to those lessons which is always the trick.

KellyAnn: Yeah. And then we go back to your original question of, “Why should we study medievalism,” right?

Stephen: Yup.

KellyAnn: So not just the medieval period but the manifestations of the medieval over and over again. I think it’s vitally important to understand the way that we are using these tropes in our own kind of rhetoric and arguments today.

And just one of the easier examples is the trope that we have is that men go out and fight and women should be at home.

Stephen: Right.

KellyAnn: I mean that’s something that we locate in the Middle Ages often with… “Rescuing the princess from the dragon” is probably one of my favorite fairy tale medieval tropes. That’s something that gets held up and used to perpetuate different types of gender roles. Even in something like Tolkien, and I love Tolkien and I grew up loving Tolkien, but I’m reading about him. I’m like this is a story about boys. I don’t wanna to be the elf princess who sits around or the one woman who gets to go and pretend, you know dress up as a man and fight. I want to be Gandalf.

Stephen: Right, right, right.

KellyAnn: Right? So that’s definitely something where medievalism is not going away. And if we just kind of let it happen without being critical of it in any way, shape, or form, there’s a lot of danger there.

Stephen: Yeah and I hadn’t really thought about that. But I think it’s been interesting I guess to see… I think you said this earlier, to essentially turn it on its head in some degrees. Like in other words I’m thinking when you were talking princess and rescued by the prince, it made me think of “Shrek.” Right?

KellyAnn: Mm-hmm.

Stephen: Where you have obviously a comical retelling but a comical, like, “All right, well let’s turn this trope and let’s turn it around.” Actually, the princess is quite capable and doesn’t need to be rescued theoretically and so on. So I think, yeah, it is interesting to think about how medievalism has obviously influenced many of the movies that a lot of us saw as kids in Disney movies. And what that passes on to us in the form of things that we don’t even think about. We just take for granted.

KellyAnn: Yeah.

Stephen: Like, “Oh, a princess in a castle, must be rescued.” Well, why? You know, is that essentially a gender stereotype that we want to persist or is that something that is not helpful that we need to retire?

KellyAnn: Yeah and “Shrek” is a great example, I’m glad you brought it up. Because that’s… In my field, we kind of look at that as a movement in medievalism that gets categorized under the name of neo-medievalism, which is a term that is still in development. But the idea of to take these tropes and turn them on their head and see the idea of the medieval being constantly reinvented and “Shrek” is a good example for it. As you point out, it takes the “rescue the princess” element and turns it on its head.

But probably one of the most pointed ideas in that movie is that Fiona, the princess, she’s patiently waiting in that castle. Because it’s kind of like she’s been told that this is how this needs to be even though she is this very capable princess. She almost goes through it as a ritual and then when she… It’s not gonna work out that way, she actually kind of gets to come into her own.

Stephen: Yeah. Yeah. No. Like I said, I think history is always very useful I think for… you know, as a way of removing ourselves from a situation and allowing to view it critically. Right? Because sometimes obviously it can be difficult to evaluate current culture sort of in a self-critical way. Right?

Because you get caught up in “Oh, I know somebody like that” or “that’s like my parent.” You’re basically too close to it. So by removing sort of… or introducing distance via different time periods, like I said, I think you have the opportunity to use it as a mechanism to shine a light on things that are present day. You know, that maybe, well at least according to narrative hundreds or thousands of years removed from the actual events.

But anyway, okay. So we’re sort of getting close. One question I want to get in before we wrap up here. As somebody who obviously is a gamer today, I wanted to make sure to sort of touch on that at least. Since we’re talking about medievalism and gaming, what do you think the role of gaming is? So we’ve been talking about medievalism as a way of learning about ourselves and learning about different aspect to our society. What role do you see gaming playing either through medievalism or on just its own?

KellyAnn: That’s a big question.

Stephen: It is.

KellyAnn: I think it probably has a big answer. The super short answer is that gaming right now it’s a commodity. Right?

Stephen: Mm-hmm.

KellyAnn: It’s something that is commercialized and for the most part kind of sold and bought. And the industry that drives it is one that is mostly driving for profit. That’s the simple answer. So that the reason we keep seeing medieval style games is because people will pay for them. Right?

That said, I almost suspect that if we stopped making them, people would make their own.

Stephen: Yeah.

KellyAnn: That there is a phenomenon in and of itself separate from this exchange of goods that people would use as a way to kind of create them on their own. And, I think people already do that.

The other answer is the idea of gamifying things is becoming very prolific outside of just the industry of game production. To the point that where any time we’re talking about teaching, we’re talking about how we can work some type of gamification into that or training or anything like that. There are entire fields of study around game theory. So it’s a very big field. And I’m glad about that because it’s, to me, a very interesting one.

Stephen: Yeah. So last question from me then. How do… when you know, people making movies, people making TV shows, people making games that are medieval in nature, how do they… How do you think or how do you know they educate themselves? Like how do they get up to speed? Because a lot of them are just writers. They don’t know anything about the time period. What is the interplay there? In other words, how do experts like yourselves connect with people in those disciplines? And make sure that, all right, you’re not gonna adhere necessarily to the , you’re not gonna replicate the setting exactly. But don’t make obvious mistakes and hey, here are some things you haven’t taken advantage of?

KellyAnn: That’s a good question too. The idea is, is the job of a film to be historically accurate? Especially if it’s based on something that’s like fictional. And the best example I can think of that is like “The Lord of the Rings” and “The Hobbit” movies where it’s not necessarily historical accuracy but attempting to somehow be authentic to Tolkien. But at the same way, translate it into this other type of field. I know that that when those films were made, there were a number of Tolkien experts brought in, probably experts that I don’t… that I can’t even think about. Like how do we make this armor?

Stephen: Yeah, yeah.

KellyAnn: How do we make these weapons? How do we put together a fighting style?

Stephen: Yep.

KellyAnn: So it’s not necessarily just historical accuracy but all these other types of things we tend look at and be like “Is this authentic or is it not?” So I guess it really depends on the purpose of the project and what we consider to be accurate. If we’re thinking of history or if we’re thinking of being close and true to somebody else’s vision or work.

That said, I do know in terms of game development often there’s a role given a name something like the “Lore Master” whose job is to at least make sure that in…

Stephen: Lore Master. That’s awesome!

KellyAnn: I know. I love that. I think that’s what I might want to be when I grow up. But their job is to make sure there is some type of integrity at least within the world they are creating.

Stephen: Sure.

KellyAnn: To have that type of like internal logic be held whether or not it is tied to something on the outside.

Stephen: Yeah. Sure, sure.

KellyAnn: I think that’s what we tend to see. Like we would look at a film or game and be like “This makes no sense” or “This at least has a coherent vision.”

Stephen: Well, that has to be difficult too because I’m sure in many of the fictional worlds there are people who know them a lot better than the writers do. Because they obsess over it in the way that writers… You know, writers have a job of advancing narrative and worrying about characters. But fans can be obsessive in a totally, totally different way.

KellyAnn: Yes. The cliché example of that is at Star Trek Convention like…

Stephen: Yeah.

KellyAnn: Trekkies know more about it than like the actors.

Stephen: That’s right. That’s right. I hadn’t even thought of that. All right. So this has been great. I have one last question for you and that question is what animal are you most frightened of?

KellyAnn: Aside from humans…

Stephen: Aside from humans.

KellyAnn: Because I have to qualify it that way. I think mosquitoes.

Stephen: Mosquitoes? Disease bearers you mean?

KellyAnn: Yeah, disease bearers.

Stephen: Yeah.

KellyAnn: I mean we’ve got malaria pretty much dealt with. We have quinine. We have like tonic and our gin and tonic and everything like that but…

Stephen: True.

KellyAnn: All these other things that mosquitos can spread are kind of frightening.

Stephen: Yeah, the Zika virus is terrifying.

KellyAnn: Yeah.

Stephen: Yeah. Okay. That’s pretty fair. With that, KellyAnn, thank you so much for being on Hark.

KellyAnn: Thank you.

Stephen: Thanks again for listening to Hark. As a reminder, you can find us in Google Play, iTunes, Pocket Cast, and Stitcher. You can also listen directly or find links to all the above by heading over to If you have questions, feedback, or suggestions, you can hit us up on Twitter @harkpodcast or via email at [email protected] We’ll be back next month with episode four and until then, enjoy your time.

Categories: Podcasts.

There and Back Again: The MongoDB Cloud Story


Before it was a database company, MongoDB was a cloud company. Founded in 2007 and originally known as 10gen, the company originally intended to build a Java cloud platform. After building a database it called MongoDB, the company realized that the infrastructure software it had built to support its product was more popular than the product itself, and the PaaS company pivoted to become a database company – eventually taking the obvious step of renaming itself to reflect its new purpose.

For the better part of the last decade, MongoDB has more or less played the part of the traditional on premise software vendor. Its model necessarily differed in important ways from that of its closed source, relational predecessors, but in general the software was bought, sold and used much as Oracle or SQL Server might have been. Like every other open source vendor, it may never have matched its closed source competitors in margin or size, but MongoDB generally prospered, emerging from a very crowded open source, non-relational market as a significant and growing commercial vendor.

The landscape around MongoDB, however, was changing.

Two years after the company formerly known as 10gen was founded, Amazon released the first version of its Relational Database Service (RDS) which supported MySQL databases, as a service. RDS support for Oracle followed two years after that, SQL Server a year later, then Postgres and continuing with MariaDB and the MySQL-compatible but AWS proprietary Aurora engine.

As has become typical, others followed in Amazon’s footsteps. Google, Microsoft and other cloud providers offered similar traditional relational database systems online. Closer to home for MongoDB, several companies began offering commercial support for the Mongo database, but as a service. San Francisco-based mLab is an independent provider of MongoDB-as-a-Service; offered MongoDB-as-a-Service among other offerings, and a year ago this month was acquired by the increasingly services-minded IBM.

All of which fit the model outlined within The Software Paradox, a model in which on premise software would give way at an accelerating rate to cloud based implementations. Databases, in other words, would decline in popularity as database-as-a-service grew. Just this morning, an on premise database vendor who considered Amazon, not Oracle, as its greatest threat, admitted that “40% of his customers were in the cloud, and 80% of those were on AWS.”

On Tuesday, MongoDB announced a new product, Atlas, at its annual user summit. Atlas represents an important new step in the company’s history, as it enters the Database-as-a-Service (DBaaS) market. For the first time since its earliest days, MongoDB is once again a cloud company.

Against the wider market backdrop, it is not only not a surprise that MongoDB has decided to directly enter the DBaaS market, this move has been expected. It could be argued, in fact, that Atlas is overdue.

Of all of the various services offered by cloud providers, databases are among the most sticky. This is in part due to the inherent inertia that comes with large datasets that pose logistical challenges simply to move from one place to another, but also to the risks of changing production database infrastructure.

This stickiness can be a positive for vendors, as it decreases the likelihood of customers exiting the platform. But it also means that unlike other workloads such as compute, customers that are lost today may be lost permanently. While a competitive market exists for on premise MySQL databases, for example, with multiple avenues available for commercial support, a far smaller percentage of workloads that enter RDS or other MySQL-as-a-Service products will ever leave those platforms.

Which means that Atlas couldn’t get here soon enough for MongoDB, because workloads lost to Compose or mLab would be difficult to win back. It also implies that other on premise software companies, and on premise database software companies in particular, would do well to add as-a-service capabilities as quickly as possible.

Services are more convenient than on premise alternatives, and as one well known senior engineer said privately yesterday, “Everyone will pay for compute. It’s less clear that people are willing to pay purely for software value up the stack.” Hence MongoDB following in the footsteps of other on premise providers that have made the journey to the cloud and thus have conflated their product with compute in Atlas.

If you’re an on premise software company, you might think about doing the same. You might not believe in The Software Paradox, but the market certainly appears to.

Disclosure: Amazon, Compose, IBM, Microsoft, MongoDB, and Oracle are all RedMonk customers. Google and mLab are not a RedMonk customers.

Categories: Cloud, Databases.

Why LinkedIn and Microsoft Isn’t Crazy

View post on

As far as $26 billion acquisitions paying nearly a 50% premium go, Microsoft’s intent to buy LinkedIn is proving to be one of the most amusing. While most transactions of this magnitude have analysts and reporters racing to assess fit and infer intent, this transaction is most notable for its comedic game. Instead of being besieged on Twitter with financial metrics, the announcement has instead unleashed a torrent of pointed commentary on “Joining my professional network,” Clippy or both. Our local Portland Slack even managed to find a Coldplay/U2 connection.

All of which is fair enough, some of which are legitimately funny, but none of which tells us anything about the transaction and whether it is likely to succeed.

For my part, that is a less interesting question than why the transaction occurred. Forecasting how this acquisition will play out is, to some degree, boring. It is true, for example, that the probability of success for major acquisitions in general is low because acquisitions are hard, and as many observers have pointed out, Microsoft’s recent track record here is not particularly encouraging. It’s also true that many of its highest profile failures, such as Nokia, were transacted before Nadella’s tenure and, it’s said, against his wishes.

The short answer to whether or not Microsoft will eventually be writing down LinkedIn a few years from now the way it has Nokia then is: no one knows.

What we do know, however, is that this acquisition is interesting both in terms of what is being acquired and the valuation Microsoft attached to it, and that the market didn’t seem to appreciate either.

Let’s start with what Microsoft acquired. For all of the focus on LinkedIn the product, it is unfortunately being lost that Microsoft stands to acquire a capable set of technologists. Whatever one might think of the LinkedIn UI – and Paul Ford is not the only one who believes it’s a tire fire – LinkedIn’s infrastructure engineering and data science teams are both smart and capable. Many of the engineers currently lampooning LinkedIn that are happily relying on Kafka, as but one example, typically don’t realize that the latter was built and released as open source by the former. According to its engineering page, LinkedIn is responsible for 75 open source projects across a wide range of categories. Ten years ago at Microsoft, this would not have been an asset. Today, with Microsoft emerging as a surprising champion for open source projects in the cloud and releasing its own software as open source, LinkedIn’s engineering team and their experience with open source are useful.

Not $26 billion useful, of course. The bulk of the valuation depends on LinkedIn the product. Which is another way of saying that Microsoft spent took on debt equivalent to a quarter of its available cash and short term investments on data. Which only makes sense if you value data, obviously, but more importantly have a clear idea of how to use it in meaningful ways to generate revenue.

Queue the Clippy jokes.

But the truth is that Microsoft has acquired a unique asset in LinkedIn, the company’s financials notwithstanding. There’s no other comparable dataset that contains a reasonably up-to-date employment history and professional network for a large portion of the working world. That information has value to any third party hungry for personal information, from Amazon to Facebook to Google to IBM, but it’s potentially more leverageable for Microsoft, which has spent the better part of the last three decades building out an ever more comprehensive and ubiquitous suite of tools for office workers via the creatively named “Microsoft Office.”

As an exercise, what would you do if you first were in Nadella’s shoes and second believed the Software Paradox to be a real phenomenon. One answer would be to begin acquiring unique, exclusive datasets.

First, because they can serve as the foundation for a variety of AI and machine learning products and services. Second, because data – unlike software – is ground that cannot be made up easily. LinkedIn’s software could be replicated, and perhaps improved upon, within a reasonable timeframe by at least half a dozen different large scale internet vendors. The corpus of data it accumulated from its users could not be replicated in twice as much time and at a large multiple of the cost. Hence, presumably, Microsoft’s valuation. Lastly, Microsoft presumably has attached value to denying the LinkedIn dataset to one or more competitors. There’s no guarantee that Google, as one example, would have been able to leverage the LinkedIn dataset into more robust and information rich business services – but now Microsoft doesn’t have to wonder.

The opportunity costs to an acquisition of this size and type are considerable, of course. Acquisitions also being inherently problematic, there is also a reasonable risk of failure. But those focusing on existing LinkedIn or Microsoft services and attempting to evaluate this transaction through that lens are missing the wider market context.

If this were a decade ago, or half that even, the Redmond giant’s core answer to strategic questions – any of them – would be software. Nadella appears to have internalized the reality, however, which is that software is not the commercial engine it once was. Microsoft’s past and present with Office and Windows was and is selling standalone assets, typical of traditional shrinkwrapped software. Increasingly, it’s harder to sell software for the returns that once were common – even for Microsoft.

But if standalone software isn’t where commercial value is to be found, what’s the alternative?

Consider the current state of the market. Broadly speaking, there is widespread agreement that the focus for both buyer and seller moving forward is in two major areas: cloud and mobile. Enterprises are moving to the cloud and cloud based services at incredible rates, and the story of how mobile annihilated the traditional PC market is well understood at this point. The good news for Microsoft is that it’s performing reasonably well in the cloud market at present. The bad news is that it was too little and too late in mobile, so the company has largely punted on the market for the time being.

The better news is that Microsoft has an opportunity to transcend both markets by combining hardware and software. For the enterprise buyer, could Microsoft use LinkedIn’s dataset to add predictive analytics around retention to its CRM offering, for example, or make this available as a third party service? This seems plausible. Could the company use LinkedIn data about your employer and teammates to improve mobile sharing, say, for individual users? Certainly. Is Microsoft one of the companies with the software and hardware resources required to build and perhaps even sell a bot that would answer, on demand, questions about a given employee’s work history and professional skills?

“Cortana: Where has Jane Smith worked?”
Jane Smith has spent ten years as a Java programmer for companies in the insurance and healthcare industries.”
“Cortana: What are Jane’s professional certifications or memberships?”
Jane is an Oracle Certified Master, Java Enterprise Architect, a certification that less than 5% of the applications for this position claim.”
“Cortana: What languages does Jane Smith speak?”
Jane is fluent in English, French and Spanish.”
“Thank you, Cortana. Can you schedule an interview with Jane next week?”
Of course.”

As long as they keep Tay away from LinkedIn related offerings, this does not seem like too much of a stretch.

Which makes the most important questions around this acquisition ones of execution, it would seem, more than valuation. Given the price tag and visibility of this acquisition, LinkedIn will be an interesting test of the new, Nadella-led Microsoft. But whatever the outcome, the process that led to the acquisition is nowhere near as crazy as the Clippy asking you to join his professional network-GIFs would suggest.

Disclosure: Amazon, IBM and Microsoft are RedMonk customers. Facebook, Google and LinkedIn are not currently RedMonk customers.

Categories: AI, M&A Announcements.

Hark Episode 2, “The Software Paradox”: Guest, Kent Beck

As newsletter subscribers are aware, Episode 2 of my new podcast Hark, “The Software Paradox,” dropped last week. In this month’s episode, Kent Beck – yes, that Kent Beck – dropped by to discuss The Software Paradox. What does someone responsible for some very popular software think about the thesis that software’s up front commercial value is headed in the opposite direction from its strategic importance? What were his experiences trying to monetize data telemetry with JUnit Max? We covered that and more – I can’t speak for Kent, but from my end it was a really conversation.

For those of you who don’t do podcasts, however, we offer a transcription of the episode below. Enjoy.

Steve: A little while back I wrote a book called The Software Paradox. Because the book claims, among other things, that software markets that were performing well at the time would be performing less and less well overtime, some people were unsurprisingly less than thrilled, which is fine of course, you’re not going to win too many popularity contests as an analyst. It’s certainly not why you get into this business. What I was always curious about however was what authors of popular software thought about the idea, the core idea that software would be less valuable over time, at least from the commercial standpoint. I talked to a great many developers over the course of writing the book and certainly over the course of our day to day jobs, but still, there was always that curiosity, there was always that question as to what they thought around the idea. Could they challenge the perspective or even refute the base thesis of The Software Paradox?

Enter Kent Beck. Many of you may know Kent for JUnit, maybe you know him for Extreme Programming, or maybe as a signer of the Agile Manifesto most recently from his time at Facebook. However you know him, however, it is very likely that you do know him because Kent has had a very successful career as a software developer. He was kind enough to join the show for discussion around his thoughts and feelings on The Software Paradox both as a software developer and the author of some very, very popular software projects. Welcome to part episode two, The Software Paradox. So excellent. Welcome to the show, Kent. I wanted to start with a quick background. So as we’d like to do when we open the show, can you tell me who you are and what you do?

Kent: My name is Kent Beck. I am currently employed as a technical coach at Facebook. I run a variety of education programs for younger engineers and at this point, they’re almost all younger engineers. Before that, I was a independent consultant, probably best known for Extreme Programming, test-driven development, the use of patterns in software development, the JUnit, the family of testing frameworks that was something that I developed with Erich Gamma, and am I leaving anything out, Steve?

Steve: That’s the background as I know it, I’m sure. I’m sure as it is true with all of us, I’m sure there are things that are going to fall by the wayside but no, I think that sums it up pretty well.

Kent: Okay.

Steve: So to get the conversational ball rolling so to speak, this little podcast started out of sort of an interchange where you were kind enough to point people at a book I wrote called The Software Paradox, and just for the background of people who are unfamiliar with that concept, this is how I would sum it up and we’ll see if you agree or disagree, Kent. So Software Paradox from my perspective is essentially the idea that the value of traditional on premise, it was sort of paid upfront commercial software is in decline, and is in decline broadly, across a wide sector of categories, consumer to enterprise. But this is occurring even as a strategic importance of software is actually going up by the day. Is that your understanding? Does that definition work for you?

Kent: So I probably would use slightly different words. The value is increasing, the value of the software is increasing, but the revenue pool available for people who create that software is shrinking.

Steve: Yeah, I’d say that’s fair. So what is it about that idea that resonated with you? In other words, you had a response and you were, as I said, good enough to point people over to it. So what was it about that idea that sort of piqued your interest?

Kent: Well it’s just so backwards. In any other case, it defies experience. So if something becomes more valuable, its price goes up. It sends a signal out. Yeah, I’m not an economist, but I’m a, what would you call me?

Steve: You know enough to be dangerous.

Kent: I do know enough to be dangerous. Thank you. Yes, yes, that’s good enough. So prices rise to send out signals to make more of this stuff that’s valuable. Except in software world, sometimes more of it makes it less valuable like when you got too many integrated development environments out there. You like to have the cliché more wood bind, fewer arrows [SP], except then every once in a while everything turns topsy turvy and you get these old incumbents that have slowed down and then you want Kuhn’s Extraordinary Science, you want fulfillment [SP] and lots of little things growing up. But in general, the way the market works is if something’s valuable, it raises prices so that there’s more of it. And here’s the case where that’s just the opposite. So if you can’t point to examples, I don’t know if you have other examples in world history where this kind of inversion took place, I guess that’s my first question.

Steve: That’s a great question actually. I don’t know, I’m trying to think if I came across any sort of in the background because it’s one of those things people always ask us how do things like The Software Paradox or before that, things like The New Kingmakers come up. And really they’re born out of conversations that we have, lots of them, right? So we obviously as sort of just by function of being an analyst, you have a lot of conversations with a lot of different people and you begin to see patterns, and then you begin to see sort of trends over time, and The Software Paradox was essentially that’s exactly what happened. I kept talking to businesses that were struggling to monetize and this is true again across a variety of sectors in my experience. And yet at the same time, you have essays like Mark Andreessen’s Software’s Eating the World, which is largely true I think at least from my perspective, that suggest that a lot of companies even in traditional industries are going to be increasingly defined by software, which means the software’s playing sort of this more important role. But it’s a great question, I don’t know that there’s another historical example that I can point to.

Kent: Okay, I assume this is a thinking on our feet kind of conversation?

Steve: Oh, very much so.

Kent: Okay. So what came to my mind was the Model T, which as it became more valuable because there were more roads and more gas stations, having a car became more valuable but the price dropped. But at the same time, I think the revenue pool didn’t shrink. I think the revenue pool expanded dramatically.

Steve: Yeah, I think there’s a lot of examples of products that would basically make a transition from what I would call sort of a margin opportunity to a volume opportunity, right? That’s something that we see sort of over and over and over again. I think there’s a couple things that are different I think in this case and that you pointed out one of them, right, which is that I think that the total revenue pool is in many cases actually shrinking. There’s fundamentally less dollars to go around, which is again a change because in the case of the Model T or any other number of examples we could point to from history, the actual size of those industries grew dramatically as they transition from margin to volume. The other interesting twist that software adds at least in my view, is that it takes the floor out, right? So in other words, when you produce something like a Model T, there’s a certain cost of goods, certain cost of manufacturing that goes into that. So in other words, whatever that cost of the materials is to put it together and whatever the cost of labor is, there’s a reasonable floor, right, that goes into putting that together, which is not to say that software is sort of inherently without some of those costs, but obviously the cost of distribution is effectively zero. The cost of replication is zero.

So in many cases, it takes out sort of the bottom of that market. So for the cost of a physical good, it may go down to a near [SP] cost. In many cases, what we see in software is that it doesn’t go down to near cost, it goes down to zero. And that’s a big change, and that’s a big issue for many that the vendors from a software standpoint to grapple with because it’s one thing if you sell near cost, if you have to sell for zero dollars, what is your business? What is your market?

Kent: Right. And there’s this argument about well, the price drops to the cost of replication, which is essentially zero. But there’s also the amortized cost of development that needs to go in there. And so if I’m a…here I am a 55 year old man, if I think, “Oh, let me start this new open-source project,” how is that ever going to contribute to my retirement? Because if I took say 6 months out of my life, which I did about 10 years ago now. I built a JUnit add on called the JUnit Max, and I couldn’t figure out how to turn it into a business. Now partly that’s because I’m a lousy businessperson. I’ve read too many books and I have too little talent or something like that.

Steve: Yeah, I know the feeling.

Kent: But it’s also because like, how am I going to get paid for that six months? And eventually I just decided there’s not a way to get paid for that time, so that leads me to not build tools that might be really useful. That’s another part of the paradox is that the signals the market sends by dropping the revenue available for software, it sends this signal that this stuff shouldn’t be created. And then we don’t create it and then the value available to everybody drops, and yet somehow it doesn’t go, it doesn’t go to zero. Facebook, for example, spends a lot of investment dollars on open-source software for which we receive zero revenue, but we got React and React Native and all the changes we made to MM [SP] Cash and there’s just a huge laundry list of things which makes sense for Facebook to invest in, but where does the…I don’t see where the engine is that starts the next thing.

Steve: Yeah, and I think that’s sort of an interesting segue to one of the questions that I wanted to sort of get at on this which is, do the underlying economics, and I’m a big believer that economics are one of the most powerful if not the most powerful change agents, do the economics here not necessarily stop the flow of open-source software or other software in general, but does it change the nature of the creation? And to get to that point is kind of…your example I think is perfect, right? Because on an individual level, it’s very difficult for a lot of developers, I think, to justify the effort, the resources that go into producing a given piece of software if there’s not a clear financial return for them. Now in many cases, we can all think of exceptions I’m sure, where it’s, “I’m going to create this project because I want to get hired by this other company. I’m going to create this project because it solves the problem that I have,” or what have you, but one of the key used cases of course historically has been, “I’m going to create a piece of software because I want to make a living and I want to make money off that.” And The Software Paradox would suggest that on an individual level, that’s sort of difficult to maintain. And yet, as you know, we have large entities, Facebook certainly is one of them, Google is another, Apple surprisingly is now one with Swift, and so on.

We have all these large entities that are now producing software and they are essentially releasing it for free, which again, from an economic perspective, suggest that they’ve looked at it and essentially determined that they have a higher return from releasing it than they would from selling it, which makes sense because none of those companies are in the business of selling infrastructure software at least. So I guess from my perspective, I guess the question I’m curious about from your end, having been an individual software developer and working for Facebook now, do you expect or anticipate a shift in terms of where the software’s coming from and who it’s produced by?

Kent: My next question is for whom does it make sense? It doesn’t have to make economic sense, but it has to not be fatal. I had kids in college and a mortgage to pay and I had to get a payoff some place. So for whom does it make sense to get software project started? Well, I think young people with low net and kind of nothing to lose and nobody relying on them. For them it makes sense to start that snowball rolling downhill. Most of the snowballs are going to go two feet and then stop, but every once in a while, one of them is going to start an avalanche.

My metaphors are getting a little mixed up here, but so I think you’re going to…what I would predict from my understanding of the model is that the innovation you’re going to see, the beginnings of innovation is going to come from people with nothing to lose and lots to gain. The refinement of those innovations into something enterprisey, something that a company like Facebook can deploy on a 100,000 servers, that refinement is going to come from those bigger companies because there’s no way that Jane grad student is…knows how to prioritize some piece of infrastructure at Facebook’s scale. So Facebook’s happy to pay for it, but it isn’t going to be money that triggers the beginnings of all these new piece of infrastructure. Who I feel sorry for in this whole thing is a company that wants to sell infrastructure. You’re just getting squeezed hard.

Steve: Yeah, it’s hard because it’s…we work with a lot of software companies and we work with software companies that are huge, we work with software companies that are just a couple of people. And again, that was a large part of the impetus for creating the book in the first place was that in many cases, they are getting squeezed. They’re getting squeezed at the bottom by open-source software. So for many of the commercial products, there are free as in beer alternatives that are…will do the job credibly, and exist, at the very least, sort of a downward price pressure. And in many cases, you’re getting squeezed at the top end by businesses that have things that still matter in today’s economy, things like account control or they have the sort of you’re the CIO and so on. And it’s a difficult position to be in, there’s no doubt. And a lot of them frankly… so we have as an example, I think I mentioned this in the book, I can’t remember. One of the businesses that we’ve spoken with basically looked at this and said, “We don’t necessarily agree across the board with the idea of The Software Paradox.” And I discussed some exceptions with them, but essentially they looked at it and said, “For our business specifically, we see our long term revenue from an upfront licensing standpoint going to zero then we plan for that.” And that’s a conversation that you don’t have four or five years ago, right?

Kent: Right.

Steve: That’s a conversation that is something that’s new and it’s a conversation that frankly, again, is surprising because you go to industry conferences today that have nothing to do with technology and they’re all talking about technology because technology in so many industries is now the differentiator. So the fact that the strategic values headed straight up in the sort of realizable commercial return is in many cases cratering, it’s a hard thing for a lot of companies to deal with.

Kent: So I’m a musician too and a writer so I’m interested in the evolution of those markets. The situation is…the analogy is not perfect because if you’ve got lots of bands putting out music for free, it’s not like that the value of that music…I’ve only got so many hours and I’ve only got two years. The value of that music isn’t skyrocketing. The economic leverage, you can’t take that music and turn it into a ten billion dollar company instead of a one billion dollar company the way that you can with software. So I’ve been scowering those markets for quite a while and it didn’t occur to me that just now that the analogy’s not perfect. So even if I figured out how to make money as a musician or I figured out how to make money as a writer, it wouldn’t necessarily inform the software situation.

Steve: Well, yeah. I think that there are definitely parallels though, right, because I think one of the things that you see in a lot of markets is that the experience as a whole is subsidized by some portion of it. So what do I mean by that? In other words, if you can sell at a software business, if you can sell one product for a tremendously outsized return, right? So for example, prior to the introduction of open-source databases or open-source application servers, where businesses felt they had no other choice, businesses would spend lots and lots and lots of money on sort of enterprise class application servers or databases or what have you. And those returns and those outsized margins can fuel investments and other products, right? They can fuel essentially experimentation.

They basically can subsidize a lot of other areas of the business. And clearly, I think, from a writing standpoint, you see that at least in journalism, right, where that whole business was for years and years and years subsidized by classified ads, sort of if nothing else, and in many cases, a sale of print editions and so on, and as both of those have fallen off, journalism and writers in general are left searching for, “All right, what’s the economic model here?” And again, I would say the same thing is I think is true to some degree of music where if you go back, certainly when I was growing up, you didn’t have downloadable singles for 99 cents or 79 cents or 89 cents or whatever they might be, you have to go out and buy a record for 16 or 17 bucks. And that’s again, subsidizes a lot of other investments and a lot of other economic opportunities for artists theoretically. A lot of that money obviously went into the hands of the recording business owners, but in other words, everything disintegrates. I said this before, everything disintegrates in subsidy for something else. And the challenge in a lot of today’s markets is that the original subsidy is gone and we need to find a new one. And I think in the case of software, I have ideas certainly in terms of what those are. Data is one of them, services is another, but I think that a lot of those original subsidies, “Hey, I’m going to charge some outlandish fee over and over and over again, and have 90%, 100%, 110% margins.” Those days, they’re not gone, they’re certainly businesses that are still realizing those kinds of returns, but they’re increasingly few and far between. Yeah.

Kent: Yeah, it’s such a puzzle. I think that’s the thing. When I get into a puzzling situation, first I look for analogies and I don’t have an analogy. And then I look for principles, like economic principles in this case. And I can’t even find economic principles where this makes sense.

Steve: Well, and it’s a difficult thing to me. You see this all the time with software, right? Software is just a fundamentally different animal and it’s really difficult for people to grasp that and not to go down a whole rat hole, but the most obvious example of this recently is the FBI versus Apple case, right, where you saw a lot of people sort of making the argument that hey, this is no different than essentially the FBI having the ability to go and search somebody’s house. And you basically have to step back and say, “If you think that, you don’t understand software, because software is inherently scalable and inherently sort of theftable or stealable in a way that a house key is not.” In other words, it’s not practical. I can’t go out and search million houses in 10 minutes. I can do that theoretically with a phone and with some of these vulnerabilities because software is just a fundamentally different animal. And it’s true economically as well, right? We see this sort of over and over and over again where people are trying to apply the economic rules for physical goods to software and to digital goods and it just doesn’t work that way.

Kent: And it did for a long time. I think that’s the confusing thing. I was involved in a startup called Agitar and we had a good product, a product that 10 years earlier would have made a, I don’t know, would have turned into a billion dollar software company. And because it was based on license revenue, poof, just gone, it’s just not a viable model. So that model worked for quite a while and now it’s stopped working. So here’s why I wanted to talk with you is we could probably list four, five classes of people, new graduate with technical abilities, investors, MBA who wants to go into technology, aging programmers, one that strikes near and dear to my heart, kind of mid-career programmers who are being squeezed by this.

What can we say? That was the piece of a book that was missing for me, what can you say to them, to any of those or all of those classes of people? And I don’t have any answer because I don’t have a model, I can’t. Usually I have got some kind of model and then I can just if I project forward in time far enough, then I sound outrageous and visionary and then, but it’s just a trick because I project 20 years, another people are projecting 10 years, then they go, “Oh yeah, you’re saying crazy stuff.” But here I just don’t even have a model. So what do I say to my daughter who’s just getting started in a software career? What do I say to myself?

Steve: Yeah. Yeah, I think that there are a couple different things. And I think a lot of the answer to that question to me depends on time frame, all right? So for the, say short to medium term, I’ll say out to four or five years let’s say because I don’t want to, well, one of the things that we find at RedMonk is that we tend to predict things and then they end up happening five years later than we think. I think in this case, I think for the foreseeable future, you’ll definitely have commercial opportunities and a fair number of them within the sort of commercial infrastructure software space. And were those opportunities be the size or the nature of what they would have been 10 years ago, no. You won’t have as many of them necessarily. But the fact of the matter is that Microsoft for example, is still making tens of billions of dollars annually on each of their twin revenue engines in Office and Windows, and that’s not going to change. And there’re a lot of businesses that are in that same boat, where they may be in decline, they may not be seeing the growth that they would have a couple of years ago, but they’re still making money and businesses are still paying for software. And for conversing…

Kent: Sorry to interrupt. As long as you can cut cost faster than revenue drops, then you got a viable business. So there’s a market out there, there’s a skill, a market out there both business and technical for a software hospice. You know it’s going to die, it’s a matter of time. You can stretch it out, give it some quality of life for a few more years. So yeah, I think there’s a book to be written there and I’m not going to write it because it’s the wrong side of…

Steve: Well I was going to say, I don’t think I’ll write a book called The Software Hospice. I don’t think that’s going to win me many fans.

Kent: [inaudible 00:26:54] software.

Steve: Yeah, people are already upset enough about The Software Paradox. Software Hospice might push them off the ledge.

Kent: It’s a natural follow up.

Steve: Yeah, yeah, exactly. As a follow-on to some of these businesses that are in decline, I think that there’re absolutely opportunities and good opportunities to build smaller businesses around open-source products. And the way that you do that typically is by saying, “Hey look, hey there’s a software product. We know it better than you do Mr. and Mrs. business. Do you really want to manage this sort of on an ongoing basis? Do you want to patch it, do you want to keep it up to date?” All those kinds of things. The answer is probably not. So look, this is what we do. We can sell that to you.

To sweeten the pot, a lot of businesses are turning to sort of “open core”, right, which is you have the sort of core of a product, all of the core features are available for free, they’re an open-source and then you have some proprietary layer that sits on top of that, that you have to pay for, whether that’s management or some other feature that isn’t necessarily integral to the experience but is a valuable add on and something that a business might pay for. So you can definitely build those kinds of businesses. We see that over and over today.

Kent: So here’s my problem with the open core model, is a question of internal moral hazard. Your part of the business is motivated to add features to the core because they want more adoption in general. And part of the business is motivated to exclude features from the core in order to make the commercial opportunity more attractive. I’m not saying it’s not navigable, but I think that there’s kind of this…

Steve: There’s a constant…

Kent: …seeds of a destruction.

Steve: There is a constant tension, there’s no doubt. Absolutely no doubt. Now, the way around that from my end, as you say, you can navigate that in the short term, right? In other words, it’s never a comfortable discussion. We’ve worked with many, many clients on just that subject in terms of, “Look, you can’t close that feature. You’re going to get killed.” So yes, it’s absolutely attention. It can be navigated in the short term. To me, the best solution to that particular problem over time is to go the services route, because all of the sudden from a services standpoint, one of the things that tends to happen is that all of your incentives begin to align with customers in ways that they don’t.

So as an example, if you have an open-source business and you’re selling support and service, well, theoretically if you do a great job of manufacturing a product, you’ve just put yourself out of business because why would I pay for support for products that works well basically all the time? Now of course, we all know that’s not how things work and there’s always bugs in software and so on, but there is a fundamental misalignment if you will, of needs, that the vendor needs customers to have problems, but the customer wants a product that has as few problems as possible, right? So that’s a fundamental misalignment. So what do you do? One of the things that you end up doing is that you offer these software assets as services. And all of a sudden, a lot of those question go away.

So for example, the tension between releasing features or keeping in private, gone. In other words, you want people to sign up, you want to retain customers, so what do you do? You continue to innovate and iterate on that product. So all of a sudden, that dries up and blows away. The concern for example of okay, what is the purchasing trigger, right? How do I get somebody to actually buy this software? Again, that goes away, right, because all of the sudden there are very few people who are looking for infrastructure software in a hosted setting for free, the way that they are for on premise, sort of open-source software.

Kent: I wouldn’t trust it if it was.

Steve: Exactly. In other words, even tiny services that we use, everything is paid, because otherwise where is that? You want the service to be around. It’s easier to justify paying for something where you know that they’re cost baked in versus software. A lot of people will sort of make the excuse to themself which is, “Well, look, they’ve already paid to develop the software and if I use it, there’s no additional cost to them to develop it again because the cost of replication is free and so on.” You don’t have that with services. There’s none of that sort of internal moral discussion.

So a lot of those issues go away if you go the services route. And the difficulty that we have in terms of talking to a lot of open-source businesses is that they’re aware of this, right? They’re aware of a lot of the issues that they have with The Software Paradox, they’re aware of some of the opportunities that exist in services business, but that is a much different business to start and run than is sort of the support and your traditional open-source important [SP] service business, right? All of the sudden, you need to hire a different caliber of person, you need to hire a different type of person because you’re not just developing a piece of software and sort of releasing it and testing it and so on, you need to keep infrastructure up and running 24 hours a day, 365 days a year. So how do you do that? Where do you that? Your capital costs are entirely different. Your cost of customer acquisition is all upfront, whereas the return from that is amortized over time. So it’s a difficult…as simple as it seems for a lot of analysts, people like myself on paper, that’s a hard question for a lot of businesses in terms of, “How do I get into that?”

Kent: Sure, it’s easy to put the spreadsheet together though.

Steve: Yeah, oh yeah. Yeah, we do it all the time.

Kent: For 20 years I’ve been on the board of a company in Switzerland called Lifeware that does life insurance contract management. So it was one of the earliest software as a service businesses. They basically run the whole backend of life insurance for medium-sized insurers. And because they’re really good software developers, their costs are a fraction of what a big life insurer pays for management a contract for a year. They only get paid by the contract, so their incentives are really closely aligned with their customers’ incentives. And it’s a tough business to run. As you say, you got all these upfront costs and then in the insurance business, you’re talking about a revenue stream that’s going to last 30, 40, 50 years. So you have to have access to capital, thank goodness, Switzerland, but you also have to have a lot of patience in a way that you don’t talk to a lot of recent MBA wannabe start up a software business and you’re talking about a 50 year timeline.

Steve: Yeah. And again, it’s just that things have changed, right? Because in other words, you go back 10 years ago, you can start with a relatively small piece of software. You can grow that into a sizable business quickly and you can either become sort of a huge entity in and of yourself, or you can sell out, you can exit for some large sum of money and the mechanics of those businesses have changed, right? And that’s something that a lot of the businesses that we talked to are struggling with.

Kent: Yeah. And with JUnit, I managed to completely miss that whole exit thing.

Steve: Yeah. So actually that was interesting. Talk to me about The Software Paradox lens as applied to JUnit. So what was your experience like, what would you have done differently if you had thought about it?

Kent: We couldn’t. We very explicitly had that conversation, Erich Gamma and I. “If we charge for this, no one’s going to use it. If we don’t charge for it, there’s no revenue. Duh, what are we going to do?” And we both had day jobs so we were the kids at the top of the mountain kicking a snowball down, because we were willing to just throw away that investment because we didn’t care if it paid off or not. Initially we had three hours on an airplane before the batteries ran out, flying from Zurich to Atlanta and so yeah, why not? And then this particular snowball rolled and rolled and picked up size and speed and turned out there was a lot of loose snow and so it ended up being very successful.

But very early on, we realized that if we tried to monetize it in any kind of way, that stops the snowball dead in its tracks, and it’s the end of the story. And we wanted people, we wanted programmers to have the benefits of automated testing, so we gave it to them. And I always get that argument, “Well, it’s going to pay off in other ways.” The last time I was doing consulting, my daily rate was half of what it had been 10 years previously, so I don’t see the payoff. And that’s part of the emotional trigger of your book for me is I faced this paradox in a very explicit way, and I didn’t get a result that, I don’t know, that made sense to me. I have a farm in southern Oregon and I have my goats and I live a really nice life. I still have to work for a living. So I did okay, but I’m not financially independent out of it. In the global world, I’m a ridiculous number of zeros, .001%, but at the same time, if a success at that scale had happened 10 years earlier, it would have been rational software, that was kind of the previous generation’s version of that which turned into a whole bunch of money for a bunch of people.

Steve: Yeah. Yeah, it’s interesting because the question of return is always an interesting one, right? So in other words, we just hired a new analyst. So I did, oh I don’t know, 25, 26 phone interviews and one of the questions that we got high percentage of time, we don’t get as much these days, but certainly we get it from time to time is, why do you release your content, your research for free? James and I had almost exactly the same conversation that you and Eric did, which was, I think it was year one actually which was, “All right, we have this research. If we charge for it, it’s going to be difficult,” because basically one of the biggest drivers for research isn’t the research itself, but somebody wants a name on it to say that, “Somebody told me to buy this so if it blows up, I’m not going to get fired.”

And if you’re a small firm, your name doesn’t count for anything in that analysis. But conversely, if we try to charge for it, nobody’s ever going to read it. And we made the decision at the time to say, “All right, you know what, this is the best economic decision for us is to release this as essentially open-source, if there was a term for that, certainly Creative Commons is the closest.” So release it under Create Commons license. And the return for us has been great because basically what ends up happening is it will put out a piece of research and businesses will say, “Hey, this is great, but how does this apply to my business?” Okay, awesome. That’s a consulting project, yeah.

So the return for us, it’s not always direct of course. It’s not like every piece of research produces something like that, but the attachment is close enough that it makes sense, right? It’s a justifiable economic decision for us. I think in a lot of cases for open-source, there are real questions as to whether or not it will be, because one of the things we haven’t talked about which is certainly a factor these days, is that there is so much open-source software that’s standing out and been sort of achieving that, “Hey, we’re going to become the snowball that starts an avalanche,” is increasingly difficult because standing out from the rest of the projects is more problematic than it was 5 or 10 years ago when there was just less software to compete with.

Kent: So I’m going to go ahead and disagree with you on that one, Steve. I think that it’s always been difficult to get attention. The success of software projects is always going to be distributed along some kind of power law distribution, which means that the vast majority of projects are going to get zero attention. And if you’ve never had a hit, if you’ve had a thousand zero attentions in a row, then that feels pretty unfair to you, but that’s actually a pretty small sampling. I’ve probably started 2, 3, 400 programs like JUnit, put in that amount of initial effort just because I was curious and I wanted to see. And that one’s by far the one that paid off the most. And I think given how much energy I’ve given to programming, I’m above, I’m exceeding variants. I got more than my share if I measure by people using and finding valuable the work that I’ve done.

Steve: Yeah, yeah. And to be clear, I don’t mean to imply that it was 5 or 10 years ago, hey, it was just simple, you throw something out there and it’s successful. I guess what I’m thinking of more is that take the database space in particular, right? If you go back 5 or 10 years ago, right, really what are we talking about? We’re talking about a small handful of projects, right? You’re talking about probably three from the commercial standpoint in terms of the most successful, Oracle, DB2 and SQL Server, and then you’re talking about two effectively from an open-source standpoint, MySQL and Postgres.

So that’s a relatively small sample size of projects. If you fast-forward to today, right, we have, depending on how you define the different categories, you probably have four or five different categories, right, in terms of relation [SP] obviously is still around, it’s still a big deal. You have graph, you have key value, you have sort of larger-scale data operations sort of in the [inaudible 00:42:37] type category and so on. And there’s probably two or three different more that we could list that are popular enough and that they’re sort of quasi-mainstream. And in every single one of those categories you now have, oh I don’t know, anywhere from say three to six legitimate contenders, legitimate sort of projects vying for attention. So it’s a crowded marketplace, right? There’s a lot going on and that’s a factor. When you start thinking about, “Okay, if I want to make my mark, where am I going to do that? Where am I going to sort of invest my time that hasn’t been done sort of over and over and over again, sort of done to death and I don’t have tons and tons of competition? The number of those areas is getting smaller.

Kent: Sure. If you were putting money in, expecting to get money out, you’d have to be nuts to start a competitor, to start, “Okay, I’m going to have another relational database and it’ll be the next MySQL,” because that’s just such a stupid bet. So I would say if you’re calculus is money in and money out, you’re going to have to wait, kind of seed funding the cost of planting the seed is so far below the transaction costs for a seed round that it just doesn’t make any sense to put money in to try and get money out.

Steve: Yeah. Well, I was just going to say, I think honestly for me, the biggest opportunity, the best return, I think, for a lot of businesses today honestly is going to come from data. And the difficulties is that it’s a very fraught conversation to have with vendors, right, because there’s all sorts of sensitivities in terms of, “Hey, if I tell customers that I’m collecting their data, they’re going to go nuts and not use my product and so on.” But here’s the thing, is that if you look at, if you talked to any of these customers who supposedly won’t use software that spies on them or watches any sort of what they do, the next logical question to ask them is, do you use any software as a service offerings? And the answer of course, in every case is yes, they use it somewhere for something. And in that case, then everything they do is being watched, right? Everything they do is being monitored, everything they do is…

Kent: Better be.

Steve: Exactly. You’re not doing your job if you’re a software as a service vendor. You’re not paying attention to things like usage patterns, right, or what are customers struggling with, what queries are taking longer and so on. So that to me is the…when I talk to businesses today and a lot of them are actually beginning to sort of take steps in this direction, the way out, and this goes back to sort of what you might tell your daughter or what I tell the startups that we speak with, is begin to think about not necessarily short term. You don’t have to pivot your business overnight, but begin to make preparations to create data as an asset, because if we look at your employer, Facebook, or if we look at businesses like a Google or a Twitter and so on, a lot of the value that they have at this point isn’t in the software, right, it’s in the data, it’s in the data that they generate. I use this example all the time. If you give me Google software from two years from now and you give me all the people and all the resources necessary to run that software, all the data centers and so on, just magically you grant that to me, it doesn’t matter because I don’t have the corpus of data to give you the returns that they can. In other words, the data that’s been built up overtime.

And obviously, it’s not a linear comparison, it’s not a one-to-one comparison comparing Google’s business to say your traditional infrastructure provider. But there are comparisons to be made because look, infrastructure software generates tons and tons of telemetry, we all know this. Anybody who’s used Spelunker, any other of the other sort of logging tools is aware of how much data it generates. That data has value and that data can be put to work.

Kent: Right, especially if you’re going to aggregate. That was the idea behind JUnit Max, this product that I worked on that we would log all the test run results from everybody, every program or everywhere in the world, and then you can answer questions like, do my test fail more often than other people’s? Or which languages seem to encourage better testing or worst testing and what can we find out about that? It is a hard flywheel to get started because you need a lot of data before it to starts to be valuable enough that you attract more data and you get the positive feedback going. But I agree with you.

I think if you’re looking for money in, money out that…so, okay. I live on a farm and this whole John Deere anti-hacking stuff, that’s a topic of conversation where I live, and what idiots John Deere is. If John Deere was, okay, so I’m just going to say it the way I would say it here in Oregon. If John Deere was really smart, they would let people do anything they want with their tractors as long as John Deere gets to collect the telemetry data from those tractors…

Steve: Totally agree.

Kent: …[inaudible 00:48:24]. And they could turn that around and they could run the most profitable commodity arbitrage business in the world. They could, oh god, there’s just a million ways they could sell that data. Anyway, they don’t have that vision.

Steve: No, no, they don’t. And that’s one of the things I really struggle with around The Software Paradox. The conversations we have with our clients is trying to get customers to have that kind of vision to see some of those opportunities, right? Because the difficulty is that when you have these conversations, it’s kind of an analogy I’ve used in the past, it’s kind of like trying to sell a relational database, right, in the sense that a relational database is a fantastic piece of software, it’s tremendously versatile, it’s behind sort of basically every application you touched on some level. But if you’re just trying to sell it to somebody in a vacuum, it’s difficult, right? Because okay, well, what can you do with a relational database?

Kent: It’s like selling an engine.

Steve: Yeah.

Kent: Which only works if you’re selling to a Formula One team.

Steve: Exactly.

Kent: If you’re selling to me, an engine isn’t doing me any good.

Steve: Yeah, I mean great. Okay, what can I do with that? Lots of things. Well, okay, what kinds of things? And then you have to basically try to find a way to scale that conversation, so it’s difficult, right? It’s a difficult conversation to have. Now the thing that’s helping is that you begin to see little examples here and there of people doing interesting things with their data. So in other words, they cancelled it now, but I thought what New Relic was doing with the App Speed Index was fascinating, right, because New Relic has access to whatever it is, tens of thousands, or hundreds of thousands of nodes sort of all over the world. And they can begin to give you a baseline in terms…

Kent: Oh, is that all?

Steve: Whatever it is, yeah. It’s not quite the Facebook experience.

Kent: I am spoiled at Facebook [inaudible 00:50:18].

Steve: No, no, no, I know.

Kent: Had to throw my little snotty snark in.

Steve: Of course, of course. No, it’s really appreciated.

Kent: Hundreds of thousands, okay.

Steve: Yeah, who knows, maybe it’s millions. I don’t have the actual number for them. But I think the point is that whether you’re talking about a New Relic, whether you’re talking about a Facebook, once you get to even a modest level of traction, you can begin to make some sort of interesting assumptions, you can begin to make some interesting conclusions in terms of, “Hey, what’s going on? How do I compare to a baseline? How do I compare to people in my industry? How do I compare to businesses of a similar size?

Kent: How do I compare to six months scale?

Steve: Exactly. And those are the kinds of things that the advantage is that they really only become more valuable and more defensible from a market standpoint overtime, because we saw this in the case of Apple Maps, where Apple goes out and drops 200 and some odd billion dollars on six or seven different startups, come out with a really nice, aesthetically pleasing, well-designed mapping product and it’s an absolute disaster because they hadn’t spent the last 10 years or 15 collecting data. And you can’t make up that ground inorganically. So yeah, I don’t know that The Software Paradox is necessarily easily answered in every case by sort of data or telemetry based models, but I think particularly in infrastructure software, I think it’s going to be a common answer and I think it’s going to be a good one.

Kent: Yeah, I was just trying to think of what’s changed to make that true. As bandwidth gets cheaper and cheaper, then collecting the data becomes cheaper, there’s fewer barriers.

Steve: Yeah. And also you allow customers to acclimate, right? So for example, when I was a systems integrator in the ’90s, right, we ran around and talked to lots of different businesses. And one of the things that we would talk to them about was, “Hey, you guys are not good at implementing CRM software. Basically half of the implantations fail. Why don’t you let us run this stuff for you in a data center, there will be dedicated hardware, nobody else has access, etc.” All those businesses came back and said, “Yeah, you guys are nuts. The customer data’s the most valuable data we have. It’s never leaving our firewall, over my dead body.”

And we’re five years later, every single one of them is running in Salesforce, because your initial reaction, your initial apprehension and so on will give way overtime if value is demonstrated. And that, I think, is going to be the trick with a lot of these businesses is, all right, you need to give customers time to get used to the idea of, “Okay, look, I’m not giving away my customer data,” for example because none of the vendors want that, that’s more liability than it’s worth. Basically they, in many cases, if not all cases, just want, “I want the telemetry. I want the data about how you’re operating. The actual data itself, that’s actually toxic. I don’t want anything to do with that.”

So as customers gets used to that, and more importantly as you can begin to show them value which is, “Okay, look, if you share this data with us, this is an example of the kind of data that we’ll give you in return.” “Oh, okay.” I’ve used the example with Gmail. If you walk up to somebody in a vacuum and say, “Hey, do you want an email client that’s going to scan your email and sort of mine it to present you with better ads?” Everyone says, “Absolutely not.” If you put Gmail in front of them and then say, “Oh, by the way, this is the cost to that.” Everyone says, “Oh, okay. Yeah, that’s fine. I can do that.” So a lot of it is just how you present it.

Kent: Yeah. So can we wrap this up?

Steve: Yes.

Kent: Here I am, driving your podcast, with concrete suggestions for those classes of people that I mentioned. That’s why I wanted to talk with you, like, “Oh, what do I do, Stephen?”

Steve: Yeah, so let’s see. So the classes of people were…

Kent: Recent graduate.

Steve: Recent grad, MBA…

Kent: MBA, mid-career.

Steve: Mid career and what was the last one?

Kent: Geezers like me.

Steve: Geezer, okay.

Kent: And then investors.

Steve: Okay, and then investors. So I would say for the recent grads, I think open-source is a great way to sort of build your visibility. In other words, you need to think about return, not necessarily in financial terms and not necessarily expecting to be sort of the runaway success of a JUnit, but a lot of the businesses we speak with are looking for profiles and they’re looking for contributions, they’re looking for sort of demonstrated capabilities, and open-source is a fantastic way to do that. So your return as a recent grad in terms of releasing projects of your own or contributing to other projects is going to be, I think, reasonably high from a career standpoint. For folks mid-career, I think you need to think sort of more about okay, presumably at that point you’ve made your name, you have some reputation and so on, and obviously you probably have more responsibilities in terms of your life and sort of dependence and so on, in which case, your concern shifts, right? You need to think about, all right, maybe I don’t release this project as open-source software, or if I’m going to release it as open-source software, it’s only going to be sort of as an incentive to essentially move along to other forms of business, like a services arm [SP].

And likewise for an MBA, if I’m an MBA, I would look at the numbers sort of across the board. I would look at The Software Paradox and basically say, “I’m bullish on services, I’m bullish on data, and I’m…” A software business that wanted to recruit me out of an MBA program would need to really prove to me that they have an answer for this. In other words, again it’s not impossible, there are businesses that are exceptions to the rule, but I want to see you prove it and I want to be convinced. And then for the geezers, basically I think a lot of it comes down to what your goals are, right? In other words, if you’re a Kent Beck and your reputation’s assured, then I think that’s not a big deal. I think that there are lots of different things that you can contribute to, whether that’s open-source, whether that’s businesses of all shapes and sizes. Again, if I was going to get hired, I wouldn’t want to buy a software company, I [inaudible 00:56:40] understand and sure to what their answers were.

Kent: I wonder if this hospice model is kind of a natural landing ground?

Steve: It could be. Certainly could be because if you are comfortable sort of not being necessarily in a high growth market, a lot of those businesses are going to be ones that are familiar to them. A lot of those products are going to be ones they probably use or worked on or build competitors to. So yeah, I think there’s opportunities there.

Kent: Okay. Cool.

Steve: All right, well I have one last question, one quick last question for you.

Kent: Sure.

Steve: It’s a fun one that we’d like to close on, which is, what animal are you most frightened of?

Kent: Okay, we had a ram here who figured out, he was kind of a silvery gray color, he figured out how to open the latch to his house. So you’d go out and feed at night and he’d be literally lurking behind a tree and you couldn’t see him, and the first thing you knew, you were flying through the air. And Smiles, his name was Smiles because it looked like the joker face had been painted somehow on his face especially after he had just sent you. And Smiles sadly is no longer with us, but he really scared the crap out of me.

Steve: That’s fantastic. Well, with that I think we can bring it to a close. Thanks so much, Kent, for the conversation. It’s been a lot of fun.

Kent: Thank you. Oh, it’s been my pleasure. Thanks, Stephen. Bye-bye.

Steve: Thanks again for listening to Hark. As a reminder, you can find us on Google Play, iTunes, Pocket Casts and Stitcher. You can also listen directly or find links to all of the above or by heading over to, which will take you to SoundCloud. If you have questions, feedback, or suggestions, you can hit us up on Twitter, @harkpodcast, or via email at [email protected] We’ll be back next month with episode three and until then, enjoy your time.

Categories: Economics, Open Source, Software-as-a-Service.