Blogs

RedMonk

Skip to content

Revisiting the 2011 Predictions, Part 1

Predicting is an easier business than it once was. True, technology is hysterically accelerating rates of change and disruption, but that’s only relevant if the substance of your predictions matters. Which all too often, these days, it doesn’t. Analysts and pundits are able to prognosticate with relative impunity; who has the time to go back and check their accuracy? Pageview driven models, in fact, reward wilder predictions because the error cost is, generally, approaching zero. Unless you predicted, say, that Linux would be killed off by Windows NT, nobody will remember later.

I find value in reviewing my annual predictions, however. If they prove correct, that’s useful. If they were not, understanding the reasons why is important to adjusting our models moving forward.

Because I made the mistake of making better than a dozen predictions last year, this year’s review will be delivered in two parts. Part 1, below, will cover my predictions for browsers, the cloud, data, developers and programming language frameworks. Part 2, covering predictions within hardware, mobile, NoSQL, open source and programming languages, will hit tomorrow.

If you’d prefer to read last year’s first, they can be found here.

Browsers

Firefox Will Cede First Place to Chrome, But Not Without a Fight

Browser Usage

According to RedMonk Analytics, whose data reflects our developer-heavy audience, Firefox was able to hold off Chrome for two quarters. On the first of June, Firefox held a 32.58 share of our audience to Chrome’s 32.08. By the second, Firefox was in second place and would remain there for the balance of the year, widening the gap in the process. At present, our browser metrics peg Chrome at 36.38 with Firefox a distant second at 25.48.

I feel safe counting this one as a hit.

Cloud

PaaS Adoption Will Begin to Show Traction, With Little Impact on IaaS Traction

The first Platform-as-a-Service providers essentially asked developers to trade choice for development speed. Like Ruby on Rails – itself the basis for multiple first generation PaaS platforms – PaaS was built for those that would embrace constraints. But PaaS platforms never saw the type of growth that Rails experienced, in part because of the further loss of control that the cloud represents. It’s one thing to have a web framework like Rails dictate the way that you build web applications; having PaaS platforms also choose the operating system, database, version control systems and more was too much.

Which is why second and third generation PaaS providers have furiously removed barriers to entry, adding additional runtimes, open sourcing the underlying platform and allowing you to pick your provider. Which, in turn, is why adoption of PaaS is accelerating. VMware CEO Paul Maritz calls PaaS “the 21st-century equivalent of Linux,” which explains not only why they feel compelled to compete in the space, but also why Red Hat might.

Virtually every vendor in this space is reporting growth similar to the Hacker News trajectories for Cloud Foundry and Openshift (below).

Cloud Foundry / Open Shift

In spite of the growth of PaaS, however, none of the metrics we track reflect any decline in usage of general infrastructure platforms. Quite the contrary, in fact.

I count this as a hit.

Data

Firms Will Increasingly Seek to Leverage the Data They Generate

Turning data into revenue has been one of the core themes of the past year, as well as the focus of my talk at the Open Source Business Conference in May. We’ve long held SpiceWorks up as a model of monetizing data, and as customers adjust to the reality that they’re already sharing data and vendors cease to regard it as a third rail issue, we’re seeing more businesses embrace data based revenue streams, as with Sonatype Insight. From 10gen to Black Duck, vendors are increasingly positioning themselves to be purveyors of data as much as software. Data is no longer the byproduct, but a product itself.

I count this as a hit.

Hadoop Will Become the MySQL of Big Data

EMC, HP, IBM, NetApp and even Oracle all have Hadoop – or in EMC’s case, MapReduce – plays in market. Microsoft actually deprecated its own Dryad initiative in favor of the Apache project. Players from AsterData to CouchBase to EnterpriseDB to MarkLogic to Tableau to Vertica have purpose built Hadoop connectors. The commerical distribution space, once essentially owned solely by Cloudera, has expanded to multiple third parties with varying points of differentiation.

Hadoop interest elsewhere, meanwhile, has not slowed.

Hadoop

Need I say more about the growing ubiquity of Hadoop? I count this as a hit.

Developers

Talent Shortages Will Continue

Granted, predicting a shortage of qualified development talent will be seen in some quarters as controversial as predicting that the sun will rise in the east. But part of this is context: certainly in January, the economic direction was less than certain. And in spite of an unemployment rate that has hovered just south of 10% for the better part of the last calendar year, hiring continues to be an issue for the majority of our clients. To the extent that several are spinning up offices solely for purposes of recruitment. This is not surprising, given the historical growth in employee headcounts (see above) that has, to date, been relatively resistant to the global economic crises.

Demand varies by skillset, as might be predicted, but 2011 remained – by our metrics – a tight market. Other market watchers support this assertion.

“Hiring talent in Silicon Valley is the toughest since the last bubble and investors are starting to openly wonder how this one will end.”
Steve Blank

“There is a war for talent, particularly developer talent, going on. Not just in Silicon Valley but also in NYC and many other places around the country.

Companies, small and large, are resorting to all sorts of creative ideas to recruit. Free lunches, free yoga, pushing code day one, cool schwag, options, RSUs, pretty much whatever it takes.”
Fred Wilson

While we don’t have good data then on market specific hiring (Bureau of Labor data is not fine grained enough), the evidence available to us seems to support the contention that shortages of tech talent remain.

I count this as a hit.

Frameworks

Node.js Will Continue its Growth Trajectory

October was a rough month for Node.js, with posts like Node.js is Cancer and node.js Is VB6 – Does node.js Suck? following the tradition of March reddit discussions like Is NodeJS Wrong? The Trough of Disillusionment, it seemed, had arrived well ahead of schedule.

Except that interest metrics showed no commensurate decline. Node took – again – three of the Top 5 spots in inbound search queries within RedMonk Analytics. Which is unsurprising against the backdrop of Google’s Insights for Search numbers.

Over on GitHub, meanwhile, which itself has achieved dramatic growth, Node.js is the second most popular watched repository, ahead of Rails, jQuery, HTML5-Boilerplate, and Homebrew. Microsoft clearly perceives this growth, because it has worked with Joyent to create a stable build of Node for Windows which in turn led to an SDK for Azure.

All of which means nothing except that Node’s growth trajectory continues.

I count this as a hit.


Part 2, tomorrow.

by-nc-sa

Categories: browsers, Cloud, Data.

Bottom Up Adoption: The End of Procurement as We’ve Known It

From the beginning of time two forces have vied for influence over us. One is bottoms-up, decentralized, and emergent. The other is top-down, centralized, and directed.

The first force catalyzes change and divergence, while the second tends toward order and convergence. The first gives birth to new ideas, and the second enshrines them.”
- Adam Ludwin, From LOLcats To Occupy Wall Street, Everything Is Happening From The Bottom Up Now

Traditionally, industry analyst firms have been oriented around top down adoption patterns. CIOs and other IT decision makers comprise both the research subjects and purchasing audience for the majority of firms in this industry, large and small. Which was logical given traditional procurement patterns. When hardware, software and services are available only at high prices, command and control is an appropriate management structure. Attempting to scale the decision making process for big ticket items across a large body of middle managers is not likely to yield acceptable outcomes.

An approach that makes sense in one context, however, may be misapplied in another.

The technology purchasing landscape today looks very different than it did even five years ago. Where once CIOs might reasonably expect to have the clearest understanding of what technologies are leveraged within their own organizations, today they are, as Billy Marshall put it, “the last to know.” This pattern manifests itself every day within the majority of businesses. Not because CIOs are failing, but because of trends that have fundamentally and likely permanently disrupted their ability to centralize the technology adoption process.

The four trends we see as most important in driving this are arranged here in rough chronological order.

Open Source

In the late nineties, startups and enterprises alike were effectively beholden to commercial suppliers for the majority of their software needs. Because each piece of the requisite software infrastructure had to be licensed, the capital expenses associated with new initiatives was high. This represented a barrier to entry, and thus a brake on innovation.

With the popularization of open source software, developers from enterprises and startups alike were able to operate independently. For the first time, the actual software practitioners were free to choose their own software rather than having it selected for them and subsequently imposed upon them by upper levels of management. Even in situations where the ultimate production infrastructure targets remained commercially licensed software, open source software like Linux and MySQL allowed for prototyping and rapid development without the attendant costs, both financial and in procurement latency.

This was the first major shift affecting procurement, and perhaps the most profound. None of the infrastructure we take for granted today – Linux, Apache, MySQL, PHP, etc – were originally adopted from the top down. Their adoption was, instead, a fait accompli. CIOs – the last to know – gradually became aware that increasingly significant portions of their infrastructure, unbeknownst to them, were running on free and open source software. The inevitable demand for production support options for this software is what fueled, in time, the valuations of MySQL, Red Hat and others.

Bring Your Own Device

In October, Apple CEO Tim Cook asserted that 92% of the Fortune 500 were “testing or deploying iPad in the course of less than 18 months,” which may help explain why the iPad revenue stream by itself would place within the top third of that group. The interesting thing about this is that the majority of businesses appear uncertain about precisely why they’re deploying tablets: “Most participants, 51 percent, indicated that they did not have a clearly articulated strategy.”

The answer, in most cases, is that there isn’t one. iPad adoption, much like the penetration of iPhones and Android handsets is being driven by users who simply want the device. Faced with a choice between users – chief executive officers among them – who will employ their own devices for work purposes with or without the permission of IT, many businesses are compelled to support the platforms even without concrete business justifications for them.

The consumerization of the enterprise is decentralizing the process of technology selection, but its importance may lie rather in design. Like all products, technology is designed and built to be sold to a specific buyer. For enterprise products, historically, the actual user has been a secondary concern; the buyer – typically centralized IT – was the priority. Consumer technology companies like Apple, however, are designed for a user. What they give up in IT friendly features they more than make up for in usability and the ability to delight.

The Bring Your Own Device trend, therefore, may well improve user productivity by driving devices designed to be used rather than managed into organizations, from the bottom up.

Software as a Service

Software as a Service is a classic case study in timing with respect to market acceptance. Not many remember today that the model actually failed the first time around, when its practitioners were known as Application Service Providers. Pyschologically, few enterprises were prepared for either the idea of renting software or externalizing critical data like that stored in customer relationship management systems. By the midpoint in the last decade, however, these concepts were sufficiently commonplace to see Salesforce.com a publicly traded company with a valuation north of a billion dollars.

Consumer markets, meanwhile, had adapted much more quickly. Hotmail debuted in 1996, Yahoo Mail the year after and Gmail dropped in 2004.

Some of those same consumer services were pressed into service by enterprise workers, in fact; it was once common for Exchange users to forward all of their email to Gmail due to the disparity in storage limits between typical Exchange implementations and Google’s webmail product.

This pattern has played out repeatedly over the years, from webmail to CRM to project management software to website hosting to online helpdesks. All were adopted from the bottom up. By making applications available to anyone with a browser, often at low or no cost, SaaS has surged up through the ranks of enterprises. The inexorable nature of the model is reflected by the growth of providers large (Salesforce.com) and small (37signals).

Cloud

The single most important feature of the cloud has nothing, or at least very little, to do with technology. It is, rather, the pay as you go economic model. As Flip Kromer puts it, “EC2 means anyone with a $10 bill can rent a 10-machine cluster with 1TB of distributed storage for 8 hours.”

What this means in practical terms is that for the first time, hardware procurement is democratized. From an accessibility and availability standpoint, cloud is the hardware equivalent of open source software. Where open source allowed developers to bypass traditional procurement channels by making quality infrastructure and development software freely available, so does the cloud allow the growing class of devops technologists to leave the world of high latency hardware procurement – where same day server provisioning is a feature – behind. Armed with nothing more than a credit card, instances can be spun up and ready for use in ninety seconds.

Cloud is the final piece of the bottom up puzzle. Open source software and to a lesser extent SaaS allowed for the decentralization of enterprise technology development, but at some point hardware would become necessary which was the insertion point for IT. With public clouds, it is possible for the first time to entirely bypass the traditional gatekeepers.

The Net

It should be evident that traditional procurement and purchasing is not dead, just increasingly bypassed by a more efficient process. Also, that a great many enterprises continue to function largely as they always have: top down. More important than the question of whether this model is sustainable in the face of the trends above is whether it should be.

Before lamenting the fact that the above forces are disrupting and destabilizing your enterprise IT, consider that that may be a net gain. If the primary drivers of BYOD, Cloud, Open Source, and SaaS include ease of use, lower costs, frictionless availability, and speed of provisioning, are these trends worth opposing? Particularly since efforts to do so will, in all probability, fail?

Or are they instead assets to be strategically leveraged? There is little debate that businesses that move the most quickly have a competitive advantage. It’s not clear how businesses that prohibit the same tools that enable this will benefit.

Either way, bottom up adoption is here to stay: use it or lose.

by-nc-sa

Categories: Bottom Up Adoption, Cloud, Open Source, Software-as-a-Service.

What’s Holding Back the Age of Data

After six months of development, Grit had become complete enough to power GitHub during our public launch of the site and we were faced with an interesting question:

Should we open source Grit or keep it proprietary?

…After a small amount of debate we decided to open source Grit. I don’t recall the specifics of the conversation but that decision nearly four years ago has led to what I think is one of our most important core values: open source (almost) everything.”
- Tom Preston-Werner, “Open Source (Almost) Everything

It should be clear at this point that commercial valuations of software assets by both practitioners and public markets is in decline. This model of ours, once controversial, is now self-evident. We examined the public markets aspect in detail in May; little has changed since.

Of PwC’s Global Software Top 20 – the top twenty firms globally as measured by software based revenue – the youngest is 22 years old, and the median age of the top ten is 35 years. It has been over twenty years, in other words, since the software industry produced a business to rank in the Top 20.

Here is a chart of the market capitalizations of PwC’s top five vendors.

PwC Top 5

If we remove the artificially narrow lens of revenue from software sales, however, the picture looks very different.

PwC Top 5 + GOOG

The technology industry has produced entities that would place within the top five; they just aren’t in the business of selling software.

PwC Top 5 + GOOG, VMW, RHT

Firms that make money with software rather than from software are outperforming their counterparts in recent years; compare the respective valuations of Google, Red Hat and VMware.

Generationally, attitudes towards software are shifting, as evidenced by Werner’s comment above, and the release as open source of assets like Cassandra, FlockDB, Hadoop, Hip Hop, Hive, Jekyll, Nginx, Pig, Resque, Storm, Thrift and so on. Entrepreneurs and public markets alike are turning their attention away from software sales and towards data oriented revenue models in search of outsized returns. There is and will continue to be an enormous market for software, but real growth is increasingly coming from areas other than software sales and service. This is true even for those currently in the business of selling code; for Sonatype and many other startups, data is increasingly the product.

As obvious as this is to most industry participants, however, the Age of Data remains hobbled by its lack of a free market.

In the early days of commercial open source consumption, the exploding number of so-called vanity licenses – those that were non-standard and vendor specific – and a general lack of understanding of their legal implications inhibited the market for open source, relegating it in many settings to unacknowledged, behind the scenes usage. Gradually, however, license proliferation was curbed and licenses generally began to coalesce around permissive (Apache, BSD, MIT, etc) or file (EPL, MPL, etc) / project (AGPL, GPL, etc) style reciprocal licenses. The net impact of which was increased adoption, because the previous uncertainty necessarily implied risk which in turn throttled adoption.

Which is essentially where the data market is today. Everybody understands that data has value; there is little consensus on how, where and via what mechanisms it should be distributed, licensed and sold. Startups like Buzzdata, Datamarket, Factual and Infochimps cannot by themselves make a market, and with rare exceptions like Microsoft with the data section of its Windows Azure Marketplace, large providers tend to be heavily risk averse with respect to the liabilities posed by data.

Absent a market with well understood licensing and distribution mechanisms, each data negotiation – whether the subject is attribution, exclusivity, license, price or all of the above – is a one off. Without substantial financial incentives, such as the potential returns IBM might see from its vertical Watson applications, few have the patience or resources to pursue datasets individually. We’ve experienced this firsthand, in fact; as we’ve looked for data sources to incorporate into RedMonk Analytics, conversations around licensing have been very uneven. After gaining verbal approval from one content provider for our proposed usage of their API, this was the message we received in response to our request for an actual license:

The short answer is… we are not exactly sure what you are planning to do and whether or not it would violate our terms of service, so the best answer I can give you is that if you are not sure, you’re best off consulting with your own attorney.

As a rule, we don’t have the staffing resources internally (or the legal resources, given the high cost of lawyers!) to make one-off contracts or licenses or exceptions to the generic terms of service.

Hope this helps!

Given the current state of data licensing, it’s impossible to blame them for taking this position, even as it prematurely terminated what could have been a mutually beneficial relationship. Why dedicate legal resources to license review when the projected return on the asset is uncertain? Nor would a commercial negotiation be any more straightforward.

Market efficiency is a function of volume; the more participants, the better an understanding we have of an asset’s worth. With limited mainstream market participation in the business of data, opinions of the value of a given asset tend to be asymmetrical. Put more simply, when we ask what a given dataset is worth, the only correct answer at present is: we just don’t know.

All of which helps explain why Infochimps and others startups targeting the data marketplace opportunity are not as visible as their significance suggests they should be.

For all of the current inefficiency in data procurement, however, it is a temporary condition. Apart from the fact that history tells us risk assessment and licensing are solvable problems, the financial incentives are sufficient to guarantee progress if not complete solutions. Consider the case of Watson. IBM has historically avoided data collection due to legal concerns, but imagine the liabilities of the first diagnostic miscalculation. What will happen, in other words, when Watson commits the healthcare equivalent of “What is Toronto?” while assisting in the diagnosis of a patient? For IBM, the answer is clearly that the potential rewards more than offset the theoretical risk. Enough so, at least, to justify massive investments in development and marketing.

Life may be marginally easier for those that capture their own data, but this will by no means diminish the appetite for more. Google generates sufficient data to be able to predict the flu better than the CDC, but still felt obligated to license Twitter’s data. A relationship that ended, notably, because of a license termination. No matter how valuable an internal dataset might be, it will be more valuable still when recombined, remixed and correlated against complementary external data; see, for example, FlightCaster.

With respect to opening the throttle for data marketplaces, escalating demand virtually guarantees the supply. The real questions are who will play a meaningful role in reducing the friction, and when. Because the attendant opportunities are large indeed.

Disclosure: IBM, Microsoft, Red Hat and VMware are customers. Datamarkets, Facebook, Factual, Infochimps, Google and Twitter are not.

by-sa

Categories: Data, Marketplaces.

Napster: Lessons for The Enemies of Shadow IT

Napster logo

In 1999, Napster was unleashed upon the world. A year later they were sued by Metallica and Dr. Dre. A year after that the service peaked with 26.4 million users. A year after that the company filed for Chapter 7 to liquidate its assets.

While the record industry would have you believe that Napster’s meteoric rise was driven exclusively by thieves, the reality was that it was a desire for a product that record companies would not provide. Napster offered music available for dowload without draconian restrictions as well as the ability to download individual tracks rather than an entire album. Services that the record industry reluctantly agreed to years later. Years that the record industry also spent fighting a multi-front scorched earth legal battle against users of file sharing services that sprang up in the wake of the fall of Napster. When they finally did break down and sell music online, they were compelled to work with a much stronger player than Napster ever was.

But what if the music industry had been a rational actor and made the decision not to fight the tide, seeking agreements with Napster similar to the ones they employ today with Amazon, Apple and eMusic? What if they had recognized in those 26 million Napster users not thieves but potential customers and given them what they implicitly asking for: a more convenient way to obtain music?

The question is important because it’s essentially the same question facing enterprise IT today.

Napster made music available to anyone with an internet connection. For decades, enterprises have endured provisioning delays measured in months. Today, as Flip Kromer says, “EC2 means anyone with a $10 bill can rent a 10-machine cluster with 1TB of distributed storage for 8 hours.” It’s the end of procurement as we’ve known it.

Enterprise IT faces the same decision that the record industry once did: fight the tide or get out in front of it. Even setting the public relations damage aside, the returns of the former strategy for the record industry have been unimpressive. Given that developers have an increasing portfolio of accessible open source software and cloud services available to them, it’s unlikely that an enterprise crackdown on so-called shadow IT will be materially more effective. And then there’s question of whether throttling the constituency within your business that wants to move fastest is generally a good idea.

Why not enable them, then? Instead of firewalling the services Shadow IT wants, provide them centrally. Turn the tools that you are wasting your time fighting into an enticement to come out of the shadows. You’ll have better, if still imperfect, visibility into consumption and usage patterns as well as shorter development cycles. What’s not to like?

The RIAA missed their Napster opportunity. You don’t make the same mistake.

Bonus: My slides from the RightScale conference address this subject in more detail if you’re interested.

by-sa

Categories: Cloud, Shadow IT.