“This paper described Dynamo, a highly available and scalable data store, used for storing state of a number of core services of Amazon.com’s e-commerce platform. Dynamo has provided the desired levels of availability and performance and has been successful in handling server failures, data center failures and network partitions. Dynamo is incrementally scalable and allows service owners to scale up and down based on their current request load.

Dynamo allows service owners to customize their storage system to meet their desired performance, durability and consistency SLAs by allowing them to tune the parameters N, R, and W.”
- “Dynamo: Amazon’s Highly Available Key-value Store [PDF],” Giuseppe DeCandia, Deniz Hastorun, Madan Jampani, Gunavardhan Kakulapati, Avinash Lakshman, Alex Pilchin, Swaminathan Sivasubramanian, Peter Vosshall and Werner Vogels

In October 2007, Amazon published a paper describing an internal data store called Dynamo. Incorporating ideas from both the database and key-value store worlds, the paper served as the inspiration for a number of open source projects, Cassandra and Riak being perhaps the most visible of the implementations. Until yesterday, these and other derivative projects were the only available Dynamo implementations available to the public, because Amazon did not expose the internally developed database as an external service. With Wednesday’s launch of Amazon DynamoDB, however, that is no longer true. Customers now are able to add Amazon to their potential list of NoSQL suppliers, although to be fair they’ve technically been in market with SimpleDB previously.

The following are some points of consideration regarding the release, its impact on the market and likely customer questions.

AWS versus Hosted

The most obvious advantage of DynamoDB versus its current market competition is the fact that it’s already in the cloud, managed and offering consolidated billing for AWS customers. Requiring minimal setup and configuration versus native tooling, a subset of the addressable market is likely to be of a similar mindset to this commenter on the DataStax blog:

“Cassandra’s tech is superior, as far as I can tell. But we’ll probably be using DynamoDB until there is an equivalent managed host service for Cassandra. Moving to Cassandra is simply too expensive right now.

…

All those are clearly better served by a service like DynamoDB than trying to run their own Cassandra clusters unless they happen to be very proficient in Cassandra administration and want to dedicate precious human resources to administration. That takes a lot of the benefits of “cloud” away from small and mid-sized companies where cost and management are the limiting factors.”

For many, outsourcing the installation, configuration and ongoing management of a data infrastructure is a major attraction, one that easily offsets a reduced featureset. Like Platform-as-a-Service (PaaS) offerings, DynamoDB offers time to market and theoretical cost advantages when required capital expense and resource loading are factored in.

Like the initial wave of PaaS platforms, however, DynamoDB is available only through a single provider. Unlike Amazon’s RDS, which is essentially compatible with MySQL, DynamoDB users will be unable to migrate off of the service seamlessly. The featureset can be replicated using externally available code – via those projects that were originally inspired by DynamoDB, for example – but you cannot at this time download, install and run DynamoDB locally.

It’s true that the practical implications of this lack of availability are uncertain. NetFlix’ Adrian Cockroft, for example, asserts that migration between NoSQL stores is less problematic than between equivalent relational alternatives, because of the lower complexity of the storage, saying “it doesn’t take a year to move between NoSQL, takes a week or so.” It remains true, however, that there are customers that postpone upgrades to newer versions of the same database because of the complexity involved. And that’s without considering the skills question. Given the uncertainty involved, then, it seems fair to conclude that the proprietary nature of DynamoDB and the potential switching costs will be – at least in some contexts – a barrier to entry.

The question for users is then similar to that facing would be adopters of first generation PaaS solutions: is the featureset sufficient to compel the jeopardizing of later substitutability? Amazon clearly believes that it is, its competitors less so. EMC’s Mark Chmarny, additionally, notes that Amazon may be advantaging adoption at the expense of migration in its pricing model.

Competition

DynamoDB clearly has the attention of competitive projects. Basho – the primary authors of Riak – welcomed DynamoDB in this post while pointing out the primary limitation, and DataStax wasted little time spinning up a favorable comparison table. One interesting aside: the Hacker News discussion of the launch mentioned Riak 23 times to Cassandra’s three.

Basho and Datastax are right to be concerned, because the combination of Amazon’s increasingly powerful branding and the managed nature of the product make it formidable competition indeed. The question facing both Amazon and competitors is to what extent substitutability matters within the database space. Proprietary databases have had a role in throttling the adoption of PaaS services like Force.com and Google App Engine in the past, but we have very few market examples of standalone, proprietary Database-as-a-Service (DaaS) offerings from which to forecast. Will DaaS or more properly NoSQL-as-a-Service be amenable to single vendor products or will they advantage, as they have in the PaaS space, standardized platforms that permit vendor choice?

The answer to that is unclear at present, but in the meantime expect Amazon to highlight the ease of adoption and vendors like Basho and DataStax to emphasize the potential difficulties in exiting, while aggressively exploring deeper cloud partnerships.

NoSQL Significance

It’s being argued in some quarters that DynamoDB is the final, necessary validation of the NoSQL market. I do not subscribe to this viewpoint. By our metrics, the relevance of distinctly non-relational datastores has been apparent for some years now. Hadoop’s recent commercial surge alone should have been sufficient to convince even the most skeptical relational orthodoxies that traditional databases will be complemented or in limited circumstances replaced by non-relational alternatives in a growing number of enterprises.

Throughput Reservation

Perhaps the most compelling new feature of Amazon’s new offering isn’t, technically speaking, a feature. Functionally, the product is (yet) another implementation of the ideas in the Dynamo paper; Alex Popescu has comprehensive notes on the feature list. Receiving the most attention aren’t technical capabilities like range queries but rather the concept of provisioned throughput, levels which can be dynamically adjusted up or down.

This type of atomic service level provisioning is both differentiating and compelling for certain customer types. Promising single digit latency at a selected throughput level with zero customer effort required is likely to be attractive for customers that require – or think they require – a particular service level. And by requiring customers to manually determine their required provisioning level, Amazon stands to benefit from customer overprovisioning; customers will feel pain if they’re under-provisioned and react, but conversely may fail to observe that they’re over. Much like mobile carriers, Amazon wins in both scenarios.

Timing

With DynamoDB having been extant in some form since at least 2007, one logical question is: why now? Amazon did not detail their intent with respect to timing when they prebriefed us last week, but their track record demonstrates a willingness to be first to market balanced with an understanding of timing.

In 2006, Amazon launched EC2 and S3, effectively creating the cloud market. This entrance, however, was built in part from the success of the Software-as-a-Service (SaaS) market that preceded it; Salesforce, remember, went public in 2004. With enterprises now acclimated to renting software via the network, the market could be considered primed for similar consumption models oriented around hardware and storage.

Three years later after the debut of EC2 and S3, and one year after MySQL had achieved ubiquity sufficient to realize a billion dollar valuation from Sun [coverage], Amazon launched the first cloud based MySQL-as-a-Service offering [coverage]. That same year, the first year that Hadoop was mainstream enough to justify its own HadoopWorld conference, Amazon launched Elastic MapReduce.

The pattern is clear: Amazon is unafraid to create a market, but attempts to temper the introductions with market readiness. Logic suggests that the same tactic is at work here.

NoSQL has, as a category, crossed the chasm from interesting science project to alternative data persistence mechanism. But while NoSQL tools like Cassandra and Riak are available in managed form via providers like Joyent and Heroku, DynamoDB is, in Popescu’s words: “the first managed NoSQL databases that auto-shards.”

It is also possible that SSD pricing contributed directly to the launch timing, with pricing for the drive type down to levels where the economics of a low cost shared service finally make sense.

SSDs

One underdiscussed aspect to the Dynamo launch is the underlying physical infrastructure, which consists solely of SSDs. This is likely one of the major contributing factors to the performance of the system, and in some cases will be another incentive to use Amazon’s platform as many traditional datacenters will not have equivalent SSD hardware available to them.

The Net

While discussion of the DynamoDB offering will necessarily focus on functional differentiation between it and competitive projects, it is likely that initial adoption and uptake will be primarily a function of attitudes regarding lock-in. For customers that want to run the same NoSQL store on premise and in the cloud, DynamoDB will be a poor fit. Those who are optimizing for convenience and cost predictability, however, may well prefer Amazon’s offering.

Amazon would clearly prefer the latter outcome, but both are likely acceptable. Amazon’s history is built on releasing products early and often, adjusting both offerings and pricing based on adoption and usage.

In any event, this is a notable launch and one that will continue to drive competition on and off the cloud in the months ahead.

Disclosure: Basho is a RedMonk client, while Amazon and DataStax are not.

The cost of delaying my 2012 predictions is that one has already come to pass. Nginx – the web server now powering all of the redmonk.com properties – passed IIS according a January 4 Netcraft release. Because the quantitative data available to us has indidicated surging interest in the alternative web server – the logical result of which was a commercial response – we’ve been expecting something like this. But of course we can’t count this as a prediction any longer because it’s January 13th.

Here instead are a few things that have not yet come to pass, but will, I believe, in the year ahead. These predictions are informed by historical context and built off my research, quantitative data that’s available to me externally or via RedMonk Analytics, and the conversations I’ve had over the past twelve months, both digital and otherwise. They cover a wide range of subjects because we at RedMonk do.

For context, my 2010 predictions graded out as 66% accurate while 2011′s were 82% correct.

With that, the 2012 predictions.

Data & The Last Mile

It is not technically correct to assert that large scale data infrastructure is a solved problem. Decades of innovation remain, as the Cambrian explosion of projects demonstrate. It is nevertheless true that relative to the user interface, data storage and manipulation is a solved problem. Since the original creation of Hadoop in 2006, for example, we have seen multiple user interfaces applied: connectors (e.g. R), standard MapReduce, scripting (e.g. Jaql/Pig), SQL (e.g. Hive), spreadsheets (e.g. BigSheets), client tooling (e.g. Karmasphere). Each has its strengths, none bridges the last mile: putting the power of Big Data in the hands of ordinary users.

Which is perhaps unsurprising; even the mature relational database world uses abstractions of varying levels of complexity to interface with business users. But with data driven decision making on the rise, premiums are being placed on tooling which can expose in sensible fashion data to those without degrees in computer science. Hence, the elevated visibility of startups such as Metamarkets, who excite data scientists with tools like Druid but whose valuation may ultimately depend on its last mile expertise.

At this point in time, whatever my preferred model for data storage and whatever the type, there will be greater than one credible option for a data engine. The same cannot be said for presentation. Which would be less problematic if the market for Big Data talent were not so desperate; outsourcing to shops like Mu Sigma will be an option in some quarters, but comes with its own inefficiences and risks, not to mention per inquiry premiums.

This, then, will be an area of focus in 2012, for both innovation (look for assisted anomaly and correlation identification, a la Google Correlate) and M&A.

Desktop Importance Declines

The most interesting characteristic of the forthcoming Windows 8 release isn’t the technology, which is curious because it’s revolutionary from a Microsoft standpoint. From the support for ARM to the addition of the Windows Store to the ability to author in JavaScript and HTML5, there is much to digest. Instead, the single most defining characteristic of the pending launch is apathy.

Overall inquiries and discussion of the platform demonstrate curiosity but limited interest; the visibility of the once dominant Windows platform is secondary to mobile platforms like Android and iOS.

While this is not a function of any specific or general design failures on the part of Microsoft – indeed, the platform is incorporating important changes while making itself more developer accessible – it is symptomatic of a broader and more difficult to attack problem: the declining role of the desktop.

The desktop is simply not as important as it once was. Mobile usage is eroding the central role PC’s once played; while they are still the dominant form of computing, the trendline is declining and there is no reason to expect it to invert. It’s been suggested that mobile computing in general is additive; that it’s being used to extend the usage of computing to areas where PCs were not employed, and is thus non-competitive. But our data as well as Asymco‘s indicates that, at least in part, mobile usage is coming at the expense of traditional platforms. General search volume data, as we’ve seen, validates this assertion.

There are two implications here. Most obviously, Microsoft’s ability to generate interest in and thus leverage for its flagship operating system is jeopardized. Worldwide developer populations are not necessarily zero sum as skills overlap, but they tend to be rivalrous; an Android or iOS developer is often a lost potential Windows developer – experiments like BlueStacks aside. We can therefore expect Microsoft to have to expend more effort to attract fewer developers to their platform, a negative cycle which becomes cyclical. Second, as the desktop’s primacy abates, we can expect to see greater competition in the marketplace. As enterprises become by necessity more heterogeneous, incorporating Android and iOS devices, the costs of supporting second operating systems drifts towards marginal, which means that forecasts of greater Apple penetration become more probable.

Developer Shortages

It’s become axiomatic that industry hiring is all demand and short supply, and none of our clients expect any relief in the year ahead. Nor will they receive it. Shortages for in demand skillsets will continue over the next twelve months, advantaging entities that are either geographically positioned to leverage markets less competitive than the Valley or with the logistical ability to incorporate remote hires.

That said, we will in 2012 see the first steps towards a more rational market, through a combination of cultural shift and educational model innovation that will increase supply. Regarding the former, it’s no secret that technology has had a profound impact on the erosion of middle class jobs. In Race Against the Machine, MIT Professors Andrew McAfee and Erik Brynjolfsson document the role that rapid innovation has had on jobs:

Digital technologies change rapidly, but organizations and skills aren’t keeping pace. As a result, millions of people are being left behind. Their income and jobs are being destroyed, leaving them worse off in absolute purchasing power than before the digital revolution.

Even skill industries are not immune. From John Markoff’s New York Times piece, “Armies of Expensive Lawyers, Replaced by Cheaper Software“:

“From a legal staffing viewpoint, it means that a lot of people who used to be allocated to conduct document review are no longer able to be billed out,” said Bill Herr, who as a lawyer at a major chemical company used to muster auditoriums of lawyers to read documents for weeks on end. “People get bored, people get headaches. Computers don’t.”

While Brynjolfsson and McAfee are ultimately optimistic about the prospects of technical progress as they relate to employment, the outcome is far from certain.

What is becoming clear, however, is that unemployment rates that have been north of 8% in the US since February of 2009 are driving people into industries that are desperate for help. For some, this means oil & gas employment in traditionally underpopulated environments like North Dakota. For others, however, technology – long an enemy – is becoming a refuge.

We’re seeing a spike in inquiries about transitioning to technology careers. Lawyers, management consultants, teachers and others are seeking – and often finding – homes for themselves within the technology sector. Some are self-taught or trained on the job, others merely apply existing skills in new contexts, but both represent a potential cultural shift. Which begs the question: could technology be the next major middle class employment sector?

For that to happen, the education system needs to improve, because even an industry which has been one of the few economic bright spots of the last decade can only absorb so many unskilled workers without slowing. This is the real significance of applications like Code Academy or programs like Harvard’s free CSCI E-52, MITx or Stanford Engineering Everywhere: they are one potential solution to the perpetual shortage of talent. For all of the limitations of distance learning, the scale means that some subset of motivated students will become productive developers, and by extension, contributors to the larger economy.

This is a long term process, so obvious progress within 2012 will be minimal, and talent shortages will continue. But we will in the next twelve months begin to see distance trained students hired at scale, and this will be one of the first steps towards lower talent costs as well as, possibly, the restoration of middle class employment opportunities.

Monitoring as a Service

We are not oriented around category definitions at RedMonk; we prefer market driven names to those conceived and marketed by the analyst industry. That said, it seems clear that the time of Monitoring-as-a-Service (MaaS) is at hand. New Relic’s growth led to a $15M round in November, Boundary took $4M a year ago this month, Monktoberfest speaker Theo Schlossnagle’s Circonus has been in market for over a year, and virtually every vendor that we speak with today is adding monitoring and management facilities, from 10gen’s MMS to Cloudera’s Cloudera Manager.

The proliferation of these services is a direct response to the increasingly heterogeneous nature of application architecture and the reality that the substrate is frequently network based, rather than local. Given accelerating rather than declining consumption of network resources, we predict a strong increase in interest and adoption of MaaS tools. Much as I don’t care for the term itself.

Intelligent usage of generated telemetry – which we’ll come back to – will further cement adoption, delivering previously unseen value.

Open Source and the Paradox of Choice

Gartner in March of last year asserted that open source had hit a tipping point, saying:

“Mainstream adopters of IT solutions across a widening array of market segments are rapidly gaining confidence in the use of open source software.”

We concur, although we would argue that the tipping point actually occured ten years or more prior. The Apache web server and MySQL were originally written in 1995. In 1999, we saw the public offering of Red Hat and the creation by IBM – as mainstream a technology brand as there is in the enterprise – of the Linux Technology Center. Firefox was first released in 2003. None of these reached their relative levels of popularity in the past twelve months; they have instead been the de facto infrastructure for the better part of the last decade.

Regardless of when one asserts that open source crossed the chasm, however, it remains that it is a model whose popularity is increasing over time. As understanding of the benefits increases and concerns about the risks abate, more organizations are not only consuming open source but contributing to it. Evidence suggests, in fact, that perceptions of the value of software are in decline – we’ll come back to that too, and that the end result of this is that more proprietary code is being released as open source software.

Widely perceived as a net benefit, however, the influx of new projects does present problems for would be adopters. Specifically, the paradox of choice implies that developers will increasingly be forced to select from a growing sea of projects which may or may not be suitable for their needs. And while the nature of open source guarantees developers the ability to apply this code to their projects without restriction or commercial engagement, this is a process with a limited ability to scale. Consider the NoSQL space, as an example. Presuming for the sake of argument that the developers in question understand the different categories of database – key value stores, document databases, columnar databases, MapReduce engines, graph databases and so on – well enough to understand their high level needs, there are at least two and sometimes as many as half a dozen credible options to consider.

This paradox of choice, or too much of a good thing, will become more problematic over time rather than less as contributions will continue to rise. The net impact is likely to be increased commercial opportunities around selection, and therefore attention to vendors like Black Duck, Open Logic, Palamida and Sonatype.

PaaS: The New Standard

It has been evident for some time that runtime fragmentation – an aggressive diversification of programming languages and frameworks, specifically – will change the development landscape. The market failure of the first generation PaaS providers, in fact, was primarily a function of their over-prescriptive natures. The benefits to outsourcing management and scale were obsoleted by the constraints; Java shops were never likely to rewrite their application stack in Python or Ruby strictly to benefit from a platform. Which is why virtually every relevant PaaS provider today offers a choice of runtimes, so as to maximize their addressable market.

But in a fragmented world, what might emerge as a standard? From a developers’ perspective, the standard is most often the framework they’re deploying to, whether that’s Django, Node.js, Lift, Play, Rails, Spring, the Zend Framework or another. From a vendor perspective, however, the new standard is likely to be one level of abstraction up from individual language frameworks: the platform itself. Certainly this is VMware’s opinion, as they are in Maritz’ words trying to construct “the 21st-century equivalent of Linux” – i.e. the substrate that everything else is built on top of.

In 2012, this will become more apparent. PaaS platforms will emerge as the new standard from a runtime and deployment perspective, the middleware target for a new generation of application architectures.

Service Proliferation

With the inevitable adoption of multiple third party services – varying cloud resources, multiple, possibly overlapping, management and monitoring services and so on – will come challenges in making sense of the whole. Overall, instrumentation and visibility on a per service level is improved, but aggregating these views into a cohesive picture of overall architectural health and performance is likely to be highly problematic. Not least because the services themselves may present conflicting information and data. Google Analytics and New Relic, for example, are frequently at odds over load times and other delivery related performance metrics. Introduce in to that mix services like Boundary or CloudWatch and the picture becomes that much more complex. Connecting their data back to underlying log management and monitoring solutions such as 10gen’s MMS or Splunk is more complicated still.

The challenges of service intregration will create commercial opportunities for aggregating services which consume individual performance streams, normalize it and present customers with a consolidated single picture of their network performance. Commercial solutions will not fully deliver on this vision in 2012, but we will see progress and announcements in this direction.

Telemetry Usage

Five years ago, we began publicly discussing revenue models based around what we termed telemetry, or product generated datastreams. The context was providing open source commercial vendors with a viable economic model that better aligned customer and vendor needs, but the approach is by no means limited to that category: Software-as-a-Service vendors, as an example, are well positioned to leverage the data because they maintain the infrastructure. In 2011, we finally began seeing vendors besides Spiceworks take the first steps towards incorporating data based revenue models. For products like Sonatype Insight [coverage], data is not a byproduct, but the product.

In 2012, this trend will accelerate as necessary monitoring capabilities are added to product portfolios and industry understanding and acceptance of the model overcomes conservative privacy concerns. Many more vendors will begin to realize that like New Relic, which observed a decline in commercial application server usage, their accumulated data is full of insights on customer behaviors and wider market trends both.

Value of Software Will Continue to Decline

Capital markets have not, traditionally, been overly fond of software firms, perhaps because comparatively few of them eclipse annual revenue marks of a billion dollars – less than twenty, by Forbes‘ count. Microsoft’s share price has languished for over a decade in spite of having not one but two licenses to print money. The mean age of the PwC’s Top 20 software firms by revenue is 47 years; a fact which cannot be encouraging to startups.

Higher valuations instead are being awarded to entities that employ software to some end, rather than attempting to realize revenue from it directly. Startups today realize this, and the value of software in their models has commensurately been adjusted downward. Tom Preston-Werner, for example, describes the GitHub philosophy as “open source (almost) everything.” Facebook, LinkedIn, Rackspace, Twitter and others exhibit a similar lack of protectiveness regarding their software assets, all having open sourced core components of their software infrastructure that would have been even five years ago fiercely guarded.

This is becoming the expectation rather than the exception because it is nothing more or less than an intelligent business strategy. Businesses can and will keep private assets they believe represent competitive differentiation, but it will be increasingly apparent that less and less software is actually differentiating. As a result, 2012 will see even less emphasis on the value of software and more on what the software can be used to achieve.

Bonus: Facebook’s Most Important Feature

In 2012 will be Timeline. Mark it down.

Disclosure: Black Duck, Cloudera, GitHub, IBM, Microsoft, Sonatype and VMware are RedMonk customers, while 10gen, Boundary, Circonus, Facebook, Open Logic, Palamida, and New Relic are not.

Predicting is an easier business than it once was. True, technology is hysterically accelerating rates of change and disruption, but that’s only relevant if the substance of your predictions matters. Which all too often, these days, it doesn’t. Analysts and pundits are able to prognosticate with relative impunity; who has the time to go back and check their accuracy? Pageview driven models, in fact, reward wilder predictions because the error cost is, generally, approaching zero. Unless you predicted, say, that Linux would be killed off by Windows NT, nobody will remember later.

I find value in reviewing my annual predictions, however. If they prove correct, that’s useful. If they were not, understanding the reasons why is important to adjusting our models moving forward.

Because I made the mistake of making better than a dozen predictions last year, this year’s review will be delivered in two parts. Part 1, below, will cover my predictions for browsers, the cloud, data, developers and programming language frameworks. Part 2, covering predictions within hardware, mobile, NoSQL, open source and programming languages, will hit tomorrow.

If you’d prefer to read last year’s first, they can be found here.

Browsers

Firefox Will Cede First Place to Chrome, But Not Without a Fight

According to RedMonk Analytics, whose data reflects our developer-heavy audience, Firefox was able to hold off Chrome for two quarters. On the first of June, Firefox held a 32.58 share of our audience to Chrome’s 32.08. By the second, Firefox was in second place and would remain there for the balance of the year, widening the gap in the process. At present, our browser metrics peg Chrome at 36.38 with Firefox a distant second at 25.48.

I feel safe counting this one as a hit.

Cloud

PaaS Adoption Will Begin to Show Traction, With Little Impact on IaaS Traction

The first Platform-as-a-Service providers essentially asked developers to trade choice for development speed. Like Ruby on Rails – itself the basis for multiple first generation PaaS platforms – PaaS was built for those that would embrace constraints. But PaaS platforms never saw the type of growth that Rails experienced, in part because of the further loss of control that the cloud represents. It’s one thing to have a web framework like Rails dictate the way that you build web applications; having PaaS platforms also choose the operating system, database, version control systems and more was too much.

Which is why second and third generation PaaS providers have furiously removed barriers to entry, adding additional runtimes, open sourcing the underlying platform and allowing you to pick your provider. Which, in turn, is why adoption of PaaS is accelerating. VMware CEO Paul Maritz calls PaaS “the 21st-century equivalent of Linux,” which explains not only why they feel compelled to compete in the space, but also why Red Hat might.

Virtually every vendor in this space is reporting growth similar to the Hacker News trajectories for Cloud Foundry and Openshift (below).

In spite of the growth of PaaS, however, none of the metrics we track reflect any decline in usage of general infrastructure platforms. Quite the contrary, in fact.

I count this as a hit.

Data

Firms Will Increasingly Seek to Leverage the Data They Generate

Turning data into revenue has been one of the core themes of the past year, as well as the focus of my talk at the Open Source Business Conference in May. We’ve long held SpiceWorks up as a model of monetizing data, and as customers adjust to the reality that they’re already sharing data and vendors cease to regard it as a third rail issue, we’re seeing more businesses embrace data based revenue streams, as with Sonatype Insight. From 10gen to Black Duck, vendors are increasingly positioning themselves to be purveyors of data as much as software. Data is no longer the byproduct, but a product itself.

I count this as a hit.

Hadoop Will Become the MySQL of Big Data

EMC, HP, IBM, NetApp and even Oracle all have Hadoop – or in EMC’s case, MapReduce – plays in market. Microsoft actually deprecated its own Dryad initiative in favor of the Apache project. Players from AsterData to CouchBase to EnterpriseDB to MarkLogic to Tableau to Vertica have purpose built Hadoop connectors. The commerical distribution space, once essentially owned solely by Cloudera, has expanded to multiple third parties with varying points of differentiation.

Hadoop interest elsewhere, meanwhile, has not slowed.

Need I say more about the growing ubiquity of Hadoop? I count this as a hit.

Developers

Talent Shortages Will Continue

Granted, predicting a shortage of qualified development talent will be seen in some quarters as controversial as predicting that the sun will rise in the east. But part of this is context: certainly in January, the economic direction was less than certain. And in spite of an unemployment rate that has hovered just south of 10% for the better part of the last calendar year, hiring continues to be an issue for the majority of our clients. To the extent that several are spinning up offices solely for purposes of recruitment. This is not surprising, given the historical growth in employee headcounts (see above) that has, to date, been relatively resistant to the global economic crises.

Demand varies by skillset, as might be predicted, but 2011 remained – by our metrics – a tight market. Other market watchers support this assertion.

“Hiring talent in Silicon Valley is the toughest since the last bubble and investors are starting to openly wonder how this one will end.”
Steve Blank

“There is a war for talent, particularly developer talent, going on. Not just in Silicon Valley but also in NYC and many other places around the country.

Companies, small and large, are resorting to all sorts of creative ideas to recruit. Free lunches, free yoga, pushing code day one, cool schwag, options, RSUs, pretty much whatever it takes.”
Fred Wilson

While we don’t have good data then on market specific hiring (Bureau of Labor data is not fine grained enough), the evidence available to us seems to support the contention that shortages of tech talent remain.

I count this as a hit.

Frameworks

Node.js Will Continue its Growth Trajectory

October was a rough month for Node.js, with posts like Node.js is Cancer and node.js Is VB6 – Does node.js Suck? following the tradition of March reddit discussions like Is NodeJS Wrong? The Trough of Disillusionment, it seemed, had arrived well ahead of schedule.

Except that interest metrics showed no commensurate decline. Node took – again – three of the Top 5 spots in inbound search queries within RedMonk Analytics. Which is unsurprising against the backdrop of Google’s Insights for Search numbers.

Over on GitHub, meanwhile, which itself has achieved dramatic growth, Node.js is the second most popular watched repository, ahead of Rails, jQuery, HTML5-Boilerplate, and Homebrew. Microsoft clearly perceives this growth, because it has worked with Joyent to create a stable build of Node for Windows which in turn led to an SDK for Azure.

All of which means nothing except that Node’s growth trajectory continues.

I count this as a hit.

Part 2, tomorrow.

“From the beginning of time two forces have vied for influence over us. One is bottoms-up, decentralized, and emergent. The other is top-down, centralized, and directed.

The first force catalyzes change and divergence, while the second tends toward order and convergence. The first gives birth to new ideas, and the second enshrines them.”
- Adam Ludwin, From LOLcats To Occupy Wall Street, Everything Is Happening From The Bottom Up Now

Traditionally, industry analyst firms have been oriented around top down adoption patterns. CIOs and other IT decision makers comprise both the research subjects and purchasing audience for the majority of firms in this industry, large and small. Which was logical given traditional procurement patterns. When hardware, software and services are available only at high prices, command and control is an appropriate management structure. Attempting to scale the decision making process for big ticket items across a large body of middle managers is not likely to yield acceptable outcomes.

An approach that makes sense in one context, however, may be misapplied in another.

The technology purchasing landscape today looks very different than it did even five years ago. Where once CIOs might reasonably expect to have the clearest understanding of what technologies are leveraged within their own organizations, today they are, as Billy Marshall put it, “the last to know.” This pattern manifests itself every day within the majority of businesses. Not because CIOs are failing, but because of trends that have fundamentally and likely permanently disrupted their ability to centralize the technology adoption process.

The four trends we see as most important in driving this are arranged here in rough chronological order.

Open Source

In the late nineties, startups and enterprises alike were effectively beholden to commercial suppliers for the majority of their software needs. Because each piece of the requisite software infrastructure had to be licensed, the capital expenses associated with new initiatives was high. This represented a barrier to entry, and thus a brake on innovation.

With the popularization of open source software, developers from enterprises and startups alike were able to operate independently. For the first time, the actual software practitioners were free to choose their own software rather than having it selected for them and subsequently imposed upon them by upper levels of management. Even in situations where the ultimate production infrastructure targets remained commercially licensed software, open source software like Linux and MySQL allowed for prototyping and rapid development without the attendant costs, both financial and in procurement latency.

This was the first major shift affecting procurement, and perhaps the most profound. None of the infrastructure we take for granted today – Linux, Apache, MySQL, PHP, etc – were originally adopted from the top down. Their adoption was, instead, a fait accompli. CIOs – the last to know – gradually became aware that increasingly significant portions of their infrastructure, unbeknownst to them, were running on free and open source software. The inevitable demand for production support options for this software is what fueled, in time, the valuations of MySQL, Red Hat and others.

Bring Your Own Device

In October, Apple CEO Tim Cook asserted that 92% of the Fortune 500 were “testing or deploying iPad in the course of less than 18 months,” which may help explain why the iPad revenue stream by itself would place within the top third of that group. The interesting thing about this is that the majority of businesses appear uncertain about precisely why they’re deploying tablets: “Most participants, 51 percent, indicated that they did not have a clearly articulated strategy.”

The answer, in most cases, is that there isn’t one. iPad adoption, much like the penetration of iPhones and Android handsets is being driven by users who simply want the device. Faced with a choice between users – chief executive officers among them – who will employ their own devices for work purposes with or without the permission of IT, many businesses are compelled to support the platforms even without concrete business justifications for them.

The consumerization of the enterprise is decentralizing the process of technology selection, but its importance may lie rather in design. Like all products, technology is designed and built to be sold to a specific buyer. For enterprise products, historically, the actual user has been a secondary concern; the buyer – typically centralized IT – was the priority. Consumer technology companies like Apple, however, are designed for a user. What they give up in IT friendly features they more than make up for in usability and the ability to delight.

The Bring Your Own Device trend, therefore, may well improve user productivity by driving devices designed to be used rather than managed into organizations, from the bottom up.

Software as a Service

Software as a Service is a classic case study in timing with respect to market acceptance. Not many remember today that the model actually failed the first time around, when its practitioners were known as Application Service Providers. Pyschologically, few enterprises were prepared for either the idea of renting software or externalizing critical data like that stored in customer relationship management systems. By the midpoint in the last decade, however, these concepts were sufficiently commonplace to see Salesforce.com a publicly traded company with a valuation north of a billion dollars.

Consumer markets, meanwhile, had adapted much more quickly. Hotmail debuted in 1996, Yahoo Mail the year after and Gmail dropped in 2004.

Some of those same consumer services were pressed into service by enterprise workers, in fact; it was once common for Exchange users to forward all of their email to Gmail due to the disparity in storage limits between typical Exchange implementations and Google’s webmail product.

This pattern has played out repeatedly over the years, from webmail to CRM to project management software to website hosting to online helpdesks. All were adopted from the bottom up. By making applications available to anyone with a browser, often at low or no cost, SaaS has surged up through the ranks of enterprises. The inexorable nature of the model is reflected by the growth of providers large (Salesforce.com) and small (37signals).

Cloud

The single most important feature of the cloud has nothing, or at least very little, to do with technology. It is, rather, the pay as you go economic model. As Flip Kromer puts it, “EC2 means anyone with a $10 bill can rent a 10-machine cluster with 1TB of distributed storage for 8 hours.”

What this means in practical terms is that for the first time, hardware procurement is democratized. From an accessibility and availability standpoint, cloud is the hardware equivalent of open source software. Where open source allowed developers to bypass traditional procurement channels by making quality infrastructure and development software freely available, so does the cloud allow the growing class of devops technologists to leave the world of high latency hardware procurement – where same day server provisioning is a feature – behind. Armed with nothing more than a credit card, instances can be spun up and ready for use in ninety seconds.

Cloud is the final piece of the bottom up puzzle. Open source software and to a lesser extent SaaS allowed for the decentralization of enterprise technology development, but at some point hardware would become necessary which was the insertion point for IT. With public clouds, it is possible for the first time to entirely bypass the traditional gatekeepers.

The Net

It should be evident that traditional procurement and purchasing is not dead, just increasingly bypassed by a more efficient process. Also, that a great many enterprises continue to function largely as they always have: top down. More important than the question of whether this model is sustainable in the face of the trends above is whether it should be.

Before lamenting the fact that the above forces are disrupting and destabilizing your enterprise IT, consider that that may be a net gain. If the primary drivers of BYOD, Cloud, Open Source, and SaaS include ease of use, lower costs, frictionless availability, and speed of provisioning, are these trends worth opposing? Particularly since efforts to do so will, in all probability, fail?

Or are they instead assets to be strategically leveraged? There is little debate that businesses that move the most quickly have a competitive advantage. It’s not clear how businesses that prohibit the same tools that enable this will benefit.

Either way, bottom up adoption is here to stay: use it or lose.

In 1999, Napster was unleashed upon the world. A year later they were sued by Metallica and Dr. Dre. A year after that the service peaked with 26.4 million users. A year after that the company filed for Chapter 7 to liquidate its assets.

While the record industry would have you believe that Napster’s meteoric rise was driven exclusively by thieves, the reality was that it was a desire for a product that record companies would not provide. Napster offered music available for dowload without draconian restrictions as well as the ability to download individual tracks rather than an entire album. Services that the record industry reluctantly agreed to years later. Years that the record industry also spent fighting a multi-front scorched earth legal battle against users of file sharing services that sprang up in the wake of the fall of Napster. When they finally did break down and sell music online, they were compelled to work with a much stronger player than Napster ever was.

But what if the music industry had been a rational actor and made the decision not to fight the tide, seeking agreements with Napster similar to the ones they employ today with Amazon, Apple and eMusic? What if they had recognized in those 26 million Napster users not thieves but potential customers and given them what they implicitly asking for: a more convenient way to obtain music?

The question is important because it’s essentially the same question facing enterprise IT today.

Napster made music available to anyone with an internet connection. For decades, enterprises have endured provisioning delays measured in months. Today, as Flip Kromer says, “EC2 means anyone with a $10 bill can rent a 10-machine cluster with 1TB of distributed storage for 8 hours.” It’s the end of procurement as we’ve known it.

Enterprise IT faces the same decision that the record industry once did: fight the tide or get out in front of it. Even setting the public relations damage aside, the returns of the former strategy for the record industry have been unimpressive. Given that developers have an increasing portfolio of accessible open source software and cloud services available to them, it’s unlikely that an enterprise crackdown on so-called shadow IT will be materially more effective. And then there’s question of whether throttling the constituency within your business that wants to move fastest is generally a good idea.

Why not enable them, then? Instead of firewalling the services Shadow IT wants, provide them centrally. Turn the tools that you are wasting your time fighting into an enticement to come out of the shadows. You’ll have better, if still imperfect, visibility into consumption and usage patterns as well as shorter development cycles. What’s not to like?

The RIAA missed their Napster opportunity. You don’t make the same mistake.

Bonus: My slides from the RightScale conference address this subject in more detail if you’re interested.

There are two interesting takeaways from this week’s – and last, if we include EnterpriseDB’s news – flurry of PostgreSQL news. Most obviously, it’s a win for the Postgres community. Whether you believe that Salesforce or VMware are in this for the long haul as strategic database suppliers, each of the new guys is a large, publicly traded enterprise software provider visibly committing to Postgres. Which is big.

The tortoise to MySQL’s hare, Postgres has built its reputation slowly, incrementally over the years. Well regarded by those in the market for relational databases, the project’s biggest limitation historically has been its more limited visibility.

With Heroku and VMware bringing Postgres to the cloud the same week – no coincidence there – the problem of visibility should presumably be lessened. Confidence in the future of the platform coupled with improved accessibility should have a positive material effect on overall traction for the project.

The second impact of this news, however, is perhaps less obvious. With two more market entrants, the importance of the wider cloud database market, and therefore the need to have a credible cloud story, increases.

Cloud database offerings are generally greenfield; it’s often difficult to justify the costs of even internal database migration, let alone transitioning to an offsite platform. But as enterprises consider their big picture database investments and strategy, the ability to consume their platform of choice from the cloud as necessary will become more strategic and less tactical.

Which means that existing database vendors lacking a cloud story for their database strategy – i.e. most of them – should accelerate their efforts in this regard, whether that’s through development or partnership.

Update: Missed the EnterpriseDB news while I was out on vacation, so that’s been updated.

Disclosure: Both Salesforce.com and VMware are RedMonk customers.

Some people build their own house. A geometrically larger number of people rent.

This has been, to date, the subtext to Red Hat’s cloud narrative. Privately, Red Hat will tell you about the role they’ve played in building large clouds, public and private. Publicly, apart from table stakes efforts such as making RHEL available on EC2 [coverage], their story has been littered with phrases like “Plan => Build => Manage” or “Reference Architectures.” All of which is fine if your target market is those in the market for private home construction. It is less ideal if you intend to service the rental market as well.

Red Hat is Home Depot, then, to Amazon or Microsoft’s landlord. Or at least Red Hat was, prior to the announcements this morning. To explore the news, let’s turn to the Q&A.

Q: Before we begin, do you have anything to disclose?
A: Yes. Red Hat is a RedMonk customer, as are competitors such as Joyent, Microsoft or VMware. Amazon is not a RedMonk customer.

Q: For those that haven’t seen the news, can you tell us what Red Hat announced?
A: Red Hat has announced the non-production availability of a PaaS offering they are calling OpenShift. The URL for the service went live this morning.

Q: Before we get into the specifics, what’s the background here? Where did OpenShift come from?
A: Although their product catalog to date doesn’t necessarily reflect this, Red Hat has been focused on the cloud for some time. A few years ago during a conversation with CTO Brian Stevens, he mentioned in an offhand way a company called Makara, and recommended that we look at them. Which we did [coverage]. Almost a month to the day after we shot that video, Red Hat acquired Makara. Post acquisition, CEO Issac Roth was tasked with building out Red Hat’s PaaS capabilities and given access to resources across the company, including personnel from Red Hat internal IT and the support teams.

OpenShift is the result of those efforts. The Makara code plus some RHEL isolation features, the JBoss EE runtime, and other pieces gives you a top to bottom technology stack.

Q: Taking it apart, what is OpenShift?
A: OpenShift is a PaaS software layer that Red Hat will run and manage on top of third party providers.

Q: Which providers? Who is Red Hat partnering with?
A: Amazon to begin with, with more to follow.

Q: What is the PaaS? Is it framework oriented?
A: No, the PaaS appears to be focused at the runtime level. At launch, OpenShift is supporting Java, obviously, but also PHP, Python and Ruby. The frameworks are intended to run on OpenShift unaltered, and their early results indicate that that’s the case, with Django, Rails, Twisted, etc running without issue. Like VMware’s Cloud Foundry, OpenShift is essentially a container. Unlike Cloud Foundry, it is not yet available as open source.

Q: It’s not open source? Isn’t that unheard of for Red Hat?
A: It’s a bit of a departure, but the company has promised to make the code available. For now, however, they want to observe it in use as a free service to get a better idea of how it needs to be improved or updated.

Q: Have they provided a timeframe for when the code will be available?
A: Not to me they haven’t. Update: following the publication of this piece, I had the chance to speak to CTO Brian Stevens about the open source question. His comment: “we unequivocally and unambiguously commit to [OpenShift] being open sourced.”

Q: The service is free, however?
A: One level is, yes. At launch, there are two offerings: Express (developer oriented) and Flex (operations oriented). The free Express service is roughly analagous to Heroku, just with more languages. You git push your application to OpenShift, it handles the rest: scaling, provisioning, etc. The actual IaaS foundation – Cloudforms – is opaque, however, to Express users. For those who want more granular control over the environment, there’s Flex. It’s got monitoring and configuration features unavailable in the entry level service, and some higher levels of service – it’s not entirely clear what those are at launch – will be paid offerings. A third offering level – Power – will arrive later, and that adds complete control: root access, custom topologies, the whole kit.

Q: How will Red Hat make money off of the service, then?
A: By charging users who consume more than a certain entry level of service money. From the conversations I have had, it does not appear that Red Hat has finalized the exact pricing model yet. That will presumably be announced once it is production ready.

Q: But if I sign up for the free level of OpenShift, which is running on Amazon, Red Hat is paying my Amazon bills for me?
A: For a certain level of service, yes.

Q: What databases are available?
A: At launch, they led with MySQL for relational needs and MongoDB for document database tasks. Their partner page, however, indicates that Couchbase is up now with EnterpriseDB arriving soon.

Q: Put this in context for us: what is OpenShift comparable to?
A: It’s got features in common with a great many offerings, but the closest is probably Microsoft’s Azure. Like Azure, OpenShift is a service managed and run by the vendor, with platform abstractions layered over the underlying infrastructure. Unlike, Azure, however, OpenShift will be open sourced, will be available on multiple hosts, and supports Python and Ruby. Its scope, meanwhile, is broader than Cloud Foundry, Cloud.com, Eucalyptus, and OpenStack.

Q: Why is Red Hat doing this?
A: Big picture, Red Hat cannot afford to be left behind in the cloud market. As a platform player, they must guarantee their relevance to the next generation of enterprise workloads, whether they’re deployed locally or to the cloud. With regard to lockin and the cloud, Red Hat and its customers’ needs are aligned: both are afraid of users being locked into closed platforms.

If you talk to potential cloud adopters, the risk of being locked in is front and center alongside traditional enterprise concerns such as security and compliance. Red Hat, for its part, obviously needs to avoid history repeating itself. Should the cloud market coalesce around a single, closed source platform as happened in the operating system market, Red Hat’s ability to compete there effectively will be limited.

Preventing lockin, then, is important for vendor and customer alike. Which explains many of OpenShift’s design decisions: the ability to choose from multiple hosts, the support for multiple runtimes, and the eventual release of the code as an open source project.

Q: What does this mean for PaaS more broadly?
A: PaaS has gotten substantially more attention from vendors in recent months, beginning this past December with the acquisition of Heroku by Salesforce [coverage] through last month’s Cloud Foundry launch [coverage] and continuing today, with the introduction of OpenShift. Historically, the platform approach has failed to gain significant ground versus its more elemental alternative, IaaS. But this is, in our view, more attributable to design decisions on the part of initial PaaS providers than an inherent and fatal defect of the model [coverage].

As Heroku proved, PaaS platforms built around open models and standardized componentry are able to attract the developer attention that closed models have thus far been unable to. Vendors have made this adjustment, as we expected:

With a multiple year track record of of anemic adoption, PaaS vendors will adapt to customer demand or they will lose ground. Specifically, expect PaaS vendors to borrow from Heroku’s model [coverage], offering platforms assembled from standard or near standard componentry.

If nothing else, then, OpenShift will serve as further validation of the “open” PaaS model given Red Hat’s visibility and market presence.

Q: What does Red Hat need to do to attract even more developers to the platform?
A: Two things. First, support JavaScript. Second, release the code.

Q: Summing it up, what does OpenShift mean for Red Hat?
A: It means that they finally have a single SKU for cloud products. And that can do nothing but help widen their addressable market.

Attempts to address questions of private vs public cloud adoption necessarily involve detailed examinations of the economics. Obscured in these discussions, however, is the influence time has on perceptions of the economic costs and the theoretical return. Current return versus future costs is an important equation, and how it is parsed depends heavily on the availability of local resources.

The fundamental value proposition of the public cloud is accessible economics, as we discussed with Marten Mickos recently. As Infochimps’ Flip Kromer puts it, “EC2 means anyone with a $10 bill can rent a 10-machine cluster with 1TB of distributed storage for 8 hours.” The pricing established by the original market entrant, Amazon, have served two purposes. First, they have made resources available to even individual developers that would have been out of reach absent the pay per use model. Second, and perhaps more importantly, Amazon imposed a ceiling on cloud pricing; a ceiling which they continue to lower as they’re able to leverage larger economies of scale. Whatever the larger ambitions of other systems vendors with respect to the cloud market, Amazon defined the context in which they all must compete. At a pricepoint sufficiently low that the majority have chosen not to.

Developers have flocked to Amazon’s platform [coverage] not because it is the lowest cost option, but rather because it is the most accessible. From an economic perspective, they’re trading up front capital expenses for potentially higher ongoing operational costs. This trade is most attractive when resources – hardware and people – are few. When a developer can’t outsource the task of hardware acquisition and setup, and wouldn’t have the hosting facilities even if they could, the cloud is a compelling alternative.

It is less apparent, however, to larger entities that the public cloud economics compare favorably to their internal operational costs. Setting aside the question of whether their conclusions are sound, discussions with those implementing private clouds as an alternative to public implementations focus on the sustainable aspect of the economics. The accessibility of public cloud services is of less value to larger institutions, both because their expectations in terms of deployment speed are very modest and because they are typically not resource poor with either available hardware and IT resources.

Instead, larger enterprises focus on the operational margins of public cloud. Amazon’s pricepoint has historically commanded a margin above the cost of traditional hosting suppliers [coverage]. Given that larger enterprises generally argue that they can deliver infrastructure at a lower cost than traditional suppliers, the economics tilt even more strongly against large scale public cloud implementations.

The benefits of the public cloud remain of interest, however. What enterprise would not want a more elastic, easily provisioned infrastructure? Private cloud is the inevitable compromise. Promising feature benefits similar to those available on public infrastructure but with what is perceived by enterprises to be a more sustainable economic model, the private cloud is an increasingly attractive proposition for large enterprises.

Supporters of one or the other approaches may question the substance of the above characterizations, but these are the behaviors we have observed repeatedly. If you’re selling private cloud solutions, then, you’d do well to understand the appeal of accessible economics. Conversely, public cloud vendors may want to more clearly articulate the sustainability of their economics over time.

Disclosure: Amazon is not a RedMonk client.

It is true that predictions for the new year are best made before it begins. And predictions regarding consumer technology trends specifically should certainly be made prior to CES. All fair complaints.

Which I will now ignore.

The following are my predictions for the upcoming calendar year. They are informed by historical context and built off my research, quantitative data that’s available to me externally or via RedMonk Analytics, and the conversations I’ve had over the past twelve months, both digital and otherwise. They cover a wide range of subjects because we at RedMonk do.

With respect to their accuracy, as with all predictions these are best considered for what they are: educated guesses. For context, last year’s predictions graded out as approximately 66% correct.

On to the predictions.

Browsers

Firefox Will Cede First Place to Chrome, But Not Without a Fight

From Screenshots

We built RedMonk Analytics to track developer behaviors, and what it is telling us at present is that Firefox and IE both are losing share amongst developer populations to Chrome. Chrome is highly performant, but also benefiting from significant marketing investment (e.g. billboards, site sponsorships) and related product development (e.g. Chrome Web Store). The conclusion from this data is that Chrome will eclipse Firefox from a marketshare standpoint (speaking specifically of developers, not the wider market where Firefox is sustainably ahead), likely within a quarter.

But having tested the 4.0 version of Firefox for several weeks, it’s clear that Mozilla’s browser is responding to the evolutionary threat. Firefox 4.0 is faster and less stale from a user interface perspective, but more importantly differentiated via features like Panorama.

The 4.0 release is unlikely to be sufficient in preventing Chrome from assuming the top spot among developer browser usage, but it is likely to arrest the free fall. Expect Chrome and Firefox to be heavily competitive in 2011.

Cloud

PaaS Adoption Will Begin to Show Traction, With Little Impact on IaaS Traction

The conventional wisdom asserts that, at present, the majority of cloud revenue derives from IaaS offerings over PaaS alternatives. The conventional wisdom is correct. Infrastructure-as-a-Service has benefited from its relative simplicity and by virtue of its familiarity: IaaS offerings more closely resemble traditional infrastructure than PaaS. Platform-as-a-Service adoption, for its part, has been slowed by a variety of factors from legitimate concerns regarding platform lock-in to vendor design decisions. The result has been a marketplace that heavily advantages IaaS.

This will not invert in 2011, but the wide disparity in relative adoption will narrow as PaaS adoption climbs. With a multiple year track record of of anemic adoption, PaaS vendors will adapt to customer demand or they will lose ground. Specifically, expect PaaS vendors to borrow from Heroku’s model [coverage], offering platforms assembled from standard or near standard componentry.

Assuming that PaaS will never be successful because it has yet to be successful is illogical. Historical precedent demonstrates adequately that some markets take longer to establish than others (e.g. SaaS). Watch PaaS in 2011.

Interestingly, the rise in PaaS adoption will have little impact on Amazon. Most obviously because its traction has become, to some extent, self-fulfilling, but also because the vendor has anticipated demand and added platform-like features to its infrastructure offerings. With cloud being far from a zero sum market, it’s reasonable to expect Amazon and select PaaS vendors to both be successful.

Developers

Talent Shortages Will Continue

It is counterintuitive to speak of talent shortages when the overall labor market hovers near 10% unemployment, but the data supports no other conclusion. Employers that we speak with, large and small, are desperate for people. RedMonk Analytics query histories regularly feature searches on named individuals, a frequent precursor to recruiting efforts. Our @monkjobs account, for its part, has more positions that we can reasonably post.

It is unclear where this demand will lead. Fred Wilson is correct when he argues that the fundamentals of this hiring war are unsustainable in the longer term. But even should the hypercompetitive Silicon Valley market experience a major correction, it is probable that wider industry trends will remain biased towards talent, assuming the economy does not substantially recede in the next twelve months.

What this means, then, is that employers will be forced to be creative about talent acquisition. Besides traditional benefits such as high end workstations and stock options, employers may be forced to consider allowing employees to release their work as open source, granting them access to data not available elsewhere or giving them the right to work relatively autonomously within the larger organization. In many cases, it may involve leaving positions unfilled in favor of consumption of externally produced software. Other counterintuitive approaches to easing hiring pains include publishing guidelines on your proprietary engineering approaches (e.g. Hadoop), which affords some of the benefits of open source software – namely academic familiarity – without the attendant risks to competitive advantage.

Expect hiring to be a challenge in 2011. If identifying and employing qualified resources proved challenging during the worst recession since 1930, it is unlikely to become less so as global economies gradually recover.

Frameworks

Node.js Will Continue its Growth Trajectory

Of all of the technologies we tracked in 2010, none generated the same interest that node.js did. Using RedMonk Analytics to rank the incoming developer queries from January 1st, 2010 to December 31st, 2010, node took the top two spots on our list. Nor is our experience unique: Google Trends reflects the same spike in traffic.

We expect this to continue. Node.js is not just another framework; it’s a fundamentally different way of approaching challenges like concurrency that limit even moderately sized properties today. Couple this with the ascendancy of Javascript and the aggressive evolution of the V8 stack and we see no reason to expect Node.js to plateau.

Data

Much of what I would predict in this space has already been said, better. O’Reilly’s Edd Dumbill, for example, expects data marketplaces to come of age in 2011. Given that we expected data to emerge as a revenue generating asset in 2010, we concur. And both Edd and Hilary Mason predict substantial data related job volume expansion, and thus a tight labor market. Again, we are in agreement. But here are two predictions for what we expect to see in data.

Firms Will Increasingly Seek to Leverage the Data They Generate

With the cost of storage falling in response to declining memory prices as well as the introduction of commodity hardware and software, organizations have greater capacity for telemetry capture. They also have an increased ability, via open source software, to leverage this data in new ways. Better and more accessible big data technology is enabling entirely new lines of inquiry. As a result data which was once considered a byproduct becomes an asset, economically speaking.

Predictably, this is causing friction between business and IT. Business is more cognizant of the value of their data by the day, but traditional IT departments which are less familiar with the emerging class of big data tools are hitting the brakes on putting it to work. Patience with this tactic is exhaustible, however. Lines of business will have their results, whether they have to go outside the organization or not.

We will undoubtedly see more public data made available, as Hilary predicts, and more private data sold, as Edd anticipates. But we will also see organizations become introspective in search of internal, high value data.

Hadoop Will Become the MySQL of Big Data

This is already arguably true, but by the end of 2011 there will be no argument. Much as MySQL emerged as the dominant player among many candidates for the lightweight backend to websites large and small, Hadoop is becoming the de facto standard platform for Big Data. While Hadoop’s MapReduce and distributed filesystem implementations are already effectively the standard outside of proprietary alternatives, its extensibility is what will prolong its longevity. Hive and Pig bring SQL-like interaction to the store, while projects like Hbase bring to Hadoop GFS-like key-value store capabilities.

There are undoubtedly shinier proprietary technologies (e.g. Dremel, Percolate), but Hadoop’s market presence will be cemented in 2011.

Hardware

• Desktop:
  Workstations Will Make a Comeback:
  For all the talk of machine availability driven by the cloud, most of the developers I know – particularly those who play in big data spaces – are rotating away from MacBook Pro-style portable hardware to workstation-class hardware. When you’re operating on large datasets or virtualizing multiple operating systems, the combination of outsized display, fast processor and large memory footprint makes life easier. Laptops will not die out, in spite of the coming tablet wave, but they will be less primary in their role moving forward.
• Servers:
  ARM Will Emerge as a Server Player:
  It may seem obvious in light of the news that Windows will run on ARM and Nvidia’s Project Denver, but the growing importance of ARM has been apparent for months now. Speak with hardware suppliers, and they will all privately admit that they’re experimenting with ARM based server designs. Software development trends, meanwhile, reflect this. Observe the spike in ARM discussions on the primary Linux development mailing list in 2010.

  From Screenshots

  Doubtless a majority of the conversations center around mobile architecture and design, as Android and other projects have driven Linux on ARM into consumer markets in volume. But the attraction of the platform, most particularly in power consumption, don’t end in mobile.

  It’s not clear to me that Nvidia’s ambitions of realizing ARM markets in consumer desktops will meet with success, but the evidence for a significant server presence is there.

• Tablets:
  Tablets are a Real Market:
  There’s speculation in some quarters that tablets are a passing phenomenon. I do not subscribe to this view. Tablets are, at present, seriously limited by input mechanisms. But they excel in other areas. The Motorola Xoom tablet, as an example, is claiming 10 hours of battery life displaying video. More important is the anecdotal evidence that machine usage patterns seem to be changing. If workstations become more common and use of laptops as workstations declines, there’s a natural market for lightweight, portable machines. Particularly for those who need to demo things.

Mobile

Challenges of Native Development Will Drive Interest in HTML5 and Hybrid Approaches

The inevitable product of multiple successful mobile development platforms is a fragmented development landscape. Further complicating the development process is the challenge of navigating competing marketplace options. Enterprises privately express frustration with the status quo, and even developers are questioning the efficiencies of reimplementing the same application for multiple platforms. As a result, expect cross platform approaches such as HTML5 and hybrid-compilation solutions such as PhoneGap to attract more interest as the challenges of supporting the Android tablets that will begin arriving in Q1, Windows Mobile, and so on become more acute.

NoSQL

There are many relevant trends in the NoSQL space worth discussing, but two predictions that can be made with a reasonable degree of confidence are:

• The NoSQL Marketplace Will Experience Consolidation:
  We’re in the beginning stages of a cycle which is likely to output a handful of successful projects. Much as the relational database market cannot support an infinite number of comparable projects, the NoSQL space will begin to contract from its 2010 height as technically credible but not popularly adopted projects are supplanted by competitors.
• NoSQL Will Look More Like Pro-SQL:
  One of my original objections to the NoSQL term was its implicit rejection of a widespread, well understood technology in SQL. The limitations imposed by this designation have been clear to a variety of projects, which in turn have led to the reintegration of features common to traditional RDBMS systems but typically omitted from first generation NoSQL stores. These include indexing and, ironically, SQL access. NoSQL stores will remain differentiated from relational alternatives moving forward, but less so than they were in years past. We’ve seen this trend play out, before, remember, with MySQL. The enterprise tension requires projects to walk a fine line [coverage].

Open Source

1. Open Source of Non-Strategic Infrastructure Assets Will Increase
   Historically, businesses that developed software have considered it a differentiating asset, one worth protecting. Over time, many of these in house development efforts have given way to packaged applications. An insurance company, for example, is unlikely to develop Customer Relationship Management software that’s superior to what an open market can provide. While this is widely understood and accepted, open source software is not yet viewed in the same context.

   This is changing. Intelligent enterprises are increasingly aware of the lack of differentiation that infrastructure software provides, and are therefore rethinking the economic model for software development. If software is non-differentiating, the benefits to making it publicly available may easily offset the cost of maintaining it internally [coverage]. Such is the logic that has led to the release of projects like Cassandra, Hadoop, and Hbase.

   Expect to see more of this moving forward, as the benefits to open development become more obvious and the anticipated downside proves to be less than substantial.

2. Forking: How Development Gets Done
   Development, historically, has been a serial activity. A developer passes changes to another developer who adds to the codebase and so on. The advent of version control systems such as Subversion made this process more efficient, but didn’t really alter the dynamics. Distributed source code management technologies such as Git or Mercurial, however, enable a concurrent, parallel development that has more in common with bacterial replication than centralized version control.

   Networked implementation of these toolsets such as Github have fundamentally and permanently altered the nature of development [coverage]. This approach has become so common, in fact, that where once open source projects would do everything in their power to discourage forking, today their websites actively incite it (see below).
   
   

   In 2011, then, forking won’t be a curiosity: it’s how development will get done. Most frequently on network-backed implementations on Github.

3. Ubuntu is the New SuSE
   It was six years ago last November that we first speculated on the possibility of IBM supporting its applications on a Debian based distribution. Last year, this became reality for two reasons. First, because Canonical was more aggressive in embracing greenfield opportunities on the cloud than its competitors, who were myopically focused on their customers; customers who were collectively well behind the curve. But just as important has been the end of Novell [coverage]. For all of Attachmate’s promises to give SuSE the support it needs to continue servicing its install base, the market has already largely reacted away from the acquired entity in favor of, primarily, Canonical’s Ubuntu.

   Red Hat will maintain a firm grip on the commercial Linux distribution lead as a result of its consistent performance and excellent account control, but Ubuntu will emerge as the de facto alternative at the expense of SuSE.

Programming Languages

James stated this perfectly: at one point, learning Javascript meant you were not a serious programmer. At this point, not learning the language makes the same statement. Javascript is resurgent, and buoyed by the V8 runtime and adjacent frameworks like Node.js will expand its footprint on the server.

Bonus

The mistake often made with Dropbox is its categorization as merely another synchronization application. All of the quantitative data available points to a larger opportunity. Alexa, for example, shows Dropbox’s growth trajectory.

From Screenshots

Google Trends, meanwhile, shows a more pronounced 2010 spike in interest:

From Screenshots

This growth is driven by a variety of needs. The most basic is simple backup or cross-platform file availability. But underlying most of these behaviors is a more subtle realization: Dropbox is a virtual, cross-platform filesystem for a substantial and influential population. Whether you use a MacBook and an iPad, Windows and a Blackberry, or Ubuntu and Android, as I do, your filesystem is available. This is particularly important for device types – smartphones and tablets, primarily – that don’t natively expose their own filesystem. Dropbox doesn’t message this, but features like the shareable links for music are an indication of the possibilities to an abstraction of the filesystem that not only spans devices but that can be exposed via a browser.

The net of this growth and its drivers is that Dropbox will become an attractive acquisition target. Expect Apple, Google and Microsoft to be among the interested parties. Acquisition scenarios are likely to be challenging, however, not just because of the potential competition for the asset, but because Dropbox is relatively successful in monetizing usage. The data available does not support prognostications of a specific exit, then, but it does indicate that those discussions will accelerate in the coming year.

“Most of my friends at some point have heard me say, ‘In hindsight we are either geniuses or idiots.’ It’s particularly true for someone like me who makes a living writing and analyzing things.” – Om Malik, “The Perils of the Prediction Game”

Failure is part of predictions. Certainty demands comprehensive intelligence, which is by definition absent in the business of forecasting. But in prediction, the question isn’t whether you fail: that’s expected. The metric by which success is determined is rather how often. That is what separates the genius from the idiot.

Let us score last year’s predictions, then, so that you may know whether to pay attention to the 2011 iteration.

Cloud API Proliferation Will Become a Serious Problem

The December partnership between Eucalyptus and Red Hat highlights the issues with API proliferation. Granted, for Eucalyptus, the deal was likely driven more by pragmatism than customer pull with respect to Deltacloud. But the fact that an API superset was discussed on equal footing with the partnership itself speaks to the central role APIs are taking in cloud deployments.

Google Trends validates the assertion that 2010 saw increased activity around cloud APIs.

From Screenshots

This activity – which led, in part, to the rebranding of Sonoa – represents interest, yes, but also the challenge. As the software industry reorients around the cloud, APIs become their primary customer interface. Which, given the proliferation of vendors in the space, is problematic today and likely to become more so: an API per provider is an approach that will not scale.

I feel safe counting this one as a hit.

Collaboration Will Never Be the Same

Scoring this prediction is a challenge. The cited projects – Google Wave and Mozilla Raindrop – have failed to deliver, with the former officially deprecated and the latter effectively so.

Nevertheless, collaboration did fundamentally change in 2010. Priority Inbox introduced algorithmic email ranking to the masses. Tungle continued to facilitate scheduling. Rapportive injected social networking into Gmail. Dropbox launched synchronized file sharing capabilities for teams. New Twitter fostered conversations on the service. And Facebook has promised to reinvent communication with its forthcoming Facebook Messages offering.

I’ll call this a push.

Data as Revenue

To the best of my knowledge, we have yet to see data recognized as a balance sheet worthy asset. That’s a miss.

And yet with Goldman valuing Facebook as a $50B entity on the basis of a reported $2B in revenue, it’s clear that the value of data is well understood, if implicitly. The venture community appears to agree, funding data marketplace startup Infochimps to the tune of $1.2M.

I’ll call this one a miss, if only because the prediction, aggressively stated, was unfulfilled. I stand behind the conviction, however, and I think the market bears it out.

Democratization of Big Data

Consider that RedMonk, a four person analyst shop, has the technical wherewithal to attack datasets ranging from gigabytes to terabytes in size. Unless you’re making institutional money, budgets historically have not permitted this. The tools of Big Data have never been more accessible than they are today.

Hadoop, in paricular, has made remarkable strides over the past year, with improvements to core, projects such as Hive evolving and others like Oozie emerging. The center of gravity around the infrastructure is remarkable, with a few properties even bending it towards traditional back end web roles, ill suited as it is for that generally.

Nor is Hadoop the only example of the ready availability of tools. Non-relational stores continue to evolve at an impressive rate, with hardware access not far behind.

This one is a hit, as far as I’m concerned.

Developer Target Fragmentation Will Accelerate

If anything, I didn’t predict this aggressively enough. There is observable fragmentation in virtually every area of infrastructure technology: operating system, database, programming language, development framework, mobile platform, cloud platform, cloud API [coverage]. Even hardware is less straightforward than it once was, with ARM a driver in mobile with larger potential.

All of this has happened before, of course: the Cambrian explosion of diversity we’re currently witnessing is not unprecedented. It is, however, potentially further reaching as the boundaries between consumer and enterprise break down, and distinctions between on premise and network hardware blur.

This is a hit.

It’s All About the Analytics

The launch of Tableau Public in February [coverage] and its subsequent success is enough to validate this prediction by itself. The introduction, a month later, of the Google Public Data Explorer is gravy: analytics are fast becoming ubiquitous. It’s hard to miss infographics in the New York Times or the Economist, and as the data from Indeed.com indicates, hiring in the analytics space spiked in the last twelve months.

From Screenshots

This is a hit.

Marketplaces Will Be Table Stakes

When browser makers feel compelled to launch marketplaces effectively rebranding websites as applications – as Google did in May – it’s safe to conclude that marketplaces are table stakes. We’re even seeing marketplace proliferation within a single ecosystem, in Android. Vendors are at last perceiving the demand for marketplace features – centralized discovery, standardized payment, etc, but the real driver is the realizable revenue they represent, model depending.

In other words, if you have platform ambitions, you’d better be bringing a marketplace. So this is a hit.

New Languages to Watch

This is a miss. The conclusion is partially correct: in JavaScript, a language did emerge and begin separating itself from the pack. But the languages cited – Clojure and Go – didn’t realize their potential in 2010. Clojure remains popular amongst a niche audience – it was roughly as popular as Haskell and Scala, in our survey of Hacker News – but Go is more or less invisible from a general usage standpoint.

Ergo, the miss.

NoSQL Will Bid for Mainstream Acceptance

Even if we do not include Hadoop, which has clearly crossed the chasm towards mainstream adoption, it’s clear that NoSQL is finding fertile ground in mainstream enterprises. Most NoSQL vendors and technologists we work with have done substantial work over the past year for name brands. From Riak deployments at Comcast to MongoDB instances at the New York Times, it’s safe to say that NoSQL is making a bid for mainstream markets.

This is a hit.

I also made five brief, FOSS specific predictions.

1. Prediction: Usage of dual licensing will continue to decline, in part because of the Oracle and EU dispute over MySQL.
   Comments: Dual licensing, which is distinct from what is commonly called dual core, is in evident decline. With the exception of long time practitioners such as MySQL, dual licensing is becoming increasingly rare. I’ll call this a hit.
2. Prediction: FOSS advocates will increasingly turn their attention from licensing to the related mechanisms of copyright and trademark.
   Comments: Copyright and trademark have both taken center stage over the past year; the former most obviously with respect to the acquisition of MySQL via the Sun transaction, the latter most recently with Oracle’s mishandling of the Hudson assets. This is a hit.
3. Prediction: Permissive licensing will continue to gain at the expense of reciprocal licensing, albeit slowly
   Comments: Comparing the Black Duck license adoption figures I have at hand (meaning no January 2010 numbers), since August 2009 I have the GPLv2 down 4.02% (though v3 is up 1.34%), while Apache is up .65% and MIT 3.9%. This reflects our experience and our expectations, with permissive licenses trending up and reciprocal licenses trending down. Hit.
4. Prediction: The value of project code will be eclipsed, in a few cases, by the data the project generates.
   Comments: This is more difficult to assess. Much of Facebook’s software infrastructure, as an example, is made available via open source, indicating that for that organization, software is non-differentiating. The selection of Hbase over Cassandra as the foundation for Messages serves as further validation of the substitutability of the underlying infrastructure. But neither example yields hard valuations of the underlying code, making an assessment of this prediction difficult. This one’s a push.
5. Prediction: Open source, building from its mainstream acceptance, will emerge as the most credible alternative to proprietary cloud and mobile platforms
   Comments: At the infrastructure-as-a-service level, Cloud.com, Eucalyptus and OpenStack are the primary alternatives at this point to roll-your-own cloud offerings. All make at least a portion of their source available. At the platform level, open source frameworks such as Django (Djangy, GAE), Rails (Heroku, Engine Yard), and Spring (VMware) are emerging as developer preferred alternatives to vendor specific platform implementations. This is a hit, in other words.

The Final Tally

Scoring the 2010 predictions, then, we get six of nine correct for the main category against two misses and one push. Amongst the FOSS specific predictions, four of five were hits with one push. A sixty-seven percent success rate in non-FOSS predictions may seem unimpressive, but anything better than fifty percent can produce market value so I am comfortable with the results.

If you enjoyed last year’s predictions, then, please stay tuned for the 2011 version.

Disclosure: Apigee, Basho (Riak), Black Duck, Eucalyptus, and Red Hat are RedMonk customers. 10gen (MongoDB), Facebook, Oracle and Google are not.