For years now, it has been self-evident to us at RedMonk that programming language usage and adoption has been fragmenting at an accelerating rate [coverage]. As traditional barriers to technology procurement have eroded [coverage], developers have been empowered to leverage the runtimes they chose rather than those that were chosen for them. This has led to a sea change in the programming language landscape, with traditional language choices increasingly competing for attention with newer, more dynamic competitors.

The natural consequence of this tectonic shift has been uncertainty. Vendors for whom supporting Java and Microsoft based stacks was once sufficient are being forced to evaluate the array of alternatives in an effort to maximize their addressable audience. Platform-as-a-Service (PaaS) stacks like Cloud Foundry and OpenShift are perhaps the best example of this; the differentiation for each at launch was in part their support for multiple independent runtimes from JavaScript to Ruby.

While the question is obvious – which languages should I support? – the answer, and mechanisms for determining an answer, have been considerably less so. There is no canonical metric for determining platform traction; we employ half a dozen or more internally at RedMonk, depending on context, which incorporate everything from GitHub LOC rankings to LinkedIn group membership data.

But one of our favorites is the one originally developed by Drew Conway in 2010. It compares and contrasts the rankings of programming languages on GitHub and Stack Overflow to provide a broader view of language popularity. Our first snapshot using this model came in September. Five months later, we recompiled the data and plotted it to see what – if anything – had changed. Herewith the updated plot.

In general, the addition of languages like Dylan, Turing or Rust aside, little has changed six months on. We still have two clearly defined upper language tiers, with two to three less visible below that. There are, however, several developments worth discussing in more detail.

• CoffeeScript: billed as a more syntactically approachable alternative that compiles to JavaScript, CoffeeScript made subtantial performance gains relative to its Stack Overflow tag volume (63%), but also jumped significantly in terms of its popularity on GitHub. Since September 1st, CoffeeScript was not only one of 11 languages to increase in popularity, it jumped the furthest, going from #19 to #13. The jump is even more significant since six new languages were added to GitHub’s list in that span. With all due apologies to Bryan Cantrill, the numbers indicate that CoffeeScript is one of the fastest growing platforms by this metric.
• Java: as recently as a year ago, Java was widely regarded as a language with a limited future. Between the increased competition from dynamic languages and JVM based Java alternatives, while the JVM had a clearly projectable future, even conservative, enterprise buyer oriented analysts – the constituency most predispoed to defend Java – were writing its obituary. As we argued at FOSDEM last February, however, these conclusions were premature according to our data. One year in, and the data continues to validate that assertion.

  Apart from being the second highest growth language on GitHub next to CoffeeScript, Java – already the language with the second most associated tags on Stack Overflow – outpaced the the median tag volume growth rate of 23%. This growth is supported elsewhere; on LinkedIn, the Java user group grew members faster than every other tracked programming language excepting C# and Java. This chart, for example, depicts the percentage of LinkedIn user group growth for Java and JVM based alternatives since November of 2011.

  

  This outperformance is even more impressive when the overall member numbers are factored in.

  

  Our data, then, indicates that Java remains – in spite of the fragmented programming language landscape – a viable, growing language.

• Rust: a C/C++ like syntactical language originally developer in 2006, with the 0.1 of its compiler completed only last month, Rust has surprising traction on GitHub. On February 1st, it sat at 21 on GitHub Explore, ahead of Clojure, Groovy, Erglang, R, Go and a half dozen other relatively popular languages. While this is almost certainly a product of Mozilla’s involvement in the project, it has caught the eye of more than a few prominent technologists. There are a mere 4 questions tagged with Rust on Stack Overflow, so it’s clearly early days, but Rust is on the radar.

Other quick hit observations:

• C dropped 2 spots on GitHub’s rankings, from 5 to 7
• Go posted the fourth highest growth percentage on Stack Overflow, R was sixth
• Java passed PHP
• Prolog jumped six spots on GitHub from 30 to 24
• Scala may be separating itself from the other Tier 2 languages
• Viml is popular on GitHub

“This paper described Dynamo, a highly available and scalable data store, used for storing state of a number of core services of Amazon.com’s e-commerce platform. Dynamo has provided the desired levels of availability and performance and has been successful in handling server failures, data center failures and network partitions. Dynamo is incrementally scalable and allows service owners to scale up and down based on their current request load.

Dynamo allows service owners to customize their storage system to meet their desired performance, durability and consistency SLAs by allowing them to tune the parameters N, R, and W.”
- “Dynamo: Amazon’s Highly Available Key-value Store [PDF],” Giuseppe DeCandia, Deniz Hastorun, Madan Jampani, Gunavardhan Kakulapati, Avinash Lakshman, Alex Pilchin, Swaminathan Sivasubramanian, Peter Vosshall and Werner Vogels

In October 2007, Amazon published a paper describing an internal data store called Dynamo. Incorporating ideas from both the database and key-value store worlds, the paper served as the inspiration for a number of open source projects, Cassandra and Riak being perhaps the most visible of the implementations. Until yesterday, these and other derivative projects were the only available Dynamo implementations available to the public, because Amazon did not expose the internally developed database as an external service. With Wednesday’s launch of Amazon DynamoDB, however, that is no longer true. Customers now are able to add Amazon to their potential list of NoSQL suppliers, although to be fair they’ve technically been in market with SimpleDB previously.

The following are some points of consideration regarding the release, its impact on the market and likely customer questions.

AWS versus Hosted

The most obvious advantage of DynamoDB versus its current market competition is the fact that it’s already in the cloud, managed and offering consolidated billing for AWS customers. Requiring minimal setup and configuration versus native tooling, a subset of the addressable market is likely to be of a similar mindset to this commenter on the DataStax blog:

“Cassandra’s tech is superior, as far as I can tell. But we’ll probably be using DynamoDB until there is an equivalent managed host service for Cassandra. Moving to Cassandra is simply too expensive right now.

…

All those are clearly better served by a service like DynamoDB than trying to run their own Cassandra clusters unless they happen to be very proficient in Cassandra administration and want to dedicate precious human resources to administration. That takes a lot of the benefits of “cloud” away from small and mid-sized companies where cost and management are the limiting factors.”

For many, outsourcing the installation, configuration and ongoing management of a data infrastructure is a major attraction, one that easily offsets a reduced featureset. Like Platform-as-a-Service (PaaS) offerings, DynamoDB offers time to market and theoretical cost advantages when required capital expense and resource loading are factored in.

Like the initial wave of PaaS platforms, however, DynamoDB is available only through a single provider. Unlike Amazon’s RDS, which is essentially compatible with MySQL, DynamoDB users will be unable to migrate off of the service seamlessly. The featureset can be replicated using externally available code – via those projects that were originally inspired by DynamoDB, for example – but you cannot at this time download, install and run DynamoDB locally.

It’s true that the practical implications of this lack of availability are uncertain. NetFlix’ Adrian Cockroft, for example, asserts that migration between NoSQL stores is less problematic than between equivalent relational alternatives, because of the lower complexity of the storage, saying “it doesn’t take a year to move between NoSQL, takes a week or so.” It remains true, however, that there are customers that postpone upgrades to newer versions of the same database because of the complexity involved. And that’s without considering the skills question. Given the uncertainty involved, then, it seems fair to conclude that the proprietary nature of DynamoDB and the potential switching costs will be – at least in some contexts – a barrier to entry.

The question for users is then similar to that facing would be adopters of first generation PaaS solutions: is the featureset sufficient to compel the jeopardizing of later substitutability? Amazon clearly believes that it is, its competitors less so. EMC’s Mark Chmarny, additionally, notes that Amazon may be advantaging adoption at the expense of migration in its pricing model.

Competition

DynamoDB clearly has the attention of competitive projects. Basho – the primary authors of Riak – welcomed DynamoDB in this post while pointing out the primary limitation, and DataStax wasted little time spinning up a favorable comparison table. One interesting aside: the Hacker News discussion of the launch mentioned Riak 23 times to Cassandra’s three.

Basho and Datastax are right to be concerned, because the combination of Amazon’s increasingly powerful branding and the managed nature of the product make it formidable competition indeed. The question facing both Amazon and competitors is to what extent substitutability matters within the database space. Proprietary databases have had a role in throttling the adoption of PaaS services like Force.com and Google App Engine in the past, but we have very few market examples of standalone, proprietary Database-as-a-Service (DaaS) offerings from which to forecast. Will DaaS or more properly NoSQL-as-a-Service be amenable to single vendor products or will they advantage, as they have in the PaaS space, standardized platforms that permit vendor choice?

The answer to that is unclear at present, but in the meantime expect Amazon to highlight the ease of adoption and vendors like Basho and DataStax to emphasize the potential difficulties in exiting, while aggressively exploring deeper cloud partnerships.

NoSQL Significance

It’s being argued in some quarters that DynamoDB is the final, necessary validation of the NoSQL market. I do not subscribe to this viewpoint. By our metrics, the relevance of distinctly non-relational datastores has been apparent for some years now. Hadoop’s recent commercial surge alone should have been sufficient to convince even the most skeptical relational orthodoxies that traditional databases will be complemented or in limited circumstances replaced by non-relational alternatives in a growing number of enterprises.

Throughput Reservation

Perhaps the most compelling new feature of Amazon’s new offering isn’t, technically speaking, a feature. Functionally, the product is (yet) another implementation of the ideas in the Dynamo paper; Alex Popescu has comprehensive notes on the feature list. Receiving the most attention aren’t technical capabilities like range queries but rather the concept of provisioned throughput, levels which can be dynamically adjusted up or down.

This type of atomic service level provisioning is both differentiating and compelling for certain customer types. Promising single digit latency at a selected throughput level with zero customer effort required is likely to be attractive for customers that require – or think they require – a particular service level. And by requiring customers to manually determine their required provisioning level, Amazon stands to benefit from customer overprovisioning; customers will feel pain if they’re under-provisioned and react, but conversely may fail to observe that they’re over. Much like mobile carriers, Amazon wins in both scenarios.

Timing

With DynamoDB having been extant in some form since at least 2007, one logical question is: why now? Amazon did not detail their intent with respect to timing when they prebriefed us last week, but their track record demonstrates a willingness to be first to market balanced with an understanding of timing.

In 2006, Amazon launched EC2 and S3, effectively creating the cloud market. This entrance, however, was built in part from the success of the Software-as-a-Service (SaaS) market that preceded it; Salesforce, remember, went public in 2004. With enterprises now acclimated to renting software via the network, the market could be considered primed for similar consumption models oriented around hardware and storage.

Three years later after the debut of EC2 and S3, and one year after MySQL had achieved ubiquity sufficient to realize a billion dollar valuation from Sun [coverage], Amazon launched the first cloud based MySQL-as-a-Service offering [coverage]. That same year, the first year that Hadoop was mainstream enough to justify its own HadoopWorld conference, Amazon launched Elastic MapReduce.

The pattern is clear: Amazon is unafraid to create a market, but attempts to temper the introductions with market readiness. Logic suggests that the same tactic is at work here.

NoSQL has, as a category, crossed the chasm from interesting science project to alternative data persistence mechanism. But while NoSQL tools like Cassandra and Riak are available in managed form via providers like Joyent and Heroku, DynamoDB is, in Popescu’s words: “the first managed NoSQL databases that auto-shards.”

It is also possible that SSD pricing contributed directly to the launch timing, with pricing for the drive type down to levels where the economics of a low cost shared service finally make sense.

SSDs

One underdiscussed aspect to the Dynamo launch is the underlying physical infrastructure, which consists solely of SSDs. This is likely one of the major contributing factors to the performance of the system, and in some cases will be another incentive to use Amazon’s platform as many traditional datacenters will not have equivalent SSD hardware available to them.

The Net

While discussion of the DynamoDB offering will necessarily focus on functional differentiation between it and competitive projects, it is likely that initial adoption and uptake will be primarily a function of attitudes regarding lock-in. For customers that want to run the same NoSQL store on premise and in the cloud, DynamoDB will be a poor fit. Those who are optimizing for convenience and cost predictability, however, may well prefer Amazon’s offering.

Amazon would clearly prefer the latter outcome, but both are likely acceptable. Amazon’s history is built on releasing products early and often, adjusting both offerings and pricing based on adoption and usage.

In any event, this is a notable launch and one that will continue to drive competition on and off the cloud in the months ahead.

Disclosure: Basho is a RedMonk client, while Amazon and DataStax are not.

The cost of delaying my 2012 predictions is that one has already come to pass. Nginx – the web server now powering all of the redmonk.com properties – passed IIS according a January 4 Netcraft release. Because the quantitative data available to us has indidicated surging interest in the alternative web server – the logical result of which was a commercial response – we’ve been expecting something like this. But of course we can’t count this as a prediction any longer because it’s January 13th.

Here instead are a few things that have not yet come to pass, but will, I believe, in the year ahead. These predictions are informed by historical context and built off my research, quantitative data that’s available to me externally or via RedMonk Analytics, and the conversations I’ve had over the past twelve months, both digital and otherwise. They cover a wide range of subjects because we at RedMonk do.

For context, my 2010 predictions graded out as 66% accurate while 2011′s were 82% correct.

With that, the 2012 predictions.

Data & The Last Mile

It is not technically correct to assert that large scale data infrastructure is a solved problem. Decades of innovation remain, as the Cambrian explosion of projects demonstrate. It is nevertheless true that relative to the user interface, data storage and manipulation is a solved problem. Since the original creation of Hadoop in 2006, for example, we have seen multiple user interfaces applied: connectors (e.g. R), standard MapReduce, scripting (e.g. Jaql/Pig), SQL (e.g. Hive), spreadsheets (e.g. BigSheets), client tooling (e.g. Karmasphere). Each has its strengths, none bridges the last mile: putting the power of Big Data in the hands of ordinary users.

Which is perhaps unsurprising; even the mature relational database world uses abstractions of varying levels of complexity to interface with business users. But with data driven decision making on the rise, premiums are being placed on tooling which can expose in sensible fashion data to those without degrees in computer science. Hence, the elevated visibility of startups such as Metamarkets, who excite data scientists with tools like Druid but whose valuation may ultimately depend on its last mile expertise.

At this point in time, whatever my preferred model for data storage and whatever the type, there will be greater than one credible option for a data engine. The same cannot be said for presentation. Which would be less problematic if the market for Big Data talent were not so desperate; outsourcing to shops like Mu Sigma will be an option in some quarters, but comes with its own inefficiences and risks, not to mention per inquiry premiums.

This, then, will be an area of focus in 2012, for both innovation (look for assisted anomaly and correlation identification, a la Google Correlate) and M&A.

Desktop Importance Declines

The most interesting characteristic of the forthcoming Windows 8 release isn’t the technology, which is curious because it’s revolutionary from a Microsoft standpoint. From the support for ARM to the addition of the Windows Store to the ability to author in JavaScript and HTML5, there is much to digest. Instead, the single most defining characteristic of the pending launch is apathy.

Overall inquiries and discussion of the platform demonstrate curiosity but limited interest; the visibility of the once dominant Windows platform is secondary to mobile platforms like Android and iOS.

While this is not a function of any specific or general design failures on the part of Microsoft – indeed, the platform is incorporating important changes while making itself more developer accessible – it is symptomatic of a broader and more difficult to attack problem: the declining role of the desktop.

The desktop is simply not as important as it once was. Mobile usage is eroding the central role PC’s once played; while they are still the dominant form of computing, the trendline is declining and there is no reason to expect it to invert. It’s been suggested that mobile computing in general is additive; that it’s being used to extend the usage of computing to areas where PCs were not employed, and is thus non-competitive. But our data as well as Asymco‘s indicates that, at least in part, mobile usage is coming at the expense of traditional platforms. General search volume data, as we’ve seen, validates this assertion.

There are two implications here. Most obviously, Microsoft’s ability to generate interest in and thus leverage for its flagship operating system is jeopardized. Worldwide developer populations are not necessarily zero sum as skills overlap, but they tend to be rivalrous; an Android or iOS developer is often a lost potential Windows developer – experiments like BlueStacks aside. We can therefore expect Microsoft to have to expend more effort to attract fewer developers to their platform, a negative cycle which becomes cyclical. Second, as the desktop’s primacy abates, we can expect to see greater competition in the marketplace. As enterprises become by necessity more heterogeneous, incorporating Android and iOS devices, the costs of supporting second operating systems drifts towards marginal, which means that forecasts of greater Apple penetration become more probable.

Developer Shortages

It’s become axiomatic that industry hiring is all demand and short supply, and none of our clients expect any relief in the year ahead. Nor will they receive it. Shortages for in demand skillsets will continue over the next twelve months, advantaging entities that are either geographically positioned to leverage markets less competitive than the Valley or with the logistical ability to incorporate remote hires.

That said, we will in 2012 see the first steps towards a more rational market, through a combination of cultural shift and educational model innovation that will increase supply. Regarding the former, it’s no secret that technology has had a profound impact on the erosion of middle class jobs. In Race Against the Machine, MIT Professors Andrew McAfee and Erik Brynjolfsson document the role that rapid innovation has had on jobs:

Digital technologies change rapidly, but organizations and skills aren’t keeping pace. As a result, millions of people are being left behind. Their income and jobs are being destroyed, leaving them worse off in absolute purchasing power than before the digital revolution.

Even skill industries are not immune. From John Markoff’s New York Times piece, “Armies of Expensive Lawyers, Replaced by Cheaper Software“:

“From a legal staffing viewpoint, it means that a lot of people who used to be allocated to conduct document review are no longer able to be billed out,” said Bill Herr, who as a lawyer at a major chemical company used to muster auditoriums of lawyers to read documents for weeks on end. “People get bored, people get headaches. Computers don’t.”

While Brynjolfsson and McAfee are ultimately optimistic about the prospects of technical progress as they relate to employment, the outcome is far from certain.

What is becoming clear, however, is that unemployment rates that have been north of 8% in the US since February of 2009 are driving people into industries that are desperate for help. For some, this means oil & gas employment in traditionally underpopulated environments like North Dakota. For others, however, technology – long an enemy – is becoming a refuge.

We’re seeing a spike in inquiries about transitioning to technology careers. Lawyers, management consultants, teachers and others are seeking – and often finding – homes for themselves within the technology sector. Some are self-taught or trained on the job, others merely apply existing skills in new contexts, but both represent a potential cultural shift. Which begs the question: could technology be the next major middle class employment sector?

For that to happen, the education system needs to improve, because even an industry which has been one of the few economic bright spots of the last decade can only absorb so many unskilled workers without slowing. This is the real significance of applications like Code Academy or programs like Harvard’s free CSCI E-52, MITx or Stanford Engineering Everywhere: they are one potential solution to the perpetual shortage of talent. For all of the limitations of distance learning, the scale means that some subset of motivated students will become productive developers, and by extension, contributors to the larger economy.

This is a long term process, so obvious progress within 2012 will be minimal, and talent shortages will continue. But we will in the next twelve months begin to see distance trained students hired at scale, and this will be one of the first steps towards lower talent costs as well as, possibly, the restoration of middle class employment opportunities.

Monitoring as a Service

We are not oriented around category definitions at RedMonk; we prefer market driven names to those conceived and marketed by the analyst industry. That said, it seems clear that the time of Monitoring-as-a-Service (MaaS) is at hand. New Relic’s growth led to a $15M round in November, Boundary took $4M a year ago this month, Monktoberfest speaker Theo Schlossnagle’s Circonus has been in market for over a year, and virtually every vendor that we speak with today is adding monitoring and management facilities, from 10gen’s MMS to Cloudera’s Cloudera Manager.

The proliferation of these services is a direct response to the increasingly heterogeneous nature of application architecture and the reality that the substrate is frequently network based, rather than local. Given accelerating rather than declining consumption of network resources, we predict a strong increase in interest and adoption of MaaS tools. Much as I don’t care for the term itself.

Intelligent usage of generated telemetry – which we’ll come back to – will further cement adoption, delivering previously unseen value.

Open Source and the Paradox of Choice

Gartner in March of last year asserted that open source had hit a tipping point, saying:

“Mainstream adopters of IT solutions across a widening array of market segments are rapidly gaining confidence in the use of open source software.”

We concur, although we would argue that the tipping point actually occured ten years or more prior. The Apache web server and MySQL were originally written in 1995. In 1999, we saw the public offering of Red Hat and the creation by IBM – as mainstream a technology brand as there is in the enterprise – of the Linux Technology Center. Firefox was first released in 2003. None of these reached their relative levels of popularity in the past twelve months; they have instead been the de facto infrastructure for the better part of the last decade.

Regardless of when one asserts that open source crossed the chasm, however, it remains that it is a model whose popularity is increasing over time. As understanding of the benefits increases and concerns about the risks abate, more organizations are not only consuming open source but contributing to it. Evidence suggests, in fact, that perceptions of the value of software are in decline – we’ll come back to that too, and that the end result of this is that more proprietary code is being released as open source software.

Widely perceived as a net benefit, however, the influx of new projects does present problems for would be adopters. Specifically, the paradox of choice implies that developers will increasingly be forced to select from a growing sea of projects which may or may not be suitable for their needs. And while the nature of open source guarantees developers the ability to apply this code to their projects without restriction or commercial engagement, this is a process with a limited ability to scale. Consider the NoSQL space, as an example. Presuming for the sake of argument that the developers in question understand the different categories of database – key value stores, document databases, columnar databases, MapReduce engines, graph databases and so on – well enough to understand their high level needs, there are at least two and sometimes as many as half a dozen credible options to consider.

This paradox of choice, or too much of a good thing, will become more problematic over time rather than less as contributions will continue to rise. The net impact is likely to be increased commercial opportunities around selection, and therefore attention to vendors like Black Duck, Open Logic, Palamida and Sonatype.

PaaS: The New Standard

It has been evident for some time that runtime fragmentation – an aggressive diversification of programming languages and frameworks, specifically – will change the development landscape. The market failure of the first generation PaaS providers, in fact, was primarily a function of their over-prescriptive natures. The benefits to outsourcing management and scale were obsoleted by the constraints; Java shops were never likely to rewrite their application stack in Python or Ruby strictly to benefit from a platform. Which is why virtually every relevant PaaS provider today offers a choice of runtimes, so as to maximize their addressable market.

But in a fragmented world, what might emerge as a standard? From a developers’ perspective, the standard is most often the framework they’re deploying to, whether that’s Django, Node.js, Lift, Play, Rails, Spring, the Zend Framework or another. From a vendor perspective, however, the new standard is likely to be one level of abstraction up from individual language frameworks: the platform itself. Certainly this is VMware’s opinion, as they are in Maritz’ words trying to construct “the 21st-century equivalent of Linux” – i.e. the substrate that everything else is built on top of.

In 2012, this will become more apparent. PaaS platforms will emerge as the new standard from a runtime and deployment perspective, the middleware target for a new generation of application architectures.

Service Proliferation

With the inevitable adoption of multiple third party services – varying cloud resources, multiple, possibly overlapping, management and monitoring services and so on – will come challenges in making sense of the whole. Overall, instrumentation and visibility on a per service level is improved, but aggregating these views into a cohesive picture of overall architectural health and performance is likely to be highly problematic. Not least because the services themselves may present conflicting information and data. Google Analytics and New Relic, for example, are frequently at odds over load times and other delivery related performance metrics. Introduce in to that mix services like Boundary or CloudWatch and the picture becomes that much more complex. Connecting their data back to underlying log management and monitoring solutions such as 10gen’s MMS or Splunk is more complicated still.

The challenges of service intregration will create commercial opportunities for aggregating services which consume individual performance streams, normalize it and present customers with a consolidated single picture of their network performance. Commercial solutions will not fully deliver on this vision in 2012, but we will see progress and announcements in this direction.

Telemetry Usage

Five years ago, we began publicly discussing revenue models based around what we termed telemetry, or product generated datastreams. The context was providing open source commercial vendors with a viable economic model that better aligned customer and vendor needs, but the approach is by no means limited to that category: Software-as-a-Service vendors, as an example, are well positioned to leverage the data because they maintain the infrastructure. In 2011, we finally began seeing vendors besides Spiceworks take the first steps towards incorporating data based revenue models. For products like Sonatype Insight [coverage], data is not a byproduct, but the product.

In 2012, this trend will accelerate as necessary monitoring capabilities are added to product portfolios and industry understanding and acceptance of the model overcomes conservative privacy concerns. Many more vendors will begin to realize that like New Relic, which observed a decline in commercial application server usage, their accumulated data is full of insights on customer behaviors and wider market trends both.

Value of Software Will Continue to Decline

Capital markets have not, traditionally, been overly fond of software firms, perhaps because comparatively few of them eclipse annual revenue marks of a billion dollars – less than twenty, by Forbes‘ count. Microsoft’s share price has languished for over a decade in spite of having not one but two licenses to print money. The mean age of the PwC’s Top 20 software firms by revenue is 47 years; a fact which cannot be encouraging to startups.

Higher valuations instead are being awarded to entities that employ software to some end, rather than attempting to realize revenue from it directly. Startups today realize this, and the value of software in their models has commensurately been adjusted downward. Tom Preston-Werner, for example, describes the GitHub philosophy as “open source (almost) everything.” Facebook, LinkedIn, Rackspace, Twitter and others exhibit a similar lack of protectiveness regarding their software assets, all having open sourced core components of their software infrastructure that would have been even five years ago fiercely guarded.

This is becoming the expectation rather than the exception because it is nothing more or less than an intelligent business strategy. Businesses can and will keep private assets they believe represent competitive differentiation, but it will be increasingly apparent that less and less software is actually differentiating. As a result, 2012 will see even less emphasis on the value of software and more on what the software can be used to achieve.

Bonus: Facebook’s Most Important Feature

In 2012 will be Timeline. Mark it down.

Disclosure: Black Duck, Cloudera, GitHub, IBM, Microsoft, Sonatype and VMware are RedMonk customers, while 10gen, Boundary, Circonus, Facebook, Open Logic, Palamida, and New Relic are not.

This is the concluding half of the exercise in which I review my predictions for the calendar year just ended. If you’re looking for the original 2011 predictions, those are here. Part 1, meanwhile, can be found here. With that, on to the predictions.

Hardware

Workstations Will Make a Comeback

This prediction is not supported by data nor even anecdotal evidence. Even in the small sample of my contacts, migrations away from tower-style hardware profiles towards laptops accelerated. Wider market developments seem to confirm this; Apple may be retiring their existing workstation platform, the Mac Pro.

This is a (big) miss.

ARM Will Emerge as a Server Player

Whether they will ultimately emerge as a credible mainstream alternative remains to be seen, but ARM is indeed emerging as a server player. Though virtually all of them discuss it privately, HP (via Calexda) this year became the first major systems player to publicly detail plans for ARM servers – perhaps banking on the fact that the upcoming A15 processor is more server friendly,

Intel is predictably skeptical of ARM’s viability in its core markets, with CEO Paul Otellini bluntly dismissive: “It ain’t gonna work.” And while it certainly hasn’t proven to work thus far, and there are real architectural and software issues to address, the power profile continues to pique the interest of server manufacturers and customers alike. Even marginal power savings mean real dollars at scale.

I count this as a hit.

Tablets are a Real Market

From the New York Times, late January, 2011:

“The iPad, introduced in April — is on track to deliver $15 billion to $20 billion in revenue in its first full year of sales, estimates A. M. Sacconaghi, an analyst at Sanford C. Bernstein. At that size, if the iPad were a stand-alone company, it would rank within the top third of the Fortune 500.”
Steve Lohr, “The Power of the Platform at Apple“

Any questions? It’s true that to date the tablet market is more accurately characterized as an iPad market, but irrespective of the particular vendor dynamics in the space, the hardware form factor appears here to stay.

I count this as a hit.

Mobile

Challenges of Native Development Will Drive Interest in HTML5 and Hybrid Approaches

In August, we did a quick pass at some developer metrics and confirmed what our qualitative research had already indicated: that interest in PhoneGap was booming. Here’s a chart of StackOverflow traction, for example.

Interest was booming enough, in fact, that Adobe acquired the talent behind PhoneGap, the code of which was submitted to Apache. This interest was unsurprising in light of the frustrations experienced by enterprises and developers alike, who are collectively slowed by the process of building an application on one platform and then porting to a second. Developers are frustrated enough, in fact, that they are in certain cases actively stalling development. While opinions differ on individual platform trajectories, most would agree that the status quo is unlikely to remain static. Meaning that at least some native development effort is likely to be wasted.

Couple that with improving mobile browser capabilities, and interest in HTML5 and hybrid approaches is likely to remain strong, in spite of inherent advantages to native development like discovery.

I count this as a hit.

NoSQL

The NoSQL Marketplace Will Experience Consolidation

The merger of CouchOne and Membase into CouchBase in February provided some evidence that the long anticipated wave of consolidation in this space was beginning, but the balance of the year provided little evidence to support this aside from the acceleration of a few individual players such as MongoDB [coverage]. I remain convinced that the marketplace will be unable to sustain the current volume of would be commercial entities, but from our conversations with both those in a position to potentially impact consolidation and those interested in partnering with various NoSQL players, it is clear that consolidation will depend on clearer winners and losers to proceed. This should occur in 2012.

I’ll count this as a push in light of the CouchBase merger which subtracted one player but otherwise saw very few exits.

NoSQL Will Look More Like Pro-SQL

The implicit rejection of the Structured Query Language in the NoSQL term is ironic in light of the fact that a variety of projects are now adding similar features. Continuing in the proud tradition of Hive and Pig, which provide query language interfaces to Hadoop, DataStax announced CQL in June while CouchBase and SQLite announced UnQL in July [coverage].

Whether we’ll see a unified interface or a variety of engine-specific implementations as Alex Popescu would prefer remains to be seen, but query languages will be coming to the majority of NoSQL stores one way or another.

I count this as a hit.

Open Source

Open Source of Non-Strategic Infrastructure Assets Will Increase

From Twitter open sourcing the Storm assets it acquired via the BackType transaction to the New York Stock Exchange’s donation of OpenMAMA to the Linux Foundation, it is increasingly clear even to traditional parties that the release of non-strategic code as open source has multiple benefits. GitHub’s Tom Preston-Werner’s list of same is difficult to improve upon:

• “Open sourcing code is great advertising for you and your company.”
• “If your code is popular enough…you will have created a force multiplier that helps you get more work done faster and cheaper. “
• “When you open source useful code, you attract talent.”
• “If you’re hiring, the best technical interview possible is the one you don’t have to do because the candidate is already kicking ass on one of your open source projects.”
• “Dedication to open source code is an amazingly effective way to retain that talent.”
• “[Assuming code will be open sourced] leads to effortless modularization.”
• “By getting code out in the public we can drastically reduce duplication of effort.”
• “It’s the right thing to do.”

It may or may not be beneficial to open source core strategic assets, as VMware did with Cloud Foundry, but it is increasingly hard to justify protecting those that are purely tactical in nature. The benefits in many if not most cases will outweigh the costs, which is why we’re seeing an increase in contributions to open source projects.

The data from the annual Eclipse surveys is one example of this. If we examine the percentage of organizations that contribute back to open source versus those that do not from 2007 to 2011, it is clear that comfort levels with open source generally are rising.

I count this as a hit.

Forking: How Development Gets Done

The benefits to distributed version control and the beneficial forking model it encourages have been sufficient to convince even large projects such as Eclipse: the majority (44.2%) of its projects have migrated from CVS or Subversion to Git. The popularity of this model is also on display in the graph above, which depicts the relative performance among commit volume of four major forges. GitHub is the youngest of the forges and yet commands a significant majority of the overall commits observed by Black Duck.

While it is difficult to separate the success of Git from GitHub, it is not necessary for this exercise, because both by design encourage forking as a developmental best practice. Forking is increasingly how development is conducted.

I count this as a hit.

Ubuntu is the New SUSE

In March of last year, I explored a set of metrics evaluating the relative performance of SUSE with developers, from jobs data to community traction. None of the metrics favored SUSE at the time, and the observed trends that favored Ubuntu persist. Consider, for example, the job trends:

As pointed out at the time, despite the relative outperformance of SUSE relative to Ubuntu, the respective trajectories were problematic for the former. Revisiting the data today, we can see that the trendlines produced a logical outcome.

Commercially, Ubuntu has continued its emergence as a server player. While initial claims that it was the primary operating system behind HP’s public cloud may have been [overstated],(http://arstechnica.com/business/news/2011/10/ubuntu-will-power-hps-new-cloud-service.ars) in HP’s words “that they are the first one in our current private beta.” This is an announcement that would not have been possible prior to 2011.

I count this as a hit, though it will be interesting to see if Mint can become the new Ubuntu, in turn.

Programming Languages

JavaScript is resurgent

In April, we were fortunate to be able to analyze data from Black Duck regarding open source project commits by programming language. When we compared a 2011 snapshot to the volume of all time commits, the rise of JavaScript was apparent.

While dynamic languages were ascendant across the board, JavaScript outperformed even high growth languages like Python and Ruby.

As “The Rise and Rise of JavaScript” notes, the prospect of using the same language on browser and server is compelling, but it doesn’t stop there:

JavaScript’s serialization form, JSON, is becoming ubiquitous as a lighter-weight alternative to XML for streaming structured data, and NoSQL databases like mongo are happily using JSON and JavaScript in the database as a query language. This means, for the first time, you can have the same JavaScript function in the browser, on the server and in the database.

JavaScript may only be the most popular language within forward communities like GitHub, but its future in the wider world is bright, Dash notwithstanding.

I count this as a hit.

Bonus Prediction

Dropbox will become an attractive acquisition target

One the one hand, I got this wrong, because Steve Jobs apparently first attempted to acquire Dropbox in 2009.

“Jobs presciently saw this sapling as a strategic asset for Apple. Houston cut Jobs’ pitch short: He was determined to build a big company, he said, and wasn’t selling, no matter the status of the bidder (Houston considered Jobs his hero) or the prospects of a nine-digit price (he and Ferdowsi drove to the meeting in a Zipcar Prius).

Jobs smiled warmly as he told them he was going after their market.”

It was in 2011, however, that Dropbox – a firm that had previously raised a mere $7.2M – justified that decision, with a $4B valuation.

So I’ll go ahead and call this one a push.

The Final Tally

Scoring the 2011 predictions, then, we get 14 of 17 correct against two push and one miss. Generally, an 82% success rate in forecasting means that the game is rigged; 51% is enough to make a substantial profit in public markets. And in reviewing the predictions, it’s certainly true that a few – the continuing developer shortage, for instance – were likely obvious enough to not merit the “prediction” label.

But we do believe, as our website suggests, in William Gibson’s claim that the future is already here, it’s just unevenly distributed. Conclusions that are obvious to us are not to many traditional enterprise technology buyers, so in that sense this game is, in fact, rigged. We’ve predicted the rise of dynamic languages, Node.js, NoSQL, REST, Software-as-a-Service and such then not because we’re Nostradamus, but because we know who to listen to.

If you want to hear what that audience is telling us about 2012, then, stick around. Those are due next week.

Predicting is an easier business than it once was. True, technology is hysterically accelerating rates of change and disruption, but that’s only relevant if the substance of your predictions matters. Which all too often, these days, it doesn’t. Analysts and pundits are able to prognosticate with relative impunity; who has the time to go back and check their accuracy? Pageview driven models, in fact, reward wilder predictions because the error cost is, generally, approaching zero. Unless you predicted, say, that Linux would be killed off by Windows NT, nobody will remember later.

I find value in reviewing my annual predictions, however. If they prove correct, that’s useful. If they were not, understanding the reasons why is important to adjusting our models moving forward.

Because I made the mistake of making better than a dozen predictions last year, this year’s review will be delivered in two parts. Part 1, below, will cover my predictions for browsers, the cloud, data, developers and programming language frameworks. Part 2, covering predictions within hardware, mobile, NoSQL, open source and programming languages, will hit tomorrow.

If you’d prefer to read last year’s first, they can be found here.

Browsers

Firefox Will Cede First Place to Chrome, But Not Without a Fight

According to RedMonk Analytics, whose data reflects our developer-heavy audience, Firefox was able to hold off Chrome for two quarters. On the first of June, Firefox held a 32.58 share of our audience to Chrome’s 32.08. By the second, Firefox was in second place and would remain there for the balance of the year, widening the gap in the process. At present, our browser metrics peg Chrome at 36.38 with Firefox a distant second at 25.48.

I feel safe counting this one as a hit.

Cloud

PaaS Adoption Will Begin to Show Traction, With Little Impact on IaaS Traction

The first Platform-as-a-Service providers essentially asked developers to trade choice for development speed. Like Ruby on Rails – itself the basis for multiple first generation PaaS platforms – PaaS was built for those that would embrace constraints. But PaaS platforms never saw the type of growth that Rails experienced, in part because of the further loss of control that the cloud represents. It’s one thing to have a web framework like Rails dictate the way that you build web applications; having PaaS platforms also choose the operating system, database, version control systems and more was too much.

Which is why second and third generation PaaS providers have furiously removed barriers to entry, adding additional runtimes, open sourcing the underlying platform and allowing you to pick your provider. Which, in turn, is why adoption of PaaS is accelerating. VMware CEO Paul Maritz calls PaaS “the 21st-century equivalent of Linux,” which explains not only why they feel compelled to compete in the space, but also why Red Hat might.

Virtually every vendor in this space is reporting growth similar to the Hacker News trajectories for Cloud Foundry and Openshift (below).

In spite of the growth of PaaS, however, none of the metrics we track reflect any decline in usage of general infrastructure platforms. Quite the contrary, in fact.

I count this as a hit.

Data

Firms Will Increasingly Seek to Leverage the Data They Generate

Turning data into revenue has been one of the core themes of the past year, as well as the focus of my talk at the Open Source Business Conference in May. We’ve long held SpiceWorks up as a model of monetizing data, and as customers adjust to the reality that they’re already sharing data and vendors cease to regard it as a third rail issue, we’re seeing more businesses embrace data based revenue streams, as with Sonatype Insight. From 10gen to Black Duck, vendors are increasingly positioning themselves to be purveyors of data as much as software. Data is no longer the byproduct, but a product itself.

I count this as a hit.

Hadoop Will Become the MySQL of Big Data

EMC, HP, IBM, NetApp and even Oracle all have Hadoop – or in EMC’s case, MapReduce – plays in market. Microsoft actually deprecated its own Dryad initiative in favor of the Apache project. Players from AsterData to CouchBase to EnterpriseDB to MarkLogic to Tableau to Vertica have purpose built Hadoop connectors. The commerical distribution space, once essentially owned solely by Cloudera, has expanded to multiple third parties with varying points of differentiation.

Hadoop interest elsewhere, meanwhile, has not slowed.

Need I say more about the growing ubiquity of Hadoop? I count this as a hit.

Developers

Talent Shortages Will Continue

Granted, predicting a shortage of qualified development talent will be seen in some quarters as controversial as predicting that the sun will rise in the east. But part of this is context: certainly in January, the economic direction was less than certain. And in spite of an unemployment rate that has hovered just south of 10% for the better part of the last calendar year, hiring continues to be an issue for the majority of our clients. To the extent that several are spinning up offices solely for purposes of recruitment. This is not surprising, given the historical growth in employee headcounts (see above) that has, to date, been relatively resistant to the global economic crises.

Demand varies by skillset, as might be predicted, but 2011 remained – by our metrics – a tight market. Other market watchers support this assertion.

“Hiring talent in Silicon Valley is the toughest since the last bubble and investors are starting to openly wonder how this one will end.”
Steve Blank

“There is a war for talent, particularly developer talent, going on. Not just in Silicon Valley but also in NYC and many other places around the country.

Companies, small and large, are resorting to all sorts of creative ideas to recruit. Free lunches, free yoga, pushing code day one, cool schwag, options, RSUs, pretty much whatever it takes.”
Fred Wilson

While we don’t have good data then on market specific hiring (Bureau of Labor data is not fine grained enough), the evidence available to us seems to support the contention that shortages of tech talent remain.

I count this as a hit.

Frameworks

Node.js Will Continue its Growth Trajectory

October was a rough month for Node.js, with posts like Node.js is Cancer and node.js Is VB6 – Does node.js Suck? following the tradition of March reddit discussions like Is NodeJS Wrong? The Trough of Disillusionment, it seemed, had arrived well ahead of schedule.

Except that interest metrics showed no commensurate decline. Node took – again – three of the Top 5 spots in inbound search queries within RedMonk Analytics. Which is unsurprising against the backdrop of Google’s Insights for Search numbers.

Over on GitHub, meanwhile, which itself has achieved dramatic growth, Node.js is the second most popular watched repository, ahead of Rails, jQuery, HTML5-Boilerplate, and Homebrew. Microsoft clearly perceives this growth, because it has worked with Joyent to create a stable build of Node for Windows which in turn led to an SDK for Azure.

All of which means nothing except that Node’s growth trajectory continues.

I count this as a hit.

Part 2, tomorrow.

“From the beginning of time two forces have vied for influence over us. One is bottoms-up, decentralized, and emergent. The other is top-down, centralized, and directed.

The first force catalyzes change and divergence, while the second tends toward order and convergence. The first gives birth to new ideas, and the second enshrines them.”
- Adam Ludwin, From LOLcats To Occupy Wall Street, Everything Is Happening From The Bottom Up Now

Traditionally, industry analyst firms have been oriented around top down adoption patterns. CIOs and other IT decision makers comprise both the research subjects and purchasing audience for the majority of firms in this industry, large and small. Which was logical given traditional procurement patterns. When hardware, software and services are available only at high prices, command and control is an appropriate management structure. Attempting to scale the decision making process for big ticket items across a large body of middle managers is not likely to yield acceptable outcomes.

An approach that makes sense in one context, however, may be misapplied in another.

The technology purchasing landscape today looks very different than it did even five years ago. Where once CIOs might reasonably expect to have the clearest understanding of what technologies are leveraged within their own organizations, today they are, as Billy Marshall put it, “the last to know.” This pattern manifests itself every day within the majority of businesses. Not because CIOs are failing, but because of trends that have fundamentally and likely permanently disrupted their ability to centralize the technology adoption process.

The four trends we see as most important in driving this are arranged here in rough chronological order.

Open Source

In the late nineties, startups and enterprises alike were effectively beholden to commercial suppliers for the majority of their software needs. Because each piece of the requisite software infrastructure had to be licensed, the capital expenses associated with new initiatives was high. This represented a barrier to entry, and thus a brake on innovation.

With the popularization of open source software, developers from enterprises and startups alike were able to operate independently. For the first time, the actual software practitioners were free to choose their own software rather than having it selected for them and subsequently imposed upon them by upper levels of management. Even in situations where the ultimate production infrastructure targets remained commercially licensed software, open source software like Linux and MySQL allowed for prototyping and rapid development without the attendant costs, both financial and in procurement latency.

This was the first major shift affecting procurement, and perhaps the most profound. None of the infrastructure we take for granted today – Linux, Apache, MySQL, PHP, etc – were originally adopted from the top down. Their adoption was, instead, a fait accompli. CIOs – the last to know – gradually became aware that increasingly significant portions of their infrastructure, unbeknownst to them, were running on free and open source software. The inevitable demand for production support options for this software is what fueled, in time, the valuations of MySQL, Red Hat and others.

Bring Your Own Device

In October, Apple CEO Tim Cook asserted that 92% of the Fortune 500 were “testing or deploying iPad in the course of less than 18 months,” which may help explain why the iPad revenue stream by itself would place within the top third of that group. The interesting thing about this is that the majority of businesses appear uncertain about precisely why they’re deploying tablets: “Most participants, 51 percent, indicated that they did not have a clearly articulated strategy.”

The answer, in most cases, is that there isn’t one. iPad adoption, much like the penetration of iPhones and Android handsets is being driven by users who simply want the device. Faced with a choice between users – chief executive officers among them – who will employ their own devices for work purposes with or without the permission of IT, many businesses are compelled to support the platforms even without concrete business justifications for them.

The consumerization of the enterprise is decentralizing the process of technology selection, but its importance may lie rather in design. Like all products, technology is designed and built to be sold to a specific buyer. For enterprise products, historically, the actual user has been a secondary concern; the buyer – typically centralized IT – was the priority. Consumer technology companies like Apple, however, are designed for a user. What they give up in IT friendly features they more than make up for in usability and the ability to delight.

The Bring Your Own Device trend, therefore, may well improve user productivity by driving devices designed to be used rather than managed into organizations, from the bottom up.

Software as a Service

Software as a Service is a classic case study in timing with respect to market acceptance. Not many remember today that the model actually failed the first time around, when its practitioners were known as Application Service Providers. Pyschologically, few enterprises were prepared for either the idea of renting software or externalizing critical data like that stored in customer relationship management systems. By the midpoint in the last decade, however, these concepts were sufficiently commonplace to see Salesforce.com a publicly traded company with a valuation north of a billion dollars.

Consumer markets, meanwhile, had adapted much more quickly. Hotmail debuted in 1996, Yahoo Mail the year after and Gmail dropped in 2004.

Some of those same consumer services were pressed into service by enterprise workers, in fact; it was once common for Exchange users to forward all of their email to Gmail due to the disparity in storage limits between typical Exchange implementations and Google’s webmail product.

This pattern has played out repeatedly over the years, from webmail to CRM to project management software to website hosting to online helpdesks. All were adopted from the bottom up. By making applications available to anyone with a browser, often at low or no cost, SaaS has surged up through the ranks of enterprises. The inexorable nature of the model is reflected by the growth of providers large (Salesforce.com) and small (37signals).

Cloud

The single most important feature of the cloud has nothing, or at least very little, to do with technology. It is, rather, the pay as you go economic model. As Flip Kromer puts it, “EC2 means anyone with a $10 bill can rent a 10-machine cluster with 1TB of distributed storage for 8 hours.”

What this means in practical terms is that for the first time, hardware procurement is democratized. From an accessibility and availability standpoint, cloud is the hardware equivalent of open source software. Where open source allowed developers to bypass traditional procurement channels by making quality infrastructure and development software freely available, so does the cloud allow the growing class of devops technologists to leave the world of high latency hardware procurement – where same day server provisioning is a feature – behind. Armed with nothing more than a credit card, instances can be spun up and ready for use in ninety seconds.

Cloud is the final piece of the bottom up puzzle. Open source software and to a lesser extent SaaS allowed for the decentralization of enterprise technology development, but at some point hardware would become necessary which was the insertion point for IT. With public clouds, it is possible for the first time to entirely bypass the traditional gatekeepers.

The Net

It should be evident that traditional procurement and purchasing is not dead, just increasingly bypassed by a more efficient process. Also, that a great many enterprises continue to function largely as they always have: top down. More important than the question of whether this model is sustainable in the face of the trends above is whether it should be.

Before lamenting the fact that the above forces are disrupting and destabilizing your enterprise IT, consider that that may be a net gain. If the primary drivers of BYOD, Cloud, Open Source, and SaaS include ease of use, lower costs, frictionless availability, and speed of provisioning, are these trends worth opposing? Particularly since efforts to do so will, in all probability, fail?

Or are they instead assets to be strategically leveraged? There is little debate that businesses that move the most quickly have a competitive advantage. It’s not clear how businesses that prohibit the same tools that enable this will benefit.

Either way, bottom up adoption is here to stay: use it or lose.

“After six months of development, Grit had become complete enough to power GitHub during our public launch of the site and we were faced with an interesting question:

Should we open source Grit or keep it proprietary?

…After a small amount of debate we decided to open source Grit. I don’t recall the specifics of the conversation but that decision nearly four years ago has led to what I think is one of our most important core values: open source (almost) everything.”
- Tom Preston-Werner, “Open Source (Almost) Everything

It should be clear at this point that commercial valuations of software assets by both practitioners and public markets is in decline. This model of ours, once controversial, is now self-evident. We examined the public markets aspect in detail in May; little has changed since.

Of PwC’s Global Software Top 20 – the top twenty firms globally as measured by software based revenue – the youngest is 22 years old, and the median age of the top ten is 35 years. It has been over twenty years, in other words, since the software industry produced a business to rank in the Top 20.

Here is a chart of the market capitalizations of PwC’s top five vendors.

If we remove the artificially narrow lens of revenue from software sales, however, the picture looks very different.

The technology industry has produced entities that would place within the top five; they just aren’t in the business of selling software.

Firms that make money with software rather than from software are outperforming their counterparts in recent years; compare the respective valuations of Google, Red Hat and VMware.

Generationally, attitudes towards software are shifting, as evidenced by Werner’s comment above, and the release as open source of assets like Cassandra, FlockDB, Hadoop, Hip Hop, Hive, Jekyll, Nginx, Pig, Resque, Storm, Thrift and so on. Entrepreneurs and public markets alike are turning their attention away from software sales and towards data oriented revenue models in search of outsized returns. There is and will continue to be an enormous market for software, but real growth is increasingly coming from areas other than software sales and service. This is true even for those currently in the business of selling code; for Sonatype and many other startups, data is increasingly the product.

As obvious as this is to most industry participants, however, the Age of Data remains hobbled by its lack of a free market.

In the early days of commercial open source consumption, the exploding number of so-called vanity licenses – those that were non-standard and vendor specific – and a general lack of understanding of their legal implications inhibited the market for open source, relegating it in many settings to unacknowledged, behind the scenes usage. Gradually, however, license proliferation was curbed and licenses generally began to coalesce around permissive (Apache, BSD, MIT, etc) or file (EPL, MPL, etc) / project (AGPL, GPL, etc) style reciprocal licenses. The net impact of which was increased adoption, because the previous uncertainty necessarily implied risk which in turn throttled adoption.

Which is essentially where the data market is today. Everybody understands that data has value; there is little consensus on how, where and via what mechanisms it should be distributed, licensed and sold. Startups like Buzzdata, Datamarket, Factual and Infochimps cannot by themselves make a market, and with rare exceptions like Microsoft with the data section of its Windows Azure Marketplace, large providers tend to be heavily risk averse with respect to the liabilities posed by data.

Absent a market with well understood licensing and distribution mechanisms, each data negotiation – whether the subject is attribution, exclusivity, license, price or all of the above – is a one off. Without substantial financial incentives, such as the potential returns IBM might see from its vertical Watson applications, few have the patience or resources to pursue datasets individually. We’ve experienced this firsthand, in fact; as we’ve looked for data sources to incorporate into RedMonk Analytics, conversations around licensing have been very uneven. After gaining verbal approval from one content provider for our proposed usage of their API, this was the message we received in response to our request for an actual license:

The short answer is… we are not exactly sure what you are planning to do and whether or not it would violate our terms of service, so the best answer I can give you is that if you are not sure, you’re best off consulting with your own attorney.

As a rule, we don’t have the staffing resources internally (or the legal resources, given the high cost of lawyers!) to make one-off contracts or licenses or exceptions to the generic terms of service.

Hope this helps!

Given the current state of data licensing, it’s impossible to blame them for taking this position, even as it prematurely terminated what could have been a mutually beneficial relationship. Why dedicate legal resources to license review when the projected return on the asset is uncertain? Nor would a commercial negotiation be any more straightforward.

Market efficiency is a function of volume; the more participants, the better an understanding we have of an asset’s worth. With limited mainstream market participation in the business of data, opinions of the value of a given asset tend to be asymmetrical. Put more simply, when we ask what a given dataset is worth, the only correct answer at present is: we just don’t know.

All of which helps explain why Infochimps and others startups targeting the data marketplace opportunity are not as visible as their significance suggests they should be.

For all of the current inefficiency in data procurement, however, it is a temporary condition. Apart from the fact that history tells us risk assessment and licensing are solvable problems, the financial incentives are sufficient to guarantee progress if not complete solutions. Consider the case of Watson. IBM has historically avoided data collection due to legal concerns, but imagine the liabilities of the first diagnostic miscalculation. What will happen, in other words, when Watson commits the healthcare equivalent of “What is Toronto?” while assisting in the diagnosis of a patient? For IBM, the answer is clearly that the potential rewards more than offset the theoretical risk. Enough so, at least, to justify massive investments in development and marketing.

Life may be marginally easier for those that capture their own data, but this will by no means diminish the appetite for more. Google generates sufficient data to be able to predict the flu better than the CDC, but still felt obligated to license Twitter’s data. A relationship that ended, notably, because of a license termination. No matter how valuable an internal dataset might be, it will be more valuable still when recombined, remixed and correlated against complementary external data; see, for example, FlightCaster.

With respect to opening the throttle for data marketplaces, escalating demand virtually guarantees the supply. The real questions are who will play a meaningful role in reducing the friction, and when. Because the attendant opportunities are large indeed.

Disclosure: IBM, Microsoft, Red Hat and VMware are customers. Datamarkets, Facebook, Factual, Infochimps, Google and Twitter are not.

In 1999, Napster was unleashed upon the world. A year later they were sued by Metallica and Dr. Dre. A year after that the service peaked with 26.4 million users. A year after that the company filed for Chapter 7 to liquidate its assets.

While the record industry would have you believe that Napster’s meteoric rise was driven exclusively by thieves, the reality was that it was a desire for a product that record companies would not provide. Napster offered music available for dowload without draconian restrictions as well as the ability to download individual tracks rather than an entire album. Services that the record industry reluctantly agreed to years later. Years that the record industry also spent fighting a multi-front scorched earth legal battle against users of file sharing services that sprang up in the wake of the fall of Napster. When they finally did break down and sell music online, they were compelled to work with a much stronger player than Napster ever was.

But what if the music industry had been a rational actor and made the decision not to fight the tide, seeking agreements with Napster similar to the ones they employ today with Amazon, Apple and eMusic? What if they had recognized in those 26 million Napster users not thieves but potential customers and given them what they implicitly asking for: a more convenient way to obtain music?

The question is important because it’s essentially the same question facing enterprise IT today.

Napster made music available to anyone with an internet connection. For decades, enterprises have endured provisioning delays measured in months. Today, as Flip Kromer says, “EC2 means anyone with a $10 bill can rent a 10-machine cluster with 1TB of distributed storage for 8 hours.” It’s the end of procurement as we’ve known it.

Enterprise IT faces the same decision that the record industry once did: fight the tide or get out in front of it. Even setting the public relations damage aside, the returns of the former strategy for the record industry have been unimpressive. Given that developers have an increasing portfolio of accessible open source software and cloud services available to them, it’s unlikely that an enterprise crackdown on so-called shadow IT will be materially more effective. And then there’s question of whether throttling the constituency within your business that wants to move fastest is generally a good idea.

Why not enable them, then? Instead of firewalling the services Shadow IT wants, provide them centrally. Turn the tools that you are wasting your time fighting into an enticement to come out of the shadows. You’ll have better, if still imperfect, visibility into consumption and usage patterns as well as shorter development cycles. What’s not to like?

The RIAA missed their Napster opportunity. You don’t make the same mistake.

Bonus: My slides from the RightScale conference address this subject in more detail if you’re interested.

In March of 2010, I sat on a panel with Justin Erenkrantz (Apache), Mårten Mickos (Eucalyptus), and Jason van Zyl (Maven/Sonatype) at the Eclipse Conference debating the future of open source [coverage]. The audience asked questions on licensing, development models and the direction of open source generally. One of the questions concerned the role of foundations like Eclipse, and whether they represented the future or if that would be written instead by commercial producers of open source.

My answer to that question was simple: neither. GitHub, instead, is the shape of things to come.

Mikeal Rogers came to a similar conclusion in Apache Considered Harmful. His arguments copncerning the importance of GitHub are compelling, as the evidence in favor of decentralized version control generally and Git/GitHub specifically is overwhelming. Quantitatively, virtually all of the metrics available to us reflect trajectories similar to those reflecting the relative popularity amongst Debian installations depicted above. As Chris Aniszczyk documents, legacy centralized tools reflect volume usage but Git’s massive growth has it poised to eclipse them. Git based services, meanwhile, are both profiting from and fueling said growth; an analysis of first half commit data for four major forges collected by Black Duck indicates that GitHub has in three years become the most popular open source repository.

Qualititatively, the benefits to decentralized development have been apparent to us since at least 2007. What GitHub calls “social coding” is the ability of decentralized version control tools like Git or Mercurial to inject collaborative development into a previously individual practice. The byproduct of which is the parellelization of development; like bacteria, GitHub developers may now evolve by swapping material directly to and from one another as required coverage. Which means, in turn, that forking is no longer a bad word but the way development should be done.

In the face of GitHub’s ascendance, the implications for open source foundations are unclear. In Apache Considered Harmful, Rogers’ argues that Apache specifically has essentially outlived its usefulness.

The problem here is less about git and more about the chasm between Apache and the new culture of open source. There is a growing community of young new open source developers that Apache continues to distance itself from and as the ASF plants itself firmly in this position the growing community drifts farther away.

But while I also subscribe to Clay Shirky’s maxim that “Institutions will try to preserve the problem to which they are the solution,” it is far from clear that it applies in this case. For disclosure purposes, I’ll note here that we count both GitHub and open source foundations like Apache and Eclipse as clients.

The argument that foundations have become vestigial in a post-GitHub world necessarily focuses on functional overlaps. Historically, project hosting has been one of the services offered by foundations. The data suggests that foundations who reject decentralized version control systems will fall behind, which is why structures like Eclipse are implementing Git. Even assuming a given foundation is able to transition from centralized to decentralized mechanisms, however, there is no guarantee that this will be sufficient. Iit’s unrealistic expect any foundation to compete with GitHub on functionality or community size, given the respective areas of focus.

The value of foundations, however, has never been principally hosting. They are, rather, the manifestation of a particular mission. The Free Software Foundation, the Apache Software Foundation, the Eclipse Foundation: all serve as the focal point for a group of developers. It is possible that their respective purposes have been fulfilled by GitHub, but the surging code repository seems likely to make foundations more relevant, rather than less.

As Rogers states:

Apache was founded about 12 years ago, a time when companies were still very afraid of open source and many people in the open source community were very afraid of companies. The world hasn’t changed that tremendously, big companies still use an open source stamp as a marketing tool, commonly referred to as “open washing”, and some in the enterprise are still wary about open source, particularly when it comes to certain kinds of licensing.

But, you would be hard pressed to find a single company that didn’t use some amount of open source software nowadays.

The obvious implication is that as acceptance of mainstream open source increases the importance and relevance of Apache’s mission decreases. I believe this to be incorrect. Aside from the many important non-infrastructure services offered by foundations – including IP management, project governance, legal counsel, event planning, and predictable release schedules – foundations have value as brands.

As the volume of open source assets grows, the paradox of choice presents itself to users. This problem in part already sustains a sizable marketplace of commercial products from vendors such as Black Duck, Open Logic, Palamida, and Sonatype. With GitHub and similar tools reducing the friction associated with development, we are likely to see selection problems get worse rather than better; as the volume of open source rises, fueled in part by GitHub’s success, so too does the difficulty of making a choice.

Which is one reason foundations are important. Much as McKinsey advantages Baker and Rhodes scholars in their hiring process, certain audiences will prefer to leverage open source software associated with a known brand such as Apache or Eclipse.

We have long argued that developers are the new kingmakers. As developers begin to make more choices within the enterprises they populate, they will inevitably face the same dilemma that their management predecessors did, which is the burden of choice. GitHub is a center of gravity with respect to development, but it is by design intensely non-prescriptive and inclusive, and thus home to projects of varying degrees of quality, maturity and seriousness. Consider the following:

GitHub, Inc. (“GitHub”) supports the protection of intellectual property and asks the users of the website GitHub.com to do the same. It is the policy of GitHub to respond to all notices of alleged copyright infringement.

Notice is specifically given that GitHub is not responsible for the content on other websites that any user may find or access when using GitHub.com.

GitHub, in other words, disavows responsibility for the projects hosted on the site. Foundations, conversely, explicitly assume it, hence their typically strict IP policies. These exclusive models offer a filter to volume inclusive models such as GitHub’s.

If your continued employment depends not just on the quality of the software you employ, then, but perceptions of the quality of the software you employ, the halo effect offered by foundations that actively triage their assets is likely to be of benefit. For better or for worse. If you’re choosing between one project of indeterminate pedigree hosted at GitHub and an equivalent maintained by a foundation like Apache, the brand is likely to be a feature. Managers used to say “you won’t get fired for buying IBM.” The developers making the decisions in the future may well have their own version: “you won’t get fired for using Apache.”

Rogers appears to have legitimate concerns concerning Apache’s acceptance of Git, and I concur that it’s a GitHub world and we’re all living in it. But I find it difficult to build the case that foundations more broadly won’t have a role to play in it.

“IBM designed IMS with Rockwell and Caterpillar starting in 1966 for the Apollo program. IMS’s challenge was to inventory the very large bill of materials (BOM) for the Saturn V moon rocket and Apollo space vehicle.” – Wikipedia, IBM Information Management System

While the practice has a long history, the formalization – and more particularly, popularization – of open source is a relatively recent development. Until the process of releasing of Netscape Navigator rallied supporters of the term open source, the industry lacked a consistent vocabulary for dealing with the concept of source code available under a license. Which gave proprietary software the stage by default.

In the absence of a widely understood model for the generation and consumption of shared code, innovation in software was largely the product of commercial vendors producing proprietary software. Starting from scratch, with few exceptions most organizations outside of finance and defense could not reasonably expect to sustainably compete with vendors whose business it was to write software. Rather than try, businesses predictably chose to outsource the process of development to vendors like IBM, Microsoft, Oracle and SAP. This is the dynamic that gave us a PwC Top 10 Global Software leaderboard with a median age of 34.5 years.

With open source systemically lowering the overhead associated with development over the last decade, however, in house development has become increasingly viable from an economic perspective. The inevitable result has been greater availability of open source code. As users and vendors alike discover the benefits to open source – via models direct or indirect – the number of projects has swelled along with the rate of contributions. The database market, for example, has gone from less than half a dozen relevant open source projects to several dozen.

None of which is news even to casual observers of the open source market. What is interesting, however, is how these projects are being developed. In some ways, development today is a return to its roots. Consider the following list of open source projects:

• Cassandra
• Git
• Hadoop
• MongoDB
• Nginx
• Rails

Besides the availability of their source, what do these projects have in common? None were originally authored to be sold. All were built for purpose rather than sale; this is the return of roll-your-own [coverage]. Much as IBM once extracted IMS from an engagement that sent Americans to the moon, each of the above widely used projects was the byproduct of a business problem. Cassandra was written to manage the Facebook Inbox, Git to manage the Linux kernel source tree, Hadoop to power Yahoo’s search indexing, MongoDB to back 10gen’s original Java cloud vision, Nginx to serve pages for Rambler. And as for Rails, it was extracted from Basecamp.

This “extracted” software model is becoming routine. The innovation inherent in these projects, however, is anything but. Extracted software typically exists because a perfect solution does not, which means that in many cases it is introducing new capabilities rather than recreating existing products.

With substantial history behind it, the extracted model seems to be fairly well understood, conceptually. The open question now is about the volume of latent innovation that might emerge from extracted software in the years ahead. Projects like the list above indicate that internal innovation has accelerated over the last decade, driven by trends ranging from the greater availability of open source code to an industry-wide shift towards horizontally scaled-out architectures.

But as concern about the risks of open source thaws and is offset by wider understanding of the benefits, it is probable that waves of new internally developed projects will be released as open source. The majority of which will generate little activity and interest. But from the volume, we might expect the next Git, Hadoop or Rails.

As open source trends go, then, the extracted software model is one to watch.