Most of the commentary regarding the donation of the CloudStack assets to the Apache Software Foundation by Citrix yesterday has centered around the landscape implications. This is understandable, because CloudStack’s break from OpenStack has an impact on multiple communities.

Given the stakes involved with the cloud market, the growing number of market entrants is no surprise. Incumbents like Microsoft and VMware cannot afford to be locked out of the cloud; what’s more, each vendor’s leaership understands the financial opportunities associated with owning the platform. Amazon, which is for all practical purposes the Microsoft or VMware of the cloud, can and will leverage their position to simultaneously undermine competitors and preemptively defend lines of attack. If this means creating ancillary private cloud markets and leaving money on the table in the process – as they may with Eucalyptus and now CloudStack – so be it. Everyone who’s not Amazon or aligned with them, meanwhile, from Joyent to Rackspace, is intent on ensuring that Amazon’s tenure as the Microsoft of the cloud is as short as possible. If that means open sourcing what would have been considered core software in the process – as with Rackspace and OpenStack – so be it.

That landscape, as convoluted as it appears, is fairly well understood within the industry. While everyone wants to predict outcomes on project and API futures, the fact is that it’s too early in most cases to project accurately. The CloudStack transaction, however, does make obvious certain facts regarding the licensing mechanisms and governance models employed by the various projects.

Permissive Licensing Continues to Make Gains

Since 2009 we’ve been predicting (and observing) gains for permissive licensing at the expense of copyleft or reciprocal alternatives. Reciprocal licenses, which require that any changes to a licensed asset that are distributed be made available under the exact same terms, have in the past been employed to disincent forks and to create artificial purchase triggers (e.g. dual licensing). As forks have become something to encourage rather than fight, however, one justification for the usage of copyleft licenses has faded.

Faded to the point that permissive licenses are increasingly seen as a license of choice for maximizing participation and community size. It’s not true that copyleft licenses are unable to form large communities; Linux and MySQL are two of the largest open source communities in existence, and both assets are reciprocally licensed. But the case can be made that this will in future be perceived as anachronistic behavior.

Commerical entities in particular favor permissive licenses like the Apache Software Foundation’s , because they impose very little overhead. As Cloudera CEO Mike Olson – whose first business was built around reciprocal licensing, but whose Hadoop business is built around Apache – says of permissive licenses, “There’s very little legal complexity for people to deal with.” Permissively licensed assets can be repackaged and sold as closed source software, for example, per the terms of the license itself. For projects, then, that wish to encourage the participation and engagement of commerical vendors, the permissive license can be a good, differentiating, choice.

Citrix is being less than explicit about this, but it is probable that as they continue to build an ecosystem around CloudStack, one of the “features” they’ll cite is the permissive license which allows would be players to leverage the code however they see fit. Contrast this, for example, with Eucalyptus, which is at present governed by the reciprocal GPLv3 license.

The same license, in fact, that CloudStack left behind in favor of the ASF’s more permissive license.

The Asymmetry of Permissive vs Reciprocal

Permissive licensing isn’t an unequivocal win for the CloudStack project, however. Underreported is the licensing asymmetry created by Citrix’s relicensing of the CloudStack assets. Because of the licensing mechanisms involved, the Eucalyptus project will now be able to incorporate code from CloudStack – or OpenStack, for that matter – at will. Technical innovations within Eucalyptus, meanwhile, are protected from usage by permissively licensed cloud stacks. Code sharing, in this case, is unidirectional towards Eucalyptus. CloudStack also forfeits similar protections from the OpenStack project; anything that OpenStack wishes to consume from CloudStack, they will now be able to, per the terms of the new licensing scheme.

Historically, this has not been a particularly important dynamic. MySQL, for example, has no major history of incorporating code from the more permissively licensed PostgreSQL. But with the cloud stack projects’ broader scope and mandate, it is not out of the realm of possibility to believe that Eucalyptus may opportunistically incorporate features or components from CloudStack, OpenStack or both.

You Won’t Get Fired for Using Apache

In the wake of GitHub’s meteoric ascension, there have been many questions about the role of open source foundations like Apache or Eclipse today. There are those who argue, in fact, that they have outlived their original purpose; see, for example, Mikeal Rogers’ “Apache Considered Harmful.” But while it is certain that foundations will need to adapt to changing needs, there are many reasons yet to justify their existence [coverage]. One of which is branding.

Citrix was very careful to put the Apache Software Foundation front and center in their announcement. This does two things. First, it allows them to benefit from the halo effect of the Apache brand and the goodwill of becoming a sponsor. Second, it differentiates them from two of the more visible alternative open source cloud stacks. Eucalyptus is primarily a single vendor initiative, much as MySQL once was. OpenStack is developed by a broader community of participants, and is being transitioned to a foundation. But that process has not been without its growing pains, with one of the co-founders questioning the governance structure.

Contrast that with CloudStack and the ASF. The latter is a known quantity, and vendors like IBM have historically advantaged Apache at the expense of independent foundations (e.g. their OpenOffice.org participation). What the ultimate impact of the Apache brand will be on CloudStack’s visibility and traction remains to be seen, but it’s undeniable that the Apache brand is being positioned as a feature for the project.

Disclosure: The ASF, Citrix, Eclipse Foundation, Eucalyptus, GitHub, Joyent, Microsoft, and VMware are clients. Rackspace has been a client in the past. Amazon is not a RedMonk client.

The asymmetry of open source technologies’ ability to penetrate the enterprise datacenter is not difficult to understand. Besides questions of maturity, the fact is that different product categories carry different risk profiles, a major factor for enterprises more afraid of making the wrong choice than interested in the right one. Red Hat’s sustained growth, for example, indicates that operating systems are experiencing minimal friction in terms of adoption. Non-relational databases, on the other hand, while widely used aren’t trusted by corporate buyers yet in quite the same way, with Hadoop a notable exception.

One area that hasn’t endured the same level of skepticism is open source configuration management software. While there are many options, system adminstrators and developers are advantaging Chef and Puppet at the expense of competing solutions. But with that success comes an obvious question: which do I pick, and why?

Although discussions of the platforms’ relative technical merits can be interesting – the comments on this HN thread display the usual range of opinions on the subject – we’re typically more interested in usage patterns. Quality of implementation is an important consideration in technology selection, but history demonstrates adequately that technically inferior solutions can and often do outperform competitors. Because there is no single canonical source for usage, we instead examine a variety of proxy metrics, looking for patterns that indicate a broader narrative at work. Here’s a non-comprehensive run through some of the metrics that we regularly evaluate.

Debian

Update: Spoke with Opscode’s Jesse Robbins who wanted me to be aware that Chef’s under-representation on Debian is likely due in part to the fact that they run and manage their own repositories, but also because they recommend deployment via the RubyGems package manager over Debian’s apt-get. So consider yourselves caveated; the charts otherwise remain untouched.

Each running instance of the Linux distribution Debian has the ability to phone home telemetry of the packages installed on the system. Called Popularity Contest, this provides insight into what the relative adoption rates of various software packages are within the subset of the Debian community that has elected to self-report application information. This graph, then, reflects adoption of Chef relative to Puppet within the Debian community.

As you can see, Puppet substantially outperforms Chef in this context. Part of this is the fact that Puppet is by four years the older project, and thus has had additional years to build adoption numbers. But if we look more closely at the data, however, there are indications that adoption may also be a function of packaging issues. In mid-July of last year, Opscode (the company behind Chef) made updated packages available for Debian and related distributions. Almost immediately thereafter, according to Debian Popularity Contest data, adoption spiked on that platform.

More interestingly, the adoption of Chef enabled by the new packages may have led to a transient decline in reported Puppet adoption. If we examine a three month period of Puppet adoption beginning in July, the impact to the overall trajectory is apparent.

From a macro perspective, the data indicates Puppet still remains more broadly adopted within the Debian community than Chef. But Chef is growing, and the evidence does seem to confirm growth for one is negatively correlated with growth from the other.

GitHub

As one of the largest and most important developer communities in existence, we track individual project performance on GitHub closely. With author backed repositories for both Chef and Puppet available, it’s possible to compare the performance of the two projects in basic fashion. GitHub gives a slight edge to Puppet in terms of total contributors; 121-118. Puppet also saw more pageviews on GitHub over the past 90 days, 30735 to 22361.

But in metrics relating to explicit interest in the project – specifically the numbers of forks and watchers per project – Chef outperformed Puppet.

The results from GitHub then, are relatively inconclusive. Implicit metrics like pageviews point one way, explitic metrics like forks another. There is no clear winner of this category.

Hacker News

On Hacker News, neither Chef nor Puppet are dominant from a discussion perspective. Mentions of one tend to closely track discussion of its counterpart, in fact, according the Hacker News search APIs.

The correlation is unsurprising except for the timeframe; Chef shows substantial traction in discussion on Hacker News well in advance of its 2009 availability. This suggests artifacts in the returned data, because mailing list traffic archives only date back to 2009.

Shared Scripts

One of the concepts shared by both Chef and Puppet is scripts that implement common patterns. In Chef, these are called cookbooks, for Puppet users, modules. While it’s impossible to effectively measure the number of scripts per project – their library size, so to speak – we can attempt to evaluate first how many each company hosts themselves. Second, we may attempt to imperfectly infer community size by comparing query returns for both projects on GitHub, as it is, not surprisingly, common practice to host cookbooks and modules on the site.

For the former, here are the returns. As a side note for those interested, these results were obtained by Puppet’s forge site and Opcode’s Cookbook Site API, respectively.

As you can see, Opscode currently hosts approximately thirty percent more scripts than does Puppet Labs, though to what extent either vendor is focusing their attention on these sites versus their efforts on GitHub is not entirely clear. Comparing general GitHub query results, meanwhile, we see that the lead has changed hands.

As measured by general GitHub queries, Puppet enjoys a slight (~10%) advantage in returns over Chef. It’s a rough metric because it includes all matching repos, but as a broad indicator of traction it has some utility.

The Gist

Ultimately, the data we’ve looked at – aside from the Debian usage information – doesn’t prove the case for either platform. Puppet backers can take heart in its dominance on Debian as well as its GitHub pageview lead and repo traction. Chef advocates, on the other hand, may take comfort in the fact that a project that’s four year younger is outperforming more mature competition in some metrics. For our part, we have (full disclosure) worked with and admire both companies.

And while there is understandably friction between the two communities at times given the functional overlap, it is likely that there’s more than enough oxygen to support both projects indefinitely. Apart from the various community numbers discussed above, both can point to impressive customer rosters and partner bases. Given the opportunity size and scope, however, as well as the legitimate traction behind both projects, the ultimate leadership role for the category may not be who can create the best software but who can best leverage the data it generates. With software valuations in decline, data should increasingly be a product focus.

In the meantime, it will be interesting to watch these projects compete with each other moving forward.

Afterword: In case anyone’s curious, we did look at StackOverflow metrics as well, but the differences were slight enough that we omitted them from the above.

Guess which open source license is more popular than the MIT, Artistic, BSD, Apache, MPL and EPL put together? Surprise: it’s the GPL. True, usage appears to be in steep decline. Since August of 2009, the GPL is down around 8%, according to data from Black Duck. Over that same span, usage of permissive licenses is up: MIT by 8%, Apache 2% and BSD 1%. But while developers may be increasing their usage of non-copyleft licenses, is this a problem?

With all due respect to Red Hat counsel Richard Fontana, for whom the waning usage of the GPL appears to be alarming (as an aside, I find it mildly ironic that the project that built those slides is itself permissively licensed), this seems to be little more than a normal market adjustment. The unnaturally dominant role copyleft licensing played for many years was, in my view, as much an artifact of the extraordinary visibility of projects like the Linux kernel and the MySQL database as project owners’ affection for the reciprocal protections offered by copyleft licensing. As such, it never appeared to be sustainable from this vantage point, which is why we predicted back in 2009 precisely what has occurred: gains from permissively styled licenses at the expense of reciprocal alternatives.

The GPL is an enormously important mechanism, as we’ve asserted since at least 2005. It simply could not expect to be the only mechanism, indefinitely. Licenses are tools, and should be selected and employed based on a desired outcome. As those desired outcomes have changed over time, it’s only logical that licensing patterns change to accomodate.

We have argued, both observationally and based on public market valuations, that the value of software as a differentiated asset is in decline. The evidence suggests that native web businesses assign a substantially lower value to written software than did their predecessors. Facebook, for example, originally wrote Cassandra to manage their Messages feature, subsequently releasing the code into an Apache project. When they rebuilt Messages, they chose Hbase – an Apache project originally created by a separate organization, Powerset – over their own Cassandra. GitHub’s Tom Preston-Werner, for his part, recommends open sourcing all but those features that represent “core business value.”

What both organizations have realized is that very little code, in practice, is competitively differentiating. Which makes open source a logical course of action, because the potential benefits of making the source code available are likely to substantially outweigh the costs. And as far as licensing is concerned, if the code is not a competitive advantage, it is likely not worth protecting. For those who view the code they produce as a generally fungible asset, the additional protections afforded by a reciprocal license may not only be unnecessary, but unwanted. In this scenario, permissive licenses are a perfect alternative.

Which should be ok. Open source licenses are, ultimately, different tools. Employing them towards different ends is nothing more than logic.

The cost of delaying my 2012 predictions is that one has already come to pass. Nginx – the web server now powering all of the redmonk.com properties – passed IIS according a January 4 Netcraft release. Because the quantitative data available to us has indidicated surging interest in the alternative web server – the logical result of which was a commercial response – we’ve been expecting something like this. But of course we can’t count this as a prediction any longer because it’s January 13th.

Here instead are a few things that have not yet come to pass, but will, I believe, in the year ahead. These predictions are informed by historical context and built off my research, quantitative data that’s available to me externally or via RedMonk Analytics, and the conversations I’ve had over the past twelve months, both digital and otherwise. They cover a wide range of subjects because we at RedMonk do.

For context, my 2010 predictions graded out as 66% accurate while 2011′s were 82% correct.

With that, the 2012 predictions.

Data & The Last Mile

It is not technically correct to assert that large scale data infrastructure is a solved problem. Decades of innovation remain, as the Cambrian explosion of projects demonstrate. It is nevertheless true that relative to the user interface, data storage and manipulation is a solved problem. Since the original creation of Hadoop in 2006, for example, we have seen multiple user interfaces applied: connectors (e.g. R), standard MapReduce, scripting (e.g. Jaql/Pig), SQL (e.g. Hive), spreadsheets (e.g. BigSheets), client tooling (e.g. Karmasphere). Each has its strengths, none bridges the last mile: putting the power of Big Data in the hands of ordinary users.

Which is perhaps unsurprising; even the mature relational database world uses abstractions of varying levels of complexity to interface with business users. But with data driven decision making on the rise, premiums are being placed on tooling which can expose in sensible fashion data to those without degrees in computer science. Hence, the elevated visibility of startups such as Metamarkets, who excite data scientists with tools like Druid but whose valuation may ultimately depend on its last mile expertise.

At this point in time, whatever my preferred model for data storage and whatever the type, there will be greater than one credible option for a data engine. The same cannot be said for presentation. Which would be less problematic if the market for Big Data talent were not so desperate; outsourcing to shops like Mu Sigma will be an option in some quarters, but comes with its own inefficiences and risks, not to mention per inquiry premiums.

This, then, will be an area of focus in 2012, for both innovation (look for assisted anomaly and correlation identification, a la Google Correlate) and M&A.

Desktop Importance Declines

The most interesting characteristic of the forthcoming Windows 8 release isn’t the technology, which is curious because it’s revolutionary from a Microsoft standpoint. From the support for ARM to the addition of the Windows Store to the ability to author in JavaScript and HTML5, there is much to digest. Instead, the single most defining characteristic of the pending launch is apathy.

Overall inquiries and discussion of the platform demonstrate curiosity but limited interest; the visibility of the once dominant Windows platform is secondary to mobile platforms like Android and iOS.

While this is not a function of any specific or general design failures on the part of Microsoft – indeed, the platform is incorporating important changes while making itself more developer accessible – it is symptomatic of a broader and more difficult to attack problem: the declining role of the desktop.

The desktop is simply not as important as it once was. Mobile usage is eroding the central role PC’s once played; while they are still the dominant form of computing, the trendline is declining and there is no reason to expect it to invert. It’s been suggested that mobile computing in general is additive; that it’s being used to extend the usage of computing to areas where PCs were not employed, and is thus non-competitive. But our data as well as Asymco‘s indicates that, at least in part, mobile usage is coming at the expense of traditional platforms. General search volume data, as we’ve seen, validates this assertion.

There are two implications here. Most obviously, Microsoft’s ability to generate interest in and thus leverage for its flagship operating system is jeopardized. Worldwide developer populations are not necessarily zero sum as skills overlap, but they tend to be rivalrous; an Android or iOS developer is often a lost potential Windows developer – experiments like BlueStacks aside. We can therefore expect Microsoft to have to expend more effort to attract fewer developers to their platform, a negative cycle which becomes cyclical. Second, as the desktop’s primacy abates, we can expect to see greater competition in the marketplace. As enterprises become by necessity more heterogeneous, incorporating Android and iOS devices, the costs of supporting second operating systems drifts towards marginal, which means that forecasts of greater Apple penetration become more probable.

Developer Shortages

It’s become axiomatic that industry hiring is all demand and short supply, and none of our clients expect any relief in the year ahead. Nor will they receive it. Shortages for in demand skillsets will continue over the next twelve months, advantaging entities that are either geographically positioned to leverage markets less competitive than the Valley or with the logistical ability to incorporate remote hires.

That said, we will in 2012 see the first steps towards a more rational market, through a combination of cultural shift and educational model innovation that will increase supply. Regarding the former, it’s no secret that technology has had a profound impact on the erosion of middle class jobs. In Race Against the Machine, MIT Professors Andrew McAfee and Erik Brynjolfsson document the role that rapid innovation has had on jobs:

Digital technologies change rapidly, but organizations and skills aren’t keeping pace. As a result, millions of people are being left behind. Their income and jobs are being destroyed, leaving them worse off in absolute purchasing power than before the digital revolution.

Even skill industries are not immune. From John Markoff’s New York Times piece, “Armies of Expensive Lawyers, Replaced by Cheaper Software“:

“From a legal staffing viewpoint, it means that a lot of people who used to be allocated to conduct document review are no longer able to be billed out,” said Bill Herr, who as a lawyer at a major chemical company used to muster auditoriums of lawyers to read documents for weeks on end. “People get bored, people get headaches. Computers don’t.”

While Brynjolfsson and McAfee are ultimately optimistic about the prospects of technical progress as they relate to employment, the outcome is far from certain.

What is becoming clear, however, is that unemployment rates that have been north of 8% in the US since February of 2009 are driving people into industries that are desperate for help. For some, this means oil & gas employment in traditionally underpopulated environments like North Dakota. For others, however, technology – long an enemy – is becoming a refuge.

We’re seeing a spike in inquiries about transitioning to technology careers. Lawyers, management consultants, teachers and others are seeking – and often finding – homes for themselves within the technology sector. Some are self-taught or trained on the job, others merely apply existing skills in new contexts, but both represent a potential cultural shift. Which begs the question: could technology be the next major middle class employment sector?

For that to happen, the education system needs to improve, because even an industry which has been one of the few economic bright spots of the last decade can only absorb so many unskilled workers without slowing. This is the real significance of applications like Code Academy or programs like Harvard’s free CSCI E-52, MITx or Stanford Engineering Everywhere: they are one potential solution to the perpetual shortage of talent. For all of the limitations of distance learning, the scale means that some subset of motivated students will become productive developers, and by extension, contributors to the larger economy.

This is a long term process, so obvious progress within 2012 will be minimal, and talent shortages will continue. But we will in the next twelve months begin to see distance trained students hired at scale, and this will be one of the first steps towards lower talent costs as well as, possibly, the restoration of middle class employment opportunities.

Monitoring as a Service

We are not oriented around category definitions at RedMonk; we prefer market driven names to those conceived and marketed by the analyst industry. That said, it seems clear that the time of Monitoring-as-a-Service (MaaS) is at hand. New Relic’s growth led to a $15M round in November, Boundary took $4M a year ago this month, Monktoberfest speaker Theo Schlossnagle’s Circonus has been in market for over a year, and virtually every vendor that we speak with today is adding monitoring and management facilities, from 10gen’s MMS to Cloudera’s Cloudera Manager.

The proliferation of these services is a direct response to the increasingly heterogeneous nature of application architecture and the reality that the substrate is frequently network based, rather than local. Given accelerating rather than declining consumption of network resources, we predict a strong increase in interest and adoption of MaaS tools. Much as I don’t care for the term itself.

Intelligent usage of generated telemetry – which we’ll come back to – will further cement adoption, delivering previously unseen value.

Open Source and the Paradox of Choice

Gartner in March of last year asserted that open source had hit a tipping point, saying:

“Mainstream adopters of IT solutions across a widening array of market segments are rapidly gaining confidence in the use of open source software.”

We concur, although we would argue that the tipping point actually occured ten years or more prior. The Apache web server and MySQL were originally written in 1995. In 1999, we saw the public offering of Red Hat and the creation by IBM – as mainstream a technology brand as there is in the enterprise – of the Linux Technology Center. Firefox was first released in 2003. None of these reached their relative levels of popularity in the past twelve months; they have instead been the de facto infrastructure for the better part of the last decade.

Regardless of when one asserts that open source crossed the chasm, however, it remains that it is a model whose popularity is increasing over time. As understanding of the benefits increases and concerns about the risks abate, more organizations are not only consuming open source but contributing to it. Evidence suggests, in fact, that perceptions of the value of software are in decline – we’ll come back to that too, and that the end result of this is that more proprietary code is being released as open source software.

Widely perceived as a net benefit, however, the influx of new projects does present problems for would be adopters. Specifically, the paradox of choice implies that developers will increasingly be forced to select from a growing sea of projects which may or may not be suitable for their needs. And while the nature of open source guarantees developers the ability to apply this code to their projects without restriction or commercial engagement, this is a process with a limited ability to scale. Consider the NoSQL space, as an example. Presuming for the sake of argument that the developers in question understand the different categories of database – key value stores, document databases, columnar databases, MapReduce engines, graph databases and so on – well enough to understand their high level needs, there are at least two and sometimes as many as half a dozen credible options to consider.

This paradox of choice, or too much of a good thing, will become more problematic over time rather than less as contributions will continue to rise. The net impact is likely to be increased commercial opportunities around selection, and therefore attention to vendors like Black Duck, Open Logic, Palamida and Sonatype.

PaaS: The New Standard

It has been evident for some time that runtime fragmentation – an aggressive diversification of programming languages and frameworks, specifically – will change the development landscape. The market failure of the first generation PaaS providers, in fact, was primarily a function of their over-prescriptive natures. The benefits to outsourcing management and scale were obsoleted by the constraints; Java shops were never likely to rewrite their application stack in Python or Ruby strictly to benefit from a platform. Which is why virtually every relevant PaaS provider today offers a choice of runtimes, so as to maximize their addressable market.

But in a fragmented world, what might emerge as a standard? From a developers’ perspective, the standard is most often the framework they’re deploying to, whether that’s Django, Node.js, Lift, Play, Rails, Spring, the Zend Framework or another. From a vendor perspective, however, the new standard is likely to be one level of abstraction up from individual language frameworks: the platform itself. Certainly this is VMware’s opinion, as they are in Maritz’ words trying to construct “the 21st-century equivalent of Linux” – i.e. the substrate that everything else is built on top of.

In 2012, this will become more apparent. PaaS platforms will emerge as the new standard from a runtime and deployment perspective, the middleware target for a new generation of application architectures.

Service Proliferation

With the inevitable adoption of multiple third party services – varying cloud resources, multiple, possibly overlapping, management and monitoring services and so on – will come challenges in making sense of the whole. Overall, instrumentation and visibility on a per service level is improved, but aggregating these views into a cohesive picture of overall architectural health and performance is likely to be highly problematic. Not least because the services themselves may present conflicting information and data. Google Analytics and New Relic, for example, are frequently at odds over load times and other delivery related performance metrics. Introduce in to that mix services like Boundary or CloudWatch and the picture becomes that much more complex. Connecting their data back to underlying log management and monitoring solutions such as 10gen’s MMS or Splunk is more complicated still.

The challenges of service intregration will create commercial opportunities for aggregating services which consume individual performance streams, normalize it and present customers with a consolidated single picture of their network performance. Commercial solutions will not fully deliver on this vision in 2012, but we will see progress and announcements in this direction.

Telemetry Usage

Five years ago, we began publicly discussing revenue models based around what we termed telemetry, or product generated datastreams. The context was providing open source commercial vendors with a viable economic model that better aligned customer and vendor needs, but the approach is by no means limited to that category: Software-as-a-Service vendors, as an example, are well positioned to leverage the data because they maintain the infrastructure. In 2011, we finally began seeing vendors besides Spiceworks take the first steps towards incorporating data based revenue models. For products like Sonatype Insight [coverage], data is not a byproduct, but the product.

In 2012, this trend will accelerate as necessary monitoring capabilities are added to product portfolios and industry understanding and acceptance of the model overcomes conservative privacy concerns. Many more vendors will begin to realize that like New Relic, which observed a decline in commercial application server usage, their accumulated data is full of insights on customer behaviors and wider market trends both.

Value of Software Will Continue to Decline

Capital markets have not, traditionally, been overly fond of software firms, perhaps because comparatively few of them eclipse annual revenue marks of a billion dollars – less than twenty, by Forbes‘ count. Microsoft’s share price has languished for over a decade in spite of having not one but two licenses to print money. The mean age of the PwC’s Top 20 software firms by revenue is 47 years; a fact which cannot be encouraging to startups.

Higher valuations instead are being awarded to entities that employ software to some end, rather than attempting to realize revenue from it directly. Startups today realize this, and the value of software in their models has commensurately been adjusted downward. Tom Preston-Werner, for example, describes the GitHub philosophy as “open source (almost) everything.” Facebook, LinkedIn, Rackspace, Twitter and others exhibit a similar lack of protectiveness regarding their software assets, all having open sourced core components of their software infrastructure that would have been even five years ago fiercely guarded.

This is becoming the expectation rather than the exception because it is nothing more or less than an intelligent business strategy. Businesses can and will keep private assets they believe represent competitive differentiation, but it will be increasingly apparent that less and less software is actually differentiating. As a result, 2012 will see even less emphasis on the value of software and more on what the software can be used to achieve.

Bonus: Facebook’s Most Important Feature

In 2012 will be Timeline. Mark it down.

Disclosure: Black Duck, Cloudera, GitHub, IBM, Microsoft, Sonatype and VMware are RedMonk customers, while 10gen, Boundary, Circonus, Facebook, Open Logic, Palamida, and New Relic are not.

This is the concluding half of the exercise in which I review my predictions for the calendar year just ended. If you’re looking for the original 2011 predictions, those are here. Part 1, meanwhile, can be found here. With that, on to the predictions.

Hardware

Workstations Will Make a Comeback

This prediction is not supported by data nor even anecdotal evidence. Even in the small sample of my contacts, migrations away from tower-style hardware profiles towards laptops accelerated. Wider market developments seem to confirm this; Apple may be retiring their existing workstation platform, the Mac Pro.

This is a (big) miss.

ARM Will Emerge as a Server Player

Whether they will ultimately emerge as a credible mainstream alternative remains to be seen, but ARM is indeed emerging as a server player. Though virtually all of them discuss it privately, HP (via Calexda) this year became the first major systems player to publicly detail plans for ARM servers – perhaps banking on the fact that the upcoming A15 processor is more server friendly,

Intel is predictably skeptical of ARM’s viability in its core markets, with CEO Paul Otellini bluntly dismissive: “It ain’t gonna work.” And while it certainly hasn’t proven to work thus far, and there are real architectural and software issues to address, the power profile continues to pique the interest of server manufacturers and customers alike. Even marginal power savings mean real dollars at scale.

I count this as a hit.

Tablets are a Real Market

From the New York Times, late January, 2011:

“The iPad, introduced in April — is on track to deliver $15 billion to $20 billion in revenue in its first full year of sales, estimates A. M. Sacconaghi, an analyst at Sanford C. Bernstein. At that size, if the iPad were a stand-alone company, it would rank within the top third of the Fortune 500.”
Steve Lohr, “The Power of the Platform at Apple“

Any questions? It’s true that to date the tablet market is more accurately characterized as an iPad market, but irrespective of the particular vendor dynamics in the space, the hardware form factor appears here to stay.

I count this as a hit.

Mobile

Challenges of Native Development Will Drive Interest in HTML5 and Hybrid Approaches

In August, we did a quick pass at some developer metrics and confirmed what our qualitative research had already indicated: that interest in PhoneGap was booming. Here’s a chart of StackOverflow traction, for example.

Interest was booming enough, in fact, that Adobe acquired the talent behind PhoneGap, the code of which was submitted to Apache. This interest was unsurprising in light of the frustrations experienced by enterprises and developers alike, who are collectively slowed by the process of building an application on one platform and then porting to a second. Developers are frustrated enough, in fact, that they are in certain cases actively stalling development. While opinions differ on individual platform trajectories, most would agree that the status quo is unlikely to remain static. Meaning that at least some native development effort is likely to be wasted.

Couple that with improving mobile browser capabilities, and interest in HTML5 and hybrid approaches is likely to remain strong, in spite of inherent advantages to native development like discovery.

I count this as a hit.

NoSQL

The NoSQL Marketplace Will Experience Consolidation

The merger of CouchOne and Membase into CouchBase in February provided some evidence that the long anticipated wave of consolidation in this space was beginning, but the balance of the year provided little evidence to support this aside from the acceleration of a few individual players such as MongoDB [coverage]. I remain convinced that the marketplace will be unable to sustain the current volume of would be commercial entities, but from our conversations with both those in a position to potentially impact consolidation and those interested in partnering with various NoSQL players, it is clear that consolidation will depend on clearer winners and losers to proceed. This should occur in 2012.

I’ll count this as a push in light of the CouchBase merger which subtracted one player but otherwise saw very few exits.

NoSQL Will Look More Like Pro-SQL

The implicit rejection of the Structured Query Language in the NoSQL term is ironic in light of the fact that a variety of projects are now adding similar features. Continuing in the proud tradition of Hive and Pig, which provide query language interfaces to Hadoop, DataStax announced CQL in June while CouchBase and SQLite announced UnQL in July [coverage].

Whether we’ll see a unified interface or a variety of engine-specific implementations as Alex Popescu would prefer remains to be seen, but query languages will be coming to the majority of NoSQL stores one way or another.

I count this as a hit.

Open Source

Open Source of Non-Strategic Infrastructure Assets Will Increase

From Twitter open sourcing the Storm assets it acquired via the BackType transaction to the New York Stock Exchange’s donation of OpenMAMA to the Linux Foundation, it is increasingly clear even to traditional parties that the release of non-strategic code as open source has multiple benefits. GitHub’s Tom Preston-Werner’s list of same is difficult to improve upon:

• “Open sourcing code is great advertising for you and your company.”
• “If your code is popular enough…you will have created a force multiplier that helps you get more work done faster and cheaper. “
• “When you open source useful code, you attract talent.”
• “If you’re hiring, the best technical interview possible is the one you don’t have to do because the candidate is already kicking ass on one of your open source projects.”
• “Dedication to open source code is an amazingly effective way to retain that talent.”
• “[Assuming code will be open sourced] leads to effortless modularization.”
• “By getting code out in the public we can drastically reduce duplication of effort.”
• “It’s the right thing to do.”

It may or may not be beneficial to open source core strategic assets, as VMware did with Cloud Foundry, but it is increasingly hard to justify protecting those that are purely tactical in nature. The benefits in many if not most cases will outweigh the costs, which is why we’re seeing an increase in contributions to open source projects.

The data from the annual Eclipse surveys is one example of this. If we examine the percentage of organizations that contribute back to open source versus those that do not from 2007 to 2011, it is clear that comfort levels with open source generally are rising.

I count this as a hit.

Forking: How Development Gets Done

The benefits to distributed version control and the beneficial forking model it encourages have been sufficient to convince even large projects such as Eclipse: the majority (44.2%) of its projects have migrated from CVS or Subversion to Git. The popularity of this model is also on display in the graph above, which depicts the relative performance among commit volume of four major forges. GitHub is the youngest of the forges and yet commands a significant majority of the overall commits observed by Black Duck.

While it is difficult to separate the success of Git from GitHub, it is not necessary for this exercise, because both by design encourage forking as a developmental best practice. Forking is increasingly how development is conducted.

I count this as a hit.

Ubuntu is the New SUSE

In March of last year, I explored a set of metrics evaluating the relative performance of SUSE with developers, from jobs data to community traction. None of the metrics favored SUSE at the time, and the observed trends that favored Ubuntu persist. Consider, for example, the job trends:

As pointed out at the time, despite the relative outperformance of SUSE relative to Ubuntu, the respective trajectories were problematic for the former. Revisiting the data today, we can see that the trendlines produced a logical outcome.

Commercially, Ubuntu has continued its emergence as a server player. While initial claims that it was the primary operating system behind HP’s public cloud may have been [overstated],(http://arstechnica.com/business/news/2011/10/ubuntu-will-power-hps-new-cloud-service.ars) in HP’s words “that they are the first one in our current private beta.” This is an announcement that would not have been possible prior to 2011.

I count this as a hit, though it will be interesting to see if Mint can become the new Ubuntu, in turn.

Programming Languages

JavaScript is resurgent

In April, we were fortunate to be able to analyze data from Black Duck regarding open source project commits by programming language. When we compared a 2011 snapshot to the volume of all time commits, the rise of JavaScript was apparent.

While dynamic languages were ascendant across the board, JavaScript outperformed even high growth languages like Python and Ruby.

As “The Rise and Rise of JavaScript” notes, the prospect of using the same language on browser and server is compelling, but it doesn’t stop there:

JavaScript’s serialization form, JSON, is becoming ubiquitous as a lighter-weight alternative to XML for streaming structured data, and NoSQL databases like mongo are happily using JSON and JavaScript in the database as a query language. This means, for the first time, you can have the same JavaScript function in the browser, on the server and in the database.

JavaScript may only be the most popular language within forward communities like GitHub, but its future in the wider world is bright, Dash notwithstanding.

I count this as a hit.

Bonus Prediction

Dropbox will become an attractive acquisition target

One the one hand, I got this wrong, because Steve Jobs apparently first attempted to acquire Dropbox in 2009.

“Jobs presciently saw this sapling as a strategic asset for Apple. Houston cut Jobs’ pitch short: He was determined to build a big company, he said, and wasn’t selling, no matter the status of the bidder (Houston considered Jobs his hero) or the prospects of a nine-digit price (he and Ferdowsi drove to the meeting in a Zipcar Prius).

Jobs smiled warmly as he told them he was going after their market.”

It was in 2011, however, that Dropbox – a firm that had previously raised a mere $7.2M – justified that decision, with a $4B valuation.

So I’ll go ahead and call this one a push.

The Final Tally

Scoring the 2011 predictions, then, we get 14 of 17 correct against two push and one miss. Generally, an 82% success rate in forecasting means that the game is rigged; 51% is enough to make a substantial profit in public markets. And in reviewing the predictions, it’s certainly true that a few – the continuing developer shortage, for instance – were likely obvious enough to not merit the “prediction” label.

But we do believe, as our website suggests, in William Gibson’s claim that the future is already here, it’s just unevenly distributed. Conclusions that are obvious to us are not to many traditional enterprise technology buyers, so in that sense this game is, in fact, rigged. We’ve predicted the rise of dynamic languages, Node.js, NoSQL, REST, Software-as-a-Service and such then not because we’re Nostradamus, but because we know who to listen to.

If you want to hear what that audience is telling us about 2012, then, stick around. Those are due next week.

“From the beginning of time two forces have vied for influence over us. One is bottoms-up, decentralized, and emergent. The other is top-down, centralized, and directed.

The first force catalyzes change and divergence, while the second tends toward order and convergence. The first gives birth to new ideas, and the second enshrines them.”
- Adam Ludwin, From LOLcats To Occupy Wall Street, Everything Is Happening From The Bottom Up Now

Traditionally, industry analyst firms have been oriented around top down adoption patterns. CIOs and other IT decision makers comprise both the research subjects and purchasing audience for the majority of firms in this industry, large and small. Which was logical given traditional procurement patterns. When hardware, software and services are available only at high prices, command and control is an appropriate management structure. Attempting to scale the decision making process for big ticket items across a large body of middle managers is not likely to yield acceptable outcomes.

An approach that makes sense in one context, however, may be misapplied in another.

The technology purchasing landscape today looks very different than it did even five years ago. Where once CIOs might reasonably expect to have the clearest understanding of what technologies are leveraged within their own organizations, today they are, as Billy Marshall put it, “the last to know.” This pattern manifests itself every day within the majority of businesses. Not because CIOs are failing, but because of trends that have fundamentally and likely permanently disrupted their ability to centralize the technology adoption process.

The four trends we see as most important in driving this are arranged here in rough chronological order.

Open Source

In the late nineties, startups and enterprises alike were effectively beholden to commercial suppliers for the majority of their software needs. Because each piece of the requisite software infrastructure had to be licensed, the capital expenses associated with new initiatives was high. This represented a barrier to entry, and thus a brake on innovation.

With the popularization of open source software, developers from enterprises and startups alike were able to operate independently. For the first time, the actual software practitioners were free to choose their own software rather than having it selected for them and subsequently imposed upon them by upper levels of management. Even in situations where the ultimate production infrastructure targets remained commercially licensed software, open source software like Linux and MySQL allowed for prototyping and rapid development without the attendant costs, both financial and in procurement latency.

This was the first major shift affecting procurement, and perhaps the most profound. None of the infrastructure we take for granted today – Linux, Apache, MySQL, PHP, etc – were originally adopted from the top down. Their adoption was, instead, a fait accompli. CIOs – the last to know – gradually became aware that increasingly significant portions of their infrastructure, unbeknownst to them, were running on free and open source software. The inevitable demand for production support options for this software is what fueled, in time, the valuations of MySQL, Red Hat and others.

Bring Your Own Device

In October, Apple CEO Tim Cook asserted that 92% of the Fortune 500 were “testing or deploying iPad in the course of less than 18 months,” which may help explain why the iPad revenue stream by itself would place within the top third of that group. The interesting thing about this is that the majority of businesses appear uncertain about precisely why they’re deploying tablets: “Most participants, 51 percent, indicated that they did not have a clearly articulated strategy.”

The answer, in most cases, is that there isn’t one. iPad adoption, much like the penetration of iPhones and Android handsets is being driven by users who simply want the device. Faced with a choice between users – chief executive officers among them – who will employ their own devices for work purposes with or without the permission of IT, many businesses are compelled to support the platforms even without concrete business justifications for them.

The consumerization of the enterprise is decentralizing the process of technology selection, but its importance may lie rather in design. Like all products, technology is designed and built to be sold to a specific buyer. For enterprise products, historically, the actual user has been a secondary concern; the buyer – typically centralized IT – was the priority. Consumer technology companies like Apple, however, are designed for a user. What they give up in IT friendly features they more than make up for in usability and the ability to delight.

The Bring Your Own Device trend, therefore, may well improve user productivity by driving devices designed to be used rather than managed into organizations, from the bottom up.

Software as a Service

Software as a Service is a classic case study in timing with respect to market acceptance. Not many remember today that the model actually failed the first time around, when its practitioners were known as Application Service Providers. Pyschologically, few enterprises were prepared for either the idea of renting software or externalizing critical data like that stored in customer relationship management systems. By the midpoint in the last decade, however, these concepts were sufficiently commonplace to see Salesforce.com a publicly traded company with a valuation north of a billion dollars.

Consumer markets, meanwhile, had adapted much more quickly. Hotmail debuted in 1996, Yahoo Mail the year after and Gmail dropped in 2004.

Some of those same consumer services were pressed into service by enterprise workers, in fact; it was once common for Exchange users to forward all of their email to Gmail due to the disparity in storage limits between typical Exchange implementations and Google’s webmail product.

This pattern has played out repeatedly over the years, from webmail to CRM to project management software to website hosting to online helpdesks. All were adopted from the bottom up. By making applications available to anyone with a browser, often at low or no cost, SaaS has surged up through the ranks of enterprises. The inexorable nature of the model is reflected by the growth of providers large (Salesforce.com) and small (37signals).

Cloud

The single most important feature of the cloud has nothing, or at least very little, to do with technology. It is, rather, the pay as you go economic model. As Flip Kromer puts it, “EC2 means anyone with a $10 bill can rent a 10-machine cluster with 1TB of distributed storage for 8 hours.”

What this means in practical terms is that for the first time, hardware procurement is democratized. From an accessibility and availability standpoint, cloud is the hardware equivalent of open source software. Where open source allowed developers to bypass traditional procurement channels by making quality infrastructure and development software freely available, so does the cloud allow the growing class of devops technologists to leave the world of high latency hardware procurement – where same day server provisioning is a feature – behind. Armed with nothing more than a credit card, instances can be spun up and ready for use in ninety seconds.

Cloud is the final piece of the bottom up puzzle. Open source software and to a lesser extent SaaS allowed for the decentralization of enterprise technology development, but at some point hardware would become necessary which was the insertion point for IT. With public clouds, it is possible for the first time to entirely bypass the traditional gatekeepers.

The Net

It should be evident that traditional procurement and purchasing is not dead, just increasingly bypassed by a more efficient process. Also, that a great many enterprises continue to function largely as they always have: top down. More important than the question of whether this model is sustainable in the face of the trends above is whether it should be.

Before lamenting the fact that the above forces are disrupting and destabilizing your enterprise IT, consider that that may be a net gain. If the primary drivers of BYOD, Cloud, Open Source, and SaaS include ease of use, lower costs, frictionless availability, and speed of provisioning, are these trends worth opposing? Particularly since efforts to do so will, in all probability, fail?

Or are they instead assets to be strategically leveraged? There is little debate that businesses that move the most quickly have a competitive advantage. It’s not clear how businesses that prohibit the same tools that enable this will benefit.

Either way, bottom up adoption is here to stay: use it or lose.

In March of 2010, I sat on a panel with Justin Erenkrantz (Apache), Mårten Mickos (Eucalyptus), and Jason van Zyl (Maven/Sonatype) at the Eclipse Conference debating the future of open source [coverage]. The audience asked questions on licensing, development models and the direction of open source generally. One of the questions concerned the role of foundations like Eclipse, and whether they represented the future or if that would be written instead by commercial producers of open source.

My answer to that question was simple: neither. GitHub, instead, is the shape of things to come.

Mikeal Rogers came to a similar conclusion in Apache Considered Harmful. His arguments copncerning the importance of GitHub are compelling, as the evidence in favor of decentralized version control generally and Git/GitHub specifically is overwhelming. Quantitatively, virtually all of the metrics available to us reflect trajectories similar to those reflecting the relative popularity amongst Debian installations depicted above. As Chris Aniszczyk documents, legacy centralized tools reflect volume usage but Git’s massive growth has it poised to eclipse them. Git based services, meanwhile, are both profiting from and fueling said growth; an analysis of first half commit data for four major forges collected by Black Duck indicates that GitHub has in three years become the most popular open source repository.

Qualititatively, the benefits to decentralized development have been apparent to us since at least 2007. What GitHub calls “social coding” is the ability of decentralized version control tools like Git or Mercurial to inject collaborative development into a previously individual practice. The byproduct of which is the parellelization of development; like bacteria, GitHub developers may now evolve by swapping material directly to and from one another as required coverage. Which means, in turn, that forking is no longer a bad word but the way development should be done.

In the face of GitHub’s ascendance, the implications for open source foundations are unclear. In Apache Considered Harmful, Rogers’ argues that Apache specifically has essentially outlived its usefulness.

The problem here is less about git and more about the chasm between Apache and the new culture of open source. There is a growing community of young new open source developers that Apache continues to distance itself from and as the ASF plants itself firmly in this position the growing community drifts farther away.

But while I also subscribe to Clay Shirky’s maxim that “Institutions will try to preserve the problem to which they are the solution,” it is far from clear that it applies in this case. For disclosure purposes, I’ll note here that we count both GitHub and open source foundations like Apache and Eclipse as clients.

The argument that foundations have become vestigial in a post-GitHub world necessarily focuses on functional overlaps. Historically, project hosting has been one of the services offered by foundations. The data suggests that foundations who reject decentralized version control systems will fall behind, which is why structures like Eclipse are implementing Git. Even assuming a given foundation is able to transition from centralized to decentralized mechanisms, however, there is no guarantee that this will be sufficient. Iit’s unrealistic expect any foundation to compete with GitHub on functionality or community size, given the respective areas of focus.

The value of foundations, however, has never been principally hosting. They are, rather, the manifestation of a particular mission. The Free Software Foundation, the Apache Software Foundation, the Eclipse Foundation: all serve as the focal point for a group of developers. It is possible that their respective purposes have been fulfilled by GitHub, but the surging code repository seems likely to make foundations more relevant, rather than less.

As Rogers states:

Apache was founded about 12 years ago, a time when companies were still very afraid of open source and many people in the open source community were very afraid of companies. The world hasn’t changed that tremendously, big companies still use an open source stamp as a marketing tool, commonly referred to as “open washing”, and some in the enterprise are still wary about open source, particularly when it comes to certain kinds of licensing.

But, you would be hard pressed to find a single company that didn’t use some amount of open source software nowadays.

The obvious implication is that as acceptance of mainstream open source increases the importance and relevance of Apache’s mission decreases. I believe this to be incorrect. Aside from the many important non-infrastructure services offered by foundations – including IP management, project governance, legal counsel, event planning, and predictable release schedules – foundations have value as brands.

As the volume of open source assets grows, the paradox of choice presents itself to users. This problem in part already sustains a sizable marketplace of commercial products from vendors such as Black Duck, Open Logic, Palamida, and Sonatype. With GitHub and similar tools reducing the friction associated with development, we are likely to see selection problems get worse rather than better; as the volume of open source rises, fueled in part by GitHub’s success, so too does the difficulty of making a choice.

Which is one reason foundations are important. Much as McKinsey advantages Baker and Rhodes scholars in their hiring process, certain audiences will prefer to leverage open source software associated with a known brand such as Apache or Eclipse.

We have long argued that developers are the new kingmakers. As developers begin to make more choices within the enterprises they populate, they will inevitably face the same dilemma that their management predecessors did, which is the burden of choice. GitHub is a center of gravity with respect to development, but it is by design intensely non-prescriptive and inclusive, and thus home to projects of varying degrees of quality, maturity and seriousness. Consider the following:

GitHub, Inc. (“GitHub”) supports the protection of intellectual property and asks the users of the website GitHub.com to do the same. It is the policy of GitHub to respond to all notices of alleged copyright infringement.

Notice is specifically given that GitHub is not responsible for the content on other websites that any user may find or access when using GitHub.com.

GitHub, in other words, disavows responsibility for the projects hosted on the site. Foundations, conversely, explicitly assume it, hence their typically strict IP policies. These exclusive models offer a filter to volume inclusive models such as GitHub’s.

If your continued employment depends not just on the quality of the software you employ, then, but perceptions of the quality of the software you employ, the halo effect offered by foundations that actively triage their assets is likely to be of benefit. For better or for worse. If you’re choosing between one project of indeterminate pedigree hosted at GitHub and an equivalent maintained by a foundation like Apache, the brand is likely to be a feature. Managers used to say “you won’t get fired for buying IBM.” The developers making the decisions in the future may well have their own version: “you won’t get fired for using Apache.”

Rogers appears to have legitimate concerns concerning Apache’s acceptance of Git, and I concur that it’s a GitHub world and we’re all living in it. But I find it difficult to build the case that foundations more broadly won’t have a role to play in it.

“IBM designed IMS with Rockwell and Caterpillar starting in 1966 for the Apollo program. IMS’s challenge was to inventory the very large bill of materials (BOM) for the Saturn V moon rocket and Apollo space vehicle.” – Wikipedia, IBM Information Management System

While the practice has a long history, the formalization – and more particularly, popularization – of open source is a relatively recent development. Until the process of releasing of Netscape Navigator rallied supporters of the term open source, the industry lacked a consistent vocabulary for dealing with the concept of source code available under a license. Which gave proprietary software the stage by default.

In the absence of a widely understood model for the generation and consumption of shared code, innovation in software was largely the product of commercial vendors producing proprietary software. Starting from scratch, with few exceptions most organizations outside of finance and defense could not reasonably expect to sustainably compete with vendors whose business it was to write software. Rather than try, businesses predictably chose to outsource the process of development to vendors like IBM, Microsoft, Oracle and SAP. This is the dynamic that gave us a PwC Top 10 Global Software leaderboard with a median age of 34.5 years.

With open source systemically lowering the overhead associated with development over the last decade, however, in house development has become increasingly viable from an economic perspective. The inevitable result has been greater availability of open source code. As users and vendors alike discover the benefits to open source – via models direct or indirect – the number of projects has swelled along with the rate of contributions. The database market, for example, has gone from less than half a dozen relevant open source projects to several dozen.

None of which is news even to casual observers of the open source market. What is interesting, however, is how these projects are being developed. In some ways, development today is a return to its roots. Consider the following list of open source projects:

• Cassandra
• Git
• Hadoop
• MongoDB
• Nginx
• Rails

Besides the availability of their source, what do these projects have in common? None were originally authored to be sold. All were built for purpose rather than sale; this is the return of roll-your-own [coverage]. Much as IBM once extracted IMS from an engagement that sent Americans to the moon, each of the above widely used projects was the byproduct of a business problem. Cassandra was written to manage the Facebook Inbox, Git to manage the Linux kernel source tree, Hadoop to power Yahoo’s search indexing, MongoDB to back 10gen’s original Java cloud vision, Nginx to serve pages for Rambler. And as for Rails, it was extracted from Basecamp.

This “extracted” software model is becoming routine. The innovation inherent in these projects, however, is anything but. Extracted software typically exists because a perfect solution does not, which means that in many cases it is introducing new capabilities rather than recreating existing products.

With substantial history behind it, the extracted model seems to be fairly well understood, conceptually. The open question now is about the volume of latent innovation that might emerge from extracted software in the years ahead. Projects like the list above indicate that internal innovation has accelerated over the last decade, driven by trends ranging from the greater availability of open source code to an industry-wide shift towards horizontally scaled-out architectures.

But as concern about the risks of open source thaws and is offset by wider understanding of the benefits, it is probable that waves of new internally developed projects will be released as open source. The majority of which will generate little activity and interest. But from the volume, we might expect the next Git, Hadoop or Rails.

As open source trends go, then, the extracted software model is one to watch.

In making the case that the window for Linux “worldwide domination” in the mobile space has elapsed – its role as the foundation of the open source Android platform notwithstanding – Forrester analyst Mike Gualtieri argues that open source isn’t fundamentally innovative.

Open source never seems to be the innovator. Instead, it seems to disrupt pricing power for established technologies.

There are numerous counterexamples to this; my analyst colleague from the 451 Group Rachel Chalmers cites Unix, others the underlying protocols of the internet and I myself would point to the more recent work that browser teams like Chrome and Mozilla are doing or the pre-Cambrian explosion currently occurring in the non-relational database market. But superficial questions like “can open source innovate” obscure real, fundamental changes in the way that software is being developed today. Changes that are important.

The fact is that neither open source nor proprietary development is intrinsically innovative. The availability of source code has no real bearing on how unique the underlying ideas contained within it may be. What people mean when they say open source cannot innovate is that the profit motive, or more accurately profit models, associated with proprietary software development create greater incentives to innovate.

Which may or may not be true; my own view is that attitudes towards monetization of software generally appear to be somewhat cyclical. Tracing the history of software revenue models from IBM to Microsoft to Google to Facebook, attitudes towards the importance of software as the primary revenue mechanism come full circle [coverage].

In any event, the profit motive is only an incentive for innovation for software that is written to be sold. Which a great deal of software today was not, originally.

What does the following list of projects have in common?

• Cassandra
• Git
• Hadoop
• jQuery
• Memcached
• MongoDB
• Rails

The answer is that none of them were originally written for purpose of sale. Unlike the Oracle database or Windows, these projects – all of which have achieved mainstream success – were written not to be sold but to solve a problem.

This inverts the traditional software startup model, in which a market opportunity is identified and solutions are built towards it.

The more common software development path today is reverse engineering projects from existing solutions. Rails, the framework built by 37 Signals, was a byproduct of the construction of their primary SaaS products (one reason their founders recommend selling your same). Git was written by Linus Torvalds to replace BitKeeper in managing changes to the Linux source tree. 10Gen, the authors of MongoDB, were originally a Platform-as-a-Service company, not a database vendor. Cassandra was built by Facebook to manage their Inbox. Memcached was written by Danga for Livejournal. Hadoop was written to build Yahoo’s search index. And so on.

That many of these technologies have been commercialized after the fact changes nothing about the substance of the original idea. The history clearly demonstrates that the incentives for software innovation may vary widely. The acceleration of this extracted-software model is not entirely attributable to the commercialization of the internet, but certainly that provoked substantial adaptation – much of what we consider innovative in the software infrastructure space has come from the web – while enabling would be sellers of these technologies by making marketing, sales and customer acquisition more efficient.

The reality is that the nature of how, where and why software is developed has fundamentally changed. There are many reasons to develop software. What the incentive is for a given project, and what that dictates with respect to the development model employed, is likely to say less about whether the software is innovative than the nature of the problem it is intended to solve.

Innovation is a function of incentive, not the software development model.

In early July, I ran across Jeffrey Breen’s post on doing sentiment analysis in R a bit before two in the morning. It was interesting enough that I stayed up into the wee hours of the morning digesting his approach, downloading the necessary dependencies in R and outputting very basic sentiment histograms. Since then we’ve included it as a supplement to our research, as in examples like our post on PhoneGap’s marketplace trending.

The approach has its limitations, of course. Breen’s presentation discusses them, as does Nathan Yau when he used the code to compare brand performance for everything from airlines to pizzas. The results can’t be considered representative in a statistical sense, for one, because it’s not technically a random sample. But with the caveats understood, there is evidence that the scoring is good enough to be useful as one of many inputs to decision making processes.

Interesting as this code was, however, it was less than accessible to those without experience using R. Even if you could fully document the process of running an analysis, as I did in internal RedMonk documents, it required a local implementation of R, and more specifically a version that worked with the twitteR library. Which only one of us at RedMonk had.

Given these barriers to entry, we decided to layer a more accessible front end on the back end R script, in this case a web UI. Via Hacker News, we connected with Alex Henning – a college student – and tasked him with building us a simple front end for Breen’s script. He called this Project Blue Bird.

Always intending to open source this code, we needed first to seek the permission of the authors of our two dependencies: Breen’s R sentiment code and the sentiment lexicons (available for download here) it uses that were assembled by Bing Liu and Minqing Hu. Coincidentally – as we’d had no contact with Breen while building Blue Bird – his code was made available on GitHub under the precise license we had intended to use, the Apache 2.0. Which left the sentiment lexicon. Fortunately for us, Professor Liu yesterday graciously gave us permission to include the sentiment lexicon in our repositories. Which means that we are now free to open source Project Blue Bird. On GitHub, obviously. Find it and fork it here.

For those interested in such things, the project was written in Python and uses the low level rpy2 interface to talk to R. Graph output is done via the excellent ggplot2 R package. The README has the complete list of dependencies.

A couple of caveats for those of you interested in using it. The UI is basic. For scheduling reasons, we advantaged function over form. The latency of the application, in addition, is high because the retrieval of matching tweets is slow. We cache what we can, but initial queries are going to take a little bit. And as mentioned above, we caution against reading too much into the sentiment graphs themselves; they’re useful and insightful rather than authoritative.

All of that said, we hope that our community, the R community and anyone interested in general sentiment analysis will find this useful. At RedMonk, we try to give back wherever we can, and if there are developers out there that can use this, that’s a win.

We’d like to thank Alex Henning for his excellent work on the project, Jeffrey Breen for the sentiment analysis code and Bing Liu and Minqing Hu for the lexicon that’s at the heart of this analysis.

Enjoy, and let us know what you think.