Previously, we’ve argued that the lack of breakout successes and the repositioning of vendors like Infochimps suggests that the data marketplace opportunity is years instead of months away. If we assume, for the sake of argument, that this is plausible (Edd Dumbill is more bullish), the question is what this means. The obvious answer is the frictions that are the inevitable consequence of marketplace inefficiencies, one result of which is decreased appetites for software ranging from data management to visualization and analytics. The logic is straightforward, if data remains hard to get, consumption will experience slower growth. Slower growth and less external data, means less demand for tools to process and analyze same.

The less obvious risk, however, may be content monopolies. While these are relatively uncommon at present, they represent an obvious barrier to entry to would be content oriented businesses. Content licensing for IBM’s work around Watson in the healthcare space was inefficient, for example, because the lack of wider marketplace valuations made each negotiation a one off. But as IBM demonstrates with its continued push into the space, the costs and effort are acceptable relative to the size of the opportunity. If IBM were to explore opportunities for Watson in the legal industry, on the other hand, the costs would likely be prohibitive. Unlike other industries, control of content in the legal vertical is largely consolidated in the hands of two players, LexisNexis and Westlaw. This control has allowed them to keep licensing costs for their product, and thereby margins, high. Bloomberg is attempting to compete as a lower cost offering at $450 per seat cost per month. And even Bloomberg, with its business and technical acumen, has an uncertain competitive path ahead of it. Superior technology or no, its access to content will be asymmetrical, limited by the exclusive licensing agreements enjoyed by LexisNexis and Westlaw.

It may be that would be content monopolists in other sectors find market sentiment a significant impediment to their efforts to consolidate control. Reed Elsevier – the parent company of LexisNexis – is facing a boycott of better than 8400 researchers for its business practices relating to scientific journal subscriptions. But the economic incentives are high enough that speculators, at least, are likely to try and anticipate some of the content areas that will be in demand. From real estate to science, data will be at a premium. And without efficient markets to smooth distribution, it is probable not only that overall costs will be higher, but that certain areas will look a lot like legal: sufficiently costly so as to make data oriented businesses and greater competition impractical.

“After six months of development, Grit had become complete enough to power GitHub during our public launch of the site and we were faced with an interesting question:

Should we open source Grit or keep it proprietary?

…After a small amount of debate we decided to open source Grit. I don’t recall the specifics of the conversation but that decision nearly four years ago has led to what I think is one of our most important core values: open source (almost) everything.”
- Tom Preston-Werner, “Open Source (Almost) Everything

It should be clear at this point that commercial valuations of software assets by both practitioners and public markets is in decline. This model of ours, once controversial, is now self-evident. We examined the public markets aspect in detail in May; little has changed since.

Of PwC’s Global Software Top 20 – the top twenty firms globally as measured by software based revenue – the youngest is 22 years old, and the median age of the top ten is 35 years. It has been over twenty years, in other words, since the software industry produced a business to rank in the Top 20.

Here is a chart of the market capitalizations of PwC’s top five vendors.

If we remove the artificially narrow lens of revenue from software sales, however, the picture looks very different.

The technology industry has produced entities that would place within the top five; they just aren’t in the business of selling software.

Firms that make money with software rather than from software are outperforming their counterparts in recent years; compare the respective valuations of Google, Red Hat and VMware.

Generationally, attitudes towards software are shifting, as evidenced by Werner’s comment above, and the release as open source of assets like Cassandra, FlockDB, Hadoop, Hip Hop, Hive, Jekyll, Nginx, Pig, Resque, Storm, Thrift and so on. Entrepreneurs and public markets alike are turning their attention away from software sales and towards data oriented revenue models in search of outsized returns. There is and will continue to be an enormous market for software, but real growth is increasingly coming from areas other than software sales and service. This is true even for those currently in the business of selling code; for Sonatype and many other startups, data is increasingly the product.

As obvious as this is to most industry participants, however, the Age of Data remains hobbled by its lack of a free market.

In the early days of commercial open source consumption, the exploding number of so-called vanity licenses – those that were non-standard and vendor specific – and a general lack of understanding of their legal implications inhibited the market for open source, relegating it in many settings to unacknowledged, behind the scenes usage. Gradually, however, license proliferation was curbed and licenses generally began to coalesce around permissive (Apache, BSD, MIT, etc) or file (EPL, MPL, etc) / project (AGPL, GPL, etc) style reciprocal licenses. The net impact of which was increased adoption, because the previous uncertainty necessarily implied risk which in turn throttled adoption.

Which is essentially where the data market is today. Everybody understands that data has value; there is little consensus on how, where and via what mechanisms it should be distributed, licensed and sold. Startups like Buzzdata, Datamarket, Factual and Infochimps cannot by themselves make a market, and with rare exceptions like Microsoft with the data section of its Windows Azure Marketplace, large providers tend to be heavily risk averse with respect to the liabilities posed by data.

Absent a market with well understood licensing and distribution mechanisms, each data negotiation – whether the subject is attribution, exclusivity, license, price or all of the above – is a one off. Without substantial financial incentives, such as the potential returns IBM might see from its vertical Watson applications, few have the patience or resources to pursue datasets individually. We’ve experienced this firsthand, in fact; as we’ve looked for data sources to incorporate into RedMonk Analytics, conversations around licensing have been very uneven. After gaining verbal approval from one content provider for our proposed usage of their API, this was the message we received in response to our request for an actual license:

The short answer is… we are not exactly sure what you are planning to do and whether or not it would violate our terms of service, so the best answer I can give you is that if you are not sure, you’re best off consulting with your own attorney.

As a rule, we don’t have the staffing resources internally (or the legal resources, given the high cost of lawyers!) to make one-off contracts or licenses or exceptions to the generic terms of service.

Hope this helps!

Given the current state of data licensing, it’s impossible to blame them for taking this position, even as it prematurely terminated what could have been a mutually beneficial relationship. Why dedicate legal resources to license review when the projected return on the asset is uncertain? Nor would a commercial negotiation be any more straightforward.

Market efficiency is a function of volume; the more participants, the better an understanding we have of an asset’s worth. With limited mainstream market participation in the business of data, opinions of the value of a given asset tend to be asymmetrical. Put more simply, when we ask what a given dataset is worth, the only correct answer at present is: we just don’t know.

All of which helps explain why Infochimps and others startups targeting the data marketplace opportunity are not as visible as their significance suggests they should be.

For all of the current inefficiency in data procurement, however, it is a temporary condition. Apart from the fact that history tells us risk assessment and licensing are solvable problems, the financial incentives are sufficient to guarantee progress if not complete solutions. Consider the case of Watson. IBM has historically avoided data collection due to legal concerns, but imagine the liabilities of the first diagnostic miscalculation. What will happen, in other words, when Watson commits the healthcare equivalent of “What is Toronto?” while assisting in the diagnosis of a patient? For IBM, the answer is clearly that the potential rewards more than offset the theoretical risk. Enough so, at least, to justify massive investments in development and marketing.

Life may be marginally easier for those that capture their own data, but this will by no means diminish the appetite for more. Google generates sufficient data to be able to predict the flu better than the CDC, but still felt obligated to license Twitter’s data. A relationship that ended, notably, because of a license termination. No matter how valuable an internal dataset might be, it will be more valuable still when recombined, remixed and correlated against complementary external data; see, for example, FlightCaster.

With respect to opening the throttle for data marketplaces, escalating demand virtually guarantees the supply. The real questions are who will play a meaningful role in reducing the friction, and when. Because the attendant opportunities are large indeed.

Disclosure: IBM, Microsoft, Red Hat and VMware are customers. Datamarkets, Facebook, Factual, Infochimps, Google and Twitter are not.

“Most of my friends at some point have heard me say, ‘In hindsight we are either geniuses or idiots.’ It’s particularly true for someone like me who makes a living writing and analyzing things.” – Om Malik, “The Perils of the Prediction Game”

Failure is part of predictions. Certainty demands comprehensive intelligence, which is by definition absent in the business of forecasting. But in prediction, the question isn’t whether you fail: that’s expected. The metric by which success is determined is rather how often. That is what separates the genius from the idiot.

Let us score last year’s predictions, then, so that you may know whether to pay attention to the 2011 iteration.

Cloud API Proliferation Will Become a Serious Problem

The December partnership between Eucalyptus and Red Hat highlights the issues with API proliferation. Granted, for Eucalyptus, the deal was likely driven more by pragmatism than customer pull with respect to Deltacloud. But the fact that an API superset was discussed on equal footing with the partnership itself speaks to the central role APIs are taking in cloud deployments.

Google Trends validates the assertion that 2010 saw increased activity around cloud APIs.

From Screenshots

This activity – which led, in part, to the rebranding of Sonoa – represents interest, yes, but also the challenge. As the software industry reorients around the cloud, APIs become their primary customer interface. Which, given the proliferation of vendors in the space, is problematic today and likely to become more so: an API per provider is an approach that will not scale.

I feel safe counting this one as a hit.

Collaboration Will Never Be the Same

Scoring this prediction is a challenge. The cited projects – Google Wave and Mozilla Raindrop – have failed to deliver, with the former officially deprecated and the latter effectively so.

Nevertheless, collaboration did fundamentally change in 2010. Priority Inbox introduced algorithmic email ranking to the masses. Tungle continued to facilitate scheduling. Rapportive injected social networking into Gmail. Dropbox launched synchronized file sharing capabilities for teams. New Twitter fostered conversations on the service. And Facebook has promised to reinvent communication with its forthcoming Facebook Messages offering.

I’ll call this a push.

Data as Revenue

To the best of my knowledge, we have yet to see data recognized as a balance sheet worthy asset. That’s a miss.

And yet with Goldman valuing Facebook as a $50B entity on the basis of a reported $2B in revenue, it’s clear that the value of data is well understood, if implicitly. The venture community appears to agree, funding data marketplace startup Infochimps to the tune of $1.2M.

I’ll call this one a miss, if only because the prediction, aggressively stated, was unfulfilled. I stand behind the conviction, however, and I think the market bears it out.

Democratization of Big Data

Consider that RedMonk, a four person analyst shop, has the technical wherewithal to attack datasets ranging from gigabytes to terabytes in size. Unless you’re making institutional money, budgets historically have not permitted this. The tools of Big Data have never been more accessible than they are today.

Hadoop, in paricular, has made remarkable strides over the past year, with improvements to core, projects such as Hive evolving and others like Oozie emerging. The center of gravity around the infrastructure is remarkable, with a few properties even bending it towards traditional back end web roles, ill suited as it is for that generally.

Nor is Hadoop the only example of the ready availability of tools. Non-relational stores continue to evolve at an impressive rate, with hardware access not far behind.

This one is a hit, as far as I’m concerned.

Developer Target Fragmentation Will Accelerate

If anything, I didn’t predict this aggressively enough. There is observable fragmentation in virtually every area of infrastructure technology: operating system, database, programming language, development framework, mobile platform, cloud platform, cloud API [coverage]. Even hardware is less straightforward than it once was, with ARM a driver in mobile with larger potential.

All of this has happened before, of course: the Cambrian explosion of diversity we’re currently witnessing is not unprecedented. It is, however, potentially further reaching as the boundaries between consumer and enterprise break down, and distinctions between on premise and network hardware blur.

This is a hit.

It’s All About the Analytics

The launch of Tableau Public in February [coverage] and its subsequent success is enough to validate this prediction by itself. The introduction, a month later, of the Google Public Data Explorer is gravy: analytics are fast becoming ubiquitous. It’s hard to miss infographics in the New York Times or the Economist, and as the data from Indeed.com indicates, hiring in the analytics space spiked in the last twelve months.

From Screenshots

This is a hit.

Marketplaces Will Be Table Stakes

When browser makers feel compelled to launch marketplaces effectively rebranding websites as applications – as Google did in May – it’s safe to conclude that marketplaces are table stakes. We’re even seeing marketplace proliferation within a single ecosystem, in Android. Vendors are at last perceiving the demand for marketplace features – centralized discovery, standardized payment, etc, but the real driver is the realizable revenue they represent, model depending.

In other words, if you have platform ambitions, you’d better be bringing a marketplace. So this is a hit.

New Languages to Watch

This is a miss. The conclusion is partially correct: in JavaScript, a language did emerge and begin separating itself from the pack. But the languages cited – Clojure and Go – didn’t realize their potential in 2010. Clojure remains popular amongst a niche audience – it was roughly as popular as Haskell and Scala, in our survey of Hacker News – but Go is more or less invisible from a general usage standpoint.

Ergo, the miss.

NoSQL Will Bid for Mainstream Acceptance

Even if we do not include Hadoop, which has clearly crossed the chasm towards mainstream adoption, it’s clear that NoSQL is finding fertile ground in mainstream enterprises. Most NoSQL vendors and technologists we work with have done substantial work over the past year for name brands. From Riak deployments at Comcast to MongoDB instances at the New York Times, it’s safe to say that NoSQL is making a bid for mainstream markets.

This is a hit.

I also made five brief, FOSS specific predictions.

1. Prediction: Usage of dual licensing will continue to decline, in part because of the Oracle and EU dispute over MySQL.
   Comments: Dual licensing, which is distinct from what is commonly called dual core, is in evident decline. With the exception of long time practitioners such as MySQL, dual licensing is becoming increasingly rare. I’ll call this a hit.
2. Prediction: FOSS advocates will increasingly turn their attention from licensing to the related mechanisms of copyright and trademark.
   Comments: Copyright and trademark have both taken center stage over the past year; the former most obviously with respect to the acquisition of MySQL via the Sun transaction, the latter most recently with Oracle’s mishandling of the Hudson assets. This is a hit.
3. Prediction: Permissive licensing will continue to gain at the expense of reciprocal licensing, albeit slowly
   Comments: Comparing the Black Duck license adoption figures I have at hand (meaning no January 2010 numbers), since August 2009 I have the GPLv2 down 4.02% (though v3 is up 1.34%), while Apache is up .65% and MIT 3.9%. This reflects our experience and our expectations, with permissive licenses trending up and reciprocal licenses trending down. Hit.
4. Prediction: The value of project code will be eclipsed, in a few cases, by the data the project generates.
   Comments: This is more difficult to assess. Much of Facebook’s software infrastructure, as an example, is made available via open source, indicating that for that organization, software is non-differentiating. The selection of Hbase over Cassandra as the foundation for Messages serves as further validation of the substitutability of the underlying infrastructure. But neither example yields hard valuations of the underlying code, making an assessment of this prediction difficult. This one’s a push.
5. Prediction: Open source, building from its mainstream acceptance, will emerge as the most credible alternative to proprietary cloud and mobile platforms
   Comments: At the infrastructure-as-a-service level, Cloud.com, Eucalyptus and OpenStack are the primary alternatives at this point to roll-your-own cloud offerings. All make at least a portion of their source available. At the platform level, open source frameworks such as Django (Djangy, GAE), Rails (Heroku, Engine Yard), and Spring (VMware) are emerging as developer preferred alternatives to vendor specific platform implementations. This is a hit, in other words.

The Final Tally

Scoring the 2010 predictions, then, we get six of nine correct for the main category against two misses and one push. Amongst the FOSS specific predictions, four of five were hits with one push. A sixty-seven percent success rate in non-FOSS predictions may seem unimpressive, but anything better than fifty percent can produce market value so I am comfortable with the results.

If you enjoyed last year’s predictions, then, please stay tuned for the 2011 version.

Disclosure: Apigee, Basho (Riak), Black Duck, Eucalyptus, and Red Hat are RedMonk customers. 10gen (MongoDB), Facebook, Oracle and Google are not.

When we’re talking about data, we typically start with open data. What we mean, in general, is access and availability. The ability to discover and procure a given dataset with a minimum of friction. Think Data.gov or Data.gov.uk.

The next logical direction is commerce. And while this idea isn’t high profile at the moment, at least outside of data geek circles, it will be. Startups like Data Marketplace, Factual, and Infochimps perceive the same opportunity that SAP does with its Information On Demand or Microsoft, with Project Dallas.

What follows commerce? What’s the next logical step for data? Github, I think. Or something very like it.

In the open source world, forking used to be an option of last resort, a sort of “Break Glass in Case of Emergency” button for open source projects. What developers would do if all else failed. Github, however, and platforms with decentralized version control infrastructures such as Launchpad or, yes, Gitorius, actively encourage forking (coverage). They do so primarily by minimizing the logistical implications of creating and maintaining separate, differentiated codebases. The advantages of multiple codebases are similar to the advantages of mutation: they can dramatically accelerate the evolutionary process by parallelizing the development path.

The question to me is: why should data be treated any different than code? Apart from the fact that the source code management tools at work here weren’t built for data, I mean. The answer is that it shouldn’t.

Consider the dataset above from the US Census Department, hosted by Infochimps. Here’s the abstract, in case you can’t read it:

The Statistical Abstract files are distributed by the US Census Department as Microsoft Excel files. These files have data mixed with notes and references, multiple tables per sheet, and, worst of all, the table headers are not easily matched to their rows and columns.

A few files had extraneous characters in the title. These were corrected to be consistent. A few files have a sheet of crufty gibberish in the first slot. The sheet order was shuffled but no data were changed.

Translation: it’s useful data, but you’ll have to clean it up before you go to work on it.

What if, however, we had a Github-like “fork” button? Push button replication of dataset, that would presumably have a transform step applied with the resulting, modified dataset made as freely available. Rather than have to transform the dataset, then, I merely picked up one that someone else had already taken the time to clean.

What are the problems with this approach?

• Data Quality/Provenance:
  Discussing this idea with a few data people, they expressed skepticism because the transform might negatively affect the dataset. False or inaccurate data could be created inadvertently, or worse maliciously, to create a damaged dataset from which the wrong conclusions could be drawn. But again, is this any different than code? Why should bad data be any easier to insert into a dataset than a virus into a codebase?

  But for the cautious, each fork could presumably include a record of the specific transforms such that a user could replicate the steps and check the result against the fork for reassurance.

• Data Ownership:
  Some data owners would undoubtedly be opposed to free public transformation of their datasets, particularly because it could pose pricing issues in commercial scenarios. Who gets paid if you’re not buying the original, commercially priced dataset but a cleansed version of same? Still, the solution to this is simple if less than ideal: disallow forking on select datasets.
• Storage:
  With a great many datasets of manageable size, duplication via forking would be less of an issue than is commonly supposed. There is little question, however, that for large datasets (1 GB+), rampant forking could pose substantial infrastructure costs.

I’m sure there are others. Still, the future to me in this area seems clear: we’re going to see transformation of datasets incorporated into the marketplaces. As the demand for public data increases, the market will demand higher quality, easier to work with data. With that demand will come supply, one way or another. There’s little sense in having each individual consumer of the data replicate the same steps to make it usable. The question will be which one of the marketplaces learns from Github and its brethren first.

Expect collaborative development to beget collaborative analysis, in other words. Soon.

For about five years now, I’ve been anticipating the rise of application and developer marketplaces. It baffles me that no one has perceived commercial opportunity in the massive inefficiencies in a customer’s ability to discover, procure and purchase apps or services for a given platform. I’ve done Q&A’s on the subject, I’ve used speaking engagements to evangelize the topic, and I’ve even articulated the pieces you might need and how most of them are available from Amazon. What has been the response to my business model for the 21st century?

Crickets, mostly.

Mobile’s an obvious exception. Once Apple built themselves a (massively successful) marketplace, vendors couldn’t build their own fast enough. And we’ve seen steps in the right direction here and there, such as the Eclipse marketplace or the Ubuntu Software Center. But the enterprise market has largely yawned at the concept of marketplaces,

But as of Tuesday, it would appear I finally have an answer to this question. The answer, at least for now, is Google.

As predicted by the Wall Street Journal last month, Google announced this week the immediate availability of what they call the Google Apps Marketplace. To explore what this means, let’s turn to the Q&A.

Q: Before we begin, do you have anything to disclose?
A: For once, not a ton. Some of the vendors who might be impacted by this announcement, such as IBM and Microsoft, are RedMonk customers, as are the mentioned Canonical and Eclipse, but Google is not a RedMonk customer, nor is Amazon. RedMonk, however, is a Google Apps customer.

Q: What is a marketplace?
A: There is no set definition, because it depends on the nature of the platform. But here are some rough stages of evolution for marketplace type offerings.

• The Repository:
  An aggregation of applications, libraries, themes and other packages appropriate – and generally packaged for – a given platform. Optimally, installation is integrated as in the case of Linux distributions (apt, Portage, YaST, Yum, etc), but it may simply be a central portal collecting related assets, as is the case with WordPress. Users can browse, discover and (optionally) install free applications which will improve their platform, but the experience essentially ends there.
• The Store:
  The Store also aggregates platform related applications and packages, but introduces the commercial element. Paid applications are made available, and generally the purchasing process is integrated. The canonical example in this category is the iTunes Application store. Users can browse free and paid applications using the interface, acquiring/purchasing and installing either type.
• The Marketplace:
  The Marketplace – a true marketplace – includes all of the above, but adds one critical element: people. While Repositories or Stores make the process of acquiring and installing applications more efficient, they offer little to no benefit subsequent to that process. Need help with set up? Configuration? Tuning? Integration? You’re on your own: try Craigslist or eLance. A true Marketplace, however, adds the critical element of people to this equation, connecting customers to individuals or companies with related expertise.

Q: And you think Google’s built a true marketplace?
A: They appear to be pretty close, yes. It’s not quite a wide open marketplace for human resources yet; there don’t appear to be multiple individual implementers for the different software options, for example. But just look at the Small Business Implementation services: lots of small shops in there competing on pricing for implementation. And that single example illustrates to me why this is a good idea for everyone involved.

Q: How’s that?
A: Consider the small business market. Many SMBs are using ISP based email to run their business, which is why you email them at something like joesflowershop67@comcast.net. They’re doing this not necessarily because they believe it’s the best solution for their business, but because everything else is too hard. But what if you could tell them that they could have mail, calendaring, online documents and everything else that comes with Google Apps for around $20 or $30? How many would go for that? Quite a few, I’d bet, and I doubt Google would take that bet.

But the set up for Apps, which involves making changes to domain MX settings and such, is a little too much to ask from the kind of people that are trying to login to Facebook from Google.

For someone who’s experienced at it, however, the setup is trivial: about 30 seconds work. So you arbitrage the delta in difficulty and come up with a fair price (as determined by a competitive market), and everyone wins. The SMB gets a better infrastructure at a very reasonable cost, the implementer gets a decent margin on a few minutes worth of work. Google, meanwhile, gets a $100 per developer/org and customers that may or may not pay them but will provide them with more data in the meantime while not using a competitor’s solution. Everybody wins.

Q: But as popular as Gmail and Apps might be, isn’t their ecosystem dwarfed by that around products like Exchange?
A: Undoubtedly. And that’s one reason the Marketplace is so interesting.

How do you find solutions if you’re using Exchange? Where do you go? If you’re relatively knowledgable, you might think to visit the Exchange homepage and search for partners. Which might lead you here, to an uninspiring partner search page. If you keep hunting, you might wind up at Microsoft’s version of the Solution Marketplace. Perhaps from there I find a listing: what are my options at that point? Show a phone number, send an email inquiry. The pricing? Per seat, that’s all I’m told.

Contrast that with similar products from its Google counterpart, where the pricing is described, you can read actual reviews of the product or services and – most differentiating – you can click to install the product. The SaaS nature of Google Apps has its disadvantages, particularly for conservative buyers, but ease of installation is not one of them.

Q: The primary advantage of Google’s Marketplace then, vs something like Microsoft Exchange, is in product acqusition?
A: The advantages of a true marketplace as Google’s built it exist at every layer: inefficiencies are significantly reduced or eliminated in discovery, acqusition/implementation, purchase, and experiental ratings/context.

Q: So do you think that Google’s community is going to approach the size of Microsoft’s soon?
A: Rome wasn’t built in a day, and Google’s got a long way to go before it can rival the size and scope of competitive communities like Microsoft’s. That said, if I was competing with Microsoft, this is how I would do it.

Q: What didn’t Google get right?
A: It’s a little difficult to discover more general resources: I have questions or needs, say, around complex spreadsheets, or the construction of templates in Docs. It’s also vendor focused at the expense of individual practitioners. Nor is it obvious what the Google Qualifications mark is. And as far as the applications implementation experience goes, it can be a bit clunky: our TripIt installation is still hung up.

Q: What’s in it for developers?
A: Access to the market. Say you do Google Apps implementations; how do you do customer acquisition? Locally or regionally, you might employ traditional mechanisms such as print or radio, but it’s probably just as easy for you to set up apps for a customer in Auckland as it is next door. By listing yourself in the marketplace, you have the opportunity to access a wider and more targeted market than you would with, say, AdWords.

Q: What about the competition? Won’t it be difficult to be heard? How differentiated can you be as someone who sets up Apps, for instance?
A: That’s an excellent question, and the answer is we don’t yet know. It will be interesting to see how Google manages the volume of developers. They’re already acting to throttle participation with the one time $100 signup fee, which should keep out most of the drive-by, non-serious implementers.

But whether they manage the volume tightly or loosely, this will be an important channel, simply by virtue of the volume that apps can generate.

Q: What about the applications space? Is it in a provider’s best interest to list with Google?
A: Why wouldn’t you? Google’s claiming that 2M businesses have already chosen Apps as their platform, and even if that assessment is inflated by 50% you’re still talking about a sizable market. Even better, a sizable market that is likely underpenetrated relative to more traditional marketplaces.

Q: Underpenetrated how?
A: The businesses running on top of platforms from IBM or Microsoft have had years, decades in some cases, to build up solutions around Notes or Exchange. Google Apps, on the other hand, is a relative greenfield simply because the integration of applications on the server side is new. It’s kind of a gold rush, in that respect.

Q: Big picture, what does all of this mean?
A: Let’s look at it by audience. For the customer, it means either a streamlined route to implementing Google Apps, an exponentially more accessible portfolio of resources once you get there, or both. For developers or application providers, it’s a relatively less crowded channel opportunity, one where the infrastructure for both installation and payment is already built for you. And for Google, it’s nothing less than an opportunity to dramatically expand their ecosystem, which – in addition to the obvious direct benefits in terms of customer acquisition and partner opportunities – acts to reassure conservative enterprise buyers that this platform is for real.

Q: And for competitors?
A: It means that they should have been building marketplaces all along, as I’ve been saying

Market Place TTV, originally uploaded by cybertoad.

“An app store lets companies tap into ideas from third-party innovators while retaining firm control over their brands. And that’s both its charm and its flaw. “The way Apple runs the App Store has harmed its reputation with programmers more than anything else they’ve ever done,” wrote Paul Graham, cofounder of the venture firm Y Combinator, on his blog.

That’s a miscalculation, because the App Store’s true rival isn’t a competing app marketplace. Rather, it’s the open, developer-friendly Web. When Apple rejected Google Latitude, the search company’s nearby-friend-mapping program, developers created a nearly identical version that works perfectly on the iPhone’s Web browser. Google looks to be doing something similar with Voice, another app that Apple barred from its store. Last fall, Joe Hewitt, the Facebook developer who created the social network’s iPhone app, quit developing for Apple in protest of the company’s policies. Where did he go? Back to writing mobile apps for Web browsers.” Farhad Manjoo, “Why App Stores Are Not the Business Model for the 21st Century“

Manjoo gets a lot right in this piece, as Nick Carr notes. Which is why it’s disappointing that I must disagree so fundamentally with the thesis.

Even in the wake of the endlessly hyped iPad announcement, which is certain to extend Apple’s developer footprint, it’s difficult to take issue with his characterization of the Apple application approval process as “heavy handed.” Or with the fact that approval is necessary in the first place.

But while the piece is purportedly concerned with the idea of app stores as a concept, in reality it’s an indictment of Apple’s implementation. Nothing more, and nothing less. Why must an application catalog and decentralized management, for example, be mutually exclusive?

There are any number of vendors who will be unpleasantly surprised by the failures of their application store intitiatives, having followed – to the letter, in many cases – Apple’s example. Few would debate the assertion that the Apple application store has been successful in spite of, rather than because of, the byzantize and frustrating application approval process. It’s a testament to Apple’s brand, in fact, that the store has been successful at all. Companies that simply reimplement Apple’s app store using their brand will fail.

At which point there are two potential responses: abandon the idea as fundamentally unworkable for entities that aren’t Apple, or iterate. I think we’ll see more of the latter, because we need app stores. Seriously.

Not as a gatekeeper, but rather a filter. Manjoo talks about the promise of the “the open, developer-friendly Web” without mentioning the downside: choice. Consider the example of WordPress, which is about as open and developer friendly as software gets. Early in the project’s lifecycle, plugins – which can be considered the WordPress application equivalent – were spread all over the web.

Want a spam reduction plugin? Great. Hit Google, there are dozens. Literally. How to pick one? What did users think of each? What versions of WordPress were they compatible with? All excellent questions: good luck answering them.

Enter the “app store,” better known as the Plugin Directory. It’s far from perfect, but at least it streamlines the plugin acquisition process by centralizing the choice. On top of that aggregation is layered the kind of metadata beneficial to users: reviews, rankings, compatibility data, and so on. And nary a gatekeeper among them.

Nor is this type of application inherently incompatible with commercial interests; it’s easy to imagine any number of potential revenue streams from this type of marketplace. Advertising, data sales, preferential listings, support and service integration, etc. That Automattic chooses not to pursue this course of action means little about its commercial viability, generally.

If Manjoo wants to argue that heavily walled and gated marketplaces are not the business model for the 21st century, more power to him. But to assert that marketplaces in general are doomed because Apple’s approach has its issues is not only an inappropriate conflation of concept and execution, it ignores what we see from consumers every day: a passionate affinity for marketplaces.

Which is why I think we’ll see more, not less, even if they look different.

sad iPod, originally uploaded by benjibot.

“Middle men exist to reduce the cost of getting a product from A to B, and as long as that cost is significant, they will be useful. However, the moment the middle man monopolizes the means of distribution, he becomes a gatekeeper, and creators can be made to fail not by the merits and popularity of their products, but by the whims and short-term interests of the gatekeeper.” – “On Middle Men,” Joe Hewitt

There are 474 words in Joe Hewitt’s call to action. Apple is not one of them. If anything, the omission speaks more loudly than a mention ever could have.

The backdrop of the piece is this: last week, Hewitt made the decision to hand off the development of the Facebook iPhone application to another developer. His reasoning was equal parts disappointed and realistic:

My decision to stop iPhone development has had everything to do with Apple’s policies. I respect their right to manage their platform however they want, however I am philosophically opposed to the existence of their review process. I am very concerned that they are setting a horrible precedent for other software platforms, and soon gatekeepers will start infesting the lives of every software developer.

For those that might argue that this is a bluff – a futile attempt to influence Apple’s behavior by boycott – two things argue against it. One, Hewitt acknowledges that his decision will make effectively no difference on the popularity of the platform for developers. Two, he’s right.

If Android can avoid death by fragmentation, a Unix wars of its own making, it will be, in time, a credible alternative to the iPhone platform. Palm may be as well, if it can address its anemic hardware and its developer relations issues; certainly they have two of the best in the business in Ben Galbraith and Dion Almaer for the latter. RIM will remain strong within the enterprise, but I’ve seen little to convince me that they get the consumer. As for Nokia, my view is that their present, at least in terms of marketshare, looks much brighter than their future.

Until one or more of the above can deliver a platform with the same polish and application volume that Apple has, however, they’re really just squabbling over second place. Not a close second.

Which makes the questions that Hewitt raised important. Not new, of course. This is how I described the App Store last February:

Apple seems to have figured this out in its iTunes store, which is succeeding wildly in spite of the fact that it’s governed by policies that are fundamentally broken at every level.

Nor was that exactly a revelation at the time. That iTunes has succeeded is a testament to just how impressive the device was designed initially and the power of the Apple brand.

Apple, for its part, doesn’t comment much on the application approval process. Perhaps believing, correctly, that in the face of the application store’s continued success, there’s no upside to a more candid stance.

This is all too bad. Not because it impacts my choice as an Apple customer and iPhone user. Not because it’s resulted in hundreds if not thousands of wasted hours from developers who find themselves indicted, like Josef K, for crimes that are never revealed to them.

It’s a shame because it could, by proxy, leave marketplaces as collateral damage.

Hewitt, to his credit, acknowledges that middle men can add value. And indeed they do, as I use them daily. As does anyone who uses apt, or Portage, or Yum. What is package management but a middle man, sitting in between me and the developer as it does.

One of Apple’s stated objections to a more open marketplace is the stability of the device. Which, in Apple’s defense, is a more credible point than is commonly acknowledged. Many of us will experiment with laptops and servers in ways that we would not with a phone, given the differences in the relative importance and substitutability between them.

But the perceived tension between a stable platform and less stringently vetted software need not exist. For all that it is a real problem, it is also a solved problem. Consider the Linux distributions’ approaches. Debian uses distributions, Ubuntu PPAs, Gentoo overlays overlays, and so on. Or, closer to home, there is the seriously relevant example of jailbroken iPhones. Which Apple, predictably, claimed were illegal.

Is it so difficult to imagine a world in which Apple could officially enable jailbroken applications with the caveat that support ends the minute that takes place? It shouldn’t be, but probably is.

The point here is as simple as it is obvious: Apple’s justifications are belied by the evidence, which indicates that the problem they are solving is in fact no problem at all. Which leaves their secretive and opaque application approval process as an artifact of their own culture of intense control, their codependent relationship with AT&T or a combination of the two. None of which consumers or developers should have to care about. Who’s actually in favor of the Permission Based Web?

Ultimately, Hewitt and I might differ on the potential value add of middlemen, because I am a firm believer in the value of marketplaces and centralized distribution. Never more so than after using the package management-less OS X, in fact.

But what we agree on, at least as far as I can tell, is that it would be a real shame if a bad Apple ruined the whole bunch because they were intent on preserving their monopoly on distribution.

“The only thing we know about the future is that it will be different.” – Peter Drucker

So let’s do some predictions, shall we? True, I dislike the entire business of prediction, close cousin that it is to guessing. Which I hate. But James’ excellent thoughts on what we might see in the year ahead got me thinking about what I’m anticipating.

Maybe we see over the hill imperfectly, but the following assertions are not without their substance either. Feel free to take them with a grain of salt, several grains, or not at all. We’ll see how we did a year from now.

One thing to keep in mind about our predictions: we’re looking a bit further out than, say, Gartner. Where they are predicting that cloud computing will a strategic technology for 2010, then, we instead consider that a given. So if you’re looking for predictions like, “open source will be a mainstream option,” you’ve come to the wrong place: we figure you know that already. It doesn’t mean that Gartner’s wrong, of course; merely that we’re having an entirely different conversation.

Cloud API Proliferation Will Become a Serious Problem

When I meet with cloud providers these days, the default answer to questions about the openness or lackthereof with respect to their software is “we have an open API.” But this is, unquestionably, the wrong answer for customers. It’s not that open APIs are bad, individually: far from it. You’d rather have one than not. But how are customers to manage them as they multiply? Cloud providers should be considering Kant’s Categorical Imperative:

“Act only according to that maxim whereby you can at the same time will that it should become a universal law.”

Unsurprisingly, however, they are not. Which means that cloud API proliferation will reach new, frightening heights in the year ahead. Or maybe you want to individually review and compare the APIs as they iterate. Watch the Deltacloud project for traction as a result; platforms with an API compatibility story like Eucalyptus should benefit as well.

On a semi-related note, I expect IaaS to remain more popular than PaaS for 2010.

Collaboration Will Never Be the Same

Google Wave was quite a splash when it landed; Mozilla Raindrop far less so. Or so says Google Trends. But both will play an important role in the fundamental reshaping of the interfaces – and in the case of Wave, infrastructure – that we all use to collaborate in the year ahead. Nor will the impacts be limited to the early adopter market those products are aimed at. As James noted, Lotus sold 1M licenses of its Connections product in two weeks to six customers. The appetite for next generation collaboration toolsets is strong, whether we’re talking about Rogers’ innovators or laggards.

But all of that may end up being the least interesting trend we see from collaboration in 2010. Of potentially greater impact are those that go beyond the interface. Github, for example, strongly incents social coding and cross pollination in ways that change the way development is done. Offerings like Gist and Threadsy, meanwhile, take a business intelligence-like approach to email, attempting to both consolidate multiple streams and process the content algorithmically according to its inter-relation. Message from your boss? Important. Someone you hear from once every two months? Less so. Neither are ready for primetime, by my testing, but they point the way forward.

And collaboration will never be the same.

Data as Revenue

I’ve written about this fairly extensively already, so I won’t belabor the point. But we’re going to see datasets increasingly recognized as a serious, balance sheet-worthy asset. Twitter pointed the way with its Bing and Google deals, and then Infochimps reinforced that value by making available, commercially, data they harvested from everyone’s favorite micro-blogging service.

This will continue. I’m fully in agreement with IBM’s Steve Mills when he says that we’re “moving into an era of information led transformation.” As margins slim and economies continue to stagnate, enterprises of all sizes will increasingly turn their eyes to data based assets, both for their latent commercial value as well as for improved decision making.

As a result, fear and concern over the privacy implications will spike.

Democratization of Big Data

Yes, Facebook and its 24 terabytes of new content per day is an outlier. But what about the individual developer that wants to make sense of the 1.7 GB Twitter dataset that Infochimps is making available? OpenOffice.org, as I can personally report, doesn’t want anything to do with it.

Fortunately, the democratization of big data is well underway. Hadoop puts MapReduce within reach, Pig puts Hadoop within reach, and with the Cloudera desktop you even have a nice, shiny browser based GUI. Throw in Amazon, and you have as many machines as you could possibly want. We’re still a little light in the front end space, with the ability to visualize the data lagging far behind the ability to process it, but that will come. Maybe in the next year, maybe not.

But either way, the ability to work on big data will increasingly be available to any business, large or small. Democractization of Big Data, commence.

Developer Target Fragmentation Will Accelerate

Between cloud fabrics, programming language proliferation, mobile application development and the spike in development framework popularity, development targets have been fragmenting for several years now. We are more or less in full retreat from the one time promise of write once, run anywhere as an industry. I see nothing on the horizon that will throttle or even slow this trend; if anything, the increasing volume of cloud platforms and the surge in interest in mobile development will accelerate this trend.

This has significant implications for purveyors of middleware, application development tools and cloud platforms, but also for those charged with setting enterprise technology standards. The CIO’s job is going to get harder in 2010, because picking a winner from the myriad language, framework and platform options will be much more difficult than picking a safe option.

It’s All About the Analytics

Flowing Data, a blog run by PhD candidate Nathan Yau, is one of my favorites. The visualization of data is as much an art as a science, and there are few practitioners more talented. The challenge of taking data and hammering it into a form that conveys meaning and supports conclusions is, of course, an age old challenge. But the tools at our disposal are getting better, fast.

Consider the simple analytics that are available, for free, to virtually anyone today: Google Analytics for the web, Feedburner for feeds, Bit.ly for links, About:Me for the browser, Flickr Stats for pictures and so on. Emerging services like PostRank will even extend that value by consolidating various streams into a meaningful, single glance assessment of performance.

I’ve never subscribed to the idea that only what can be measured can be managed – open source, in particular, belies that claim – but there’s no debate that metrics can be immensely important in maximizing returns, and to an extent, profits.

We’re going to see analytics become, as James said, ubiquitous to the extent that they’re not already. Two projects to keep your eyes on in this space, both from IBM: the chronically underleveraged ManyEyes, and the Hadoop-backed M2. Both could – should, in my view – be important at advancing the state of analytics forward in the next year.

Marketplaces Will Be Table Stakes

Why has it taken so long for the idea of marketplaces to catch on? Don’t look at me; I’ve been banging on about them since 2006 or so. The equation has long seemed like a no brainer to me: developers and ISVs get a centralized channel and wider audience, platforms get a wider ecosystem, and customers get a more efficient discovery and acquisition process – at a minimum.

Whatever the initial reluctance, that’s over. Two plus billion Apple iTunes store downloads later, mobile players are falling all over themselves to roll out marketplaces to compete. Canonical, sponsors of the Ubuntu project, are moving towards their own software store (though, regrettably, it still doesn’t include developers as I’ve hoped for). Amazon, meanwhile, has most of the pieces it would need to sell apps, and a sustained rate of innovation that is more or less unmatched in the industry at present.

What does 2010 hold, then? Marketplaces, and a lot of them. Mobile is quickly becoming staturated, web apps and the desktop are probably next, and data marketplaces may ultimately eclipse them all. If you want to play next year, bring your marketplace. Or go home.

New Languages to Watch

Seems like we have a new hot programming language every year. Some are in it for the long haul, some fade away, and some linger in between like the undead. I’m not prepared at this point to call the winners for the next year, but two that a.) might lend themselves well to cloud and cloud-like environments and b.) are receiving disproportionately more attention relative to their erstwhile competition are Clojure and Go. The former is essentially Lisp reborn on top of the JVM, while Go borrows from C syntaticly but adds in modern language conveniences such as garbage collection without taking too much of a hit performance-wise (Go is 20-30% slower than C/C++, reportedly).

It seems unlikely that either will make significant inroads at the expense of the currently popular compiled languages such as C#/Java or the dynamic alternatives (PHP/Python/etc), but the level of attention – and the people paying attention – distinguish them from other languages aimed at concurrency like Erlang and Haskell.

NoSQL Will Bid for Mainstream Acceptance

Maybe the NoSQL label is a misnomer, and maybe Michael Stonebraker is right that NoSQL has nothing to do with SQL. Either way, I am not ready to predict that the NoSQL moniker will retired in favor of, say, AltDB.

What I will claim, however, is that projects in this space will individually and collectively make serious bids for mainstream acceptance. Cassandra, CouchDB, InfiniDB, MongoDB, Riak, Tokyo Cabinet and the like – different as they all are from one another – will position themselves not as relational replacements but complementary technologies that solve a different set of problems.

While I don’t believe the bid for mainstream acceptance will be successful generally – enterprises are too wedded to the RDBMS model, the tooling for the NoSQL projects is generally weak, etc – they will find a fertile ground in areas illsuited to the traditional relational, row-based model.

So that’s my nine. As a bonus, five predictions for free and open source software:

FOSS Predictions

1. Usage of dual licensing will continue to decline, in part because of the Oracle and EU dispute over MySQL
2. FOSS advocates will increasingly turn their attention from licensing to the related mechanisms of copyright and trademark
3. Permissive licensing will continue to gain at the expense of reciprocal licensing, albeit slowly
4. The value of project code will be eclipsed, in a few cases, by the data the project generates
5. Open source, building from its mainstream acceptance, will emerge as the most credible alternative to proprietary cloud and mobile platforms

But that’s just what I’m seeing. What are your predictions for 2010?

Update: Nat Torkington’s done an excellent follow up that looks at the opportunity side of the above: highly recommend you go read it.

Disclosure: Basho (Riak), Canonical, Cloudera, IBM, Oracle, Sun (MySQL) are RedMonk customers. Apple, Google and Mozilla are not.

Let’s say you wanted to try and correlate taxi data, as one hedge fund reportedly did recently, with market performance. Where, precisely, would you go for that data?

Exactly.

That’s why we’re going to see, and soon, the rise of data marketplaces. It’s going to be driven by two simple realizations. First, that every well run organization needs to become more creative in its usage of data analytics, because the margins are slimmer and the competition wider. Second, that just about every organization on the planet is carrying a moderately to heavily underutilized asset on its balance sheet in its data. Yes, even your organization.

The solution to both of those problems, quite obviously, lies in facitilated connections between buyers and sellers. Which means marketplaces. Marketplaces like Infochimps.

The site run by Flip Kormer and Joe Kelly promises to “Find Any Dataset in the World,” and is – to my way thinking – the shape of things to come. Currently hosting 5648 datasets, ranging from “Price Indexes for Personal Consumption Expenditures” to “100,000+ official crossword words” to “Resident Population by Race, Hispanic-Origin Status, and Age” to “Retrosheet: Major League Baseball Awards and Honors,” Infochimps is positioning itself as an independent marketplace for data owners to market and sell their datasets. Thanks to Josh of Jones-Dilworth, I spoke to the founders last week and found their ideas of the opportunity ahead similar. And they’re just great guys, to boot.

There are already information marketplaces, to be sure. SAP, as James passed along, sells data via its Information OnDemand store. And Acxiom and the credit bureaus have been trafficing in your details since the 60′s, according to Wikipedia.

But as the tools of data analytics are being democratized, as I mentioned yesterday, so too will the means of data acquisition. Infochimps illustrates this perfectly; if anything’s, it’s too democratic. The licensing – because it’s set by the user, are all over the place – and there’s not much in the way of client integration.

Consider, however, what becomes possible if they were able to build a bridge between Infochimps and something like IBM’s ManyEyes. What would be the difference between this visualization (Java required for both, sorry) of Brunswick temperatures I did a while back with data that I ferretted out from NOAA’s FTP servers:



and this one, which I did a few minutes ago using Austin weather data downloaded from Infochimps?



Nothing except for the time it took to find the data. Well, that and the fact that one’s Brunswick, ME and one’s Austin, TX.

The importance of data marketplaces is less, though, for data like weather that can be retrieved, albeit with some difficulty, than it is for data that’s currently just not made available, period. The success of marketplaces will depend initially on staples like weather and financial data, but like mobile app stores, broaden significantly as critical mass is gained. Eventually we may even see co-op type sales; while the data from one small business, for example, isn’t likely to be that compelling, the information from 500 of them would be. Marketplaces should eventually be able to facilitate such sales.

In the meantime, I’d settle for one place to get my baseball data, because I’m tired of pulling it down manually from sites like this.

Postscript: Many of you probably read the above and are screaming “you’re going to sell my data??? what about my privacy!?!” Which is a fair question, and as a result one that I’ll tackle shortly.

A month ago this upcoming weekend, I got massively delayed out of Portland, blew a connection in JFK and missed a wedding in Denver. This won’t surprise anyone who knows me or is a regular around here: I never get anywhere on time.

What was different this time was the fact that I used an iPhone application that I’d bought, built by a Y Combinator backed startup, that was supposed to algorithmically predict all of the above. Unfortunately, this trip out at least, it failed. But the user experience wasn’t all bad, as I’ll get to.

The basic premise is this: the application inputs a variety of data – departure and arrival airport statuses and weather, inbound aircraft location and so on – runs the numbers and gives you a best guess as to whether or not you’ll be delayed. Great idea, in my opinion.

Anyway, when we arrived at the airport, I was greeted by this screen.

That, for those of you not in the business, is what we call suboptimal. Still, when I turned to FlightCaster, it asserted that I was “Probably on Time.” Which would have been great news; I wanted to get to Denver both for the wedding and to see all of my friends out there.

The only catch was that it continued to assert this long after we were delayed.

Note the timestamp at the top of the screen. Worse, for FlightCaster, it had exactly the same trouble when we gave up and decided to return back to Portland.

Almost three hours after we were supposed have taken off but hadn’t, FlightCaster was still saying that we were “Probably on Time.” That’s a bit of a problem, I think.

The good news in all of this is that in spite of the applications failure and a disastrous twelve hours spent flying to and from New York, one of the FlightCaster developers picked up on my comments on Twitter and was both immediately responsive and apologetic. Which made a big difference.

Was I happy that I’d spent $4.99 on an application that failed? Not at all. But the ability to interact with one of the people building it made a difference in my perception of the application.

I still can’t recommend it, but I’ll continue using it and might change my opinion at some point in the future. Which is pretty much all you could hope for given the quality of the initial experience.