(photo courtesy the rab)

As far as I’m aware, the Monktoberfest is the only conference that began its life as a punchline. It became more than that because the attendees wanted it to. It was, to borrow a metaphor from the ITSM guys, Field of Dreams in reverse: you wanted it, so we built it.

Here’s how that happened. February 17th, I tossed out a throwaway line on Twitter, kidding. None of the people bombarding me with @ replies, email, IMs, and voicemail, however, thought it was funny. They were very serious, and very persistent.



So we looked into it. First, asking the Lion’s Pride if we could shut them down for an evening. Next, scouting potential conference locations. Then, speakers. Rinse, lather, repeat.

Seven months and nineteen days later, we were all together at ten in the morning in the downtown Portland Public Library holding new smelling t-shirts.

The volume of reactions to the day have been, frankly, humbling. All the more so because credit for the event rightfully belongs to everyone who pushed us to make it happen, and everyone who took a few days out their schedule to be with us.

What we set out to build for you was something different. There are too many conferences already; there’s no point to running a conference for the sake of running a conference. We wanted to rethink at a fundamental level what a conference was for and about.

Here’s how we did that.

The Draw

We began with the beer, of course, as the name implies. We’re fortunate to have one of the world’s best and highest rated beer venues here. But that meant that would be attendees would have to come to Portland, ME rather than us bringing the event to them in Boston, New York or San Francisco, as is more common. We decided that was a plus, because we wanted this show to be populated by people who actively wanted to be there, rather than those who might attend out of convenience because the venue’s a ten minute drive from San Jose or similar. And if people got the opportunity to see that Maine generally and Portland specifically are a beautiful place to live and a great place to work, so much the better.

The Content

But you need more than beer to hold a conference. You need content. Believing that some of the best content at any conference is the hallway track, we optimized for that. Start time was 10 AM, so that attendees could enjoy themselves at our welcome event the night before, and self-organize for breakfast the next day if they chose. Talks, meanwhile, were single track and loosely scheduled. This meant that we were all having the same conversation when there were talks, and that there was time in between to discuss and dissect.

The Theme

For our theme we chose social. Not as in something narrow and prescriptive like social media, but social as a broader trend. Because if it’s true that software is eating the world, it’s equally true that most software is social these days. GitHub’s changed the way we develop software, permanently. Untappd – the official social network of the Monktoberfest, and a speaker at the event – has changed the way people drink, and where they drink from. And they have the data to prove it.

And so on. We didn’t want to create an Everybody conference, or a Specific Agenda conference: the Monktoberfest was built from day one as a Welcome to Our Home Conference. We hope everyone felt that way.

The Swag

Instead of giving away the usual conference bag, we gave our attendees glasses. The t-shirts were a bit traditional, admittedly, but they went over well for all of that.

The Website

Thought went into the website as well. By regular conference organizer standards, our simple WordPress instance was primitive. But we tried to actively think about what attendees would want to know, rather than what we wanted to broadcast to them. Hence a travel page that linked to NY Times reviews of Portland and an FAQ that told you when to arrive and depart and who should attend.

The Food

We go to a great many conferences, and the food is more often than not poor. Which isn’t a big deal for me, as I’m not a food person, but if you can improve it for those that are, why wouldn’t you? Why not try and turn what is usually an afterthought into a feature? So lobster rolls and clam chowder it was for lunch, with vegetarian wraps for those that who aren’t meat eaters. Granted this was easier for us to do in Maine from a costs perspective, but that’s just one more reason for us to leverage this non-traditional destination.

The Sponsors

What we offered our sponsors was simple: exposure and, at higher spend levels, a few free tickets. We weren’t allowing product talks – that was, in fact, the only rule we had for our speakers – because we didn’t believe they were a fit for this audience. There are other shows to pitch your product; that wasn’t why we were here. That Red Hat’s OpenShift project, deCarta, Basho, Splunk, Zendesk and Crowd Favorite shared that vision with us, and didn’t insist on being given their fifteen minutes of fame at our show is to their credit, in our opinion. They were the reason we were able to do everything we were able to do.

The Audience

The size of the Monktoberfest was as much about embracing constraints as planning. We worked backwards from the capacity of our venue, and the result was the type of intimate conference that we’d hoped for, or so the attendees told us. Some things just work better at a small scale; this is especially true of things that involve people. Discussions ran well over their allotted time because people were asking so many questions, and the questions were all good. This is the type of problem that, from the perspective of a conference organizer, you want to have.

We’ve had a few people tell us they met more people at this conference than any other that they’ve attended. Even better, the staff at the various venues reported that they enjoyed themselves nearly as much as we did, and that everyone was polite both to them and to the other attendees. This is a testament to the quality of the people we drew.

The Credit

The credit for this event belongs everywhere else. What we did best was find good people, then get out of their way.

Logistically, Anna Melbin and Stu Brydon did a wonderful job before and at the show keeping things running smoothly; my life would have been miserable without them. My wife Kate has likewise been doing an immense amount of coordination work in the run up to the event while juggling her various other professional obligations, and things like the glasses were her idea. Marcia Chappell, who runs RedMonk operations, has been doing yeoman’s work for the past few weeks, coordinating with everything from the bus company to the venue. Rori at Novare Res took great care of us Wednesday night, and Lurie Sprague-Palino and her team at Seacoast catering did an excellent job, as always, with our Maine-themed lunch.

And as for Leigh and Ryan Travers and their team at the Lion’s Pride, as Joe Brockmeier put it, “I’d write more about it, but I think it might be cruel to those who didn’t attend to go into detail.” All that you need to know about their effort level was that they drove nine and a half hours to procure one of the beers for our tasting. Ryan and Leigh are among the best in the world at what they do, and we couldn’t be happier to share that with everyone.

On the content front, James said it best:

To a person, our speakers were excellent, and had put in real effort to craft their talks. Matt LeMay, for example, had the Bit.ly team run a computationally expensive query just for his talk. The results spoke for themselves, as they did for each of our speakers.

Whatever success we’ve had is ultimately due to the people who came together to make the Monktoberfest happen. Our sponsors, our attendees and everyone who’s encouraged us along the way. Take a bow, all of you: this was your conference as much or more than it was ours.

The Blame

While we had surprisingly few real hiccups given that it was our first time running an event, the things that went wrong are my responsibility. Not having power strips strung through the venue, for example. And as the otherwise very kind ReadWriteWeb review dinged us, our speaker diversity was poor. It’s true that we did try and get three different female speakers, only to be foiled by scheduling, but we still could have tried harder on that front. That’s my fault, and something we’ll try to rectify if we do it again. Which brings us to…

The Future

Everyone has questions about the prospect for future Monktoberfests, from which I infer that they enjoyed this one. For me personally, however, it’s a lot like asking the mother who just gave birth about plans for her next child. Conferences are an immense amount of work, and this one was no exception. So I need to recharge the batteries a bit before I think about the next show, though James is already talking up a London version.

Like the original, however, future conferences are likely to be a function of demand: if you want it, we’ll build it for you.

On behalf of everyone at RedMonk, we thank you.

For the first time since 2005, I will not be attending OSCON.

I am aware that there has been some controversy around the conference’s lack of a harrassment policy; I want to be clear that my absence is not related to this. We at RedMonk have known Tim O’Reilly for almost a decade and many of the organizers and conference chairs like Edd Dumbhill for nearly as long. Nothing in our experience with O’Reilly Media has demonstrated them to be anything but an organization with great integrity. We do believe that everyone has the right to attend a conference – OSCON or otherwise – and feel both safe and respected, but in our experience, O’Reilly always tries to do what is right. I have confidence that that will continue to be the case here.

I am unable to attend OSCON this year instead because of a health issue involving my wife. If you follow either of us on Twitter, you are already aware that she has been dealing with a severely herniated disk in her back; a herniation between the L4 and L5 vertebrae, more specifically. This herniation has been debilitating and degenerative, and after meeting with a specialist this past Friday morning at the Spine Center in South Portland, the recommendation has been made for surgery. The good news, for all of you that have expressed concern via email or Twitter – and we both thank you for that – is that the prognosis for recovery is promising and the surgery itself relatively straightforward.

While we have no guarantee that she will be slotted for the procedure this week, she is in enough day to day pain that she must take the first available opening. I therefore cannot take the chance of being caught 3100 miles away in the wrong Portland while she undergoes a procedure that involves general anesthesia. Which is why I apologized and informed O’Reilly first thing this morning that I was now unable to attend.

This was an easy decision for me on the one hand because my wife comes first. Easy in spite of the fact that she has been trying to talk me out of cancelling all weekend. It is a sad occasion still because OSCON is the conference I look forward to most every year. It is the only conference I will travel to during the summer months because of the content and the people. This is the only opportunity per year I have to meet with many of my friends from across the industry.

I will miss all of you that will be there, and I hope that you will all have a beer for me at the Kells at some point this week. If you can, come to the Monktoberfest in October; RedMonk will buy you an even better beer, and we can catch up here in the other Portland and talk social/tech.

In the meantime, I wish O’Reilly and all the attendees of OSCON an excellent week. I look forward to seeing everyone there next year.

This year marked my fifth year at the MySQL Conference. With some distance between the Oracle acquisition, this year’s show provided an interesting glimpse into the status of MySQL, both the project and the ecosystem. Let’s get to the questions.

Q: Before we begin, do you have anything to disclose?
A: Yes. Prior to its acquisition by Oracle, Sun was a RedMonk client. And prior to its acquisition by Sun, MySQL was a RedMonk client. In addition, multiple entities that compete directly or indirectly with MySQL are RedMonk clients, including Akiban, Basho, IBM, Lucid Imagination, Membase, and Microsoft.

Q: With that out of the way, how did the show do, logistically?
A: I’m not aware of what the actual attendance figures were, but they were reported to be down from last year. The show floor, at least, was sparsely populated. In general, the show is clearly down from the height of its popularity.

Q: What is this a symptom of, do you think?
A: Many things. Market fragmentation, scheduling conflicts, but Oracle probably played the most direct role.

Q: How so? What did Oracle do, or not do, to impact the MySQL Conference?
A: As with the prior year, Oracle’s commitment to the show was anemic. To begin with, their Collaborate conference was scheduled direct opposite the MySQL conference. Financially, they were not even the lead sponsor for a conference focused on their product; that spot went instead to a MySQL competitor, EnterpriseDB. More broadly, Oracle’s relationship with its ecosystem is more complicated than it used to be. MySQL partners deemed competitive with Oracle, for example, have had their relationships terminated.

Q: So Oracle is then a poor steward of MySQL?
A: Of the community and ecosystem, perhaps. With respect to the product itself, however, Oracle appears to be living up to its EU commitments: 5.6 looks like an excellent release, and reactions have been very positive.

Q: What did this conference say about the future of MySQL?
A: First, that the community remains large and vibrant. Second, that MySQL is potentially facing an Android like future of multiple implementations. Last, that MySQL is an option these days, rather than the option.

Q: Let’s take those in order. How did this conference validate the size of the MySQL Community, particularly if attendance was down?
A: Through the sheer variety and scope of sessions and attendees. I spoke with services people, hardware people, software product people, developers, engineers, DBAs, and sysadmins…and all in the space of a few hours. The speakers included the usual suspects: members of the MySQL family (Data Differential. Monty Program, MySQL, Percona, SkySQL, etc) and long time web users (Craigslist, Facebook, Google, Twitter, Yahoo, Zynga, etc). But it also featured talks from AOL, Amazon, Blizzard, BYU, Canonical, Metric Insights, Rackspace, and Recorded Future, not to mention erstwhile competitors such as Akiban, Aster Data, Cloudera, Membase, Mongo, and PostgreSQL. That’s a reasonably diverse base.

Q: And what about the fragmentation?
A: MySQL users increasingly face an interesting question: where do I buy MySQL support? The obvious answer, and certainly the safe one for larger businesses, is Oracle. The difficulty, as it usually is with Oracle, is in pricing. As in, it goes up. Regularly. At some point, for some customers, Oracle’s offerings become less relevant as a function of their pricepoint. Fortunately for these customers, there is no shortage of alternative options. You can purchase MySQL, or something very like it, from Amazon. Or you could buy support from SkySQL, launched late last year and home to many of the original MySQL staffers. Percona will support its flavor of MySQL for you. Or canonical MySQL, Drizzle, MariaDB, or RDS. Pythian provides MySQL support services. And so on.

If you’re a customer, which do you choose, and why? It was easy – or easier – when MySQL was an independent. As part of Oracle, the support calculus for some users will change. But the plethora of available support options is counterintuitively a negative for customers, as they contemplate the array of options available to them.

This is a natural consequence of the relatively recent acquisition of Sun/MySQL by Oracle; roiled markets take time to settle. But the issue of fragmentation remains. Henrik created MepSQL for a reason, and that reason is that there is a high volume of decentralized development occurring around the codebase. This is a positive for functional development, obviously, but it poses challenges from a customer adoption standpoint. Centralization would be useful, but under what mechanism? A commercial vendor? Neither are likely, though you should watch Amazon here [coverage].

Q: And lastly, how is MySQL just an option now, rather than the option?
A: As we’ve documented previously [coverage], the wider industry trend is away from general purpose and towards specialization. The database market is no exception to this: we’ve seen an accelerating acceptance of specialized datastores within businesses of non-specific types or industries. While these typically non-relational technologies are not typically displacing traditional databases, they are absorbing substantial volumes of new workloads that were once serviced by RDBMS systems. Rather than bending MySQL into a key value store, users instead are selecting built-for-purposes persistence mechanisms such as Bitcask, Membase, memcached, Redis or Voldemort. Or column databases. Or document databases. Pick your datastore type of choice.

The end result of this process is a more diverse, competitive marketplace. One in which MySQL is a popular option for certain workloads, but no longer the de facto only option.

Q: Bigger picture, what do you think of the state of MySQL, circa 2011? What would you tell a MySQL user?
A: Like Brian Aker, I’m generally positive. Whatever you think about the fragmentation, the increased market competition, or Oracle’s lack of support for community oriented events, the fact is that MySQL remains an immense ecosystem under active development. Support options are varied, the code is being improved, and the ubiquity of the database is a massive advantage. As such, I see no reason to recommend against current or future MySQL usage.

Q: And for developers?
A: Most developers have already made the adjustment to Oracle’s acquisition. Many were already leveraging MySQL side-by-side with specialized or alternative datastores – the interest in NoSQL predates the Oracle acquisition – and those that had issues with Oracle’s support have moved on to the likes of Percona and SkySQL.

Q: What about the idea that it won’t be called the MySQL Conference anymore, but the open source database conference? Will the 2011 MySQL Conference be the last one?
A: That’s more a function of conference logistics than the health of MySQL. MySQL AB, while it was independent, outsourced the conference organization to O’Reilly in a partnership that worked well for both parties. For better or for worse, Oracle appears to have no interest in similar community focused events, and as a result it would be impossible to blame O’Reilly if they retired the MySQL branding considering that the product owner is only peripherally involved. And if you factor in market context, the timing is appropriate: the open source database space is exploding.

Frankly, whatever O’Reilly calls the show next year – assuming they have one, of course – I’ll be there. The ecosystem was always bigger than just MySQL, and if the naming reflects that, so be it.

In November of last year, Forrester analyst Mike Gualtieri published the provocatively titled “Java Is A Dead-End For Enterprise App Development.” This January, the firm’s John Rymer followed up with a more balanced but similarly pessimistic “The Future of Java.” Collectively, these may be considered representative of the conventional wisdom of the enterprise.

The reality that we at RedMonk observe differs substantially, as might be predicted given our sources. What you think depends on who you listen to, remember. Forrester, as we have discussed previously [coverage], primarily concerns itself with enterprise software buyers and “platform software decision-makers.” RedMonk, meanwhile, is oriented around practitioners; practitioners like those attending FOSDEM, for example. We advantage this audience simply because we believe that bottom up adoption is more predictive of technology direction than top down procurement, but reasonable minds may obviously disagree.



The original rise of Java has been well chronicled, so it needs not be repeated here. The fall is perhaps less well understood.

The Fall of Java

Java, both the language and the platform, have largely stagnated the past few years. Some of this was the inevitable result of a decade plus developmental cycle, but the decline and subsequent acquisition of Sun further stalled progress. As have a variety of other factors, such as the historically contentious ineffectiveness of the JCP, the developmental tension of managing large enterprise demands against developer accessibility, and the sheer diversity of platform deployment.

Perhaps the biggest black eye for Java from a developer perspective has been the refusal of first Sun and then Oracle to grant Apache a TCK for Java SE over field of use restrictions [coverage]. The end result of this process – the December departure of the ASF from the JCP Executive Committee – was a blow to open source developer perceptions of Java broadly and an indictment of Oracle’s stewardship.

External factors have also played their part. The litigation of Google by Oracle over Java related copyrights and patents has had a chilling effect on Java as developers try to understand Oracle’s ultimate ambitions. According to RedMonk Analytics, Oracle’s litigation was the 12th most searched for topic by developers during the 2010 calendar year.

Finally, there’s the problem of context. In the middle of the last decade, Java was one of two dominant software ecosystems for enterprise software development, with Microsoft providing the primary alternative. While development was never quite as homogeneously Java or .NET as CIOs believed – PHP, in particular, was everywhere – Java enjoyed a relative popularity that it is not likely to ever see again. The developmental landscape has shifted, and the trend is away from general purpose platforms towards more specialized alternatives [coverage]. This means that where once fewer languages were more popular, today more languages are less popular on a relative basis. We see sanctioned usage – even in the enterprise – of dynamic languages like Perl, PHP, Python, Ruby, tactical usage of things like Clojure, Erlang, and Scala, and a real surge in interest around serverside Javascript. Put more simply: the runtime market is both more diverse and more competitive than it was previously, making it more difficult for Java to dominate in the manner it is accustomed to.

On the surface, then, we seem to have validated Forrester’s version of the conventional wisdom: Java really is a “dead end.” The only problem is that the data does not support this contention.

The Rise of Java

When we at RedMonk assess technology popularity and direction, we look at a variety of metrics public and private. Generally we prefer observational data to surveys, not least because of the biases inherent to survey work. We also focus on developers over decision makers, as discussed above. We do not generate conclusions from a single dataset, because individually they are hardly gospel. In the aggregate, however, their conclusions are compelling.

Among the sources we look at is jobs data. And while Java’s performance on a stand-alone basis may seem somewhat anemic:



The inclusion of competing platforms such as Javascript, Python and Ruby offers useful perspective.



From a job growth perspective, then, Java seems relatively healthy.

Tiobe’s language popularity metrics support this claim, for in spite of a multi-year downward trajectory, Java easily leads the majority of surveyed languages with the exception of C where the gap is narrower.



You may also recall that our survey of the Hacker News dataset found surprising traction for Java in that forward looking community:



Other developer observational data such as the performance of Java across open source mailing lists indicates substantial traction along with an anomalous spike in activity in the 08-10 timeframe, one possibly associated with pre/post Sun acquisition discussion. While the 2010 plateau indicates retrenchment, it by no means is indicative of a dying platform.



But perhaps most problematic from those who would condemn Java to the history books, damning it via the faint praise of comparisons to COBOL, is the continued employment of Java in critical open source projects like Cassandra, Jenkins, Hadoop and HBase.



Even as the rapid expansion of the Hadoop ecosystem permits the usage of more accessible languages like Python (Dumbo) and Ruby (Wukong), Java is the foundation upon which the entire edifice rests.

Even Android and the Damoclean sword of litigation that hangs above it are beneficial for the Java ecosystem, because they are indoctrinating thousands of developers – for better and for worse – with the convoluted syntax charateristic of that platform.

The Gist

How then might we reconcile the above data with the conclusions of the Forrester analysts and the conventional wisdom of the enterprise they represent? For my part, the answer is simple: I break with them.

According to the data at our disposal, it is apparent that on a relative basis, Java has peaked. It is not as popular as it once was, and is not likely to return to its former prominence in future. It is equally clear, however, that it is still a dominant platform, and the data we have on current usage and employment indicates that this position is sustainable moving forward.

Which means that, from our perspective, Java is anything but a dead end. But as always, it depends on who you ask.

Disclosure: Google and Oracle are not RedMonk clients, while Sun was. The Apache Software Foundation is a RedMonk client.

Thanks & Acknowledgements

My sincere thanks to Tom Marble for his invitation to speak at FOSDEM; I enjoyed it immensely, and the event was aggressively well run thanks to his and others’ efforts. I’d also like to point to Simon Phipps’ writeup of the event, which is too kind in my direction. His talk was exceptional, and I highly recommend viewing if and when you have the opportunity. Lastly, I’d like to thank the Free Java hackers @ FOSDEM: I thank you for lining up for the talk, and I apologize to those of you who weren’t able to get in to the room – which was packed – for the talk. You guys are why we do what we do.

FOSDEM Slides

:

“The basic reality is that the risks that scare people and the risks that kill people are very different.” – Peter Sandman, the New York Times via Freakonomics

There has never been a time, in my opinion, that MySQL has faced a more diverse set of threats than at present. Of these, one gets a disproportionate amount of the attention: Oracle’s stewardship, and the implications this has for the future of the database.

This is understandable. For most in the MySQL orbit, Oracle was the enemy, making it a suboptimal home for the most popular open source database on the planet. But the real question facing that community is whether or not Oracle should be the primary concern, or whether it’s focusing on the risks that scare it at the expense of those that could kill it. To explore this question, let’s turn to the Q&A.

Q: Before we begin, do you have anything to disclose?
A: Yes indeed. MySQL and Sun were, prior to the acquisition, RedMonk customers. We’ve also worked with Oracle in the past. Certain MySQL and Oracle competitors such as IBM and Microsoft are also RedMonk customers. So read into the following what you will.

Q: Ok. Let’s start with some of the easy stuff first: how was the conference attendance and such?
A: Pretty good. The show floor was pretty sparse, but that, I’m told, is partially due to the disruption caused by the extended acqusition process of Sun/MySQL by Oracle. This acted to depress sales of booths and so on. The attendee head count was only slightly down, from what I’m told, with the keynotes packed and many of the sessions similarly full.

Q: How about the general product news? Good or bad?
A: Users certainly seem happy. Facebook’s Mark Callaghan said in his keynote that 5.1 had turned out to be a good release – “surprise!” – and Craigslist’s Jeremy Zawodny and Smugmug’s Don MacAskill are both very excited by what they see coming in 5.5.4. So the immediate future looks positive from a development standpoint.

Q: Let’s turn back to the opener at the top. I’m confused: are you arguing that Oracle’s stewardship is not a concern? That we should just trust Oracle?
A: No. I’m arguing that there are greater and more pressing concerns than Oracle’s handling of the project.

Q: Given the fact that Oracle owns the copyright and trademark to the MySQL asset, still employs most of the developers, and is the only entity entitled to dual license the codebase, can there be a more important question than its stewardship?
A: Let’s consider the plausible outcomes:

1. Oracle intends to keep developing MySQL as a weapon against, among other products, Microsoft’s SQL Server
2. Oracle intends to kill the project, and will do so explicitly by limiting or narrowly focusing development.
3. Oracle intends to kill the project, and will do so implicitly by deliberate indifference.

Personally, I still believe the first outcome is the most likely. I’m also on record as arguing that the second is much less of a problem than people believe, because if Oracle acts publicly in bad faith, this would create – immediately – massive commercial opportunity for a second player or players. The last scenario is clearly the most problematic, as an Oracle that didn’t do anything bad but simply failed to do anything good could leave the community paralyzed and unable to muster a suitable response. Still, I consider this the least likely of the three potential outcomes.

Is Oracle’s handling of the MySQL project important? Quite obviously the answer is yes. Whether you agreed with Sun’s valuation of the project at one billion, MySQL is clearly a valuable asset.

It’s my contention, however, that there are more pressing concerns than this facing the MySQL community.

Q: And those would be?
A: What MySQL is to be going forward.

Q: What does that mean?
A: How would you describe the high level trajectory of the MySQL project? A superficial sketch might read something like this: a small, easy to use developer database goes open source, gets major traction in the web stack and becomes in a relatively short period of time the most popular database, period, on the planet. With revenue a challenge, the company makes two decisions important to the future of the project. First, they leverage the dual license model which trades the ability to incent and consume community contributions for revenue. Second, they target the enterprise relational database market. The good news is that enterprise customers actually want to pay for their software. The bad news is that the enterprise wants features that make the database less relevant and less usable for the web customers, MYSQL’s base. MySQL is able to finesse the needs of different markets well enough to turn its sub $100 million revenue stream into a 10X multiple, but the tension between what enterprise users ask for and what everyone else wants is more and more apparent, until some of the developers cry uncle and fork the project.

Q: The fork being Drizzle?
A: Correct. Drizzle can be thought of as a return to MySQL’s roots. A refactoring that repositions the MySQL codebase as the default database for web providers. Except that web providers, not getting what they wanted out of MySQL or the alternatives, were frantically rolling their own datastores and releasing them as open source projects.

Q: Which web providers? And what were they building?
A: Web providers such as Amazon, Facebook, Google and Twitter. Amazon has Dynamo, Google BigTable, Facebook tried to combine the best of both of those in Cassandra, and Twitter’s using that project as well as rolling their own graph database, FlockDB. These developments represent a challenge for MySQL.

Q: Why?
A: Because the projects are frequently directly competitive with MySQL. While NoSQL and relational database advocates may passionately debate the merits of one approach or the other, the fact is that some very smart people working for these properties looked around, found the available relational solutions – MySQL included – wanting and built their own non-relational engines. Twitter, for example, currently runs off of MySQL but has parallel Cassandra nodes running continuously. The latter has advantages for Twitter, most notably that they can do rolling restarts while the MySQL infrastructure cannot be efficiently taken down.

Q: So MySQL is doomed? Facebook et al are wholesale migrating to NoSQL alternatives?
A: Not at all. Callaghan, for example, gave an excellent talk at the MySQL conference about how Facebook uses the relational database to service some tremendously impressive workloads: 170 million reads a second, for example. MySQL still has, and will in all likelihood continue to have, a role to play at the web firms. Few if any of the NoSQL data stores are mature enough to handle the broad range of workloads a RDBMS can, so they are most often used in conjunction with one another or with databases like MySQL. The issue is that it is no longer the default choice for these players; it’s but one choice among many.

Q: Meaning that there’s a potential identity crisis brewing?
A: Exactly. If MySQL isn’t the default database of the web anymore, what is it? It’s not likely to be the default database of the enterprise – the big three are too entrenched, and Postgres is preferred by many for those kinds of workloads. Adding to the problem are the distributions.

Q: What distributions?
A: MySQL has had for a few years now distributions of the original MySQL code that are distinct from one another. Monty Widenius, one of the founders of MySQL, maintains one such over at Monty AB called MariaDB. Percona has another, OurDelta one more. Drizzle was mentioned above. One of the points of discussion at this conference were the relative identity needs for these flavors of MySQL. Customers need to understand why they might pick one over another given their individual requirements, and with the exception of Drizzle which has a fairly clearly mandate, this is not currently the case. Confusion about choices is never good for adoption.

Q: Where does the cloud fit into all of this?
A: The cloud’s a challenge for a few reasons. The Platform-as-a-Service cloud platforms act to conflate previously distinct software layers, for one. What database will you use with App Engine, Azure or Force.com, for example? The one they give you, which in none of those cases is a flavor of MySQL. Hosted versions of MySQL such as Amazon’s RDS are certainly available, and have been for years from providers like Rackspace, but the majority of the available PaaS cloud platforms use proprietary databases over MySQL. Second, the cloud poses certain architectural challenges. Drizzle’s approach acts to mitigate these, which probably explains why Rackspace has hired that entire team, but the traditional MySQL codebase needs to adapt itself to the cloud.

Q: Between the NoSQL stores and some of the non-relational cloud databases like the version of BigTable available in App Engine, isn’t the value of SQL itself questionable at this point?
A: You hear that a lot, and certainly there are some use cases where a non-relational store is simply a better fit than a traditional relational database. Key-value stores like Redis or Voldemort, for example, will be a fundamentally better option for a certain class of persistence needs than MySQL or any other relational database. There’s a reason people use memcached and MySQL side by side, remember.

That said, reports of the death of SQL are, to borrow from Twain, greatly exaggerated. There’s a reason that Hive is among the most popular interfaces for the Hadoop project: as it turns out, a lot more people know how to query a dataset using SQL than how to write a MapReduce job. It’s far from perfect, but SQL is effective, well known and understood, and time tested. It’s not going anywhere, in other words, NoSQL or no NoSQL.

Q: What do you think MySQL and its ecosystem should be doing about all of the above?
A: A few things.

1. Oracle needs to begin setting expectations. Bad news, we like to say at RedMonk, is always better than no news. While it’s understandable that Oracle’s being cautious with its messaging given that many of MySQL’s worldwide employees are not yet integrated as Oracle employees, some questions can and should be answered now. Oracle’s silence is causing problems and generating uncertainty. Even if some of the decisions being made are likely to be bad news, being definitive about them can only help.
2. The individual members of the MySQL ecosystem need to do a better job of articulating their particular focus areas. Granted, they’re all MySQL derivatives so there’s going to be a fair amount of overlap, but they tend to have differentiated customers. Being clear and articulate about the kind of customers each is targeting would be of substantial benefit to newcomers to the community, many of whom have only recently become aware there are such things as MySQL “distributions.”
3. Related to the above, MySQL and its derivatives need some clarity on the types of workloads they intend to service. The project has had success, particularly in the revenue area, straying from its base customer – the web properties, but future success in either area is not guaranteed. What is MySQL today, and what does it want to be in future? It was the database of the web, but that’s increasingly a title that it is ceding not to a single competitor, but to dozens. It is probably not, in Oracle’s hands, going to be the database of the enterprise. What does that leave? What’s the market it intends to own?
4. MySQL should be attempting, wherever possible, to interoperate and integrate with NoSQL stores. Rather than trying to fight the tide, as many in the RDBMS space are, MySQL should figure out – as it clearly has with memcache – how to play nicely with the other children. The database that is friendliest towards these data stores is going to be a popular database indeed.

Q: Has MySQL peaked?
A: On a relative popularity basis, probably. It is not likely to ever eclipse the current marketshare in future, if only because there is so much more competition than there used to be. Time was that MySQL had a few competitors in the open source RDBMS space like PostgreSQL, a few competitors in the embedded space, and obvious competiton for enterprise workloads with the web a virtual greenfield. These days there are quite literally new datastores coming out of the woodwork weekly. Most of these will not survive, but in such a crowded marketplace, it will be difficult if not impossible for any competitor to achieve again the share that MySQL enjoys today. Including MySQL itself.

But on an actual relevance basis, it is far from clear that MySQL has peaked. MySQL’s open source nature and ability to consume other data stores as storage engines makes it flexible enough to adapt to a fast changing data persistence landscape. When I spoke with Eric Day of Rackspace and the Drizzle team, we discussed the possibility of combining Cassandra and Drizzle in some interesting and creative ways. These are the conversations that the MySQL ecosystem should be having, and it’s a real positive to hear that that is taking place.

Either way, MySQL remains the most popular database in the world. If it stopped being developed today, it’s still not likely that that fact would change in the next five years. So whatever Oracle decides to do, it remains one of the most important open source projects in the world. And if MySQL can look beyond the short term Oracle concerns to the bigger picture changing data persistence landscape, it will be one for a long time to come.

Update: Somehow autosave got mixed up and forgot to include a thank you to the Drizzle project’s Brian Aker, both for organizing the MySQL Ecosystem Summit at the conference and extending to me an invitation. I also need to thank Tim O’Reilly and his team for providing space and logistical support, not to mention power strips.

If I told you that the most interesting thing I heard from Dell at their analyst event last week was the fact that their design point for their hardware is the medium sized business, would you consider that an indictment? Well, you shouldn’t.

Dell, like a lot of larger enterprises these days, is bullish on virtualization, billing the present as “The Virtual Era.” And while it might seem illogical for a maker of physical hardware to be heavily touting the benefits of virtual hardware, the mismatch is not so great as you might suppose. In spite of the ever improving ability of virtualization providers to cram more VMs onto a given piece of physical infrastructure, VMs need to sit on hardware, somewhere. Virtualization simply makes that hardware easier to utilize. Lowering the barriers to consumption, in turn, generally has one effect: it increases consumption. Think about it. If you lower the price of gas, do you sell more or less gas? So goes the thinking from purveyors of infrastructure, and that thinking is correct.

Part and parcel of virtualization in the “Virtual Era,” of course, is cloud computing. Dell had that angle well covered, calling themselves the #1 provider of cloud infrastructure today. And with customers like Facebook and Microsoft (Azure), and to a lesser extent Ask.com, that claim may be correct. Even it is not, however, and Google’s sprawling datacenters housing their homemade hardware outweigh Dell’s deployed volume, Dell is for many the defacto hardware platform of the web. Why? Because Dell, unlike the larger systems players, is able to sell to small and medium sized businesses. The kinds of small businesses that grow up to be Facebook, in fact. Cue Joyent’s tale of woe with HP and Sun versus their success with Dell.

Speaking of Joyent, back to the cloud. As my colleague wryly observed, Dell’s got the marketing angle covered, stealing a page from Oracle’s “G” branding with their PowerEdge C1100, C2100, and C6100 offerings. “C” for cloud, get it? The Dell marketing calls them “hyperscale-inspired” and “purpose-built for cloud computing, HPC, Web 2.0″ and so on, but ignore that. What they are are machines well suited to the tasks of cloud computing. High memory server for workloads like memcached? C1100. Analytics/big data/database machine? C2100. Etc etc.

But hardware is of course just one piece of the equation, and at least when it comes to the cloud, potentially the least interesting part. Good news for Dell is that their software story is quite credible. Much of the public discussion centered on Dell’s selection of Joyent as the software layer for their cloud solution for web applications, for good reason. Because while most Platform-as-a-Service fabrics like Force.com, Windows Azure or the software layer powering Google App Engine are available in public cloud form only (with the rare exception), Dell’s giving customers the opportunities to run the same cloud software layer that powers Joyent, privately. An interesting differentiation point.

Nor is that the extent of Dell’s cloud partner story. Also high visibility at the event were Canonical (Ubuntu) and Eucalyptus. The former being the most ubiquitous Linux distribution on a non-commercial basis, the latter an open source cloud middleware offering with VMware and Amazon compatibility, the combination is a cloud platform enjoying wide circulation these days. Meaning that it’s yet another option for Dell customers interested in moving to the cloud. As are both Microsoft and VMware, yet more launch partners covered at the event.

It will be interesting indeed to see how Dell manages this crowded stable of competing partners, but you certainly can’t fault them for choice. Whether enterprises are moving towards the cloud in what Dell terms revolutionary (platform based, application oriented) or evolutionary (infrastructure based, virtualization oriented) steps, they’ll have options. For both hardware and software. Not just platforms for software, either: two of the other software partners were aster data and Greenplum, indicating that Dell perceives the same big data opportunities that the rest of do.

While all of the above is interesting, however, and a sound strategy for Dell, it all hinges on the design point. By building their products for and orienting their sales force towards even the small and medium businesses, rather than the typical design for the Fortune 50 and then arbitrarily ripping out features, Dell’s able to sell even to the little guys.

Which is important, not just because some of those little guys could grow up to be Facebook or Google. It’s important because it doesn’t matter how good your technology is if you can’t sell it to lots of people. At least not if you’re Dell. My one complaint was that Dell spent an inordinate amount of time talking about the CIO, who are more often than not no friend to the PHP, Python, Ruby, et al workloads explicitly and actively courted. But as long as the design point remains the medium sized businesses, those boxes will continue to find their way into the right hands, with or without CIO approval.

Disclosure: Dell is a RedMonk customer, and comped hotel for this trip. Canonical, Eucalyptus, Joyent, Microsoft are also RedMonk customers. aster data, Facebook, Google, Greenplum, Salesforce.com and VMware are not RedMonk customers.

IBM Project Vulcan, originally uploaded by ed brill.

It’s easy to view the recently previewed Project Vulcan from Lotus as the Facebookification of Notes because, well, it is. In the waning years of the last century in particular, the Lotus enterprise collaboration suite looked it, lagging significantly the next generation collaboration user interfaces evolving, daily, on the web. But midway through the ought’s or whatever we’re calling it, the folks from Lotus began expanding their horizons, taking their cues from the web. The first product manifestations of this design evolution were Connections and Quickr.

Project Vulcan continues this trend.

But to dismiss it as merely a pretty face, a gussied up nod to the Gen-Y crowd flooding the workplace, misses the point. Under the covers, IBM is attempting to solve the same problem that anyone with an email account these days has: volume. Or, by extension, attention, referring to the time required to process said volume.

There’s no need to rehash the specifics of the systemic attention deficit. There might not be an app for that, but we do have a generally accepted definition of the condition. In addition to being a massively successful social network and the connectivity fabric for a couple of hundred million people, Facebook is part of the problem. The Facebook primary News Feed, for example, becomes increasingly unusable with each “friend” you add. Manageable at first, each additional contact becomes an input. These inputs, manageable individually, are problematic in the aggregate.

But consider the problem faced by those that would transition that experience to the workplace. If I lose the latest pictures of a “friend’s” kids or an account of their vacation amidst the noise of Facebook, it’s a pity. If I miss, on the other hand, an action item from my boss or a customer request in my Inbox, that’s a potential crisis. But it gets worse. Facebook, for the most part, is charged with handling little more than text and photos. That won’t work in an enterprise context, where businesses need the ability to collectively track everything from calendar to email to office documents to Twitter.

This might be, in other words, one of those cases where consumer simplicity genuinely isn’t enough for enterprise usage. Business apps can and do have far more bells and whistles than they need, but Facebook by itself just isn’t enough.

An enterprise Facebook clearly needs to be able to consume a variety of assets. Further, it needs to help users effectively manage that firehose, by any means necessary. Of those two, I would submit that it’s the latter that’s the real challenge. Google or Zoho, for example, offer tools that permit users to collaborate on presentations, share calendars, and the like. But who’s helping users triage their extended inboxes?

There are startups like Gist – who announced a partnership today with IBM, perhaps not coincidentally – and Threadsy attacking the space, but the short answer is that no one yet has a good solution to the over-hyped but very real problem of information overload. IBM included, Project Vulcan or no.

But of the firms with an eye towards attention management, IBM has perhaps the widest array of assets to bring to bear. They have the table-stakes collaboration pieces, as well as some interesting experience in the synchronization and federation of persistence mechanisms (think Notes databases or, more recently, CouchDB), whether that’s done client or server-side. All of which is interesting. What’s more so is their analytical expertise, from Cognos to IBM Research.

We know IBM sees analytics as vital moving forward, because everyone from Ambuj Goyal to Steve Mills have said as much. You don’t bother to create college curricula, after all, if you don’t need the people. Desperately.

In Vulcan, we may see why analytics are at a premium. For all the benefit of methodologies like GTD, the solution to the information overload problem will be algorithmically derived, because it must be. We can only ask the likes of GTD for so much; at 34 GBs per person, per day and growing, something has to give. What that is is likely to be determined, ultimately, by an algorithm. We’ll be using analytics, in other words, to tell us what we don’t need to pay attention to. It won’t be trusted at first, but that’s a problem for another day.

The idea of marrying analytics to collaboration is not new – I’ve been asking for it since 2004, actually – but its time is almost at hand. Vulcan could be the first implementation for the enterprise, if not the masses, slated as it is for a 2011 release. But it will certainly not be the last implementation. Whether Lotus gets it right or not, we’re all going need more analytics, all the time. Collaboration being no exception.

Don’t be fooled, then: Vulcan is more than an enterprise Facebook. It’s analytically assisted collaboration, and thus one to watch.

Disclosure: IBM is a RedMonk customer, as are competitors such as Microsoft and SAP. Gist, Facebook, Google, and Threadsy are not RedMonk customers.

“Baseball Prospectus’ preseason PECOTA projection for the Boston Red Sox: 95-67, second place
Actual record for the Boston Red Sox: 95-67, second place” – Kiss ‘Em Goodbye: Boston Red Sox

Even if you don’t follow baseball, you can probably appreciate that predicting the performance of a baseball team over a 162 game season is a difficult task. Injuries, abnormally good or bad performances, trades, even weather can affect the outcome of any given season significantly. And yet the Baseball Prospectus guys were able to nail it to the game.

Who says predicting the future is hard?

Or maybe you think that sports are too anomalous, too trivial, and that the above is a superficial and ultimately irrelevant example. How, then, do you feel about politics? Because happily, Nate Silver – the original creator of the PECOTA algorithms used above to predict the Boston Red Sox’s unfortunate second place finish – has turned his extraordinary talents to the problem of predicting elections. Does it work? Yep. Really, really well. As Wikipedia puts it, “Silver’s predictions matched the actual results everywhere except in Indiana and the 2nd congressional district of Nebraska, which awards an electoral vote separately from the rest of the state.”

Still not convinced? Consider then, as we have before, the case of Google Flu trends. It’s using data that indicates, as the diagram below shows, a spike in flu related activity here in Maine.

Which seems like an accurate conclusion in the wake of today’s piece in the Portland Press Herald entitled, “H1N1 widespread in Maine, top health official says.”

How are these predictions made possible? By an awareness and ability to use data. Maybe algorithms aren’t the magic bullet, but they’re damn close.

Which brings us back to the conference. When IBM’s head of Software Group Steve Mills talks, it’s usually in the vernacular of enterprise businesses, with a message tailored at same. Which makes sense; speaking in a language your customers understand is good business, and IBM’s been doing pretty well for itself recently. But one of his Monday comments at this week’s Information On Demand show in Las Vegas should, I think, be fairly universally appreciated. So much so that I’m inclined to believe that even Silver, whose business is a fair distance removed from enterprise software, would subscribe.

He said, in part, “that we’re moving into yet another wave of transformation.” More specifically, that “we’ve been living in a decade of process led transformation, and that we’re moving into an era of information led transformation.” Translated, the time of data driven decision making is at hand. So yes, designers: you may occasionally be asked to prove why 3 pixels is better than 5.

It would be naive, of course, to suggest that data driven decision making is somehow new. Repeat after me: there’s nothing new in technology. Everything from where products are placed within a supermarket to when your favorite TV shows are placed on the calendar is, and has been for a long while now, determined by a particular set of numbers. What’s different these days is that we have exponentially more numbers than we did before, better tools to attack them, and an understanding that if you aren’t using the numbers to improve your business – every aspect of it – your competitors will.

There are three primary components to this transition: market, server and client. There’s the data, too, but that deserves a post of its own.

Market

The technology industry has for the past several years been long on storage growth trivia, detailing for us all by the byte the incomprehensible amounts of data being generated yearly, monthly, even hourly. What we’ve been short on, however, have been stories about how that data is actually being put to good use. The idea certainly isn’t new; warehousing and business intelligence have been staples of the technology industry for decades. But just as cloud computing makes available resource volumes and types that were once out of reach of all but the largest businesses, the growing availability of better and lower cost technologies with an ever increasing supply of datasets dramatically upleveled the visibility of analytics for the average technologist. Perhaps more importantly, it’s received a boost from people like Silver and bestsellers like Freakonomics. For what seems like the first time you can have a conversation with a non-technologist about data and they get it. That’s important, because it widens the addressable market from numbers geeks to include people with actual budget. Not a bad development.

Server

One of the conversations I had at IOD followed a path you’ve heard here before many times: the breadth of persistence options for application developers has never been wider, and while the relational database will remain a popular option for years to come, the days of it being the only option are almost at an end. Consider IBM. For all of its investments in the relational space, the folks from New York seem to get the complementary roles non-relational tools can play as well as anyone; from CouchDB, where they employ Damien Katz, to Hadoop, where their M2 demo is one of the more interesting I’ve seen recently to Cassandra, which they are apparently contributing to. All three of which are part of – along with other projects like Drizzle, MongoDB, Riak, Tokyo Cabinet, and Voldemort – a new set of alternatives.

Whether you know the category as NoSQL or AltDB doesn’t really matter – they’ll still smell as sweet. And be as good a fit for those workloads that just aren’t quite appropriate for the relational databases you grew up with.

Fueling their respective popularity, not surprisingly, is the fact that most of the NoSQL/AltDB options are open source. The tools for making sense of mega data are available to anyone and everyone, which is a far cry from the days when warehouses meant an early retirement for data center systems integrators and software salespeople.

Client

It’s not enough, as anyone who’s viewed raw Apache server logs can tell you, to have access to a lot of data. You need a way to make sense of it; to parse it and present it back in a meaningful fashion. As far as we’ve come on the server side is as far as we’ll need to go with the client. True, the democratization of Business Intelligence tools is well underway – just take a look at what Flowing Data can do with information. It’s so useful it’s pretty. Excel may be enough for geniuses like Bill James, but us mere mortals will needs some help seeing the patterns. Think ManyEyes, but even more accessible.

Look for data visualization clients to be the next category for data driven innovation, then. Because it’s great to be able to manipulate the data, but it’s even better to be able to use it. With something this side of a Java query, I mean.

Disclosure: IBM, the organizer of the IOD show, is a client and comped hotel. Cloudera, a commercial sponsor of the Hadoop project, Basho (Riak), and Sun (Drizzle) are also RedMonk clients.

It’s true that the trip out to California is a bit more involved from Maine than from Denver, but the Zend Conference was – predictably – worth the trip. Even if it was in the cavernous San Jose conference center. As one of the larger gatherings of PHP types, it’s an opportunity to take the pulse of an increasingly important language ecosystem. Herewith a brief report – I’m coming off a red eye, so it’s in everyone’s best interests – on the present and future for PHP.

PHP and the Mainstream

Enterprise technology adoption is an interesting process, one that people can and have written books about, but perhaps its most defining characteristic is speed. Or, rather, the lack of it. As CEO Andi Gutmans put it, there’s a difference between being ready for the mainstream and actually being mainstream. PHP has, in the eyes of many including IBM, been ready for a while now. Years, actually. But having Deutsche Telekom on stage to discuss its decision to leverage PHP in a fashion far more strategic than tactical is just another validation point: the one time second class citizen of the enterprise is poised for a role alongside Java and .NET.

Perhaps the most impressive thing I heard from Petra Ruehl, DT’s VP of Product Development, was their willingness to challenge the conventional wisdom. Where other enterprises question PHP’s ability to scale for no other reason than the fact that it’s not Java, the German telecom giant (234K+ employees / 63B+ euros revenue) looked around and saw that there were in fact a few social networking sites that seemed to be scaling just fine with the dynamic language. And they took a chance on it two years ago, and were happy they did. So happy they came to talk about that decision here.

The question for PHP is whether Deutsche Telekom is an outlier or an indication of things to come. My bet is the latter; the momentum behind the language just can’t be denied.

PHP and the Cloud

The Zend folks were kind enough to ask me to moderate a panel for them entitled Developing on the Cloud. It was an interesting session for me, not least because all of the questions I took from the audience came in via Twitter, as did some very candid real time feedback on the panel’s worth. The panelists and I discussed the cloud and its implications for developers, from framework to management to security. What was interesting, I thought, were the repeated calls for an App Engine like option for PHP.

This to me is an open opportunity, either for Zend or another interested party. Most of the other dynamic languages have dedicated cloud platforms available for usage. Ruby has Heroku and Engine Yard, Python Google App Engine and so on. But while PHP developers have made support for the language the most requested feature on App Engine and even accumulated just shy of 3500 signatures on a App Engine based petition, there is no Heroku for PHP at the current time. I can use PHP in the cloud, for sure, whether it’s on IaaS offerings like Amazon’s or more PaaS-style offerings like Azure.

But sooner or later it seems probable that a commercial entity will conclude that the ubiquity of PHP and the inevitability of the cloud might make a winning combination.

PHP and Observability

Those of you that have followed my interest in Sun’s DTrace project will understand why I think the code tracing feature introduced in the latest iteration of Zend Server is an excellent idea. I can’t decide if it’s more horrifying or surprising that it’s 2009 and developers are still, for the most part, building on top of what are – effectively – black boxes. You can tell what happened before, and you can tell what happened after, but in between you’re pretty much limited to turning errors on and throwing debugging statements around like confetti. The demo I got yesterday however showed a server capable of contextually applying, via rules, a tracing function to specific error conditions. The performance hit is not major, but even better it will essentially look at a hash of the error fingerprint before writing it all out to the database. So if you get a flood of the same error, you’re not overwhelming the server by recording the same trace hundreds of times.

Part of making tracing useful, however, is intelligently and cogently rendering the collected telemetry. The data’s not particularly useful if you can’t read it, after all. Back in my mainframe days we had one guy working for us who could read a hex dump like it was a comic book – to the rest of us it was Greek written in Kanji. The Zend Server, to their credit, has a usable interface for the tracing functionality. It’ll be improved over time, I’m sure, but the current state will provide developers with a pretty good idea of precisely what went wrong, when things inevitably do go wrong.

You won’t be surprised to learn that I think that what’s really interesting to consider is what might become possible if that telemetry is centrally collected and analyzed in the aggregate, but more on that later.

Disclosure: Forgot this earlier because I was fresh off the plane: Zend is a RedMonk customer, as is IBM, Microsoft and Sun. Amazon, Heroku, Engine Yard and Google are not customers.

To their credit, Red Hat runs a concise analyst event. Three hours of presentations from senior executives and a customer (Qualcomm), some Q&A time, lunch, and you’re out and on to your next engagement. The larger vendors can’t realistically attempt this, but everyone who’s not huge would do well to learn from the open source pure play. You get the big picture, the opportunity to ask a few questions, and the chance to engage in person. Sort of a “No Fluff, Just Stuff” for analysts.

And if that doesn’t convince you, think of it this way: you know they’ve done a great job when I don’t mind getting up at four and putting on a suit to attend.

But of greater concern to most of you would be the substance: what did Red Hat want to tell the mixed crowd of financial and industry analysts? The big picture message from Jim Whitehurst (CEO) and Charlie Peters (CFO) was stability and modest growth in spite of the worst economic climate we’ve seen since the thirties. The last time Red Hat convened this crowd, the market dropped 570 points, which was a record for two whole weeks. A couple of reactionary trends, such as a shift from multi-year support and service contracts to single year commitments, have begun to right themselves, and overall the picture painted by the staff was of a confident, well run organization with tailored performance metrics tempered by an understanding that its installation to revenue ratios will always be biased to the customer relative to its proprietary competition.

From a technology perspective, the content was audience appropriate, which is to say that it was kept at a high level. Financial analysts, after all, can hardly be expected to appreciate the architectural distinctions between, say, KVM and Xen. But below were the key themes on the day.

Fighting the Good Fight Against Lock-In

EVP Products and Technologies Paul Cormier’s presentation differentiated Red Hat from a historical backdrop of proprietary lockin. From the 80′s proprietary stacks of HP, IBM, Sun, and DEC to today’s Windows and VMware virtualization offerings, the same patterns repeated themselves according to Cormier. Just as the open source antipattern has, at the middleware, operating system, and virtualization layers; in the form of Red Hat’s own product offerings, in fact. Though we could debate the minutiae of his case, the fact is that lock-in has, historically, been a popular tactic within the software industry. As the cloud is so adeptly proving at the moment.

Given that Oracle has effectively cloned Red Hat’s primary software product, I think it’s safe to say that Cormier’s claim that Red Hat is open is eminently defensible. The question for me, then, is this: do customers care? Logic would suggest that they would, or at least that they should: who would choose, voluntarily, to be singularly dependent on a vendor? The facts, however, suggest otherwise. As a species, we’re ever willing to trade the future for the present, and the businesses we create reflect this. Whether that means that your average enterprise is continually living Santayana’s Law of Repetitive Consequences or merely that they’re Getting Things Done depends on your philosophy; either way, the practical outcome is the same – businesses lock themselves into technology every day, every hour, every minute. There’s one doing it as I write this sentence.

So while we know, empirically, that lock-in is best avoided, we also know that businesses have, are and will continue to lock themselves into providers, logic be damned. Will customers really advantage Red Hat in a competitive evaluation against, say, Windows because they’re more open? Some will. But the evidence suggests that the market as a whole will not.

That said, there are good reasons for Red Hat to impress upon its analyst followers the importance of a lock-in alternative position. One, because it differentiates the firm from many of its larger competitors. Two, it provides historical perspective: as popular as lock-in might be, it is not an indefinitely viable strategy. Three, when the product is competitive, a certain percentage of customers will pick the more open of two choices. Four, we’re in the midst of a significant shift in technology purchasing and deployment, and lock-in is a more serious possibility than ever. And lastly because lock-in, like its negative image openness, are easy high level messaging tools because they’re binary characterizations. White/black, good/bad, it’s good copy.

If the analysts can understand and appreciate the fact that open source can earn Red Hat the goodwill of developer communities the world over, so much the better. But either way, it sets Red Hat apart.

Parting the Clouds

Because you can’t have an analyst event these days without talking about the cloud, the subject was sprinkled throughout the day’s presentations and discussion. Brian Stevens, CTO, however, got a slot on the agenda to discuss Red Hat’s ambitions. At last year’s event, Stevens discussed briefly his plans to open source the cloud. This time around, Red Hat was able to talk execution alongside of ambition. Between the KVM equipped 5.4 RHEL and API abstraction projects like Deltacloud, Red Hat’s doing quite a bit, project-wise, to drive forward their vision of an open source cloud.

Talk of getting into the cloud hosting business was dismissed; the ambition is purely to be an arms supplier. And given that they supply at least one of the major cloud providers, that ambition seems to be well founded.

The timing for an open source cloud will be important, given the accelerating customer acquisition efforts of proprietary vendors and the increasing volume of PaaS-style abstraction layers that trade the freedom to leave for convenience and time to market. While cloud offerings have to date been constructed largely from open source components, the functionality exposed to users has tended trended away from standardized interfaces. Witness the truckload of quote unquote open and yet highly differentiated APIs. If Red Hat is to help customers stave off a write-once, run-in-one-place future in which applications deployed to one cloud are guaranteed not to work on another, the time is now.

What’s Missing from the Stack

One of the threads running through the day was the idea of an end to end stack. Whether it was a standardized hardware and software combination running at an internal datacenter, or one spun up on an external cloud instance, Red Hat wants to be able to provide a complete and open alternative to its customers, current and future. Via its acquisition of JBoss and its build out of the management capabilities, they’re closer than ever to that goal. But an important question remains: can an end to end stack not include a persistence capability – database or otherwise?

If the scope of the language was less broad, perhaps. But when the big picture strategy is to compete in the stack game from top to bottom with open alternatives, it’s going to be difficult not to play in some portion of the database game, I think. Microsoft and Oracle will certainly leverage their databases competitively, and while VMware lacks a database, it did recently add a framework and management tooling in SpringSource and its Hyperic piece. Consolidation, as has long been anticipated, is afoot, and Red Hat has responded aggressively in every area but databases.

Besides the competitive pressures, the lack of a database offering tends to contradict the uber anti-lock-in messaging. Cormier’s presentation, for example, spoke about lock-in as an endemic problem to the industry, one that negatively impacted customers repeatedly in the operating system, middleware, and virtualization markets. Creating the market opportunity, or one of them, that Red Hat is built upon. Cormier’s macro assertions are pretty reasonable. But the slides noticeably and conveniently omitted one of the largest – and most lock-in susceptible – markets in the industry: relational databases.

As my analyst colleague Gordon Haff covers, the vendor has finally left the management software sidelines, and appears committed to those capabilities via the RHEV-M component. Meaning that Red Hat offers just about every piece of infrastructure software an enterprise could need…except a database. Pressed on a similar question during the day, Whitehurst assured the audience that Red Hat “was not looking to move into new markets,” with a database the given example.

Which is fine for now, because management’s conservative, measured pace has proven itself valuable over time. But as I’ll discuss later, the time may come when Red Hat is compelled by market forces to deliver a true end to end stack: one that includes the ability to persist data. The really interesting question is this: how would they do that? But that’s a post of its own.

Disclosure: Red Hat is not presently a RedMonk client, IBM, Microsoft and Oracle are.