It’s been a crazy week here at the RedMonk home office, what with our 1and1 downtime, the bad news on my car, and the bittersweet hunt for its replacement (which is now over, as I mention on this week’s RedMonk Radio podcast). Fortunately, the week is almost over for me as I’m out of the office today, flying out to Washington DC for a Memorial Day weekend wedding. Before I’m done for the weekend, I thought I’d bring you all up to speed on some of our internal debates and thinking about our IT infrastructure. It’s RedMonk specific, yes, but touches on topics that I think are of general interest. One additional caveat: this is a relatively technical conversation, so if you’re looking for analysis of say, the business models behind open source, this won’t be the post for you. In other words, consider yourselves warned.
Anyhow, over the past couple of days, which saw me in Denver for a full week for the first time in I don’t know how long, I’ve had the opportunity to actively research options with respect to our various IT ailments. This involved demoing applications, soliciting quotes and answers from providers, conferring with my colleagues, talking to some Friends of RedMonk, doing some basic web queries (“PRODUCT NAME sucks”, etc), and – of course – digesting the feedback you all have very graciously provided following our request(s) for assistance.
As a result of that collected effort, we’re poised to address some long standing issues within the next few weeks, and we’re pleased to announce (some of) the winners for RedMonk’s business going forward. We have had two major decisions to make: one, what do we do about our messaging, calendaring, etc situation, and two, what’s our path with respect to hosting following the 1and1 debacle.
Because I have de facto responsibility for all of RedMonk’s IT infrastructure, my first and overriding concern is for the needs of our business. While I cannot therefore treat our infrastructure as a testing environment, it would be inaccurate to state that our experiences with some of the decisions below should not be expected to impact our analysis. We try, as Michael kindly notes, to actually use some of the technologies we cover. Strange concept, I know.
Anyhow, without any further delay, here’s what we/I have decided:
- Messaging/Calendaring:
Winner – Zimbra, hosted by Maccius
Finalists – Google, Joyent
This was a difficult decision for us, as two of the three of us (guess who’s not 😉 are big fans of Google services, and I’m particularly pleased with Google Calendar. Joyent is an interesting player, and is probably the most visually attractive out of all three options, but ultimately the scheduling/calendaring experience – our single most pressing problem at the moment, IT-wise – was not impressive, so they fell out early on that account. From there, we were forced into a Google v Zimbra decision, and rather than follow the path that most of the customers from my systems integration days did – designing huge and bloated feature matrixes – I shortcut the process by asking whether or not we could disqualify the players for any simple reason.Via this method, Google was deselected. The reasoning was very simple: I did not feel comfortable trusting RedMonk’s email and MX records to a host that a.) I’m not paying and b.) doesn’t have commerical level support. So despite having very capable products, Google isn’t right for us at the current time; we’ll see if they get into the supported, for-pay applications business down the line, because they would have been a serious contender should those have been available.
Having axed Google and Joyent, we were left with Zimbra, but the good news is that we may very well have selected them anyhow purely on features. Their scheduling and calendaring functionality is simple and easy to use, and the client is pure heaven after using the Firefox version of Outlook Web Access for the past couple of years. As stated previously, however, I have no interest in being in the business of running an email server, so we’re going outside for that functionality. We’ll be over at Zimbra partner Maccius for the forseeable future on a special offer. I’m hoping to make the migration sometime in the next week or two, and will keep all of you posted on any potential disruptions. Once we’re over, however, I’d expect scheduling our time to become a whole lot simpler. Finally.
- Hosting:
This decision is actually several smaller decisions, two of which have been made, one of which is still pending. Breaking them down, here are the winners:- Type of Hosting:
Winner – Colocation
Finalists – Dedicated, Shared, Virtual
As discussed previously, I was very reluctant to consider shared hosting again after some highly unpleasant prior experiences – so that option was cut early (even TextDrive’s higher end business shared offering). That left colocation, dedicated, and virtual. While the latter was an interesting option, as Ryan noted in some earlier comments, we actually have the budget for dedicated so the primary driver for selecting that route – cost – is minimized. That left colo and dedicated servers on the card, and after considering it I’ve decided to go the colocation route. There’s no question that we’ve got much better hardware than we could afford on a dedicated plan, but as I discussed previously I had serious misgivings as far as backup and security (firewall, etc). Fortunately, I’ve been assured by all three of our hosting finalists that they’ll handle backup and security more or less transparently – leaving me with basic server maintenance, applicaiton installation, and so on – tasks I’m more than capable of handling with a little help from the operating system.At lunch yesterday, however, Alex raised another important issue – patching. He queried me on whether I’d feel comfortable keeping Apache, PHP and so up-to-date with the appropriate security patches, and floated the notion that I might be better off with a managed server much like we have now. He makes an excellent point, but I remain convinced that colocation remains our best option. My answer to Alex was multi-part. First, I don’t have much faith – given the fact that they can’t even keep MySQL up and running for us on a consistent basis – that 1and1 is in keeping our currently managed, dedicated server up to date in such a fashion. Second, I’m looking to the operating system for assistance here, as will be discussed below. Third, of the hosts that remain as options, most are not providing managed servers – but basic dedicated boxes on which users are more or less left to their own devices. As near as I can determine, we’re simply not willing to pay enough to get adequate attention from a managed provider. So as long as we’re getting a root server which we’ll be responsible for, it might as well be the best server available – which means colo. Lastly, in a worst case scenario, we’re simply not storing anything of real, significant value. No credit card numbers, no SSNs, etc. Assuming our backup strategy is adequate, security is an issue but not one I’ll lose a tremendous amount of sleep over.
The net of all of this? Within the next week or so, I’ll be journeying down to our current colocation provider to extract our V20Z from the datacenter – then figuring out how to ship it wherever it’s going given that I’ve lost the original packaging.
- Operating System:
Winner – Ubuntu
Finalists – Debian, Gentoo, Nexenta, Solaris
This was perhaps the most difficult decision of all, because there are solid reasons for me to use every operating system on that list. Debian’s got the biggest community and is famously stable, Gentoo I know very well, Nexenta marries the Solaris featureset with Debian tooling, and Solaris itself offers features like ZFS that are of definite interest (I’d be using Express, rather than the official build). Windows Server 2003, for those who are wondering, is not on that list because I don’t want to pay for a copy of it – it’s not any intrinsic bias against the platform. CentOS, Fedora/Red Hat, and OpenSuSE/SuSE are not on the list because I believe there are larger communities (though not commercially, of course) behind the other Linux distributions.So why Ubuntu? It’s fairly simple. Ubuntu represents a compelling intersection of commercial interests, community support and superb tooling. As a bonus, Ubuntu’s forums are near Gentoo-level in their overall friendliness and willingness to help. In short, Ubuntu seems like the best choice for us, and one that hopefully will mitigate some of the aforementioned patching challenges via its Debian package management tooling. Now if only someone would marry the strength of the community network with a commercial equivalent for security patches and the like, as I proposed almost a year ago, I’d be sitting pretty.
- Host:
Winner – Undetermined
Finalists – Austin Web Development, Johncompanies, Rackmounted
Thus far, we’ve decided how we’re going to host our account, and we’ve decided what we’re going to host, all that remains is figuring out where to host it. Our hosting history is interesting; we began on a shared account at AIT, a small North Carolina host. Our experience there was a disaster, and I decided that small hosts were not the way to go, and we departed for a shared account at DellHost – a joint venture of Dell and Sprint. All was well there, until they decided to exit the hosting business and we were transitioned two or three times in a matter of months. Along the way, we had our MX records – the ones that point your email to the right place – erased two times in five days. Which led me to select the biggest player of all – 1and1. This seemed like an ideal situation for us; massive host, lots of customers, etc. Unfortunately, while big is good – in that it’s less likely that your small host will get acquired multiple times and screw up your account in the process – it’s also bad, because they have less and less incentive to support individual customers when things go awry. So having gone from small host to big host, I’m now ready to go back to smaller, more specialized hosts. But this time, I’ve got the benefit of a network – you guys – that I didn’t have last time I made this type of decision. That benefit and knowledge base is not to be underestimated.I’ve gotten excellent feedback on all three of the above hosts from a variety of sources, and they all differentiate themselves to some degree on either price, features, or service. This will be a tough call, but it’s one I’ll probably have to make within the next week or two. Any last testimonials or warnings on the above would be very much appreciated.
- Type of Hosting:
Having carved out plans to migrate our messaging and hosting infrastructure, you’d think my job was more or less done. Not so. I’m also actively contemplating a departure from the Movable Type platform, which would presumably coincide with the cut over from 1and1 to our new host. The primary reason for this is simple: scalability. While I cannot prove definitively that Movable Type is in fact having difficulty scaling to meet our needs, there have been signs for some time that we may have outstripped its capabilities.
- First and most crucially are the HTTP 500 errors. In MT’s defense, the recent spate of these has been as covered before, attributable to the incompetence of 1and1 as opposed to the platform itself. But I’ve had multiple reports that our HTTP 500 problem with MT predates by months at least our MySQL difficulties. That in an of itself is not damning, because it could merely indicate that 1and1’s been having database connectivity problems longer than we realized. But there’s more.
- Besides the HTTP 500 errors delivered to our would-be commenters and trackbackers – a terrible failing, there’s the simple fact that at present there are a number of actions within the tool that I am unable to perform, as they invariably return HTTP errors of several types. I can no longer, for instance, rebuild my blog when I make a change. MT seems incapable of rebuilding the several hundred pages from scratch. Nor can I mark as Junk more than 20 or 30 comments; this typically returns an HTTP 500, and when I’m able to return to the interface only about half of the marked comments are junked.
As if I didn’t have enough on my plate already, but such is the life of an administrator. Should I conclude that it’s necessary to replace MT, WordPress would be the leading candidate for a successor. Given the lack of multi-blog functionality, however, it seems that it would be necessary to maintain multiple instances of the tool, in which case I may break from my colleagues and run on Typo as an experiment. We shall see. Before anyone suggests Roller – a blogging platform I have the utmost respect for, and is likely more than scalable enough for our needs – I have no intention at the current time to run a JVM on our server. PHP with certainty, Ruby probably, but not Java as well. While it’s a good thing, IMO, that Java’s now available in our future platform, I have little interest in running it.
So there you have it: that is the current state of RedMonk’s IT infrastructure. As always, suggestions, corrections, warnings and such are much appreciated.
In the meantime, enjoy your weekend everybody. More from me next week.
