Blogs

RedMonk

Skip to content

The RedMonk IT Report: Mint, Rsync/Dirvish, Ubuntu, and Zimbra

Screenshot of Mint

When last we touched on the State of RedMonk IT, things were looking fairly grim. We were fresh off a multi-day outage (the latest in a series of outages that led to some deserved criticism of our infrastructure), we’d more or less given up on Exchange as a means of scheduling each other’s time due to its inability to function effectively on non-Windows platforms, and our blog infrastructure was reaching its limits as we crossed the 2000 entry threshold.

I’m happy to report, however, that while we still have issues to address, we’re in a much better place than we were back in May. The first step towards solving our problems, of course, was the introduction of Hicks, hosted (very happily so far) with Johncompanies on the recommendation of many of you (including DeWitt). The combination of a dual Opteron Sun V20Z running Ubuntu gave us a.) more hardware than we’d ever had access to, and b.) an operating system I was more or less capable of administering. And by going the colocation route, I finally had a degree of control; if something went wrong, it was in my hands, and I at least theoretically could solve the problem. This is in stark contrast to the situation with 1and1, where we were more or less subject to the whims of their support staff, who seemed extraordinarily uninterested in even hearing about our problems, let alone solving them.

Since our initial transition, we’ve had (as far as I and our monitoring software are aware) zero outages (with the exception of a 10 second period where I was attempting to get an Allow,Override parameter working in Apache, which crazily caused the server to stop serving pages), and vastly improved performance. So that’s all good. But we still had some problems to solve. In case any of you are interested, here’s what we’ve been doing about them.

  1. Backup:
    Reluctantly, I’ll throw backup into the haven’t-quite-fixed-this-yet bucket. As an aside, this screams opportunity to me, but I’ll get to that in a later post. Thus far, my backup strategy consists primarily of doing manual dumps of our MySQL DB’s (e.g. mysqldump -h localhost -u dbname -p dbusername > dumpfilename.sql) and then retrieving the filesysem to a local server via rsync (rsync -rv [email protected]:/web/directory /target/local/dir). While this works, it’s hideously inefficient and clearly unsustainable longer term. Even if I automate the whole thing via cron jobs, it’s not going to capture everything we need to capture. So within the next few days I’m going to sit down and try to puzzle out a simple Dirvish based solution, storing backups either at local Johncompanies backup share or routing them back to a local server over SSH. Once I get this puzzled out, I’ll be sure and post my solution. In the event that some of you have already done so, would love to get a look at your scripts – particularly if you’re using Dirvish’s pre-command facility to dump MySQL DB’s before you run the incremental backup.

  2. Mail:
    That our email difficulties had reached a crisis level is likely not news to most of you. But after a week or two of being full-time Zimbra users, I know that several of you have been curious to see how we’re making out. The answer is, pretty well so far. More specifically:
    • The Good

    • Calendar:
      While the initial process of sharing my calendar with James and Cote was more complicated than it needed to be – I would have preferred this just be done ahead of time – once accomplished, it’s quite nice. With a single click, I can load and unload my colleagues’ calendars to try and make scheduling easier; my only complaint here is that the interface doesn’t seem to remember when I add them, so I can’t load them by default. Compared to Exchange, where my only option for viewing James’ calendar was via a three step process of a.) create an appointment in OWA (even if I wasn’t sure if I’d need it), b.) add James to the required list, and c.) view availability (free/busy only), Zimbra’s a significant improvement.

    • Filters:
      I know that on Windows, the Outlook/Exchange combo allows for fairly efficient rules processing on incoming mail because I used to use the capability fairly extensively. But once I was on Linux, it is not possible – so far as I know – to use the OWA to create and manage custom filters to help process your mail. As a result, as someone who subscribes to a number of high traffic development email aliases, my Inbox was usually overrun until I could process them client-side with Evolution & IMAP. With Zimbra, however, I can set up and configure custom lists so that with no client whatsoever all of my mail is routed to the appropriate folder I have set up, and these folders can subsequently be replicated on Evolution via IMAP. This, more than anything, has allowed me to reclaim my Inbox within the web interface. Such a simple thing, but it’s been a big help.

    • Tags:
      I’ve been asking for tags within the context of my productivity tools for some time now – at least as far back as October 2004, as a matter of fact. So as you might expect, I’m enormously happy to finally have them, and they are just as useful as I’d hoped. I can’t tell you how much more efficiently I was able to process the email glut following my vacation using tags; one quick pass through, and all the client related email received a “Client” tag. Another quick pass and I’d marked the “Sales” related messages. Then the “Briefings” and so on. While it would be possible, I suppose to replicate some of this ability with folders, that approach would not allow for multiple tags – say, “Client” and “Briefing,” versus “Client” and “Sales.”

      Net net, tags are good. I’m a big fan.

    • The Less Good:
    • Migration:
      Migrating from one email application to another is perhaps the single largest barrier to entry for prospective messaging platform switchers. Even before retraining costs and so on are considered, is how to get the email and other platform data from the old system into the new one.

      Fortunately, Zimbra’s developed a nice little tool that uses your Exchange MAPI connection to populate the Zimbra account with your Exchange data, and with one exception (not having access to Active Directory, it was unable to attach names to email from James or Cote) it populated our mail accounts completely. That’s excellent.

      What’s less ideal is that at least in James’ case, it did not successfully migrate his calendar/contacts data. We’re still trying to determine precisely what happened there, but the migration tool gets a grade of Incomplete for anything other than mail data.

    • Provider:
      I’m by no means ready to characterize our provider as either good or bad, but they’re definitely not transparent. Thus far we’ve had two brief outages that I’m aware of – though no email lost, as near as I can determine – and neither has been communicated to us in any way. I’m hopeful that MACCIuS and other hosted providers will take their cue from the likes of Salesforce.com and begin posting information about uptime/downtime. We’re fairly tolerant of limited downtime, I think, but the old adage of “bad news is better than no news” is one we believe in strongly. We’ll give you the full verdict on these guys once we have a better read on how they perform.

    • Tags Just in the Web Client:
      Because tags are so hugely useful, I miss them immensely when I work outside of the web client. This is not Zimbra’s problem to solve, obviously; it’s rather something for the Evolution, Outlook, etc guys to consider, but it’s too bad.

      On the positive side, I’ve been working almost exclusively in the web client for several days now and I haven’t really missed the rich client. Evolution’s absurdly fast search is about the only thing I wish I had, but I’m surviving without it.

    • Still Focused on Other Zimbra Users:
      Lastly, I have to at least acknowledge the fact that like virtually every other provider of enterprise messaging, Zimbra seems focused on internal users to the detriment of contacts outside your system. Granted, Zimbra needs to concern itself primarily with its customers and their needs, but as I’ve told IBM, Microsoft and Novell in the past this type of behind the firewall thinking is more or less how we ended where we are today, with all sorts of collaboration solutions that don’t work with each other. I believe that Zimbra does support iCal export – I think this is how Cote’s using it on OS X – but there’s no way that I can see within the UI to share my calendar with non-Zimbra users, or to publish free/busy information. If there was one thing I’d like Zimbra and other messaging providers to focus on, it would be that: play nicely with everyone, not just users of your application.

    Interestingly, the one question that everyone’s been asking me, and the concern everyone seems to have about Zimbra, performance, really hasn’t been an issue. Apart from the occasional slow response to a click – which could be just as easily attributable to the poor network connectivity I have up here in Maine – Zimbra’s been more than adequate for my needs.

    To sum up then, while Zimbra still has some work to do, it’s a night and day improvement over our previous situation – assuming, of course, MACCIuS doesn’t go down the 1and1 reliability path.

  3. Operating System:
    Ubuntu was selected as our operating system on the basis of three simple factors: 1.) the availability of cost effective support (under a grand per server), 2.) a massive application library to choose from, and 3.) a highly effective package management infrastructure (I still prefer Portage, but apt is pretty solid in its own right). Nothing in the several months that I’ve been running Ubuntu has given me any cause to second guess that decision, or my reasoning. Just yesterday, we determined that one of the Mint Peppers (see below) had a hard dependency on Curl. Via apt, it was simple: sudo apt-get install php5-curl. Done, and I didn’t even have to leave the command line. What I haven’t quite yet figured out is how to do a security update, like I could on Gentoo with the glsa tools. I’d love to have an ‘apt-get install stuff that needs to be updated for security reasons’. And for that matter, what’s the apt-get equivalent of emerge -p world?

  4. Web Statistics and Logging:
    As many of you may be aware – either because I’ve mentioned it or from seeing the Javascript load when you click on one of our pages – we have been using Statcounter.com to provide us with some basic metrics. While it doesn’t cost us much, around $10 a month I believe, neither was I thrilled with the package. Unfortunately, 1and1’s database restrictions prohibited us from doing anything more sophisticated with our database, such as Mint.

    But then we’re not on 1and1 anymore. So yesterday afternoon, I gave Mint a quick look and was initially underwhelmed. It was more Ajax-y than Statcounter, and more visually appearing, but seemed to differentiate itself little from our existing solution. Until I checked out its plugin/extension library. Remember when I asked developers and ISVs to be more like Firefox? Mint gets that, in a big way. Rather than try and deliver every feature under the sun, they charge a modest fee ($30 per domain) and let individuals build out features that they want. So far I’ve added Peppers (Mint’s term for a plugin) that add our Feedburner stats into Mint, one that renders our traffic into pretty graphs using SVG, one that analyzes the data for trends, and yet another that collapses our sizable list of referrers into easily navigable, Ajax-ified lists.

    Mint has its limitations, of course, but so far I’m pretty happy with it. Well worth the $30 we’ve spent so far.

With all that accomplished, what remains to be done – apart from the aforementioned backup scripts? Several things. First, I need to explore the possibility of moving to a different blog infrastructure, which is more tedious than technically difficult considering our naming structure. Second, we need to transition from our hideously old website to a brand new site, and wireframes are apparently coming this week. Third, we need to replace our library with a wiki-like infrastructure – still haven’t found a great package for that. Will keep looking. And last, Cote and I are considering some sort of chat/IRC/etc infrastructure – whether it’s Campfire or our own dedicated IRC server. In addition to having more opportunity to interact with all of you, I’d like to begin running ESPN-like chats where we get the opportunity to address and answer the questions that you folks have in an interactive way. So more on that later. I should add also that if you’re having any current issues, or have suggestions about what you think we could improve, I’m all ears.

Until next time, this has been the RedMonk IT Report ;)

Categories: RedMonk IT Report.

  • http://www.unto.net/ DeWitt Clinton

    Great status report. It is useful to hear what you’re up to.

    Regarding backups, I have been happily using rsnapshot. The rsnapshot tool automates the rsync process (over ssh) to create rolling copies on a remote disk. I use it on unto.net to keep hourly and daily snapshots of critical directories. So if anything goes bad in production I have an exact duplicate waiting offsite.

  • http://theotherthomasotter.wordpress.com Thomas Otter

    Glad to see you are being so transparent about this stuff. Your blog is your shop window, your pipeline and a major part of your distribution channel, your ERP if you like.

    For your wiki stuff, check out socialtext. Been using it internally here.

  • Toby

    Considering Zimbra also (on Solaris 10, which isn’t really supported yet).

    For instant messaging: I strongly recommend Jabber (which is to irc as Subversion is to CVS!)

  • Toby

    Oh, and for wiki, we use TWiki extensively. Check it out…

  • http://www.zimbra.com Kevin Henrikson

    Hey Stephen,

    Thanks for the update/feedback. We just pushed Zimbra 4.0 RC1 this week at Linux World, and plan to have it GA in a few weeks. That said several of the items you mentioned are already part of 4.0.

    – Calendar: We now remember which calendars you have checked so it will load your favorite calendars at login.

    – Migration: The current version of the Exchange and PST migration tools only handled Email/Contacts. In 4.0, calendar is now supported. Not a factor for you anymore but FYI none the less.

    – Tags: Today we do our best to expose tags outside of the Web UI. In Outlook we map them to categories so you can filter, sort, search locally on them if you wish. It will also let you put categories on a message and that syncs back to the server so they show up on the web UI. For IMAP your only option today is to create a search folder (aka saved search) which can be enabled to show up as read-only folders in IMAP.

    – Your Data: We provide many ways to get at and share your data. We’ve got a REST interface on the server that allows you to access any item in Zimbra via SOAP, JSON, ZIP, iCal, CSV, IFB (free busy), sync (full meta data), and atom/rss.

    Roland has written several posts on the subject:
    http://www.zimbra.com/blog/archives/roland_schemers/

    ZCS 4.0 adds the ability to share calendars and documents publicly or to specific external users (email address and password access.) Just right-click on a folder and setup the sharing or grant/deny access.

  • http://www.redmonk.com/sogrady stephen o’grady

    DeWitt: rsnapshot looks interesting – i think i ran across it before in Senor Pilgrim’s del.icio.us links. any idea how it compares to Dirvish (yes, i’m so lazy i haven’t googled it yet ;)?

    Thomas: thanks. we try to make opacity the exception here rather than the rule, if only to practice what we preach. more to the point, i don’t see a whole lot of downside: we have had problems, everyone knows that, so why not be open about what caused them? it’s easier, i’ll grant, because running the infrastructure is but one of my jobs rather than my only job so i’m not as invested in protecting my reputation in that space, but even then i think it’s better to be up front than obscure.

    in addition, thx for the SocialText recommendation. i’ve used it in the past, and like it quite a bit, and we run MediaWiki internally along with DokuWiki, but i’m looking for something sort of wiki-like but with other capabilities. i’ve got something in mind, we’ll see if it’s packaged and ready in time.

    toby: i’ve been after the Zimbra guys to support S10 for a while, we’ll see if we can make progress there. from what we’re hearing there’s actually a lot of interest in the OS X port, which i would not have predicted.

    as for Jabber, we know those guys and have a great deal of respect for them. Joe, in fact, set us up with a conference room during the initial part of our IRC chat today. longer term we’ll explore that as an option, but for now a lot of the users we’re looking to chat with will be running IRC.

    thx for the TWiki recommendation as well.

    Kevin: excellent, comprehensive feedback. now i just need to find out when MACCIuS might but 4 out ;)

    but on a more serious note, i have followed the blogs for a while, and was aware on a superficial level both from them and talking to guys like Scott and John about the REST support, but it hadn’t really sunken in until i went back and reread some of those posts. there is a *ton* that can be done with that data – the most important thing now is building out the ecosystem.

  • http://spyderous.livejournal.com/ Donnie Berkholz

    Amanda’s also worth checking out for backup.

  • https://wiki.ubuntu.com/JamesStansell James Stansell

    “I’d love to have an ‘apt-get install stuff that needs to be updated for security reasons’”

    When you have the security repository in the sources.list then an ‘apt-get update && apt-get upgrade’ will do it. (Assuming you don’t have non-interesting repositories also listed.)

    (Of course, I’m replying so late here that 1) you hopefully have already learned this elsewhere; and 2) you may never see this post.)

    Regards,

    -james.