Who Turned Out the Lights? – Or, the Straw that Broke 1and1’s Back

Share via Twitter Share via Facebook Share via Linkedin Share via Reddit

As some of you have obviously realized, given the number of emails we’ve received on the topic, the RedMonk blogs went dark several days ago – Thursday evening, to be precise. The pages themselves remained available, because they’re static HTML and require nothing more than a functioning web server to operate, but your ability to comment and our ability to post was taken away. Why? Because we had a reoccurence of the problem that knocked us offline on April 25th; our MySQL databases became unreachable.

For what they term security reasons, 1and1 chooses to host their MySQL environment not on dedicated boxes themselves, but within a separate DMZed environment. In architectural terms, this is an entirely appropriate decision and is defensible on both security and scalability grounds. As a systems integrator, this is the approach that I would generally prescribe. For our needs, however, this is serious overkill. While we’re running at around 2 million page views a month now (thanks in large part to our audience of highly irritating spammers), our needs are minimal. And when the data, presentation and application tiers are separated, there’s always the possibility of connectivity issues.

Those are to be expected, and frankly we expect some downtime with our hosted services. All that we ask is that the downtime is kept to a minimum, and that we’re kept apprised of the situation so that we can plan accordingly. Unfortunately, 1and1 failed miserably on both of these counts. I frankly have never before experienced such appallingly poor customer service; 4+ days to fix a simple database outage? They could have built a machine from scratch and reimaged it in maybe 2. Even when we had a multi-day outage of email a few years back, ASP-One at least kept us informed as to progress. 1and1? No such luck. They seem reluctant to be the bearers of bad news, never having learned the crucial lesson that bad news is always better than no news.

Despite probably a dozen calls to 1and1’s customer support service in the past 4 days – at 30 minutes plus each, no less – I still have no real conception of what the problem is. 1and1 has clearly grown too quickly, and their customer support department is obviously totally overwhelmed. With each call, I was told by the tier 1 support that the situation was escalated, and in the hands of admin. When I requested that they call admin and get a latest update, I was kept on hold for 20 or 30 minutes and told that they couldn’t get through. Tried the supervisor route, same deal. Tried an email address that turned up in a Google of “1and1 sucks,” no joy.

There is no more helpless feeling professionally than having the tools of your livelihood taken out of your hands, with no real recourse open to you. Being at the mercy of an organization that doesn’t – and probably can’t, in 1and1’s case – care at all about your business is a terrible feeling.

All of which leads me to a question I’d like to ask you guys (presuming that the database is still working tomorrow and you can actually comment): what should we do next? That we’re leaving 1and1 is a given, at this point. Fortunately, porting the site shouldn’t be terribly difficult, based on my experiences uploading my blog in TextPattern and Typo.

Our options, as I see them, are as follows:

  1. Shared Hosting:
    Where we now have a dedicated box to ourselves, we could go back to a shared hosting plan. Given our experiences doing this before, I’m not keen to try this again. Whenever something goes wrong, admins are usually unable to determine whether it’s you or some of your fellow tenants.

    • Pros: Price
    • Cons: Control, flexibility, performance, space, etc.
  2. Dedicated Hosting:
    This is what we have now, and it’s been a much better experience than sharing a server with lots of other parties who may or may not be doing evil things to the box.

    • Pros: Flexibility, performance, space, etc.
    • Cons: Control, price
  3. Colocation:
    We currently have available to us a tremendously underutilized dual Opteron, 2G ram Sun V20Z. It’s a lot more hardware than we’d probably be willing to pay for from a host, but is basically sitting there crying for a greater workload.

    • Pros: Even more flexibility, performance, space, etc.
    • Cons: Workload (backup, firewall, etc)

There is also the Virtual Server option using Xen or equivalents, which in theory offers some of the advantages of both shared and dedicated hosting. Donnie, noting our distress via my del.icio.us links, was kind enough to send along a recommendation of Linode (thanks for that sir), and he speaks very highly of them. My first reaction was to dismiss the notion, believeing that we need the full power of our own machine, but I’ll actually have to think on that more.

When I thought of hosts, I initially thought of TextDrive, but their prices are an order of magnitude greater than we’re currently paying: we’re sub $100/month for our dedicated machine, TextDrive starts at a grand. Even presuming their service lives up to that pricing, that’s not in my IT budget given our rather pedestrian requirements. Alex recommended Chris over at Austin Web Development, and we’ll certainly look into that and appreciate the tip.

How about the rest of you? Do you have any recommendations you might pass along? Any hosts that you can speak very highly of? While we’re migrating, I might as well get us onto a platform I’m comfortable with: does anyone know hosts that are running Ubuntu? For those of you running your own production servers (I know all about running dev ones), how much of a pain is it to backup and protect your machine? I’m not talking about SELinux or Trusted Solaris level of protection, but I’m used to having basic firewall needs taken care of for me. It’d be nice if we could simply eliminate 1and1 and shift everything over to the V20z; are there remote backup services that work effectively and cost efficiently over a WAN? Any thoughts or assistance in this matter greatly appreciated.

On the good news front, we’re nearing what we hope will be a final decision on our messaging/calendaring problem and anticipate being more reachable and more easily scheduled in the very near future. More on that hopefully shortly.

Lastly, I’d like to apologize on behalf of myself and my two colleagues for any inconvenience you may have experienced during our outage, and rest assured we’re taking measures to ensure it doesn’t happen again. In addition, I officially retract all the positive things I’ve said about 1and1in the past, and hereby recommend that future customers avoid them. They’ve simply gotten too big to care about their customers, which is a shame because their uptime has generally been good and their administration front end is quite nice.


  1. I found JohnCompanies via a thread on Slashdot that also mentioned Linode. At various times, we had a virtual server and a 1U dedicated server from them, and I was quite happy with their support and responsiveness in general.

  2. How about considering TextDrive’s Business hosting plan? It’s basically midway between their Shared hosting and Dedicated hosting plan. (I actually wouldn’t recommend their regular Shared hosting until they finish undergoing some growing pains that tend to occasionally affect the Shared plans- however, they are currently planning to remedy this by making a massive switch from FreeBSD to Solaris any day now, so hopefully I’ll also be able to recommend the Shared plans before the summer is out…)

    Right now they have a lifetime special going on for their Business plan at http://textdrive.com/three-martini-lunch , which might be worth considering even if you decide to go somewhere else with your site. You get StrongSpace and Joyent for life as well.

    As far as responsiveness to technical issues such as those you had with 1and1, TextDrive is definitely a great place to be.

  3. maccius? dunno if they support what we need, but it might be an interesting/dangerous/useful consolidation, alongside the mail ideas

  4. Paul: have seen positive things said about Johncompanies previously over at DeWitt’s site (unto.net). will definitely give them a look.

    Dan: i did consider that, and i suppose it’s still on the table. but it’s a difficult proposition economically when, for only slightly more money, i can have a box of my own. we’ll see what we end up doing.

    James: hmm…hadn’t considered Maccius in that light. will have to look into that.

  5. Stephen,

    I’ve had several small business friends host at Lunarpages. They only use the $8/month plan, but even at that LP support is very fast and answers questions well – I’d imagine that they’d be still more responsive to dedicated hosting customers. They have a $99/month dedicated plan:

    A couple of folks I know host at RBW (http://rbwtech.com/) and are very satisfied. One has significant traffic and things seem to have been quite reliable.

  6. Hosting my site with TextDrive was a very unpleasant experience for me. The server I was on crashed almost daily, and always took at least two hours to come back. I had been with pair.com nearly 10 years when I gave TextDrive a shot. I wound up leaving a few months later. The last straw was when I went to the pair.com updates page and saw that a server’s hard drive had crashed and that in the process of bringing it up, they went ahead and replaced the motherboard too. Total downtime: 20 minutes. When I compared that to daily two hour fscks, I knew that it was time to go.

    I have since leased a dedicated server with Layered Tech. They are very affordable, the support has been excellent, and I have root. What’s not to like? I think I pay about $68 a month.

  7. I have to recommend http://rackmounted.com. Good prices on dedicated boxes and I know two of the admin’s first names / direct emails.

    I haven’t messed around w/ shared or virtual hosting for some time now – dedicated is so insanely cheap (rackmounted entry level: $64/mo) that it’s kind of silly to consider anything else. Shared hosting still has a market but it’s not you!

    Consider throwing $230/mo at this Xeon and just go crazy:


    Or, save $10/mo on what you’re paying now on this guy:


    You guys need to get crackin on your suite of simple Analyst 2.0 tools and services and that should do you just fine for quite a while 🙂

  8. Rafe: Were you on the Shared plan? I’d definitely advise people not to sign up for the Shared plan until Textdrive gets things in better shape (i.e., until after they switch to Solaris, perhaps giving them a few months to see whether things are stable). But as far as the responsiveness and tech savvy of the staff is concernd, I feel like TextDrive is great; and I don’t think the Business plans have faced the same level of problems their Shared plans did. Having said that, I’ve been on the Shared plan myself and it was definitely rough for a while, so I understand where you’re coming from.

  9. rick: thanks for the advice. my default behavior at this point when evaluating hosts is to search “HOSTNAME sucks,” and LunarPages has some negative returns there. RBW, on the other hand, doesn’t return much, so they seem to not have pissed too many people off.

    rafe: layered tech looks decent; they’re on the shortlist. my concern is that they bill themselves as a budget unmanaged server option – if i need unmanaged, i’ll probably just colo – it’s all the same.

    ryan: thx much – rackmounted looks solid. have an inquiry into them right now.

    Dan: you’re right; i’ll be interested to see how they fare once they transition to S10.

  10. My recommendation is to host it yourself. I’ve tried just about every host out there (1and1 was one of my first abismal experiences) and most recently GoDaddy.com left me dry when the moved to SenderID for email “protection”.

    Try out OpenSuse 10.1 – it’s very easy to setup firewall permissions at the server with YaST or my preference, control ports at the router and server. Backing up isn’t difficult – since you’re using MySQL on the backend you can just create a backup with phpmyadmin in seconds or setup a script to do it automatically. With OpenSUSE 10.1 you can also easily run setup servers in Xen.

    I hate poor hosting experiences and prefer to have complete control over the architecture and administration. I’ve been trying it out myself recently and have had no real issues thus far on a “testing” domain.

  11. I can recommend johncompanies.com and have also been trying out their new product rsync.net. I also have larger boxes at theplanet.com and ev1servers.net (who have just merged) although they are no longer price leaders in this category. johncompanies do colo too.

  12. Hmm… been with those guys 2 years. No issues and nothing like this out there when I signed up. I’ll have to keep an eye on them…

  13. I’m honestly not sure what the difference is between managed and unmanaged. Layered Tech provided me with a bare Fedora Core 4 box and some IP addresses, and the rest was up to me. I upgraded the kernel and something broke, and they logged in and fixed it for me (and didn’t charge me anything).

  14. I have been using 1&1 for domains for a while now. I like them. Sorry to hear about bad experiences, but I have no beef with them. My friends use them for hosting and have no problems either.

  15. hi Rob:

    so did we, until all of this started happening. if you do a search here, you’ll even discover a thread from me thanking them for their uptime.

    if you’re happy with them, great, and i hope your experience stays as positive as it is now.

    what i would be concerned about is if it suddenly went sour as ours did, because they simply are unable to service their clients.

    best of luck either way.


  16. If you are ever in the need for hosting again please check out my site http://www.thehostguru.com as I list only the most reliable web hosts.

Leave a Reply

Your email address will not be published. Required fields are marked *