Last Monday, (February 25, 2008) at 435 JST, someone at Bluehost decided to migrate our account to a new server with a different server and with a different configuration. Normally, as a service provider, you would notify your client that you plan to migrate their data and make them aware of the new system. (Matt Heaton,Owner Pictured Left)
In this case, Bluehost did nothing of the sort. They just started migrating a site. At that point, they broke our CMS application which is the backbone of our website. Calling customer support we could get no clear response from Bluehost as to what happened. Just guesses as to what they thought was happening.
Our developer immediately checked the website and the CMS files were all intact. Our system engineer discovered Bluehost had changed the IP address for our server. Our main dot com DNS was pointed at the original IP address. Bluehost’s call center person said that we should just wait for the DNS to update. We waited and waited but the site was broken 24 hours. The next AM, we contacted our local ISP and also asked them to update their DNS. Our dot com was still broken.
Bluehost was saying there was something wrong with our local ISP. We got the local ISP on a conference call with them. Step by step, Bluehost admitted it was not our ISP’s problem. We let the ISP off the call. The website was still broken. (Please keep in mind that every hour that someone works on the site or our engineers are involved costs us money).
On Tuesday night, we called in the original developer (about midnight) as our site was still down. He got on it and fixed it that night. We were without a website for 31 hours. His summary was the following
From: L S[mailto:email@example.com]
Sent: Wednesday, February 27, 2008 9:18 AM
Subject: Site problems resolved
I had another look last night after some digging around and was able
to add back in index.php to the URL’s so the site (and sub pages)
were working again as of last night. I spent a couple of hours
digging around some more this morning and discovered that Bluehost
had apparently upgraded their Apache installation and changed the way
a certain function in PHP is handled. That in turn affected how URL’s
were working on the company, which is why you would get the
front page but none of the others. They appeared to try and add a
server rule to fix this, but that completely broke how the files were
being served. Anyway, I commented that out, and found a fix for the
URL’s problem so they should be acting normally now and you should be
able to browse around the sites as normal. Pretty amazing they would
change something like that and either not notify you, or notify their
techs that it might be an issue, but there you go!
So after all that it looks like it wasn’t a DNS issue after all (at
least as far as I could see, though it might have been a problem
earlier) – I assumed the domain wasn’t resolving at all, so when I
saw the domain was resolving to the correct location (and was
receiving traffic), I assumed it was fine. But like you mention the
requests for sub pages weren’t working, hence the need for the
solution above for the company site.
I’ll invoice you for the extra hours support if that’s
After figuring out what happened (nobody at Bluehost seemed to know, or want to tell), I tallied our loses. $2155 of loses that we could count, not sure how much traffic (usually 50-100 visitors per day).
We sent an email on 2/28 at 1215am to Matt Heaton, the Bluehost President, explaining the situation. It is one thing for us to take our site down, it is much worse for the company you’ve entrusted your site to, to take it down for you and not be able to fix it or at least cover the costs they have generated for you.
I have really come to admire what Matt is doing at Bluehost and enjoy reading his blog. However, in my almost two years of experience, I think Bluehost is lacking in two areas
1) They are a bit disorganized and should notify customers of changes. (Ie. an additional paid spam filtering service vanished overnight and we had to call to get our money back).
2) If you fail in serving your customers, you should do something to show them you care.
In this case, you should help pay the costs that were incurred by the customer. If you couldn’t cover all the costs, the least you could do for example is cover X years (or whatever years) hosting (if they are going to stay with you) as that would cost you nothing as you’ve already got 800 users crammed onto those servers. Matt’s blog talks about his trouble with Google and his frustration with Google’s customer service. After resending the email to him on 1 March, I am feeling his “Google” frustration. In this case, it is about him and his team in Utah.
Overall, I have enjoyed my experience with Bluehost. I’ve even written about it. If we are going to raise the level of customer service in the market, then everyone in management needs to do it in our own companies first. If we all do that we would have a better world one company at a time.