So, here’s the deal. We sent out emails to all of our clients — if you are one of them, you should have received it. If you did not receive it – – leave a comment here if you see this and let me know.
We’ve experienced a fast growth rate in the passing months. More than we expected, actually. We experienced some down time today and some random down time sporadically throughout this month. None of this, by the way, was related to the power outage in our data center earlier this week . . that was bad timing all around.
Between yesterday and this morning, we did a full audit of our servers and determined that it’s time to add more servers to our rack space immediately and to put in place some procedural limitations in place for our servers as we grow and add more domains and services.
In the email – we acknowledged our unexpected growth and our strong need to compensate for that growth. We are providing a 50% credit to all clients to compensate everyone for the unexpected downtime and to show our gratitude for their continued business and patience with us during this time. In all – we’ve experienced approximately 8 hours of downtime in the last month, which is unacceptable as far as we are concerned. It is definitly time to add servers to the rack space. We continue to be committed to providing ongoing reliable services.
The servers have been ordered and will be in our possession by this time tomorrow. We’ll spend about a day configuring them and optimizing them for use – – and will then start the process of transferrring sites over. In the end, everyone will experience greater reliability, speed and connection rates.
There is no good excuse for a host to experience downtime and I’m about as happy about the downtime as anyone – – which isn’t very happy, by the way. Hard lessons learned, for sure – – and at the very least, in the future we will be able to better anticipate, and better equipped to handle, the growth versus the need to add more space.
There is never a good time for server downtime, that is for sure. Timing for it usually sucks. But today – this was my exact quote to Chris while he was working on getting it back up:
“I don’t care WHAT you have to do – – what strings you have to pull – – who you have to pay – – but you better have that server up by the time the Jackson verdict is read!!”
Damn if he didn’t, too. 🙂
I used to work for an ISP, not the big honken kind of ISP with redundent servers mirrored to the 9’s, backup generators, or even spare bandwidth for that matter. Users used to get so pi$$ed at the shortest of outages, regardless of the reason for them (my favorite was trying to explain sun transit to someone who *had* to send an email right then and wanted it “fixed now”) – that has tended to make me pretty darned laid back about stuff like this.
It’s hardware…. sometimes it breaks, sometimes it gets over burdened, sometimes the gnomes get in there and wander around doing their thing for no apparent reason (that’s why I use Enlightenment btw) – and then ultimately it all comes together again and life goes on.
Hey lisa, I have a DC that could support every server/rack in the building and still generate power for the city. I have a OC48 and 2 OC3 from different carriers. Want to move your servers to Phoenix? We have
Oh, By the Way, Michael is excellent.
Hey Michael – I think every DC has a problem here or there . . none are perfect, and like stageleft said – – these are machines . . sometimes it breaks 🙂 Overall, I’m very happy with our DC. They provide us a great service and do believe we won’t ever be seeing this type of power problem again in the future 🙂
In terms of our other issues? That was just a matter of our growth and our needed to stay on top of that grown and accomodate for it, as needed.
And Michael IS excellent – I completely agree with you 🙂 🙂
Thanks!
Thank you for your prompt explanations of what had been going on — we thought it was something WE did. (LOL)
As always, Blogs About blows the others’ doors off with their fantastic, nay, SUPER customer service.
You ROCK!
Love & Hugs,
Margi & KoolAid
Margi – you are the best!
Thx! xoxox.