Ouch! I just spoke with one the Bluehost tech support engineers, inquiring about the mysterious outage of not just the NITA Online and the NNIC servers, but even its domain name servers, which is quite rare, indicating a major incident. Well, I wasn’t off the mark: indeed, Bluehost has been hit with an unprecedented power outage, knocking even their own computers offline. The tech support engineer himself didn’t have access to a computer; it also explains why they couldn’t put some alert up on their server status page… Some extremely unusual collision of circumstances must have taken place to reach that level of snafu, given the multiple layers of fall-back assets and redundancies that their datacenter has. I surely hope that they won’t be forced to reach back to a system-wide restore of regular system backups (typically those run on a weekly cycle) so they won’t have to contend with too much fall-out, and indubitably nervous customers. But I sure as heck am curious as to the circumstances of such for a datacenter catastrophic outage.
So, let’s just sit tight and keep our fingers crossed for an expedient resolution, as their team is working hard on getting everything back on its feet again.
Update: it looks like it was a controlled shutdown on account of Provo City Power, who are (ahem) replacing Bluehost’s transformer. Yay foresight, planning and proactive communication (e.g. via their own Twitter account)
Update2: the transformer was replaced somewhat hastily as it appears the old one had a bad case of being on fire. Meaning, there is little foresight could have done here. Except, of course, using their Twitter account to spread the good news of being back at around 11pm PDT. And not leaving Twitter languishing to become a point of mockery, such as this little gem in their one and only tweet to date. I’m not kidding, this is what it says: [Read more…] about NITA Online server at nitaonline.org hit by power outage