Nothing Is Guaranteed
By Ian Scott
This past weekend, one of the top web hosting firms, ThePlanet.com had an explosion and fire in a data centre, located in Texas. This affected thousands and thousands of websites as although apparently none of the servers in the facility were damaged, backup power to the facility was not allowed by the fire fighting staff that arrived on the scene.
There was apparently also some structural damage to the facility itself where the explosion (which occurred in a transformer room) but it would seem that for the most part, back up power to the servers and air conditioning would have brought the servers back on line. Of course, having backup power systems is supposed to account for situations like this.
But nothing is guaranteed and things don’t always work out like they are supposed to. Having backup power available but not being allowed by the fire fighting staff to turn it on was likely a frustrating situation for ThePlanet.com management. In all, 28 hours of downtime were seen by many of the webhosting firm’s customers. ThePlanet.com did a great job of communicating updates through the crisis via a message board.
The situation reminded me in some ways of the “Great Power Outage” of 2003, where no power was available to a large swath of North America. Many hosting firms did have backup power available for their servers but had not accounted for air conditioning requirements. With the event occurring on a warm day in August, some had no choice but to shut down servers due to heat buildup.
Another issue that came up and that had not been thought about was the ability to power generators. At my own location, we had plenty of backup power available, but our uplink provider did not. The reason? They don’t store fuel at their sites. Their emergency plan did not account for a large geographical area losing power. Instead, their plan that was in place at the time was to load up fuel for the onsite generator from a local gas or fuel station and transport that to the site that required generator backup. But of course, gas and fuel stations were unable to pump gas as today, most require electricity to pump.
In retrospect, not a great plan. Assuming 4 hours of battery backup, this plan means that you are expecting to be able to obtain fuel for your generators within a 2 hour driving distance. And even without such a wide geographical area hit by a power outage, weather conditions could affect what exactly a “2 hour driving range” is. In my part of the world, what is often a half hour driving distance can be more than 2 hours during winter snow storms.
It goes to show that even with our best laid security plans, we often overlook something that we haven’t accounted for.
Read more in: Uncategorized |