California's Power Crisis: A Warning For Corporate America

LOS ANGELES -- (BUSINESS WIRE) -- Behind the state of California's high-profile struggle to keep the lights on, an equally serious problem has gone largely unreported because companies aren't aware of it or don't want to talk about it, according to reliability experts at Syska & Hennessy's OnlinEnvironments (www.onlinenvironments.com). The problem? Thousands of companies across California and the U.S. have never adequately tested their infrastructure or information systems to see if they could survive a manmade outage or natural catastrophe that could shut down their operations and cost them millions of dollars in lost business. "Companies need to address this issue with the same energy and attention as any business threat," commented Jim Szel, a partner with OnlinEnvironments, a leading creator of critical facilities including data centers and Web hosting facilities. "Many executives are completely unaware of their risk exposure," Szel said. "A crisis is not the time to find out if your people and systems are ready," he warned. Companies are easily lulled into a false sense of security, Szel observed. They assume redundant systems such as internal emergency generators or uninterruptible power supply (UPS) batteries will function in a crisis. "Some of these systems are redundant in name only," Szel said. "They won't work as expected -- or at all." In one emergency, batteries expected to operate for 30 minutes failed after only two minutes, leaving the company in the dark -- and in total crisis. The time required getting systems online after a power failure or other crisis can substantially add to a company's business losses. Szel said it could take 24 hours or more for large, complex facilities such as data centers to fully recover from a hard crash and resume normal operations. If a company's business losses equal $1,000,000 an hour, its losses every 24 hours could exceed $24 million. OnlinEnvironments works with companies in conducting surprise tests that measure the ability of managers and technical staff to respond to a crisis. These simulations are designed to detect points of failure or vulnerability in any operation. As companies jolted by California's power crisis have learned, trouble can strike at any time, anywhere. How should companies plan for the unexpected? "Start with your people," Szel said. Most preventable down time is caused by human error. People must be well trained to minimize the risk of error such as a Web server or data center crashing. Human error caused Microsoft's web sites to go offline for several hours recently. And before trouble strikes, "you have to evaluate whether people in critical positions will keep cool and make the right decisions in a timely manner," Szel commented. A company must not only have the right people, but also the right processes in place. In one company, the engineers were carrying around sets of inaccurate and incomplete notes for managing a crisis. The company created an electronic database containing a standardized, complete and accurate plan. Companies also have to regularly test their equipment and systems, Szel noted. In the event of a loss of power, for example, a company has to ensure a smooth transition from utility power to UPS batteries to internal generators. In many facilities, generators are designed for fire and life safety only, he noted, and not for the larger demand of providing power to critical business services.