Feed Description

  HTML News Feed     ---     Current IT News    ---  Links  --- WAP Mobile Feed

Now that the earthquake that hit in January and the November firestorms are behind us, we have an opportunity to look at how various disaster recovery plans worked in getting businesses back on line. As a first step we will look at the impact of our plans for our offices.

Our offices are approximately 5,000 square feet in a new complex (owned by the IBM Pension Fund). During the earthquake our building moved as it was designed. The only problem was that the sprinkler system did not flex -- it broke. We arrived in our offices three hours after the quake and were greeted by a rain forest. It seems no one turned off the water until significant damage had occurred. When we finally drained the offices three days later they sucked twenty-five (25) 55 gallon drums of water out of our carpet. It is with this background that we will discuss lessons learned.

We were able to get our network out of our offices quickly and within six (6) hours we had it operational in another facility. Our losses for physical damage of computers, furniture and office supplies was almost $200,000. Given the extent of damage we were fortunate to be able to be back up and in limited operations with less than our full complement of equipment. We set priorities early in the process and people took responsibility for tasks that needed to be completed.

The most important lesson learned is that there is nothing that you can do after the disaster that will minimize your exposure unless you planned for it. After the fact does not work.

There are some unique issues associated with losses due to earthquakes. The first and most critical is no insurance policy covers earthquakes automatically. When they are covered there are major exclusions as well as inclusions.

By choice we did not have earthquake damage insurance. The cost was prohibitive and the deductible amount was too high. We did have three very important riders.

1. Computer equipment repair and replacement - all perils
2. Sprinkler water damage - all perils except earthquake
3. Earthquake sprinkler water damage - only earthquake.

This on top of our normal business interruption insurance provided the best form of security. With this behind us we were able to repair and replace critical computer and network equipment without having to worry about the financial impact on our firm.

When we walked into our suite water was POURING in. Our first point of business was to remove as many of our critical network components and files as possible. This was done at the risk of life and limb as after shocks were frequent and strong. Simple necessities such as lights, construction hats, and elevators to move equipment were not available. Everything that we used was in place and ready for us to use immediately.

A generator backup system implies that the wires that feed the power were intact. This was not the case for at least three major data centers in the valley.

As quickly as we could we accounted for all personnel and made sure that everyone knew what they were to do. This included limiting the number of people involved with the implementation of the recovery plan.

In order to facilitate the on-going operations of the business, we stabilized our voice mail system and used it as a primary means of communication. Second, we provided our staff with the capability to dial into a remote location for access to our file servers. With this accomplished we were able to see that 80% of our staff was focused on providing our customers with service. In one case, one of our consultant's drove a water truck for a client who distributed bottled water. That seemed to be a much more appropriate task than completing an AS 400 conversion.

Voice mail and cellular telephone technology are a god send. Without them we would not have been able to do half of what we accomplished.

Two of our suppliers were at our site the day after (standing in water) getting things fixed -- AT & T to repair some minor water damage to our telephone switch (which continued to operate in 90% humidity through the entire period of time) and Canon who came to wrap our copiers in case we had to move them out.

We can only say good things about most of our suppliers. Having good relations with all of our suppliers allowed us to focus on getting operational.

Standards are the road to success. Everything that was standard worked or was replaced quickly. Everything that was not standard - or not current was a problem. Some items that caused us the most difficulty:

• IBM - Micro Channel Computers (Limited options and supply of replacement parts)
• Microsoft WFW 3.11 (Not responsive to problems and replacement documentation lost due to water damage)
• DEC Alpha Chip computers (Non-responsive to getting replacements for damaged documentation)
• Non-3Com network cards (Not responsive to problem)

Some items that were no problems at all:

• AT & T Telephones and Voice Mail
• HP Printers and plotters
• Disaster Recovery standards that were followed