Call Center Continuity Planning

CALL CENTER CONTINUITY PLANNING

Jim Rowan Sharon Rowan

Boca Raton London New York Washington, D.C.

LIMITED WARRANTY CRC Press LLC warrants the physical diskette(s) enclosed herein to be free of defects in materials and workmanship for a period of thirty days from the date of purchase. If within the warranty period CRC Press LLC receives written notification of defects in materials or workmanship, and such notification is determined by CRC Press LLC to be correct, CRC Press LLC will replace the defective diskette(s). The entire and exclusive liability and remedy for breach of this Limited Warranty shall be limited to replacement of defective diskette(s) and shall not include or extend to any claim for or right to cover any other damages, including but not limited to, loss of profit, data, or use of the software, or special, incidental, or consequential damages or other similar claims, even if CRC Press LLC has been specifically advised of the possibility of such damages. In no event will the liability of CRC Press LLC for any damages to you or any other person ever exceed the lower suggested list price or actual price paid for the software, regardless of any form of the claim. CRC Press LLC specifically disclaims all other warranties, express or implied, including but not limited to, any implied warranty of merchantability or fitness for a particular purpose. Specifically, CRC Press LLC makes no representation or warranty that the software is fit for any particular purpose and any implied warranty of merchantability is limited to the thirty-day duration of the Limited Warranty covering the physical diskette(s) only (and not the software) and is otherwise expressly and specifically disclaimed. Since some states do not allow the exclusion of incidental or consequential damages, or the limitation on how long an implied warranty lasts, some of the above may not apply to you. DISCLAIMER OF WARRANTY AND LIMITS OF LIABILITY: The author(s) of this book have used their best efforts in preparing this material. These efforts include the development, research, and testing of the theories and programs to determine their effectiveness. Neither the author(s) nor the publisher make warranties of any kind, express or implied, with regard to these programs or the documentation contained in this book, including without limitation warranties of merchantability or fitness for a particular purpose. No liability is accepted in any event for any damages, including incidental or consequential damages, lost profits, costs of lost data or program material, or otherwise in connection with or arising out of the furnishing, performance, or use of the programs in this book. Library of Congress Cataloging-in-Publication Data Catalog record is available from the Library of Congress This book contains information obtained from authentic and highly regarded sources. Reprinted material is quoted with permission, and sources are indicated. A wide variety of references are listed. Reasonable efforts have been made to publish reliable data and information, but the author(s) and the publisher cannot assume responsibility for the validity of all materials or for the consequences of their use. Neither this book nor any part may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, microfilming, and recording, or by any information storage and retrieval system, without prior permission in writing from the publisher. The consent of CRC Press LLC does not extend to copying for general distribution, for promotion, for creating new works, or for resale. Specific permission must be obtained in writing from CRC Press LLC for such copying. Direct all inquiries to CRC Press LLC, 2000 Corporate Blvd. N.W., Boca Raton, Florida 33431. Trademark Notice: Product or corporate names may be trademarks or registered trademarks, and are used only for identification and explanation, without intent to infringe. © 1999 by CRC Press LLC Auerbach is an imprint of CRC Press LLC No claim to original U.S. Government works International Standard Book Number 0-8493-9982-3 Printed in the United States of America 1 2 3 4 5 6 7 8 9 0 Printed on acid-free paper

© 1998 by CRC Press LLC

Contents Introduction PART 1

Principles of Voice Recovery

1 Planning Philosophy Business Continuity What Is a Call Center? What Is a Disaster? What Is a Business Continuity Plan? Why Voice Recovery? How to Build a Fire Engine

2 Six Principles for Developing an Effective Plan Principle 1 Principle 2 Principle 3 Principle 4 Principle 5 Principle 6

3 Illusions of Recoverability Illusion 1 Recovery from One Fully Functioning Call Center to Another Problem Example Illusion 2 Recovery to a Cold Site (Dark Site) Problem Example Illusion 3 Recovery to a Virtual Call Center Problem Example

4 Rationalizing Procrastination The Executive Will Not Support the Idea We Are Too Busy We Do Not Have the Budget for It We Do Not Know How Where Would We Begin?

© 1998 by CRC Press LLC

You Cannot Plan for Everything It Cannot Happen Here PART 2

Planning

5 Plan Stages Preplan Recognize Problem Justify Need Management Buy-in Dollar and Time Commitment Planning Recovery Team Selection Business Impact Analysis Risk Analysis Plan Contents Team Responsibilities Backup Procedures Disaster Implementation Tasks Return to Normal Operations Postplan Plan Testing Plan Maintenance

6 Business Impact Analysis Senior Management Buy-In Identify Business Functions Operations Data Communications Utilities Risk Implications Lost Sales Fines/Jail Lawsuits Loss of Market Share Loss of Good Will Loss of Employee Productivity Loss of Shareholders Business Impact Summary Outline Recovery Window How Much Speed to Be Cost-Effective? How Much Delay Is Deadly? BIA Report Why You Must Issue a Document

© 1998 by CRC Press LLC

Executive Summary Summary of Topics

7 Risk Analysis What Are The Risks? Regional, Economic, Seasonal, Political Statistics and Their Sources Finding the Risks Risk Analysis Risk Categories Insurance Industry Input Risk Assessment Matrix Examples of Levels Probability Degree of Predictability Duration of the Disaster Severity Frequency of Types of Disasters Speed of Onset Amount of Forewarning Risk Mitigation Philosophy Prevent What Is Preventable Minimize Vulnerability Planning and Preparation Chicken Little and His Friends Plan to Update Your Continuity Plan

8 Recovery Team Selection Team Structure Primary Personnel and Alternates Tasks and Checklists Keep Updated Shift Schedule — Every Shift Must Have a Representative Representatives from Security and Maintenance Incident Manager Centralized Reporting Emergency Communications Plan Emergency Operations Center Authority What the Incident Manager Does Not Do Department Managers No Bureaucratic Sign-Off Full Support Essential Keep Subordinates Informed Keep Departmental Operations Information Updated Updating Departmental Checklists Notes

© 1998 by CRC Press LLC

9 Plan Contents Mission Statement Not a Career State the Obvious, Then Start Planning! Planning Process Definition of Disaster “All Events Are Potential Disasters” Many Disasters Start out as a “Bad Day” “Any Event That Adversely Impacts” Time Frames RTO That Fits Your Company Rationale Team Responsibilities Checklists Deputies Contact Information Active Passive IVR Situation Reports Reports to Incident Manager Up-to-Date Escalation Plan Levels of Response Declaring an Emergency Scalable: Same Procedures at Higher Pitch Critical Documentation Files (Letters of Agreement, Contracts) and Databases Vendor Contact List Equipment and Software Manuals Critical Functions Other Critical Tasks Sales Fulfillment Accounts Receivable, Credit Card Payments Public Relations Business Communications and Purchasing Critical Applications Call Center Computer–Telephony Integration Customer/Sales Database Accounting Software Word Processing and E-Mail Call Volume Management Plan Key to Call Center Continuity Recovery Site Inventory Backup/Recovery Process

© 1998 by CRC Press LLC

Implementation Plan Assessment and Declaration of an Emergency Command Center Alternate Personnel List External Contact List Inventory of Critical Documentation Inventory of Equipment Inventory of Forms Distribution Register IBM Business Recovery Services

10 Team Responsibilities Plan Update Requirements Every Station Every Change in Procedure or Contact Every Department Representative or Alternate, Every New Employee Every Change of Telephone Extension Frequency of Updates Predisaster Responsibilities Checklist Updates Training Sessions Tests, Exercises, Simulations On-Call Pager/Cellular Phone Handover to Alternate Updates of BIA Disaster Responsibilities Eyes and Ears of Incident Manager Passing Messages and Instructions Checklist Tasks and Notes Head Counts and Condition Reports (Personnel) Team Responsibilities by Expertise Covering for a Disabled or Absent Team Member Diffusing Panic Recall/Call Lists (Company Personnel) Conditions for Use (Privacy) Ensuring Correctness of Information Alternatives during Wide-Area Disaster Constant Updates

11 IS Department Recovery Planning What IS Does Information Systems Data Center Site Teams Checklists and Objectives Damage Disaster Recovery Interim Call-Handling Team

© 1998 by CRC Press LLC

The Hot Site Recovery Team Telco Data Storage IS Recovery and the Call Center

12 Disaster Implementation Tasks Whether to Declare a Disaster (or Not) Every Disruption a Potential Disaster Minor Disruptions with Major Consequences Declaration of Disaster, Disaster Type When There Is No Warning When You Can See It Coming When It Looks Innocuous, Then Embarrassing, Then … Disaster of Infrastructure Escalation Chart Stages of Disaster Ripple Effect Same Methods in All Circumstances Disaster Checklists First Person to Notice Sounds Alert Alerting Recovery Team Assessment Declaring a Disaster Enacting the Plan Following the Plan Life Preservation People First Training Takes Over First Aid Evacuation and Rescue Call Switching from the Parking Lot (Get Safe First) When the Whole Recovery Team Is Incapacitated Disaster Plan Survivability Likelihood of Being Affected by the Same Disaster Single Point of Failure What Happens If Backups Fail? Battered but Not Beaten

13 Return to Normal Operations Cold Site Too Slow for Emergency Use Useful for Long-Term Recovery from Outsource Call Center Alternate Locations for Permanent/Temporary Relocation Precontracted Premises Similar to Building a Cold Site Dependence on Assisting Call Center Assisting Call Center Maintains Company Cash Flow Restoration Procedures

© 1998 by CRC Press LLC

Ensure Power, Telephony, Supporting Systems Personnel Information Systems Telephony System Restoration Technical Personnel Equipment Fitness LAN Restoration Server Desktop Computers Booting Up Applications Call Volume Management Restoration Call Management Software Is Operational Telco Confirms Connectivity Alerting Assisting Call Center Switching Call Flow

14 Plan Maintenance Procedures Operational Changes We Do It Differently This Week Planned Changes Changeover Date Departments Responsible to Announce Personnel Changes Preparation to Hire Includes List Change He Moved to Another Company Human Resources Monitoring of Turnover Rate Promotions and Task Changes Hardware Changes Documenting New Installation Manuals and Specifications Checklist Updates Plan Book Supports Insurance Claim (Damage or Destruction) Boot-Up and Maintenance Procedures The Essential Person

PART 3

Call Volume Management

15 Call Volume Management Issues External CVM Solutions Real-Time Overflow Interim Call Handling Hot Site Company Cold Site Other Solutions High-Volume Call Answer Direct CSR Link

© 1998 by CRC Press LLC

Preestablished Procedures for Disaster Call Routing Real-Time Backup for Mission-Critical Calls Internal CVM Solutions Redundant Site Distributed Architecture (Virtual Call Center) Reciprocal Agreement

16 ICH and RTMO Interim Call Handling What Is Interim Call Handling? How Does it Work? Scripting Call Completion What Should You Look for in an Interim Call-Handling Facility? IBM Call Protect Familiarizing External CSRs with Your Products, Services, and Customers Aid in Answering Calls during a Disaster Real-Time Minimum Overflow Definition Contract with Assisting Call Center Prearranged Routing of Calls Benefit to Your Company Familiarizing External CSRs with Your Product, Services, and Customers Quality of Service “Can-Do” Attitude Needs Support Initial Training on Dummy Databases Pop-Up Screens and Scripts Building Skill Assistance for Peak Time-of-Day Call Volumes Benefit for Customer Benefit to Your Call Center Personnel Aid in Answering Calls during a Disaster

17 Other Uses for Your CVM Backup Managing Call Volumes to Reduce Rates of Abandoned Calls Call-Handling Goal Caller Response to Being on Hold Did Anyone Ask If Caller Had Time to Wait? Hold Time Music or Message (Why, How) Option to Receive Callback Adapting to Seasonal or Time-of-Day Peaks PreChristmas Rush (Season) Cold Season Holiday Rush (Season) Student School Supply Rush (Season) Accountant’s Tax Rush (Season) Other Rush Seasons — General Discussion Noon Hour (Time)

© 1998 by CRC Press LLC

Cellular Rush Hour Calls (Time) Utility Outages (Event) Storm Tow Truck Rush (Event) Monitoring Call Blocking of Faxes Dealing with Too Much E-Mail Scheduling Telephone Conferences and Videoconferences outside Peak Periods Supplementing the Existing Call Center How to Know If Your CSRs Are Losing Ground

18 Checklists Components of a Checklist Phases of a Crisis Types of Checklists Example Checklists PART 4

Testing

19 Six Questions to Ask When Designing a Test Where Are You Testing? Live Call Center Backup Call Center What Are You Testing? Revisiting Disaster Types and Causes When Is the Best Time to Test? End of Peak or Off-Peak Volume Period Nonvacation Time Holiday Weekend Midnight Shift, CSRs Off Shift, One Hour before Shift Commencement How Are You Going to Test? Notice or No Notice Logical Review (Look for Flaws) Simulated Input Operational Evaluation (Talk-Through) Physical Verification Exercise (Walk-through) Evacuation Exercise (Back-Up, Hand Over, and Walk Out) Computer Simulation Testing (Heartbeat Drill) Full-Function Tactical Evaluation (from Exhibit Event to Resumption of Operations) Who Are You Going to Test? Recovery Teams Team Leaders Call Center by Department Facility Managers Why Are You Testing? (Result) Training Value Logical Faults Prove Plan Functionality

© 1998 by CRC Press LLC

Prove Plan Current (Vendor and Other Contact Lists) Instill Confidence (Team!)

20 Types of Testing Simulated Input Operational Evaluation Paper Test Planned Inputs Critical Business Functions Not Time Sensitive Physical Verification Exercise Walk-Through with Checklists Where Are Emergency Exits Where Are the Backup Disks What Equipment Is Used for Handover of Calls Time Sensitive Evacuation Exercises Fire Drill (out of Building) Handover of Calls and Head Counts Computer Simulation Testing AT&T WinCAPS IQ Services Full-Function Tactical Evaluation Yes, It Is a Big Deal Real Emergencies Come at Worst Time Obtaining Cooperation of Emergency Organizations Obtaining Cooperation of Your Critical Vendors Obtaining Cooperation of Company Personnel Public Relations Handing over Calls: Keeping Customers out of It Security Time Sensitive

21 Test Objectives Prove Functionality of the Plan Do Not Be Distressed If Tests Seem Like a Mess Finding the Leak in the Fire Engine Tire Test More, Not Less Test Continuity of Business Processes Backing Up Data Handing over Calls to an Alternate Site If You Plan for the Alternate Site to Make Sales If You Plan for the Alternate Site to Receive Payments If You Plan for the Alternate Site to Provide Fulfillment If You Plan for the Alternate Site to Process Payables If You Plan for the Alternate Site to Handle Business Communications If You Plan for the Alternate Site to Handle Customer Service and Public Relations If You Plan for the Alternate Site to Handle Purchasing

© 1998 by CRC Press LLC

Testing Function and Timeliness Test Backup Plans Test Call Flow Patterns Peaks and Lows; Orders and Customer Service; Intuition and Graphs IVR Programming Loops (“IVR Hell”) Prove Logistical Plans Delivering Calls to Alternate Site(s) Delivering Backup Data to Alternate Site(s) Receiving Replacements of Equipment Receiving Electrical Generators Receiving Emergency Personnel Supplies Delivering Personnel to Hot Site Testing Use of the Continuity Fund Testing Vendor Response Testing Insurance Arrangements Wide-Area Disaster: Dealing with Distant Vendor Representatives Walk-Through and Time-Sensitive Tests Test Restoration Procedures Restoration of Building and Systems Data Reconciliation Restoration of LAN Restoration of Call Volume Restoration of Critical Business Functions Prove Network Connectivity Familiarize Personnel with Emergency Procedures Virtues of a Messy First Test Breaking the Plan into Bite-Sized Chunks. Five-Minute Walk-Through Test Teaching “Second Nature” (Learning to Live by Checklist) Explaining Plan Updates Make Testing Part of Normal Routine Announcing Test by Name After-Test Morale Boost (T-Shirts, Congratulatory E-Mail, Perspective) Running Your Test Use Responses Laid Out in Checklists and Manuals Note What People Really Do in (Simulated) Crisis Flag Deviations from the Plan in Your Test Notes Use Only Resources That Are Available at the Time of the Test Act as if the Event Were Real (for Training Value) Use Monitoring Staff from Departments Not Being Tested Be Prepared to Terminate Test If Someone Suffers Panic (and Reschedule) Identify Each Incident with Its Own Test Name Plan Test Objectives in Advance Plan Test Assumptions in Advance Test Evaluation All Participants Attend After-Test Assessment Meeting

© 1998 by CRC Press LLC

Distant Participants Attend by Telephone or Videoconference Record Proposed Plan Amendments Update All Plan Books and Checklists PART 5

Crisis Management

22 Crisis Leader, Internal Command, and Control Authority to Deal with All Aspects of the Crisis Chain of Delegated Authority Discussion of Crisis Communications Internal Command and Control Procedures Public Relations Coordination Authorized Release of Information Designated Meeting Place

23 Basic Crisis Management Event Alarm Assessment Notification Phases of a Crisis

24 Personnel Issues Location Wide-Area Disaster Makes Commuting Arduous Alternative Call Center Cost Capacity Coordination of Transportation Postdisaster Trauma Willingness to Return to Work Family Care Issues Replacement Workers PART 6

Conclusion

25 Continuity Planning for Your Call Center The Well-Equipped Fire Engine Brief Summary Discussion: Continuity Might Be of Essentials Only Just Reading This Book Is Not Enough! Successful Recovery Testing Continuity Planning for Continuity in Banking Planning for Continuity for Utilities Discussion: Not Just Like It Was Call to Action: Plan and Test

© 1998 by CRC Press LLC

Call to Action: Keep Plans Up-to-Date Contact Us

PART 7

Appendices A Tables B Sources Major Sources Minor Sources

C A Wake-Up Call for the Call Center Industry The Storm Returning to Work: Anecdotes Why Planning Is Essential: The Costs Associated with Wide-Area Disaster “Wake-Up”

D Glossary Index

© 1998 by CRC Press LLC

Introduction

T

his book will help you to plan for continued operations of your call center and, therefore, your business, during a crisis. Whether your company is large or small, plans and budgets in millions or hundreds of dollars, operates according to strategic plans generated by committee or by the seat of your pants, this book contains helpful advice to help you continue business in virtually all circumstances. This book has three major sections. The first section is a discussion of the principles and philosophy of call center (voice) continuity planning. The second section is an examination of what is involved in preparation for planning and “call volume management.” The third section is a discussion of what comes after the planning process — testing and, in case of a disaster, crisis management. A Note about Examples: Throughout this book, we will illustrate aspects of the continuity-planning process by use of examples. Some of the examples that we use will be totally fictional. Some of the companies that we describe will be composites of many companies that have had similar experiences. Sometimes we will describe the triumphs and woes of real companies, with identifying details disguised. Any resemblance to any real company, situation, or person is completely coincidental.

© 1998 by CRC Press LLC

Acknowledgments A THANK YOU NOTE

T

his book would not exist, except for the help of some very special people. We wish to acknowledge our deep and abiding debt to Alan J. Rowan, CDRP. His wise advice and contributions concerning checklists and virtually every part of this book were exceedingly valuable to us. Throughout his 35 year military career, he has participated in, planned, and taught disaster preparedness. More, he has taught us the disaster preparedness attitude. The concepts of planning, preparedness, and redundancy of equipment and supply apply to both military and business continuity planning. We can truly say that without him this book would not have been written, because we would not have ever known about plan writing or plan testing, risk assessment or mitigation. We wish to thank Michael Lavoie of Auerbach Publications for his enthusiasm and patience. It is good to have an editor who appreciates the significance of the subject matter to businesses large and small. We thank the specialists, who contributed their expertise on various subjects, and who are named in Appendix B as Sources. We would also like to thank various other people who gave us insights into the process of planning for business continuity and resumption for large corporations in diverse industries. Most of all, we would like to express our appreciation to the large number of people who told us the stories of disasters at their companies (large and small) “anonymously.” The stories we liked the best were the ones that ended, “and the plan worked just as it was supposed to.” Our thanks to Jason R.D Page, BA, of All Things Page cannot be overstated. His help in producing graphic images and transcribing endless audiotape materially aided us in preparing the manuscript. Last, we thank our family for their great patience with us while we were planning and writing this book.

© 1998 by CRC Press LLC

Part 1

Principles of Voice Recovery

T

his section discusses some basic philosophic issues about planning. What should a good plan contain? How can you make a continuity plan effective? What are some fallacies that might fool you into thinking that your present precautions are “okay”? We believe that you will find here information that will be useful to you even if you have already gone through the call center continuity-planning process. Part 1 also discusses some rationalizations for not planning. We encourage you to read them with your company in mind.

© 1998 by CRC Press LLC

1 Planning Philosophy BUSINESS CONTINUITY What Is a Call Center? Senior management often underestimates the importance of the call center to a company. Some managers do not even realize that they have a call center. Almost every company having more than one hundred employees has some sort of telephone customer interface or help desk, yet very few companies identify themselves, in telephone directories or on the Internet, as call centers. Any company that has a facility whose purpose is to handle a significant number of telephone calls or other electronically delivered messages is indeed a call center, whether or not it is called by that name. The consequence of operating a call center, but not recognizing or naming it as such, is the real possibility that the call center might be ignored during business continuity planning. The managers of such a company might know what they consider their “primary” business very well, whether it is the sale of vacuum cleaners, skiing holidays, insurance, or providing solutions to customer complaints. Because of their lack of a dedicated “call center” focus, in adverse circumstances, the failure to recognize the importance of the call center, as a call center, might impair the ability of the company to carry on business. In fact, this neglect could hamper the survival of the company. Once upon a time the management of a company was debating whether or not it should expand the capabilities of its call center. A vice president was heard to say, “Are we in the vacuum cleaner business, or in the call center business?” The call center manager stated that because the company made most of its sales by telephone, it was really in the call center business. Although management had not become accustomed to thinking that way, vacuum cleaners were only the product that it provided. But the call center was what made the company “go.” A call center is any group of individuals in a company or other organization, who perform similar task groups, whose primary contact with the customer (or client) occurs by telecommunications.

This includes use of telephone, facsimile, e-mail, Internet Web site, and videoconferencing. Such activities include (but are not limited to) customer service, sales, help desk functions, order inquiries, credit approval, revenue protection, and so on. Please have a look at Exhibit 1-1, which shows many of the major kinds of calls. Even “experts” can work in a call center. Specialized licensed services are done in a call center environment if there are more than five people communicating by telephone, all in the same offices, doing the same activity at the same time. Such licensed activities include life insurance agents selling life insurance, stockbrokers selling investments, and real estate agents selling properties. Contacting clients or former clients, servicing existing client accounts, and following up on past sales are all activities that could be thought of as taking place in the call center environment. If your company does any of these activities, operating a call center is how the people contacting the public do them. Management of their activities would benefit by being thought of in that light.

© 1998 by CRC Press LLC

EXHIBIT 1-1. CALL TYPE CHART Type of Call

Type of Call

❑ Order taking

❑ Telemarketing outbound

❑ Reservations

❑ Answering inquiries for general information

❑ Product location inquiries

❑ Answering sales

❑ Channeling customer complaints to others for resolution

❑ Collections

❑ Solving customers’ problems

❑ Outage reporting

❑ Answering inquiries about product/service use

❑ Scheduling repair/warranty service

❑ Contest lines

❑ (Advertising)

❑ Advising senior management on customer issues

❑ Market research

❑ Public relations

❑ Quality management

❑ Crisis handling

❑ Billing

❑ Defining customer service/policy/strategy

❑ Specialized certification/licencing

Call types below the double line may require additional skills or technical experience.

What Is a Disaster? Events can be call center disasters that in other senses you might term inconsequential. Suppose that a fire alarm rings. Is that a disaster? Let us suppose that the alarm is the result of an electrical fault in the alarm assembly. Let us further suppose that no one is injured while exiting the building, no one suffers heart palpitations, and since there is no fire there is no smoke or water damage. If it takes as little as a half hour, or up to an hour and a half, for the call center staff to return to the telephones, how much revenue have you lost? Suppose that one or more CSRs (customer service representatives call-takers) was on the point of making a sale when interrupted by the loud alarm bell. That sale is lost. What is the dollar value per call minute of the average sale? How many minutes have been lost per call-taker? Add this all up, and you arrive at the cost of this “minor” event. How to find the “cost per call minute” is discussed in a later chapter. “A disaster is any unplanned event that prevents an organization from performing its critical business functions over a period of time, resulting in unacceptable damage or loss of employees, customers or assets.” So says Gary Via, a Senior Consultant with Lucent Technologies.1 In the above definition, “any” means any unplanned event that causes disruption to the operation of your company, for a period of time that would mean a significant dollar loss. The disaster could also be a planned event that has unforeseen results. For instance, suppose that the fire alarm in the previous example had an intermittent fault that caused frequent “nuisance” alarms, but was hard to find and repair. Suppose that the call center staff felt under pressure to make high numbers of sales, that the morale in the call center was falling because of the alarms, and that some supervisors felt compelled to browbeat the call-takers because of falling sales volumes. In such a tense environment, a

© 1998 by CRC Press LLC

fire alarm, even if caused in the course of an electrician’s maintenance, might be the additional stressor that caused an overburdened employee to resign. As defined above, “loss of employees” is also a “disaster.” We discuss employee morale later in this book. Although we have drawn out the above example to an extreme degree, surely everyone has heard of telephone cables cut during construction or renovation. Or water mains broken during road repair. Most readers have heard of electrical work that required the turning off of electrical power for a time. Although the primary event is expected (construction), the impact on business activities might not have been anticipated when the plans were made. How do you find out how much disruption is “significant” to your call center? The topics of testing and statistics are discussed in Part 4 of this book. Some events are unmistakably “disasters,” and road repair had nothing to do with it. How many people are affected by such events each year? The International Federation of Red Cross and Red Crescent Societies has reported that in 1994 more than 160 million people throughout the world were affected by disasters not associated with wars. The United States National Archives and Records Administration reported that only 43% of businesses that suffer a disabling disaster ever resume operations. A disabling disaster is a disaster that shuts down a company long enough to have a profound impact upon the company. On average, a disaster that shuts down a company for more than four days will seriously disable or cripple that company. Of the 43% that survive the initial disaster, only 29% are still in business two years later. Turn this statistic around, and you will see that of businesses that suffer such a disruption, about 57% never resume operations. And about 71% of those that resume cannot continue as long as two years. Only about 12.5% of all such unfortunate businesses have the capability of continuing for more than two years after the disaster.2 Ironically, we have found in various seminars and workshops that only about 10% of companies sending attendees have any sort of emergency plan. Most people think of disasters as “something bad,” but disasters can also result from “something good” as well. If your company had too many calls coming into its call center, prospective customers wishing to order goods might become irritated. It is a disaster, in a certain sense, if callers cannot get through, because the company is losing sales. But this type of disaster might not be noticed if the company management (you) is thinking of these sales from a “bottom-line” point of view rather than a “call center” point of view. For example, take the case of a cable television company changing its channel lineup and rates. Customers of this company wishing to subscribe to additional channels had to call repeatedly, for weeks, to get their message heard, often with no success. The cable company call center was heavily overbooked, and did not have enough incoming lines. It was usually impossible to get through to a “live” agent. That company did not know it, but it was experiencing a disaster. The “unknown disaster” consisted of its telephone lines experiencing call blocking, with resultant frustration for a large portion of its customers. Management was not even aware of this disaster. Had the telephone lines been cut altogether, managers would have known that something was happening. They would have been vigorously following up with the telephone company to get repairs completed as quickly as possible. However, since the cable company did not know how many of its customers could not get a call into the call center, the managers also did not know how many customers were deeply frustrated, even angered by the problem. Eventually, the company (possibly realizing that responses were not meeting the marketing department projections) offered an e-mail address, a shopping mall kiosk for letter mail, and a shopping mall direct-link telephone line. Whether the cable TV company received flaming e-mail is unknown; however, it was plain that the direct telephone line went ignored by most customers. Perhaps some of the customers declined to contact the company because the provided telephone was too public and there was no provision for privacy. Others (possibly with reason) thought such a call would be as much a waste of time as

© 1998 by CRC Press LLC

calling from their homes. They anticipated having to wait for 15 or 30 minutes, or more, for an answer, and no chair had been provided. Coincidentally, the cable TV company was facing heavy competition from satellite TV providers at the time. The numbers of cable TV subscribers who were frustrated by their inability to reach a live call center agent had grave consequences for the cable TV company revenues and subscriber base. How many calls were blocked, “peaked out,” and were lost to a busy signal? How many customers sat on “hold” for long lengths of time, and abandoned their calls? How many customers, having called in repeatedly, on subsequent calls abandoned them quickly? How many subscription upgrades were not made, because customers were calling on a whim, which died of old age? How many customers lost their tempers, and on finally reaching a call-taker, canceled their subscriptions rather than upgrading them? Because the management of the cable TV company was thinking about cable television service, instead of a call center service intended to serve happy cable TV subscribers, the financial scope of the disaster was “unknown.” Perhaps in due course the satellite TV companies will publish, as part of their advertising, how many former cable subscribers they win in the affected geographic area. You might easily “cure” this “unknown disaster” by outsourcing to an overflow call center or by adding additional telephone lines, equipment, and CSRs to the company call center to handle the call volume. This “disaster” would not be unknown if the cable TV company received call volume statistics from its telecommunications provider, and paid attention to them. Additionally, the company might run what we call “heartbeat drills”3 with the assistance of a company such as IQ Services, which we describe later in this book. Knowing that there is a problem is the first step in solving it. But at the time we wrote this book, the unknown disaster at the cable TV company continued.

What Is a Business Continuity Plan? The disaster recovery industry no longer calls plans made in case of disaster “disaster recovery” plans. In fact, the Disaster Recovery Institute (DRI International)4 no longer calls certified planners “Certified Disaster Recovery Planners.” They are now known as “Certified Business Continuity Planners.” Picking the pieces up after a disaster only does a company so much good. By the time a disaster has occurred and the damaged equipment has been replaced, electrical power has been restored, and so on, a company might already have sustained a substantial loss in business. Customers might already have gone elsewhere to purchase the goods and services they sought. Although some lost assets can be replaced, some others have to be redesigned, or replanned, or are on the supplier’s back-order equipment list. Sometimes, the revenues, customers, employees, and assets lost to a company during a disaster can never be recovered. If the company is lucky, this results in a “bad year.” If unlucky, the company could fail, and cease operations. The aim of a company planning to overcome a disaster to its call center should be to continue doing business even during a disaster. A disaster recovery or business resumption plan is better than nothing. But a business continuity plan conveys the ability to carry on doing business notwithstanding a disruption. Changing the focus to business continuity provides the valuable means to ensure that the current year of a business will not be its last, and that its current customers, revenues, assets, and employees will continue beyond the present. A good business continuity plan will likely include the means of dealing with both known and unknown disruptions. For instance, the business continuity planner can plan ahead for known risks, such as regional potential for earthquake or flood, seasonal disruptions like hurricanes, tornadoes, or thunderstorms, the possibility of outbreaks of influenza or other disease, and so on. As well, the planner can take account of risks as yet unknown,

© 1998 by CRC Press LLC

such as terrorism, meteor strikes, public relations events like scandals or accidents, and other undreamed of events. With the emphasis placed on continuity rather than resumption or recovery, the planner puts in place a mechanism (the plan) and a team (designated personnel) that will enable the company to respond flexibly to events as they occur. The business continuity plan will likely even contain a mechanism for carrying on business while dealing with a situation that is totally unanticipated. We will discuss all the above topics in more detail later in this book. Consider again the example of the cable TV company suffering its unknown disaster. Because of the overflow of unanswered calls, it has a “disaster in progress.” Numerous subscribers or potential subscribers suffer great frustration in trying to communicate with the company offices. On the plus side, because of the nature of cable TV service, the majority of customers who give up trying to reach a live agent do not actually “go away” forever. They do continue paying their monthly cable bill. But on the minus side, the company revenues suffer because new subscriptions and upgrades in subscriptions can be made by only the most-determined callers. As a side note, even those determined callers would be likely to be in an irritated mood when recounting to the call-taker(s) that they had to call sixty times before finally getting through! Suppose that the TV cable company instituted a business continuity plan focused on the call center, which delivers the bulk of the company’s revenues. The company management would require that the condition and efficiency of the call center not be a mystery. “Telco,” the telephone utility, would report on call blocking, that is, the percentage of times callers had to dial in before reaching a live agent and completing their call. Such a plan would enable the company to learn quickly what percentage of inbound lines were experiencing “call blocking.” In addition, the company could establish a quality assurance program. Computer-generated test calls (the heartbeat drills) and follow-ups with customers will show what ease or difficulty callers might have experienced in contacting the call center, and whether superior or inferior service were being given to callers. We discuss Telco reporting and “quality assurance” in more detail later in this book.

Why Voice Recovery? The military and civil defense organizations made the first “disaster recovery” plans decades ago. It was their job to “think the unthinkable” and plan for the consequences of war and disaster, to defend the citizenry and safeguard their own personnel. Their plans included fallout drills, secure document and weapons storage, emergency rations, bomb shelters, plans for evacuation by various means of transportation, and secure communications. They also had to plan for the actual waging of war, while advancing upon and retreating from the enemy. At some point, managers of industry took notice of this and raised some new questions — What do we do if our plant closes for a day? What are our alternate supply lines? — and, commercial disaster recovery planning was born. The computer industry got involved. (If we lose our computers, all data processing comes to a halt.) Computer companies made plans establishing procedures to “recover” or bring back into operation the computers of a customer. These plans come under the general heading of “data recovery.” There are plans in use which talk about how to get people safely out of burning buildings, how to find alternate suppliers, or how to repair or retool an assembly line “quickly.” There are plans for how to recover your computer data successfully and get the data to computers at other locations. In the past, most of these plans ignored the company telephones or only mentioned them in passing.

© 1998 by CRC Press LLC

In the 1980s, toll-free 800 numbers came into wide use. It was relatively easy for anyone able to pay the telephone utility fees to obtain the use of such a telephone line. Recently, tollfree “888” numbers were brought into operation. There were no more 800-series numbers available, because they were all in use. Planners already know that the time when the 888 numbers will all be in use is imminent, and are looking for an alternative. And it is now possible to obtain an eleven-digit global 800 telephone number, which can be called toll-free from virtually anywhere on planet Earth. This shows the vast increase in business being done on the telephone. Now planners must consider voice recovery, that is, measures to safeguard the operation of the telephone communications of a company. In case of disaster, a company can recover its buildings and people, but if it does not have the capability of using its telephones for communication, it will still be out of business. The ability to restore communications is essentiala to the survival of a company. Some managers now realize that “recovery” or restoration of communications lines and equipment is also becoming a matter of immediacy. At one time, an RTO (recovery time objective) of one day (at that time, of computer data) was considered good. Now, the recovery time for the voice communications of a busy call center must necessarily be accomplished in 30 minutes (or, preferably, less!). In today’s communications environment, 30 minutes of lost calls represents a large amount of business that might never come back. An AT&T-sponsored study found that a stock brokerage could lose $6.5 million for every hour that its call center is inoperative. Inbound calls for credit card reservation of travel arrangements could represent a loss of $2.5 to $3.0 million for every hour of lost communication with customers. Even small companies, such as florist shops, restaurants that sell “take-out” orders or take table reservations, hair salons, and landscaping or gardening contractors — in fact, any company that receives orders by telephone — could lose hundreds or thousands of dollars during every hour of a communications outage.

HOW TO BUILD “A FIRE ENGINE” The business continuity plan is like a fire engine. Different compartments of the truck hold tools, such as defibrillators, Scott air packs, and “jaws of life” cutting tools. The firefighters bring along fire hoses, tanks of chemical fire extinguisher, one or more winches, and connections for fire hydrants. The truck carries rescue gear such as stretchers, blankets and ropes, bunker gear for entry into the heat of fire-engulfed buildings, and more. The location and capabilities of these tools must be known to the firefighters, and they must be trained in their use. The fire engine, by itself, is of no use without trained personnel to operate it. If you have a fire and do not have a fire engine (at all), the chances of being able to put out the fire are slim. You need both the truck and the people. The truck alone is of no use without the trained firefighters, and the trained firefighters alone can do little without the truck. Even the most experienced firefighters (together with their truck) require the following to be effective in performing their duties: a plan, training, testing of their capability to use the plan, and regular updates to the plan. Your company is similar to a trained group of firefighters. Your people, also, require a plan, training, and proper equipment to meet adverse circumstances and triumph over them. Like firefighters, the people of your company will need a “vehicle” to provide them with what they will need in case a disaster should threaten your company operations. That “vehicle” is the business continuity plan. The organization of a fire department, the training of the personnel, and the stocking of firefighting and rescue equipment aboard the fire engine require detailed planning. So does the making of a call center continuity plan. Yes, you will indeed have to plan to plan! The plan stages for preplanning and planning are discussed in Chapter 5. When (sorry, it

© 1998 by CRC Press LLC

would be dishonest to say, “if ”) a crisis arises, the time spent on this stage will show its value. Doing a business impact analysis will show what parts of your company are the most important to its continuation and survival (see Chapter 6). Risk analysis (see Chapter 7) helps to show what dangers are prevalent in your industry and region. Risk analysis includes talking with the local fire department. You need to determine what known hazards exist in your local area that could affect your company. Selection of the “recovery team” who is responsible for enacting the plan during a crisis is like putting together a team of fire marshals (see Chapter 8). Whether for fire control or call center continuity, there is no “one-size-fits-all” plan. Your business continuity plan can be as well equipped as a fire engine. You can make it useful for handling most crises that might arise. If you are lucky, some parts of the plan might seldom be used, just as a backboard for care of spinal injuries is not required by firefighters every time they assist at the roadside scene of an automobile accident. Some parts of the plan might never, ever, be used. Conceivably, there are fire departments in the world that have never needed, or will need, a geiger counter to measure radioactivity, or that have never needed, or will need, protective clothing and gas masks, for responding to a chemical fire or spill. Even though the probability is that some parts of the continuity plan will never be needed, the likelihood is that some parts of it will be required some day. On that day (or days), it will be highly important for your company to have a viable plan ready. Company personnel will need to be totally familiar with all aspects of it. The plan will have to be up-to-date, because old procedures or contact information would cause confusion and costly, or even deadly, errors. The costs for not being ready on that day or days will be measurable.5 Your company might suffer lost sales, loss of company prestige, lost trust of customers, employee injuries or deaths, lawsuits, action by regulatory bodies, even the risk that the company itself might not “continue.” The unready company might not survive to continue paying salary, wages, or dividends. Planning is the key to continuity, to survival.

NOTES 1. Lucent Technologies has been among the leading companies in providing for communications continuity. The company produces telephone-switching equipment and interactive voice response units, among other hardware. 2. The 12.5% survival figure (for more than two years after a disabling disaster) has been derived by us from the statistics, and was not produced by the United States National Archives and Records Administration. 3. These test functions provided by Interactive Quality Services, Inc., are described in detail in Part 4, “Testing.” 4. Disaster Recovery Institute International, 1810 Craig Road, Suite 125, St. Louis, MO 63146. 5. See Chapter 6, “Business Impact Analysis,” for a discussion of risk implications.

© 1998 by CRC Press LLC

2 Six Principles for Developing an Effective Plan

T

he following six principles originated with Ted Ging (see more about him in the Contacts section in Appendix B), and should be part of the foundation of every call center’s business continuity plan.

PRINCIPLE 1 Your communication plan should mirror the way you would normally operate. People revert to familiar methods in a disaster, rather than using different, unfamiliar procedures.

Sometimes people’s minds go blank in a crisis. Take the case of an explosion followed by a fire, which was caused by a microscopic crack in a natural gas pipeline across the highway from a remote establishment. The “emergency plan” called for members of some departments to do things in ways that were totally different from their normal procedures. Virtually all the people who were supposed to do something different forgot what they were doing. Their minds went blank and the people acted by habit. Other departments had emergency plans that called for their members to act in ways that were similar to what they did every day. Almost all of those people acted according to their departmental plan. Because of those people, the situation as a whole remained under control. Despite a roaring gas fire, which shook the ground and caused people to have to shout to be heard, the physical plant sustained only minor damage. Some windows cracked from heat, some paint blistered, but no lives were lost. Your company operates in a certain fashion in its regular operations. Causing people to switch jobs in case of a crisis would almost guarantee that the people in your organization would not remember their emergency tasks. If you find that you absolutely, for some reason, must change the jobs of certain people during a crisis, then you should also provide those people with checklists. Your people can use checklists to follow step-by-step instructions and ensure that they perform every part of the plan. You should make the checklist part of their daily routine too. Then, they would become accustomed to checking the list for the next step. If a crisis arises, they would then have the “check the list” habit, which would enable them to follow the “crisis” list successfully.

© 1998 by CRC Press LLC

PRINCIPLE 2 You must appoint a single clearly defined incident manager to coordinate the event. Committees cannot provide quick and decisive action.

It has been said that the camel is a horse that was designed by a committee. Undoubtedly (if such a thing could be true) that committee would have had long, boring meetings where not much was accomplished. Many, many such meetings would have been needed, and each decision that had to be made would have been the subject of many long memoranda and reports, amid wrangling over budget and resource allocation. During a crisis, there is no time for short meetings, let alone long ones! Decisions, such as the pivotal decision of whether or not to declare a disaster, must be made with a minimum of delay. All departments need to know that there is one person (or in a round-the-clock operation, one office) to whom they should report, and from whom they should expect to receive instructions. There is a place for “other hands” to help the incident manager. These people would be the incident manager’s delegates, and be entrusted by him or her with essential tasks. The people would receive their instructions, ask only essential questions (to obtain necessary information), and exit without indulging in debate or back talk. They are the recovery team, and we discuss their functions more fully in Part 2, “Planning.”.

PRINCIPLE 3 You should manage every event the same way. Different plans for different events result in confusion and contention.

What is the difference in dealing with a major fire in your call center, compared with a minor one? Answer: more smoke. Your fire response plan should be written in such a way that the same alarm bells would always ring in the same way, so that no one could get confused and continue working while everyone else was evacuating the area.1 No one should argue with an evacuation order. The same designated people would be handing inbound calls to a hot site (alternate call center, see Chapter 12) in either situation, while another group of designated people would take care of data backups. While all this was happening, one person or department would be calling the fire department while another designated person or group was investigating to determine if the fire could be put out with fire extinguishers. All of these people should have checklists, so that they cannot “freeze up” or forget a step. Whether for a major or a minor fire, the same procedures should always be followed in making head counts of employees (and visitors, if any), so that everyone can be accounted for. Few situations could be more tragic than the discovery of an injury or fatality. Having the same procedures (and checklists) for any event of a given type should prevent you ever hearing that a person who could have evacuated the building did not. You would not want anyone on the premises ever to have thought an alarm was “only a test” and not a full-scale evacuation. And having put a standard procedure in place, you would never have to apologize for the disruption of work, since everyone would understand the plan and its purpose.

© 1998 by CRC Press LLC

PRINCIPLE 4 All parties involved should know the communications process in advance.

At the point of impact, there is no time available to train or create new procedures. In what way would the recovery of inbound telephone calls to an alternate call center be different during a fire, compared with that performed after a bomb threat was received? Probably, these procedures would not be different at all. However, this is a question that must be planned for beforehand. As described above, all the personnel in a company, not just a few department heads and supervisors, must know about critical procedures and the checklists for them.2 Your company should have a toll-free number to a coordinating call center that your company personnel could call, even from a cellular telephone or pay telephone booth. Everyone should know that number, or have it in his or her wallet or purse. If the crisis were so severe that the company building was a smoking ruin, the one or two people remaining conscious would have to count on that assisting call center not only to take over inbound calls, but also to call for fire engines and ambulances. Having such a plan in place, known to all, and everyone already trained to enact it, could spell the difference between lives and business lost or you, your co-workers, and your company surviving to do business in the future.

PRINCIPLE 5 Not everyone requires the same type of communications. Identify in advance who needs what information, and how will it be provided.

As part of the business impact analysis during the planning process (Chapters 5 and 6), you will be assessing the company business functions and call patterns. This will enable you to know which departments need what sort of communications capability. The tables in Appendix A include call-type and income-loss information. The gathering of this information will enable you to determine: • Which departments use the telephone most heavily? • Which departments generate the most income? • Which departments have similar requirements? • Whether or not some departments could “piggyback” on the equipment of another department.

Note that facsimile transmissions, your Internet Web site (if you have one), e-mail, and videoconferencing are also “communications” and must be included in the survey. This kind of survey is also helpful in identifying which functions can be handed over to an “outsource” or alternate call center in the event of a crisis, and which cannot. Some departments utilize specialized knowledge for handling calls. These calls could not be handled by another call center without either extensive training of their call-takers or your company personnel being physically present at the other site.

© 1998 by CRC Press LLC

On the other hand, maybe your company has a smaller branch or secondary office that could handle the call types that require expert knowledge. The coordinating call center, helping out during a crisis, could do initial call handling and routing — and hand over calls to your experts in the “smaller” office when necessary. That smaller office might not be a “branch.” Sometimes, the head office itself might be the facility remaining operational while other parts of the company are knocked out of commission. Frequently, the various departments are located in different parts of a building, in entirely different buildings separated by lawns or fire lanes, or in different municipalities or regions. This underscores the importance of a companywide call inventory. Your continuity plan should cover all departments, divisions, and locations. We will discuss the advantages of this later, in Chapter 3, ”Illusions of Recoverability” and Chapter 15, “Call Volume Management Issues.” If your company depends on its communications for earnings (and to prevent loss), you must know which departments need what communications facilities and why. You do not want to find out how necessary the fax machine is to the sales department, or how many new sales orders are first addressed to “Public Relations,” on the day of a disaster.

PRINCIPLE 6 The incident manager will have no other responsibilities. A person performing multiple jobs will perform multiple jobs poorly.

On a day-to-day basis, the head of the “recovery team” (designated the incident manager) would perform regular duties of benefit to the normal operations of the company. The incident manager might normally manage call center operations, personnel, information systems, or some other company function. Whoever this individual is, he or she should be poised and experienced, and have capable subordinates who can take over emergency tasks in that regular department during a crisis. The incident manager is like the captain of a ship. Once a crisis starts, this individual must drop all other tasks and concentrate on safeguarding the lives, assets, and business of the company, exclusively. This person could not, for instance, forget about a bomb threat to take a call from an important client. Nor could the incident manager answer a panic call from a subordinate about a computer file while arranging for alternate call handling or staffing because of poisonous fumes, an approaching tornado, or rising floodwaters. The incident manager must be known to all as • Having the authority to act decisively and swiftly in enacting the continuity plan; • Not being the contact point for his or her normal department — such duties being handled by a designated subordinate; • Not being required to make simultaneous reports to the CEO, the entire board of directors, all the shareholders, whatever journalists might happen by, and any interested employees — at least, not until the crisis has been handled! Reports, sent and received, should be routed via the designated members of the recovery team.

Preferably, the incident manager would have an EOC or emergency operations center — an office, van, even a tent, depending on the circumstances — where crisis management communications could be sent and received with a minimum of interference. An

© 1998 by CRC Press LLC

incident manager must be left, undisturbed by nonessential distractions, to follow the business continuity plan to its successful conclusion. We discuss other aspects of this matter in Chapter 22, “Identify the Crisis Leader” and Chapter 28, “External Command and Control Procedures.” One would hope that every member of the company would respect the importance of letting the incident manager manage the “incident.” Ineffective plans include unexpected emergency procedures, muddy delegation of authority, crisis communications unknown to all but a few “team” members, or other failures of planning. Plans showing such weaknesses could be expected to have only limited success in preserving the continuity of communications, business transactions, or perhaps even lives. In conclusion, to establish effective crisis management for your company, your continuity plan must incorporate the foregoing six principles. Telephone directories from past years are full of listings for companies that have gone out of business. You and your co-workers have a personal stake in not joining them in that oblivion.

NOTES 1. We have heard of people continuing to work, or going for a coffee, while alarm bells rang and people all around them were evacuating the building. 2. The toll-free number to the assisting or coordinating call center is also discussed in Chapter 8 in the subsection, “Emergency Communications Plan.”

© 1998 by CRC Press LLC

3 Illusions of Recoverability

W

e have found that there are three predominant plans that many call center managers think will work for them. The individual plans are as follows:

• Recovery to another fully operational call center; • Recovery to a backup facility (dark site); • Recovery to agents’ or representatives’ individual homes.

We have met a large number of call center managers who have grabbed hold of one or other of the individual plans. They announce proudly that they have their “solution” already, and do not need to look further for backup of their call centers. These plans, used together, form the basis of most call volume management plans. However, these plans when used separately, if made without analysis of the call volume of a call center and a complete business impact analysis of the whole company, are likely to contain fatal flaws. We refer to these plans as the “illusions of recoverability.” Mary Kay Morgan invented the term illusions of recoverability. More information about her is available in the sources list in Appendix B. The authors developed the concept, which we explain below.

ILLUSION #1 Recovery from one fully functioning call center to another If you have succumbed to Illusion of Recovery #1 (see Exhibit 3-1), your company has two call centers in different locations, and you have been planning to recover inbound calls from one existing call center to another, fully operational one. The idea is that the spare capacity of the “recovering,” or assisting, call center can be used to take up all the calls from the disabled call center.

Exhibit 3-1. Illusion 1.

© 1998 by CRC Press LLC

Exhibit 3-2. Problem with Illusion 1.

Problem The problem with Illusion #1 (see Exhibit 3-2) is that the “spare capacity” of the assisting call center is finite. Suppose that the disabled call center was operating at 80% of what it could do, and that the assisting call center was operating at 70% of capacity. When the disabled call center experiences a “disaster,” be it a power outage or failure of the PBX or associated equipment, all calls are rerouted (by arrangement with the telecommunications provider) to the assisting call center. The disabled call center is then operating at 0% of capacity, and the assisting call center is operating at 100% of its capacity. Since the assisting call center only had extra capacity of 30%, only 30% of the calls of the disabled call center can be answered at the assisting call center. Therefore, the remaining 50% of the disabled call center calls go unanswered. The result is an overburdened assisting call center. The call-takers (CSRs) are liable to burn out, since there is no excess capacity available to provide for rest, sick days, or other purposes. The 50% of the disabled call center calls that cannot be answered represent a large number of customer or other messages that cannot be received. If these calls represent sales orders, your company has (in this example) lost 33% or one-third of its telephone sales income for the duration of the outage of the disabled call center. The lost calls represent business that might migrate to your competitors, or customers/clients who might become disgruntled or conclude that you have gone out of business. If anything happens to the assisting call center before the disabled one can be “recovered,” or brought back online, the problem gets even worse. There is no slack remaining at, and there is no backup for, the assisting call center. Meanwhile, your call-takers are showing a loss of morale and are at risk of becoming ill from overwork, or quitting to escape the stress of the situation. Consequently, if one call center goes down, you effectively wind up with two disasters. If you have more than two call centers, there are more call-takers to share the incoming calls. The problems of finite spare capacity, increased stress on personnel, and increased vulnerability to other disruptions still remain.

Example Suppose that you have not only two, but three call centers in your company. Two are regular inbound call centers doing direct telemarketing for your company manufactured products. One call center is at your head office, and receives mainly inbound customer service calls relating to complaint resolution or product location. A small department does collection calls to retailers that have been slow to pay.

© 1998 by CRC Press LLC

The remaining calling, which is done from the third (head office) location, consists of executives and other personnel sending or receiving calls related to the management of the company affairs. A number of faxes are received each day, mainly directed at the sales department, although some relate to the purchase of raw materials for the factory floor. Because the CFO takes his or her duties to the shareholders seriously, there is very little excess capacity in any of the call center facilities. Any “slack” is mainly due to seasonal fluctuations in the numbers of calls, not to any planning on your part! Suppose that a chemical spill nearby causes the evacuation of the personnel of one call center. Suppose also that this event occurs during the pre-Christmas sales push, and the other telemarketing call center is tasked to capacity. (Even in summer, it would still have only limited capability to assist.) Where are there extra personnel to take the calls upon which the company revenues depend? Even if the head office telephones were used for this purpose, there are two problems to consider: • Who would make the calls, considering that head office functions also are important? • What information would they use to make such calls? Unless there is a real-time data transfer taking place to update the sales department database, there is no way to tell which names on a list might have been called already or, indeed, what names are on the outgoing call list at all.

Very likely the telemarketing call centers together would account for as much as 60% of the company revenues. In the case of an outage, how long could the company do without that 30% of its revenues? If the seasonal element of pre-Christmas rush were considered, that might be a major part of the company earnings for the year. The outage of the first call center might not be a chemical spill. It could be, just as easily, a power failure, computer hardware failure, lightning strike, or sabotage, such as a computer virus picked up on the Internet. Even a wastebasket fire could put a call center out of commission for hours, or even days if it should spread into the surrounding offices. This and other situations that could be classified under the heading, “Illusion of Recoverability 1” would be more easily handled by a recovery team utilizing an up-to-date call center continuity plan. Such situations would be better handled yet if the liabilities of planning according to this illusion were rectified. See Part 3, “Call Volume Management.”

ILLUSION #2 Recovery to a Cold Site (Dark Site) If your company is operating with one functioning call center and one vacant backup facility (dark site), you probably feel that your potential problems are all taken care of (see Exhibit 3-3). If you have succumbed to Illusion of Recovery #2, in the event of disaster you have been planning to recover inbound calls from the operational call center to your waiting, nonoperational one at a distant site. The idea is that the dark site, or backup call center, can be used to receive all the calls from the disabled call center. Suppose that your original call center has a disaster, whether it is food poisoning of employees or an evacuation of the street by a police SWAT team chasing armed robbers or terrorists. Perhaps its something longer term, like condemnation of your office building because of a dangerously weak foundation, or a heating system prone to explosion of combustible fumes. While the original call center has been operating at 100% of its capacity, the waiting backup facility has been operating at 0% of its capacity.

© 1998 by CRC Press LLC

Exhibit 3-3. Illusion 2.

Problem Some dark sites or backup call centers already have electrical power and telephone service hooked up and functioning. The company has been patiently paying those utility bills against the day of need. In that case, call-takers need to be transported to the site, which takes time. Computers need to be loaded with the current database information, which takes time. The telecommunications provider (Telco) must be requested to switch incoming calls from the original to the backup call center. That too takes time, but if a plan to do such a thing is already in place, it should not take too much time. We have heard of backup facilities that consisted of vacant offices that really were “dark,” because they had no electrical or telephone hookups. Some of these had warehoused office furniture, chairs and desks, stacked up one on top of one another. Some had desks but no computers, or had computers but no software. Some dark sites served as an archive for old files and equipment, but had no current files, computers, software, or mailing lists. Some had telephones served by an inadequate switchboard or PBX, or not enough trunk lines, because capacity had not been updated as the company grew. The worst backup facilities would have to be those lacking heat or water. They really would be “cold” sites. Who could work there? The result of trying to recover from a disaster at an inactive facility is a continuation and development of the disaster. Before your CSRs can get on the telephones to receive calls, all the other functions that support them must be provided for. Meanwhile, the 100% of the disabled call center calls that cannot be answered represent a huge number of customer or other messages that are just getting lost. If the lost calls represent sales orders, your company has (in this example) lost all of its telephone sales income until the backup facility can be brought online. The lost calls represent business that will almost certainly migrate to your competitors (depending on what industry you are in), or they are from customers/clients who might become disgruntled or conclude that you have gone out of business. And until you can receive calls again, they are right! If your backup facility is within a short drive from your original site, and the furniture, power and telephone hookups, water and heat are already provided for, you might be fortunate. It is possible that your call center personnel could arrive at the site in a few hours, and in only two to eight hours more have all their computers booted up and loaded with your

© 1998 by CRC Press LLC

current database information. (If your call-takers were suffering from disease or some form of poisoning, this plan still could not work, even with everything else provided.)

Exhibit 3-4. Problem with Illusion 2.

Major preparations take major amounts of time to perform. How long would it take to get electrical power online, and turn on the lights? How long would it take to unstack the desks and chairs? How long would it take to put a functioning computer and telephone on every desk? Who would do that work — the call-takers when they stepped off the bus, your information systems technicians, or crews of contractors at a large fee? Would you be satisfied with getting a small number of work stations operational quickly and putting more work stations together gradually? Probably the first group of call-takers would suffer burnout, as in the Problem with Illusion #1. Because of “loss of morale and [being] at risk of becoming ill from overwork, or quitting to escape the stress of the situation,…” they would not be happy campers. The disabled call center would still be operating at 0%, which is to say, not at all (see Exhibit 3-4). Perhaps the backup facility might operate at 10% in the first week, at 30% in Week 2, and not until Week 3 or 4 might it operate at 100%. In summary, the ramp-up time to bring the backup facility to full capacity could be days, weeks, or even months. If the disabled call center receives just 30,000 calls per day, in the situation described in the previous paragraph, over 300,000 calls would be lost. The loss of revenue or customer confidence could be as damaging as the original disaster.

Example During a hurricane, a company decided to move its call-takers from their call center to their dark site facility. It appeared that the hurricane would be savaging their area soon, and the call center was in the projected storm path. Normally, the drive there would take between four and five hours. However, because of the hurricane, the roads were choked with vehicles of people evacuating the area. It turned out that the company buses did not reach the dark site facility for more than thirty hours. By that time, the hurricane warning had been rescinded. The storm had missed the original call center, which had been thought to be in its path. Management thought that since they had had such a hard time making the trip through the storm, they should continue with the dark site recovery. They commenced bringing the computers and telephony equipment into operation. In the end, it took another whole day to get the equipment online and capable of receiving calls.

© 1998 by CRC Press LLC

If the company could have switched calls to an overflow or assisting call center, management could probably have minimized the financial loss. Even if the busloads of call center personnel had been delayed by trying to travel through the course of the hurricane, even if the dark site facility had not been ready, the company would still have been earning revenue during that time, by the efforts of the subcontracting call center. As it was, the company lost three days of revenue-producing telephone calls. The company was severely affected by it.

Exhibit 3-5. Illusion 3.

ILLUSION #3 Recovery to a Virtual Call Center A company might operate with only one call center and, therefore, have no backup facility or assistance possible from any other call center. You might have considered a telecommuting scheme. If you had succumbed to Illusion of Recovery #3, in the event of disaster you would have been planning to switch inbound calls from your disabled call center to your employees at their homes. The idea is that the telecommunications service provider (Telco) would send inbound calls to all the CSRs at their place of residence, and theoretically no calls would be lost (see Exhibit 3-5). Suppose that the company is operating with one call center only, which is operating at 100% of capacity. Something happens, maybe short term, like a power failure lasting half a business day for your city block. The utility that supplies you with electrical power is upgrading its poles, or burying the lines underground. Or perhaps the disruption is expected to be intermediate term, like replacement of your switchboard or all your computers, because of a lightning strike. Or perhaps the disruption is expected to be very long term, because that lightning strike burned down your office building.

Problem You, the call center manager, declare a “disaster.” Your call-takers return to their homes to receive calls there. Traveling only in the local area, they will probably arrive at their home telephones in 15 minutes to half an hour. During that time no calls are answered. While your disabled call center is operating at 0%, the telecommunications service provider (Telco) starts to send calls to your call-takers. If Telco waits for a signal that the calltakers are in position, the calls started flowing again upon that notification, but 15 to 30 minutes of calls are lost. If the telephone service provider was not waiting for a signal that the personnel are home, the call center personnel unlock their doors hearing their tele-

© 1998 by CRC Press LLC

phones already ringing! Those sharing accommodations with stay-at-home family members are likely to be yelled at by them. Furthermore, calls answered by anyone’s spouse or grandparent, or child would be, if anything, worse than just lost, because of the confusion to the customer. Unfortunately, the coordinating facility (your switchboard or automated call-distribution equipment — ACD) is no longer operational. Therefore, one or two individuals receive all the calls, and quickly burn out. The remaining staff receives no calls, no information, and suffers anxiety and loss of morale.

Exhibit 3-6. Problem with Illusion 3.

The few call-takers receiving all the calls would be unable to report back to you, the call center manager, because their telephones would start ringing immediately after the finish of each call that they received. Their only chance to give you feedback would come if they happened to have a cellular telephone available, or have some means to pass a note to a cooperative neighbor. And, depending on the disruption, you would also necessarily require a cellular phone, and the employees would need to know your phone number! In summary, the company coordinating computer could not direct the calls to staff in rotation. Instead of your company experiencing recovery from disaster, it would in consequence experience the occurrence of a second disaster. The one or two individuals handling the barely 1 to 5% of calls that could be distributed by this method would burn out, leaving the company again operating at 0%. The call center would be, effectively, out of business (see Exhibit 3-6). Are there ways around the problem of requiring a coordinating facility? Yes, there are, and they are discussed in Chapter 15, “Call Volume Management.”

Example We ask you to cast your mind back in time to various “wide-area” disasters such as the North Dakota floods, the Northridge earthquake, and hurricanes and typhoons Andrew, Fran, and Iniki. The happenings that we describe are drawn from accounts of the progress of Hurricane Andrew over southern Florida. However, the disadvantages to call centers would be the same in many wide-area disasters. Not only one-call-center companies, but also companies with several call centers located nearby each other (within 20 miles), are vulnerable to failure of infrastructure. If power lines, telephone lines, and water mains were destroyed by a calamity, even an undamaged call center would be useless.

© 1998 by CRC Press LLC

In the case of Hurricane Andrew, that was exactly the case. The area around south Miami was devastated. The power grid and telecommunications systems were crippled by wind damage. Water supply for firefighting became unavailable because water mains were ripped apart by the rising roots of wind-felled trees. Consequently, emergency personnel, for fear of fire, closed whole city blocks, and even call centers with their own electrical generators would have been “out of luck.” Without electricity to power computers, or telephone lines to carry calls, both actual and virtual call centers could not carry on operations. In the aftermath of Hurricane Andrew, call centers outside the disaster area and those with alternate facilities far away fared the best. It is important to note at this point that the personnel who are supposed to be the “virtual call center” staff must be considered also. During Hurricane Andrew, more than one company relied upon critical personnel who evacuated the area ahead of the storm. Even volunteers cannot be blamed for considering their personal survival, and that of their family, ahead of their work commitment. Unfortunately, if the people who have evacuated are the same ones who are supposed to enact the business continuity plan, the plan will not be employed successfully. There could be a benefit to letting some employees take their families away in advance of the storm (in the case of Andrew, to northern Florida or even Georgia) and return alone to help the company recover from disaster.1 Such people would not have to worry as much about whether their families would have shelter from the rains, or have sufficient food and potable water to support life. Therefore, they might be willing to delay surveying the damage to their residences to get the company back on its feet quickly. The company would, however, have to guarantee financial assistance to such volunteers to receive such dedicated service. Employees who had been out of the area during the storm would suffer less postdisaster trauma (see Chapter 29) and have more energy to give to the task of reconstruction. As far as we know, only very large companies such as AT&T or IBM would have been able to bring workers in from other regions to help. (For more detail, see Appendix C, “Wake-Up Call: A Case Study of Hurricane Andrew.” Let us look back now over the three Illusions of Recoverability: • Recovery to another fully operational call center • Recovery to a backup facility (dark site) • Recovery to agents’ or representatives’ individual homes (the virtual call center).

We would like to point out that use of only one of these approaches is likely to produce a call center continuity plan containing grave weaknesses. We encourage you to read this book and then undergo the planning process. Part 2, “Planning” and Part 3, “Call Volume Management” will help you whether you have no continuity plan at all or are probing your present plan for weaknesses. Part 4, “Testing” will help you plug the holes in a plan once it has been written, and Part 5, “Crisis Management” will help to prepare you for going through a disaster and its aftermath. We wrote Chapter 4, which follows, for those who are, or work for superiors who are, doubtful about whether or not the whole procedure is necessary. “Rationalizing Procrastination” is the last chapter in Part 1, “Principles of Voice Recovery.”

NOTES 1. An alternative to evacuation is discussed in Chapter 8, under the subsection heading, “Emergency Operations Center.”

© 1998 by CRC Press LLC

4 Rationalizing Procrastination

T

he excuses that follow (and that is what they are) are things that various company representatives have told us in the course of many business continuity presentations and seminars. Representatives of a company might have told us about such-and-such a calamity and how they survived it. Or, how their former company suffered a given type of calamity, and did not survive it. The representatives of these companies (surviving or failed ones) listed the excuses for not planning as some of the biggest obstacles they faced in the planning procedures and in the actual recovery from the disaster itself. People at these companies were not receptive to the idea of spending money on disaster recovery planning or business-continuity planning. This was especially so if they had already experienced the stress and expense of recovering their call centers from disaster. Senior management considered it a low-priority item. When many of these companies got themselves into trouble, there were no resources available to work on their call center recoveries, let alone on planning to avoid a repetition of the events. In a lot of cases, that itself (lack of resources) can be a big problem. Ensuring that a company is prepared to survive will not be an easy task. We hope that this discussion will help you with the situation of your company.

THE EXECUTIVE WON’T SUPPORT THE IDEA Who is the person who is pushing for planning? Unless the impetus comes from the board of directors or owner of the company, it is possible that this person (perhaps you, the reader) might feel some trepidation. If the senior executive does not support the idea and expense of call center continuity planning, you might have to point out this passage, and say, “Does this look like you?” Depending on the individual’s tolerance for new ideas, you might wish to present the paragraph in a photocopy, which can be ripped up with no lasting impact! The first part of planning for the survival of your company is “Executive Commitment.” If the senior executives are not committed to this project, their lack of support will kill this initiative before it becomes more than your possession of this book in your office. Perhaps someone has been told to go and “do this thing.” Perhaps to reduce insurance rates, or to satisfy a lender’s requirement, the company must appear to produce a continuity plan. Perhaps senior management expects a subordinate to just prepare a plan and leave it with them. But the senior executives may only plan on putting the resulting document on the shelf and forgetting about it. In such a case, the person tasked with building the plan might as well not bother to start on it. The “planner” might as well go and do something more productive, because in that case the planning process would be a waste of time. The plan must be a living, breathing document that changes with time. The “care and feeding” of the plan must be ensured for a period of time (as long as the company exists, we think) and somebody must have the responsibility for that care. The updating of telephone numbers, forwarding of copies to all the appropriate parties, and changing of departmental checklists to reflect changing operating conditions must all be done in perpetuity. Producing a plan that is meant to ensure the survival of the company, and then archiving it, is an exercise in futility. It might be that all you (the reader) can do, to try to gain executive support that has not been forthcoming, is to do a business impact analysis. This is the best way to show what can be lost if the company cannot carry on business, particularly its call center business.

© 1998 by CRC Press LLC

EXHIBIT 4-1. COST OF A ONE-HOUR COMMUNICATION OUTAGE BY INDUSTRY Type of Business

Average Hourly Impact

Airline reservations

$89,500

ATM services fees

$14,500

Brokerage operations

$6.45 million

Catalog sales

$90,000

Cellular service activation

$41,000

Credit card authorization

$2.6 million

Home-shopping channels

$113,750

Online network fees

$25,250

Package shipping service

$28,250

Pay per view services (TV)

$150,250

Source: Contingency Planning Research, Inc., Jericho, NY.

If you are the owner or CEO and remain doubtful of the benefits of continuity planning, we recommend that you, also, do such an analysis. We deal with the business impact analysis in Chapter 6. It will take you via a step-by-step process through the different types of telecommunication done by your company, and how much they mean in dollars and survival. It could be that if the customers of your company were not able to get in contact, the company could lose (depending on your industry) anywhere from thousands to millions of dollars per hour. Consider that such losses because of an outage (see Exhibit 4-1) can range from $14,000 to as much as $6.5 million per hour of call center downtime. Over the course of a day, what does that total up to? What do you figure a 24-hour communications outage would do to your company? It is possible that the senior management of your company might be thinking only in terms of large-scale disaster. If it seems that your area is not prone to hurricanes or earthquakes, perhaps your company has the illusion that disaster happens only to other people on the television news, far away. Even in a storm or quake “belt,” perhaps your area has been missed by the misfortunes that have befallen others. These comfortable illusions must be dispelled if your company is to survive the buffets of misadventure that await it. The “disasters” most likely to disrupt a business are (1) human error and (2) failure of equipment or infrastructure. Many such events do not start as an easily identifiable catastrophe. Rather, they creep up, starting as slight irritations that slowly grow in frustration-causing ability. Although sometimes a disruption could arrive with the suddenness of a drive over the edge of a cliff, in many cases the circumstance that could kill your company might take days, weeks, or even months to develop to its full destructive potential. Considering the experiences of various companies, we find that losses sustained because of “slow-motion disasters” are the worst to predict. (You can find a detailed discussion of slow-motion disasters in Part 2.) Frequently, managers are caught unawares by these “creeping” disaster situations. They do not know whether the disaster will be serious enough to start spending money to combat until it is already too late to do anything about it. Until

© 1998 by CRC Press LLC

the crisis becomes obvious, it does not appear to have an impact on company revenues or survival. After the crisis becomes obvious, managers might be unable to preserve company revenues or survival.

“WE’RE TOO BUSY” This rationalization for failure to plan for business continuity gets a lot of use. Without mentioning any names, some major companies have used this one. “We really are too busy,” they say. And yet, continuity planning, once you have done the basic preparations, can take a surprisingly small amount of time to actually work on. A lower-level person (middle management), having been given this task by supportive upper-level management, could probably maintain larger elements of a plan in a couple hours each week, once it had been “built.” Has a new procedure been brought into use by the department that fulfills your customer orders? Did the company just buy cellular phones or an emergency electrical generator? These are just “updates” once the plan itself has been written. That plan could literally save the company from being wiped out of existence. So you have to ask, how busy is too busy? Are you too busy to save your company from collapse? Let us look at the kind of information contained in a business impact analysis, to make the business case for this. Calculate how much your loss would be from being “down” for a day. As we have said before, depending on the size of your company it might turn out to be substantial amounts of money — in the millions of dollars if your company is large, hundreds or thousands if it is smaller. The point is, unless the day of your communications outage is a statutory holiday, and you would not be open for business in any case, your company would suffer a damaging loss. The situation of your company will determine if a one-day outage would “damage” or “doom” the firm. According to FEMA, it takes fifteen times the amount of money to recover from a disaster that it would have taken to carry on with a properly executed recovery plan.

Compare the amount of money that your company would lose to the amount of money it would save. Consider that 15:1 ratio. The work to build a call center continuity plan, calculated on an hourly basis, does not seem all that bad. It does take a period of time to get the planning procedure started, and it does take regular maintenance to keep the checklists up-to-date. However, once the plan is developed, even secretaries or clerks could ensure that the “contacts” (vendors, customers, emergency numbers) database is maintained. Your human resources (HR) people could ensure that all new employees were logged in on the continuity plan to keep departmental contact lists (office and home) current. HR could also ensure that all new personnel were familiarized with the basic parts of the plan, and why the company needs one. Your aim in proceeding with call center continuity planning is to build a plan that will ensure the survival of your company — its ability to “continue” — whether the difficulty is minor or severe, whether it is short in duration or lasts for months. Are you too “busy” to provide for the survival of your company?

“WE DON’T HAVE THE BUDGET FOR IT” The rationalization that your company has no budget for continuity planning is similar to the excuse of being too busy. Does the senior management of a company feel that the company really does not have the budget for building and maintaining a continuity plan? Then

© 1998 by CRC Press LLC

those executives might want to look at doing a business impact analysis to ensure that the company would not lose very much money if its call center should go down. If that did prove to be the case, your company would be invulnerable to the risks contingent upon existence in this universe. You would want to ensure that loss of buildings, electrical power, or employees would not have an impact either. If that were the case, you could forget about continuity planning. Your company would not need to do any continuity planning, and you, reader, could close this book right now. Perhaps yours is a “virtual” company that exists upon the Internet. In that case, it is even more vulnerable than most to failures of infrastructure, telecommunications faults, and the problems that can befall computers. Does your business impact analysis prove that disruptions of business would be very expensive for you? Then you should look at the costs of trying to recover from a disaster without having preplanned your way out of it by means of arranging contracts in advance. The item that might cost you hundreds of dollars if the order followed normal channels might cost thousands of dollars to buy in a hurry. Your company might have to pay large amounts to ensure its survival after the event. The costs to recover from business disruption might be huge if your whole region were beaten up by a wide-area disaster, and your supplier had to fly the goods in from across the continent, and you had no preexistent contract providing for emergency supply. For instance, your company might consider setting up a contract with a vendor to bring high-capacity diesel generators to your facility in case of a prolonged outage of electrical power. If you prearranged this in advance, you could find it did not cost too much. Perhaps a retainer paid quarterly could build up a contingency fund in case of need. In the middle of an ice storm or blizzard, when the vendor might be inundated with requests for generators, naturally the vendor’s representatives would deliver to preexisting customers first. Those not already a customer would probably wish they could get that wonderful price that the companies with forethought would enjoy. This type of contract is pretty much standard for the recovery industry. Consider the recovery service offered by IBM. Suppose that your company arranges for call center assistance at $25 per “seat” at a hot site facility run by IBM, enters into a contract, and pays a retainer. Naturally, your firm would pay less for this service than a company that called up in the middle of its disaster to ask for immediate help, no matter what the cost! Suppose that your region suffered a wide-area disaster such as a hurricane or earthquake, and every company for miles around was calling for recovery assistance. Your company, if it was already a customer of this service, would already have contact numbers for activating the call center assistance according to your contract. IBM (or another assisting call center) would already have current backups of your accounting and customer databases, and pop-up screens for your call center database(s). You might even know the person supervising your account by his or her first name. Your company customers and other contacts, especially those outside the disaster area, might not even know you had suffered a disruption. For them it would be “business as usual.” The company that did not have such a contract in place at the time of a disaster might be faced with sending already exhausted survivors of disaster across the state, the country, or even the continent, to the nearest hot site facility with spare capacity. This bedraggled group would carry whatever files, publications, or data disks could be hauled out of the wreckage of its offices. As a consequence of the failure to plan, there would be a long delay for travel, plus another delay while the excellent people of the assisting call center tried to bring some kind of customer interface into operation. All of this would be very expensive. Also expensive would be the losses from disruption of communications with the suffering company customers. The companies that had done planning and budgeting in advance, and paid relatively small amounts of money to retain services in advance, would find it similar to paying for

© 1998 by CRC Press LLC

insurance. If you think about it, continuity planning is very much like insurance. A little bit of forethought and money protects you from having to pay out a lot of money in case of some adverse event. Suppose that your company had a fire. If a company representative were to phone up an insurance company and ask to be insured for yesterday’s fire, what would be the response? The insurance company, given the opportunity to write a policy for the building that was already burned down, would probably refuse. If the insurer did accept this new business, the premium would cost as much as the building was worth (plus administration costs), or more. This might be an extreme analogy. But trying to recover a business that has undergone a disaster, after the disaster, would be a similarly expensive proposition. The importance of preplanning and prearrangement of vendor emergency contracts applies to those innocuous-seeming “slow-motion” disasters as much as to the more dramatic seeming kind of catastrophe. Whatever the cause, the company that cannot carry on business is at risk of failing to “continue.” A large pool of retained earnings is not as good as a plan. A “fairy godmother” with deep pockets is not as good as a plan. Certainly, even an interested buyer for the shambles of your company is not as good as a plan. Remember the FEMA observation that recovery from a disaster costs fifteen times the expense of using a properly enacted plan. If your company does not have the budget to plan against disaster, it will not have the budget to recover from a disaster. The alternative to planning is to say, “In case of disaster, we’ll just close up the company and go home.” Do you think your competitors have the budget to prepare and maintain a continuity plan?

“WE DON’T KNOW HOW” The plea of ignorance is another commonly heard excuse for the failure to plan. We hope that no one you know has resorted to it! There are so many consultants and software packages available. Your company can obtain help to go through the planning process step by step. LDRPS (living disaster recovery planning software) by Strohl Systems has both business impact analysis software and a complete set of planning software. That is just one avenue to this venture. Virtually every major accounting company offers a recovery planning service. Various consultants go to call center industry trade shows to offer their services. This book itself gives step-by-step instructions for building a business continuity plan with a call center orientation. After considering all the foregoing, anyone who continues to say, “We don’t know how to plan” is really saying, “We don’t want to plan.” This is like saying, while braving the U-boats on a trans-Atlantic voyage during World War II, “I don’t need to know about life jackets or life boats.” It might seem hard-nosed to say so, but the company that does not “want” to do continuity planning should shut its doors immediately. Why wait? Surely a better use could be found for the energy of that company of the creative employees. A better use could be found for the assets of that company.

“WHERE WOULD WE BEGIN?” Fortunately for those unsure of where to begin, this book itself will describe the process of call center continuity planning. Step One is the business impact analysis (BIA) we describe in detail in Chapter 6. Following this procedure will enable you to determine where your company is most vulnerable to disruption. Where are its revenues most at risk? What liability or regulatory requirement is most dangerous to the continued well-being of the company if things should go wrong?

© 1998 by CRC Press LLC

Where would disaster hurt the worst? That is where you start your plan. You gradually expand your plan through sections to take into account vulnerabilities that would hurt less, over time. You do not have to write the whole plan all at once. It does not have to be finished in five minutes, in an hour, or even in a week. How do you eat an elephant? — One bite at a time. The building of a continuity plan can be done in little bites too. Your plan can grow from providing against the most damaging vulnerabilities to the least. Plan for the critical functions first according to the BIA, in a step-by-step, bite-by-bite process.

“YOU CAN’T PLAN FOR EVERYTHING” This rationalization for not planning against disaster is not exactly an “excuse.” It is true that you cannot plan for everything. As you will see while we go through the discussion of the planning process, you would not be planning for everything. What you will do, if you proceed with the planning process, is to write a plan applicable to an escalating series of adverse circumstances. If you had a tiny fire, you would do “this.” If you had a large fire, you would do “that.” Your plan would be adaptable. You would provide your company with the “fire engine” that we talked about in Chapter 1. Your plan would show what to do, and how to do it, in the event of a disaster. Remember that a “disaster” is any unplanned man-made or natural incident that causes an interruption of your business functions. You cannot plan for everything, but you can plan how to do things in the event of emergency. It might seem amazing, but if (God forbid) an emergency should arise that you did not ever foresee, training and planning would probably allow you to deal with it effectively. You have probably heard police officers and firefighters talk about it on television many times, when they have prevailed in a bad or unusual, life-threatening situation. They say, “Our training took over.” Like you, the police and other emergency personnel cannot train for every possible event that could occur. It is impossible. What they do train for is specific procedures that will be followed in various types of situations. And even emergencies can be categorized into broad types. How these types of emergencies are categorized by severity will be described later. You cannot plan for everything. Nevertheless, you and your company would meet the unique challenge of emergency in far better shape if you had a continuity plan. That plan would give you checklists enabling you to safeguard your company data, communications, customer base, personnel, and other essentials. Having handed over the company telecommunications to an assisting call center as planned, you would be able to relax and enjoy your emergency in peace! (A small joke.) And a well-constructed continuity plan would prepare your company to deal with most eventualities that could occur.

“IT CAN’T HAPPEN HERE” People want to believe that disasters only happen to other people, far away. They might give a “ho-hum” to the nightly news report from the other side of the planet, or another part of the country. They might watch it avidly, and discuss it with their neighbors or co-workers. However, they have a deep-seated resistance to entertaining any thought that such disasters might happen in their own neighborhood or office building. Maybe someone at your company is in a similar state of denial. We had a terrible time trying to convince call center managers in “La Belle Province” of Quebec that they were at risk of disaster. Yet, within twelve months, they suffered the ravages

© 1998 by CRC Press LLC

of a flood in one area and an ice storm in another. Other regions of North America have suffered disaster after disaster at the hands of the El Niño ocean current also. The ice storm in 1998 was called “the Storm of the Century,” and was the second “Storm of the Century” to hit Canada and the northeastern United States within three years. The first storm hit the Maritimes and just touched Quebec, and the second storm hit Quebec and touched the Maritimes. The “lucky” people in New York and New England witnessed both of them. Power supply to home and business was cut off in Quebec because HydroQuebec did not have adequate disaster recovery or communication plans. Maine, however, was in better shape, because a company named Twenty First Century was handling the emergency communications of the utility. As a result, repair crews in Maine were able to discover the locations of the power outages and to restore power to the utility customers quicker. There was not as much “disaster” in Maine as there was in Quebec because of the preparation and planning that those responsible in Maine had done. Contrarily, the Quebeckers showed great fortitude in their reaction to the hardship of the ice storm. There were people who worked by candle light in unheated offices. Yet, there was no reason why the banks and other companies should have had to run their offices and communications centers without electrical power. Their stalwart employees would have benefited by having managers who did a better job of planning for this eventuality. To revisit the topic of electrical power, there are companies in the marketplace that can provide electrical generators of various sizes and capacities. Diesel generators able to provide power for houses, call centers, even whole towns, are available. We heard of a company in eastern Ontario, part of which was also hit by “Ice Storm ‘98,” that donated its excess generating capacity to its grateful municipality. If your company is located anywhere in the Northeast, the sight of trees bowed down by ice is likely to remain in your memory forever. We hope that if you suffered through this event, you have decided to ensure that provision for loss of electrical power is included in your company (and even household) continuity plan. All of your facilities should be able to continue operating without the electricity, water, or natural gas supplies that you normally depend upon. This is because all your facilities are vulnerable to the loss of these supplies. The continuity of your business depends upon provision of alternative supplies in the case of sudden loss of these necessities. Perhaps an influential executive in your company says, “How often does such an event strike?” Severe weather of one kind or another can strike any part of North America, or the world. Failure of power supply, telecommunications, human stamina and will, general sorts of machinery, computer equipment, or water supply can happen at any time. The authorities can shut down such transportation links as bridges, railway lines, highways, tunnels, or airports at any time for a variety of reasons. Exhibit 4-2 shows how often such events might occur, and what they can cost in business lost. If 450 of the Fortune 1,000 companies have an average of nine disasters (of whatever type) per year (see Exhibit 4-3), why would your company fare better? The cost per hour of downtime will, of course, vary from company to company. Going through the BIA will help you determine what the cost is for your company. What are the most likely events that could affect your company? Considering business disruptions of a general nature, what are the most likely events that could affect your company? The category of “Other” in Exhibit 4-3 includes HVAC service failure, software error, employee sabotage, burst water pipes, “contamination,” insects, riots, and transportation accidents. There is no place in the world immune from every one of the possibilities listed above.

© 1998 by CRC Press LLC

EXHIBIT 4-2. COST OF DOWNTIME AVERAGES Item

Statistic

Cost per hour of downtime

$78,000

Hours per incident

4.2 hours

Incidents per year

9 (average)

Downtime cost per year

$2,970,000

Statistics from Contingency Planning Research, Inc., of Jericho, NY. Statistics drawn from experiences of 450 of the Fortune 1,000 companies.

EXHIBIT 4-3. MOST COMMON OUTAGE CAUSES IN THE UNITED STATES* Type of Event

Percentage of Occurrence

Power outage

27.7

Storm damage

11.7

“Other”

10.1

Flood

9.6

Hardware failure

7.7

Bombing

7.2

Hurricane

6.3

Fire

5.6

Power surge/spike

5.1

Earthquake

4.9

Network outage

2.1

“Human error”

2.0

* Breakdown of causes for 5,320 outage incidents recorded by Contingency Planning Research, Inc., since January 30, 1995.

Let us move from general disruptions of business to the specific one that this book will examine most closely — loss of communications. Exhibit 4-4 does not describe how many companies would fail. The mere “34” and “17” are percentages of companies that would survive. The information does not indicate whether those companies would survive well, or in tatters. Communication with the world by whatever means (telecommunication) is an essential condition for business in modern times. Lack of the ability to communicate almost foredooms the company that suffers it. An outage of telephone lines is a difficult problem to deal with. The only effective way of dealing with

© 1998 by CRC Press LLC

EXHIBIT 4-4. DAYS OF SURVIVAL WITHOUT TELEPHONE OR TELEPHONE LONG DISTANCE SERVICES Time without Local Telephone Services

Companies That Could Survive, %

Time without Long Distance, days

Companies That Could Survive, %

Less than 1 day

34

Less than 1

22

1 day

23

1

17

2 to 3 days

17

2 to 3

18

4 to 7 days

26

4 to 7

13

Figures from Contingency Planning Research, Inc., of Jericho, NY.

it is to have a facility available for use in a nonaffected area. As with most aspects of continuity planning, arrangements for such facilities are best made before the event. Like telecommunications, the use of computers has become essential to business. The University of Minnesota found that 93% of businesses that lost their data center for ten days or more filed for bankruptcy within one year. Of those filing for bankruptcy, 50% filed immediately after the disaster. Except for such a circumstance as the failure of a switchboard or the cutting of a trunk line, most of the circumstances that affect business communications also affect business computing. One may sympathize with the impulse to avoid thinking about disaster. However, the shareholders, whose assets could be destroyed, and the employees, whose jobs could be wiped out, would not agree that such denial is proper to the executives entrusted with management of the company. It is possible that various regulatory bodies, or customers suing for breach of contract, would also not approve such negligence. (For more on this subject, see “Lawsuits” in Chapter 6.) If continuity planning for your company is opposed by someone because of the claim that disaster “can’t happen here,” or “won’t happen,” or is “unlikely,” they are wrong. Business is frequently disrupted by various adverse events large and small, everywhere! Senior management that ignores or denies that there is risk is a liability to the survival of your company. In conclusion, the rationalizations for not proceeding with continuity planning are: 1. “The Executive Won’t Support the Idea.” 2. “We’re Too Busy.” 3. “We Don’t Have the Budget for It.” 4. “We Don’t Know How.” 5. “Where Would We Begin?” 6. “You Can’t Plan for Everything.” 7. “It Can’t Happen Here.”

Various people explaining why they did not plan gave these rationalizations to us. These seven dismal excuses could be etched on the tombstones of the companies they cited. They would make fitting epitaphs, poignant words memorialized to be read by those who come afterward. We think it is better to be a survivor.

© 1998 by CRC Press LLC

Part 2

Planning

T

his section discusses business impact analysis, risk analysis, planning stages, team selection, and plan contents, such as critical documentation, inventories, and procedures. It also contains chapters on team responsibilities and backup procedures. We talk about disaster implementation tasks such as whether to declare a disaster (or not), life preservation procedures, and alternative sites. This section also discusses the tasks necessary for a return to normal operations, such as LAN and call volume management restoration — because, while welcome, the return to “normal” might not always be easy. We talk about plan maintenance at the end of this part as well. The process of reviewing a plan and deciding “what we would do next time” can reveal valuable insights about the function of your company during a disaster or other inconvenience. It can also point you toward identifying changes in your marketplace or other aspects of your operations before they can come up and catch you by surprise. Each of the stages in planning has a dedicated chapter. These stages should be followed in turn. Together they form the necessary preparation for the call volume management procedures we describe in Part 3.

© 1998 by CRC Press LLC

5 Plan Stages PREPLAN You have had the opportunity in the first four chapters to see the reasons for disaster recovery planning. What we are going to do now is take you through the stages of writing the plan. First comes the preplanning stage. This is effectively what you are doing right now.

Recognize Problem Consider the problems of continuity planning for companies having call centers. The continuity plan is adaptable. It is a set of guidelines on what to do and how to do it in the event of a disaster. A disaster is any planned or unplanned, natural or manmade incident that causes a disruption in your business function. Anything can happen, and you cannot plan for everything. However, you can consider how to accomplish most things in the event of an emergency. In anecdotes told to us, people say that it is amazing that an emergency came up that no one thought of, but their training and planning allowed them to deal with it effectively. Proportion of Companies Lacking a Plan. Most call centers are located in highly populated areas. This means that a disaster in a highly populated area such as Oklahoma City or Toronto would cause a significant problem for the North American call center industry, and for the economy as a whole. Companies unable to continue doing business during a disruption could not pay either employees or shareholders any money, nor could they pay their suppliers. Therefore, continuity planning for the call center is a must for survival, not just for any one company, but for the health of the economy as a whole. Throughout the call center industry, a large portion of companies lack a call center recovery plan. Yet it has been estimated that $550 billion worth of goods and services traded hands in North America as a result of calling. This means that this industry represents a bulk of the gross national product. Therefore, a failure of the call center industry throughout North America would be catastrophic.

Justify Need Failing Businesses Face Lawsuit, Criminal Charges. In Chapter 6 we will talk about the various legal implications to justifying a need for a disaster recovery plan. The previous four chapters all work toward justifying a need. If you are still convinced that you cannot justify the need for a continuity-planning project, go through the first four chapters again. You should also have a look at the legal implications of not having a disaster plan that will be discussed in Chapter 6.

Management Buy-In You need to have management buy in because support from the highest level is essential. This is because of the significant commitment that needs to be made to get the plan operational. In the discussion of business impact analysis (Chapter 6), we recommend that you start with a business impact analysis and a risk analysis.

© 1998 by CRC Press LLC

Support from Highest Level Is Essential. Once you have completed both the business impact analysis and the risk analysis, you then try to “sell” to management to get the necessary commitment from it for the plan. Embarking on the continuity-planning process will be costly in terms of a serious dollar and time commitment.

Dollar and Time Commitment Senior management, having seen the necessity of doing continuity planning, must follow through with the resources to complete the project. Team Time. It is important that senior management plan for a commitment of team time to write and test the plan. The formation of the recovery team will require personnel from all company departments, and this means planning to give those people “continuity project” time. Additionally, consultants from outside the organization might be required to help with the process, and their fees will need to be allocated in the budget. Equipment Allocation. Part of the planning will be for equipment and funds allocation for hardware that would be required in a standby arrangement (as equipment is sold, the vendor rotates a small inventory of equipment that is standing by). Your company will need to get commitments from vendors to supply the company with equipment in case of a disruption, and in a reasonably short period of time. Supplier and Real Estate Contracts. Your company will need to set up supplier and real estate contracts in advance. This means assessing various equipment and property options and writing up “option to rent or purchase” contracts on a conditional basis. These contracts (to be written at all) would require a commitment from senior management to pay the retainers or deposits that might be required. Having such contracts in place would provide some relief from the requirement after a large-scale disaster to do “everything” at once. Such contracts would provide for the contingency, ensure priority service to your company, and keep costs down. One Disaster Survived Justifies All Expense. James Whit from FEMA stated, “A disaster normally costs fifteen times more than the money spent preparing for that disaster.” So if you do not have a continuity plan, it could cost you fifteen times as much as committing to having a plan that works. (We have repeated this observation, because it hits home.) Note that not preparing for the disaster has the probable result that the disaster would leave your company crippled or out of business. Consider Exhibit 5-1, which will help you find out how much time it could take the teams, which you will select, to work through the plan, and how much cost is involved. Then compare it to a two-day communications outage in your business, and you will see that disasters could easily cost your company a lot of money if there was no continuity plan. The chances of your company having a two-day communications outage in the next two years are very high. Surviving one disaster would justify all of the expense that the continuity plan project entails, and the business impact analysis will show that in both tangible and intangible effects. Anyone concerned about the level of expense could consider the effect of the company not surviving a disaster. If after the preceding chapters you still think, “it cannot happen here,” consider this: “In 1994, there were 580 occurrences of natural disasters with associated economic impact of $65 billion. Just $15 billion of these losses were insured,” according to G-2 Research, Inc. Our own research shows that the years following 1994, up to and including 1998 have been progressively worse for disasters.

© 1998 by CRC Press LLC

EXHIBIT 5-1. PLAN FLOW Step One: From Exhibit 7.2 (part of the risk assessment in Chapter 7), we take information on the risk of internal fire. This category is divided into 1 through 6 levels of risk. Internal fire

Fire drill False alarm

Small fire; quickly extinguished

Small fire; Fire Department called; Required to extinguish

Small to moderate fire

Moderate fire

Large fire

Step Two: This flows into Exhibt 7.6, which is a risk result table. This table is divided into the following categories: Shortduration outage

No recovery plan activation required

Return to normal operations

Short evacuation required

Evacuation of call center

Call handover

Return calls

Return to normal operations

Moderate damage to call center or infrastructure

Evacuation of call center

Call handover

Activate hot site

Return calls

Return to normal operations

Severe damage to call center or infrastructure

Evacuation of call center

Call handover

Activate hot site

Activate cold site

Return calls to new site

Return to normal operations

Step Three: In Chapter 9, you and your planning team find out how a plan is created “in general,” to deal with each set of required actions, and how they are accomplished. Step Four: In Chapter 20, you and your planning team see how checklists are created for each position, risk, and set of required actions. Step Five: In Chapters 23 through 25, we ask you to test these checklists to find any problems with the checklists, planning, or discover risks that were overlooked. Step Six: When — not if (at least the first time around) — you and your planning team find a problem, you must begin the process over again to deal with the new risks, or problems that you had not considered.

PLANNING You should design the Plan to flow from one section to another, with the information that you derive from one area assisting in completing another. The chart in Exhibit 5-1 shows how this flow works. Note that we have not shown all parts of the planning process, although they contribute information to the plan.

© 1998 by CRC Press LLC

Recovery Team Selection As far as planning goes, the first thing you need to do is select the recovery team. We talk about overall team selection in Chapter 8. However, you need an initial team to do the work of putting together the business impact analysis and the risk analysis. Who Does What Tasks. We suggest that you start by selecting a team for each of the tasks shown in Exhibit 5-2. This is the time to assign the personnel to do each section of the information gathering for the business impact analysis and the risk analysis. Personality and Prior Experience. You should secure the services of people who know the entire scope of the operation, making sure that all departments in the company are covered. You need the help of people who are willing to look into every detail of the how the company actually operates (as opposed to how anyone might think it does) and who understand the operation as a whole. The personalities and prior experiences that are required are probably those of personnel who are at least at the middle-management level, and have shown themselves to be dependable and flexible. For reasons stated in Chapter 8 we believe that a senior representative from the accounting department should be involved from the beginning.

Business Impact Analysis Importance of Doing Complete Analysis. Once you have your team assembled to do the business impact analysis, it is important to do a complete analysis of the business. We have provided some forms in the business impact analysis chapter (Chapter 6) that you can work through. However, there will be other aspects of your call center that you will need to take a close look at which are not in these forms. Dangers of Doing a Short-Form Sketch. There are dangers to doing a short-form sketch of the business impact analysis. The most significant is that without an in-depth look at the company as a whole, management would have difficulty in justifying a continuity plan. Only after you have gone over the whole company with a fine-toothed comb can you truly have a proper view and understanding of the importance of a continuity plan. Management understanding of how much time and money in a department bring value to the company is limited unless a business impact analysis has been done. For example, the loss of a help desk function would not be immediately noticeable to most companies as far as cash impact on the bottom line. The result of this ignorance is that most companies consider their help desk to be an expense that generates little revenue. Help desks keep your customers happy, and happy customers remain customers and will buy more from your company. Even though there is no direct short-term monetary impact for the company, there is a long-term impact on the overall sales of the company. Consider, for example, if a software publisher provided a help desk that kept callers waiting for 30 or 40 minutes before they could obtain answers to pressing questions. That company would deliver a substandard level of service. Such delays could lead many customers to return the product and not purchase from that company again. Customers have the right (and usually the inclination) to switch to a company with the ability to deliver technical help quickly and efficiently when the customer calls. Before performing a complete business impact analysis, company management might not know or care what level of service the help desk subdepartment had the personnel, equipment, or training to provide.

© 1998 by CRC Press LLC

EXHIBIT 5-2. TASK ASSIGNMENT SHEET

Recovery Team Selection Team structure (see Exhibit 8-1) Business Impact Analysis Identify business functions Business impact summary Identify company applications Critical application summary Organizational impacts from disasters Develop recovery time frames Risk Analysis Risk analysis matrix Insurance broker consultation Plan Stages Mission statement Definition of disaster Time frames Escalation plan Critical functions Critical applications Command center Team Responsibilities Plan update requirements Predisaster responsibilities Disaster responsibilities Recall/Call lists Contact lists (including government authorities) Equipment inventory (and emergency replacement)

© 1998 by CRC Press LLC

How will tasks get done?

When will tasks get done?

Who will be responsible for what stages?

What is the time commitment cost?

EXHIBIT 5-2. TASK ASSIGNMENT SHEET (continued)

Recovery Team Selection Backup Procedures IS backups Hot site backup Call routing according to site outage and volume Emergency Procedures Life preservation Assessment (of whether or not to declare a disaster). Declaration (of disaster and type) Disaster checklists Emergency call handling Emergency communication procedure Escalation chart Alternate site recovery Personnel restoration Equipment restoration Crisis Management Identify crisis leader External command and control procedures Checklist procedures Return to Normal Operations Hot site Cold site Alternate locations for permanent/temporary relocation Restoration procedures a. System restoration b. LAN restoration

© 1998 by CRC Press LLC

How will tasks get done?

When will tasks get done?

Who will be responsible for what stages?

What is the time commitment cost?

EXHIBIT 5-2. TASK ASSIGNMENT SHEET (continued)

Recovery Team Selection

How will tasks get done?

When will tasks get done?

Who will be responsible for what stages?

What is the time commitment cost?

Plan Testing Test objectives/locations Announced and unannounced testing Tactical exercises Plan maintenance procedures Operational changes (see also team responsibilities) Personnel changes Hardware changes

Risk Analysis In the discussion of risk analysis (see Chapter 7) is a matrix where we look at the risks of various incidents that could endanger your company. Various organizations can help you with this section of the business continuity plan. Fire Department Input. The fire and police departments can give you a list of risks that they have identified in the local area. Most emergency response departments have public relations officers that can provide this information upon request. Since newspapers cover news of fire and police emergencies, newspaper archives are a good source of information about disasters, such as bombings, earthquakes, tornadoes, floods, riots, and so on, that have affected the locale of your company. Unlike the police and fire departments, the local newspaper is likely to charge a photocopying fee. Regional Emergency Preparedness Organization. Your regional emergency preparedness organization generally has information that can assist you with local identified risks in your region. The local offices of FEMA and the American Red Cross would likely be able to provide you with publications and anecdotes. Many publications are intended for distribution to the public and business. Anecdotes from Your Industry. Talk to other people in your industry who have dealt with a disaster of one kind or another. Find out how they dealt with the troubles that arose, and what problems they had. How did their continuity plan work (if they had one) and what would they have done to improve their plan. These people can be an excellent resource — take every opportunity to use them.

Plan Contents The Disaster Recovery “Common Body of Knowledge.” Call volume management (CVM) is what makes the difference between the call center continuity plan and other types

© 1998 by CRC Press LLC

of disaster recovery plans. In the plan contents section we will be concentrating on the call center aspects of the business. There are many other types of companies whose “recovery” or “resumption” you might be called on to manage, sometime in your career. Still, this book deals with only the call center aspects of your company business. The Disaster Recovery Institute publishes a Web site where they provide what is called the “common body of knowledge” and information on business recovery in general. Please see the source information in Appendix B. Additionally, an excellent publication Business Resumption Planning is available from Auerbach that deals with planning for areas other than call center continuity. Emergency Communications Plan. CVM is the basis of the emergency communica-

tions plan. The CVM technique provides a means of looking at all the different aspects of recovering a call center, including interim call handling, hot site recovery, cold site recovery, real-time minimum call overflow and high-volume call-answer methods. If the foregoing terms are not familiar to you, it is not surprising. These are the methods by which it is possible to take call center call volumes and manage them in an effective way. The aim is to ensure that all calls can be answered. You could think of calls as a flow of water, like a river. Just because there might be an ice jam would not mean that the flow of water would stop. Similarly, telephone calls and other forms of communication are like a river of messages. Your company disruptioninduced lack of capability to answer calls or receive faxes and e-mail would not mean that the flow of messages would immediately stop — at least, not until sufficient time without an answer had caused the world at large to think your company out of business. For your company, calls are money and you cannot afford to allow the flow to be jammed. So you have to find ways of getting around the “ice jam” and getting your incoming calls back on line. “Future Procedures” (Contracts and Safeguards Not Yet Arranged). Most companies do not have service-level agreements with their vendors to get new equipment in the event of an emergency. Instead, many companies have agreements with vendors to repair or service their existing equipment. In the event of a disaster few companies have agreements with their vendors to get new equipment within a specific period of time. A contract that does not specify an exceptional level of service will not be likely to be fulfilled with an exceptionally high service level — unless, of course, your company is so fortunate as to deal with a company that strives for excellence in every department. While preparing to write the continuity plan, this is an area where you and your team will start to record notes about “things to be done.” Finding vendors who will guarantee to supply and deliver replacement equipment within a certain period of time is one of those necessary tasks. Preparing Redundant Equipment (Back-Ups). The future procedures section should include notes concerning additional and backup equipment as well. Spare equipment is required to provide operational redundancy for the critical functions of the call center and critical departments. Keeping spare equipment on hand, like arranging for replacement equipment, is the kind of provision that companies usually do not do unless they purposefully set out to plan for their continuity.

Team Responsibilities We will talk in length on this in other chapters, especially Chapter 8. However, basically you will need to set up a recovery team comprising a number of subteams.

© 1998 by CRC Press LLC

Departments. Besides the team described above that would help you, the Planner, write the continuity plan, the following teams have to be assembled: • A disaster site recovery team, to manage repairs at the facility undergoing a disaster. If your company has more than one site, each facility requires its own predetermined team. • An interim call-handling team for the assisting call centers. (You need one for each internal or external interim call center that you have or arrange.) • A hot site recovery team for each facility to which your operations might relocate • A team for each of the regular departments of your company to safeguard critical business functions. • External liaisons for each team so that interteam and interdepartmental tasks can be provided for specific areas of the business. Management and Employees. Later in the planning process, you and your team will discover the need to plan for personnel issues like emergency shift rosters, care of dependents (children, seniors, pets) during wide-area disaster, emergency banking, and inclusion of family members in plan enactment training exercises. These are not human resources tasks in the usual sense of the term. This is the point in the preplanning stage where you might want to assign one or more persons to function as “employee liaison” and “union liaison” representatives, as applicable to the unique labor situation of your company. External Liaison. The external liaison will handle things like contact with fire, police, and ambulance departments, the EPA, and, possibly, in the case of some wide-area disasters, the military.

Backup Procedures Data. Even though this is a book primarily aimed at call center continuity, the data center forms an important part of the recovery of the call center during and after a disaster. Here are some questions to ask when starting to plan backup procedures. • How large is the IS department in terms of equipment and software managed? • Are servers isolated in a dedicated computer room or out in the open call center or offices? • If one or more servers malfunctioned, could desktop computers function alone in a limited capacity? • Are original and backup copies of application software stored together? (Why?) • How long would it take to back up current data in each department? • Where are daily departmental and call center backup files stored? • Does the IS department have a fireproof storage safe or vault for magnetic media? (see Schwab Company, Disaster Recovery Yellow Pages). • How long has it been since anyone printed out the database on paper for archiving? Inventories of Files, Forms, Documents, Contracts. In this section you will be looking at backup procedures for information recorded on paper. This includes files, forms, inventory registers, and other documents such as contracts and blueprints. You and your team will have to plan the coordination of making these files available to the original site call center or the interim call-handling facility.

© 1998 by CRC Press LLC

Coordination with Assisting Call Center. As examined above, you will have to decide whether your assisting call center(s) will be internal, at one or more facilities belonging to your company, or in virtual (distributed) form, or external at the facilities of one or more outsourcer(s). This decision will have an impact on how you and your team will plan to distribute required information (both print and computer readable) in the event of a disruption. Off-Site Storage. We will be looking at some of the aspects of off-site storage of your paper and computer data files: • Where is the best place to store items off site? • What must you prearrange? • How often should you archive paper files?

Part of the process of planning for off-site storage is deciding: • What documents to prepare. • How to prepare them. • How the documents will get to the storage site. • How updates to these documents will be handled.

The flip side of these decisions is the question of what documents not to include. Pre-prepared Duplicate Files in Distant Storage. In case of wide-area disaster, you need to vault critical documents and data files outside your local area. • How quickly can you get access to the information that is stored off site? • How far away should the storage site be from your offices and call center(s)? • What mechanism should be used to pick up or receive delivery of these files in case of need? • What security arrangement will you use to ensure only authorized personnel get access to the stored information? (Example, company ID and password.)

Disaster Implementation Tasks Checklists. Checklists are a big part of the continuity plan, especially during disaster recovery. With a checklist, almost anyone can figure out how to handle and shut down or turn on a piece of equipment properly. A checklist should be kept handy for every employee because you never know who will be around to do what in the event of a disaster. Handover to Hot Site. The checklist procedures will guide a person through the processes needed for switching to an interim call-handling facility, your backup facility, and then back to your original location when the disaster is over. Information Lines and Situation Reports (IVR). This book will also provide checklists for handing over the company data lines and voice lines to the assisting call center(s), and for generating situation reports. These situation reports, presented on a passive IVR (interactive voice response) line, will keep company personnel up-to-date with what is happening during the disaster. (See the section on “Emergency Communications,” in Chapter 12.)

© 1998 by CRC Press LLC

If your company provides a passive situation report line, this important measure can spread information to employees who call in. Up-to-date situation reports can direct personnel to shelters if required, announce changes in shift schedules during a disruption, ensure that everyone knows how payroll and banking will be handled during the disaster (of whatever magnitude), and keep morale and group cohesion up for the company as a whole.

Return to Normal Operations This is a part of the plan that tends to get overlooked. You have brought all of your personnel, equipment, and support services over to another location and now you need to know how to get them back again. Personnel. Getting your people back from their shelter, evacuation site, or distant hot

site work station involves more than putting a “return to the workplace at your normal shift time” message on the IVR. What is involved in the way of transportation (from an alternate site) and accounting? When it comes time to resume normal call center operations, will the employees be ready to return to work? How should you deal with finding replacement workers, and should they be on temporary or permanent contract? See Chapter 24,”Personnel Issues” in Part 5,:Crisis Management.” Utilities. Before returning to normal operations, your recovery team must establish that all the support systems for the facility are in working order. This means that power, water, heat/cooling, plumbing, lighting, etc. are repaired, replaced, and serviceable. It means ensuring that the roof is still on, the walls are not damaged, the water coming out of the taps is not fouled by pollutants, the carpets and decorations are not damaged by smoke or chemicals and other items not a normal part of property management. It also means ensuring that humble items like toilet paper, coffee and soft drinks, pens, refrigeration, and chairs are available for your employees to use when they come on shift. LAN — Bringing Up Your Computers. After the disaster is over and the power is on, you will have to decide when you should bring up the computers and what is involved in the process. Will you be using your previous set of computer equipment or replacements? This will have an impact on how long it would take technicians to power up and test that all application programs are up and running. If the company had to rebuild or renovate the facility, loading the programs and data backups (for the first time in replacement equipment) would take significantly longer than if the company had evacuated the area for a day or two and suffered no physical damage. CVM — Return of Calls. What is involved in bringing the flow of calls back from the assisting call center or hot site(s), or from the virtual call center, and back into your original facility? It is a little more complicated than just flipping a switch. See Chapter 13, “Return to Normal Business Operations.” Offices and Normal Business Activities. When is the disaster truly over? After personnel, data, and communications are restored to their normal configuration, your company will have to inform its vendors, and interested regulatory agencies, major clients, and perhaps the press that normal business has resumed. This is the time to task the public relations department with preparing a billing insert or mailing that thanks all parties for their patience during the disruption.

© 1998 by CRC Press LLC

Senior management, directors, and major shareholders will probably want an assessment of how the company came through the disaster. They will want to know: • What shape is the company in now? • How did the company fare in comparison to other companies undergoing similar troubles?

Postplan For planning and training reasons, you and your team will want to issue the postdisaster recovery assessment, plan improvement, and amendment of arrangements with the company suppliers as a document. This is all part of plan maintenance. Assessment. In the postplan section we will discuss assessing the workability of the plan. • How do we assess whether or not the plan will be viable, improvements to the plan, and changes that need to be made? • Did the continuity plan work? To what degree of success? • Were flaws discovered in operations of one or more departments? Improvements to Plan. What was learned from the disaster that could be used in future operations? In future disasters? Changes in Vendor Arrangements. During the postplan stage, we will discuss changes that you might decide to make in vendor arrangements. First, you might choose to revamp your arrangement with your current vendors, based upon discoveries made or plan weaknesses revealed. Alternatively, your company might decide to switch vendors. Some vendor companies might determine that they are unwilling to put the extra effort into servicing your account that participation in the continuity plan would require. You and your team might find that the vendor has no concept of continuity planning and would likely go out of business as a company if it underwent a disaster of its own.

Plan Testing Exercises That Test and Train. After writing the plan, and even after having successfully survived a disaster, you and the various teams will plan and run exercises to test and train personnel and practice the plan as a whole. Part 4, “Testing,” describes five types of exercises, going from the most basic test, the tabletop exercise, to the most complete test, which could be called the full-body workout. It is impossible to train for every single possible situation. Even so, training for specific situations and having experience in practicing emergency procedures for them allows company personnel to adapt to new situations. What personnel do train in are specific procedures for emergency situations that the people follow and learn to make into “instinct.” By categorizing disasters into types, you and your team can plan for different types of disasters or levels of disasters. As for the actual testing, we discuss this subject in greater detail in Part 4. Don’t Let Test Schedule Slip. Do not let test schedules slip. One thing that many companies are guilty of is a failure to test the continuity plan once they have one. They are afraid to test it, because they are in denial. Even after going through all the planning procedure, it

© 1998 by CRC Press LLC

seems that testing the plan would be like admitting that a disaster was possible or even likely. Testing makes the concept of risk seem more real. Well, if the management of your company does not think you can service a test and you do not test your plan, then your company does not actually have a plan after all. Having plan binders gathering dust on shelves in various offices will not save anyone’s life, and will not save the company cash flow. Training Includes the Assisting Call Center. Training and testing must include the companies that will be assisting you. This includes the interim call-handling facility, the real-time minimum call overflow (RTMO) facility, and the hot site facility.

Plan Maintenance Stay on Top of Changes. Companies often fail to stay on top of changes. They have good intentions, but “real life” tends to have a distracting effect upon even the most diligent. The day that your scheduler indicates that you wanted to remind all departments that change charts are due in your office might be the very day that you yourself are off sick with the flu. Or perhaps everyone is celebrating the success of the company in a sports championship. Perhaps everyone decides that it can wait until after lunch, and that is the day that a tornado hits, so they never get to it. Enacting the plan that everyone forgot to update could be a challenge. Company Operations. We will be giving tips in later chapters on how to make sure that the plan changes get into the company operations as well as changes in operations getting into the plan. Over time, whenever an aspect of the plan changes, then company personnel will become accustomed to rolling with it. We hope that all management and employees will make the changes to their checklists, thinking, and behavior “second nature.” Gradually, it will probably become apparent that changes in company operations suggested by changes made to the continuity plan will tend to make the company leaner, more efficient, more profitable, happier, and a safer place to work. Conversely, methods and procedures that are counterindicated by testing of the continuity plan will be shown to be counterproductive, unsafe, unprofitable, and dangerous to company survival. Contact Information List. The contact information database needs to be updated at the same time that you update your employee list. Someone on the team should also be assigned to make scheduled telephone calls, faxes, or e-mail to determine whether contact information for other parties named in the plan, such as emergency department liaisons, real estate agents, insurance brokers, and so on, remains the same or has changed. Vendor Contracts and Contacts. Every time a vendor company moves, your account

representative changes, or new contract terms are negotiated, the vendor contacts and contracts must be updated in the plan and all checklists. This applies to off-site storage of duplicate contracts and drawings as well. Changes in Departmental Procedure and Form Inventory. When departmental procedures change, the plan and all relevant checklists must change too. When the plan and checklists change, form inventories also must change to reflect the changes. Would it not be frustrating to have a plan and checklists that are up-to-date, but then personnel go to use a chart or form inventory and it contains old information? Updates and testing are like the two wings of a jet plane — one supports the other. As a philosophical choice, we would rather discover a glitch during a test than during a real disaster!

© 1998 by CRC Press LLC

6 Business Impact Analysis SENIOR MANAGEMENT BUY-IN A business impact analysis (BIA) is a procedure used to judge the effect of an interruption of services on the call center and, as a result, on the organization as a whole. The BIA can present information on the effects of a disaster on all areas of the businesses operation. The company has to make a business decision about the level of acceptable losses and, conversely, the level of unacceptable losses. The means used to prevent these losses are called control measures. It is important to have a senior executive from the finance department involved in this project, right from the start. If you have not got such a person committed to this project, then stop reading and go convince the necessary person to join you. To make the business decision (of what constitutes acceptable and unacceptable loss), management has to assess its starting point. Most managers have a very clear idea what would happen if they lost their manufacturing units. What is the impact of the call center on sales, customer support, and customer relations? Senior management commitment to the project is an essential prerequisite to analysis and planning activities. Any and all plans of voice recovery, business continuity, and disaster recovery require that senior management buy-in be the first step. In fact, this is so important that we dedicated the first four chapters in one form or another to the idea of senior management buyin. We show the “Illusions of Recoverability” in Chapter 3 in detail. Why? We did this to show senior management that the simple plan that they thought would work, will not work. If your company relies upon a flawed plan for its survival, it is dangerously at risk. Chapter 4, “Rationalizing Procrastination,” is a list of objections that senior managers often give to excuse lack of action, and the methods of overcoming those objections. The tables found in that chapter are all methods of overcoming senior management reluctance so that the planning process can begin. For example, when we have used those tables at conferences, many people have been shocked by what they discovered in them. Someone in the travel business could find a piece of information as simple as how high losses in airline reservations can run during a communication outage to be unsettling. Ironically, when we have said that such companies could lose $90 thousand an hour, our audience has corrected us. We have been told that many companies could lose anywhere between $90 to $150 thousand for every hour of an outage. The audience in such conferences was unsettled because these are inconvenient facts that people have avoided confronting. As another example, in similar presentations we have said that credit card companies could lose $2.6 million per hour of a communications outage. The credit card folks that were in the audiences thought that this would be a low figure. These are the same audiences that ruefully confess that only about 10% of them have any kind of disaster plan in place at all. Once senior management starts looking at what other companies have lost or could lose in a disaster, the idea of disaster planning does not seem so “off the wall” to them. Following the philosophy that “survivors survive,” we will make an assumption here. Chances are that the reader is determined to be a “survivor.” We believe that representatives of a lot of companies that already have a disaster plan will be reading this book. That is because this book is primarily focused toward call center recovery.

© 1998 by CRC Press LLC

We hope that if your senior management has already bought into a disaster plan of some description, it will understand some of the importance of disaster planning. This is the reason we included the principles of recovery in the introduction and the first four chapters. They are methods of helping convince management to buy into a voice recovery plan for your call center. We believe that the first step in getting management approval is the BIA. In the book Business Resumption Planning by Edward S. Devlin, Cole H. Emerson, and Leo A. Wrobel, Jr., the authors recommend that you do a preliminary risk analysis: “The preliminary risk analysis identifies risks, their probability of occurrence, and their potential impact on the organization. The risk analysis differs from a business impact analysis in two important respects: • A risk analysis only identifies overall impacts of a disaster to the organization. It does not attempt to identify financial and logistical impacts on specific organizational departments and functions; that is the purpose of the business impact analysis. • The business impact analysis does not consider what types of incidents cause a disruption to business operations; it is concerned only with identifying consequences in terms of financial loss, additional expense, embarrassment, and the expected length of the interruption. (page I-11, Step 1)

We personally do not believe that you should start with a preliminary risk analysis. We believe that you and your helpers should do the BIA and the risk analysis immediately in sequence. In this way, the two studies will buttress one another. The first shows the impact of loss (or, conversely, what your company needs to preserve), and the second shows the likelihood of risk. Together they will prepare you to begin planning to preserve the critical functions of the company from the identified risks. Proceeding immediately to the risk analysis without first defining the critical functions (in terms of revenue and possible loss) might have a quelling effect upon the continuityplanning process before it is even begun. This has happened in almost every case where we have known someone to go through all the steps of preliminary risk analysis first. We have found that they have not completed their continuity plan. Likewise, their management did not take the necessary next step, which was to do the BIA that would come after the preliminary risk analysis. Also, this tends to confuse people. It would seem that after doing both, you would have done your work twice. The difference is that doing the BIA first seems to result in getting a continuity plan finished and workable. The BIA could also function as a business function review. In other words, the BIA could show what each department of the company does. The BIA gives you an opportunity to put the spotlight on the different parts of the business to see which areas are more critical than others. This is a benefit that gives the time and effort of doing the BIA some immediate usefulness to senior management. Seeing where the company revenues come from would enable company executives to expand the company profit centers. The BIA could also highlight some of the departments or projects that might not be earning back their costs. Senior management could therefore be sold on the BIA in terms of cost justification if that should prove more useful to it than a business function review. Should senior management still be reluctant to do the BIA and begin the process of business continuity planning, then it should be reminded of the legal implications of not doing such planning. To our knowledge, there are no laws that specifically state that a company must have a disaster recovery plan. However, there are many laws with clauses relating to voice recovery

© 1998 by CRC Press LLC

plans that are in force. When we say a voice recovery plan it is because there are some recovery plans required by legislation for various industries. Some of the key regulatory issues are found in the Federal Financial Institutions Examinations Council (FFIEC) guidelines that replaced the previously issued banking circulars in the United States. In addition, Executive Order 11490, the Foreign Corrupt Practices Act of 1977, various liability statutes, life and safety statutes, and various risk-reduction statues all deal with the issue of voice recovery. These are all United States regulations. The Foreign Corrupt Practices Act requires that a corporation “make and keep books, records, and accounts which in reasonable detail accurately and fairly reflect the transactions and dispositions of the assets.” Please note that assets is the key word here. What are the assets of a corporation? Keeping the assets of the corporation has been defined in court as being the preservation of accurate computer records. This means providing for fire-resistant, off-site storage for computer-readable storage media and printouts. (Security Exchange Commission (SEC) vs. World Wide and Coin Investments.) The Foreign Corrupt Practices Act provides for personal fines of up to $10 thousand, corporate fines of up to $1 million, and prison terms of up to five years. The section of the Act that keeps it in the minds of continuity planners is the “standard of care” warning. Company management can be judged for their mismanagement of corporate assets. Neglecting this responsibility by allowing the destruction of the assets of a United States company (especially its financial records) to occur could indeed earn company executives up to five years of jail time. Do telephone calls count as corporate assets? At this point in time we are not aware of any members of any corporation being held accountable for loss of company telephone calls. However, we see no clear indication of how the telephone calls can be defined as anything other than a corporate asset. The calls are what makes sales in a number of companies possible. Accordingly, the requirements to provide safeguards for corporate assets could be held to extend to telephone calls. Sooner or later (but probably sooner), we believe that there will be test cases on this point. There might even be court cases concerning calls as assets proceeding at this time. We would hate to work for the company that some overzealous exchange commissioner or attorney might decide to pursue on this matter. The Federal Financial Examining Council and the comptroller of currency has issued various circulars. But in 1989 a joint circular was issued by a number of agencies including the Board of Governors of the Federal Reserve System, the Federal Deposit Insurance Corporation, the National Credit Union Administration, and the Office of the Comptroller of Currency. The circular states: Loss or extended interruption of business operations including central computing processing, end user computing, local area networking, and nation wide telecommunications poses substantial risk of financial loss and could lead to the failure of an institution. As a result emergency planning now requires an institution wide emphasis. In Canada there is no “law” requiring companies to maintain a disaster recovery plan per se. The Office of the Superintendent of Financial Institutions does evaluate the “operating risk” of banks and trust companies. Pressure for business continuity in Canada tends to come from the peers of banks in the financial services industry, nationally and abroad. Such other institutions might demand a written guarantee concerning the business resumption of a bank plan before accepting a tender for new business. A less-explicit requirement comes from the commercial and individual account holders of the (Canadian) bank. They expect to have access to their funds no matter what circumstance the bank might be enduring. Banks that fail to show due care and diligence will tend to lose market share to other banks that do a better job of operating under duress.

© 1998 by CRC Press LLC

Because of this policy, banks are among the leaders in finding interesting and unique ways of recovering their telephone calls. Barnett Bank in Florida (recently acquired by NationsBank) has some of the most comprehensive recovery plans we have ever seen. In Chapter 2, we quoted Tim Ging from PNC Bank, on the principles of voice recovery. So banks have been planning for voice recovery. In general, they have become quite good at it. One of the things that the reader can do when thinking about voice recovery is to talk to the local bank and see what it does as far as its own recovery planning goes. In addition to considering legislation, one of the largest hazards you should be concerned about is liability. Depending on the industry sector of your company, the liability for not being able to take telephone calls could be huge. In voice recovery planning, as in other liability discussions, the mythical “Reasonable Man” is the basis for judging if a recovery can be considered successful. Suppose that your company prepares a recovery plan that calls for recovering lost communications capability in a specific time period. If your plan calls for a five-hour recovery and that is appropriate for your industry, well and good. In contrast, if your company were to lose millions of dollars per hour as a result of your data or call processing being interrupted, then the management could be judged to have been negligent. If the five-hour recovery time were not reasonable, on the basis of revenues or expenses lost, then it could be judged inappropriate for the company to have planned for such a long recovery time frame. The time frame would not be as critical if your company and your customers did not stand to lose large amounts of money in a very short period of time. It all depends on what a reasonable man would do. If there were less vulnerability to loss, then the recovery time period could be extended. We hope that this discussion has clarified matters for you. Once your senior management realizes that it risks going to jail or being sued for negligence for nonsupport of the continuity-planning project, the holdouts will probably commit the necessary resources to the task. Remember our previous observation, that only about 10% of companies whose representatives we encounter have planned for their continuity, and only about half of those have tested their plans. We believe that in future years, the courts will become less and less sympathetic to such lack of forethought.

IDENTIFY BUSINESS FUNCTIONS You are now commencing work to prepare a plan that can protect the company communications capability in case of a future disruption. To proceed, you must know the present state of the company operations, where its revenues come from, and where it is most vulnerable. It is important to know what functions are crucial to the continuation of the company in business, and what can go “on hold” until normal operations resume. If you will follow the subsequent discussion step by step, you will probably find that the BIA unfolds for you in a methodical and logical manner.

Operations Starting Out. The planner will have to gather information on the call center for the purpose of • Deciding the priorities for restoring the different call types flowing through the call center; • Deciding the recovery time objective for the various call types; • Identifying what critical resources are required to recover the call center.

© 1998 by CRC Press LLC

You start the analysis by announcing that a survey will be made of the company business functions. Humanity being what it is, there will probably be a political aspect to discussions about the importance and needs of departments. Department heads will all argue that their department is “critical.” Personnel at various levels will all maintain that their access to communication and various tools is essential to getting their job done. They will suspect (probably correctly) that, if their part of the company is not “critical,” then it could be the target of downsizing. While you are delivering surveys to the department heads, we urge you to make clear to them that the decisions resulting from the BIA will be based on facts and logic. The goal is not to protect anybody’s favorite department. The goal is to gather information leading to a plan to ensure the survival of the company. We hope that you will no longer meet any resistance by senior management. Perhaps you work with a company that has recently reorganized (or is a new, knowledge-based company) according to the “individualized” hierarchy that allows executives to use their initiative. If your company rewards real innovation, rather than “face time,” congratulations. Your job in analyzing and planning for the continuity of company essential functions will be much easier. Survey of Business Basics. The first thing to do is to identify which of the following call types are handled by your call center. The call center manager and supervisors are the best people to fill out a survey of call types. However, look at the revenue-producing phone calls from other departments as well. For example, suppose that personnel of company sales department completed the form found in Exhibit 6-1. By analyzing the different call types in Exhibit 6-1, you begin to get an idea of the complexities involved in trying to “recover” them — trying to provide for emergency call handling of the various call types. The areas below the double rule in Exhibit 6-1 require special call volume management (CVM) techniques to recover those calls. This is because special knowledge, “expert” licensing, or special software support is required to handle those call types. Recovering those call types during a disruption would require special planning and preparation. (We discuss CVM in Chapter 15.) Next, take the chart in Exhibit 6-2 and start to assess the value of the types of calls to the call center. Note that although there are only two call types shown in the example, it is important to do assess all of the call types. If you have a problem assigning a value to any of the call types, talk to your CFO about the financial justification used to establish this function in the call center. It can be particularly difficult to establish a value for help desk functions. If all else fails, take the cost of providing equipment and personnel for handling each type of call and add 15% as a reasonable expectation of value added. Now you know what call types are handled and what impact loss of those call types would inflict upon your company. The following are the critical functions of any business. First comes sales; second accounts receivable; including credit card payments; third fulfillment (delivering service or goods to the customer who just bought from you), which may require contact with couriers or shipping firms; and fourth accounts payable and payroll. Fifth are customer service troubleshooting and public relations; sixth business communications; and, last, purchasing of the items that enable the previous functions to be performed. Since this book deals mainly with call center continuity planning, we will not belabor by lengthy discussion the reasons we assign sales the first place. However, just as agriculture is the basic industry that enables all others, we believe that sales is the basic department. The first survey will establish what types of call flow in and out of your company. Each department needs to identify its own critical tasks. If a hurricane were expected to stall over

© 1998 by CRC Press LLC

EXHIBIT 6-1. CALL TYPE CHART (ANNOTATED) Type of Call

Type of Call

Order taking Yes

Telemarketing outbound Yes

Reservations Yes

Answering inquiries for general information Yes

Product location inquiries Yes

Sales Yes

Channeling customer complaints to others for resolution Sometimes (to sales manager)

Collections

Solving customers problems

Outage reporting

Answering inquiries about product/service use Yes

Scheduling repair/warranty service

Contest lines

Advertising Sometimes

Advising senior management on customer issues Sometimes (to sales manager)

Market research Sometimes

Public relations

Quality management

Crisis handling Rarely (transfer to sales manager)

Billing

Defining customer service/Policy/strategy

Specialized certification/Licensing

your region for a week, if your heating or cooling system failed for a week, if the roads leading to your premises were closed for a week, what calls and functions would absolutely have to be handled? For sales, accounts receivable, purchasing, and accounts payable (including payroll), what would be the expected total dollar income or outgo associated with each type of call? What would be the expected dollar income or outgo per transaction or operation? What percentage of calls could be expected to yield a sale? Other than direct contact with the customer, what else in customer service could be regarded as essential? What workload (in calls and faxes) could public relations expect during a crisis, and how much more than its normal “publicity” workload would those personnel handle? What tasks would be absolutely required to fulfill customer orders? What percentage of business communications would be regarded as essential to the bottom line? What volume of communication does the call center handle for each of the above departments, and what volume could each department do itself if the call center suffered disruption? Can you the planner make do without information from any one department? The short answer is “no.” The disruption of business might be a disaster or only a limited outage of an important utility. Departmental personnel and heads of departments who cooperate fully with the BIA and the continuity-planning process could expect to carry on business during and after a crisis. The essential needs of their department would be met. No one person could run a company, and no one person could ensure the continuity of the communications or business of that company. It has to be a team effort.

© 1998 by CRC Press LLC

EXHIBIT 6-2. CALL TYPE CRITICAL IMPACT (ANNOTATED) Call type (customer requests for information)

Sales

Help Desk

Location of calls

St. Catharines, Ontario

Columbus, Ohio

Priority of call type

1

2

Income per call or cost of operations per call

4 calls = 1 sale Each sale worth $520.00 Avg. income $130.00

No direct income; avoids loss of product reputation; call cost $35.00 processing

Average duration of call

3 minutes

7 minutes

Number of stations capable or dedicated to receiving call type

150 (capable — peak) 80 (dedicated — average in use)

10 (dedicated)

Average number of calls covered per hour per station/total

15/1200

7/70

Peak number of calls per hour, total

2250

80

Estimated loss of business if down 30 minutes during peak

$146,250.00

$1,400.00

Estimated loss of business if down 1 hour during peak

$292,500.00

$2,800.00

Estimated loss of business if down for 4 hours at peak based on average call volume

$624,000.00

$9,800.00

Estimated loss of business if down for 1 day based on average call volume

(In this example the operating day is 10 hours long) $1,560,000.00

$24,500.00

Estimated loss of business if down for 2 days based on average call volume

$3,120,000.00

$49,000.00

Data In the “information age,” computer hardware, software, and databases have become essential to business operations. The manager of information systems (MIS) will therefore be a key member of the future recovery team. Let us examine the data requirements of the call center. Use of some types of application will be shared with other departments (see Exhibit 6-4). To do the calculations at this point, you should use a worst-case scenario. Assume that the equipment went down with only the current backup in place, and was destroyed. The MIS would probably have the information available that would show what the pieces of hardware, software programs, databases, and communications and analysis software would cost to replace. (Note that we are not calculating the cost in lost sales.) In addition, the MIS could determine the hardware and software necessary to the performance of the critical functions and the call handling of the departments polled in the previous survey.

© 1998 by CRC Press LLC

EXHIBIT 6-3. CALL FLOW BY CALL TYPE (ANNOTATED) Call Type: Sales Call

Primary

Subcategory

Sub-Subcategory

Originating facility (home, office, both, call center)

Example: callers home 1-800-555-8888

Routing Path 1

AT&T Long Distance

Routing Path 2

Bell Mega Link

Routing Path 3

Local service

Receiving Path 1

IVR

Receiving Path 2

CSR

Receiving Path 3

Supervisor

Processing Path 1

CTI

Processing Path 2

Screen pop

Processing Path 3

Server database

Access customer account info

Postprocessing Path 1a

Accounting billing #1

Credit card

Link to card center for approval

Postprocessing Path 1b

Accounting billing #2

Accounts receivable

Check bad debt file

Postprocessing Path 2

Fax server

Postprocessing Path 3

Shipping

Postprocessing Path 4

Customer follow-up

The MIS survey need not be exhaustively detailed at this stage. However, that manager should show in a general way what resources of personnel, diagnostic software and instruments, and what replacement components or repair equipment would be required to keep the rest of the company up and running.

Communications Each department should complete a general survey of basic communications requirements. How many telephones, facsimile machines, and computer stations for e-mail are in use? Does a department utilize other communications methods, such as cellular phone, satellite phone, two-way radio, ship-to-shore radio, digital or alphanumeric pagers? Is videoconferencing used, and if so how many cameras (and associated equipment) are there? Is the videoconferencing accomplished by satellite uplink or downlink or sent and received over the Internet? Exhibit 6-5 lists exactly what the interdependencies are with the organization. This survey will constitute a “communications inventory.” That, in turn, will form the basis for the call pattern surveys that follow later. The survey should show costs associated with the various forms of communication. If for some reason your company had no access to courier

© 1998 by CRC Press LLC

EXHIBIT 6-4. CALL CENTER DATA REQUIREMENTS (ANNOTATED)

Call Types*

Critical Applications

Data Processing Location

Time to Restore, Include Backlog**

Restoration Cost†

All

CTI hardware and software

Call center

1 day

$300,000

Sales, customer service, problems, telemarketing, location inquiry, etc

Screen pop software

Call center, sales, accounting, help desk

2 days

$70,000

All

Server database

All

2 days

$56,000

Sales order, accounts receivable, and payable including credit card payments, fulfillment, purchasing

Accounting billing

Call center, sales, accounting, help desk

2 days

$250,000

Order confirmations, business communications

Fax server

Help desk, sales, purchasing

2 days

$10,000

Shipping — fulfillment, order tracking, dispatch calls to courier/transport company

Shipping tracking and labels module

Call center, help desk, shipping

1 day

Unknown

* Group all types that have similar data requirements together. ** Backlog includes lost data. † Restoration includes equipment, personnel, and backlog costs.

delivery of documents, what critical information could not be distributed? What would an alternative method of delivery cost, for example, if another courier service billing at a higher rate were used, or if junior personnel had to carry the documents by hand, traveling by car or air flight, incurring food and accommodation costs? What would be the cost in expense and lost revenue if the call center were knocked out and slower, secondary forms of communication were used?

Utilities What basic services are required to support the critical business functions, data capabilities, and communications of the company, and the people who run them? If your company owns its premises or campus, the property manager will do this survey. If your company is a tenant, the information will have to be obtained either from the landlord’s property manager or through some digging by your accounting department. The survey should include all electrical power connections and panels, water mains, heating and cooling systems, numbers of washrooms, and numbers of toilets and sinks in those washrooms. It would not hurt if an additional list were prepared, showing numbers of coffee machines, kettles, refrigerators, and vending machines on the premises. These items are not “essential” in normal circumstances. (Except for one basic requirement, which is toilet paper!) However, in a blizzard that snows in your call center, or after an earthquake or other disruption, they might prove very useful.

© 1998 by CRC Press LLC

EXHIBIT 6-5. COMMUNICATIONS INTERDEPENDENCIES

Department:

Number of Units

Estimated Daily Usage

Estimated Loss If Unable to Use

Additional Information

Telephones Facsimile machines Computer stations Mail Couriers Internal e-mail — internal reporting External e-mail Other

EXHIBIT 6-6. UTILITIES Question

Answer

Downtime

Cost

How many power grids service the building? Can you switch from one power grid to another effectively? How long would the UPS last without power? Is there a generator in the building or reserved off site for your company? Has the generator been tested and maintained on a regular basis? How long can the company carry on operations without power?

It is important to know what utilities affect what areas of business. For example, computers are obviously affected by power loss. In addition, some computer mainframes are water-cooled, and, therefore, the water supply becomes expensive to lose even for a brief period of time. The primary repair of water pipes and the secondary repair (or replacement) of computers together could make a significant impact upon the company’s quarterly financial report. The impact of the lack of any of these utilities could be devastating. However, the loss of power is usually the most critical and the most likely to occur. Please consider the questions listed in Exhibit 6-6. You should consider them on the basis of a serious outage. In this case, that is assumed to be two days or more.

© 1998 by CRC Press LLC

EXHIBIT 6-7. LOST SALES Question

Answer

Cost

Answer

Cost

In the event of a disaster would your customers purchase from one of your competitors? If those customers purchase from other companies, would those sales be lost permanently? Do you provide an exclusive service? Would you lose customers (accounts) permanently during a disaster? Would your loss of customers be only temporary?

EXHIBIT 6-8. NEGATIVE CONSEQUENCES Question Additional operating costs Cost to replace equipment, buildings Public relations/advertising to repair reputation Cleaning, repair, decontamination (mitigation) Inadequate service to customer What are the effects of loss of business?

RISK IMPLICATIONS Lost Sales Loss of Business. Loss of sales (see Exhibit 6-7) could occur during a disruption if customers normally make impulse purchases of the company products or services by telephone. If these buying decisions are made quickly, or where time is a factor (such as in the travel and tour business), your company could lose significant revenue even from a short communications outage (see Exhibit 6-8). If your competitors should not experience a disruption that put your company out of contact, they would probably pick up the business that your firm could not transact. Perhaps in your industry the customer would usually have to think about the buying decision for some time before actually making the call to your call center. Maybe a history of repeat business has forged a “relationship” between the customer and one or more of your company representatives. In such a case, loss of a sale due to disruption might only be temporary. This is especially true in the financial sector where the client’s long-term private account or investment information is held confidentially. Relationship selling in the cosmetics or “beauty” industry would also likely suffer only temporary loss, since the customer would tend to prefer the service of a representative already familiar with the customer’s preferences.

© 1998 by CRC Press LLC

EXHIBIT 6-9. LEGISLATION AND PENALTIES Question

Answer

Cost

Answer

Cost

Answer

Cost

Are you governed by the Foreign Corrupt Practices Act? What penalties could your company and management suffer under law if they were ill-prepared for a disaster? What legislation is applicable in your state, or industry? For example, is your business governed by the Environmental Protection Act?

EXHIBIT 6-10. LICENSING Question Can the company lose its license(s)? Is your company in insurance, stock brokerage, engineering, utilities or in the field of medicine? What kind of insurance will you need? Can the company be sued for damages for not answering its telephones? What are the reporting requirements under legislation? What are the consequences of not reporting?

EXHIBIT 6-11. SAFETY LEGISLATION Question What legislation governs the safety of workers in your state or industry? If workers are killed or injured in a disaster, what are the penalties? What are the standards for preparedness?

However, there is always the risk that the customer might sample a competitor’s service and “go away” forever. Regardless, a lost sale is lost revenue. Whether permanent or temporary, losses are detrimental to the well-being of your company.

Fines/Jail Legislation. See Exhibit 6-9 for questions about legislation and penalties. Professional Licensing Bodies. See Exhibit 6-10 for questions about licensing. Occupational Safety. See Exhibit 6-11 for questions about safety legislation.

© 1998 by CRC Press LLC

EXHIBIT 6-12. CUSTOMER LOSSES Question

Answer

Cost

Answer

Cost

What are the possible losses to customers in the first hour? In the fourth hour? In the first day? After the first week of a disaster? Would customers seek a refund, or would they take your company to court? Would a judge understand your company difficulties or award a large settlement? How many customer or client accounts might be affected by a disruption?

EXHIBIT 6-13. LOST TRUST Question Would your company permanently lose customers during a disruption? How difficult would it be, and what would be the costs associated with, gaining back customers lost when they could not contact your company? What would the company have to do to assure customers that this type of disaster would not affect them again in the future? Would customers be more likely to be demanding in the future?

Lawsuits Customer Financial Damages. See Exhibit 6-12 for questions about customer losses. Loss of Reputation and Trust. See Exhibit 6-13 for questions about lost trust. Liability Issues: Company and Individual. See Exhibit 6-14 for questions about lost

trust.

Loss of Market Share See Exhibit 6-15 for questions about lost market share.

Loss of Good Will See Exhibit 6-16 for questions about lost good will.

Loss of Employee Productivity See Exhibit 6-17 for questions about lost productivity.

© 1998 by CRC Press LLC

EXHIBIT 6-14. LOST TRUST Question

Answer

Cost

Who at your company would be liable for customer damages? Would directors have to pay personally, or would the company only bear the liability? Who could be named in a lawsuit? Will breaches of contract affect a lawsuit outcome? Does the company have service-level agreements or specific performance levels that must be met? What is the risk of customer death or injury? What is the liability of the company in the event of death or injury? How much does liability increase if the company has no plan? What are the reasonable requirements according to what other companies have? What would the company be expected to plan for?* * Companies would be expected to have planned for an ice storm in northern Quebec, but probably not in southern Florida. Similarly, a heat wave in northern Canada might be unexpected, but would be normal in the southern United States.

EXHIBIT 6-15. LOST MARKET SHARE Question

Answer

Cost

How many competitors do you have? How likely are competitors to be affected by the same disaster at the same time as you are? How quickly and effectively would your competitor take advantage of the disaster? Does your competitor have a continuity plan? If so, could they carry on business when you were not able to?

Loss of Shareholders Your company must have the confidence of investors to finance future large projects, such as expansion and the inevitable technological upgrades of the future. See Exhibit 6-18 for questions about lost investors.

© 1998 by CRC Press LLC

EXHIBIT 6-16. LOST GOOD WILL Question

Answer

Cost

How would local news media react? Would you be accused of lack of sensitivity, or poor planning? How well would your employees deal with customers who were rude, upset at the outage, or irate due to delays? How would your customers be affected by your disaster? Would your disaster in turn create a disaster for your customers? Would you have to issue public relations announcements? What would happen with your technical service or customer service lines?

BUSINESS IMPACT SUMMARY Outline This is where you pause and summarize your preliminary findings. By briefly examining each portion of the procedure done up to now, you give yourself some direction and point the way for future examination. This has the benefit of helping you put your discoveries into some kind of perspective. A short report would also validate the continuity plan project for senior management, showing that even a preliminary inquiry has determined that various departments have significant vulnerability to disruption. At this point you should stop doing the BIA and proceed to the next chapter, “Risk Analysis.” After completing that, return to this chapter and finish the BIA. Exhibits 6-19 and 6-20 combine figures from the risk assessment and the levels of outage and probability of outage. From these, you would calculate the expected amount of downtime in the next two years. Insert the figures for you company in the blank boxes of Exhibit 6-19.

Recovery Window The time allocated for the recovery window depends on your industry. As you will see in Exhibit 6-21, the primary variable is the length of time within which your customer expects to make contact with your company. If the customer wants an answer immediately, the recovery time objective (RTO) must be very short. If the customer expects your answer to take a long time, the RTO can be longer. Exhibit 6-21 gives you a very rough idea how long your company can take to recover its communications before customers and business partners might complain. Exhibit 6-21 also provides a general idea of what kind of recovery strategy might be required to attain that RTO. A better method is described below.

© 1998 by CRC Press LLC

EXHIBIT 6-17. LOST PRODUCTIVITY Question

Answer

Cost

Would the simple threat of a disaster affect your employee productivity? After the disaster, would your employees be mentally and physically able to return to work at their normal productivity? How long would this process take? Do you have a plan to get psychological therapy for employees who are traumatized by the disaster so that they could return to work? Would employees’ families be affected by the disaster? How would this affect the employees? Would employees feel that management would support them through the disaster? Would employees be able to get to work or would the disaster make travel impossible? What help would employees expect for long-term trauma and injury? What help would employees expect for dependents and damage to home? Would it seem like management cares? What about casualties at the workplace? How would you replace and train workers? What are the costs of replacing a worker? What are the costs associated with disability compensation for workers? What are the costs of dealing with survivor trauma? How would you get employees back to work?

How Much Speed to Be Cost-Effective? One of the major reasons for determining an RTO is to find out how much your recovery effort is likely to cost. The less time you set for recovery, the more it will cost. The secret is to spend enough to recover in a timely fashion, without spending too much on a disaster that might not happen. This is where calculating your RTO is very helpful. The RTO tells you how much time you can take before it is necessary to recover. That, in turn, tells you what means you must use to achieve that goal, and what methods might be optional.

© 1998 by CRC Press LLC

EXHIBIT 6-18. LOST INVESTORS Question

Answer

Cost

What would be the effective investor confidence after a disaster? What would be the risk to capital? Are investors aware of your current lack of a business continuity plan? Would investors compare business continuity vs. risk to assets? Would there be a massive sell-off of company shares after disaster? What is the chance of the company surviving investor loss of confidence and sell-off of shares? What is the chance that investors pulling out might affect the ability of the company to meet its current financial obligations? What would be the additional operating costs to replace buildings and equipment? What would you need to spend in advertising to repair the company reputation? Would prior agreements be met? What litigation would be forced upon the company because of the inability to complete agreements? What would be the effect on future IPOs?

EXHIBIT 6-19. CALCULATED LOSS CHART 30 Min. Peak 1 Hour Peak

4 Hour Avg.

1 Day Avg. 2 Day Avg.

Exhibit 6-2 Exhibit 6-4

N/A

N/A

N/A

All other Exhibits in Chapter 6

N/A

N/A

N/A

N/A

How Much Delay Is Deadly? In Exhibit 6-22, we have shown the formula to determine your cost-effective RTO. The senior management of the company makes a business decision, deciding how much loss is acceptable. Then you (and your planning team) divide the acceptable loss by the cost (expense or lost revenue) of downtime per hour of outage.

© 1998 by CRC Press LLC

EXHIBIT 6-20. EXPECTED LOSS DUE TO DOWNTIME CHART

Number of Disaster Risks in Category

Ratio of Expected Events

Very high

1:4

High

1: 6

Moderate

1:12

Low

1:36

Very low

1: 64

Number of Risks Expected to Occur

Average Hours of Downtime for These Risks

Expected Dollar Cost

To use this table: 1. Take the total number of risks in the category. 2. Divide by the ratio. If you have a 90% probability of disaster it is extremely probable that 1:4 of them will strike. This equals the number of risks expected to occur. 3. Multiply by the number of hours and estimate how many working hours or working days the disaster would consume. 4. Multiply by the appropriate time box on Exhibit 6-19 to arrive at your expected dollar cost.

EXHIBIT 6-21. INDUSTRY-BASED RTO Time Frame

RTO before Call Centers

Industry Type

RTO Essential to Company Survival

Immediate

Formerly 0 to 24 hours

“Instant contact” industries; e.g., sales, utilities, “just-in-time” supply system

Recovery 0 to 30 minutes requires prerouted switching, hot site*

Intermediate

Formerly 1 to 7 days

“Return call” industries; e.g., professionals, consultants

Recovery 30 minutes to 2 days requires warm site or hot site

Long-term

Formerly 7 days or more

“Return letter” industries; e.g., manufacturing, scheduled projects

Recovery 2 to 5 days requires warm site or ramped-up cold site

* Prerouted switching would be to an interim call handling or real-time minimum overflow facility, as discussed in Parts 3, 4, and 5.

Note that depending on call volume at various times of day, the cost-effective RTO might differ. For instance, an outage at midnight might require an RTO of six hours, but at peak time the RTO might be 30 minutes. An outage 45 minutes before the end of the midnight shift change to day shift might require the RTO to be 1.5 hours.

© 1998 by CRC Press LLC

EXHIBIT 6-22. COST-EFFECTIVE RTO

Time Period

Acceptable Loss

(Divided by) Cost Per Hour Of Outage

(Equals) RTO

Peak Average daytime Average evening Average midnight

BIA REPORT Why You Must Issue a Document The BIA report that you and your report team write should not be a mere memorandum or spreadsheet table. If the tangible and intangible risks, vulnerabilities, and monetary costs are what we think they might be, you should issue the document formally, complete with a special cover. The document that you issue should bear the date of issue and the names of the contributors. Special Issue. The BIA document will form the basis of discussion by the senior management of the company. It is the senior managers’ task to confront the liabilities (company and personal) that could be incurred if the company does not take on the continuity-planning project. They must weigh the financial consequences of suffering a disaster, without preparation, against the probable costs of running the company on a continuity-based continuous-planning basis. Once the senior management has decided, as we believe they must, to embark upon the process of planning, testing, and training for continuity, that is not the end for the BIA document. It is possible that the CEO might have to present the document to the board of directors. This is the reason that it must have a formal cover and title page. No spreadsheet printout would suffice for that purpose. Date of Issue. The BIA report should bear the date of issue because this BIA is only

the first step of the continuity-planning process. Later BIA update reports will refer to it, and they will need to be dated also.

Executive Summary You and your BIA team should write the executive summary to the BIA so that it shows in short form: 1. Scope — critical functions of call center and departments; 2. Risks and Costs — the likelihood of disaster, costs in dollars per hour of outage; 3. Departmental Impact — personnel, equipment, and nontangible vulnerabilities.

© 1998 by CRC Press LLC

Summary of Topics Major Findings. You and your BIA team should write the “major findings” as a summary of the major tables. What are the major call types, inbound and outbound, that are handled by company personnel? How much revenue do these call types bring in, or how much expense do they save? What are the major risk implications for a company that has no continuity plan? What should the senior management choose as the company RTO, as shown by time-ofday call patterns? What kind of recovery method would fit such a goal? You should use the questions and answers from the risk implication tables in Chapter 7 to create a section discussing additional issues for consideration. Long Form a Living Document. As noted previously, the BIA should be a living document. The company should go through the exercise (in long form) at least every five years. Such a discipline would enable the company to stay up-to-date with changes in call patterns, changes in legislation, and even changes in sales as the marketplace evolves with time. Departmental Update Schedule (Not to Be Locked Away). To keep the raw information, from which such a document would be drawn, up-to-date, you and the BIA team should establish a schedule for updates. This is not something that you would have to do today. In fact, as you go through the risk analysis that follows, and start turning the BIA team into your planning team, we think that you might become busy for quite some time with writing the continuity plan. However, please take a moment right now to schedule a review (by all departments) of the major tables in this chapter, once per year. Once the continuity project gets going, that kind of recurring review should be enough to alert department heads if any major change should occur that the company needs to recognize.

© 1998 by CRC Press LLC

7 Risk Analysis WHAT ARE THE RISKS? It is important to know what risks are peculiar to your industry and your region.

Regional, Economic, Seasonal, Political At this stage in the game you have gone through the BIA and now you are going through the risk analysis. As you saw by examining the risk implications that you have just examined in Chapter 6 (unless you are just reading through this book first), there are not only categories of disasters but there are also different scales of disaster. One of the reasons you do a risk analysis is that it forces you to understand how much risk there is out in the world, and why your company requires business continuity planning. A fire alarm in a call center, even when there is no fire, could cost a lot of money during peak call periods. This level of disaster is going to be different for each company, depending on the unique cost per call hour of each company. (What is the possible revenue that could be lost? What is the possible expense or loss of reputation that could occur if the call is not handled?) That is why we did not fill out in hard copy most of the forms but rather we have shown you how to fill them out. This way the data of your company can go into the applicable spaces. The largest problem a company has is recognizing its requirement for a continuity plan. With the BIA combined with the risk analysis you can make a business case to management for the rest of the continuity plan, and this is the reason for the content of the chapters thus far. Large numbers of companies that have sent representatives to our seminars do not have a business continuity plan of any kind. In fact out of any one hundred companies sending representatives, roughly fifteen will say they have a plan. But even from those fifteen, when asked “when was the plan last tested or updated,” only two or three might raise their hands out of the hundred people at the seminar. Being without a tested and up-to-date business continuity plan is not a good situation to be in. If you have not tested the plan, you have no plan. To justify the need to have a plan, you can look and see how much business your company could lose. What kind of lawsuit could you personally, management, and the company itself be hit with if a disaster plan were not devised? Support from the highest level is essential. To get that support, you need to present the BIA and the risk analysis. Fortunately, this book will show you how to start. You began with the BIA, taking it step by step to determine the biggest impact to your company. Next you will find out where a disaster could hurt the company the most. You will gradually expand your plan through various sections to take into account areas of your company that could be hurt the least, making a list of your vital areas of business. You do not need to do this all at once. It is like eating a mountain of chocolate. You can do it one bite at a time. You need to take this process in little chunks. Get the critical functions done first. Once you have done the BIA to determine which are the critical functions, then you can work on protecting all of these functions on a bite-by-bite process. You cannot plan for everything, but as you will see this is not a plan for everything. This shows your company what to do in an escalating series of circumstances, so that vital functions of the company remain online or return to normal status as soon as possible. The size and magnitude of a disaster will determine what portions of your continuity plan to initiate and carry out.

© 1998 by CRC Press LLC

Exhibit 7-1. Disaster Types and Causes

When it comes to assessing the various risks that your company might be subject to, it might be that all you have to do is take a walk down memory lane, or talk with other people in your company. Failing that, you could look through a small pile of local or trade magazines, or consult the archives of your city or regional newspaper or television station. In the last ten or even five years, how many news articles do you count that mention disaster? How many have taken place in your region or city? Or in your or another region, affecting companies in your industry? How many “disasters” had economic causes, such as companies failing during recession, suffering embezzlement, or having to pay large court settlements? How many had a seasonal component, such as severe weather, wildfire, or flooding? How many disasters had a political aspect, such as calls from one or other quarter for greater government regulation, or assistance, or restraint, or funding? Almost every news article about disaster gives a dollar figure for the losses suffered in each particular incident. We encourage you to count up these financial losses. You might find the anecdotes about the disasters at other companies to be instructive.

Statistics and Their Sources In Chapter 1, we cited information from the International Federation of Red Cross and Red Crescent Societies, the United States National Archives and Records Administration, and a study sponsored by AT&T. In Chapter 4 we cited some uncomfortable statistics from Contingency Planning Research, of Jericho, New York. There is no shortage of statistics to refer to. Many of these organizations release quarterly or annual statistics to the news media, like those cited in Exhibit 7-1. Federal agencies such as FEMA or USGS (Federal Emergency Management Agency and the United States Geological Survey) publish information regularly as part of their normal activities. The government departments of both the United States and Canada publish statistics frequently, and this information is discussed in both print and electronic news media. Charitable organizations such as the American Red Cross often publish statistics in their fund-raising brochures and television messages. Readers who are comfortable surfing the Internet can obtain statistics from around the globe on any disaster or business-related topic they can phrase for their search engines. Many Web sites include lists of similar disasters grouped together and arranged by date.

© 1998 by CRC Press LLC

FINDING THE RISKS Risk Analysis Risk analysis goes hand-in-hand with the BIA to help deliver to your company a picture of how long your company could reasonably expect to be out of operation in the next couple of years and how much money that could cost. That gives management an ability to be able to determine how much to spend on business recovery, business continuity, and risk mitigation.

Risk Categories In the risk analysis matrix shown in Exhibit 7-2, we take the different risks and break them into categories. It is important to know what risks are peculiar to your industry or your region, because each region and industry has different risks associated with it. It would be impossible for any individual who is not familiar with the company, the location of the company, and the factors that are close to the company to be able to predict the degree of risk that lies in wait for that company.

INSURANCE INDUSTRY INPUT You could obtain additional information from the insurance industry. Insurance brokers and underwriters have statistics on mortality for life insurance, fire for fire insurance, and on floods and earthquakes. They have special riders (according to the region you live in) for policies to cover exceptional risks. Insurance representatives have statistics on business failure, such as that caused by the loss of facility or key personnel. They have statistics that are derived from errors and omissions, and directors and officers, insurance so that you can find out the number of times managers have been sued for failing to fulfill their responsibilities. The underwriters have a great deal of information on the likelihood of a disaster in your particular area.

RISK ASSESSMENT MATRIX We have taken the various risks shown in Exhibit 7-2 and divided them into three different areas — natural, infrastructure, and man-made. Each of these three areas appears as a separate table in Appendix A-25. Because each risk has different levels of risk to it we have divided risk into six categories. Category 1 would be the least damaging and Category 6 would be the most damaging. For instance, in the fire category, a fire drill or small internal fire is going to cause a minimum of inconvenience, yet it is very likely this will happen in the next two years in your facility. Our purpose for doing it this way (as opposed to just giving a general rating for each hazard as is done in a lot of disaster recovery manuals and is done by the Disaster Recovery Institute) is to assist you in building checklists and escalation charts later on. Other plans create a risk factor. This factor is a combination of the frequency of the event, the probability of it happening, and the severity of the event, which the other planners multiply out to get a risk factor. The other method gives you a good overall base of knowledge, but it leads people astray when they are trying to figure out the hours of downtime they could experience in the next little while. A planner could get a risk probability factor of about 36 on a given item because of its severity, but it might turn out not really likely to happen. We are recommending a

© 1998 by CRC Press LLC

© 1998 by CRC Press LLC

EXHIBIT 7-2. RISK ASSESSMENT MATRIX Disaster Type

1

2

3

4

5

6

Natural Disaster Hurricane or typhoon

Hurricane warning

Hurricane watch

Hurricane strike within 30 minutes

Hurricane strike

Tornado

Tornado warning

Tornado watch

Strike within city

Strike within six blocks

Direct strike to facility

Heavy thunderstorms/wind damage

Weather watch

Weather warning

Damage to power lines; e.g., broken branches

Power outage more than a day

Power out more than a week

Epidemic

Medical alert notice

Some personnel affected

30% personnel absent

More than 50% affected

Building or personnel quarantine

Snow and ice storms

Personnel delays at shift-time

Minor power outages

Widespread power outages

Power out 1 day; property damage

Power out less than five days; major damage

Power out more than five days.

Wide-area flooding

Flood watch

Flood warning

Minor flood damage

Major flood damage

Building under water more than a week

Building and equipment destroyed

Internal flooding

Roof leak

Roof leak near ACD

Water main to upper floor burst

Major water damage to equipment

Building flooded; major repairs required

Repairs required; 2–4 weeks

External flooding

Flood watch

Affecting some people getting to work

Building surrounded by water

Water leak in building; some damage

Major equipment failure

Building evacuated; equipment replacement required

Earthquake or seismic damage

Notice of possible quake

Personnel disruption in affected area

Minor tremor; some damage; RTO 1–3 hours

Major damage; RTO 1–3 days

Major damage; 3–5 days

Building destroyed; 1–4 weeks

Volcanic eruption

Possible eruption notice

Affecting personnel enroute to work

Fallen ash causing breathing problems

Evacuation of nonessential personnel

Lava flow; complete evacuation

Building and equipment destroyed

© 1998 by CRC Press LLC

Infrastructure Disaster Hardware failure

Minor glitch affecting only a few PCS

Power failure to some equipment 1–3 hours

Major hardware breakdown; RTO 1–5 hours

Main cable cut; repairs require 2–12 hours

Major damage to cable; RTO 2–5 days

Equipment replacement required 1–3 weeks

Software failure

Virus alert

Virus detected; 30–60 min to clear

Minor failure of main program RTO 1–3 hour

Major program failure; RTO 2–8 hours

Complete software failure; 1–3 days

Replacement required of all software; RTO 1–3 weeks

Power failure

Some lights affected

Minor disruption to equipment; 30–60 min

Major disruption; 1–4 hours

Complete power failure; RTO 4–12 hours

Major transformer/circuit damage; RTO 1–3 days

Major equipment damage; RTO 1–3 weeks

Water

NO COFFEE

NO COFFEE

NO COFFEE

NO COFFEE

NO COFFEE

NO COFFEE

Disruption to local service only

800 service lost; RTO 1–3 hours

Major disruption of service; RTO 1–6 hours

Complete loss of landlines; RTO 6–24 hours

Major repairs required; 1–3 days

Occasional disruption of local service

Lost of 800 lines; RTO 30–60 min

Minor interruption of IXC/LEC; 1–3 hours

Major failure of communications RTO 1–4 hours

Complete communications Failure; RTO 1–12 hours

Major equipment damage; RTO 1–3 days

Chemical exposure

News release of highway chemical spill

Some personnel delayed enroute to work

Minor exposure; some evacuation required

Major exposure; full evacuation of staff

Minor equipment damage due to corrosion

Major corrosion damage; RTO 1–5 days

Internal fire

Fire drill False alarm

Small fire, quickly extinguished

Small fire, fire department called, required to extinguish

Small to moderate fire

Moderate fire

Large fire

External fire

Grass fire affecting personnel enroute to work

Adjacent building on fire under control

Building threatened; evacuation of people 1–4 hours

Some external equipment damaged; RTO 4–12 hours

Major damage to building and equipment; 1–5 days

Building and equipment destroyed

Telco local CO

Telco IXC/LEC or 1-800 service

Man-Made Disaster

© 1998 by CRC Press LLC

EXHIBIT 7-2. RISK ASSESSMENT MATRIX (continued) Disaster Type

1

2

3

4

Communications failure; internal/external

Hits on the phone line

Minor interruptions to local telephones

Failure of internal communications; 1–3 hours

Major failure to internal/external communications; 1–4 hours

Human error

Minor mistakes corrected in minutes

Accidental program deletion; 1–3 hours

Allow access to restricted area

Damage caused by unauthorized personnel

Disgruntled employee

Complains continually — family problems

Mental anguish — disrupts other workers

Threat to other personnel and equipment

Strikes/picketing

Negotiations

Noon pickets

Improper handling of sensitive data

Minor embarrassment; no damage

Theft

5

6

Major repairs required; 1–3 days

Major cable replacement; 1–3 weeks

Distraught — destroys equipment and causes a calamity

Shoots supervisor, wounds co-worker

Kills himself or herself

Strike declared

Major picket line

No workers will cross; work stops

Plant closes

Some codes compromised

Some data must be replaced; 1–3 hours

Major compromise of sensitive data

Data obtained by competitor

Data used by competitor

Employees taking pens and pencils

Software disks taken

Keys and codes to restricted areas taken

Sabotage caused by theft

Major hardware stolen

Company secrets sold to competitor

Bomb threats

Bomb threat; briefings to personnel

All threats to be taken seriously

All personnel evacuated area searched

Bomb squad called; bomb located; no damage

Small explosion; some damage

Building and equipment destroyed

Civil disorder

News reports

Strikers interfering with workers

Protesters disrupting operations; tyingup equipment

Destruction of property

Sabotage of equipment

Operations forced to shut down

Chemical spill

External site affects personnel reporting to work

Internal spill; minor clean up 1–10 minutes

Major spill; some evacuation required

Major spill causes full evacuation 1–12 hours

Corrosive spill; some damage to equipment; 1–5 days

Major equipment damage; 1–3 weeks to replace

Airport proximity

Possibility of aircraft accident

Road closure due to aircraft accident

Aircraft crash near site; personnel evacuation

Aircraft crashes into site; fire damage

Major damage; 1–5 days

Complete loss of building

EXHIBIT 7-3. LEVELS OF IMPACT

Level

Duration of Interruption

Personnel Disruption

Damage

1

Less than 30 minutes

Causes for concern (distractions)

None

2

30 minutes to 4 hours

Temporary disruption

Marginal — minor effect on operations

3

4 hours to 1 day

Significant disruption of employees lives, some unable to report to work

Minimal or moderate damage to operations center

4

1 to 2 days

Serious disruption causes severe stress/anxiety

Moderate damage to operations center

5

2 to 5 days

Large number of personnel unable to report to work

Infrastructure failure; heavy damage to operations center

6

More than 5 days

Severe disruption for employees; loss of homes; evacuation, etc.

Operations center destroyed

more-detailed approach to risk assessment by breaking risk down into six categories of severity and having a look at all the different risks in each category. In Exhibit 7-2, we have given examples of levels. This is where you need to decide on the levels for each of the six categories. There are three different sections: duration of the interruption, personnel disruption, and damage. Each one of these categories has a rating from 1 to 6. An event might be a 1 on the level of interruption, but it might be a 3 or a 4 in the other categories. You should rate the event at the highest level of disruption that it would cause. You can see how in Exhibit 7-2 we have entered in some possible responses. If you choose, you can use the same information that we have used. However, we recommend that you evaluate each category yourself and make your own assessments.

EXAMPLES OF LEVELS Exhibit 7-3 displays some examples of how you may set your levels of impact. Please note that each column should be taken individually. A given event might have either a given duration or personnel disruption, or a given level of damage to the facility. Or, there might be a combination of one, two, or three factors of differing impact. In that case, take the factor having the highest impact. Once you have completed Exhibits 7-2 and 7-3, then you have information to complete the risk-type result chart in Exhibit 7-4. This chart takes the risks and groups them by their impact. Completing this should help you estimate how long the company would be put out of commission because of any of these particular risks. You would use this chart to calculate how long you would reasonably expect the call center to be out of operation. Now you have the disaster type and the impact information available. It is important in Exhibit 7-5 that you do not rate according to disaster type. You should rate by the impact rating that you assigned in the previous risk assessment chart. As for frequency of each disaster type, you can either rate this on a percentage basis or (as we recommend) rate it on the number of occurrences your area records per year.

© 1998 by CRC Press LLC

EXHIBIT 7-4. PLAN FLOW Shortduration outage

No recovery plan activation required

Return to normal operations

Short evacuation required

Evacuation of call center

Call handover

Return calls

Return to normal operations

Moderate damage to call center or infrastructure

Evacuation of call center

Call handover

Activate hot site

Return calls

Return to normal operations

Severe damage to call center or infrastructure

Evacuation of call center

Call handover

Activate hot site

Activate cold site

Return calls to new site

Return to normal operations

We have categorized the scale of disaster in the previous exhibits by type according to the paradigm described in the following sections.

Probability Based on records from previous years, how probable is the type of disaster? Is your area visited by a hurricane once in a hundred years? That would indicate a low probability. Do hurricanes swing through every three years or less? That would indicate a high probability of such an event occurring sometime in the next two years. Try to rate probability in the following groups: Very high High Moderate Low Very low

90% or more 75 to 90% 50 to 75% 30 to 50% 30% or less

Degree of Predictability Do meteorologists often predict weather of a given type for your area? Are they somewhat accurate when they do so? Are they able to predict whether an event will be minor or severe, and are they reasonably accurate when they do so? What about other events not associated with the weather? Is there any way to predict if a disgruntled employee is likely to commit theft or sabotage? Perhaps, but only in the most general way. Is there any way to predict that a piece of equipment might malfunction? Perhaps, but only by the frequency of employee complaints made about it. Yet we showed at the beginning of this chapter in Exhibit 7-1 that statistics record human error as the major cause (68%) of disasters affecting businesses, followed by technological failure (25%). We can predict on the basis of past events that severe weather will probably cause only 5% of disasters, and human intention or malice is likely to cause only 2% of disasters. If you can make an event predictable, that makes writing the plan easier for you and the planning team.

© 1998 by CRC Press LLC

© 1998 by CRC Press LLC

EXHIBIT 7-5. RISK TYPE/RESULT CHART (SHORT-DURATION OUTAGE)

Disaster Type Tornado: Watch

Fire Drill Small Fire Tornado: Direct strike

Impact Rating 3

1–3

6

Frequency of this Disaster Type

Probability

Seasonally 3

Amount of Forewarning

Speed of Onset

Duration of the Disaster

1 to 2 hours

20 to 30 minutes

15 minutes to 2 hours

At peak: $50,000 lost calls

20 to 30 minutes

Less than 5 minutes

5 days

$500,000 to $1 million

Severity

2

Seldom

Low

Duration of the Disaster How long would you expect the call center to be down because of a minor fire? How would that compare with structural failure of the building because of an earthquake? Often, duration will be a result of the severity of the event. At other times, the duration will be a result of indirect causes such as traffic tie-ups, actions of people in other organizations, or timing. A minor disruption requiring the help of a plumber or electrician that occurs on a Wednesday at noon is probably more easily resolved than one that occurs at three in the morning.

Severity Severity is a measure of the amount of damage that would be done by the disaster or interruption while it is taking place. For instance, we have classified a false fire alarm as an event that would have severity limited to the amount of time that was lost for the alarm itself. That would have a severity rating of 1. A fire alarm due to a moderate-sized fire, which necessitated an evacuation of a call center and loss of sales revenue in the thousands of dollars, would have a higher severity rating.

Frequency of Types of Disasters How often do power failures strike your area? Would you rate them as frequent or seldom? If an event can be categorized in this way, you can plan to take preventive action.

Speed of Onset How fast could a fire spread through your call center if the fire sprinklers were working? How much faster could it spread if the water supply were interrupted? How fast could your call center personnel succumb to an epidemic? How much faster could they be incapacitated if poisonous smoke were blowing in from a building on fire down the street?

Amount of Forewarning Some disasters such as tornadoes give two minutes of warning or less. Others such as hurricanes give 24 hours of notice or more. If you can receive forewarning, you and your team will have a better chance of taking preventive action. Sometimes the amount of forewarning depends on whether company personnel are paying attention to warning indicators. For instance, in addition to tornado or hurricane watches given by radio or television, the sky tends to turn dark before such severe weather. Your plan can be written to include “looking out the window” as a checklist item, if you choose.

RISK MITIGATION Philosophy We believe that it is possible to plan for the future, even though it is largely hidden from our view. The purpose of continuity planning is to reduce the effects of emergency upon companies and organizations, so that they can continue, survive, and prosper. By planning to avoid some risks and to minimize others, those willing to expend some forethought and energy can increase the likelihood of their company continuing in business.

© 1998 by CRC Press LLC

Prevent What Is Preventable As mortal humans, we do not have the capability to prevent every adverse event. We do, however, have the power to prevent some events (even severe ones) and diminish the damage from others. Can we look at a cracked window and predict that a high wind would shatter it? Then we have the power to replace the window and prevent injuries and damage from flying glass. Can we look at a funnel cloud heading our way and predict that it might sweep over the call center? If so, then we can evacuate the personnel in that facility and reroute call volume, thus preventing personal injury and loss of revenue.

Minimize Vulnerability As mortal humans, we have a degree of free will. We can choose to heed a warning, see a danger, and prepare a plan to minimize our vulnerability to disaster. Is the call center located near a river that floods in the spring? In that case, we have the technology to erect permanent barriers to channel away all but the highest floodwaters. Is the river predicted to crest at a higher level this year, higher than our barriers could hold? In that case, we can decide to relocate to another facility on higher ground. Of course, we also have free will to ignore the danger. Let us not use it for that purpose!

Planning and Preparation Even if we can see a danger, avoiding disaster takes planning and preparation. Counting up our exposure to risk must lead us to make a plan. Making the plan must lead us to preparing the means to enact the plan. Having enacted the plan and avoided the danger, we must update the plan for next time. Living in such a complex world as we do, we can predict that there will be a “next time” — and then the risk will probably be something different. Assessing risks and writing a plan will take some work. Making the preparations to enact the plan if necessary … that will also take work. Planning and preparation result in preparedness. It is worth the effort.

Chicken Little and His Friends There are so many ways that the sky could fall! How can we avoid living in a constant state of anxiety? Ice Age, Alien Conquest, and other Distractions. How likely could it really be that an ice age might start today? How possible could conquest by aliens be, considering that movie theaters sell so much popcorn with that plot device? We encourage you to plan for the possible. It is possible to predict that human error will happen. It is possible to predict that severe weather events like blizzards and heat waves will happen. What can be predicted can be planned against. However, if an ice age occurred, would not your company not relocate? If aliens landed in your city, would not your company not evacuate? If a meteor were predicted to strike somewhere on Earth (but no one knowing yet exactly where), would not your company not enact its continuity plan, and distribute its data and calls between multiple locations? Extreme “Chicken Little” types of disaster could be viewed as a massive distraction. You should not lose sleep over them or waste time dithering about unknowable dangers. If you and your planning team set out to write a robust and flexible plan, and prepare and train to enact it, the danger would have to be catastrophic on a worldwide scale before your plan might seem useless. Even in such a case, your company personnel would fare better than those who had no plan.

© 1998 by CRC Press LLC

Mobs, Panics, Global News Media, Panic on the Internet. Notwithstanding the preceding, humans are social beings. If everyone you know is enraged by some notorious court case, you might have a hard time ignoring the matter. If everyone you know is terrified that the stock market might fall, you might have a hard time avoiding debate about finance, even if you have all your funds in treasury bills or bonds. Panics can sweep through cities, countries, and even continents. Often, the flames of emotion are fanned by speculation from the news media that the terrible trend will continue and worsen, or that there is a conspiracy afoot, an untreatable disease on its way, an unstoppable calamity in the wings. If you surf the Internet, you can research the current panic (whatever it is) deeply. You can find speculation there, too. Winnowing through so much information can take (valuable) time, but you might be able to discern what limited truth might be the basis for all the fuss. If you have time after writing and practicing the new company continuity plan, you might be interested in an experiment. Go through that stack of old magazines or the archives of the local newspaper, and see how long a “panic” usually lasts. Take a look, and see how many stories you can find about the last panic after a new one has taken hold of the public imagination. What can we say about the panic before that one or the stories that dominated the news a year or two ago? Usually people become enthralled by a craze or panic that is about an extreme condition — not an inconsequential or moderately bothersome one. Very few people are likely to worry that they will be unable to buy shoelaces in the stores. Usually people become wrapped up in panics about conditions or events over which they have no control. Very seldom do the people in your city worry that someone might prevent them from mowing their lawns or buying potato chips. If the personnel in your company, and especially in your call center, should become excitable about some topic, try to ignore it. Only if call center efficiency suffers, or some physical event occurs that threatens company survival (such as a fire or riot) — only then should you become concerned. And in such a case, would not you not enact the disaster recovery plan, evacuate the call center, and transfer your call volume to another facility?

Plan to Update Your Continuity Plan You and your planning team should write a mechanism for making updates to the plan into the plan itself. Inevitably, personnel will be promoted into other positions, or leave to retire or join other companies. Almost certainly, departmental policies and procedures will change from time to time. Probably your vendor representatives will change also. Possibly, legislation concerning your industry or business in general might change. Keeping the continuity plan updated means that you and the rest of the (future) recovery team would probably stay on top of any disruption that might arise. We discuss plan maintenance in Chapter 14.

© 1998 by CRC Press LLC

8 Recovery Team Selection TEAM STRUCTURE Primary Personnel and Alternates As discussed in Chapter 5, it is important to set up the recovery team (and all its subteams) properly from the beginning. You need to start by selecting a representative group of your company personnel. When assembling the team, you will need managers, senior managers, technical experts, and a team leader. The team leader needs to be at such a level in the organization as to be able to make business decisions quickly without too many debates in the event of a disaster. We have been discussing all planning action so far as if you, the reader, are also the planner, and that makes you the most likely candidate for team leader. Los Angeles County has a single person in charge of managing any disaster, and that same person has the authority to direct all the resources in the entire city for the duration of the emergency. You need to have the same thing in your company, in addition to having representation of each section of your business. Some representatives would be managers for CSRs, the voice network, the accounting department, and so on. You start, as we described in Chapters 5 through 7, with a planning team to help you do the BIA and risk analysis. During the planning, you will assign personnel to the teams that will do the actual work of enacting the continuity plan in the event of a disaster. There are three teams that you must have available to help you. Every position should have both a primary and at least one alternate representative. This is so that, if necessary, 24 hours of the day could be covered during a disaster. Also, if someone happened to be away with a cold on the day of an important meeting, the alternate could attend and brief the primary representative after the fact. If anyone becomes unable to participate in the project on a long-term basis because of health or family problems, the position should be assigned to another person. Disaster Site Recovery Team. The first team is a disaster site recovery team. These people would stay at the site that experiences the disaster and make decisions about what would need to be done to return the company to full operation at that original site as soon as possible. The person in charge of this team cannot be the person in charge of the overall management of the disaster recovery process. We recommend that the site property manager be a member of this team. Note the location crisis manager should lead this team. The overall crisis manager should operate from an off-site emergency command center (EOC). The EOC could possibly be located at the hot site if it is close enough. Interim Call-Handling Facility Team. The interim call-handling facility team is the second team that you will put together. These people (most likely CSR supervisors) would monitor the activities of the interim call-handling facility. This monitoring would include the handing over of Telco lines, and monitoring of the calls as the ICH (interim call handling) site ramped up. They would ensure that the facility to which your call center handed

© 1998 by CRC Press LLC

over its calls met the company standards of customer service. Note that your ICH site should have remote call-monitoring capability. This is where the project manager (you, the reader/planner) should go. In Exhibit 8.1, your position is called “planning manager.” Hot Site Recovery Team. This team would go to the hot site along with the planning manager (if required) and begin to bring up the facility. As far as task allocation is concerned tasks should be as close to each person’s normal function within the company as possible. This would avoid confusion and would present team personnel with as few new situations to deal with as possible. The people that can remain calm in the event of a disaster and/or those who have experience with disaster recovery are the most likely people to assign to the team. Just as a recap, the ICH team would monitor the call handling activity of the assisting call center. The hot site recovery team would set up operations at an alternate preprepared site where your call center personnel would operate from while the original site was being repaired. The disaster site team would stay behind and handle the disaster. Accounting Department Representative. You will notice on the chart in Exhibit 8-1 that the accounting department representative is included as one of the mandatory team members. It is important that one of the senior people for the accounting department be on the team because the continuity project team would often need funds quickly to bring the company back to life. You need to have someone possibly as high as the CFO working with the planning team to establish budgets. Having a senior accounting representative on the team would make obtaining access to the money that would be needed during a disaster a lot easier. If you had to start looking for funds in the middle of a disaster, it would make meeting the challenges of the situation much harder. As we describe in the discussion of the continuity fund (see Chapter 21), funding to make repairs or replacements should be included in the company budget. Your CFO should be prepared to open a continuity fund account, not only throughout the process of planning for the company continuity, but also as a continuing obligation. If all the vendor contracts were arranged for emergency supply and no cash had been set aside to pay the invoices, that could be a corollary disaster to the first one. It is difficult to assess what things will cost during a disaster. So you should try to set up as many things in advance as possible. As we describe later in Chapter 21, advance vendor service-level agreements are a part of the continuity plan that should not be overlooked. You should discuss with vendors the possibility that in a serious disaster they might have to go beyond the service level to get the work done. In a wide-area disaster you should expect that utility suppliers and some equipment suppliers would be affected by the disaster too. If companies did not live up to their servicelevel agreements then you could sue them after the fact, but you would have to first make sure that your own company is around to do that. Consequently, the better way would be to make sure that you could get diesel fuel for your generator by selecting a vendor from an out-of-state company to supply your diesel fuel for the disaster. This is an expense that might come up that you forgot to budget for unless the accounting department and especially the CFO are fully committed to the continuity-planning project. During the planning and testing stages it will probably be difficult to know how much money you would need to recover from a disaster. Some things are easier to plan for than are others. Setting aside extra funds is always a good idea. The person you have placed in charge of finances for the disaster planning should assist in putting forward the best possible financial projection.

© 1998 by CRC Press LLC

The person in charge of the finances would also be there to make sure that there are not significant cost overruns. Also, as time goes by the finance representative would need to look back at the BIA and update it. This is because the view of the BIA going into the planning and testing stages will probably change as you revise your continuity plan. If possible, it is best to have this person involved right from the beginning of the planning process. In Exhibit 8-1, we have included the essential positions (in regular print) as the fundamental part of the team. The other members of the team are made up from the departments listed in the annotations (set in italics) and will be unique to your own company and its situation. Note that customer names in the example are fictitious.

Tasks and Checklists The planning team will also furnish many of the people for the three other teams. As the planning process gets into full swing, these people will draw on their expertise and experience to draft disaster implementation checklists for their departments. However, the people assigned to the ICH and the hot site recovery teams will be entering largely new territory. If you find that you need help on this part of the project, do not forget that the outsource call center(s) should participate in the planning process too. The external liaison representatives and the outsourcer representatives will need to consult with each other to draft a checklist that is common to both sides of the call-switching transfer.

Keep Updated As noted above, the team representative assignments must be kept current. You should plan for vacations, illnesses, parental leave, and even conventions that could cause personnel to become unavailable to the team for a period of time.

Shift Schedule — Every Shift Must Have a Representative If your call center or other departments operate outside of the nine to five, Monday to Friday “traditional” work week, you should make provision for each departmental shift schedule to assign continuity plan project team primary and alternate members. During the planning project period, these team members might make only minor contributions. However, you could assign them to plan for facility emergency supplies such as blankets and pillows, water, toilet paper, and first aid supplies. This would cover a possible disaster contingency (facility is isolated by blizzard, storm, flood, or earthquake) and take this task from the property management representative (for example) who would be busy planning for electrical power generators and portable toilet vendor contracts. During an actual disaster, especially one longer in duration than four hours, the team shift representatives would be a valuable link between the incident or crisis manager and the CSRs. This includes the enactment of the continuity plan, and the recovery to an alternate site. Upon return to the original site (or a replacement) the shift representatives would also do important work explaining the procedure to the call center employees of the various shifts. For this reason, the team shift representative and alternate should be drawn from the shift supervisor and senior call-taker level.

Representatives from Security and Maintenance As noted above, the property management representative and alternate will be very busy ensuring the provision of utilities for the facility. In addition, the team will require departmental or contractor representatives (and alternates) to plan for security and maintenance measures in the event of disaster. For example, what measures would need to be taken in

© 1998 by CRC Press LLC

the event of an evacuation of part of the building because of windows blown in by high wind? This would include security action to deny entry by unauthorized personnel and maintenance action to clean up broken glass.

INCIDENT MANAGER The incident manager is also sometimes called the crisis manager. Unless your company uses a third-party crisis management contractor, the terms can be used interchangeably (see Exhibit 8.1).

Centralized Reporting As described elsewhere, the initial activation or enactment of the continuity plan includes the following actions: • Quick mobilization of first response teams • Gathering of accurate information about the disruption • Determination of whether further mobilization of plan provisions is required • Identification of departments and personnel affected by the disruption • Controlled response to ensure a single point of direction (“command and control”) for all teams, all departments, and all personnel.

The incident manager provides the single focus for everyone’s efforts, and is the person who should receive all reports and confirmations of checklist actions performed. The purpose for this is to minimize confusion and prevent panic. The incident manager coordinates the recovery process. Having identified the root cause of the disruption, this person would be responsible to safeguard the company communications (by handing over calls as provided for by the plan) and critical business functions. The incident manager would provide for handover of calls to the ICH facility, and give the signal for relocation of critical departments. This is the person who would assess the progress of the disaster according to the reports of recovery team members (including all three subteams) and would be responsible for solving problems as they arose. The incident manager would then determine how long the disruption of normal operations might last, and ensure that plan provisions for ensuring continuity were enacted. Like everyone else in the project team, the incident manager needs at least one alternate representative. Since no one knows in advance if a disaster would be long or short in duration, this position should probably have two or three alternates. If a disaster should run for 24 hours of several days, provision must be made for the incident manager and alternates to get adequate rest to think clearly. This means planning for cots in the EOC as described below.

Emergency Communications Plan Ted Ging, of the PNC Bank, has stated that a company’s crisis communications plan is the most important part of its overall business continuity process. This is because the ability to communicate with customers and with recovery team members is the essential function for ensuring continuity of the company and its operations. In addition to the transfer of call volume to the ICH facility, the incident manager would require interactive communications to receive ongoing status reports from all team members. This includes the first people on the scene and the response teams who would follow

© 1998 by CRC Press LLC

EXHIBIT 8-1. TEAM SELECTION BY CRITICAL COMPONENTS (ANNOTATED) Company Primary Planning Manager Crisis or Incident Manager Senior Management Representative Accounting Department Representative “Disaster Site” Recovery Team ICH Facility Team Hot Site Recovery Team PBX, ACD, IVR logs Vectoring Customer Call #’s (800- and 888#’s; Call Coordinators; “Must answer” lines redirected Agents to work problems (skills, training, accommodations) Work stations/tools (includes data scopes for IE) Applications/Systems- Lotus Notes; CSS; Web tools/servers; IP Administrators; etc. Critical customer support (list) — XNS Corp, IPS Inc., banking, etc. Internet support as backup Communications (pagers, cellular phones) Customer billing — Visa, MC, remote authorization Customer fulfillment (shipping) Physical connectivity, LAN WAN — verify network plan Command centers — access to main office lines redirected IE support — outage < 2 days require support; alternate call center > 2 days outage

© 1998 by CRC Press LLC

Company Secondary

Location 1 St. Catharines

Location 2 Columbus

the incident manager’s instructions to handle the situation. The incident manager would also require the capability of communicating with members of directly impacted departments to find out about their situation and emergency requirements. Every disaster is unique. The maximum information available would be needed so that the incident manager would be able to determine the best response, in the quickest possible time. The incident manager (and team support personnel) would provide passive communications for information purposes for the duration of the disaster. Upon enactment of the continuity plan, the incident manager would assign a team member to make a verbal situation report on a dedicated interactive voice response (IVR) hotline. This would need preplanning, since if communications at the original (“disaster”) site were knocked out, the hotline would be part of the communications package handed over to the ICH facility. The hotline should probably be a toll-free line capable of being switched anywhere (see Preprogrammed Switching) to give maximum flexibility in case of wide-area disaster. This hotline IVR unit would provide information on the current status of the disaster to • Management from nonimpacted departments. • Employees directly impacted by the disaster. • Departments impacted indirectly (such as Accounting being unable to receive credit card payments by telephone if the call center were inoperative). • Major customers or vendors included in the planning project.

This information should be updated at least every eight hours during the crisis. The hotline would be important for maintaining morale and ensuring that personnel knew when essential measures had been performed. The incident manager (and alternates) would require a dedicated cellular telephone with office and mobile battery-charging units, to ensure that the incident manager would have communications in most circumstances, and even if going mobile to inspect site damage or structural integrity of personnel shelters.

Emergency Operations Center There are two types of EOCs for use by the incident manager. The on-site EOC would be an office set aside for crisis management (and no other purpose) and might be colocated with an emergency call center, if your company decided to construct one. The characteristics of such a subfacility would be that it was provided with electrical generators, food, water, and redundancy with the PBX (dedicated extra telephone lines), and no glass would be used in windows, wall panels, or as office dividers. The facility would be hardened and fortified, a strong concrete building able to withstand high winds. As noted above, the EOC would include room set aside for cots, pillows, and blankets for off-shift personnel to sleep in during the disaster. The EOC and emergency call center would be more likely to be fully staffed during a disaster of long duration if provision were also made for the safe accommodation of family and pets. There should be a list of employee volunteers willing to relocate temporarily to support the disaster recovery efforts. If family members were ensured a glass-free shelter, for instance, they would be less likely to leave to evacuate their families out of the area in case a hurricane were expected. If children and pets were to be allowed accommodation in the EOC, canned milk and pet food would have to be included in the EOC supply inventory. The virtual EOC, by utilizing the ICH facility as a centralized relay point, and exploiting the mobile communications ability provided by one or more cellular telephones, could be located anywhere. The virtual EOC could be quickly activated, and would allow the incident

© 1998 by CRC Press LLC

manager to direct the teams from outside the affected area. For example, if a chemical plant upwind from your company facility were spewing toxic smoke toward your facility, having a virtual EOC would allow the incident manager to assess the situation from a safe vantage point.1

Authority The incident manager must have the authority to act decisively and swiftly in enacting the continuity plan. The CEO, and other senior managers, should be patient and call into the crisis hotline to receive situation updates from the IVR unit. Scheduling briefing sessions at intervals would be okay so long as they do not interfere with essential actions being taken, and no interference or second-guessing is allowed to distract the incident manager. Any “second-guessing” — done with 20/20 hindsight — should take place during the plan assessment stage that follows successful recovery from the disaster. Any journalists that might drop into get quotations about the situation should be directed to the public relations team representative. Scheduled press conferences are a good idea; however, the public relations representative (or alternate) must receive clearance for the wording of the press release before giving it to media representatives (see Chapter 22). Interested employees should call into the hotline IVR for situation updates, rather than crowding around the incident manager and getting in the way. The incident manager and alternates absolutely should not have to worry about their normal departments, each of which should have its own separate primary and alternate team representatives.

What the Incident Manager Does Not Do Heroic Rescues. In the real world, in contrast to fictional dramas, the incident manager and alternates should not take it upon themselves to rescue anyone from a burning facility. They should never drive through roadblocks set up by emergency authorities without permission. They should never risk their lives or safety no matter how precious the person or document or piece of equipment that might be left inside a collapsing building, a malfunctioning elevator, or other position of jeopardy. It is the job of the security personnel to assist safely anyone who has already left a fire scene or other dangerous location. (And the limits of their actions should be clearly defined in the security provisions of the plan.) However, real rescues should always be left to the firefighters, police officers, and paramedics, who are trained to respond to danger and have the necessary equipment to do so in a prudent and judicious manner. The sole exception to this rule might be if your company had a trained firefighter/paramedic on staff, and because of wide-area disaster no help from the authorities could be expected for hours or days. Even so, only the most prudent measures should be allowed, and only those not requiring special breathing apparatus or other protective equipment. The incident manager’s value to the company is in remaining a calm, clear-thinking director of the company’s emergency response measures. This requires that the Incident Manager remain in a safe location, and watch the fire or other ongoing disaster through binoculars or by remote camera. If the Incident Manager or other team personnel endangered themselves, no Oscar would be given for “Best Dramatic Performance.” Saving the World. In case of a wide-area disaster, community outreach is certainly important, but the incident manager would have to remember that the primary mission is

© 1998 by CRC Press LLC

to ensure the continuity of the company. The incident manager should note that outreach to the community should be limited in some important ways: • Company critical functions must be protected first. • Company excess capacity only (if any) could be provided for shelters, etc. • Confiscation of resources by government authority should be a passive, not an active, provision of equipment or supplies by your company.

Little good would be served if your company (an important employer of community members) should fail because necessary resources were donated instead of using them to ensure the continuity of your employees’ jobs. That would be like robbing tomorrow to pay today. Although people might appreciate the gesture, no one could give a plaque or award to a company that no longer survived.

DEPARTMENT MANAGERS No Bureaucratic Sign-Off Department heads are a necessary part of the planning process whether or not they fill a team position. They are the ones who will update operational information in their departmental part of the plan binder. They are the ones who will approve changes to checklists before they are distributed to the relevant parties. You, the planner/planning manager, should impress upon these managers that merely signing or initialing a document every week or month is not sufficient. The department heads must be fully involved in the planning, testing, and updating of the continuity plan.

Full Support Essential Some department managers (who are not part of the recovery team in any capacity) will have subordinates who are part of the team. Team members must be allowed time to attend planning meetings and time to investigate operational procedures and draft the necessary plans and checklists. Team members should not be overloaded with other work, or criticized for putting the continuity project first. Department heads, especially the CFO or senior accounting representative, must not shunt aside the continuity-planning process and the measures that it requires. It is not satisfactory to miss planning meetings or document preparation deadlines because of conflicting meeting times. Except for preparing the company year-end reports and tax returns, few tasks could take priority over the continuity-planning imperative. In the event of disaster, even those pressing responsibilities would have to be put aside.

Keep Subordinates Informed It is the responsibility of department heads to inform people in their departments of announced test dates, operational change dates, or other continuity project–related information. In the event of disaster, it is the responsibility of department heads to call into the crisis hotline and access the IVR situation reports to keep up-to-date with the situation as it develops. Although the recovery team will be enacting the plan as written, if requested, department heads should assist in spreading situation report updates to the employees who have not heard them. This is particularly important during a serious disaster, when employees might need to know about shelter locations, emergency banking, or where to get food, water, and blankets.

© 1998 by CRC Press LLC

Keep Departmental Operations Information Updated It cannot be emphasized enough: operational change information must be put promptly into the plan document and all relevant checklists. This is not something that can be done once a year. This is not something that could be added to the far end of an assistant’s task list.

Updating Departmental Checklists The department head should ensure that all work stations are supplied with updated checklists. These checklists should be kept posted (preferably at eye level for a seated person) where they are easily found and can be used immediately when needed. Suppose that a person new in a department answered the telephone and a caller made a bomb threat. If the checklist were not beside the telephone, vital seconds and information could be lost. The checklists for each department should reflect the continuity plan provisions for that department and the recovery team structure for that department.

NOTES 1. The Hagarsville Tire Fire, in Ontario, Canada burned for weeks, and caused the population to evacuate from a wide area.

© 1998 by CRC Press LLC

9 Plan Contents MISSION STATEMENT The mission statement must be a clear statement of the intention of recovering your call center and ensuring continuity of its operations.

SCOPE The mission statement should encompass the intent and commitment of the organization to ensure continuity of the business operation. It should define what business operations are going to be covered.

COMMITMENT The mission statement should make explicit the commitment of the management to preserving the jobs of the employees and protecting the interests of customers of the organization.

RESPONSIBILITY The mission statement should establish responsibilities for call types, plan development, activation of the plan, and maintenance of the plan.

TIME FRAME The mission statement should also include a time frame for the plan to be completed. It should conclude by restating the desire of the company to continue operations and to protect and safeguard the assets of the company. It is important to note that the business continuity plan is not a book. The plan itself should be in a checklist fashion or bullet form as much as possible. The mission statement should be the only part of the plan that is be described at length.

Not a Career The writing of the mission statement should not be regarded as the prelude to writing an advertising brochure extolling the merits of the company. The mission statement should not be worded so precisely as to merit bronzing and display upon the boardroom wall. No one should take large amounts of time to decide where exactly is the best place to put a comma or period. Consider the sayings, “Brevity is the soul of wit,” and “Time is of the essence.” It could be said, “Conciseness is the heart of a mission statement.” No one should labor over drafting the perfect phrasing.

© 1998 by CRC Press LLC

EXHIBIT 9-1. PLAN FLOW Short-duration outage

No recovery plan activation required

Return to normal operations

Short evacuation required

Evacuation of call center

Call handover

Return calls

Return to normal operations

Moderate damage to call center or infrastructure

Evacuation of call center

Call handover

Activate hot site

Return calls

Return to normal operations

Severe damage to call center or infrastructure

Evacuation of call center

Call handover

Activate hot site

Activate cold site

Return calls to new site

Return to normal operations

State the Obvious, Then Start Planning! When it is time to activate the plan, the last thing you want to have people do is to read large sections of text. Checklists make a plan easier to follow. A person following a checklist can simply glance down a page and come to the topics that are most critical to that individual. In the mission statement you state the obvious and then start planning.

PLANNING PROCESS In the planning process, form follows function. If you have not completed Exhibit 5-2, go back and finish it now. That chart shows all the stages of the planning process. In case you do not recall the chart from Chapter 7, it is presented again as Exhibit 9-1. This chart shows the requirements according to the amount of damage that could occur. If you have not completed Exhibits 7-6, 7-7, 7-8, and 7-9, go back and do so now. The stages of the planning process should keep you from missing anything. The requirements show when you need to use the items that you are planning.

DEFINITION OF DISASTER “All Events Are Potential Disasters” All serious events are potential disasters. We have seen problems where there were leaks of water in the room with the ACD unit. If such a leak had gone unnoticed for a half-hour, then there could have been a serious disaster. We have seen air-conditioning units freeze right up and create a pool of water around the UPS (uninterruptable power supply). You could imagine what would have happened if those two (high-voltage electricity and highly conductive water) had made contact. That was a serious potential disaster. The UPS touching that water (and exploding) would have taken the call center out of operation for a significant period of time.

© 1998 by CRC Press LLC

Definition of a Disaster A disaster is any man-made, human-error, or natural event that disables or interrupts the ability to maintain business operations for a period of time, which results in an adverse impact to the organization, and which leads to loss of customers, loss of revenues, or loss of reputation.

Many Disasters Start out as a “Bad Day” Many disasters just start out as a bad day. Consider the UPS problem with the water that we talked about earlier. If it had turned out to be a disaster, the sequence would have been something like the following. The janitor would have noticed the water around the UPS, and, of course, consequently maintenance people would be trying to act. If the water had made a circuit with the UPS, then the batteries could have blown up. This could have been a disaster with people seriously injured or killed.

“Any Event That Adversely Impacts” It would all have started out as a bad day. What if people had been called into deal with the leak in the air conditioner? The situation could have easily escalated to a major disaster with possible the loss of life. Fortunately, that did not happen. An explosion of the UPS would have cut off communications for the call center, might have cut off power to the computer room, and could have damaged nearby offices in a 100-foot radius. If a fire had started, a whole building full of people could have been at risk.

TIME FRAMES RTO That Fits Your Company The recovery time objective (RTO) must meet the needs of your company. The purpose for the exercise in determining how much money would be lost over a given time period is to assist in setting up the RTO. Of course, with call centers, time frames for recovery are often more crucial than with other types of recovery. This is because the customer calling into your company wants to have the telephone call answered. A customer in California does not particularly care that the call center is buried in six feet of snow or on fire in some other region.

Rationale The rationale for the time frame needs some basis. It has to fit the amount of money that your company is willing to spend to have a proper continuity/disaster recovery plan. It has to take into account the amount of money that your company stands to lose if it does not have a plan. If you did not complete Exhibit 6-22, go back and do it now.

TEAM RESPONSIBILITIES Checklists The teams should be responsible for creating checklists in their own areas of operation. We cover this in Chapter 18.

© 1998 by CRC Press LLC

EXHIBIT 9-2. TYPES OF CHECKLISTS These are some examples of checklists for different positions. You will notice that the checklists become more complex at higher levels in the organization. CSR Checklists

Supervisor Checklists

Call Center Manager Checklists

Fire Tornado Bomb Threat Wide-Area Evacuation Contact Information — e.g., 1-800-Contact

Fire Tornado Bomb Threat Wide-Area Evacuation Contact Information — e.g., 1-800-Contact [same as CSR list with more details] Hot Site Relocation Procedures Cold Site Relocation Procedures

Fire Tornado Bomb Threat Wide-Area Evacuation Contact Information — e.g., 1-800-Contact Hot Site Relocation Procedures Cold Site Relocation Procedures [same as supervisor list with more details] 800 rerouting procedures Interim Call Handling RTMCO Facility Procedures HVCA Facility Procedures Distributed Architecture Activation

For example, suppose that a company had a water-cooled computer and experienced a hurricane. The water-cooled computer would have to be shut off in a specific sequence. Suppose that, unfortunately, the person who would normally do this had already left town. If there had been a checklist, then anyone from the CEO to the janitor could have shut the computer down properly. There is, predictably, some sort of equipment in a department that needs to be shut off using a specific sequence. Perhaps only a few people in the company know how to do it. It is the responsibility of the people in that department to create the checklist in layman’s terms, so that the checklist could be followed by anyone.

Deputies It is a good idea to assign a primary person and a deputy (or alternate) who will be responsible for implementing all checklists. The deputy is there in case the primary person is not available to do the job, and the checklists are there in case both people are unavailable to do the job. In Exhibit 9-2, you will see a list of possible checklists.

CONTACT INFORMATION Active The plan should include some sort of active list of contact personnel with telephone numbers. This list will change, so the best thing to do is to create if on some sort of database together with the rest of the plan. We recommend Lotus Notes or some sort of word-processing file. The contact information should be kept current. If possible, an alternate telephone number should be included just in case there might be difficulty in reaching someone in urgent circumstances.

© 1998 by CRC Press LLC

Passive IVR Situation Reports Contact information can be kept accurate and up-to-date with an IVR situation report. This is a telephone number that your personnel can call into, which records information that is needed, such as their location, condition, and information on their contact capability during a disaster. If there is a wide-area disaster, this is very important. However, even during a localized disaster (such as the building being on fire), it is important that the incident manager know where people are. People can call in and report to a single telephone number, and be given information on what is happening and where they are instructed to go. In a high-volume call-answering situation, where hundreds or thousands of calls are being received at once, this could be very useful. Suppose that an employee comes to work and the building is burning down. That employee’s first reaction could be to panic, but the person knows that there is a toll-free number (1-800 #) in place to receive an emergency call. When that employee calls in, the IVR transmits the incident manager’s update on the situation and where the regroup facility is located. In case the employee might not know how to reach the gathering point, the IVR gives directions to it, as well as a time to meet there for all employees. Employees can then enter in their ID number or personal password, and then management will know who has called in. The passive IVR hotline is also useful for receiving recovery team situation reports to the incident manager. If your company has a predictive dialer, then the dialer can call up all of the company employees, inform them of the situation, and give instructions before they call in, provided, however, that the dialer itself is not destroyed by the diaster. If it is destroyed or your company does not have one, then there are companies out in the call center marketplace that can be hired to do this specific task. The planning manager (you) and the planning team have to remember to write these options into the plan in order for employment of these communication methods to become possible.

Reports to Incident Manager Plan provisions and checklists for all departments should describe the procedure for reporting to the incident manager. Checklists should include these reports as a task to be done at regular intervals throughout the disaster, and especially if something significant has occurred.

Up-to-Date All information needs to be kept up-to-date, constantly. There should be a clerical staff person, and preferably an alternate as well, whose responsibility, as part of the job description, is to keep the contact information up-to-date in the disaster recovery plan.

ESCALATION PLAN Levels of Response One of the principles of planning for disaster recovery is that every incident should be handled in the same fashion. Now at first this seems to be a contradiction, in that you cannot predict what will happen in a disaster. Nonetheless, you can predict your responses to a disaster. We could categorize the types of disasters, for instance, by putting a tornado warning and a fire alarm in the same category. You will see as you go through the planning process

© 1998 by CRC Press LLC

that the escalation to deal with these disaster increases. You will write your plan to provide for scalable levels of response. You will see in Exhibit 9-3 the same details that we have presented before, except that we have created a chart that shows the different levels where a particular response is required. For instance, suppose that the company recovery time objective is 30 minutes. Then the initial part of the scale of, say, a fire drill that would last 15 minutes would not be activated unless it looked like the disaster was going to exceed 30 minutes. If it did seem likely to last longer that 30 minutes, then the next level would be activated which could be the interim call handling (ICH) or the real-time minimum call overflow (RTMO). The recovery time objective in this case is more than 30 minutes and less than one hour. So when a small-scale disaster occurs, the plan is to wait to see if it will become a largerscale disaster. The ICH or RTMO action in this scenario is for more than one hour of outage but less than six hours. If the call center is going to be out for more than six hours, the hot site must be activated. Most hot sites only allow the client call center to be there for 45 days. If the outage is expected to be longer than 45 days, the company must activate an alternative facility by that time.

Declaring an Emergency What procedure should be followed when declaring an emergency? The best thing for your people to do is to have a look at what the situation is and compare it with Exhibit 7-2, the risk assessment matrix. This should be a part of your checklists. With this table, personnel will know when to declare an emergency. This gives you the capability of making all of your plan measures the same. However, you can scale them on higher levels depending on expected disaster lengths and arrivals.

Scalable: Same Procedures at Higher Pitch As shown in Exhibit 9-3, plan measures to recover the call center operations to an ICH facility and provide for critical business functions are the same. The key variables are duration and damage.

CRITICAL DOCUMENTATION Files (Letters of Agreement, Contracts) and Databases Critical documentation should be a checklist item. The checklist should include critical files, letters of agreement, contracts, and significant legal agreements. Information should be given concerning where the original and duplicates are stored. Important: Do not store these items in the call center. Store them in two separate off sitelocations. These documents will force your vendors to stick to the prearranged service-level agreement and get you the equipment and technical assistance you need to recover your call center. What if your call center should burn down? What if the contracts had been on site? And if no one at a vendor company would give you any help, then you would have no document to quote. Off-site storage means never having to ask the vendor what was set in the agreement. Critical documentation should also include a list of where your database backups are held. It should include all information about when your company has scheduled them to be updated and what procedures are in place to get your databases to your backup facility. Safety deposit boxes can make excellent storage locations for contracts, service-level agreements, and other critical documentation. Originals should be kept there or in a protected archive facility and duplicates should be at the call center location, in a “ready to go” case. If using a bank for storage of important documentation, be aware of its hours of operation.

© 1998 by CRC Press LLC

EXHIBIT 9-3. SCALABLE LEVELS OF RESPONSE Category

Decision

Shortduration outage 5 Determine if fire is real

Short evacuation required

8 Determine size of fire

Moderate damage to call center or infrastructure 11 Assess damage after fire is extinguished

Severe damage to call center or IR infrastructure

Actions 1 Fire alarm

2 Take “to go” case and evacuate

3 Meet in designated area 1

6 If no, then

7 No recovery plan activation required

8 Return to normal operations

6 If yes, then

7 Call handover to ICH or RTMO facilities

9 If small fire with no damage to call center, then

10 Return to call center; begin clean up

9 If moderate to large fire, then

10 Activate hot site

12 If damage is moderate and will be able to return in 45 days or less, then 12 Activate cold site or alternate location

4 Do a head count

11 Return calls

12 Return to normal operations

13 Complete repairs as necessary

14 Return calls

15 Return to normal operations

13 Activate drop-ship provisions from vendors

14 Set up new facility

15 Return calls to new site

16 Return to normal operations

Vendor Contact List The same person who is responsible for contact updates should update the vendor contact lists. This would be a checklist giving company name, telephone numbers, and representative names for different vendors.

Equipment and Software Manuals Your plan should provide for equipment and software manual emergency copies. The IS department needs to keep a percentage of these manuals in “ready to go” cases (see

© 1998 by CRC Press LLC

EXHIBIT 9-4. CONTENTS OF INCIDENT MANAGER “READY TO GO” CASE One satellite or cellular telephone with battery-charging equipment. Maps to hot site(s) or other emergency facilities Checklists (risk type, team tasks, call/recall, inventories, etc.) Emergency communication contact list The complete disaster plan Flashlight Advanced first aid kit Summary of emergency vendor agreements Company Visa Card with minimum $10,000 limit Cash: minimum $500 Optional laptop computer with battery and power cord and fax modem and software Note: All batteries should be tested periodically.

Exhibit 9-4) so that if your call center needs to pack up and go to another facility, it can do so rapidly. These cases can be grabbed and taken along easily even if personnel are evacuating swiftly.

CRITICAL FUNCTIONS Consider Exhibit 6-3 (Call Flow by Call Type). This identifies the critical functions that are required to handle each different call type. We hope that now you have created a call flow for each and every call type that you have. We suggest that you next take the table in Exhibit 9-5, which shows sales, fulfillment, accounts receivable, credit card payments, and other functions. Chart out which are required for the different call types. Then you can chart out where the alternate location will be for each of these different functions. The information to build this chart comes from the information that you generated on each call type in Exhibit 6-3. This illustrates the critical functions that would restore most applications. Out of the list of critical applications in Exhibit 9-5, create a list of requirements for each application in Exhibit 9-6, to restore each critical application to functionality if destroyed.

OTHER CRICTICAL TASKS In Exhibit 9-7, we look at the different tasks that are performed in departments other than the call center. This section is designed to determine tasks and equipment critical to the ongoing function of each “other than call-center department” and to develop checklists that minimize loss in case of a disaster. It would be difficult to write checklists to ensure that everything is taken care of if you do not know what that is.

© 1998 by CRC Press LLC

EXHIBIT 9-5. CRITICAL FUNCTIONS (ANNOTATED) Call Type Critical Functions

Sales Call

Tech Ass’t.

Product Location

Outbound

AT&T long distance

Yes

Yes

Yes

Yes

Bell Mega Link

Yes

Yes

Yes

Yes

Local service

Yes

IVR

Yes

Yes

Yes

CSR general trained

Yes

CSR help desk Supervisor

Yes

Yes

Yes

Yes

Notes

Yes Yes

Yes

Predictive dialer

Yes

CTI

Yes

Yes

Screen pop

Yes

Yes

Server database

Yes

Yes

Access customer account info

Yes

Yes

Accounting billing #1

Yes

Yes

Credit card

Yes

Yes

Link to card center for approval

Yes

Accounting billing #2

Yes

Accounts receivable

Yes

Check bad debt file

Yes Yes

E-mail

Yes Yes

Shipping (fulfillment)

Yes

Customer follow-up

Yes

Yes Yes

Yes

Word processing

Fax server

Yes

Yes

Yes

Yes

Yes Yes

Yes

Yes

Although an enterprise-wide recovery program is beyond the scope of this book, it is important that the planner ensure that no important related sections are missed.

Sales If all the sales were done by the call center, then we would need to look at where the alternate call center is to be located. (In Chapter 15 we discuss call volume management and how it is done.)

© 1998 by CRC Press LLC

EXHIBIT 9-6. CRITICAL APPLICATION REQUIREMENTS Critical Application

Requirement 1

Requirement 2

Alternate Location

Expected Result

EXHIBIT 9-7. OTHER CRITICAL EQUIPMENT

Tasks

Frequency

Positions

Equipment, Forms, and Supplies

Software and Documents

Facilities

• Tasks: State the tasks that are performed, rated from highest priority to lowest. Please consider all functions including those provided for other departments, particularly the call center. • Positions: List the minimum staff (title and number) that is required to perform the tasks and any related fulfillment. • Equipment, forms, and supplies: Minimum equipment, forms, and supplies necessary to perform the tasks. Please ensure that you include all items necessary; e.g., Rolodex contacts file. • Software and documents: Minimum software and documentation necessary to perform the tasks; e.g., books, reference manuals, checklists, etc. • Facilities: Minimum facilities necessary to operate at the backup call center.

Fulfillment For the fulfillment aspects, you would look at where the fulfillment was being done. If it were in the same location as the call center, then you would need to establish an alternate fulfillment center. You would need to ensure that the fulfillment center was relocated at the same time as the call center was relocated. If your fulfillment center were separate from your call center, then you would need to have disaster recovery plans for both centers.

Accounts Receivable, Credit Card Payments The postprocessing path that shows the account billing function (whether by accounts receivable invoice or credit card is not critical) that shows up on the call-type chart. You would need to have alternate billing facilities available in the event of a disaster. There are plenty of companies that provide these services; however, this would probably be best handled by your ICH facility. This is because most ICH facilities are set up to accommodate billing requirements.

© 1998 by CRC Press LLC

Public Relations A public relations team should handle your interactions with the public. This would be particularly vital in case of a disaster that involved only your company, such as a fire that involved a loss of life. Your planning team public relations representative should ensure that each type of incident that has been identified in the escalation plan has a prescripted response for the public relations team or incident manager to use. All that the personnel handling the disaster would need to do is simply to fill in the details. This would give your company an opportunity to step back from the situation and say what needs to be said rather than doing a rush job. A standard form gives you this time. This also would stop people from saying things that they should not, especially when they might be exhausted or overcome with emotion. All employees should be counseled on the name and phone number of the public relations department (should be part of their checklist) in case they might be asked to comment on a situation by a journalist.

Business Communications and Purchasing Other critical functions include business communications, customer service, and purchasing. These areas of the business might have a longer RTO than the rest of the call center. These RTO might be a day to two days after your more critical business functions. Your call flow by call type chart will indicate which calls need to be recovered immediately and which calls can have later recovery times.

CRITICAL APPLICATIONS Call Center Computer–Telephony Integration These were first addressed in Exhibit 6-3. The ACD and the predictive dialer should all be looked at as part of the processing path. When you are planning ICH for the call types, look at the simplest processing paths that are required. Because you have already identified the call types, you can look at what you need to create to enable the alternate facilities to handle the calls.

Customer/Sales Database The customer sales database is another path that you should identify. It is possible that you will not need to have your customer sales database updated as frequently during a disaster as you would normally. If you can, try to make do with the database being updated only once a day.

Accounting Software The accounting software goes back to the processing path that you identified in Exhibit 6-3. The software that you have should be able to be replicated at a new facility depending on the level of response required. Your planning team accounting representative should ensure that provision is made to ensure that the ICH facility already has the same accounting software (and version) as your company. If not, your company should supply the correct software, and the planning team IS representative should ensure that the ICH facility has loaded and tested the software successfully at that site. In addition, a “last” or “current” backup (as available) of company

© 1998 by CRC Press LLC

accounting data should be loaded at the ICH facility as an interim measure until the plan is fully written and tested. If you and your team plan for a outsource service bureau to be contracted instead of using the ICH facility, the same provisions apply. If a company branch office is planned for use, your IS planning team representative should ensure that the branch office would have sufficient data-processing capacity to handle the accounting functions of your facility. If not, the CFO should immediately allocate funds and authorize purchasing of equipment to handle that essential function. This is where in the escalation plan you would look at the level of response required before making the decision to relocate the accounting department functions. The incident manager’s difficulty in deciding what level of response to employ is this: it is easy to make a decision if the building is wiped out. However, if the disruption is less severe than that, sometimes the decision is not as straightforward. This is another reason to chart out the levels of response in advance.

Word Processing and E-Mail Your planning team IS representative should ensure that the ICH facility, branch office, or service bureau that is planned to provide administrative functions has compatible wordprocessing and e-mail programs. If the assisting facility does not, sufficient capacity must be arranged. Word processing and e-mail can usually be relocated with little difficulty. Most programs are compatible with each other. If your company (as a continuity project policy) decides to use off-the-shelf software and hardware, then any computer store would be able to provide the software that would be required.

CALL VOLUME MANAGEMENT PLAN Key to Call Center Continuity Your planning team, especially the call center manager and other call center representatives (in cooperation with the CFO or other accounting department representative) should examine the options, speed of enactment, company “survivability,” and costs when drafting this part of the plan. This topic is discussed in more detail in Chapter 15. Proceed to Chapter 15 at this point and review the call volume management section. The remainder of this chapter will then be easier to complete.

RECOVERY SITE INVENTORY Exhibit 9-8 shows a sample recovery site inventory for a hot site. In addition, see Exhibit 9-4, which shows the contents of a “ready to go” case. As shown, some of the essential contents are cellular telephone, essential documents, and the complete disaster plan.

BACKUP/RECOVERY PROCESS When data from the call center and other departments is backed up, it can be sent to an assisting call center (ICH facility) if necessary to “recover” critical functions in case of disruption or disaster at the the original company facility. See Exhibit 9-9.

© 1998 by CRC Press LLC

EXHIBIT 9-8. RECOVERY SITE INVENTORY Date: _____ Time: _____

Recovery Site Inventory

Problems

Initials

Checklists — hot site recovery procedures Power and lights Heat and plumbing Telephone lines operational Computer data lines operational Computers checked and functioning Backup data tapes arrive on site Supplies checked Personnel contact/recall lists Critical business functions (applications) Interim call center online Assisting call center online (overflow, expert calls, telecommuters) Communications established with disaster site Vital records recovered from storage (e.g., service-level agreements) Vendors notified Food, water, sleeping accommodation arranged Additional items

IMPLEMENTATION PLAN Assessment and Declaration of an Emergency Personnel discovering the disrupting event should immediately refer to their checklists. These checklists should set out how to set the recovery portion of the continuity plan into motion. Following plan procedures would allow the recovery team to alert all departments and enact the appropriate measures to safeguard personnel and company assets. See Exhibit 9-10.

© 1998 by CRC Press LLC

EXHIBIT 9-9. BACK-UP AND RECOVERY ACTIONS Action

Consequence

Routine backups

Can recover data to last backup period

Special backups on declaration of emergency

Can recover data up to minute of declaration

Alerting assisting call center(s)

ICH facility stands ready for prerouted call switching

Prerouted switching of calls to assisting call center and/or telecommuters

Telco initiates call switching on receipt of coded password

Sending current backup information to hot site

CSRs standing by get data current up to minute of backup

By data line

Data arrives in seconds or minutes

By courier

Data arrives in hours

When emergency prevents current backup

Must use last backup data

Opening last backup at warm/hot site

Customer and other data known only to last backup period

Last backup with the verbal comments of your CSR

Last transactions not backed up might be replicable from remembered details

EXHIBIT 9-10. IMPLEMENTATION TASKS Action

Planning Required

Informing all departments

Contact lists, checklists

Checklists

Departmental personnel knowledge of procedures

Backups

Practice of backup procedures with checklists

Role of the recovery team

Plan for continuity of all critical functions

Informing emergency organizations

Contact lists, liaison representative, checklists

Handover of calls

Arrange Telco prerouted switching

Evacuation if necessary

Evacuation checklist, prearranged meeting place

COMMAND CENTER When a disaster is declared and the continuity disaster recovery provisions are enacted, the incident manager takes control of all response measures by the recovery team. The emergency operations center (EOC), or command center, is described in detail in Chapter 8.

© 1998 by CRC Press LLC

EXHIBIT 9-11. ALTERNATE POSITION LISTING

Position

Primary Person

Telephone

Alternate Person

Telephone

Alternate Person

Telephone

Alternate Personnel List The purpose of Exhibit 9-11 is to identify the primary and alternate personnel responsible for, or able to perform, critical duties (that are not already included with the recovery team lists) within the organization. List as many alternates as possible.

External Contact List The purpose of Exhibit 9-12 is to provide telephone numbers and contact names for organizations and vendors that are critical to the recovery process, such as police, fire, ambulance (911 + additional numbers), disaster service centers, public relations consultants, personnel agencies, insurance companies, and utility companies as well as a master list of vendors. If your plan includes using a computer-based contact list, you do not have to fill this form out. Just make sure you have a printout of your list in the “to go” case. The printout should show the file name and last change date.

© 1998 by CRC Press LLC

EXHIBIT 9-12. EXTERNAL CONTACT LIST

Name of Organization FEMA/EPA

Police Department — 911

Ambulance — 911

Fire Department — 911

Hospitals

National Guard

Post Office

Disaster Service Centers*

Public Relations Consultants

© 1998 by CRC Press LLC

Contact Person/Position

Telephone Numbers (Main/Cellular/Pager)

EXHIBIT 9-12. EXTERNAL CONTACT LIST (continued)

Name of Organization

Contact Person/Position

Television

Legal Assistance

Personnel Agencies

Utility Companies

Vendors

Security

Major Account Customers

* For example, see IBM BRS at end of Chapter 9.

© 1998 by CRC Press LLC

Telephone Numbers (Main/Cellular/Pager)

Inventory of Critical Documentation. The purpose of Exhibit 9-13 is to maintain a current inventory, by department, of Service-level agreements, contracts, checklists, recovery plan copies, computer manual(s), software manual(s), application software documentation, accounting procedures, policy/procedure manuals, insurance polices, and other documentation; Storage location of documentation — off site, on site, building, address, hours of operation, etc.; Method of replacement: backup copies stored on disk, secured in a bank, obtained from branch offices, available from vendor, etc.

Inventory of Equipment The purpose of Exhibit 9-14 is to maintain a current inventory by department of all critical equipment used throughout the call center. Information required includes the vendor of equipment, description of equipment, replacement time, and model.

Inventory of Forms Not a lot of companies use preprinted forms today. However, if you are working at one of those companies, you need to back up your forms. Exhibit 9-15 is intended to list forms used by the call center to ensure a timely replacement. Information needed includes form number, type of form, supplier(s), replacement time, and off-site storage locations.

Distribution Register The distribution register in Exhibit 9-16 is an inventory of plan binders and checklists. The register should show every department and work station, and record every time that the team updates the plan and checklists. These entries should show a plan version number, date of the update, and the initials of the person making the change. The distribution register is important for keeping track of all plan binders and checklists. The recovery team would use the register to ensure that someone in fact does update all the binders and checklists, and none is forgotten. Even if someone is on vacation when a change is made, his or her binder or checklists would still be amended. This would prevent anyone from using an obsolete checklist and performing wrong, dangerous, or out-of-date responses. Using the register would also prevent checklists for vacant work stations from being forgotten at update time.

IBM Business Recovery Services The following additional services are designed to overcome specific problems. The IBM BRS information is a useful list. Your company could contract with IBM or another outsourcer for the described services. Or you could plan for the company to locate its own vendors. Either way, the information above would provide a good starting point for planning. As stated, they are all provided by IBM BRS. However, there are other companies that can carry out the tasks stated in each category also.

© 1998 by CRC Press LLC

EXHIBIT 9-13. CRITICAL DOCUMENTATION INVENTORY Storage Locations Service-Level Agreements

Contracts

Recovery Plan Copies

Computer Manual(s)

Operating Systems Software Manual(s)

Software Manual(s)

© 1998 by CRC Press LLC

Method of Replacement

EXHIBIT 9-13. CRITICAL DOCUMENTATION INVENTORY (continued) Storage Locations Application Software Manual(s)

Disaster Plan Standards Documentation

Accounting, ATM Cards

Accounting Procedures

Policy/Procedure Manuals

Recovery Plan Copies

Other Documentation

© 1998 by CRC Press LLC

Method of Replacement

EXHIBIT 9-14. EQUIPMENT INVENTORY Type of Assistance Electrical power Telephony Information systems instrumentation Computer hardware Computer software First aid equipment Emergency food Emergency shelter Portable water Personal care items Waste control Heating/cooling Secure document storage local Secure document storage distant Magnetic media replacement Backups storage off-site local Backups storage off-site distant Backups storage on-site Banking

© 1998 by CRC Press LLC

Vendor

Description

Replacement Time

Model

EXHIBIT 9-15. INVENTORY OF FORMS

Form Number

© 1998 by CRC Press LLC

Type

Supplier(s)/ Telephone

Replacement Time

Off-Site Storage

Notes

EXHIBIT 9-16. DISTRIBUTION REGISTER Department

________________________________________

Plan version # ________________________________________ Update number ________________________________________

Station

© 1998 by CRC Press LLC

Checklist Number

Date

Name of Person Inserting Update

Initials

Crisis Response Team1 When a disaster strikes, you just call the Crisis Response Team 800 number. A certified team of experts — available 24 hours a day, 365 days a year — will provide you with initial risk and damage assessments. They coordinate and manage all recovery-related issues and report back to you. If there is a regional disaster, our Crisis Command Center may even contact you — before you call us — to get your recovery efforts under way quickly. As part of the client’s Business Resumption Services subscription, the Crisis Response Team will focus on providing the client with the following services, depending on the client’s situation: • Assess damage and risk • Coordinate site recovery resources • Establish communication and documentation procedures with client executive management team • Manage and solve all recovery conflicts • Schedule and conduct crisis status meetings • Provide expertise on industry related regulations • Solve leasing disputes associated with moving from a leased building • Identify insurance issues • Develop business cases for insurance • Assist in insurance claims validation

Relocation Services* If the facility is so severely damaged that it is declared unsafe, we will assist the client in locating an alternate site. We can review available IBM sites, and use our relationships with commercial real estate firms to help you. Once you make your selection, we can provide timely site fit-up, packing, moving and storage services.

Construction Services* In the event that the clients’ facility is significantly damaged or destroyed, we can also provide construction services to assistance in repairs to existing facilities, or build new ones.

Acquisition Services* IBM will acquire and provide replacement PCs, displays, network services, IPS units, furniture, fixtures, telephones and other equipment as specified by you. We have contracts for a variety of equipment needs through a wide range of vendors. A low-cost retainer fee allows us to have access to the type of equipment you will need when disaster strikes — equipment that is typically in short supply in a disaster’s aftermath.

1

The marked terms are trademarks or registered trademarks of IBM Corporation in the United States and other countries. The quoted text is from IBM published advertising material.

© 1998 by CRC Press LLC

Technological and Environmental Cleaning Services* An important part of our Business Resumption Service includes restorative CPU Cleaning, maintenance and recovery as well as mitigating any building moisture, corrosion and humidity problems. Our specialists will assess the damage and make recommendations.

Press and Media Management* IBM can provide solutions for managing news and media rotations, plus strategies for communicating with your employees, customers, stockholders, and others in the event of a disaster.

Workplace Assessment* In an effort to limit client business exposure before a disaster occurs, we also provide complete facility analysis. This study covers the building, surrounding hazards, electrical and mechanical facilities, geological conditions, logistics, communications and safety. This assessment also considers key personnel, employee communication plans, family counseling, first aid, search and rescue evacuation, and relocation logistics.

Consultation Services* Business Recovery Consultation Services provide our clients with the capability to design, implement, and manage effective enterprise-wide recovery programs. Our capabilities span all facets of disaster recovery planning, from identifying a client’s vital business processes and applications to determining financial impacts of disasters, identifying potential risks, and defining critical technology and resources during disaster situations. Built upon years of disaster recovery knowledge and expertise, and incorporating IBM systems management methodologies, Business Recovery consulting engagements deliver comprehensive and effective recommendations and solutions tailored to customer specific requirements, addressing one or all of the following elements: • Vital business processes and applications • Future operating environment requirements • Financial impact of disasters • Acceptable recovery windows • Recovery mode performance • Critical technology and resource • Threat, vulnerability and asset relationships • Plans for managing communications with employees and the external media during a disaster situation

These recovery strategies are designed to include the rapidly changing environments of client/server, LAN, and end user as well as host systems. Plans can be developed to address all levels of a client’s organization — from corporate, campus, or department to the data center.

© 1998 by CRC Press LLC

Business Recovery Consultants can provide clients with the support needed to develop a plan, review an existing plan or evaluate the loss of a business process. With the assistance of IBM’s knowledge, experience and project management skills, the client’s staff will be prepared and ready for immediate action should their business suffer a disaster. Our consultants will work with them as they plan their recovery strategy using one or more of our key tools, educational and analytical services. These services encompass the three key components of successful business recovery.

Managing Risk and Avoiding Disasters* • Business recovery education • Integrated risk management/risk analysis • Anti-virus services

Developing an Enterprise-Wide Recovery Strategy* • Customer Environment Analysis (CEA) • Business Impact Analysis (BIA) • Client/Server Customer Environment Analysis • Enterprise Solutions Study • Voice • Network

Developing and Implementing the Recovery Plan* • Business Recovery Plan Development • PC Based Plan Development Software • Plan Testing • Plan Maintenance • Business Recovery Audit • Recovery planning seminars and workshops

IBM Business Recovery Consultants can help with all three phases of successful business recovery, from managing risks and defining strategy, to creating the plan that will ensure the continuation of a customer’s business.

Recovery Management Services* Testing and recovery of your critical business functions can be time consuming, taking the focus away from your core business. Recovery Management Services (RMS) provides an alternative to the skills and resources required for the management and delivery of recovery testing and disaster support. BRS will perform various recovery functions and services for our customers under an IBM agreement where customers may choose the level of complexity.

© 1998 by CRC Press LLC

10 Team Responsibilities PLAN UPDATE REQUIREMENTS At conferences, we have asked the specific question of how many attendees have updated their plans in the past year. Generally only two or three people out of a hundred raised their hands. Virtually every company will have a change of employees within a one-year period, and this means that the plan is out of date as far as those employees and work stations go. As we will discuss in Chapter 14, updating the plan is a major responsibility of the recovery team. The plan cannot be once written and then gather dust on a shelf. It needs updating no less than monthly, possibly even weekly. Depending on what kind of changes occur, this could be set up as an administrative function. Are there new vendors or new customers? Has your company signed a new contract with a security firm or other service provider? If new equipment has been brought online in the call center or other department, then the plan needs updating. Suppose that the one person who knew about a significant change was in the hospital, on vacation, or had recently left the company. The plan must include all significant new information, or else no one would know about it if the continuity plan were enacted. Below, we provide a list of the kinds of changes that need to be updated into the plan.

Every Station Every station needs to be updated once changes have been made to the station. For instance, when the company gets a new computer system, someone needs to write an update memo. There is new equipment at that station. This also applies to changes in type of office equipment, such as desk (for inventory checklist), fax (for inventory and communication station checklists), phone, and so on.

Every Change in Procedure or Contact As the departments of the company change their procedure of operation, the change has to be transmitted throughout the entire organization. This is an excellent time to update the procedures in the disaster recovery plan. If the disaster recovery team was trying to update or recover with a procedure that was out of date, it might not work. As systems change, the plan must change. We also discuss operational changes that impact upon the constitution and activities of the recovery team in Chapter 14, “Plan Maintenance.”

Every Department Representative or Alternate, Every New Employee As personnel changes, so does the disaster plan. Normally, specific people are assigned in the disaster plan to do specific tasks. Therefore, if the person doing the task changes, the name assigned to that task in the plan has to change as well. Even (especially) your telephone recall and contact lists have to change according to the changed personnel. If you no longer have Joe working for your department, then you do not want him to be on your recall list or responsible to phone other people.

© 1998 by CRC Press LLC

Every Change of Telephone Extension Changing the company telephone directory is important, because some companies change their telephone extensions quite frequently. People should not have to try to remember what the correct telephone extension is in an emergency. They should be able to look at the plan and dial the correct number without puzzling over the task. The fact that telephone extensions are going to be changing all the time means that you need to store your disaster recovery plan in a medium that is accessible during a disaster and easy to change. Those two requirements would seem mutually exclusive. However, the “to go” case contents list includes a laptop computer (optional). This is included so that the incident manager has the option of checking the contents of the plan on computer. We recommend that things that are not easily changeable like checklists should not include very many items. A checklist should not be written full of details, like an insurance policy. If checklists change, for instance, because of a change to departmental operation, we recommend that someone change both the data file and print a paper copy as well.

Frequency of Updates The company policy on updating the plan should depend on the rate of change at your company. However, the plan should be updated no less than once a year. Your team should also update the Plan to record changes after any major testing, installation of new equipment, or expansion of company facilities.

PREDISASTER RESPONSIBILITIES Checklist Updates New checklists and checklist updates are a big responsibility of the team during the planning phase. The checklist is what your team would need to get through a disaster. No one would want to have an out-of-date checklist. Furthermore, having a faulty checklist might cause a person to panic. Instead of evacuating the facility, the person might just freeze up, stay at the spot where the checklist information departs from intelligibility, and die. You and your planning team should ensure that every checklist gets checked periodically to make sure that its instructions are correct. Evacuation routes especially need to be updated regularly. Suppose that a checklist gives a specific route for someone to get down to a shelter. What if the floor or stairs on that route were damaged by some circumstance? The checklist must describe a route that is safe to use. It would be even better if the checklist gave one or more alternate routes as well.

Training Sessions You and your planning team will have to hold training sessions if company personnel are to become familiar with the plan. Members of the organization must be familiar with their part of it. This means that all employees need training sessions to ensure that they know how to use the checklists and when to use the checklists. It is also important to train the senior management in using the checklists and updating departmental changes. Generally they are the most resistant to going to a training session. It might be partly because they cannot be ordered to go to the training course. But usually those busy senior executives feel that they have more important things to do. Nevertheless, the training sessions are important to ensure that, if necessary, the CEO can be called at four in the morning. Even if he wakes up groggy, if he is to be told that there is a disaster going on at the facility, he must have some idea of what is going on.

© 1998 by CRC Press LLC

Tests, Exercises, Simulations In Part 4, we describe rather extensively the tests, exercises, and simulations that you and your team will be running. We also describe the team responsibilities for planning and running those tests in detail. However, at this point, we would just like to mention that testing is a big team responsibility. You and your planning team will have to make sure that departments make personnel available to test the disaster recovery plan. “If you have never tested your disaster recovery plan, then you do not have one.” Even though we have said this in many other spots, it is very important.

On-Call Pager/Cellular Phone One of the predisaster responsibilities to prepare is the “to go” case. We recommend that a cellular phone or a satellite phone be packed in the case. There should also be someone from the disaster recovery team on call 24 hours per day with a pager or a cellular phone. This is necessary so that when a disaster strikes, the people at the call center can immediately get in contact with a member of the recovery team. Naturally, you and the rest of the team will be very familiar with the plan. That is important when the person on the other end of the telephone line is asking questions in a shaking voice or seems totally mystified by some event.

Handover to Alternate If the person that is responsible to manage the recovery team response is not available, then that person should hand over responsibility (and the cellular phone) to an alternate. As soon as a disaster was declared, the on-duty recovery manager would become the crisis manager. Immediately. You should have one crisis manager always available to your organization depending on what the hours of your organization are. If the on-duty person is junior in experience, it is okay for that individual to hand over the incident manager position to the primary or a more senior manager once the more senior person arrives at the facility. However, if the disruption is short in duration, it might be all over with by the time the primary person can be alerted. The important thing is that the on-duty crisis manager should act in accordance with the plan to respond to the disruption and safeguard personnel, calls, and equipment. You can check out the role of the crisis manager (also called incident manager) in Chapter 8.

Updates of BIA The (BIA) should be updated from time to time to ensure that the scope of the business is being covered by the disaster recovery plan. If you recall, the amount of money that the senior accounting representative on your team budgeted to be spent on the disaster recovery plan was a direct result of the risk analysis and the BIA. So from time to time the BIA and risk analysis should be updated. If a new risk starts looming on the horizon, then the entire plan might have to be updated to reflect this new risk.

DISASTER RESPONSIBILITIES During the disaster, according to the roles assigned to the team members in the plan, the recovery team would have certain responsibilities.

© 1998 by CRC Press LLC

EXHIBIT 10-1. TEAM TASK ASSIGNMENTS Incident Manager (Crisis Manager) Hot Site Recovery Team

Disaster Site Team

Immediate Call Handling Team

Personnel Transport and Housing Coordinator

Personnel Safety Coordinator

Call-monitoring Team

Telco Hot Site Activation and Call-sharing Team

Public Relations Team

Telco Emergency Reroute Coordinator

IS Restoration Team (from backup data)

IS Preservation Team

IS Active Data Team

Hot Site Support Services Team

Property Management Coordinator (assesses building and utility condition)

Critical Business Function Team

Legal Coordinator Security Team

Eyes and Ears of Incident Manager The team would be the eyes and ears of the crisis manager. The recovery team would divide into the three subteams that we talked about before. All team members would have their own specific tasks to do, and all of them would report back to the crisis manager. The recovery team members would be thoroughly familiar with the plan because of working to write, test, and update it. They are the people who would be responsible to provide the crisis manager with information. You and the team would not be able to count on the rest of the organization functioning properly during a crisis. The team members would have to make sure that the crisis manager was kept abreast of the situation.

Passing Messages and Instructions Recovery team members would be responsible for passing messages and instructions from the incident manager to the various parts of the organization. For instance, the interim callhandling facility would be the responsibility of the various team members assigned to the IVH facility subteam (see Exhibit 10-1).

Checklist Tasks and Notes It would be important during a disaster to ensure that the members of the organization were using their checklists. The best way for that to happen is for the team to go through their checklists and then confirm with anyone who was supposed to have carried out an action that the action was in fact carried out. The team should have a master checklist listing everybody’s required activities so that it could check and log the actions taken. Suppose that a supervisor had a checklist directing that person to ensure that the calls were all exited properly when a disaster was declared. The team member assigned to confirm that activity would investigate to see that the supervisor has in fact checked the call exit procedure. If the responsibility of the call center manager was to

© 1998 by CRC Press LLC

check with a supervisor to see if all calls were exited properly, then the team member could call the call center manager and confirm it.

Head Counts and Condition Reports (Personnel) Suppose that a disaster arose and personnel had to evacuate from the call center. Once the people had arrived in the shelter area or predetermined meeting place (depending on the disaster), it would be the responsibility of a team member to ensure that a head count had been done. The head count would show whether anybody had been left behind (for example, during a tornado) and whether any personnel were injured. In case of injuries, a team member would fill out the condition report and confirm that an ambulance had been called and emergency personnel notified. The report would also record the names of the injured people, and what their conditions were when they left the area. In case of a disaster of extended duration, such as a hurricane, the condition report would show the number and condition of injured personnel. Once travel was possible again, team members would be able to bring paramedics to the most severely injured personnel first.

Team Responsibilities by Expertise It would be best to assign the team responsibilities according to the jobs that team members knew best. What work does each member do on a regular basis? That would be that person’s area of expertise. You would want to assign the most expert people to each function. If someone did not have expertise in first aid or some other essential area, then you should send him or her on the appropriate training course.

Covering for a Disabled or Absent Team Member It would be an excellent idea to ensure that every one of your team members was sent to get a St. John’s Ambulance first aid certificate with CPR (cardiopulmonary resuscitation) training. Then, if someone should be injured, the team members would know what to do. You should assign two or more people to each team position. If the team had redundant personnel (by design), every member would have an alternate to cover in case he or she happened to be absent or became disabled in the crisis. The team members should all receive cross training in each other’s duties. In case of wide-area disaster, this would improve the chances of the team as a whole being able to perform its tasks.

Diffusing Panic One of the big responsibilities of each team member is to dissipate panic. The team members should carry out their functions in a professional and calm fashion. During a disaster, if the team were performing its tasks proficiently, then company personnel would be reassured. This would dispel fear. If a person should start losing self-control (and “go off the deep end”), then the team personnel should try to calm the individual. If necessary, team members should assign a security guard to take that person outside of the facility. Team members and security personnel (who by definition must be represented on the team) should receive training in conflict resolution and dealing with people who are having hysterics. Sometimes a terrified person might just need to be distracted by walking around and getting fresh air. Sometimes a person who appeared to be panicking might be in shock from an injury, or be suffering delirium from a fever.

© 1998 by CRC Press LLC

Probably, as you planned and ran more training exercises, company personnel would become more confident and assured. Company personnel would become more at ease with the plan and checklists, and less disposed to anxiety.

RECALL/CALL LISTS (COMPANY PERSONNEL) Conditions for Use (Privacy) The personnel of the team would have access to information about all the employees in the company. The information would show where to contact people and would give their alternate contact numbers, such as at their mother-in-law’s house. You must enforce the conditions for using the call and recall lists to ensure the privacy of all individuals. Some companies and organizations keep the personnel contact lists in sealed envelopes. When the incident manager activates the disaster plan, these envelopes must be annotated with the date, circumstance, and name of the person accessing the information. Some companies keep the call and recall contact lists on diskette. In such a form it would be accessible to the incident manager’s “to go” case laptop computer. However, a printout is necessary too, in case of dead batteries, corrupted disks, electromagnetic pulse, damage to the computer because of acidic fumes or volcanic ash, or other adverse circumstance.

Ensuring Correctness of Information If you can ensure the privacy of the individuals, it will be easier to get the information that you need from all of the employees. Your team is also more likely to get updates from personnel who move, or change telephone numbers. Some personnel might not cooperate, because of abusive spouses (the positive view) or criminal intent such as commercial espionage (the negative view) — and your team might never know. We suggest that numbers for essential personnel (including team members) should be confirmed by running a communications test using the IVR/predictive dialer combination.

Alternatives during Wide-Area Disaster Your team members should find out if there are alternate locations where the employees could go in the event of a wide-area disaster. For instance, in Florida during hurricane season, the team could maintain a register of each employee’s planned evacuation destination, whether it was a town to the north or relatives in Georgia. It is easy to cite hurricanes in the Gulf States and earthquakes on the West Coast, but the probability is that any part of the continent could suffer a wide-area disaster. Every city could be vulnerable to fuel explosions, terrorism, mudslides, or other disasters. Your team should store the possible evacuation destination list in an envelope with the other contact information in the incident manager’s “to go” case. Red Cross. In the event of a wide-area disaster, the American Red Cross maintains lists of people who have registered or are seeking their family and friends. Your team should make all company personnel aware of this. Shelters. If some company personnel have not preselected an evacuation destination, a team representative should check out your local emergency organizations. A team member could find out where shelters would be set up in the event of a disaster. Then employees could know in advance where the shelters would be, and perhaps your company could provide maps

© 1998 by CRC Press LLC

to personnel so that they could seek shelter. Team members should record the notation “would seek local shelter” in their evacuation destination list for personnel that indicated such a choice. Toll-Free Line to Assisting Call Center. Do not forget to institute a toll-free line into which your personnel can call to report where they are sheltered. (See the passive IVR hotline described in the Emergency Operations Center section of Chapter 8.) The hotline for personnel contact with the incident manager should be toll-free so that if it gets switched to an assisting call center, employees would still be able to call in, even from a phone booth. “Would the employees of ABC Company please give this number a call and report where you are?” You might have heard it on the radio when companies have advertised a request for their employees to call in. There was a situation in Florida when most of the employees of a company left due to an expected hurricane. The hurricane was predicted to hit the city directly, but as hurricanes often do, it veered off at the last second and went elsewhere. As a result, the company was not struck at all. The fictitiously named ABC Company had a problem, because it could not reach its employees for a number of days. As a result, the company was out of commission even though there was no damage to its facility. The hurricane that went through caused such severe damage that most of the employees disappeared to different areas. They assumed that the call center was out of business, and so therefore they did not report into the call center as quickly as they should have. Therefore, the company lost a lot more money than was necessary.

Constant Updates If the employees were trained to call a toll-free number and register their condition and location, then your assisting call center could contact them if required. Team members could then call them with situation updates and times to report back to work. Personnel lacking a contact telephone number (for instance, if they were at a local shelter) could call back at regular intervals to receive messages. The recall list system requires constant updates, training, and testing for it to work properly. However, in case of disaster, the ability of your team to use current contact information would be immensely valuable to the company continuity. If the team were distributing food, water, and blankets after a wide-area disaster, it would be important to the employees personally as well.

© 1998 by CRC Press LLC

11 IS Department Recovery Planning WHAT IS DOES Information Systems The IS department is formally referred to as the information systems department. The call center cannot function without the functions provided by the IS department. For the purpose of a disaster recovery (DR) plan, we have included the following items in the IS department responsibilities: 1. Programming of systems and their application 2. Data gathering 3. Data center operations 4. Database functions (including sending/receiving data) 5. IVR (interactive voice response unit) 6. Compiling and reporting on IVR 7. RAN messages 8. ACD (automatic call distribution) 9. CTI (computer–telephony integration) 10. LAN (local-area networks) 11. WAN (wide-area networks).

Most call centers do not have a large IS department. Normally they have four or five individuals that take care of the management of all of these functions. For the purpose of plan data recovery, we can divide the functions into three different areas: 1. Communications (IVR, ACD, CTI); 2. Data center (data entry/collection, database, LAN, WAN, overall systems); 3. Software application (management of the software).

Data Center The data center is for the most part the hardest to recover. Without the data center or access to the database, the call center functions are extremely limited. CSR could gather information, but they could not do much with it. The call center would be effectively reduced to working with pens and pencils. Since the data center is so vital to the call center, it is generally the area that must be recovered first. The IS technicians in the data center have to work with the three recovery subteams that have already been established to ensure that data will be recovered properly. The IS

© 1998 by CRC Press LLC

representatives on those teams would, in case of disaster, join the other team members at the applicable locations.

Site Teams The disaster site recovery team IS personnel stay at the call center location, if possible, to take care of the recovery of the primary call center. This site might be slightly or severely damaged. The hot site recovery team goes to the hot site and performs the ramp-up there. CSRs and IS personnel work to initiate and test all data and telephony systems at the site. The interim call-handling (ICH) team works with the RTMO and the ICH facilities and the virtual call center (distributed architecture) to ensure that the immediate DR call recovery plan is adhered to. The first step in setting up the IS recovery is to select the IS recovery team. Most call centers as was stated earlier have a very small IS department so generally the IS recovery team would be the same people who work in the IS department on a daily basis. If you have enough personnel, we advise that you divide your team up so that the disaster site recovery team, hot site recovery team, and the interim call handling team all have members from the IS department. Then, redistribute your IS members further into the communications, data center, and software applications teams. Once you have your IS team, then you need to decide the scope of the recovery process. Exactly what you are going to recover? Trying to recover full CTI at a hot site is generally very difficult if not impossible. So you need to decide how much functionality you have to bring back and what kind of systems the CSRs are going to have. If the system at the hot site is going to be different than the system at your call center, then you need your employees to practice using that system as part of their plan tests. At the disaster site, you have to decide what information is going to be saved and what applications will be removed from the call center or backed up to another facility as quickly as possible. What are your backup procedures going to be to get your data backed up at another facility? The ICH group will have to decide how to get data to that facility? How much data does that facility need? All of these decisions should be preplanned.

Checklists and Objectives One of the considerations that needs to be addressed by the IS department is how can it create checklists for its part of the DR plan. If it is impossible to create a checklist for any particular item, then the IS department will need to decide how badly that portion of the DR plan is required. Without a checklist, there is a good chance that any particular section of the DR plan either will not get completed properly or perhaps will not be done at all. There needs to be checklists made for disaster types, but there also needs to be checklists for each piece of equipment (no mater how simple it would be to shut down or start up). Some people who work on computers all day do not know how to shut them down properly. If the equipment needed to be shut down during a disaster, then you would need a shutdown procedure. For example, if water is leaking onto the UPS, is there a procedure to shut it down in this situation? Water has caused more problems than any other type of disaster — because of leaking ceilings, air-conditioning units, fire department high-pressure spray, backed-up drains, and similar situations. Remember that if you have a fire, it is likely that the water will do as much damage as the fire (if not more) to your call center.

© 1998 by CRC Press LLC

If you want to ensure that your checklist can be used by anyone, do a test. Get the cooperation of a CSR or a janitor, or some other employee that has little or no knowledge of the systems involved in the checklist. See if that person can follow the checklist properly. If not, then fix the checklist until the person can make it work. The recovery plan that these teams create must have certain objectives. The checklists that they create must reflect these objectives. The first objective for all portions of the DR plan is to ensure the safety of all employees. The second major objective is to get your call center calls recovered. The data must be recovered to the hot site, the RTMO, and ICH facilities in a fashion that is rapid and effective. The general goal in any portion of the DR plan is to limit the amount of loss in any disaster. Obviously, you cannot control how much is destroyed by any given disaster. You can, however, reduce the amount of downtime as a result of the disaster. An additional objective of the IS department DR plan is to assess the amount of damage to the information systems. It will need to determine whether or not the IS department will be able to function in a reasonably short period of time. This is one of the factors that will help to determine when or if the call center should go to the hot site and whether or not to activate that portion of the DR plan. There will need to be an assessment scale in the plan to assess accurately the damage and the recovery time needed to get the IS department fully functional again. This is a major part of the overall recovery of the IS department. The IS team must ensure the systems are brought up, and that the IS department identifies all that needs to be done to recover data systems. Each DR team will have objectives to accomplish, outlined in their checklists, when any portion of the disaster recovery plan is initiated for any reason, be it a test or a disaster. The responsibilities of the disaster site recovery team will include shutting off the computer systems to limit damage to them. The members should perform transmission of a electronic vaulting, diskette, or tape backup. They should turn off or down as many systems as possible, (air conditioning, water, and heating) to reduce damage to electrical systems in the time available to them.

Damage Through our research we have discovered that when a hurricane comes, many landlords will shut down their buildings to reduce damage to them. Unfortunately, if you are still operating when the landlord decides to do this, then you will have added damage to your equipment. Most call center equipment does not benefit by simply being turned off with an instant power loss. You do not want a UPS damaged by power spikes or instant power loss. You will need to make some sort of agreement with your landlord, so that the building manager will not shut your systems down until you have shut them down internally first. Once the disaster is over, the disaster site recovery team must determine the amount of damage to your telephony equipment, data systems, to utility services, heating, and air conditioning. It will also need to determine how much of the documentation will need to be in place to begin operations again. It will be responsible for ordering equipment that needs to be replaced and setting up that equipment. What if you have to operate at a cold site or you need to look for a new site altogether? The DR team would have participated in the selection of alternate sites. Should the decision be made to go to a cold site or hot site, then the DR team will be the first to arrive at that location to begin ramping up for operation. One of the problems with the IS department during a disaster involves security. You have to leave the doors to the IS department unlocked during a disaster. Yet you cannot afford to leave your doors unlocked in the case of a false alarm. So you need to have a lockout

© 1998 by CRC Press LLC

procedure or a shutdown procedure for your database and main equipment areas. You need to make these areas difficult to sabotage. Yopu need to maintain security in these areas. In fact, you should even boost security.

DISASTER RECOVERY Interim Call-Handling Team The ICH team is responsible for activating the IS departments at the ICH facility and the RTMO facility. It will need to contact those departments and ensure that they have begun their recovery processes. RTMO facilities already have ongoing day-to-day operations with your company. Their IS departments will already be operating. However, you will still need keep their IS department informed of developments on your end of the disaster. You will need to ensure that the RTMO and the ICH are able to access your data at its new location. The ICH facility will need checklists to know how to bring up your systems, as you require them. A checklist for this facility would include going through the steps of bringing up the call types that they will be handling as well as all of the data needed to handle these call types. Generally, once the ICH facility is up and running, the job of the ICH team will then be to monitor the ICH and the RTMO facilities. It can now begin to work with the hot site recovery team to ensure that the hot site is operational quickly.

The Hot Site Recovery Team The hot site recovery team is responsible for, first, getting to the hot site and, second, getting the hot site operational. Most hot sites are commercial hot sites, and they will have their own recovery team to work there with you. Your team is responsible for getting the backup data to the hot site, initiating operating systems, and preparing the hot site to receive calls. The ICH facility and the hot site recovery teams have to ensure the backup of data during the recovery process. This is because disasters have been known to strike companies in more than one location at any given point in time. Just because you have experienced one disaster does not mean that you should not be prepared for another. Make sure you continue to make proper backups.

Telco The IS department will also need to make sure that the preprogrammed call routing is in place and functional when a disaster strikes. This will have to be done in accordance with the requirements of the ICH facility, RTMO, the virtual call center (distributed architecture), and then according to the requirements of the hot site. You might have as many as six preprogrammed call-routing requirements. If Telco will do this, then the best thing to do is to have these requirements set up in advance so that the IS department can select the proper route for the DR plan that is being activated. It is best if you can code your routing methods so that when you call your Telco, you simply tell them, “Implement call routing Alpha for DR plan Tornado. The pass word is….” This is a quick routing method that leaves little room for error.

Data Storage All of this planning will not do you any good unless you have an off-site data storage location. You must ensure that the backup site is contracted and operational. Generally, the hot

© 1998 by CRC Press LLC

site facility can assist in this as well. The IS department will be responsible for ensuring that the data is rotated to the hot site and then to the off-site storage premises so that backups are kept up-to-date. This can be done with dedicated data transfer lines (electronic vaulting) or by tape backup. Systems programming and applications must be protected. Every time you create a new application, the system protection should be updated to reflect the changes made. All backup facilities should be updated at the same time that you update your call center. Your database does you no good if you do not have the proper programs to run it. You must ensure that the materials and manuals that your IS team will require all are copied off site, as well. There is a critical forms inventory that the IS personnel should fill out to ensure that they have everything they need and that all of the required materials are stored in a safe location. Finally, the IS department must ensure that the reconstruction and replacement procedures are in place in case a cold site must be made operational. The IS team members should call all of their vendors to find out about their “quick-ship” programs. Most of what might be needed should be included in service level agreements. There are also a number of data restoration companies available to assist. They are listed in the Disaster Recovery Yellow Pages. These companies could fix any damaged databases to restore lost data. Some of these companies do this kind of work for the police as well, when they are investigating crimes involving computer data storage.

IS Recovery and the Call Center IS recovery planning has been around since the mid-1970s. Most of the smaller companies started paying attention to this in the 1980s. This book is not meant to advise you on how to recover your data systems. However, call center recovery and data system recovery have to work together if a recovery is going to work. We have designed this chapter to help do that. If you require a complete data recovery plan, then you should have a look at Business Resumption Planning published by Auerbach Publications, or talk to one of the companies listed in Exhibit 11-1.

© 1998 by CRC Press LLC

EXHIBIT 11-1. COMPANIES Comdisco Disaster Recovery Services, Inc.

Metropolitan Emergency Disaster Services — MEDS

611 North River Rd.

2500 W. Roosevelt Rd.

Rosemont, IL 60018

Chicago, IL 60608

800-272-9792

312-226-6337

Data Assurance Corp.

NCR Business Recovery Services

12503 East Euclid Dr., Suite 250

1611 S. Main St., SDC-3

Englewood, CO 80111

Dayton, OH 45479

800-654-1689

800-587-0911 x4

Delaney Recovery Services, Inc.

IBM Business Recovery Services

540 Township Line Rd.

Long Meadow Rd.

Blue Bell, PA 19422-2701

Sterling Forest, NY 10979-0700

215-992-1081

800-599-9950

Global Link Communications, Inc.

SunGard Planning Solutions, Inc.

1021 Mill Creek Dr., Suite 2

1285 Drummers Lane

Feasterville, PA 19053-7326

Wayne, PA 19087

800-494-5465

800-448-6850

IBM Business Recovery Services Long Meadow Rd. Sterling Forest, NY 10979-0700 800-599-9950 Names supplied courtesy of the Disaster Recovery Yellow Pages

© 1998 by CRC Press LLC

12 Disaster Implementation Tasks WHETHER TO DECLARE A DISASTER — OR NOT The decision of whether or not to declare a disaster will have far-reaching consequences for the call center and the whole company. Upon the enactment of the continuity plan, the recovery team will cease to do its regular tasks, and bend every effort to the noble enterprise of ensuring the survival of the company. The call center will commence the handover of calls to the assisting call center(s) in whatever manner the continuity plan dictates. Throughout the company, all personnel will cease their normal way of operating and refer to their continuity plan checklists. All of these changes in the way your company behaves will have consequences. Some personnel will experience high blood pressure. Some departments will spend a lot of money. Some departments will have very little work to do, of their regular kind, and will send their people to the aid of the recovery team. For sure, everyone in the company will sit up and take notice of the person who declares a disaster, and what happens next. How are you supposed to know if all this bother and expense is justified? What if there really is not a disaster, just the misapprehension of someone who should have thought before acting? What if a competitor or some malicious prankster is hoaxing your company? 1. There must be some way to determine what response to a questionable situation is appropriate. We will discuss the question of whether to declare a disaster. Just as important is the question of when “or not” is the appropriate response. 2. This is the area where most of the unsuccessful plan enactments that we have heard about fell apart. Companies have had a plan but decided not to implement the plan, and what started off as a bad day turned into a major problem.

Every Disruption a Potential Disaster One of the useful characteristics of call volume management is the fact that if you start suspecting that things are going astray on you, your call center can start handing over its simple calls reasonably quickly. Most of the time, the actual implementation of the interim callhandling (ICH) measures and real-time minimum call overflow (RTMO) is not a great deal more expensive than your normal day-to-day operations. So declaring a disaster is a “better safe than sorry” situation. The cure is far less expensive than the disease. Every disruption is a potential disaster. It is important to remember that there have been many disruptions that have turned into disasters. At any gathering connected with the call center industry, you could (if you asked) hear all kinds of stories about situations that call center managers did not think were serious — at first. We hope that by giving you some food for thought, we will help you write the guidelines for declaring a disaster that people in your company would depend upon to help them decide what to do in borderline circumstances.

Minor Disruptions with Major Consequences This relates to the section on escalation in Chapter 9. Here are some minor disruptions with major consequences.

© 1998 by CRC Press LLC

The Smell of Tar. A facility was getting some roofing work done, and everybody expected to smell tar. The roofing company parked its tar-making machine (“tar pit”) next to the ventilation intake. No one saw where it had been parked. The smell of tar inside the building kept getting stronger. Because it started off smelling faintly and increased gradually, the people did not notice at first. Suddenly, however, personnel were complaining because they had become ill. The management did not feel that this was a major disruption and did not begin to implement the disaster recovery plan. In this case following the plan would have meant sending people home and working from the distributed architecture environment (i.e., they would convert to a virtual call center staffed by telecommuters). By the time that management noticed that it had a significant problem, most of the work force was ill. It developed that most of the employees were so ill that they did not return to work the next day. Because the management assessment was faulty, it treated the smell of tar in the building as inconsequential. Nevertheless, this “minor” disruption became a disaster that got away. Instead of having maybe a 15-minute period of transfer while it gradually handed over calls to its call-takers as they arrived at their homes, the company ended up with two days of sharply reduced operations. Would you have declared a disaster in this circumstance? Please refer back at this point to Exhibit 6.2 for examination of a communications outage. Consider again the possible lost revenue per hour of sales or the possible cost per hour of customer service/complaint resolution (at peak volume). Your company “cost per outage hour” figure might be lower or higher than those given in our example. However, once you have gone through the BIA and risk analysis exercise, you can determine what the cost would be to your own company. Exhibit 12-1 examines the costs to compare when deciding to declare a disaster or not, for example, during a bomb threat. When it comes to deciding to escalate your response from “assessment” to declaration of a disaster, it is usually better to take a better-safe-than-sorry attitude. Even making precautionary backups, and alerting the assisting call center that handover of calls might take place, is better than just ignoring the situation as unimportant and hoping it will go away. The following scenarios are covered in Exhibit 12-2. “After-Lunch Epidemic” Caused by Food Poisoning: What first appeared to be an outbreak of influenza or other infectious digestive disorder became identifiable when call center supervisors learned that all the affected personnel had eaten at the same restaurant. Those who were ill were taken to the hospital emergency room. Decision point: since food poisoning might be potentially lethal in serious cases, should the incident manager enact the preliminary stages of the disaster plan? The ICH facility handled partial volume of calls until spare and on-call CSRs could be called into the call center to take over. Personnel were warned about the suspected link to the restaurant, so no more CSRs ate there. “The Newly Cleaned Carpet” Causes Noxious Fumes: Fumes from a chemical reaction due to an overnight cleaning of the call center carpet caused call center personnel to suffer watering of the eyes, headache, dizziness, and nausea. Decision point: Should the incident manager start enacting the plan provisions right away, or wait until personnel were so seriously overcome as to need medical attention? Variables: would emergency personnel cordon off the building until the Environmental Protection Agency (EPA) could determine how poisonous the fumes were? How long would it take to ventilate the building? Should the incident manager instruct the public relations representative to prepare a precautionary press release?

© 1998 by CRC Press LLC

EXHIBIT 12-1. BOMB THREATS AND HOAXES (ANNOTATED) Decision Point

Possible Consequence

Cost of Declaring a Disaster for a Hoax Services of assisting call center

Declaration cost = $3500 $25 estimated cost per work station, per hour × 150 = $3750 Total = $7250

Effect on morale, reputation

Successful “test” of continuity/disaster recovery plan proves skill and diligence

Cost of Ignoring a “Hoax” and Discovering a Disaster Cost of lost calls

Estimated loss of business if down 1 hour during peak = $292,500.00 Estimated loss of business if down for 4 hours at peak based on average call volume = $624,000. 00 Estimated loss if building unexpectedly destroyed by explosion = minimum 10 hours (in this example the operating day is 10 hours long) = $1,560,000.00 or more Who would survive to enact handover of calls?

Damage to assets, loss of life

Replacement of building = $15 million Replacement of equipment = $3 million Replacement workers = 150 × $15,000/yr = $2.25 million Total lost = $20.25 million

Damage to morale, reputation

Negative morale = Crushed Reputation = Destroyed

Total cost if “disaster” really a hoax

$7,250.00 only

Total possible loss if “hoax” really a disaster

$21,810,000.00 or more

Cost saving to think about if having trouble deciding whether to enact plan provisions

$21,802,750.00 or more

Note: Cost of lost calls taken from example at Exhibit 6.2. CSRs at $6/hour (U.S. dollars) estimate $15,000/year earnings.

Pandemic Crowding Hospitals Incapacitates Call Center Personnel: A serious outbreak of disease caused widespread illness throughout the region that the call center was in. Almost all the CSRs were either ill or had family members who were confined to the hospital or in bed at home. The outbreak was so serious that hospitals were overtasked, and patients were being put on gurneys in the crowded hallways. Television news reports started to mention the Centers for Disease Control in Atlanta. Medical authorities discovered that the watershed had become contaminated with cholera. Radio news reports carried lists canceling meetings of clubs, organizations, and sports contests because so many were ill. The incident manager decided, “Recover Calls to Distant ICH. Do NOT Activate Hot Site.” The incident manager declared a disaster and handed over call volume to an assisting call center several hundred miles away. This was because so many company personnel were

© 1998 by CRC Press LLC

EXHIBIT 12-2. ILLNESS AND POISONING (ANNOTATED) Decision Point

Problem

Plan Action

“The after-lunch epidemic”

CSR food poisoning

Partial handover of call volume to ICH; send affected personnel to hospital

“The newly cleaned carpet”

CSR illness due to noxious fumes

Handover of calls to ICH, evacuate building; ventilate

Chemical spill (hazardous material or “HAZMAT”)

Emergency personnel order evacuation

Hand over calls to ICH; evacuate building; assess duration — will hot site be needed?

Pandemic crowding hospitals

CSR serious absenteeism due to areawide epidemic

Recover calls to distant ICH; Do not activate hot site (see discussion below)

Call-taker efficiency — when to hand over calls

Monitor calls in queue; average length of call; number of transfers and calls put on hold

Compare to plan preestablished handover level; at predetermined points institute precautionary backups and partial handover of calls

sick with the disease that the company no longer had the personnel resources to do the work necessary to move operations to the hot site facility. Even if the hot site had been utilized, it could not have been staffed with call center personnel. After this action, the incident manager went to the hospital, having come down with the disease, too. The incident manager’s alternates voted the healthiest one of them to be the “deputy incident manager” for the time being. This individual stayed in contact with the distant ICH to ensure that call-handling quality levels were being met. Decision point: Should the deputy incident manager seek the partial return of calls from the ICH when convalescing personnel started coming back to work, although they were still in a feeble condition? “Weather happens.” Depending on your region, you can predict that some kind of severe natural event will strike … sometime (see Exhibit 12-3). If you and your team can plan well, your company will be prepared. Your company personnel will meet the challenge, and the company will survive with cash flow and reputation intact. Consider the possibility of establishing multiple sites and contracting with ICH facilities in widely separated areas, to reduce risk. You and your planning team should write the plan in such a way that the preliminary actions of plan enactment are the same in all circumstances (see Exhibit 12-4). Once the recovery team is alerted, the incident manager will follow checklists to recover communications and critical functions. Most major plan actions will be the same in all circumstances. (See the Disaster Checklists section later in this chapter.) However, the incident manager (or alternate) should be prepared to think creatively when managing a disaster that does not fit preplanned options.

DECLARATION OF DISASTER, DISASTER TYPE The previous section discussed the decision points for declaring a disaster. We will now look at some aspects of declaring a disaster. These are basically the disaster types. In addition to the basic characteristic of whether disasters are natural or man-made, you can classify a

© 1998 by CRC Press LLC

EXHIBIT 12-3. SEVERE NATURAL EVENTS (ANNOTATED) Decision Point

Consequence/Action

Loss of roof (wind storm)

Primary task: safety of personnel Handover to ICH facility If raining: water damage to computers If dry: cover electrical equipment with tarp and try to carry some components to alternate site

Loss of power (lightning)

Use cellular phone to hand over to ICH facility; instruct personnel to remain indoors until storm over; after power restored, test equipment for damage

Flooding, slow or swift

Hand over to ICH; shut off power; evacuate personnel to high ground; if time available, sandbag around building(s)

Mud slide

If expected: evacuate personnel and hand over to ICH facility. If unexpected: hand over to ICH facility while evacuating or rescuing personnel; assess site for damage, danger of further mudslides

Earthquake, aftershock, tsunami

Hand over to distant ICH facility Ensure safety of personnel; enact wide-area disaster provisions; assess disaster site for structural damage; prepare hot site team; if danger of tsunami, evacuate immediately!!

Assessment of successful recovery

Safeguards company revenues, personnel, equipment, and building

EXHIBIT 12-4. SCALABLE RESPONSE (ANNOTATED) Plan Action

Remarks

Precautionary backups

Take off site to hot site, or arrange electronic vaulting

Alerting assisting call center “in case”

Arrange two or more ICH facilities in separate areas

Partial handover of calls

Arrange prerouted switching to ICH facilities

Precautionary alert of recovery team

Write and distribute checklists

Enactment of checklist usage

Write checklists in logical steps, including actions on declaration of disaster

disaster by the speed of onset. The speed of onset makes a great deal of difference in the impact of the disaster.

When There Is No Warning The first disaster classified by speed of onset is the type that gives no warning. This is generally an explosion, rapid fire, or infrastructure failure. These disasters are the most life-threatening. The most important thing that you have to be concerned about is the safety of your staff. You must get people out alive. Your company plan checklist must include this.

© 1998 by CRC Press LLC

The only nice thing about a “no warning” disaster is that you have no doubt that you are in a disaster. You know that you must immediately implement your disaster plans. All personnel must grab their checklists and follow whatever disaster procedure is in place. For instance, suppose that a tornado touches down right outside your building. You know that you must immediately get people down to the tornado shelter. This is where having the ready-to-go case is very nice. If you have to get out of your building very quickly, you can grab this case as you go. This will ensure that your incident manager has everything (cellular phone, contact names and numbers, and copies of the plan, etc.) that is needed to begin operations immediately to recover communications and critical functions, thus ensuring the continuity of the company.

When You Can See It Coming The disaster that you see coming gives you plenty of warning. Consider the following: hurricanes, large storm fronts, or fires that are raging in the building next door. These are things that you can see coming in advance. They give you some period of advance warning. It is generally not easy to determine when you should declare a disaster or state of emergency and begin to enact your recovery plans. Hot site recovery in this situation is generally difficult because typically, if you can see the disaster coming, then it is a wide-area disaster for example, is flooding. Consider slow flooding, where you know that a peak is coming. You know that the river crosses a road at such and such a place, and a bridge could wash out. You know that if the dike breaks, then the town is going to be flooded out. But if you are downstream, you can know as much as a week in advance when the crest is going to be. You can assess whether there is a reasonable expectation that your location is going to be flooded out. The danger with this particular disaster is in trying to decide just when exactly to declare a disaster. Lots of people wait too long, and put their operations in peril before they actually declare the disaster. Personally, we would prefer to declare the disaster sooner rather than later. There would be time for the ICH facility to take the calls. There would be time for the hot site recovery team to get a skeleton crew up to the hot site to prepare for opening it as an alternate site. This way, if it were necessary to get the rest of the folks out of the original facility, at least the critical operations would be taken care of while everyone was evacuating.

When It Looks Innocuous, Then Embarrassing, Then … The third type of disaster is what we call the slow-motion disaster. It starts out looking innocuous enough, and then it gets embarrassing, and then “boom!” you are down.

Slow-Motion Disaster The problem with the slow-motion disaster is that the disaster is sitting right in front of you. The “smell of tar” incident that we cited earlier is an example of a slow-motion disaster. It was right there and everybody saw it. However, nobody realized what was going on until it was too late. Exhibit 12-5 lists possible examples of slow-motion disasters. Here is another illustration. Personnel of a company came into work and saw thick, black smoke pouring out of their building. They called the fire department. Their recovery plan called for them to travel four hours to a hot site. They had no ICH outsource contract, so for them to declare a disaster, they had to spend a lot of money. After the fire department got to the building, the firefighters told them that the smoke was not being caused by fire but by some kind of chemical reaction. The EPA had to be called in. Firefighters told the company representatives that the EPA team would be there shortly. But it transpired otherwise. The EPA arrived four to five hours later.

© 1998 by CRC Press LLC

EXHIBIT 12-5. TYPES OF SLOW-MOTION DISASTERS (ANNOTATED) Equipment

Telephony

Personnel

Natural

Infrastructure

Computer virus

Call-blocking

Epidemic

Flooding

Electricity

Incompatibility of program and O/S

Cut cable

Mass transit failure

Hurricane

Water

Year 2000 (Y2K) problem

PBX or ACD failure

Local politics

Tornado or wind storm

Telephone grid

Corrupted data storage media

Outdated dontact Lists

Labor dispute

Earthquake and aftershock

Road or street blockage

When the EPA team examined the site, it determined that the cause of the smoke was indeed a hazardous chemical. Next, it had to try to determine what was causing this hazardous chemical reaction. By this time, half the day had gone by for the company. The EPA did not give the company an indication of how quickly it would make the determination of the cause. The company executives were hopeful that it was some minor equipment failure. They wanted to get back to operational status reasonably quickly. The company delayed activating its disaster plans until the EPA got back to them. Well, the EPA team replied to the company executives the next morning. It told them that no one could get back into the call center facility that day. This is when the company decided to implement the disaster recovery plans. By that time, it was difficult to contact the employees. It took until the end of that day (Day Two) to gather the call center personnel and get everybody up to the recovery facility. The alternate facility was a “semi-hot site.” That meant that it was in only a partial state of readiness. It took the recovery team most of that day and half of the next day to get the recovery site operational. The company communications were down for two and a half days, which was very significant for them. You Cannot See It. From what we heard, this incident very nearly put the company out of business. It was a reasonably simple disaster, but it was slow-motion disaster. The decision to activate the disaster recovery plan was not right there in front of their eyes. This is why the escalation chart is so crucial, and why you should make the decision to declare a disaster reasonably easy. If it is not going to be horribly expensive to declare a disaster, then get the ICH activated. This makes the decision to declare a disaster easier. You Acclimatize. We have all seen disaster movies where the person who is in charge does not make the critical decision because he just cannot believe the disaster is actually happening. When a disaster sneaks up on people, suddenly they cannot believe their eyes. • Why is the volcano blowing up? It was only rumbling a little this month. • Why are rioters destroying the building? They were demonstrating outside for a week with no trouble. • Why are we snowed in at our call center facility? The blizzard was supposed go light on our area. • Why can’t we breathe? The fire down the street seemed under control.

© 1998 by CRC Press LLC

Graphing Falling Efficiency. From a management point of view, one of the ways of detecting a slow-motion disaster (like the “smell of tar” example) is to graph falling efficiency. We hope that you have a call-blocking reporting mechanism to show you how many calls are not getting through. Telco can also give you the call-attempt/call-abandon rates. This can give you an indication about what is happening with your call volume. A lot of call centers have display boards that show how many calls are waiting at any given time. The number of CSRs working in the call center, noticing how many have got up and left, is a pretty good indication if something adverse is happening. This could indicate such problems as the “after-lunch flu,” for example, where a lot of people get food poisoning after lunch. If personnel get sick and leave quickly going toward the washroom, and the call volume is backing up, it is time to take action. Consider the risks in your area, and those that might be unique to your industry and company.

Disaster of Infrastructure Almost by definition, a failure of infrastructure is beyond your control. You cannot, personally, go out and fix the problem. As the planner/planning manager, you can only plan the best way you can to minimize the impact upon your company. Failure of Power Station or Water Plant. What operations of your company would be affected by loss of electrical power? What would be affected by loss of water? Electricity is necessary for most data-intensive operations of the modern office. It is needed for much of what happens in the shipping/receiving/warehouse area, which cannot operate without lights. Your fulfillment department could be crippled without a computer to keep track of shipments and inventory or print labels and bills of lading. Even the humble coffee machine or kettle requires electricity to work. Water is not just necessary for drinking fountains or washrooms. If water is cut off, the fire department might evacuate the building, because fire sprinklers would be inoperable. This might apply to whole city blocks or quarters, especially after a wide-area disaster such as a hurricane or earthquake. Perhaps you remember the bucket brigades of volunteer citizens who helped firefighters in San Francisco after the Bay Area earthquake. If water for fire control should be cut off in such circumstances, it is not beyond possibility to consider a firestorm sweeping through building after building, and producing great danger to human life. Wide Area: Ice Storm, Blizzard, Hurricane, Tornado, Earthquake. As we have seen from news coverage of ice storms and hurricanes, if severe weather visits an area, infrastructure of many different types could be affected over a wide area. Blizzards and ice storms could make roads impassable and would endanger human life because of freezing temperatures and possible electrical power outages. Likewise, infrastructure damage from an earthquake could affect a large area (depending on the strength of the quake) in a diminishing amount out from the epicenter to the fringes of the disaster area. Tornadoes could do severe damage to the area in the twister’s path. In one way, a tornado could cover a wide area. In another, depending on the strength and size of the storm, damage could be more localized than in other types of wide-area disaster. It is not unknown for a tornado to damage buildings in part of a city block, and leave others nearby totally untouched. Assisting Call Center Should Be Outside Local Area. Depending on the resources of your company, you and your planning team might decide to use only one, or more than

© 1998 by CRC Press LLC

one, ICH facility. If your company can only afford to retain one ICH facility, then you should ensure that it is far enough away to be useful in the event of a wide-area disaster. If your company can afford multiple outsource contracts, your emergency communications plan should provide criteria for deciding to use local or distant ICH assistance, depending on the scale of the disaster.

ESCALATION CHART Next is the escalation chart. To know how to use the escalation chart, you need to know the stages of a disaster. The basic stages of a disaster are as follows: 1. Event 2. Alarm 3. Assessment 4. Notification 5. Declaration 6. Evacuation 7. Escalation of the plan including backup activation 8. Recovery of calls 9. Restoration of the systems 10. Restoration of the calls 11. Return to normal operations 12. Plan evaluation and update.

However, life is rarely this simple. In a real disaster, the stages of assessment, escalation, and recovery may have to repeat several times. Consider the exhibits in the following sections. We also discuss this in more detail in Chapter 23, “Basic Crisis Management.”

Stages of Disaster Exhibit 12-6 is the chart that appeared in Chapter 9. In this example the recovery time objective in this case is more than 30 minutes and less than one hour. So when a small-scale disaster occurs, the plan is to wait to see if it will become a larger-scale disaster. The ICH or RTMO action in this scenario is for more than one hour of outage but less than six hours. If the call center is going to be out for more than six hours, the hot site must be activated. Most hot sites only allow the client call center to be there for 45 days. If the outage is expected to be longer than 45 days, the company must activate an alternative facility by that time. Exhibit 12-7 displays this sequence of events against the stages of disaster. This can be turned into a chart that escalates for other types of risks, as seen in Exhibit 12-8. You will note that both of these are sudden-onset-type disasters.

Ripple Effect The interesting thing about a disaster is the ripple affect. Some disasters do not directly affect your facility or your business. For instance, you could have a fire down the street. People in your facility would hear the fire engines arriving nearby. Next thing, you might be told that the firefighters have blocked off access to and from your workplace. People would therefore have trou-

© 1998 by CRC Press LLC

© 1998 by CRC Press LLC

EXHIBIT 12-6. SCALABLE LEVELS OF RESPONSE Category

Decision

Short duration outage

9 Determine if fire is real

Short evacuation required 12 Determine size of fire

Moderate damage to C.C. or infrastructure 18 Assess damage after fire is extinguished

Severe damage to C.C. or infrastructure

Note: C.C. = call center.

Actions 1 Fire

2 Fire alarm

3 Checklist states begin evacuation

4 Notify backup call center of fire alarm

5 Advise staff of evacuation

6 Take “to-go” case and evacuate

7 Meet in Designated Area 1

8 Do a head count

10 If no, then

11 No recovery plan activation required

12 Return to normal operations

13 Evaluate evacuation

10 If yes, then

11 Call hand-over to ICH or RTMCA facilities

13 If small fire with no damage to call center, then

14 Return to call center begin clean up

15 Return calls

16 Return to normal operations

17 Evaluate evacuation and update plan if necessary

13 If moderate to large fire, then

14 Activate hot site

15 Activate all three recovery teams

16 Notify ICH or RTMCA facilities of situation

17 Coordinate with local fire authorities

19 If damage is moderate and will be able to return in 45 days or less,

20 Complete repairs as necessary

15 Return calls

16 Return to normal operations

Evaluate fire recovery and update plan if necessary.

19 If more than 45 days., activate cold site or alternate location

20 Activate dropship provisions. Set up new facility.

16 Return calls to new site

17 Return to normal operations

Evaluate fire recovery and update plan if necessary.

© 1998 by CRC Press LLC

EXHIBIT 12-7. STAGES OF DISASTER

Short-Duration Outage

Short Evacuation Required

Moderate Damage to Call Center or Infrastructure

Severe Damage to Call Center or Infrastructure

Event

Fire

Fire

Fire

Fire

Alarm

Fire alarm

Fire alarm

Fire alarm

Fire alarm

Assessment

Checklist states begin evacuation

Checklist states begin evacuation

Checklist states begin evacuation

Checklist states begin evacuation

Notification

911 – Notify backup call center of fire alarm

911 – Notify backup call center of fire alarm

911 – Notify backup call center of fire alarm

911 – notify backup call center of fire alarm

Declaration

Advise staff of evacuation

Advise staff of evacuation

Advise staff of evacuation

Advise staff of evacuation

Evacuation

Take “to-go” case and evacuate

Take “to-go” case and evacuate

Take “to-go” case and evacuate

Take “to-go” case and evacuate

Meet in designated area

Meet in designated area

Meet in designated area

Meet in designated area

Do a head count

Do a head count

Do a head count

Do a head count

Assessment

Determine if fire is real

Determine if fire is real

Determine if fire is real

Determine if fire is real

Escalation

No

Yes

Yes

Yes

Recovery of calls

No recovery plan activation required

Call handover to ICH or RTMCA facilities

Call handover to ICH or RTMCA facilities

Call handover to ICH or RTMCA facilities

Assessment

Determine size of fire

Determine size of fire

Determine size of fire

Escalation

If small fire with no damage to call center

If moderate to large fire

If moderate to large fire

© 1998 by CRC Press LLC

EXHIBIT 12-7. STAGES OF DISASTER (continued)

Short-Duration Outage Recovery plan

Short Evacuation Required Return to call center

Moderate Damage to Call Center or Infrastructure

Severe Damage to Call Center or Infrastructure

Activate hot site

Activate hot site

Activate all three recovery teams

Activate all three recovery teams

Notify to ICH or RTMCA facilities of situation

Notify ICH or RTMCA facilities of situation

Coordinate with local fire authorities

Coordinate with local fire authorities

Assessment

Assess damage after fire is extinguished

Assess damage after fire is extinguished

Escalation

Damage is moderate, will be able to return in 45 days or less

Large amount of damage, will not be able to return for more than 45 days

Recovery plan

Continue operations in hot site

Activate cold site or alternate location

Complete repairs as necessary

Activate drop-ship provisions

Restoration of system

Begin cleanup

Set up new facility Restoration of calls

Return calls

Return calls

Return calls to new site

Return to normal operations

Return to normal operations

Return to normal operations

Return to normal operations

Return to normal operations

Plan evaluation and update

Evaluate evacuation and update

Evaluate evacuation and update plan if necessary

Evaluate fire recovery and update plan if necessary.

Evaluate fire recovery and update plan if necessary

EXHIBIT 12-8. ESCALATION CHART WITH OTHER TYPES OF RISKS Event

Fire

Tornado

1. Alarm

a. Fire alarm b. Smoke c. You hear “FIRE, FIRE, FIRE”

a. Radio reports sighting nearby b. Tornado sirens

2. Assessment

Checklist states call 911, recovery team, and begin evacuation

Checklist states call recovery team and begin evacuation

3. Notification

a. 911 b. Notify recovery team of fire alarm

Notify recovery team of tornado

4. Declaration

Advise staff of evacuation

Advise staff of evacuation

5. Evacuation

Take “to-go” case and evacuate a. Meet in designated area b. Do a head count

Take “to-go” case and evacuate a. Meet in designated shelter b. Do a head count

6. Assessment

Determine if fire is real

Determine if the tornado is going to effect the call center

7. Escalation

If Yes, go to next If No, go to 18

If Yes, go to next If No, go to 18

8. Recovery of calls

Call handover to ICH or RTMCA facilities.

Call handover to ICH or RTMCA facilities

9. Assessment

Determine size of fire

Determine extent of damage

10. Escalation

a. If moderate to large fire, go to next b. If small fire with little or no damage to call center, go to 16 or 18

a. If power failure, go to power failure escalation chart b. If tornado hits call center, go to next c. If tornado does little damage, go to 16

11. Recovery plan

Activate hot site

Activate hot site

Activate all three recovery teams

Activate all three recovery teams

Notify ICH or RTMCA facilities of situation

Notify ICH or RTMCA facilities of situation

Coordinate with local fire authorities

Coordinate with local fire authorities

12. Assessment

Assess damage after fire is extinguished

Assess damage after tornado has passed

13. Escalation

a. If damage is moderate, will be able to return in 45 days or less, go to 15a b. If large amount of damage, will not be able to return for more than 45 days, go to 15 b, c, and d

a. If damage is moderate, will be able to return in 45 days or less, go to 15a b. If large amount of damage, will not be able to return for more than 45 days, go to 15 b, c, and d

© 1998 by CRC Press LLC

EXHIBIT 12-8. ESCALATION CHART WITH OTHER TYPES OF RISKS (continued) Event

Fire

Tornado

14. Recovery plan

a. Continue operations in hot site or b. Activate cold site or alternate location c. Activate drop-ship provisions d. Set up new facility

a. Continue operations in hot site or b. Activate cold site or alternate location c. Activate drop-ship provisions d. Set up new facility

15. Restoration of system

a. Begin cleanup b. Complete repairs as necessary or

a. Begin cleanup b. Complete repairs as necessary or

16. Restoration of calls

Return calls

Return calls

17. Return to normal

Return to normal operations

Return to normal operations

18. Plan evaluation and update

Evaluate evacuation and update

Evaluate evacuation and update

ble getting in (this could impede the change of call center shifts). Smoke might be blown toward your building. If it were sucked into the air intakes of your building, it could force evacuation of your facility even though the fire was not taking place in your building.

Same Methods in All Circumstances This is the reason that for crisis management we suggest your plan deal with the same actions on every occasion. You should be planning for events to happen in a predictable sequence. This will aid the recovery team and other personnel to take plan actions according to checklist procedures without confusion or second-guessing. After the onset of the disaster situation, personnel would have a moment to evaluate the disaster. They would be able to see if there were be a possibility that the disaster might affect company operations, either directly or indirectly because of the ripple affect. If such an effect started to look less “possible” and more “probable,” then you would start looking at the escalation chart to see what kind of outage time would be likely. It is important that all personnel use the same methods in all circumstances. This includes the planning team when responses to possible situations are planned out. This includes the recovery team and subteams when they meet the challenge of the disaster and respond to it. All events should be handled in the same fashion when managing the escalation of emergency response measures. First, the personnel have a look at the disaster and decide whether or not the event is worth declaring as a disaster. Some of these decisions are pretty easy. If your building starts shaking to pieces around you, then the decision of declaring a disaster happens almost instantaneously. The major disasters are reasonably easy to make a decision on, so the chart is intended for helping with the less obvious decisions.

DISASTER CHECKLISTS You will notice in the disaster checklists like the fire checklist given in Exhibit 12-9 that the employee who first notices the disaster situation has specific actions to perform. For example,

© 1998 by CRC Press LLC

please examine the first part of the bomb threat checklist, Exhibit 20-3 (Multiple Problem Exercise Dependencies). The employee has a number of things that he or she tries to glean from the conversation. This is something that the employee does not do on an everyday basis. That is why it is important that all personnel be trained to grab their checklists. As soon as they suspect that something is going amiss, they should try to figure out what information they will need to deliver to the recovery team. In a bomb threat situation, the checklist prompts the employee to note what the caller sounds like, whether the caller knows the building, or whether the caller sounds like he or she is faking being familiar with the building. A case in point would be if someone phones in and says that there is a bomb on the fourth floor. If there are only three floors in your building, then there is a good chance that the call is a hoax. Suppose that somebody calls and claims that there is a bomb somewhere behind the main supervisor. If the caller names that supervisor, then it can be reasonably assured that this is probably more serious than a mere hoax, because the caller has knowledge of the facility. In Exhibit 12-9, the first person that notices the event sounds the alarm. This brings us to the second phase of the crisis, which is the alarm.

First Person to Notice Sounds Alert All employees must receive training so that in the event of discovering a disaster situation (or potential disaster situation) they would know the procedures to sound the alert. The more people who know about that requirement, the less likely that a potential danger could exist “unknown” and ignored by all but a few. It has happened in the past that people have noticed an event like a fire, and just left the building. We have heard of this happening. That is fine for the person that exited, but everyone else is left ignorant that there is a potential danger. People have to understand the procedures involved in notifying management or emergency authorities of situations. Please examine the fire checklist given in Exhibit 12-9. Most checklists for CSRs or other nonsupervisory personnel should include a preliminary information-gathering phase, and an “alert” phase, the first action of which is notification of the person’s immediate supervisor. That supervisor would usually be tasked with alerting the call center manager, who in turn would alert the recovery team.

Alerting Recovery Team The call center manager or senior supervisor on duty at the time would commence alerting the recovery team. Usually it would be the on-duty recovery manager who was alerted. This person would begin an assessment of the situation. If your recovery team were unavailable at the time, the individual would alert whoever could be contacted. The on-duty recovery manager is the head of the recovery team at that point in time. Other call center personnel would be assigned to begin notifying other members of the recovery team.

Assessment The on-duty recovery manager would do the assessment of the situation with the aid of the recovery chart. Each event should be rated according to the response level. The first person must assess the situation in accordance with the chart and determine whether to declare a disaster or continue normal operations.

© 1998 by CRC Press LLC

EXHIBIT 12-9. FIRE CHECKLIST (CSRS) 1. If you detect the fire: a. Sound the alarm by • shouting “Fire, Fire, Fire” • pulling nearest fire alarm b. Ensure immediate supervisor is aware of the situation c. Report on all details of the alarm to your supervisor 2. Upon hearing a fire alarm: a. Determine if it is an evacuation alarm (in some buildings two short rings indicate not to evacuate but to stand by for further instructions) 3. If you must evacuate: a. Exit the call using emergency scripting and exit screen b. Exit building using nearest stairwell (do not use elevators) c. Report to your predesignated meeting point d. Report to your immediate supervisor that you are present Remember: In any fire situation, remain calm and do not panic. 4. Upon returning to your station (only when directed), a. Determine if your station has been damaged b. If no damage is apparent, use start-up procedure 3 when directed 5. If sent to a hot site, secure a hot site CSR checklist and follow the instructions 6. If sent home, report to phone number 1-888-555-help and await a return call More checklists are given in Chapter 18.

Declaring a Disaster The on-duty recovery manager decides if the situation is a disaster, by using the chart. The next step is to take the disaster to the next stage and declare the disaster. This seems to be the most obvious part, but many companies get it “wrong.” The manager who fears making the wrong decision will, in the end, make no decision at all. Some events move so quickly that they do not allow managers to postpone the decision of whether or not to declare the disaster. In testing, we recommend that your company test the decision of whether or not declare a disaster to the point where folks get very comfortable it. The consequence of the on-duty recovery manager or even the incident manager floundering for minutes or quarter-hours without being able to make a decision could be severe.

Enacting the Plan The recovery team should emerge from the testing-and-training phase, knowing when it is time to declare a disaster. Then they will have confidence when the time comes to enact the plan.

Following the Plan The important thing is to follow the checklist according to the disaster that is happening. This would include practice in evacuating the employees, getting the backup online, and other components basic crisis management (see in Chapter 23).

© 1998 by CRC Press LLC

To recapitulate, the basic stages of a disaster are as follows: 1. Event — Something happened 2. Alarm — First person follows event checklist 3. Assessment — Supervisor informs call center manager, follows checklist 4. Notification — Call 911 — call center manager informs on-duty recovery manager (checklist) 5. Declaration — On-duty recovery manager declares disaster and enacts plan 6. Evacuation — Safety of personnel is the primary concern

LIFE PRESERVATION People First The most important thing a disaster plan can do is to keep people alive. In your disaster planning, think people first. This cannot be overemphasized. Think hard about how to get your people out of the facility alive.

Training Takes Over Training takes over in case of a disaster. People should train with the applicable checklist in their hands, so that during a disaster it would not be unfamiliar to them. Training and testing exercises should be as close to the anticipated events and actions that might occur during a real disaster as possible. Personnel who have practiced an action many times would not be likely to panic and lose their heads under the pressure of an emergency. Case Study: An explosion near a remote military base of a natural gas pipeline occurred in the middle of the night. It made a loud roar that shook the ground and threw children out of their beds. The roar did not stop, because a 900-foot (300-meter) tower of fire was thundering at such a volume that it had to be shouted over. The cold night air heated up to a summertime noon temperature — in the tropics. The volunteer firefighters, and other emergency personnel of the community, all did the tasks they had trained to do. This was because these people performed the very same steps in the same order every time they went to work. The firefighters went around hosing down all the buildings, to prevent them from catching fire from radiated heat. In the meantime, the fire chief worried over the question of whether or not to evacuate the community, which had only one exit — one that was at risk of being blocked if the fire should ignite the surrounding forest. Finally, the natural gas–pumping station supervisor contacted the fire chief and informed him that the cutoff valves were being closed, and that the fire was anticipated to die out slowly over the next four hours. A team of firefighters determined that the area around the site of the explosion was normally a swamp, so there was little likelihood that the forest would be affected. Afterward, it was determined that the firefighters (all volunteers) had all done their jobs properly. They had training and practice in facing the heat, noise, and smoke of fires (even though the pipeline explosion was much bigger than any they had previously faced.) The fire chief was experienced in assessing fires and planning to safeguard human life. The other emergency personnel (such as police officers) all did their jobs properly because setting up roadblocks and other emergency activities were tasks that they were trained and experienced in doing.

© 1998 by CRC Press LLC

However, most of the people who were not part of the emergency personnel panicked. They did not have the training and practice in dealing with such events that could keep them in control of themselves. There was nothing to prevent them from running away at top speed, so they did!

First Aid Life preservation planning and the disaster plan should include training in first aid for key personnel — or all personnel, if you can afford it. While evacuated personnel were gathering at the predetermined meeting place, those trained in first aid would be able to make any injured personnel comfortable and identify who might be at risk of having a heart attack. This would be particularly important in a wide-area disaster, when official emergency personnel might take a long time to arrive.

Evacuation and Rescue Evacuation training should be done with all personnel and rescue training should be done with key personnel, particularly your recovery team. If you cannot convince the CFO to hand over calls to the ICH facility just for training, consider holding the training after the next call center shift has come on the job. A small expense for overtime pay could save a lot of grief in case of a life-threatening disaster. Consider asking your local fire department for training aid. Most such departments would be more than pleased to help your company prepare and train.

Call Switching from the Parking Lot (Get Safe First) One of the values of the “ready to go” case is that having it allows you to do call switching from the parking lot. If something happens very quickly (such as, the room begins to fill with smoke) you can just grab the “to go” case and get out of the facility. You and your team can make your recovery plan happen from the parking lot if necessary. Remember that the evacuation plan includes the call center manager and the incident manager, too. No one should feel the necessity to stay (and face physical jeopardy) just for the sake of a few hundred or thousand telephone calls or e-mail messages. The virtue of the Telco-coded password for prerouted switching is that it can be done from anywhere. The incident manager does not have to be standing at the call center manager’s desk, or in the Telco room, to do it.

When the Whole Recovery Team Is Incapacitated If the whole recovery team should become incapacitated, all personnel should know where to grab the checklist that tells how to activate the recovery call center. All Personnel Know How to Contact the Assisting Call Center. The “to go” case

should be the subject of training with supervisors and other senior personnel, so that any of them can get the communications handed over and keep the company critical business functions operational. If all supervisors and senior managers of the company were stricken down (for instance, by an “after-lunch illness”), then all junior personnel should at least know how to call an offduty supervisor at home. Should junior personnel be able to call the assisting call center and ask that facility to receive a handover of calls without sending a coded password to Telco? We would recommend against that. Communications security must be enforced to safeguard your company from any (unlikely but possible) disgruntled employees.

© 1998 by CRC Press LLC

DISASTER PLAN SURVIVABILITY Likelihood of Being Affected by the Same Disaster. Can lightning strike twice in the same place? Yes. Can a river flood your area twice? Yes. However, you can take precautions by adding lightning rods or semipermanent flood barriers to minimize damage another time. Is the same piece of equipment likely to fail again? That depends on whether there is a design or software fault. Otherwise, it is unlikely. If necessary, you can pursue the matter with the vendor. Would the same problem cause the carpet to smoke again, or the company managers to delay so long in making the decision to recover to a hot site? Probably it would not. This is the purpose of updating your plan and checklists. If the cause of the disaster is something that can be controlled by human ingenuity, your company does not have to be a sitting duck for it again. Single Point of Failure. In this book we address single points of failure when it comes

to selecting Telco lines to ensure that you do not have any lines that could be subject to a single point of failure. This is a good philosophy to adopt for the rest of the plan as well. No part of your plan should be vulnerable to a single point of failure rendering it inoperable, whether that single point be infrastructure, location, building, transportation system, or personnel. Your continuity plan should provide for redundancy in equipment, location, infrastructure, and even personnel. You do not necessarily have to invest in multiply redundant systems. Knowing what to do, or where to go, if the plan does not work out the way it should sometimes is enough. What Happens If Backups Fail? If the current backup of data does not open at the ICH facility because it was damaged in transit, the assisting call center could use the “last backup” information. Although not everything might be “up to the minute,” most customers would be served. If the switching of calls to the ICH or hot site fails, the incident manager would just call Telco and tell it to try again. This is the purpose for sending a test message to establish that the communications link is serviceable before transferring the entire call volume over. If the situation were a “smoking hole” disaster, even if several minutes worth of calls were lost, the company would probably survive it. The important thing would be to restore communications as soon as possible. The failure of one attempt should not cause anyone to panic. This is the reason the plan must be regularly tested. All parts of the plan, including backup and switching, need to be proved operational in case of a real disaster. Battered but Not Beaten. Suppose that a real disaster has come along after all that planning, and all those tests and training exercises. Your personnel have all been evacuated safely. The ICH facilty is answering your calls and receiving credit card payments for sales orders. The fulfillment outsourcer is shipping goods to fill those orders. The hot site recovery team is hard at work getting a temporary site ready for your personnel. The disaster site recovery team is hard at work arranging for repair of the building and replacement of equipment. At this point, the “disaster” is still not over. However, you and your team can relax a little. Your company survival has been safeguarded. The company is battered, but not beaten, and all the company personnel will still have jobs tomorrow.

© 1998 by CRC Press LLC

13 Return to Normal Operations

R

eturning to normal operations is one of the most difficult parts of a disaster. In this chapter we are going to walk through some things that might have happened during a disaster. We will discuss how to get the company call handling back to normal operation. In this example, “you” are acting as the incident manager. Let us assume that during the first 20 minutes of a crisis, you advised the real-time minimum overflow (RTMO) facility and the interim call handling (ICH) facility that they were about to receive calls. After alerting the ICH facility, you informed the carrier that the calls were going to be switched to the two facilities. After the switching took place, you confirmed that it had all happened as planned, and that the companies were receiving the calls. You sent your more difficult calls to the RTMO facility. The RTMO facility had already been receiving some of your calls, and you confirmed that they had received the additional calls and were dealing with them. You then took your people to a hot site location. The first thing that the hot site recovery team did at the hot site facility was to bring up the call-taking capability. This included all the software and scripts that would normally be in use. You then started taking calls back from the ICH and RTMO facilities. In addition, you had the ICH recovery team monitoring what was going on at the other two locations, to ensure that the process went smoothly. Our example assumes that a day or two has passed, during the course of which you discover that your primary facility has been completely destroyed. You and your recovery team begin the long process of returning your company to normal. If your original facility had not been destroyed, it would be a matter of getting a company into clean up the damage. Then your company could resume operations again at the primary facility. The assisting call centers would hand back the calls in an orderly fashion. Suppose that your hot site team had started handling all the calls at the hot site location. You would notify the RTMO facility and the ICH facility that you were going to begin handing calls over to them again. You would notify Telco, and the calls would be sent to the assisting call centers. This would enable you to transport the hot site recovery personnel back to the original facility. But let us move back to the fact that you and your recovery team migrated to the hot site and you have restored your calls (received hand back) there. Now that you know that the primary (disaster) site is destroyed, you have a problem. Most hot site facility providers will not let you stay in their facility for more than 45 days.

COLD SITE Too Slow for Emergency Use Since your originating facility was destroyed, you are going to have to start looking for alternate facilities. This is where a cold site is handy, although it is too slow to ramp up in a normal emergency. Having a cold site is useful in a long-term recovery from your hot site or your other call-handling facilities.

© 1998 by CRC Press LLC

Useful for Long-Term Recovery from Outsource Call Center If you had a cold site, then it might be possible to turn that cold site into a permanent location. You and your team would then migrate from the hot site to the cold site. You would turn it into your regular call center. If you did not have an alternate location or cold site picked out already, then you would have to get a team together to select a new facility and begin to get cables installed. Remember that you and your team would only have 45 days to accomplish this.

ALTERNATE LOCATIONS FOR PERMANENT/TEMPORARY RELOCATION Precontracted Premises You should contact a realtor in your current city to help you and your team search for a permanent or temporary location in case of need. It would be a good idea to make a list of sites (ten or eleven vacant facilities) that would be available at short notice. You should assign someone on the recovery team to keep a list of vacant sites and update it about every two months. That person should ensure that the ongoing list was kept with your disaster recovery plan binder, with a copy in the “to go” case. It is possible to set up precontracted premises and have an emergency vendor list of landlords who could make their facilities available to you in the event of a disaster. If the landlord or owner of a superior site (that met all company needs very well) agreed, your company could if it chose make an “option to lease” agreement in case of adverse contingency. Such an agreement would be duplicated and the original document stored off site.

Similar to Building a Cold Site The emergency vendor list would also be stored with the plan. You and your team could refer to it to get all the support systems that such a facility would require installed. Starting work on an alternate site would be like building a cold site. You would be starting with the basic building, which would not yet have any equipment installed. Nor would there be any telecommunications or power lines going to the facility. In the Exhibit 13-1 we have selected a number of categories that you should consider when you are creating your emergency vendor list.

Dependence on Assisting Call Center If your facility were destroyed, your company would have to depend on the assisting call center(s). This would include the ICH and/or RTMO facilities, and also any other call centers internal to your company. Here is where preplanning could make all the difference. You should make sure that the ICH and RTMO service-level agreements that you put in place provide for long-term service if necessary. This is so that the outsourcers would continue to take the company calls during the entire restoration time. Then they would not be able to say that a preestablished time limit had been reached and they could no longer accommodate you — leaving you high and dry.

Assisting Call Center Maintains Company Cash Flow You should set up some sort of assistance for your fulfillment. There is a list of companies in the Disaster Recovery Yellow Pages that provide this service, so that you can set up fulfillment arrangements in advance. This would ensure that, during a disaster, the sales orders taken by the ICH facility could be fulfilled and goods shipped to your customers.

© 1998 by CRC Press LLC

Exhibit 13-1. Categories for Emergency Vendor List Selected from Disaster Recovery Yellow Pages. Actuaries Air Ambulance Charter & Evacuation Appraisers Architectural/Engineering Services Art Restoration Audio Tape & Record Restoration & Preservation Ballast Recyclers (Lighting) Banking Equipment Repair Building Materials, Bulk Quantity Brokers Burglar & Fire Alarm, Consultation & Design Cleaning Companies, Data Processing Site Specialists Cleaning Companies, Electronic Equipment PCs on site & off Site Cleaning Companies, Insurance/Disaster Specialists Cleaning Companies, Medical Cleanup: PostCrime Cleanup & Restoration Computer Repair Specialists Concrete & Masonry Repair Conservation: Art, Library, & Museum Construction, Emergency Cooling Services Crisis Management Services Data & Records Recovery Data Backup/Recovery Software Data Distribution Service Data Recovery Data Recovery from Broken Hard Disks Data Recovery from Damaged Media Data Recovery from Damaged Tape Data Recovery from Optical Disks Dec Equipment Repair Decontamination Equipment, Portable Detective Agencies Disaster Specialists Disk Access Control Dispatch Software Disposal, Electrical Equipment Disposal, Hazardous Waste Dry Cleaning & Laundry: Specialty & Emergency Drying & Dehumidification Education, Training & Awareness Electric Heaters, Dehumidifiers Electronic Vaulting Emergency Information Location Emergency Management Software Emergency Response Logistics Emergency Staffing Software Environmental Cleanup

© 1998 by CRC Press LLC

Erased & Damaged File Recovery Utilities Facilities Preparation for Data Processing Equipment Fire & Water Damage Restoration Fire Protection Flood Damage Restoration Food, Lodging, & Sanitary Services, Emergency Food, Lodging, & Sanitary Services — Mobile Forensic Accountants Forensic Engineers Full-Service General Contractors: Restoration & Reconstruction Furniture Restoration Glass Board-Up Government Services Guard Services Hard Disk Repair Hazardous Material Management Software Hazardous Materials Heating Services Housing, Temporary IBM PCs & Compatibles — Repair Incident Management & Planning Software Information Integrity Verification Software Information Management & Protection Instrument Repair Insurance Insurance Adjusters, Independent Insurance Adjusters, Public Insurance, Special Situations Item Processing Jewelry, Fire Damage Repair Jewelry, Replacement Laser Printer Repair Legal Issues, Environmental Legal Issues, Insurance Life Support Services & Supplies Lighting, Emergency Lighting, Solar Generated Liquidators/Salvage Log-Off, Automatic Loss Prevention & Investigation Macintosh — Repair Mailing, Printing, & Inserting Services Maintenance Services, Computer/Networks Microfilm Drying & Re-Wash Microfilm Drying & Re-Wash Service Microfilming & Record Restoration Mirror-Imaging Software/Hardware Misc. Peripherals Repair Missing Source-Code, Recovery

Exhibit 13-1. Categories for Emergency Vendor List (continued) Mobile Administrative Offices & Command Centers Mobile Bank Branches Mobile Buildings & Teller Facilities Mobile Cargo Storage Space Mobile Classrooms Mobile Communication Centers Mobile Computer Rooms & Data Centers Mobile Computers Client-Specified Configurations Mobile Computers Hewlett Packard Mobile Computers IBM — Midrange Mobile Computers Wang Mobile Computers, Delivered On Site Mobile Disaster Response Trailers Mobile Homes Mobile Satellite Communication Dishes, Emergency Museum Services Network Backup/Recovery Software Network Census & Inventory Software Network Problem Analysis & Management Network Recovery & Recovery Planning Notification & Dispatch Services, Phone, Fax, Pagers Office Equipment Repair Office Furniture Restoration Off-site Record & Hard-Copy Storage Off-site Tape, Film, Optical, etc. Data Storage Online Software Backup/Recovery Outside Audit of Disaster Plan Adequacy Outside Evaluation Audit of Disaster Plan Adequacy Overspray Removal Services Password Management & Recovery Utilities Personal Computer Repair Power Protection & Continuity Power-Loss-Detect-&-Shutdown Utility Software Printer Repair — General Printing, Emergency — Business & Bank Forms Psychological Services Crisis Management Psychological Services Human Factors Psychological Services Trauma Counseling Public Relations Records Management Recover Water-Damaged Books & Documents Referral Service, Disaster Cleanup Remote-Operation Software Restoration: Audio Materials Restoration: Video Materials Salvage & Surplus Buyers Satellite Dishes, Mobile Scrap Computer & Banking Equipment — Buyers Scrap Fluorescent Lamps & Ballast — Buyers

© 1998 by CRC Press LLC

Scrap Hard Drives — Buyers Search & Rescue Secretarial Services, Temporary Security — General Security & Access Control Security Systems — Repair & Replacement Site Remediation Smoke & Odor Counteracting Products Smoke & Odor Counteracting Services Snow Removal Plows, Trucks & Equipment Steam Cleaners & Pressure Washers Stolen Computer Equipment, Registry Subrogation Services Sun Equipment Repair Surge Protectors Surplus Computer Equipment Buyers Surplus Computer Equipment: Charitable Donations Switches, Ruggedized & Specialty Tanks, Storage Tape (Cartridge) Testing Systems Tape Backup Units Tape Backup Units — Portable Tape Drive Repair Tape Library Management Systems Telecommunications Telecommunications, Specialty and Nationwide Coverage Telecommunications: Lost-Power Bypass Units Telecommunications: Wireless & Fiber Optics Telephone Answering Telephone Dialing Equipment Automated Telephone Supply Transformers, Electric Transportation, Specialized Trucking Turnkey Data Processing Site & Service Restoration Underground Rescue UPS — Uninterruptable Power Supplies Videotape Restoration Virus Detection/Prevention/Recovery Voice/Call Center Recovery Warning Systems Water Damage Restoration Water Testing Equipment Weather Forecasting Weather-Tracking Software, PC-Based Wireless Telecommunications Backup Workplace Violence Wrapping, Protective: Buildings & Structures Year 2000 Software Problems: Analysis & Solutions

This would assist your company in maintaining cash flow. The ability of our company to meet its obligations, and pay the costs of recovering from the disaster, would not be diminished. Cash flow might be very difficult during this time in any case, especially if a wide-area disaster had taken place. It is probably a good idea to bring your banker into the planning process, too, so that the bank can provide any additional services (such as acceptance of accounts receivable from the ICH facility location) that your company might require. Your recovery team accounting and fulfillment (shipping) representatives should work on this as part of setting up the plan to recover critical business functions.

RESTORATION PROCEDURES Ensure Power, Telephony, Supporting Systems As part of your restoration planning, we recommend that your company should have prearranged contracts with cleanup and restoration companies. Such companies would come into your primary facility after a fire or other disaster. They would do restoration of the building and furnishings, and clean up the computers, either on site or off site. If necessary, your company could contract with companies that do decontamination in the event of a chemical spill or a hazardous waste spill. Other firms perform water or flood damage restoration. Some companies offer turnkey data processing site and services restoration, recovery of data from damaged magnetic media, and drying and dehumidifying of paper records. There are emergency equipment sources including providers of cash registers and point-of-sale devices, equipment repair, power protection devices, and used and rental computer equipment dealers. Contact information for these vendors can be found in the Disaster Recovery Yellow Pages.

Personnel If you were restoring your original site after a disaster, you would have to ensure that the site was safe for and usable by people. For example, if bringing people back on site, you would have to ensure that the building had been cleaned sufficiently and that the equipment would work. Opening a building that had been through a fire, where walls, windows, and furniture were covered with smoke residue, would not be healthful for personnel.

Information Systems If you were going to a new (alternate) facility, then you would have to start from scratch. You would have to ensure that your power, telephony, supporting systems, information systems, and personnel were working. If you were restoring your original site, your IS disaster site recovery team would have to work with restoration vendors to clean, repair, and test all hardware. Depending on the type of disaster, the software and data backups on site might be undamaged, dirty, corrupted, or damaged beyond repair. Before deciding whether to spend the money to clean old information systems hardware of your company, you might want to contrast the cost of getting new replacements. (The recovery team representatives from accounting, IS, and purchasing might want to gather information beforehand and possibly enter into a vendor service-level agreement that established prices in advance.) As for the application software and backup data, your IS team should check the condition of duplicates and backups held in off-site storage before leaping to have magnetic media from the disaster site cleaned.

© 1998 by CRC Press LLC

Telephony Preplanning could save time if your company needed to order new telecommunications lines and equipment. The recovery team members for the accounting, IS, and purchasing departments could save time by arranging for personnel and companies to do this in advance. Vendor service-level agreements with Telco and telephony suppliers would be useful whether your team were ramping up a cold site or restoring the original (disaster) site.

SYSTEM RESTORATION Technical Personnel Your IS team members would have to restore the information and telephony systems. Whether at the primary (original, disaster) site or at a new alternate (cold) site, they would have to do a lot of work in a short time. This would include installing any new equipment that was quick-shipped from your vendors (as was prearranged), as well as performing LAN restoration, testing, or repair of the server and desktop computers, and loading or reloading all software and applications. Even if we were talking about new equipment at a new site, we would still term the action restoration because we are describing restoration of the company communications and critical functions.

Equipment Fitness Whether the equipment were brand new units fresh from the vendor, equipment that had gone through the disaster and been repaired, or equipment that had emerged from the disaster apparently unharmed, the IS technicians would have to test it. Was the equipment damaged in shipping from the vendor or during the trip from the old site to the cold site? (from original to alternate site, that is). Was the equipment properly repaired? Was it really unharmed, or was that just someone’s opinion? To test the equipment against technical specifications, the IS technicians would require their tools, instrumentation, and the relevant manuals and checklists. If they were doing this at the disaster site during the course of restoring it to operation, all these items would have to be either unharmed or replaced in order for the technicians to do the work. What if the technicians were testing the equipment at a cold site, prior to opening it up as the alternate midterm or permanent company facility? In that case, they would need either to have brought the equipment from the disaster site in the IS “to go” box at the time of the evacuation, or they would need to have replaced the tools and instrumentation with the help of a vendor. The manuals and checklists could be replaced, if necessary, from the duplicates kept in off-site storage. However, that would only be an option if the recovery team had had the forethought to store those items off site in the first place. Every item of equipment would have to be tested against its specifications to ensure that it was functioning properly before the IS team could take the next step and begin restoring the applications software.

LAN RESTORATION Server Once the IS team has confirmed that the equipment is all functioning according to specifications, the next step is to reconfigure the local area network (LAN). The first piece(s) of

© 1998 by CRC Press LLC

computer hardware that they would give their attention to would be the server for the LAN. If the facility were intended to function with redundant servers (to have a robust system more likely to be safe from equipment faults), then they would have to prepare more than once. The technicians would load the server(s) with the operating system, networking software, and other basics of the system as set out by the MIS. Then they would employ diagnostic software to confirm that everything was working properly.

Desktop Computers Part of the process that the IS personnel would perform would be setting up and confirming by diagnostic software that the desktop computers at all the work stations in the facility were functioning properly. This extensive procedure would apply to computer terminals at all the work stations in the call center first, followed by those used to perform the critical functions of the other departments. At this point, any work station computers allocated for noncritical functions would be at the far end of the work order list.

Booting Up Once the server(s) and all the critical work station desktop computers are connected and loaded with the operating system and network software, the IS technicians would bring up the system to confirm that the LAN was operating properly. It is at this point that the MIS and the technicians would probably assign a security password to the system, to replace their informal “installation” password of the previous period. Once the LAN was confirmed to be operating properly, the technicians would commence loading applications software onto the desktop computers for the various departments.

Applications The IS team would load word-processing, accounting, database, and other dedicated software onto the various desktop computer terminals. At this point they would probably also load antivirus-checking software to ensure that the newly loaded operating system and application software had not been contaminated. Once the “critical application” software was loaded, some work stations of recovery team members from the various departments could be used to keep track of the recovery effort. This would include the CFO or recovery team accounting representative keeping track of revenues from the ICH facility, shipments from the fulfillment outsourcer, and purchasing for ramping up the cold site or restoring the primary site, whichever the case might be.

CALL VOLUME MANAGEMENT RESTORATION Call Management Software Is Operational The whole purpose of ramping up the cold site (or restoring the primary or disaster site) is to provide the support systems for the call center. The call center, with its complex interactions of computers, telephony, and call-taking personnel, is the profit center for the company. Every day of delay before the call center facility could be reopened would expose the company to more expense in outsourcer fees.

© 1998 by CRC Press LLC

The computers associated with the call center critical functions, including the ACD and IVR units, would have to be reloaded with the call management software. This includes the scripts and other programming for the IVR. This would include the customer and sales databases, the pop-up screens for the CSR screen prompts, the timers for callback appointments, and the CSR comment subroutine for the customer database. It would also include the diagnostic software for the call center manager and supervisors. This is the software that keeps track of average time per call, and the number of transfers or customers put on hold. It would also include software (if management wanted it right away rather than installing it once the call center was operational again) for graphing Telco reports itemizing the number of calls blocked or abandoned.

Telco Confirms Connectivity Once the facility was ready to go (as far as equipment and application software ramped up), the recovery team would commence the preplanned procedure for recovering call volume from the ICH facility and RTMO facility. This procedure should be laid out with Telco and the assisting call center(s) in advance. We recommend that the procedure be planned in stages, during the night or lowest call volume period, as follows: • The crew of IS technicians, call center manager, and recovery team CSR representatives would confirm operability of the LAN, call volume managemnet (CVM) applications and telephony equipment. • Telco and IS representatives would cooperate to trade test messages over the Telco lines. • The call center manager would authorize partial return of call volume from one of the assisting call centers. • What if the call volume recovery team could not receive calls after a certain time that night or low volume period? In that case, the call center manager would be advised by Telco and the assisting call center to close down the recovery attempt. The recovery team would have to attempt to reopen the next day, or in the next lowest call volume period. • Telco and the team IS technicians would troubleshoot the call center equipment and lines to find the fault that was keeping the call center from returning to operability. • Once the trouble was found, Telco and the technicians would again trade test messages. At this point, a company such as IQ Services (see Part 4) could do testing with simulated messages to prove the system serviceable. • Another attempt would be arranged with Telco, the recovery site team, and the assisting call center. If a partial handback of call volume succeeded, the call center manager would (in conjunction with Telco and the assisting call center) schedule a staged return of call volume. • The call center manager would request the call center supervisors to recall CSRs on a shift schedule complementary to the ramp up of call volume.

Alerting Assisting Call Center Once the recovery team in the new (or restored) call center had confirmed that they had received the calls, then the call center manager would inform the assisting call center that had handed back call volume that the outsource personnel could then go on standby. That first outsource call center to hand back calls would remain on standby in case anything goes wrong. If some glitch were found in the new (or restored) call center system, calls would have to be handed back, and the trouble shooting process would begin again.

© 1998 by CRC Press LLC

Switching Call Flow Telco, the IS technicians, the call center manager, and selected CSRs would go through the whole process again to prove connectivity and recover call volume from the one or more other assisting call center(s) that were handling the company calls. If the first assisting call center to hand back calls were the ICH facility, the next to hand back should probably be the virtual call center composed of telecommuters, probably CSRs from the primary call center. We recommend this because these would be people handling calls from their homes, which would be a burden to their families. Also, since the call-takers are individuals, this handover of calls could be staged on a line-by-line basis. The CSRs who had handed back calls from their homes could then start working shifts at the new (or restored) call center, becoming familiar again with the call center setup. The next assisting call center to hand back calls would be the RTMO facility. The Telco/technician/manager/CSR team from the Telco confirms connectivity section above would go through the whole process again. The recovery team would go through this for each and every assisting call center, until the total call volume was returned to the company call center. We recommend that during this ramping up procedure, you and your team should keep your RTMO or ICH or virtual call center (whatever your quick-recovery plan is) available in case things go wrong. The standby period would last until a certain period of time had gone by (perhaps two or three days) during which the total call volume continued to be handled by the company call center personnel and equipment without glitches. At that point, the call center manager could declare the major part of the disaster over, and the ICH and RTMO facilities could relax and return to the regular services according to the usually applicable outsource contract. At the beginning of this chapter, we gave an example of a primary company site having been destroyed. The ICH and RTMO facilities had taken the call volume, we said, and then handed it to the company hot site recovery team of CSRs. Then, we noted the 45-day limit in operating from most hot site facilities. If that were the case, as the call volume was received back to the company at the ramped-up cold site, the company CSRs would be returning gradually from the hot site. If the handover of call volume was not complete within the 45 days, the ICH and RTMO facilities would be needed again, to bridge the gap between the hot site contract period and the full capacity of the new company call center facility. At this point it would not matter too much whether the total call volume were all handled by the company alternate (formerly cold) facility or by a combination of company and outsourcer resources (except for the outsourcer fees, that is). The company could relinquish its work stations at the hot site. Once voice call volume was recovered, the Telco/IS technician team would commence recovering the company fax lines, e-mail, and other communication arteries. Throughout this entire restoration process, the members of the recovery team would be providing situation reports to the incident manager. The incident manager would be briefing senior management on the progress of the recovery effort. Other personnel would be working to ramp up the various departments and support services of the company. What would come next? The company, starting with the recovery team, but including all personnel (both management and employee) should commence the postincident assessment of the disaster and the disaster recovery plan. See Chapter 14 (Plan Maintenance Procedures). Then would come the decision by senior management whether or not to adopt the alternate site (formerly the cold site) as permanent. What if the alternate site were located near some potential danger, such as a river prone to flooding, or a petroleum tank farm? If it had been the only facility available at the time of the disaster, it would be good enough for a temporary facility, but not safe enough for long-term use.

© 1998 by CRC Press LLC

If senior management decided to seek another site, the whole ramping-up process would take place again. However, with company operations proceeding well at the alternate (now identified as temporary) site, the recovery team could proceed at a slower pace. This would enable it to be picky about the permanent site. Probably it would not need to use the assisting call centers for the changeover to the new permanent site — at least, not unless that river flooded. The subject of CVM is discussed in detail in Chapter 15.

© 1998 by CRC Press LLC

14 Plan Maintenance Procedures OPERATIONAL CHANGES As soon as the company has “recovered” from the disaster (the word recover meaning that communications and all other critical functions have returned to normal), you as the planner should commence a postdisaster assessment. While the whole event is still vividly in everyone’s mind, you should hold discussions to determine “how we did” during the disruption. These discussions should involve everyone in the company, but not necessarily all at once. For practicality’s sake, all departments should meet separately. Call center personnel handling live calls should make their reports after their shift. Once all departments have gathered their material, the departments should send their representatives to the meeting of the incident manager and the recovery team. Each individual’s report need not be exhaustive. One or two good sentences about operations that could be handled better would suffice. Perhaps your company might use a standardized form asking what was done well, what could be done better, and what was learned during the experience (Exhibit 14-1). These short comments, gathered into the departmental report, can aid the recovery team in updating the plan. Changes to the continuity plan include team responsibility allotments. Perhaps during the disaster an additional person was needed to do a task not provided for in the plan. For instance, perhaps a person was required to allay the anxieties of worried family members of company employees, or to compute estimated bills by calculator because of a computer glitch that the assisting call center could not handle this time around. You need to write provision for this kind of “extra” assistance into the plan, so that if another disruption should occur, the personnel for that task would be provided. In the case of data-processing tasks, you might need to provide not just a person but perhaps additional computer applications and database resources to the assisting call center or warm site. If you need to refresh your memory upon some points, operational changes that impact upon the constitution of the recovery team are also discussed in Chapter 10 (Team Responsibilities). Planning to update the plan in general (not just after a disaster) as operational changes occur is one of those good ideas that people agree to on principle. When the continuity plan project is started, everyone agrees (once convinced that company survival and his or her job depend upon it) that it is a necessity. However, you as the planner will have to keep reminding all personnel that such plan maintenance cannot be let slide. Some of the excuses given in Chapter 4 will be trotted out again, as various departments or individuals argue that they are just too busy this week, but yes, they will get to it! Do not believe them. Kindly but insistently refuse to let company personnel give into their natural human laziness and tendency to ignore the unpleasant possibilities of other disasters coming their way. Although some may grumble, insisting those personnel note changes in how departments operate as they come up will save confusion and error if a disruption should occur. In the event that this topic arises in conversation at your company, yes, a second disaster could indeed occur in the same month as one that you have already successfully survived. And yes, it is worth the time and bother of updating your plan speedily, “just in case.”

© 1998 by CRC Press LLC

EXHIBIT 14-1. EXAMPLE INCIDENT REPORT Date:

Time:

Live:

Questions

Test Name:

Answers

Action

What was the incident? (disaster type) What position were you working at during the incident? How much warning did you have? How much damage was there? Which systems were affected? Did the checklist accurately provide all the information needed to deal with the incident? If no, which areas were missed? Were the affected areas recovered within the recovery time objective? Did you have to evacuate? If so, did the checklist accurately guide you to your shelter or designated area? List the recovery options used: Did you have any switching problems? Did you have any database recovery issues? Did you have any interim call handling problems? What control measures could be used in the future to ensure this does not happen again? Comments:

© 1998 by CRC Press LLC

RTMO

ICH

Dist. Arch.

Hot Site

Cold Site

Other

In Part 6 we will describe the successful recovery of Central Maine Power and the assisting call center(s) of Twenty First Century Communications, Inc., during “Ice Storm ’98. During the first storm (of two) of this disaster of infrastructure, the work orders for electrical service restoration to the Central Maine Power subscribers had to be amended. For the first time ever, their work orders had to include a new code, invented for the occasion, to give repair priority to customers who were “trapped” by downed lines and poles blocking their doors and driveways. Changing its procedures bore fruit within the same month, when a second ice storm arrived. Because plans and procedures were updated during the first storm and recovery of January 7 to January 18, 1998, everything was ready for the second storm of January 24 and recovery through to January 27, 1998. Do not let this matter fall by the wayside.

We Do It Differently This Week Company personnel must get into the habit of making a weekly summary of operational changes. It would be wonderful if they could compile this report gradually over the week on the days that changes occur. If people can be so motivated as to willingly and eagerly make these changes as a matter of routine, your job as the planner will be much easier. What kind of changes are we talking about here? Usually these are the incremental changes by which company employees show their marvelous human ingenuity. Some of the changes will be tiny, such as noting that the master vendor list for a department is kept in a different place than before, perhaps closer to the telephone, for ease of use. Some of the changes will be truly important to the operational efficiency or even survival of the company. Perhaps someone in the sales department has noted that customers are asking for a specific kind of information about products or their accounts. Having that information available on a computer-generated screen pop would render a higher quality of service to the customers who call. The assisting call center would render a similarly high level of service if that screen pop were available there as well. You should add this kind of change to the continuity plan. Perhaps someone in the property management department has realized that records concerning the various company facilities are lacking crucial information. Suppose that the company has purchased electrical generators for two sites, and has contracted for rental electrical generation at two more. The individual realizes that no one has ensured that fuel would be available for these generators, nor does anyone on staff know how to operate them. As soon as the person has this insight, that person should immediately jot down a memo noting that fuel and training are required urgently. That memorandum should be sent quickly to the planner (you) and noted in all relevant checklists immediately. Because this change could impact upon the survival of the company, you would put this item on a fast track for purchasing and human resources action. After all, what good is it to have equipment that cannot be used? In the meantime, you would want to put a local electrician on retainer at each site to start and maintain the generators in guaranteed good operating condition throughout any disruption. Therefore, you would require the electrician (at each site) to guarantee service by providing for an alternate electrician “just in case.” All relevant checklists would have to be amended to show the interim change. As fuel was obtained and personnel trained, the checklists would have to be amended to reflect those changes as well. Amending checklists and relevant parts of the continuity plan weekly sounds like a lot of work. In reality, the changes would not be too hard to make if amending the plan was part of everyone’s Monday or Friday routine. And the cost of not updating the plan with such incremental changes could be tremendous if a disaster came along that impacted upon the function that had changed. Consider. What if the person who knew where the master vendor list was kept “this week” should be stricken in a flu epidemic? What if customers should become accustomed to calling in for certain information “this week,” but during a disaster the assisting call center

© 1998 by CRC Press LLC

could not provide it? What if power should fail, and after the incident manager discovered that the portable electrical generator could not be used, the property manager told you, the planner, that no fuel was available “this week”? A lot of things can change in even a week. Many of them are important enough to note.

Planned Changes Changeover Date Let us get back to the matter of the electrical generators, fuel for them, and training of company personnel to operate them. Suppose that you as the planner, with support from senior management, and in conjunction with the recovery team, opted for a phased approach. In Phase One, the contracted electricians would provide emergency support for company power needs. In Phase Two, the three or four individuals per site who had obtained training would understudy the electricians. Preferably, they would get some “hands-on” experience. (Yes, such hands-on training would cost money, but would it not be worth it to know the personnel could really perform the task?) In Phase Three, fuel for the generators would be delivered to each site, and its location noted in the plan and checklists. In Phase Four, the contracted electricians would be put “on call” in case of problems, but would no longer have to guarantee to be available to start the generators. You would ensure that all of the phases were assigned a changeover date in the plan book (and all its copies) and in all relevant checklists. Until the noted date, the previous arrangement of the phase would be in effect. After that date, the arrangement for the next phase would be in effect. Naturally, the contracted electricians would be part of the loop. So all parties who were affected by these changes would be informed. There would not be any confusion or worrying about whether the electricians should be called or whether company personnel had the task in hand. The same rationale that pertains to the generator example would apply to any other change that could be planned for and scheduled. If it is an expected event, the date of the change should be noted in the continuity plan and added to all relevant checklists. The only time any confusion could then be expected might be at exactly midnight on the date of the change. One minute after midnight, you and everyone else in the company could expect the change to be in effect. By the way, here is another instance of the continuity plan being part of the general planning for company operations. Not only is it a benefit to the company to record changes in the plan and checklists, but planning with continuity in mind can streamline the planning. The plan provides a framework for keeping confusion at a minimum and for making each stage in the plan known to everyone who needs to know about it.

Departments Responsible to Announce When any department makes a major or minor but significant change in operations, that department must amend its portion of the plan. All checklists impacted by the change must also be amended. Verbal announcements over coffee do not count. The plan has not been amended if the department head says to you (wearing your “planner” hat) at lunch that she is thinking of making a change “sometime,” or that he thinks that “someone” should address a need that he has seen. A department has announced a change in operations only after written changes to the plan and all checklists have been written, printed or photocopied, and distributed to all affected departments and personnel. This includes the master plan binder. This includes an effective date for the change. If the change does not signal an abrupt change in policy, this includes at least the planner and members of the recovery team discussing the change. If the change does include a major change in company policy, it requires approval by senior management and an effective date.

© 1998 by CRC Press LLC

PERSONNEL CHANGES Providing for updates to operational information is only one of the plan maintenance procedures that you should set into motion. Hiring, firing, transfer, resignation, parental leave, the end of temporary contracts, death, and disability all have an impact upon every department of the company. And especially, of course, upon the call center.

Preparation to Hire Includes List Change You as the planner should ensure that the human resources (personnel) department hiring and termination checklists include changes to the continuity plan binder(s) and all relevant checklists. When new people enter the company, the company telephone directory needs to show the new employees’ names the very day (or slightly before) they arrive at their new departments. Along with their company benefits booklets, they should receive an introduction to the company continuity plan and basic emergency information such as evacuation routes and gathering points. (For more information on designated meeting places, see Chapter 22.) Contact information such as home telephone number and alternate number, next-of-kin information, and emergency skills such as first aid or CPR training should all be noted in the relevant checklists before the new employees start their first day on the job. Suppose that a tornado were seen through the windows of your facility on that first day. Once data backups, handover of calls, and the evacuation of personnel to fortified shelter was complete, would it not be a comfort to know that the departmental head counts included all employees?

“He Moved to Another Company” Mobility of the work force is a fact of life. When employees leave a company, it might be for a variety of reasons. Perhaps they are returning to school for additional training. Perhaps they are pursuing a religious vocation. Maybe they want to be stay-at-home parents or are helping with family problems across the continent. Perhaps they have left to go work for other companies, either in the local area or far away. Whatever the reason, the result is an empty spot in your company organization chart and continuity plan. Contact List. Let us return to the example of a tornado touching down near your company facility on the effective date of an employment change. A precautionary disaster has been declared, the standby backups made and e-mailed to the assisting call center(s). Calls were handed over to the assisting call center(s), and personnel were already standing by because of hearing a tornado watch report. While personnel take shelter in the fortified rooms prepared for such situations, the designated departmental representatives do a head count and compare their results to their checklists (see Exhibit A-36, in Appendix A). This is the moment when someone is likely to notice that the count is one person short. Who is missing? The (old) checklist says the missing person is the IS manager. Everyone in the shelter suffers an immediate rise in blood pressure! Maybe someone remembers that he was leaving to go work for another company. But without access to departmental or human resource files, at that moment no one can be sure if he has actually left the company or might have been left outside the shelter. What date was supposed to be his last day of work? No one in the shelter knows. The regular human resource functions include closing out the employee’s personnel file, paying vacation pay, and arranging for closure or portability of the employee’s pension and other benefits. This is the time when the master plan binder, all copies of the plan, and all plan

© 1998 by CRC Press LLC

checklists should be amended to remove the ex-employee’s name and contact information. Furthermore, all these changes should occur, if possible, before the person’s last day of work. In the event of an unexpected disaster (which most of them are!) the checklists must be as up-to-date as human ingenuity can make them. This is the moment when the plan updates and record keeping show themselves to be an essential part of plan maintenance. If the checklists were all up-to-date, or even if they contained interim notations in readable handwriting, there would be no question about the ex-employee’s last day of work. Instead of worrying about the missing person, people in the shelter could relax and cover their heads. Replacement for Plan Actions. Suppose that an absent ex-employee were a member of the recovery team, or an alternate team representative. What if he or she had some critical continuity plan task? We return to the moment when a tornado watch is announced, and the continuity plan is enacted. Precautionary data backups are supposed to be made of all critical information (see Chapters 7, 9, and 11). The accounting department has done its backup, and the call center personnel have done theirs. However, when the time comes to back up the LAN server, with its record of e-mail, faxes, and letters sent and received, among other critical functions, no one can find the recovery team member who is supposed to perform that task. Well, that’s right, someone says. The IS representative who was supposed to perform that task is not available to do the backup. She has left the building. It was her last day of work, so she went away at about noon. This leaves the company in a pickle. Because the plan binders and checklists were not amended to show her change in status and no alternate was scheduled to perform this task. Let us remember that at this moment, a tornado watch has been announced. At any moment the recovery team member who is monitoring the weather might hear that a funnel cloud has been spotted near an intersection just down the road. As the minutes tick by, someone has to be found who can perform her task. Letting someone try to do the backup who has not received proper training in how to do it without formatting the hard drive is a risky proposition and should not be attempted. If need be, the company could function with its “last backup” data, annotated by comments from the personnel who made today’s changes (see Chapters 9, 11, and 24). The incident manager (or you as the planner, depending on the size of the firm) is better off to have the alternate disaster recovery team representative paged, or wait until that person returns from lunch. Of course, this worrisome situation could have been avoided entirely if the human resources department had done its plan and checklist update promptly. Because of receiving an updated disaster recovery team schedule, the alternate IS representative would have brought lunch from home or ordered something in from a local restaurant. The backup would proceed in a planned and methodical fashion, and no one would fear that his or her multipage contract with covering letter might be lost to the void. If Not Replaced Yet (Listing Interim Substitute). Suppose that the ex-employee worked in such a specialized field that his position might be difficult to fill quickly. The company (and his co-workers) would have to deal with his sudden departure and seek to hire a replacement. In the meantime, the continuity plan (master and all copies) and all checklists showing the person’s name and tasks would have to be amended. Until such time as a replacement could be found, the alternate would have to do doubleduty. If someone in the company had prior experience in the field, that person could be deputized as the alternate disaster recovery team representative for the task. (The former alternate representative would now be the interim primary disaster recovery team repre-

© 1998 by CRC Press LLC

sentative for the position.) The new primary representative would train the interim alternate representative to return that person to a “current” skill level.

Human Resources Monitoring of Turnover Rate The human resources department would monitor numbers of personnel entering and leaving the company as a matter of course. Human resources representatives would naturally prepare monthly or quarterly statistical summaries for senior management concerning numbers of employees leaving for family reasons, going to school, departing to other companies, or transferring to other company divisions. We suggest that the human resources department also be required to issue a monthly report to you as the planner. This would allow you to discover whether the call center personnel were under increased stress lately and perhaps find the cause and determine if it might have the makings of a slow-motion disaster. (Slow-motion disaster was described in Chapter 12.)

Promotions and Task Changes All department heads, not just the head of human resources, have the responsibility to amend the plan and all relevant checklists upon promoting or transferring personnel within their departments. If an alternate was promoted into a primary position, the continuity plan should reflect that fact. If someone from accounting were transferred into another department to be the permanent alternate for that disaster recovery team position, that fact should be recorded. Naturally, the revised organization of the accounting department would have to be shown in the appropriate parts of the plan and various checklists also. If new personnel brought in at entry level gain skill and are promoted, these promotions should be recorded in the plan and checklists. Just as with effective dates of hiring or termination, promotions and transfers should have an effective date as well. Handwritten notations recording effective dates and names of interim replacements are fine, if time is at a premium. Better a handwritten note than no note at all.

HARDWARE CHANGES The purchase and installation of equipment is a hardware issue, similar to the changes in personnel described in the previous paragraphs.

Documenting New Installation Every IS department is likely to have its own local designators and procedures when cataloging equipment use and replacement. However, most companies have some sort of purchase order system. A purchase order is raised, and the purchasing department sets about obtaining a delivery date from the vendor. This can be very useful when it comes to updating the plan. When a new purchase order is generated, a reminder should be sent, noting that after the equipment is installed, a new set of procedures must be generated for the recovery plan. When the equipment is cleared for operational use, it should be recorded in the continuity plan and checklists. If the new procedures are not forthcoming within 14 days of the scheduled online changeover date, then the IS manager should be informed by senior management that this is important. The IS departmental plan for replacement of the equipment, budgetary

© 1998 by CRC Press LLC

approval, and time frame could be used by the planning team to keep track of any changes as well.

Manuals and Specifications The continuity plan needs to include the location and contents of manuals and technical specifications. The MIS or IS designate should make up checklists to be used during a declared disaster, that would show information about the brands and models of all equipment. As well, settings of the equipment and various program options should be shown in case the call center must be recovered to a cold or warm site and those settings duplicated on other equipment.

Checklist Updates Whenever a piece of equipment is installed or quarantined, the relevant checklist must show that fact, and its effective date. As quarantined hardware is repaired and returned to service, or condemned and replaced, each change should be recorded. Are technical personnel scheduled to give training to personnel from various departments in how to use a piece of equipment or software? That fact should be shown somewhere in the plan, so that an upto-date picture is available at all times of exactly what equipment is in use, what its condition is, and who in the company is skilled in its use.

Plan Book Supports Insurance Claim (Damage or Destruction) All this recording of technical information is not intended to keep IS personnel from their pressing and important technical duties. However, in case of a fire, earthquake, or even a humble fire sprinkler discharge, the continuity plan book and checklists would be available to document the company insurance claim for repair or replacement of equipment. Keeping the plan and checklists up-to-date is important work. Consider the classic “smoking hole” disaster, where a local or wide-area disaster has destroyed a whole building or a multibuilding campus. In such a situation, the continuity plan binders and checklists clutched in the hands of evacuated recovery team members might be all that remains of the company files at that site. We would trust that one or more assisting call centers would be handling the company calls and critical business functions at that point. We would hope that the off-site storage of important data files and copies of crucial documents would allow the company to resume its own operations at a cold or warm site in accordance with the procedures outlined in this book. Nevertheless, the insurance adjuster for your company would appreciate being able to refer to an inventory of equipment brands and models. Without such a list, the claims adjuster would find it difficult to identify the twisted metal frames, or charred and melted plastic moldings, that might remain in the former computer room or call center telecommunications room or be found amid the ashes of work station desktops.

Boot-Up and Maintenance Procedures What steps need to be followed, and in what sequence, exactly, to boot up the LAN or other equipment after a power failure? How would technical staff be able to know if a given computer terminal were “hung up” (needing to be rebooted) or was out of service because of a burned fuse or other malfunction?

© 1998 by CRC Press LLC

What tests should be performed during monthly or quarterly maintenance? What pieces of equipment have already been tested this month, and what equipment has yet to be examined? Probably the members of the IS department would say that they know their business. None of them would every power up or power down any equipment in improper sequence. We would hasten to recognize that your company technicians must be competent or your company would not have hired them. However, in the stress of a crisis, sometimes people forget crucial steps. Sometimes, they might start a procedure, be distracted by shouting or screaming nearby, and lose their place in a complicated series of tasks. Perhaps the technicians following the continuity plan for computers and telecommunications equipment might be overcome by smoke, and have a hard time thinking clearly. In such a case, they should, according to the continuity plan, be evacuating the area and leaving the computer shutdown to another time! Perhaps there has been a storm, earthquake, epidemic, war, or riot, and the usual person is unavailable to perform the task. What if the only technician still conscious is the summer engineering student who joined the department last week? We have heard of incidents during wide-area disasters when the “usual” technicians had evacuated the area. That left partially trained junior personnel on duty, who perhaps did not have the funds to flee out of the region. So the junior technicians were the only ones available to power down and power up the very expensive computer equipment upon which various companies depended for their data-processing and computer–telephony integration (CTI). How much money would it cost to replace all that (overheated and burned out) equipment? How much money would the vendor charge for sending its own technicians into a disaster area to figure out the problem? The boot-up/power-down checklist could save the company a lot of time and money, depending upon the circumstances. The equipment maintenance schedule could save the IS department from having to test every piece of equipment in the same week, if other records were lost or destroyed.

The Essential Person The disruption or disaster might be large or small. Suppose that a flu epidemic struck down all the technical personnel of one facility and alternate personnel came in from a branch or division across the continent. In that case the continuity plan and checklists would be the starting point for those visiting replacements to determine what tasks were pressing, and what equipment might need testing or repair. The problem rendering the IS department ineffective need not be an epidemic or widearea disaster. What if the person in charge of most maintenance and repair carries all the information in his or her head? If that person should suffer appendicitis or break a leg while washing windows at home, the company would have to go without the services for a week or more. Suppose that this person supervises junior technicians. If the person is in the hospital and out of communication, what a difference to the fortunes of the company (in case of a disruption) if the critical functions of the IS department were recorded in the continuity plan and essential checklists! The “essential person” effect is even more pronounced if you as the planner work for a small or family company. Everyone, but especially that really important person you absolutely need, must take the time to describe critical tasks in the continuity plan and checklists.

© 1998 by CRC Press LLC

Part 3

Call Volume Management

T

his section contains information you are unlikely to find in any “disaster planning” book not focused primarily toward companies operating call centers. Moreover, the call volume management (CVM) approach could be termed the “cutting edge” of call center continuity planning. This approach treats the volume of telephone calls like a fluid, which can be channeled from one place to another. Computer-managed telephone switching is like the system of aqueducts, which allows the “flood” of incoming calls to be managed without allowing your call-takers to be swept away. The CVM approach is valuable both in emergency and “normal” situations. The seven chapters of this section discuss CVM issues and strategies. Should you use external CVM solutions such as preestablished procedures for disaster call routing? Or, should you use internal solutions such as planning for a company cold site, virtual call center, or use of the excess capacity of your other call center(s) during an emergency? What about managing call volumes to cope with nonemergency seasonal or time of day peaks? Can you use your call center’s “minimum daily overflow” of calls as a benefit rather than a problem? What is involved in contracting to outsource call handling to another call center, in terms of effectiveness, expertise, technology, and human resources — and what advanced call processing techniques are available? We discuss call center professional services such as call pattern analysis and call volume analysis, among others. We talk about “what it takes” for an assisting call center to provide warm site services, including full ACD functionality and desktop and server technology. We discuss how much time to allot to testing. Chapter 21, the last in Part 3, is about acquisition of the necessary technology and managing a safe return of the call volume and updated data to your home call center.

© 1998 by CRC Press LLC

15 Call Volume Management Issues

C

all volume management (CVM) is a comprehensive call pattern analysis that results in a customized solution to the emergency communication needs of a call center. It is more difficult to develop a CVM plan than a “cookie-cutter” standardized plan, because of its comprehensive nature. However, this type of planning is more versatile and extensive than ordinary planning techniques. Exhibit 15-1 looks at the more commonly used methods of call center recovery. With CVM you will use a blend of these methods to create a plan that works for your call center specifically.

EXTERNAL CVM SOLUTIONS In Exhibit 15-1 and described in more detail below, we have given the types of CVM that are available. You should now look at the original call type chart, Exhibit 6-1; the area below the double lines are areas that could not be handled readily by an interim call handling (ICH) or overflow facility. However, an overflow facility could handle them more easily than any other alternative. You should now start assigning levels of difficulty to your call types, if you have not done so already. (Your result would be a chart of call types that assigns difficulty factors to them, as shown in Exhibit 15-2.) Once you have looked at the difficulty factors of your call types, then you can look at the call volume for each type of call. The two issues that we will consider here are the call volume and the difficulty factors. There is no one solution that will fit any “standard” call center. You should grade the call volume from 1 to 10 and the difficulty from 1 to 14. Then you can start charting which of the calls can be handled by an ICH facility, minimum call overflow facility, an IVR system, or by distributed architecture. This needs to be a management decision. Your call volume will help you to determine how many work stations you will need to have at each of these facilities, including your hot site facility. Remember that you might not necessarily to have the same number of work stations (for your own CSRs) at the outsourcer hot site as you do at your original call center. If your company has contracted with an ICH facility to handle the simple calls, then let them. The idea here is to handle the calls that you must handle. You and your recovery team should contract out as much call volume to other companies as possible until your personnel are ready to return to the original call center. This should be done as long as it is cost-effective. If it is going to be to expensive to do this, then have all your calls go to the hot site facility (and your own CSRs) as soon as possible. Despite the foregoing, we recommend that only your most difficult calls be handled at the hot site facility, unless the outsource call handling were prohibitively expensive, that is. This is because your company personnel should be employed to start getting your cold site or alternate site facility up and operational, in case your original call center could not be restored.

© 1998 by CRC Press LLC

© 1998 by CRC Press LLC

EXHIBIT 15-1. TYPES OF CALL CENTER RECOVERY Types of Call Center Recovery

Cold Site

Hot Site

Interim Call Handling

Real-Time Minimum Call Overflow

Description of service

• Site with T1 lines • No equipment setup

• Site with T1 lines • Equipment setup with computer equipment and basic phone systems • Also includes mobile sites

• A call center and/or a virtual call center setup • Scripting in place, waiting to receive calls

• RTMO continuing overflow system that facilitates the availability of trained agents off site

Recovery time

6 hours to 2 days

1 to 6 hours

30 to 45 minutes

15 to 30 minutes

Disadvantages

• • • • •

• No CTI • Manual systems • Must relocate staff

• Untrained agents on customer calls • Heavy reliance on scripting • Only able to deal with basic call types

• Requires that a small number of calls be handled by an external agency (expert calls)

Advantages

• Use own staff

• Use own staff • Equipment already setup

• Reasonably short RTO • Excellent for large volume of simple calls

• Uses best route according to RTO • Very short RTO • Variety of options available • Agents trained on customer system • Easier to implement

Emergency communications for staff (high-volume inbound-outbound IVR)

• No

• No

• Yes

• Yes

Call center planning assistance

• No

• Yes

• Yes

• Yes

Crisis management

• No

• Yes

• Yes

• Yes

Full call center testing

• No

• No

• Yes

• Yes

No CTI Manual systems Must relocate staff Hard to test Very long recovery time

Note: “Yes” indicates that we expect the companies who wish to perform these services to have the capability.

EXHIBIT 15-2. CALL TYPE CHART WITH DIFFICULTY FACTORS Duration (min.) Peak No. of Calls

Type of Call

Difficulty Factors

Order taking Reservations Answering sales calls for manufacturing Answering sales calls for insurance or professional services Answering sales calls for retail Product location inquiries Channeling customer complaints to others for resolution Solving customers’ problems Telemarketing outbound Answering inquires for general information Answering inquires about product service Answering inquires about product use Advising senior management on customer issues Defining customer service Defining customer policy/strategy Public relations Crisis management Market research Quality management Billing Collections Repair/warranty service Outage reporting Contest lines (advertising) Difficulty Levels: (L) low; (M) medium; (H) high. Difficulty Factors: 1. 2. 3. 4. 5. 6. 7.

© 1998 by CRC Press LLC

No complexity Services knowledge Features/benefits knowledge Product/services changes Knowledge base Product knowledge Experienced personnel will give seamless service to customers

8. Technology to support 9. Tracking and delivery of information for fulfillment 10. Required understanding and close working relationships with management and company 11. Effective problem solving skills 12. Sales skills/closing techniques 13. IVR or RAN message 14. Other

In addition, it could be very expensive to transport employees to your hot site location, and house and feed them there. If your company were trying to recover from a wide-area disaster, your hot site outsourcer must probably have been located distant from your original location to have remained operational. Also, if a wide-area disaster had occurred, then most of your employees would probably want to remain with their families rather than go on the road with the company.

Real-Time Overflow Real-time overflow is a method by which a call-handling facility, normally an outsourcing overflow center, handles a number of your difficult calls on a regular basis. Taking your overflow call volume regularly would enable the overflow facility personnel to become acquainted with your procedures. This would allow them to become part of your regular callhandling solution. As a result, their CSRs would blend in with yours, and seem invisible to your customers. See the next chapter for more details.

Interim Call Handling ICH is a system that is set up such that the outsourcer would take a large number of relatively simple calls and process them. An ICH facility would handle call volume such as a sales call where the CSR could read from a script that was at the ICH facility, take the information, and input that information into a common database. Generically, an ICH facility would resemble an overflow facility (or something like IBM Call Protect) that would process calls while your call center was out of commission. Note that these facilities could handle a large volume of calls of simple types. See the next chapter for more details. Emergency Overflow for Peak Hours, Days, and Seasons. One of the values of the

ICH and the minimum call overflow combination is for call handling during emergency overflow. Too much call volume (such as sales business) could be as big a disaster as not enough business. It could be very difficult to recover from having too many calls. If your company used assisting call centers to handle overflow, that could provide continuity to your call center that would not otherwise be there. With the real-time overflow facility taking expert calls and the ICH facility taking simple calls (as routed by the toll-free number the customer called in to) the caller would not know that anything out of the ordinary had happened. If your company does not use an overflow company at this point, we recommend that you get one.

Hot Site We have discussed hot sites elsewhere in this book, so we will just cover the major point to consider when selecting a hot site. First, most hot sites do not have CTI capability. When your company selects the hot site, ensure that your call center functions could operate as separate computer and telephone functions. This might mean that your call center applications would take a technological step backward. Your CSRs would need training to deal with this. Most of the of companies listed in Chapter 11 provide hot site services. Second, the subscription fees, activation fees, and duration of stay vary from hot site to hot site. These would be three factors to examine when selecting the hot site. Last, look at the location of the hot site and consider how long it would take your personnel to reach there, in the worst traffic situation imaginable. Even in a metropolitan area, CSRs are not enthusiastic about commuting a long distance every day to the alternate location.

© 1998 by CRC Press LLC

Company Cold Site A cold site is a location that is available for your company to move into temporarily. If it has telephone lines already there, it is often referred to as a warm site. Often a cold site does not have electrical power or telephone lines connected to it — depending on whose definition you use. Cold Site Drop-Ship. Cold sites and warm sites do not have equipment sitting there ready to use, with the possible exception of a PBX or an ACD unit. For that reason, in a very short period of time you would have to get equipment to the site and install it. Many vendors have drop ship or quick ship programs. These programs enable customers who have purchased equipment for their main call center to get urgent shipment of equipment to an alternate site within 24 hours. This option should be preplanned and is best arranged in a service-level agreement with your vendors. Ramped Up. If the site was partially ready for use, it would be a warm site. Whether the facility is to be “cold” or “warm” depends on a decision made by senior management. A warm site would cost more to maintain, but could be put into use far more quickly than a cold site. As seen in Chapter 13, after a major disaster the financial and personnel burden of having to ramp up a cold site could be substantial. Schedule of Inspections. One of the problems associated with cold sites is the tendency of companies to forget about inspecting them to ensure that the facility is ready to be used. If you planned to use a cold site, your company would have to commit to a schedule of inspections and to enforcing a policy making members of the recovery team accountable to do those inspections. The team members would have to check that the site was not leaking water, which could destroy the electrical equipment, as noted in previous chapters. Also, other basic conditions would need checking — such as ensuring that the roof remained on, the windows were unbroken, and the furniture had not been not stolen from the building, so that the facility would be ready for use in case of a disaster to the primary site. Periodic Testing of Telephony. If your company had decided to use a warm site, and had telephone lines and electrical power in place, then the recovery team should do testing to make sure that these were in good working order. This should be scheduled for regular attention. The testing could be done remotely by having call center CSRs phone the site and see what response they get. Alternatively, company telephony personnel or Telco representatives could run diagnostics on the equipment at the site. Outside Local Area. Normally, a cold site should be outside your immediate area. If a wide-area disaster or wide-area infrastructure failure occurred, having a distant site to go to would be important. The alternate site is not the quickest way of setting up alternate call handling. As noted above, the cold or warm site should be regarded as a later part of the call center recovery plan. The cold/warm site would be ramped up after call volume was switched to the ICH and real-time overflow facilities, and concurrently with use of an outsourcer hot site facility. Such a site would not be used for a minor disaster. The cold or warm site (depending on its degree of readiness) should be viewed as a place to go only in the event of the worst type of disaster. The condition for use would be that your primary facility had suffered a major disaster of long duration, such that there would not be a primary call center to go back to.

© 1998 by CRC Press LLC

Keep Equipment and Software Current. The equipment and software that is stored at the alternate site should be kept current with that at the primary call center. Having an ACD on site could be a complication, because it too should match your current equipment. Companies sometimes fall into the trap of upgrading the ACD at their primary call center, while the ACD at the cold site just sits there and receives few or no upgrades. If your recovery team had to use the ACD at the warm site, there might be all kinds of problems if it had not been upgraded along with the ACD at the main call center. Plan to Stay ahead of Growth. Out-of-date software could also cause problems. What if your company had upgraded the software at the primary call center but did not at the warm site? If a disaster struck, the company might have a serious problem with personnel not remembering how to use the old software. Just consider this example: if you are a Windows NT user, try going back and using the Windows 3.1 operating system. It is not that easy. Even worse would be going from Windows to an old computer running MS-DOS. When the IS department upgrades the primary site hardware and software, your CFO (as part of the continuity plan) must allocate additional funds to provide the upgrades to the alternate site as well. One useful thing about either a cold or a warm site is that it would have the potential to be ramped up into another call center for the company in a reasonably short time. This would allow the company to provide for growth by incorporating the alternate site into its plans for expansion.

Other Solutions Mobile Workshop. A number of companies provide mobile workshops or backup facilities for call centers and could travel to your site. NCR and IBM are two of them. The mobile work site solution is a call center on wheels that could come and hook up to your telephone lines and provide your CSRs with alternate work stations. The workshop could usually arrive within 24 hours. This solution is good for the “smoking hole” scenario, when your call center would be unable to function because of a site disaster. For many reasons, the mobile workshop is a complicated solution. Your company would have to find the space to operate this facility near your call center and its cabling. Although these facilities require very little room, emergency personnel might cordon off the open space available, such as a parking lot. It might be difficult to place the facility there. If your company is located downtown, it could be even harder to get the space for this facility. Another problem could come when attempting to hook the mobile workshop up to the demarcation line of your inbound cables. The hookup might take longer than expected. This might be to some degree depend on what kind of disaster had taken place, and what the weather was like. For instance, if an explosion had destroyed the call center, the cables might be damaged. If space for parking could be arranged and connecting the cables could be done rapidly, your company might find the mobile workshop to be a good alternative to a cold site. By providing a temporary facility for the company CSRs to work in, using the mobile workshop could give your company time to arrange an alternate facility or repair the primary one.

High-Volume Call Answer Another option for immediately answering your calls even during a disaster would be a system called high-volume call answer. Basically this is a sophisticated IVR unit (or units in tandem) set up either at another call center facility or inside a Telco network, as, for example, inside the

© 1998 by CRC Press LLC

AT&T network. Your company could also set up HVCA IVR unit(s) inside its own multiplefacility call center network. Assisting Call Center. Twenty First Century Communications, Inc., pioneered the idea behind this system. (We will say more about them later, in Part 6.) The idea is that your calls could be processed completely by IVR in a reasonably high volume. Twenty First Century uses this technique to deal with calls to electrical utilities when they must deal with power outages. When the customer calls up a company that is experiencing an outage, the Twenty First Century HVCA system would take the call. This would not be because the power utility would expect to have suffered damage of any kind to its communications equipment. Rather, it would be because the Twenty First Century HVCA system could receive calls from the two thousand or more customers who might be calling simultaneously. For a utility having three million or more subscribers, facing that kind of call volume could be very difficult. The first thing the customer would hear would be a message, saying, “Thank you for calling ACME HYDRO. If this is an emergency, hang up and dial 911 immediately. To report an outage press 1.” The caller would then be instructed further by IVR prompts. The customer would be prompted to enter the residential or commercial account number via the telephone numerical keypad. The system would then verify the customer’s address and access a situation report about the customer’s local area. The IVR unit would then inform the customer verbally with an estimate of the time that the customer’s area would likely have power restored. High-volume call answer is an excellent method of dealing with a massive number of calls. There are many applications where this would be appropriate. For example, an insurance company responding to a wide-area disaster could receive its customers’ initial claim information, even if it were, itself, recovering from the same disaster. Planning and Preparation. To use this option, your company would need to set up an outsourcer contract with the computer/telephony vendor and set up prerouted switching with Telco. The HVCA IVR would have to be programmed with scripts and prompts fitting the application of your company. Please note that the HVCA system would not be completely automated. The IVR equipment could answer thousands of standard calls. However, emergency calls would require live agents to troubleshoot situations for which the IVR could not be programmed.

Direct CSR Link For extremely critical applications, such as real-time contact for stockbrokers, we have pioneered an idea called “direct CSR link” in which a CSR becomes the eyes, ears, and hands of the client. This would be a person who needed to have information entered into a database immediately. By that recovery method, the client could use a cellular or satellite link to the direct CSR link. While the client was traveling to a hot site facility, the direct CSR link could keep the client up-to-date with fast-changing information, and in contact with the client’s customers. In the case of stockbrokerages, this would require the licensing of the CSR as a broker. However, the licensing issues would be reasonably inexpensive in comparison with the costs of having a brokerage go down with a communications outage. Such costs could exceed $6 million per hour. If the brokerage should go down on a day such as that when the BreX scandal broke, then the losses could be $18 to $20 million in that hour. If the brokerage happened

© 1998 by CRC Press LLC

to suffer a power outage and lose trading capability, the company could be ruined and shareholders, management and the company might be held liable. The direct CSR link is high on the scale of difficulty, but it would be the next best thing to the expert company representative performing the required actions at first hand.

Preestablished Procedures for Disaster Call Routing Telco Assistance to Set Up. One of the essential parts of CVM is a preestablished procedure for emergency call routing. The Telco can set this up for you in advance. For instance, AT&T has six or more preprogrammed routes that your company could use in some of its recovery options. In other plans, your company could have a computer on the call center manager’s desk that would perform the rerouting function. However, in the event of a disaster this should not be the only place to reroute calls — see below! Call Center Switching. Once upon a time, there was a company whose management thought that it could reroute its lines just fine. But unfortunately the only phone that it had set up to do the rerouting from was a POTS (plain old telephone set) in the middle of its call center. The idea was that this one dedicated line could reroute all the calls. This was to keep people external to the agency from rerouting their lines on them. No one even knew anymore why the system had been configured that way. In a meeting, a consultant told company representatives that they had a serious problem with their rerouting arrangement. The problem was that if the call center was burning down, someone was going to have stay with the POTS to reroute the lines. The CFO looked at everyone in the room and said that would be the job of the call center manager. The call center manager looked shocked. The consultant then asked if the call center manager would get extra pay for staying in a burning building during rerouting. Code Word to Enact Switching. You have to take two things into account. The first is your requirement to provide security for your lines. You do not want just anyone to be able to switch your call routing. The last thing you need is a disgruntled employee rerouting your lines to the local pizza parlor. The second thing you need to do is to balance that security with the ability to reroute your lines from a facility other than your call center. (In some circumstances, this switching might have to be done at high speed.) One of the best ways to arrange this would be to have the receiving facility set up so that with a command word you could have your calls rerouted. This would allow your company (together with Telco and the assisting call center) to provide controlled distribution of the password to a small number of authorized personnel.

Real-Time Backup for Mission-Critical Calls Defining “Mission Critical.” A mission-critical call is a call that would be difficult to hand over to an outside agency. The value of the call type would be such that all such calls had to be dealt with in a short period of time. The other calls to the call center facility would be valuable also, but they could be handed over to an assisting call center without the company suffering a loss of income. Mission-critical calls might also be regulated by legislation. Relevant concerns might be licensing, confidentiality, or similar issues. Data Shadowing. How could an assisting call center handle data without compromising company licenses or confidentiality agreements? Probably this would be best handled by

© 1998 by CRC Press LLC

database sampling. By using this method, the call-handling facility would only handle the database for individual transactions so that there would be a record of those transactions. A secured company such as IBM or ComDisco would handle the entire database under contract. The secure database maintenance company would set up arrangements for the recovery facility to sample only the parts of the database information that it might need to complete any one transaction. Another solution would be simply to input (restore) the new data into the database at the end of the day. The recovery facility would never access the confidential account information. To do these tasks, the assisting call center would need trained and skilled CSRs who had basic background knowledge of your product or service. The CSR level of training would need to be above average. The assisting call center personnel would need to be skilled in handling both inbound and outbound calls. These CSRs would need the ability to use popup screens and scripts on an immediate basis. If outsource CSRs had access to your company database, it would be highly important to make sure that the CSRs were bonded. Nondisclosure agreements would also have to be put in place to keep your data safe. How It Is Done. Generally this kind of information would be distributed on a daily basis. Normally it would be best to have a nighttime dump of information from your company data center to storage media. This would enable your company to retain the information at the data center (internally) rather than keeping it at the assisting call center site. This would reduce the risk of a breach in confidentiality. Some other company facilities could function as alternate backup storage locations. The shipping department, for example, would often be very busy during the Christmas rush. In contrast, for the rest of the year it would have largely idle facilities with good security safeguards that could be utilized by your company for secure alternate storage of backups. In the event that the assisting call center had to use the confidential database, the last backup could be transported or transmitted there. The vendor service-level agreement for data shadowing would specify that only authorized personnel have access to the information. Your continuity plan should also include provision of secure off-site storage of duplicate backup media. This would be important in case of a major or wide-area disaster. We recommend that at the off-site storage facility (even if distributed by electronic vaulting), the backup data should be retained offline to prevent access or mischief by hackers or other vandals.

INTERNAL CVM SOLUTIONS No individual method described below should be used alone. The solutions discussed should be used in combination. Topics include redundant sites, the other company call center(s), a virtual call center, and reciprocal agreements.

Redundant Site Other Call Center(s) within Company. Recovery from one fully operational call center to another fully operational call center was covered previously in Chapter 3. “Illusions of Recoverability.” This is the most common disaster recovery plan for call centers, but it generally does not work. Normally a company would not have a fully operational call center unless it was being used to near its capacity. Therefore, there would be little excess capacity to handle a surge in call volume.

© 1998 by CRC Press LLC

Excess Capacity. It is possible that there might be a small amount of excess capacity. If that were the case, the other company call center might be able to handle a small part of the call volume of the crippled call center. So there is some value that could be realized by adding this option into the call center disaster recovery plan. Take the basic disaster recovery plan for handling a brief outage. The second call center can take a certain amount of call volume depending on what the acceptable call blocking is for the call center. This might be the short-term solution for the company. The second facility would take as many calls as it could, and the planner would then accept that the rest of the calls might be blocked. It is important to calculate what the excess capacity would be in the event of a disaster. The IS and telephony personnel on the team would have to put safeguards in place to make sure that the additional call volume does not cause a failure of the second call center. Capacity Redistribution. If the plan employed the second company call center in association with an ICH or real-time minimum overflow (RTMO) facility, the plan would be more flexible than if the second company facility were used alone. The primary call center would reroute calls so that the external facility (ICH or RTMO) would take the simpler calls and the second company call center would handle the expert calls. In some cases, companies have dedicated one call center to one call type and a second call center to another call type. Unless there is a very strong reason for this, we recommend that the call types be split up equally among your company call centers. This is so that if there were a disaster, the personnel at the functioning call center(s) would have experience in dealing with all of the call types. If personnel at both (or all) of your company call centers were experienced with all the call types, the call centers could back each other up in case of disruption. This is a more expensive way of running your call centers; however, it is generally less expensive than losing all calls of one call type without having an alternative way of handling them. The ICH and the RTMO facility could work with your call centers to help with call volume distribution, and to stay ahead of growth. Reciprocal Backups of Data. One of the advantages of using more than one call center is is reciprocal backups of data. That means that the facilities would have redundant server backup in each location. The cost of a server is generally not much in comparison with the time saved in backing up files and in restoring lost data in a disaster. This would also give the ICH facility and RTMO facilities the capability to sample data without having to hold it within their own system. Expert Calls (HQ as Call Center). When you and your planning team did (or, if reading through this book first, will do) the company wide analysis of critical functions, the survey should have shown what calls could be handled by other areas of your company. Many of the people working at the company head office (HQ) have gone through the call center or are familiar with the products and services that are sold or provided by the call center. This is not a recommended method of call center recovery. It can be done, however, if you have a number of people at the HQ who are skilled at call handling and your HQ is in another location. Then, in an emergency, the company could employ the HQ as an emergency call center. One of the options would be to set up the HQ as a hot site facility. There exist portable desks that could be stored away, as well as additional computer systems that could be stored in excess space, so that on a very quick basis a homemade hot site could be set up and ready to go. It would probably be more expensive to do the initial setup of such a company hot site than it would be to contract with a hot site outsourcer. However, the accounting department could do a close analysis to see if this method would be cost-effective.

© 1998 by CRC Press LLC

Distributed Architecture (Virtual Call Center) The virtual call center is “distributed” in the sense that the CSRs are not all in one place; rather, the “call center” is a creature of cyberspace and is located all over the local area, where personnel are in their homes answering the company calls. This would be a solution for the “smoking hole” type of disaster, in which, when employees arrive at work, there is nothing left of the company building but a smoking hole. In such a situation (a local fire, or any situation that required an evacuation of your building), the call center would be effectively gone, but the employees’ homes would not be affected. Provision for Coordination by Assisting Call Center. This method could also work

as a temporary solution while the rest of the company personnel were traveling to an alternate site. The main problem with this method is the requirement for an external commandand-control facility to route the calls. Distributed architecture would work well when combined with an RTMO or ICH facility to accept the calls and coordinate routing of calls to CSR whose lines became free. Calls could come into the ICH or RTMO facility. There, a CSR, or the IVR system, would determine what level of handling that particular call would require. If the call were relatively easy, it would be handled at the ICH facility. If the call was of medium or moderate difficulty, then it should go to the overflow facility. If the call were very complex, then it could be handled by your CSRs at their homes. In this manner nearly all of your calls could be captured without tying up other resources. Telco Prearranged Routing. Because of the vast variety of switching methods and switching plans available from the different telecommunications providers, it is beyond the scope of this book to give you the best ways to do your switching. The prime goal would be to make sure that once your incident manager had decided on a method for handling the calls, the calls would indeed be handled. However, we will address some problems that could be a concern in this area. Switching of Telco lines. Your recovery team must work with your Telco to arrange emergency switching in advance. Attempting to do switching on the fly just does not work. Most telecommunications companies offer preprogrammed routing on demand. One thing that we have found is that transferring calls across telecommunications systems does not work well. If your company were planning to outsource to an ICH or RTMO facility that was using MCI lines, then your company also should be using MCI lines for its call center. Likewise, if your company were on Sprint or AT&T lines, we would recommend that you find outsourcing facilities on the same system. There have been toll-free (800 network) infrastructure failures on a number of occasions that have affected some of the large telecommunications carriers. For this reason, we recommend that your company and its RTMO facility both have alternative toll-free lines, with different routing methods to their central office (CO). In other words, if you are going to have multiple T1 lines, then why not have some of those come in on one provider and some on another provider. This could be difficult if you were doing a single 1-800 application, but if you were using multiple 1-800 numbers, then it would be a good idea to split up some of the numbers so that you would have traffic on different networks. For example, if you had the toll-free numbers of 1-888-555-1111 through 1-888-555-1118, you might be able to have half of these numbers sent through one Telco, and half through another. This would allow (in the event of a network failure) customers to call on an alternative number. Many companies do publish multiple toll-free numbers in their advertising. It might get a little confusing for the customer, but it would give the caller more than one option, in case one of the numbers becomes inoperable.

© 1998 by CRC Press LLC

A problem that arises is that some of the networks use the same infrastructure locally. So it might do you no good at all to have the infrastructure set up with two different telecommunications providers if they were using the same local CO to route the calls. If the CO went down, then so would the 1-800 providers for your business. Make sure you check existing local routing before you make any of these decisions. While on the topic of COs, check with your telecommunications carrier and ensure that you have route diversity to multiple COs. Most urban areas have access to two COs, sometimes three. Also, check the status of the survivable fiber protection rings. You would want to have at least two routes going to your main CO. If your Telco does not provide this level of service, you might want to check with its competitors. Check your existing longdistance carrier to see if there is access to multiple COs for your service through that carrier. If there were a fire, the local CO could be a single point of failure. If there were a cut in the line between you and the CO, you would want to make sure that the calls can still be routed through another CO or routed by another route to the same CO (survivable protection ring). Questions to ask: • Does the local routing go through the same CO? • Is there a single point of failure in the fiber ring between your call center and their CO? • Are there two COs that connect to the network through different routes? • Is there route diversity between the two sets of routes?

Another area of concern might be the local fiber loop that services your facility directly. Are there any single points of failure on your local fiber loop? This includes the question of where the fiber comes into your building. If it is be cut, could there be more than one access into your building? Or would you have a disaster if your only access in-line were cut? It would not be that difficult to set up another fiber route on the other side of the building or at least 500 feet away that could meet with another fiber ring. What you are trying to do is set up multiple fiber protection rings, so that at each step a call takes from the caller to your location there would be route diversity. When there is a network of COs, you would want to make sure that even if your company lost some of its phone calls, the company would not lose its business altogether. When the recovery team sets up call routing, first you must determine the destination that you want your calls routed to in an emergency. You would arrange the service-level agreements in the areas that you wanted to route to, and then you would set up the methods of routing according to the scale of disasters in your risk assessment matrix (see Exhibit 7-2). The idea would be to arrange a mechanism by which your incident manager or call center manager could telephone Telco and enact alternate routing by plan number. This would also be very handy information for your ICH or your RTMO facility to have. Suppose that it had a duplicate of your checklist and you informed the facilty that you must evacuate the building immediately. Then it could begin to implement the disaster recovery plan with the proper protocols, and it could call your Telco and implement the routing change as part of your disaster recovery plan. You need to look at what could happen to that routing plan with regard to disaster. What could interrupt the routing? Your call flow handling should include as many redundancies as are necessary to make your plan robust and flexible. Once you had the routing plan in place, it would be important to test the effectiveness of the plan to see if it would work. In the testing portion of this book (Part 4), you will see a section on simulated testing. You would want to run simulated traffic loads and see if the routing plan allowed you to do the switching that you had planned for.

© 1998 by CRC Press LLC

The Telco representatives could deal with routing plans. However, you, the planner, must ensure that the Telco representative knows what you have in mind. You would need enough information to make an informed decision about different alternatives. Sometimes the telecommunications carriers are reluctant to share information on routing diagrams and similar items. You might have to be insistent. What if the Telco representative were not cooperative in helping you write your preprogrammed routing plan? In such a case, your company should consider switching to another Telco carrier that would provide this essential information. Another point about telecommunications carriers is that most of them have emergency implementation procedures in case of a wide-area disaster. Amazingly enough, it seems that businesses are not high on most Telco priority lists. In fact, pay phones would probably be the first telephones to have service restored. As part of your agreement with your Telco, and depending on your company recovery time objectives, you should make specific agreements with the Telco (or at least find out from them) about the priority of your company for restoration of service. That would give you some idea in a wide-area disaster how much time it might take for your primary facility to get fully operational. If your company were low on the Telco restoration priority list, you might find that it would be days or even weeks before you could get telephone service restored. In such a case, your contracts with the ICH and the RTMO facilities would mean survival for your company. All the routing protocols (plan numbers and passwords) should be in the call center “to go” case, so that the incident manager could do the routing from by cellular phone from outside the building if necessary. (This information should also be duplicated in secure off-site storage.) Your company might want to consider maintaining more than one “to go” case in different offices, in case of a fire that might cut off the call center manager or incident manager from the route to the case. The recovery team might even plan to keep one at another location. However, your company would have to ensure that all cases were kept secure, because they would hold information vital to the company. Other questions to consider concerning the distributed architecture solution: • Cabling and equipment at homes of CSRs • Provision for IVR support during high call volume • Provision of current data back ups • Security of the home • Emergency generators • Cellular phones • What if a CSR proves unfit? • What about a wide-area disaster? • Management follow-up to ensure emergency needs met

Reciprocal Agreement We have talked with a number of people in the call center industry who have tried reciprocal agreements. It would be remiss of us not to mention this as a method of call center recovery, although we do not recommend the method. Few of these agreements seem to work, although in theory they would be a fantastic idea. Noncompeting Company. To arrange a reciprocal agreement, you would need to find a noncompeting company that had similar call types and volumes to yours. Theoretically, the companies would be best matched if they were of similar size and had similar revenues.

© 1998 by CRC Press LLC

“Next Door” Neighbor. The next door neighbor form of reciprocal agreement would not be a good idea. Most of the time any disaster affected you, it would affect the neighbor, too, because of the “ripple” effect. This ripple effect is the reason most reciprocal agreements do not work. For example, if there were a local utility outage, the company next door would probably be suffering the same outage, likewise, in case of severe weather, blockage of the street due to construction, crowd problems because of a labor dispute down the block, and other local problems. Confidentiality of Database. Some of the things that you would need to concentrate on if you decided to set up a reciprocal agreement would be confidentiality concerns. The two companies would have to reach an agreement concerning their databases. Basically, the two companies would have to provide backup storage on their servers for each other’s data. The reciprocal agreement would have to provide, first, for the other company not to access or use the database information and, second, for each company to provide access to the backup data in case the company that owned the data suffered a disaster. Considering that the database information would be required so that the company could ensure continuity of its business, there could be some problems with the arrangement. What if one company worked only on the day shift, but the other ran 24 hours a day? How would the information be retrieved during the night? How would the companies provide for security of the data, and for release only to authorized personnel? What about scheduling and delivering backups? Exchange Training of CSRs. Companies entering into a reciprocal agreement would have to provide exchange training for CSRs, so that the CSRs could do work at each other’s facilities. This would be to ensure that in the event of disaster at one company, the other company could make office and personnel resources available at the site of the second company. An interesting idea would be to use the personnel of the other company as part-time employees during busy times. This could have some advantages. However, if a disaster occurred at one company while the other was having its peak season, the burden could be too much, and personnel might burn out. Both Companies Must Ensure Excess Capacity. Generally reciprocal agreements would work best if the two companies had peak seasons at different times of the day or year. If they had different peak seasons, then the CSRs at both locations could deal with call volumes by sharing the load. However, if the peak periods were not compatible, how could the problem be addressed? Another problem might occur if one company were expanding, but the other company were static or in decline. In case of disaster, how could the company in decline cope with the larger call volume of the prospering company? How could the expanding company cope with the additional volume of the calls of the other company on top of its own heavy workload? Contracts and Safeguards. Companies entering into a reciprocal agreement would require detailed contracts and safeguards. The lawyers of both companies would no doubt be overjoyed to work on such a contract. This fact would be neutral as far as disaster recovery was concerned, but would make some small impact on the bottom line. Joint Recovery Teams. Joint recovery teams would be a must. The hot site recovery teams could be separate. These teams would have to get together frequently for consultation and joint training. As complex as planning for the continuity of one company would be, the

© 1998 by CRC Press LLC

planning project for two companies would be even more so. About twice as complex, one could say. Things to think about in case the other party had the disaster: • Sharing call center capacity • Accommodation and meals for visitors • Periodic testing (See Part 4)

Here is something to consider which, if your company did make a reciprocal agreement, might keep planners of both companies awake at night: how would the companies recover from a wide-area disaster if both companies were crippled? In such a case, neither company could help the other, yet theoretically they would be bound by contract to try. Spin-Off Benefits of Reciprocity. One of the reasons for planners dreaming of reciprocal agreements could be that people from the company marketing department keep suggesting it. Marketing people like to believe that there would be spin-off benefits from reciprocity. There would be fresh views available for problem solving, they say. They could do exciting cooperative advertising campaigns. Marketing personnel can often be very persuasive. Unfortunately, we have not found any success stories about reciprocal disaster recovery agreements, only stories of failure. The theory says that the companies would enjoy synergy, see co-op advertising cost savings, and have the ability to share capacity. They could use each other’s facilities for periodic testing of their disaster recovery plans without paying outsourcer fees to do it. But the tests are where the reciprocal agreements begin to fall apart. It turns out that during testing (let alone, actual disaster) each company tends to look after its own interests first. The interests of the other company very quickly fall by the wayside. We have not seen success stories about reciprocal agreements. This ends the overview of call volume management techniques and issues.

© 1998 by CRC Press LLC

16 ICH and RTMO INTERIM CALL HANDLING What Is Interim Call Handling? Interim call handling (ICH) is the system that uses a large number of CSRs and, depending on the configuration, distributed architecture to take a large number of calls. The CSRs work from a script to respond to calls.

How Does It Work? 1. When a disaster commences, the ICH facility is notified about a disaster and it activates the response plan according its contract with the client call center. 2. The ICH facility either recalls its on-call people or reassigns other CSRs who might be doing outbound or some other deferrable type of call handling, to deal with the large volume of calls. 3. The calls are redirected to the operators who are trained to handle the incoming calls with a script. 4. Because ICH has mass call-handling capability, it is excellent for taking a large volume of reasonably simple calls.

Scripting To obtain ICH service, your company would have to a script for each call type that must be handled. The calls that will be handled be an ICH service must be of types that can be scripted. The script that the facility would use would be processed with the systems and software available at the ICH facility. Its computers and software might not be the same as your system, and for this reason the scripts should be complete and easy to use. The scripts must be able to convey all the information that is required to complete each type of call.

Call Completion Once the call is completed, the CSR has an exit screen used to enter information about what happened with that call. The server gathers this information for the customer, and the information is stored in a standard type database.

What Should You Look for in an Interim Call-Handling Facility? You and your recovery team should look for a call center having a great deal of capacity (headroom). This is most often found in a company that does a lot of outbound calling. The reason is simple: most outbound campaigns could be suspended for a few hours or a day without causing any problems. The company you select must be reasonably large and have a lot of capability. It would also have to provide a quick ramp-up environment. This would be an absolute necessity,

© 1998 by CRC Press LLC

since the company you contract as an ICH facility would not receive any calls from your call center unless you had declared a disaster. That company would have to operate in a quick ramp-up environment, or else its service would not work very well for it or for you. Your toll-free telephone numbers would have to be redirected to the ICH facility. The call volume to be handled would have to be calculated and agreed upon. A common database would have to be set up to store the information (such as a telemarketing script and customer sales information) at the ICH facility. We recommend that, after the prerouted switching is set up, the ICH facility should do at least two tests per year to ensure that that company could meet or exceed all of your call-handling needs. The two methods of call handling (real-time overflow and ICH) tend to work well together to make a flexible plan. Suppose that you had contracted with an overflow facility that could not take all of your call volume. You could send a percentage of reasonably simple calls to the ICH facility and the more complex calls to the real-time overflow facility, and thus have a large portion of your calls answered.

IBM Call Protect The IBM Call Protect Program employs a large number of call centers with ICH seats (as many as seventeen call centers), and it also uses distributed architecture with Teloquent. This gives it the capability of handling a large number of calls. Although IBM does have several facilities dedicated to ICH, its success lies in the fact that a number of its regular call centers have a half-dozen seats set aside for ICH activities, so it spreads a client’s large volume of calls across a number of different call centers. This means that it does not have to bring in a large volume of people at any given center. It can take two or three people from what they are working on and bring them across the call center floor and put them to work handling calls for your facility. IBM uses Teloquent to create a distributed architecture environment for its ICH. This is so that it can spread the calls even further and have people working out of their homes ready for disasters. It uses distributed architecture much the same way as your company might use distributed architecture in the event of an emergency. (Some of the companies doing distributed architecture are IBM, under its Call Protect Program, Southwestern Bell, and Pacific Bell. We saw a demonstration of the program at a disaster recovery show in Atlanta. However, at the time we wrote this book, we had been unable to gather any more information.)

Familiarizing External CSRs with Your Products, Services, and Customers In the ICH case this is done entirely by scripting. The agents never get to see anything on your products or services other than the scripts that pop up in front of them during times of disaster. This means that your scripting has to be first-rate. You and your planning team must make sure that you cover all the details that the customers could reasonably expect to hear. If there are details that the ICH facility CSRs are not familiar with, you should consider providing a means for the customers to get the information themselves. This could include a reference to a Web site. Most of the time the ICH facility has a pop-up screen that is associated with your company, so design your scripts with that in mind.

Aid in Answering Your Calls during a Disaster The ICH facility is excellent for taking your calls in a disaster. In the call volume management chapter (Chapter 15), we looked at the calls and the difficulty factors. Anything that had a difficulty factor greater than 4 or 5 would probably not be able to be handled by an

© 1998 by CRC Press LLC

ICH facility. You should look at real time minimum call overflow (RTMO) for that, which follows below.

REAL-TIME MINIMUM OVERFLOW Definition RTMO as a method came about because of the problem of inexperienced CSRs finding some calls too complicated to handle. If a CSR had not dealt with your calls before, then it would be very difficult for that CSR to answer those calls without some experience. The idea is to give CSRs experience with your company calls by handing over a small percentage of calls to the overflow facility all of the time. This accomplishes two important goals of your disaster plan. The first is that it proves that the calls can be received and answered properly by the RTMO facility. The second goal attained is that of proving that the RTMO facility can access your database properly. In effect, you are testing your RTMO facility with every call it receives. The overflow facility personnel gain skill in handling your company calls by regular exposure to them. The idea is to send a predetermined number of calls to the overflow center on a scheduled basis. We recommend that 3 to 5% of your average call volume be sent there. You should arrange a service-level agreement requiring that in the event of a disaster the overflow facility would handle a specified percentage of your call volume. The word minimum in the name real-time minimum overflow denotes the minimum overflow volume of calls that the outsourcer would contract to handle on a regular basis, for training. Basically, since the RTMO facility would be contracted to handle a scheduled volume of calls, it would be as if you have the outsourcer on retainer as a satellite facility of your company.

Contract with Assisting Call Center Your service-level agreements with the RTMO facility would ensure that it handled your calls in the same fashion as you do. In addition, it would guarantee that it would be able to handle a preset number or percentage of your call volume in the event of an emergency.

Prearranged Routing of Calls What do you look for in a facility? When you make up a table like the call center tables in Exhibits 16-1 through 16-3 to send out to various RTMO call centers you need to do several things: 1. You need to ensure that it has enough capacity to handle the call volumes that you would be sending to it in the event of a disaster. Look for a company that does outbound telemarketing, because in the event of an emergency that company could take the outbound campaigns that it is running and shift them over to take your inbound calls. 2. You should look for a call center that works with equipment as similar as possible to that used in your company call center. This would make the switching of calls from your call center to its easier, although it would not be essential. 3. Ensure that your Telco is the same. As stated previously, it is difficult to switch from one carrier to another during an emergency. On an ongoing basis (during normal operations) it would be possible yet still difficult. The RTMO facility that you choose should use the same Telco as the one your company uses for 1-800 service.

© 1998 by CRC Press LLC

EXHIBIT 16-1. PLANNING YOUR CALL CENTER BUSINESS CONTINUITY SERVICE Call Center Information Company Information: Company Name: ____________________________________________ Address: ___________________________________________________ Address: ___________________________________________________ City: ___________________________________

Province/State: _________________________________

Country: _______________________________

Postal/Zip: _____________________________________

Contact Name: __________________________

Title: __________________________________________

Telephone: _____________________________

Fax: ___________________________________________

Toll Free: _______________________________

E-Mail: ________________________________________

Description of Services the Call Center Can Offer Full call center recovery billed at a ______ rate (service level 80/20) Overflow support billed at a per minute rate service level as available) Call center business impact analysis assistance Call center disaster recovery planning consultation Crisis management assistance Crisis communication (a predictive dialer and IVR capable of contacting or receiving calls from all employees) Basic inbound call handling — see definition of primary call types Single ACD queue associated with a single inbound telephone number with a queue size of 5 Two or more delay announcements associated with wait times in queue Basic data collection based on script developed jointly by ICH and customer Customer call backs Single form database with information gathering, i.e., name, address, etc.

© 1998 by CRC Press LLC

We Are Currently Performing This Service

We Have the Capability to Do This Service

We Cannot Provide This Service

EXHIBIT 16-1. PLANNING YOUR CALL CENTER BUSINESS CONTINUITY SERVICE

Description of Services the Call Center Can Offer Scheduled delivery of data to customer within predefined time frames in industry standard electronic format Voice-mail/automated attendant Custom delay announcements Enhanced/custom/detailed reporting Skills-based routing Queue size larger than 5 Multilingual capabilities Multiple inbound lines Specialized/custom training packages Multiform or relational database Credit card authorizations (please state which ones) Fulfillment (please state location of warehouse) Real-time interaction with existing customer information systems Addition of routing technology, dedicated circuits (frame relay, point to point, etc.) Originating call center may elect to locate its own dedicated equipment at support call center Dedicated equipment at support call center (own by the support call center) to support specific requirements Shared specialty equipment at support call center to support multiple customers Interconnectivity into existing customer network infrastructure, security issues and advanced network management issues Distributed architecture — facilities in selected agents’ homes with incoming calls prescreened and routed to best agent Direct CSR link — allows for specialized recovery, even from pay or cellular phones Secondary office as call center (do you have backup for your call center?)

© 1998 by CRC Press LLC

We Are Currently Performing This Service

We Have the Capability to Do This Service

We Cannot Provide This Service

EXHIBIT 16-2. CALL CENTER GENERAL INFORMATION Industry Sectors (Please indicate all sectors in which your company provides service.) Credit and Collections

Technical Response

Financial Services

Telemarketing

Government

Telecommunications

Health Care

Transportation

Hospitality

Product Support/Service

Insurance

Utilities

Manufacturing

Wholesale

Retail

Other

Services

Other

Number of sites: ____________

Hours of operation: ____________________

Time zones: ________________

Geographic area covered: _______________

Call Center Profile: Calls per year: ___________

% Inbound: ___________ % Outbound: ___________ % IVR:

Calls per day: ___________

% Inbound: ___________ % Outbound: ___________ % IVR:

Peaks:

Seasons: ______________________________________________________ Day of week: __________________________________________________ Time of day: __________________________________________________

Average length of call: ______________ mm:ss

Average handle time: ______________ mm:ss

Objectives: Telephone service factor: _______________________________________ Average speed of answer: _______________________________________ Abandon rate: _________________________________________________ Transfer rate: _________________________________________________ What languages do you offer support for?

______ English ______ French ______ Spanish ______ German

Others ________________________________________________________________________________________

© 1998 by CRC Press LLC

EXHIBIT 16-2. CALL CENTER GENERAL INFORMATION (continued) Recovery Time Capability Recovery Time Capability

Recovery Time Capability — Time of Day

15 minutes

8:00 A.M. to 8:00 P.M.

30 minutes

8:00 P.M. to 8:00 A.M.

60 minutes 2 hours 5 hours

4. As far as your long-distance service goes, you would need to have an arrangement with the RTMO facility that when your call center had achieved an overflow condition, you would send overflow calls to them. It would be best, however, to have an arrangement with your Telco. Once you had reached a certain number of calls being handled concurrently at your call center, then any more calls above the agreed-upon threshold would be rerouted to the RTMO facility. Having the Telco do this would have the advantage that your company would only pay the normal long-distance fee. The call would be rerouted in the network, so that your call center would not be accepting a call and then sending it out again to the RTMO. Also, the Telco would in this arrangement be configured so that, if your call center were not answering its calls, the calls would be rerouted to the RTMO automatically. If your call center personnel had evacuated, there might still be a number of calls coming into your call center and not being answered. If your personnel were unable to activate one of the preprogrammed call-routing plans, you would still have calls routing to the RTMO facility after a fashion. AT&T refers to this kind of routing as “Next Available Agent.”

Benefit to Your Company An ongoing benefit to having an RTMO facility working with your call center to handle your calls is that your company would be free to take on new contracts and expand your call center later. Using the RTMO facility would allow you to answer calls on a new contract almost immediately. Yet you would still have time to expand your company facilities to be able to handle the new contract. This type of freedom would give your sales team extra room to sell your services. They would no longer be constrained by the number of calls that your call center could handle directly. Your company would have the expansion capability provided by the RTMO facility. Normally the cost to do this (have calls taken by the RTMO) is only slightly higher than if your company did it with its own resources.

Familiarizing External CSRs with Your Products, Services, and Customers In order for this to work, you would need to familiarize the external CSRs with your company products and services. This would include offering training courses to the CSRs at the RTMO facility. It would be a good idea to swap CSRs from time to time to allow the RTMO agents to see exactly how your call center handles calls, to improve the system and ensure that the same quality of service was given by both sets of CSRs. The RTMO solution is difficult to set up at first, because the personnel that would be handling the calls would require training in how to deal with your calls. They might require

© 1998 by CRC Press LLC

EXHIBIT 16-3. LAN NETWORK WORKSHEET Call Center Systems Do you currently offer: ❑ Skills-based routing? ❑ Zero outs to live agents from IVR?

❑ CTI? ❑ Access to live agents in call center via Internet? ❑ Access to call center via Internet e-mail? ❑ Electronic call recording?

Dynamic?

❑ Multiple queue login? ❑ Interactive voice response applications?

Server Platform

Agent Desktop

❑ Windows NT

PCs or NCs with ❑ Windows 3x

❑ AS400

❑ Windows 95 ❑ Windows NT 4.0 WS

❑ RISC

❑ Emulation Software ❑ UNIX

❑ UNIX

❑ Other

❑ Custom Programs Terminals with ❑ Emulation ❑ Custom Applications

Please include the following if applicable: # Users on network: __________ Applications:

_______________________________________________

Router platform:

_______________________________________________

Network is:

______ LAN

______ WAN

Please attach network diagram if available. Telephony Long Distance Networks AT&T ______ 800 numbers

MCI ______

Sprint ______

Bell ______

Other ______

___________ ___________ ___________

# Local trunks

___________ # ACD queues required __________________________

# DIDs

___________ Custom delay announcements (RAN) _____________ Detailed custom reporting ________________________

© 1998 by CRC Press LLC

Continuous?

extensive training to acquire product knowledge. The outsourcer CSRs need more than just a script to quote. They need to be able to put on a convincing act as your employees. That means that your company must provide pop-up screens that include detailed information. Subjects that the overflow facility CSRs would need information on would include company policies (such as the policy on discounts or replacement for faulty products), customer service and help desk information, and sales or marketing information. Even general knowledge such as where a customer traveling on vacation could drop into see the product line in person should be included in that extensive preparation.

Quality of Service There must be a system in place enabling your supervisors to monitor the calls that were handled by the CSRs at the RTMO facility. The system should allow monitoring of the RTMO the same way as your supervisors would monitor the calls at your own facility. You should be able to record random calls and listen to them later, or call in and listen “live.” We recommend a combination of both these quality assurance techniques.

“Can-Do” Attitude Needs Support One of the hardest aspects of employing RTMO is that you need to ensure that the same work atmosphere and attitude prevails at both facilities. If your facility has a can-do attitude, then the RTMO facility needs to have the same attitude. The customer should always be given the best possible support. You need to make sure that all of the resources that are available to your CSRs are also available to the CSRs at the RTMO facility. Your call center manager might have to promote a program to encourage the use of these resources by both sets of CSRs.

Initial Training on Dummy Databases Training on dummy databases is a good idea. Have your CSRs call the RTMO facility to train its CSRs on company products. This would give the RTMO CSRs a chance to learn about your products, and at the same time your CSRs could be asking them commonly asked questions and the occasional difficult question.

Pop-Up Screens and Scripts The RTMO facility would require identical pop-up screens and scripts to those used at your call center. It would be highly important to ensure that all upgrades in software and changes to the scripts were passed on to the RTMO as the changes were implemented at your call center. Security is crucial. You must make sure that the RTMO facility takes the same or better care than your call center personnel would with your data. Confidentiality of all information must be upheld. You would also need to have strict noncompetition and confidentiality agreements in place. The noncompetition agreement should run no shorter than two years in duration, to protect your company. As well, all employees at the RTMO facility must be bonded so that if one of their employees were to run off with some of your data, then you would have some financial recourse.

Building Skill An added benefit of using the RTMO facility is that the calls come in on a regular basis. This allows its CSRs to become very familiar with your products and or services. At an ICH facility,

© 1998 by CRC Press LLC

the CSRs would handle your calls for the first time during a disaster or during a time when call volumes are so large that you need some other call center to handle the calls quickly. The RTMO personnel would see your calls on a daily basis.

Assistance for Peak Time-of-Day Call Volumes One of the benefits of employing an RTMO facility is the assistance that it would give your call center manager during peak periods of call volume or during spikes in your call volumes. This allows you to set your schedules tighter, so that if you hit peaks that your call center cannot handle, then you could shift more calls over to the RTMO facility.

Benefit for Customer Using an RTMO facility to handle spikes in your call volumes or peak periods in call volumes would cost you a little bit more on a per call basis. However, your customers would not be on hold as long. Your customer service would therefore be enhanced by an RTMO facility. In this way, you would avoid having to post an IVR message such as, “We value you as a customer, please stay on the line and the next available call representative will be on the line to assist you shortly.” Let us face it, a valued customer should not be left on hold. All of the people who have worked on this book no longer use the services of companies that have left us on hold too long.

Benefit to Your Call Center Personnel Your call center personnel would have the benefit of less stress when they were dealing with calls at your peak period. There would be less of a rush to answer calls and get to the next customer quickly. This would also enable your call center to meet its call-handling goals with greater ease. If you were aiming for an 80/20 service level and you started slipping to a 75/25 level or worse, then you could have the RTMO facility assist your call center and bring that service level up to 85/15 or better.

Aid in Answering Calls during a Disaster The most important characteristic of the RTMO facility for your call center is how it would aid your call center during a disaster. You would have to test the RTMO CSRs for their readiness for unscheduled switching of calls. The best way to do that is to send them a number of dummy calls. These could be created by either your agents or be computer-generated calls sent by IQ Services or a similar company. You would need to prearrange a test of plan enactment, to make sure that it would work properly. This would ensure that the RTMO CSRs could carry on “business as usual” during your call center disaster. The RTMO facility would be familiar with your calls and business operation and would be working the same hours as your call center. Consequently, it could make an excellent crisis management company. This could be set up as a separate service. To set this up, you would need to supply the RTMO facility with duplicates of your disaster recovery checklists and call-handling and call-switching methods. Thus prepared, the RTMO facility could switch the calls for you if you had a disaster where your personnel had to evacuate immediately and or if it were difficult to maintain communication lines with the RTMO because of the disaster.

© 1998 by CRC Press LLC

17 Other Uses for Your CVM Back-Up

W

e have found that sometimes the CFO and other senior staff need to have a good business case to institute an ICH facility. What is normally in the back of their minds is the statement “It cannot happen here.” This could be the cause of their hesitation. If you are in this situation, then this chapter offers some reasons for proceeding. If you are not in that situation, some of these ideas could be worthwhile even so. These are not recommendations. Rather, they are suggestions that should get you thinking about other uses for your call volume management backup.

MANAGING CALL VOLUMES TO REDUCE RATES OF ABANDONED CALLS CVM techniques are not only useful to handle emergency overflow, high-volume call answer, or for disaster call routing, but you can also use CVM techniques for reducing rates of abandoned calls.

Call-Handling Goal Imagine that your call center should shoot for a call-handling goal of 90/10. That is, the call center aggregate goal should be to handle 90% of calls within 10 seconds. Is this goal attainable? With state-of-the-art call handling software, yes it is. Aiming for a high goal is a worthy enterprise. It tends to be more profitable. It gives you more satisfied customers, and it builds pride of achievement in your CSRs, tending to help you make a “happy call center.”

Caller Response to Being on Hold In the “old days” it was not uncommon for callers to wait on the line for 15 minutes or more. Then again, in the “old days” even a large company might have only a few incoming telephone lines. Callers would remain on the line patiently for long periods of time. If they hung up, and called back, they would probably not get through again easily. Nowadays, people have become accustomed to calling into call centers that have a multitude of incoming lines and numerous call-takers to answer them. If they have to wait more than 15 seconds, callers are likely to become impatient and abandon their calls. Considering the possible dollar value of such calls, if customers are calling to purchase goods or services, it pays to make callers as happy as possible. Have you ever heard people discussing the telephone service of another company? Suchand-so Company, they might say, has just terrible service. You have to wait “forever” before someone answers the telephone; then you sit on hold “forever.” Finally, you reach a live person, who speaks irritably or unsympathetically to the now impatient customer. The person telling the anecdote then says that they prefer to call a competing company where the caller can get through right away. And the people on the phone there are so nice, too! You, the planner, should try to ensure that your company is the one that seldom puts callers on hold. If your program report is showing that a certain percentage of callers are

© 1998 by CRC Press LLC

put on hold for 15 seconds or more, you should consider sending the overflow to your internal or virtual assisting call center. If the problem is more serious than can be handled by the present internal capacity, you should consider outsourcing the overflow. Business Case: Callers only have to call that other “nice” call center that does not put them on hold one or two times before it will become their call destination of choice. At that point, your company has lost them as customers, probably forever. How much is each customer worth to your company in revenues, per year, or per decade? Superior customer service is worth the money that your company spends to do it. The return on investment can be computed in market share gained or preserved.

Did Anyone Ask If Caller Had Time to Wait? Sometimes callers just want to make a quick call while their kettle comes to a boil. Or their barbecue is warming up and they use that two or three minutes to call your company. The last thing they want to hear is a “Hello, please hold! [Click].” Not everyone has a portable telephone. Even people holding a portable extension might want to use both hands for cooking or some other task. For a caller, an even worse situation would be that of calling in, being put on hold, and waiting for five minutes until someone might decide to talk with them. Such callers are likely to hang up, leaving an empty line, and giving the call center a rising call abandon rate. It is an easy thing to train your call center CSRs, or program your IVR (interactive voice response) unit, to ask callers if they want to hold before transferring their calls. Asking the caller for permission to hold or transfer the call seems like a basic courtesy. Still, sometimes “the basics” are forgotten. It is never a waste to remind all personnel of the value to the customer of their politeness and thoughtfulness.

Hold Time Music or Message (Why, How) Let us assume that your call center has a policy of always asking permission before putting callers on hold. What will keep callers waiting in queue from abandoning their calls after a minute or two of waiting? We think that you, the reader, would prefer that your company CSRs handled calls quickly enough that such a situation would not arise. However, if there are many calls in queue, callers are likely to be happier if they have something to listen to while they are waiting. It is neither too hard nor too expensive to provide a “hold time” soundtrack playing recorded music, a local radio station, advertising or public service message for your callers to hear. This simple service could greatly affect caller satisfaction with your company customer telephone interface. Such a simple thing shows the courtesy of at least trying to entertain or inform the caller.

Option to Receive Callback Even if the call-takers ask permission to put callers on hold, they will not always receive it. Some callers have pressing tasks on their minds, or are in a hurry to go away and do something else. If the caller declines to be put on hold, the CSRs should be instructed to ask if the caller would appreciate getting a callback at a later time. It is important that someone from your company should follow up and make the callback at the promised time. This means that the first CSR to speak with the caller would have to log the caller’s name and telephone number onto a pop-up screen with a timer function. Someone at a later time would have to answer the computer alarm and make the callback.

© 1998 by CRC Press LLC

Failure to follow through on the promise to call back could contribute to a loss or decline of the company’s reputation for providing a caring and professional service. Perhaps your call center already provides such a callback service, in which case you, the reader, will be thinking, “how obvious.” However, perhaps your company is among the many that could improve their interface with the public. In that case, we recommend this as a means by which your CSRs (customer service representatives) could make callers feel important and esteemed. It adds just another reason for callers to dial your company telephone number rather than that of your competitor. You, as planner, should ensure that callback routines get transferred to the assisting call center(s) in the event of call volume overflow or declaration of a disaster.

ADAPTING TO SEASONAL OR TIME-OF-DAY PEAKS Is there a pattern to call blocking or an overflow of calls? You, as the planner (and the call center manager), should examine your name of report, either as numerical tables or in graph form, to see when the peaks occur. Then you can either schedule for more CSRs to come on shift at peak call-handling times or else outsource the overflow to an assisting call center.

Pre-Christmas Rush (Season) Direct sales of gift items, tours and cruises, and various other goods and services increase dramatically before Christmas. Just as in the shopping mall retail sector, a large percentage of these sales all occur in the same few short months after Thanksgiving. Your call center needs to be prepared for the rush. Such preparations include a higher level of staffing during times of day when customers are more likely to call. All shifts, even at night, should be augmented to prevent buyers hearing a busy signal. Your assisting call center(s) should be prepared to take on any overflow, and provide the same level of service as your original call center would. Your call center also needs to be prepared for disruption or disaster during this important season. A significant part of the year’s revenue would be at risk if the original call center and the assisting call center were not ready to transfer calls.

Cold Season Holiday Rush (Season) Tour operators, cruise lines, hoteliers, and operators of spas and resorts need to ensure that their call center is able to handle increased call volume starting with the Christmas rush and running through to March or April. In any but an “El Niño” year, people from the cold and frozen north are likely to seek holidays in the sunny south as soon as the mercury plummets significantly. As with the Christmas gift rush, you (in the travel industry) should provide augmented staffing for your call center during this season. You should also be prepared to restore communications as quickly as possible in the event of disruption or disaster. This requires contracting with an outsourcer, or directing call volume to an internal call center with extra capacity, by prerouted switching of calls upon enactment of the continuity plan.

Student School Supply Rush (Season) Companies marketing books, stationery, and shoes, bags, or clothing for youth need to augment staffing and prepare to handle overflow during the late summer and early autumn.

© 1998 by CRC Press LLC

Students of both grade school and college do not all buy from the local retailer of such articles. Perhaps increasing catalog sales have opened an opportunity for your company that it enjoys at no other time of the year. As with the Christmas and travel rush seasons, your call center needs to augment its shifts to deal with increased call volume. Here also, the need to ensure that your assisting call center(s) (whether consisting of an internal virtual call center composed of telecommuters or one or more outsourcers) is prepared in case of disaster could be important to the bottom line.

Accountant’s Tax Rush (Season) If your call center services a large accounting firm, you are familiar with the “tax rush” that starts in December and runs through until April of each year. In this industry also, call center shift schedules need augmenting. Although the accountancy year does not end in April, a significant part of company revenue and reputation accrues during the rush of taxpayers to file their returns on time. Therefore, not only the call center, but also all departments of an accountancy firm are likely to carry increased workloads during this time of year. This points up the reason for doing an entire business impact analysis and continuity plan even though the “front line” of communication with the public is likely to be the call center. In the event of a disaster or significant disruption of business during the “rush” season, all critical functions of the company must continue. This includes preparation of tax returns and provision of accounting advice for the clients of the company. Therefore, you, as the planner, must ensure that the continuity plan not only provides for a handover of calls to the assisting call center(s), but also that critical business functions continue. Otherwise, your company might not survive to do business in the tax rush of the following year.

Other Rush Seasons — General Discussion Rush season commerce might take place by various kinds of communication with the customer. In some industries or regions, regular letter mail is still a factor, and employees in the mailroom will feel a heavier load than at other times of year. E-mail is becoming a factor in sales sparked by customers looking over catalogs or Web sites, and this puts increased emphasis upon keeping computer equipment operational. A situation similar to call center call blocking might be the result if your ISP (Internet service provider) cannot keep your “mailbox” from becoming full. Videoconferencing is not as likely (yet) to be used for caller purchases; however, that medium is discussed later. In the call center, rush season is likely to be a time of increased stress for both caller and call-taker. Some aspects of such stress are discussed below. Irritable Caller. The phrase “shop until you drop” need not only apply to crowded department stores. Especially as the deadline or end of season for a particular industry approaches, your CSRs are likely to be in communication with stressed-out callers. Dealing with cranky, unreasonable, and impatient callers can be hard on your company personnel. We recommend that before your “rush” begins, your company should schedule a refresher in “dealing with problem callers” for all departments. Yes, even your shipping department might receive calls from customers who erroneously indicated the departmental destination of their call. It is not unknown for callers to call the wrong department on purpose. Perhaps such callers think that if their call were transferred internally, between departments of your company, they

© 1998 by CRC Press LLC

could “jump the queue” or receive special treatment. In some instances, the more “inventive” callers might feel motivated to try such a stratagem — especially if your product or service is something callers might compete for, or if the company is running a contest. All your company personnel should treat calling customers (especially those switched to departments not expecting such calls) with politeness and respect. This includes your assisting call center(s) during overflow or a declared disaster. Stressed Call-Taker (Time for a Break). “Rush” season can take a toll upon those who deal with the public. For days and even weeks on end, your CSRs could be dealing with callers who are rushed, cranky, demanding, or asking for help in deciding between alternative products or services. As part of your call center quality assurance program, your call center supervisors should be watching for fatigue or irritability in the call-takers on shift. Even a short break could make the difference. If an employee starts to become flushed, perspires, or seems to be breathing rapidly, that person needs a break! Letting personnel off the phones for short periods for de-stressing washroom breaks or trips to a lounge vending machine for a candy bar or soft drink should not be viewed as a “waste” of paid employee time. Rather, it should be viewed as a kind of investment. A “happy call center” is a productive one. Furthermore, relaxed CSRs are less likely to respond to a problem caller with irritability. This will tend to promote higher customer satisfaction and higher customer service all around. Ergonomics Increase Efficiency. Footrests, telephonic headsets, and keyboard wrist support are not a “fad” in new office furniture. These items can improve CSR efficiency and diminish fatigue. As noted above, a call-taker who is relaxed is more likely to be good-tempered and helpful to even a problem caller.

Noon Hour (Time) The lunch “hour,” which runs from 11 A.M. until about 2 P.M. could be called a “mini-rush,” and it happens every day, even on weekends. This is the time when people who are calling during their workday have a moment to call from their desk or from a pay phone to your company. The “hour” is an indeterminate period of time with no clear boundaries because of flexible or staggered lunch periods at the companies where callers themselves work. Even on weekends (or midweek if callers are on “days off”) callers might decide to call before or after partaking of lunch at home or while out shopping on foot. It makes sense to be prepared to augment call center personnel for this interval. If other “rush” periods are going on as well, your assisting call center(s) should be prepared to take over the overflow in call volume.

Cellular Rush Hour Calls (Time) In a metropolitan area, the evening “rush hour” or “drive time” for commuters moving by car, city transit, or commuter train can start at 3 P.M. or earlier, and last until 6 P.M. or later. Perhaps your call center provides a product or service likely to appeal to commuters, or one that is advertised on billboards or on placards aboard city buses or subway cars. Maybe your company advertises by radio during that time period. Consider the poor commuter, immobilized during the commuting trip for an hour or more of unproductive time. If the person has nothing to look at but an appealing photograph and a toll-free telephone number, calling that number by cellular phone could at least have some entertainment value. Perhaps the person has listened to a radio commercial and must call now, while the number is fresh to mind.

© 1998 by CRC Press LLC

Your call center personnel need to be prepared for commuters calling in who might not have their credit card information at their fingertips. In fact, if the caller’s fingertips are controlling the steering wheel of an automobile, it could be a matter of road safety that this be so. Such a caller would probably also not have access to the written information that describes that person’s current service package with your company. Perhaps the caller wants to upgrade health insurance, cable television service, or a magazine subscription. If your call-takers are prepared to provide verbal product or account information and refresh the caller’s memory concerning any forgotten details, the “entertainment” caller might be induced to trade money for the time of your CSR. In that case it would be worth the augmentation of the call center shift at the commuter “rush” in money as well as reputation.

Utility Outages (Event) The use of electrical power has become essential to most households and businesses. Any power outage prevents the use of computers required to do important private or commercial tasks. Likewise, lack of electrical power can cause failure of refrigeration (and hence damage to household, restaurant, and retailer stocks of perishable foods), failure of traffic signal lights and thus a rise in accidents, and other results varying from irritating to life-threatening predicaments. In the old days it was not uncommon for thousands of utility subscribers to call the power utility repeatedly in the faint hope of getting some information about when their electrical service might be restored. Nowadays, assisting call centers such as Twenty First Century Communications, Inc., provide a much-needed service to energy companies. The HighVolume Call Answer (or HVCA) service allows a utility to answer every call in the event of a power outage. The PUCs (Power Utility Commissions) that use the Twenty First Century HVCA service know that the assisting call center(s) have the capacity to handle 4,000 or more calls simultaneously. When a tornado or ice storm hits, these companies are in communication with subscribers through Twenty First Century, and can inform customers of when their power will be restored. This substantially increases customer satisfaction. The HVCA system analyzes the volume of calls and produces a “footprint” report that helps determine the magnitude and location of the faults and damage that cause a power outage after severe weather or other event. This means that downtime is reduced because repairs can be completed quickly. True emergency calls, such as reports of downed wires or requests for help for persons on life-support equipment, can be redirected from the HVCA system to the live CSRs at the utility. The whole call center team at the utility need not work long hours of overtime in anticipation of severe weather or in its aftermath. The HVCA connects with the computer at the utility to produce outage reports identified by telephone number, address, and pole number. A second report segregates emergency reports so that they can receive priority attention. A third report lists the number of calls received each half hour, so that the peak and decline of mass calls by subscribers can be recorded — and the end of the high-volume call period can be identified so that calls can be handed back to the originating call center. Only a small “emergency line repair” department need be on duty to deal with trouble calls requiring immediate response. If your company uses the Twenty First Century HVCA service, we congratulate you. This is a service of proven effectiveness. (Please see Part 6 to learn about how Twenty First Century Communications helped to meet the challenge of Ice Storm ‘98 in Maine.) If your utility does not use HVCA (yet), please see Appendix B for information on how to contact Twenty First Century before your next outage.

© 1998 by CRC Press LLC

Storm Tow Truck Rush (Event) Tow truck operators know that certain events will trigger a rash of calls. Although sometimes there are multivehicle crashes on highways and expressways, the majority of distressed motorists will end up calling because of severe weather. Some such calls will be the result of radiator trouble during excessive heat. However, the majority of calls for a tow truck will come as a result of storms. Winter storms that make roads all but impassable due to snow or ice will leave cars stranded in ditches or even in their own driveways. Severe cold or sometimes wind will prevent ignition, causing the driver to call for a boost with jumper cables. If traffic is all but stopped by snow that reduces lanes available to handle high volumes of vehicle traffic, some drivers will call for an emergency supply of gasoline, having run out of fuel. Sometimes lightning or windstorms will leave cars buried underneath trees or torn-off roofs. Sometimes flooding will sweep cars off their wheels and turn them over, or flood their engines. In all of these instances, tow truck operators go to the rescue. If your call center services a tow truck operation (perhaps affiliated with the AAA or CAA) we recommend the following. Your call center shifts should be augmented during expected bad weather. Your continuity plan should provide for overflow call handling by interactive voice response (IVR or VRU) or an assisting call center. Your assisting call center(s) should be put on standby during bad weather, so that in case of a power outage, even throughout a wide area, callers can still log in for a tow, even if your own call center is put out of commission. Because such a “tow rush” would affect your whole company (like accountants in tax season), we recommend that in addition to augmenting your call center personnel, you also bring additional drivers and other employees on shift to put additional trucks into service during heavy weather. Your accounts receivable department probably should be augmented as well.

Monitoring Call Blocking of Faxes Facsimile messages are a form of communication that seems to get overlooked when company call volumes are computed. Usually, people do not use the fax machine for voice messages unless they are calling for a lunchtime or after-hours pizza, or for a taxi or tow truck. Nonetheless, messages sent or received by fax can be an important part of both customer and business communications. Perhaps you yourself have suffered frustration from trying to fax repeatedly to another company. That fax number is always busy. This is the result of call blocking due to inadequate provision of telephone or dedicated data lines to handle the volume of messages transmitted. Telco is just as able to report on call blocking on a fax or data line as on a telephone line reserved for regular use. All that is required is a request to Telco to add that line to its reports to the call center manager. If Telco should report that call blocking is taking place, you (as the planner) together with the call center manager (or wearing two “hats” at once if you fill both positions) should request senior management to approve the provision of additional lines. Fax lines should not be forgotten when contracting with an assisting call center for overflow or emergency call handling, or when setting up a virtual (or distributed) call center for use during purely local disruptions. Prerouted switching of the fax lines should be included in the continuity plan and all relevant checklists for transfer when a disaster is declared.

Dealing with Too Much E-Mail We hear from time to time about company personnel, particularly in management positions, who receive burdensome amounts of e-mail every business day. They dread going on

© 1998 by CRC Press LLC

vacation, because they know that hundreds of e-mail messages will await them upon their return. These people could productively employ a clerk (or a subroutine) to sift through the waiting messages to find the truly important ones. Alternatively, a mass print job would at least make these messages readable on the commuter train or over lunch. The personal or humorous ones could be enjoyed better if saved to a disk or forwarded to a personal e-mail address to be printed out and read on the deck or balcony at home. Or, perhaps personal messages could be reserved for perusal and reply from a person’s home office or den on Saturday mornings. If you, the reader, hear complaints (even embarrassed ones) about too much e-mail from company personnel in various departments, you should discuss the matter with the MIS. Too much e-mail to wade through could rob company personnel of massive amounts of time required for other valuable tasks. In addition, facing a burden of “useless” or “nuisance” messages could seriously affect employee morale. Someone in your company should find a way to sort out valuable messages from the “Spam” (junk e-mail) and time-wasters. (We proffer our apologies to the manufacturers of the original canned meat Spam product.) If the IS department were not able to provide a program able to sort out important messages and throw out the chaff, the chances are that they could find an expert who could. The ISP Web manager could probably provide the required software upon request. What is probably required is a “kill” file. Certain key words contained in any e-mail received would activate the program to erase an offending message so that company personnel need not waste any time upon it. Small amounts of programming expertise could make such a program able to send a polite reply to the originator of the message, or invite the writer to resend the message by letter mail. If people from other branches, divisions, or companies are complaining to company personnel that they cannot seem to get their e-mail through, the following possibilities might apply. Perhaps the ISP server mailbox for a given e-mail address does not have enough capacity. In that case, your MIS should request that the ISP upgrade the memory allocated to receive such messages. An alternative might be that the senders of messages that are not received are using an incorrect e-mail address. Even if the “name” portion of the address was correct, an incorrect suffix (such an erroneously used .net, .com or .org) could cause the server to deny all knowledge of the e-mail address. As with telephone calls, sometimes people need reminding to check their directories — or your company business cards — for the correct listing.

Scheduling Telephone Conferences and Videoconferences outside Peak Periods Company personnel are finding video or audio conferences increasingly useful for virtual “meetings.” Such arrangements save on travel and accommodation costs. The time and expense for equipment and telecommunications (even including satellite uplink) compare favorably with cross-continental or transoceanic airfare and hotel accommodation. Since virtual meetings are seldom short, it makes sense to schedule them outside peak usage periods. Your Telco representative could probably tell you what hours have the most favorable billing rate per call minute. Holding conferences outside the usual office hours also makes sense if one or more participant in the conference has offices in another time zone.

SUPPLEMENTING THE EXISTING CALL CENTER Sometimes your call center needs extra capacity. Perhaps some crisis is going on in your local area that is not exactly a disaster, but puts a burden upon your call center personnel

© 1998 by CRC Press LLC

or causes them considerable distraction. Or perhaps an epidemic or period of political unrest has just concluded and company personnel, although showing up at work, have not yet returned to peak effectiveness. Perhaps there is no crisis or burden. Instead, business is booming, revenues are climbing, and your company is experiencing unexpected growth. The capacity of lines, equipment, computers, and the personnel of the call center and other departments are no longer up to the workload that is linked to the new company prosperity. Whether the reason is good or bad, there are times when the company needs “a bit extra” in the way of equipment and personnel.

How to Know If Your CSRs Are Losing Ground Your CTI (computer telephony integration) software can provide reports that will show how long callers remain on hold, how many times they are transferred from one department to another, and can give you some idea of how effective your CSRs are compared with previous reporting periods. In addition, a testing company called IQ Services (full name in Part 4, Testing, and Appendix B) can help you determine what quality of service your CSRs are providing. This company can even report to you how effective your IVR unit is and tell you if call volume has outstripped the handling capacity of your computer(s). Recognizing that failure to handle call volume is in itself a kind of “disaster,” you, as the planner, in conjunction with the call center manager (or possibly you acting as both), should bring this situation to the attention of senior management. What should you be recommending to senior management at this point? One option would be to contract with one or more additional assisting call center(s) to handle the overflow. Another option would be to hire on more CSRs and provide them with computers and dedicated telephone lines at their homes. A third option could be the renovation and expansion of your existing call center, and the provision of more computer capacity and hiring of more call-takers to handle the call volume. The best solution would depend largely upon the cause (if you could discover it) for the reported failure of your call center personnel to deal with the call volume as well as in previous reporting periods. Could there be some kind of “slow motion” disaster threatening the company and perhaps the entire local region? (Slow-motion disaster was described in Chapter 12.) In that case, handing call overflow to an assisting call center outside the local region makes good sense. Perhaps some contaminant has entered the local drinking water supply. Maybe a local factory is experiencing undiscovered seepage of a chemical agent, which is sapping energy from the local population. Maybe the local power grid requires expansion or renovation, and frequent “brownouts” are causing a degradation of data processing — but nothing that a human could see in the ordinary course of events. On the other hand, perhaps your company is experiencing rapid growth. You might determine that the company needs an entire new call center down the road from the original. This would be a good thing, right? Growth is sometimes a tricky thing to handle. If it proceeds at an even pace, wonderful. But sometimes growth can be uneven, or mirror the booms and busts of the general business cycle. Many companies have expanded, by incurring debt, at the top of a boom cycle, and then collapsed in the ensuing recession. We would encourage you, in the case of perceived increase in call volume, to proceed cautiously. If planning to build an additional call center facility, we would suggest the following. First, while planning for a new facility, you the planner should expand the contracted service of the assisting call center(s). In the near term, this would help both to handle increased call volume and to confirm whether the rise in call volume might be temporary or long term

© 1998 by CRC Press LLC

in nature. You should examine whether or not the increase might have an impact on company revenues, too. What if it was solely a response to some news article and had no lasting result after a month or two? Second, if continued call volume and higher company revenues should warrant it, we would suggest building an additional facility outside of the local electrical and communications grid. This would have the effect of widening your company “footprint” and making it less vulnerable to wide-area disasters, such as weather, war, or earthquake. Third, we would recommend a scalable expansion. Although you would probably wish to provide for more growth yet, a plan providing for the growth needed this year would protect your company from the effect of unexpected recession. On the other hand, setting out upon a program of expansion planned in multiple phases would provide for the possibility of future growth if required, without saddling the company with unmanageable debt.

© 1998 by CRC Press LLC

18 Checklists

C

hecklists are one of the simplest yet most effective tools we can use.

1. Check lists are: • Simple to Make Up. They all follow the same basic format with just some of the procedures changing. Notice in the examples that follow they have some specific information pertaining to the incident, and then common sense procedures to follow. • A Necessity in an Emergency. Although they are commonsense items, most people forget what they supposed to do when faced with an unexpected crisis. • Proven Method to Ensure People Do Not Miss Critical Items in an Emergency. The military police and fire departments all use checklists from time to time in different emergencies. 2. Members of each unit or section should have their own set of checklists for their position, from CSRs to CEO. 3. The checklists must cover all incidents that are likely for your area. The ones that are listed below are not a comprehensive list. If you are in doubt as to the types of checklists you should have, look back at the risk analysis chart. 4. The fire department and police in most areas can help you with instructions for the life-preservation section of most checklists. That is the part that says when to get out and where to go. 5. Checklists must be as simple as possible; they must cover major points and be available to all employees. Remember: any position, any time, could detect a possible disaster, or receive a telephone call about a bomb threat, fire, or a number of other possible disasters.

COMPONENTS OF A CHECKLIST 1. Concise, but thorough. The checklist should be in bullet form and cover all of the points required for the position (see the phases of the crisis below). 2. Notes as reminders. Anything that the person has to remember should be inserted in the bullets above or at the bottom of the checklist as a note. For example, at the bottom of the fire checklist, it states, “Remember: in any fire situation, remain calm and do not panic.” 3. Notification procedures. In all checklists, there must be a clear statement of who should be notified and what they need to be notified about. 4. Actions specific to a situation. In each checklist, the actions that need to be taken have to be spelled out in detail. You will notice that the actions that should be taken in the fire checklist are similar to the tornado checklist. However, the area that the people report to is different. After all, you do not want your personnel reporting outside during a tornado or staying inside during a fire. 5. Evacuation procedures. The route for evacuation should be clearly stated in the checklist. In the examples given below, we have stated “to your designated area.” However, in a real checklist, if there is any opportunity for confusion, the route must be clearly explained. 6. Pertinent information logged for postincident report. In the call center manager’s checklist, you will notice an incident report that should be filled out immediately following the incident. However, it is the responsibility of all personnel to provide sufficient details to the call center manager and the recovery team to fill out the report accurately.

© 1998 by CRC Press LLC

7. Remarks for any additional data. Any information that could help in creating better checklists or that could help prevent the incident from recurring should be logged for review by the recovery team.

The checklist should address the following phases of the crisis according to the position and responsibilities for which the checklist is written. You will notice that the supervisor has more responsibilities than the CSR, and so on.

PHASES OF A CRISIS The basic stages of a disaster are as follows (described in more detail in Chapter 23): 1. Event 2. Alarm 3. Assessment 4. Notification 5. Declaration 6. Evacuation 7. Escalation of the plan including backup activation 8. Recovery of calls 9. Restoration of the systems 10. Restoration of the calls 11. Return to normal operations 12. Plan evaluation and update

TYPES OF CHECKLISTS There are two types of check lists: 1. Position checklists. This is a book of checklists that pertain to a particular position. In the following example, we have the first position stated as “CSR.” 2. Incident checklists. These checklists make up the book. In the following example, we have the incidents starting with “Fire Checklist.”

You will notice that the number of checklists grows with the higher rank of the position. The lists in Exhibits 18-1 through 18-4 are not comprehensive; the checklists should encompass all threats that are high to medium risks.

EXAMPLE CHECKLISTS The four checklists in Exhibits 18-5 throught 18-8 are what you might see in a set of checklists for a CSR. The remainder of the checklists (Exhibits 18-9 through 18-13) define areas covered by other participants.

© 1998 by CRC Press LLC

EXHIBIT 18-1. CSR CHECKLISTS • Fire • Tornado • Bomb Threat • Wide-Area Evacuation, Contact Information — i.e., 1-800-Contact • Recall Procedures • Power Outage Procedures

EXHIBIT 18-2. SUPERVISOR CHECKLISTS • Fire • Tornado • Bomb Threat • Wide-Area Evacuation, Contact Information — i.e., 1-800-Contact • Recall Procedures • Power Outage Procedures • Hot Site Relocation Procedures • Cold Site Relocation Procedures

EXHIBIT 18-3. CALL CENTER MANAGER CHECKLISTS • Fire • Tornado • Bomb Threat • Wide-Area Evacuation, Contact Information — i.e., 1-800-Contact • Recall Procedures • Power Outage Procedures • Hot Site Relocation Procedures • Cold Site Relocation Procedures • 800 Rerouting Procedures • ICH Facility Procedures • RTMO Facility Procedures • HVCA Facility Procedures • Distributed Architecture Activation

© 1998 by CRC Press LLC

EXHIBIT 18-4. IS TECHNICIAN CHECKLISTS • Fire • Tornado • Bomb Threat • Wide-Area Evacuation, Contact Information — i.e., 1-800-Contact • Recall Procedures • Power Outage Procedures • Hot Site Relocation Procedures • Cold Site Relocation Procedures • 800 Rerouting Procedures • IVR Shutdown Procedures • Computer System Shutdown Procedures • All other equipment shutdown procedures

© 1998 by CRC Press LLC

EXHIBIT 18-5. CSR BOMB THREAT CHECKLIST 1. Note: • Keep your cool • Keep caller talking • Listen carefully • Log all information 2. Ask: • Bomb location? • When will it explode? • What does it look like? • What is the explosive? • Why was it placed? • Who are you? • Where are you? • Request more information to save lives. 3. Note characteristics of caller: • Sex • Estimated age • Accent • Voice (loud, soft, pitch) • Speech (fast, slow) • Diction (good, nasal, lisp, etc.) • Manner (calm, excited, vulgar) • Background noise • Does the caller seem familiar with the area? 4. Immediately after caller hangs up, notify: • Immediate supervisor • Site security • Co-workers • Police and other units 5. Note: Supervisor may have personnel conduct search in their immediate area • Sound alarm • Vacate building • If you must evacuate: a. Exit the call using emergency scripting and exit screen b. Report to your predesignated meeting point c. Report to your immediate supervisor that you are present d. Your assembly spot is the coffee shop across the street unless otherwise directed

© 1998 by CRC Press LLC

EXHIBIT 18-6. CSR TORNADO WARNING CHECKLIST 1. Note: • Keep cool • Get location • Listen carefully • Log all information 2. If telephoned in, ask: • Present location • Direction of travel • Speed of travel • Wind speed • Associated weather hazards • Additional information 3. Notify: • Immediate supervisor • Site security • Alert shelter staff (if applicable) • Other sections (computer, operators, etc.) 4. If seen, notify: • Authorities (911) • Immediate supervisor • Shelter staff • Other personnel 5. Seek Cover! Your shelter location in the red door in the first level of the basement If you must evacuate: • Exit the call using emergency scripting and exit screen • Report to your predesignated meeting point • Report to your immediate supervisor that you are present

© 1998 by CRC Press LLC

EXHIBIT 18-7. CSR COMMUNICATIONS OUTAGE CHECKLIST 1. If by telephone, ask: • Problem • Equipment effected • Estimated RTO (if available) • Effect on operations (minor or major) • Log all information, note time received 2. Notify: • Immediate supervisor • Maintenance section • Log information 3. If you detect a problem: • Write down all problems • Note equipment affected • Notify immediate supervisor • Check for any safety concerns • Log all information, including names of personnel notified Remember: Safety first!

EXHIBIT 18-8. CSR FIRE CHECKLIST 1. If you detect the fire: a. sound the alarm by i.

Shouting “Fire, Fire, Fire”

ii. Pulling nearest fire alarm b. Ensure immediate supervisor is aware of the situation c. Report on all details of the alarm to your supervisor 2. Upon hearing a fire alarm: a. Determine if it is an evacuation alarm (in some buildings two short rings indicate not to evacuate but to stand by for further instructions) 3. If you must evacuate: a. Exit the call using emergency scripting and exit screen b. Exit building using nearest stairwell (do not use elevators) c. Report to your predesignated meeting point d. Report to your immediate supervisor that you are present NOTE: Remember: In any fire situation, remain calm and do not panic. 4. Upon returning to your station (only when directed) a. Determine if your station has been damaged b. If no damage is apparent use startup procedure #3 when directed 5. If sent to a hot site, secure a hot site CSR checklist and follow the instructions 6. If sent home, report to phone number 1-888-555-help and await a return call

© 1998 by CRC Press LLC

EXHIBIT 18-9. SUPERVISOR FIRE CHECKLIST 1. If you detect the fire: a. Sound the alarm by i.

Shouting “Fire, Fire, Fire”

ii. Pulling nearest fire alarm a. Ensure immediate supervisor is aware of the situation b. Report on all details of the alarm to your supervisor 2. Upon hearing a Fire alarm: a. Determine if it is an evacuation alarm (in some buildings two short rings indicate not to evacuate but to stand by for further instructions) 3. Upon receiving a fire alarm or report: a. Confirm from call center manager (if unavailable, then make decision) if evacuation is required, if so: i.

Ensure all calls are exited

ii. All your personnel have left the area iii. That the doors are not locked 4. If you must evacuate: a. If you are on a call, exit the call using emergency scripting and exit screen b. Exit building using nearest stairwell (do not use elevators.) c. Report to your predesignated meeting point d. Upon arriving at the predesignated meeting point i.

Ensure all your personnel are accounted for

ii. Dispatch someone to direct the emergency crews to the fire area iii. Report to your call center manager that you are present 5. Ensure all pertinent information has been logged. 6. Report to the Call Center Manager.

© 1998 by CRC Press LLC

EXHIBIT 18-10. CALL CENTER MANAGER FIRE CHECKLIST 1. If you detect the fire: a. Sound the alarm by i.

Shouting “Fire, Fire, Fire”

ii. Pulling nearest fire alarm b. Ensure crisis manager is aware of the situation 2. Upon hearing a fire alarm: a. Determine if it is an evacuation alarm (in some buildings two short rings indicate not to evacuate but to stand by for further instructions) 3. Upon receiving a fire report: a. Determine if evacuation is required b. Initiate interim call handling c. Advise recovery facility to initiate fire checklist d. Take “to go” case e. As you exit, ensure: i.

Doors are not locked

ii. All personnel have left the building f.

If you must evacuate: i.

Exit building using nearest stairwell (do not use elevators)

ii. Report to your predesignated meeting point iii. Report to crisis manager that you are present iv. Ensure all supervisors are present v. Take report from supervisors to ensure everyone is accounted for vii. Ensure all required steps have been taken g. Collect data and logs from supervisors for postincident briefing; complete incident report h. Contact interim call handling company to ensure its checklist is complete Note: Ensure some form of communication is available to talk to the interim call handling facility.

© 1998 by CRC Press LLC

EXHIBIT 18-11. INCIDENT REPORT Date:

Time:

Live:

Questions

Test Name:

Answers

Action

What was the incident? (disaster type) What position were you working at during the incident? How much warning did you have? How much damage was there? Which systems were affected? Did the checklist accurately provide all the information needed to deal with the incident? If no, which areas were missed? Were the affected areas recovered within the recovery time objective? Did you have to evacuate? If so, did the checklist accurately guide you to your shelter or designated area? List the recovery options used: Did you have any switching problems? Did you have any database recovery issues? Did you have any interim call handling problems? What control measures could be used in the future to ensure this does not happen again? Comments:

© 1998 by CRC Press LLC

RTMO:

ICH:

Dist. Arch:

Hot Site:

Cold Site:

Other:

EXHIBIT 18-12. IS TECHNICIAN FIRE CHECKLIST 1. If you detect the fire: a. Alert personnel by i.

Shouting “Fire, Fire, Fire”

ii. Pulling the nearest fire alarm 2. Upon hearing a fire alarm: a. determine from call center manager or supervisor if evacuation is required b. If so, before leaving ensure: i.

Haylon dump is turned to “manual on”

ii. Start emergency data backup to internal system iii. Set doors to unlock c. Exit building by nearest stairwell (Do not use elevators) d. Report to the predesignated meeting point e. Log all pertinent information f.

Report to immediate supervisor or to the call center manager.

EXHIBIT 18-13. INTERIM CALL-HANDLING COMPANY FIRE CHECKLIST 1. Upon receiv-ing a fire call from the call center manager: a. Begin line transfer procedure b. Call AT&T at 1-800-xxx-xxxx c. Identify uses code word xxxx d. Request preprogrammed route # 1 2. Activate call stations 3. Confirm actions to the call center manager at 1-905-xxx-xxxx that transfer has taken place 4. Ensure all pertinent information has been logged for use at the postincident debriefing

© 1998 by CRC Press LLC

Part 4

Testing

T

his section starts with “Six Questions to Ask When Designing a Test.” A well-made test is the key to determining whether or not your continuity plan has the “right stuff” to save your company when the chips are down. We tell you about various types of tests, including verbal simulations, test evacuations, and computer simulations. Chapter 21 concludes with a discussion of that exciting event, the full-function tactical evaluation. This section also discusses test objectives and describes the multitude of items that tests measure. Not only can you find out if your continuity plan “works” as a whole, but also you can check out continuity of your critical business functions. A good test will reveal whether your logistical plans “hold water,” and test your restoration procedures, simulating the return from “disaster” to normal operations. You can learn whether your vendors of equipment and essential services would be able to help you in an emergency or whether your supplier relationships need some fine-tuning. Can you order a hundred tents, thirty portable toilets, first aid kits, drinking water, sleeping bags, and food for employees and dependents, and portable electrical power? What about a working call center or replacement computer components? Are credit arrangements in place? How do you contact the supplier in the middle of the night? If there is a glitch in your plan, it is better to find out during a test, and not while you are talking on your cellular phone in the parking lot during an ongoing emergency. Tests have another very important function. They help familiarize all involved personnel (including suppliers) with the company emergency procedures. Everyone from CEO to CSR would benefit by knowing about emergency checklists, how to reach the incident manager or third-party crisis manager, and where to gather after evacuating the building. Whether the test is a verbal walk-through or a full-function tactical evaluation (complete with transfer of calls to an assisting call center), every person who takes part becomes less likely to panic and more confident of “what to do next.” Just in case.

© 1998 by CRC Press LLC

19 Six Questions to Ask When Designing a Test

T

esting is the heart of knowing whether your call center business continuity plan works or needs major retuning. In conferences that we have attended, most of the companies that say they have a plan have not tested it. Why go to all the work of creating a plan and then not test it? If you have not tested your plan then you do not have one. The purpose of these next chapters is to take the fear out of testing. Following the basic format of Where, What, When, How, Why, Who, and then observing the Result is the simplest and most effective path to take. You should understand that no matter how well you write your plan, something is almost certain to go wrong when you test. Be happy: this is supposed to happen. Better to find out that the plan has a weakness in testing, than to discover a fault during a real disaster, when it matters. Testing could be compared with prospecting for plan faults. Although it is like panning for gold in reverse, the faults are what you are testing to find. If you find such a plan weakness, such as a missing step in a checklist, you should not be discouraged. Once you discover the weakness, you can correct the plan and related checklists, and your company will be stronger for it.

WHERE ARE YOU TESTING? Live Call Center Nothing strikes fear in a call center manager’s heart quite like someone telling him or her that someone is going to do a test on the live call center. When it becomes time for you and your planning team to begin testing the plan, we suggest that you broach the subject by asking the call center manager to help plan the testing. This will reassure that worthy individual that you and your team would not be likely to just shut down equipment, leaving the call volume nowhere to go. Of course, if the call center manager was already part of the planning team, that person would already know what safeguards were planned to prevent damage to the company interests. The best way to start this testing is with the “Methods of Contact” list below. Then you should start work on the different departments, finally working up to the call center as a whole. This process might take a year or more, and will allow everybody to become familiar with the testing process. Whole Call Center. This is normally associated with the type of test some people call the full-body workout. The more official name is the full-function tactical evaluation, described later in this chapter. This type of test is best done in the off-peak hours with simulated call flow (see the section on IQ Services). As discussed above, this is one of the last tests that you and your team should schedule.

© 1998 by CRC Press LLC

Departments within Call Center. This is probably the easiest kind of test to set up. If you arrange the call center into teams you can, from time to time, have one team test another team. The tests can run the full range of test types (see the next chapter) and can actually be fun. Methods of Contact. The following methods of contact are tested in the same fashion that the IS technicians test the database backup: 1. IVR 2. E-Mail 3. Fax 4. Telephone 5. Links to Fulfillment

You start in the off-peak hours and move everything to the backup system. You then confirm that it works properly, and move it back again. That sounds simple, right? However, it rarely goes that smoothly. When you do this test, ensure that you only move part of the system, if possible. Follow your checklist systematically. If there are any steps missing or something does not work at the other end, rework all the steps until the process works smoothly. Do not forget that when you upgrade a piece of equipment, some or all of the checklist steps may change.

Backup Call Center Internal. The other call center(s) of your company should be tested as a backup to find out its/their capabilities and how much extra capacity is available. How many calls can the other call center(s) receive from the primary site before that starts creating problems? This can either be done with live calls or, as we recommend, with simulated calls. It would also be possible to take the live call center environment and add simulated calls until you start to see call blocking. Note that if your company has more than one other (branch) call center, testing should be conducted with each and every one of them. Additionally, testing should be done the other way, to confirm if each other call center were able to obtain backup from your primary call center and all of the others. This includes your virtual or distributed call center of telecommuters, if you have one. Why is this so important? The company must have reliable backup available in case of wide-area disaster. Such a disaster might strike any of the areas where your company maintains facilities. External (Outsourcer) CVM. The company senior management might decide to contract with one or more outsource call centers, exclusively. Alternatively, external outsourcer(s) might be chosen as a supplement to the company internal facilities that are available. Interim Call Handling. We suggest that it would be best to deal with testing your ICH facility by simulated testing. Another way to test your ICH might be during a sales campaign. In such a case, you would retain your normal inbound/outbound configuration at the primary call center. You would call upon your marketing department to do an advertising campaign with a special name, product, or prize inviting contact from customers via a special toll-free telephone number. You would arrange with your ICH facility for them to take

© 1998 by CRC Press LLC

care of all the call volume caused by the sales campaign, and route the toll-free number to their call center. This scenario could include a test of your recovery team ICH facility subteam. A bonus of this “communications” test could be a corollary test of your ICH facility accounts receivable/credit card-billing subgroup. The purpose of this exercise is to test the ICH outsource arrangement. Nevertheless, the marketing department might have interesting things to tell you afterward about the cost of advertising it and what net profit might have accrued to the company bottom line from the “special” sales. RTMO. The real-time minimum call overflow (RTMO) facility receives a number of

calls from you on a regular basis. Therefore, they do not necessarily need to test whether or not they can indeed receive the calls. Instead, your tests should be designed to ensure that they could receive a large volume of calls. This would test the ability of the RTMO facility to supplement the ICH facility in case of a high-volume call answer situation. We suggest that you should also test the individual RTMO CSRs, to ensure that their knowledge level concerning the products and services of your company is what it should be. The best way to test this would be to have a large number of simulated calls coming into your facility, and then switch them over to your RTMO facility. Another way of testing this would be to hand over a large number of your calls during a busy call period, and monitor the results at the RTMO facility. Hot Site. The hot site should be ramped up and tested at least once a year. You would do the hot site test by starting at your facility. You would activate the hot site recovery team (including CSRs) and put the designated number of people onto a bus. The clock would start on this test at the moment of declaration of a test disaster. (A separate part of the test might monitor the time and expense to get the bus.) How long would it take the team to travel by bus to the hot site facility? (This establishes a benchmark, to which you could add a factor simulating crowded roads because of evacuation or other possibilities.) The hot site recovery team would be instructed to get the hot site facility up and running. When backup data had been loaded on all work station computers, including popup data screens and all CTI functions, the team CSRs would start receiving calls. Many of the hot site tests that we have seen have been tests of the people at the hot site practicing activation of the computers and telephony. Conversely, in real disaster situations most of the time a hot site has failed, it was because of difficulties in getting the personnel to the facility. Normally the hot site itself passes activation tests. Doing an activation test only might give you and your team a false sense of confidence. When you do a hot site test, make sure you test the whole procedure, including transportation of the hot site recovery team to the facility. Virtual Call Centers. The main problem with virtual call centers is the difficulty that is experienced in actually streaming the calls to the proper individuals. When you start setting up a test for your virtual call center, take care in how you configure the test. You should make sure that the persons who are supposed to be receiving the calls are, in fact, actually receiving the calls. Also, whatever external command and control facility you use should be accepting the calls and directing them in the fashion that you intend. The best way to do this is with simulated call types coming in during an off-peak period, so that you can test the handling of calls without disrupting communications with your customers. Note that testing a virtual call center includes both telecommuters and also Internet call centers.

© 1998 by CRC Press LLC

Cold and Warm Site Facility. These types of facilities require a great deal of money and energy to bring up to operational condition. Normally testing of such a facility just involves testing the equipment to ensure that it is still functioning, and inspecting the facility for weather damage. It would be difficult actually to do a test where you started sending calls to a cold/warm site facility. This is why they are not considered part of an immediate RTO. The cold site (no power or telephony) or warm site (wired for power and telephony but not operational) should be regarded as a possible alternate site in case of an extended disaster. In all probability, the site would not be used unless the following occurred: 1. 2. 3. 4.

Damage to the primary site ICH and RTMO facilities handling calls Primary site proven impossible to reopen External hot site in use under temporary contract

WHAT ARE YOU TESTING? As you can see by the hazards and possible problems listed in Exhibits 19-1 through Exhibit 19-4, there is a great deal that could go wrong at any point in time. The probability is that the more likely kinds of disaster (especially human error) are the ones you would be most likely to have to deal with in reality. However, even the unlikely events can and do happen to someone in the world often enough to keep the journalists busy. So you need to test responses to them too. We have provided you with the charts to give you ideas for tabletop tests. For instance, you could take the airport proximity case and run a plane crash or hijacking situation as a tabletop test. How would such an event effect your company? What would your incident manager or alternates, and the various recovery subteams, have to do to ensure continuity of company functions? These are the kinds of exercises that you could walk a team through. This would get them thinking about more involved forms of testing. A tabletop exercise would be a good place to start designing your tests, because it is possible to examine all the different parts of the plan and see what each situation could do to your particular company. You and your planning team could also start getting opinions from all personnel regarding which of these items would be good to include in a test.

Revisiting Disaster Types and Causes Review the probability of disaster that is listed in Exhibit 19-5. You will notice that human error is the most probable kind of adversity that your company might have to face. When you configure your tests, make sure that you script them so that most of the problems that personnel are going to run into are the human-error type. Try and follow the percentages that are listed in Exhibit 19-5, unless you find that for your call center the figures are different. If a bulk of the disasters are going to be human error failures, you should test the same number of problems. A test loses its credibility dramatically if you are testing for typhoons in the prairies. If you plan tests derived from this chart, the tests are more likely to be of value to your company.

WHEN IS THE BEST TIME TO TEST? End of Peak or Off-Peak Volume Period The military always tests on nonpeak periods, and a lot of other agencies try to do the same. We believe this to be a good idea because it saves money. If something goes wrong while you are testing, you do not have peak-level problems to deal with. Testing at off-peak is the arrangement.

© 1998 by CRC Press LLC

EXHIBIT 19-1. SOLUTIONS TO NATURAL DISASTER (ANNOTATED) Disaster type

Problems

Solutions

Hurricane or typhoon

Evacuation, damage to buildings

Fortified shelters, glass-free eoc and emergency call center, passive IVR hotline

Internal flooding

Electrical, PBX, computers, comfort

Tarps to cover electrical equipment, vendor service-level agreements including furniture

Wind damage

Roof, fallen trees, injury from debris

Fortified shelter, tarps until roof repaired, chainsaws or contractor, advanced first-aid kit

External flooding

Damage to roads, destruction of furniture and paper files

Buses transport employees to workplace, contractors repair or replace furniture and dry of paper files

Epidemic

Illness or death of key personnel, nonfunctional workers

Cross training, replacement workers, distant ich facility

Earthquake, seismic damage

Collapse or condemnation of buildings

Distant ich and hot site facility, real estate buyer’s agent finds alternate site

Snow and ice storms

Electricity and telephony, impassable roads

Distant ICH, snow removal, check employees have heat and electricity

Volcanic eruption

Evacuation, ash, fumes, fire and lava, earthquake, thrill seekers

Evacuation!!! ICH, relocate to distant alternate site, salvage equipment if undamaged, crowd control, safety of families

Tsunami (tidal wave)

Evacuation, warned or unwarned; files, building damage

Treat as aftermath of earthquake, evacuate inland to high ground, ICH, alternate site

Electromagnetic interference

Power bumps, corrupted data, solar flares

Multiple backups, distant off-site storage, protective magnetic storage media

Non-Vacation Time You should ensure that few people are on vacation when you do the test. Try and encompass the largest group that you can. However, do not pull people off their vacations to do a test. In an actual event, you would not have extra people at your disposal. Also, canceling vacations would be bad for morale and might demotivate employees, especially recovery team members, from participating in the continuity project. Consider the case of a real disaster showing up during vacation period. We suggest that if any personnel might agree to come back from their vacations to (really) safeguard the company, they should receive double time off and some sort of bonus when disaster recovery was complete.

© 1998 by CRC Press LLC

EXHIBIT 19-2. SOLUTIONS TO INFRASTRUCTURE DISASTERS (ANNOTATED) Disaster

Problems

Solutions

Hardware failure

Stoppage of data processing or communications

Redundant equipment, ICH facility

Internal fire

Evacuation, smoke and water damage

Evacuation, ICH facility, vendor repair and replacement

Software failure

Confusion, probably cannot backup current data

ICH facility, uses “last backup” data

External fire

Road blocks, evacuation, smoke

Evacuation, passive IVR hotline, ICH facility

Power failure

Equipment unusable, no lights

ICH facility, get power restoration estimate from electrical utility

Facility structural failure

Personnel to rescue, denial of access

Heating, ventilation, or airconditioning failure

Discomfort, failure of equipment

Explosion, water service interruption

No cooling, no fire sprinklers, denial of access by firefighters

Loss of physical access to resources

Closure of building or area by emergency or government officials, doors locked during dispute

Gas interruption

Lack of heating

Malfunction or failure of CPU

Server cannot drive ACD or IVR

Gas leaks

Illness, evacuation, risk of explosion

Telecommunications failure

Wide-area calls cease, cellular phone to back-up

Activate ICH by cellular phone

Chemical exposure

Illness, rescue and evacuation, denial of access

Evacuation, decontamination, ICH facility, hot site, restoration of primary site possibly lengthy

Communications failure internal/external

PBX, trunk line

Transportation blockage or failure

Relieving shift delayed

© 1998 by CRC Press LLC

Find alternate routes, ICH facility to allow leaving shift to depart

EXHIBIT 19-3. SOLUTIONS TO MAN-MADE DISASTERS (ANNOTATED) Disaster

Problems

Solutions

External criminal activity

Extortion, theft, blackmail, police investigation

Security team on alert, cooperation with authorities, inventory of equipment, cash, and company credit cards

Civil disorder/riot

Assault, vandalism, looting

Switch to ICH facility, lock file cabinets, and safes, evacuate personnel

Disgruntled employee

Vandalism, theft, violence

Security team on alert, call police, deny access, if necessary switch to ICH and evacuate

Nuclear attack

Evacuation, denial of access, electromagnetic pulse

Distant ICH, if time seek distant evacuation site, otherwise use protected EOC or basements

Strikes

Picketing, violence, intimidation

RTMO or ICH facilities, security team on alert, escorts for arriving/departing shifts, call police

Chemical spill/toxic waste spill

Illness, evacuation, denial of access, EPA investigation

Data entry error

Sabotage of information

Switch to ICH facility “last backup” data

Kidnapping/death of key personnel

Confusion, delegation, trauma, police or FBI investigation*

Deputy and alternate personnel take over duties, alert security team, report any ransom demands, cooperate with police

Improper handling of sensitive data

Loss or compromise, lawsuit

Consult lawyer, public relations team, do not cover up

Vandalism/sabotage

Damage to equipment or files

Copy from off-site duplicate files, insurance claim, replace equipment

Unauthorized physical access

Theft, violence, sabotage

In progress: enact plan, after fact: repair or replace

War

Evacuation, failure of infrastructure, violence, flight of personnel, imposition of martial law

Plan for multiple sites, safety of personnel, relocation to alternate sites, hardened shelters

Malicious damage or destruction of software or data

Confusion, hacker, virus

Multiple backups, antivirus software, off-site storage of “last backup,” hardware and software vendor service-level agreements

© 1998 by CRC Press LLC

EXHIBIT 19-3. SOLUTIONS TO MAN-MADE DISASTERS (ANNOTATED) (continued) Disaster

Problems

Solutions

Biological contamination

Disease, quarantine, evacuation, denial of access, WHO or Centers for Disease Control

Distant ICH facility, replacement workers from out of area, emergency banking for evacuated personnel

Unauthorized access to data or theft of data

Commercial espionage

Radiation contamination

Corruption of data, quarantine, evacuation, illness, denial of access, government investigation

Robbery, theft, burglary

Replacement, police investigation, insurance claim

Vehicle crash

Road rage, car theft, injuries or trauma to personnel

Deputy and alternate personnel take over duties

Bomb threats

Hoax or real? Embarrassment, damage, injury, evacuation, police investigation

Treat as real, alert security and police, evacuation and ICH

Airport proximity

Plane Crash due to Hijacking

Danger of fire, enact plan, rescue employees or passengers

* RCMP in Canada.

Holiday Weekend Testing on a holiday weekend would be very expensive, because you would have to pay people time-and-a-half overtime. However, it could be a good idea to see how many personnel you could actually expect to come in during a real disaster at an inconvenient period of time. As in the discussion of vacations above, we suggest that a bonus or “Hero” T-shirt of some kind should be awarded to participants, plus compensatory time off. Please note that pets and dependents of personnel would have to be provided for just as if a real disaster were occurring. This might be a good opportunity to include families in the testing process. That is a good idea anyway, to test preparedness in case of wide-area disaster necessitating the use of shelters, the emergency EOC/call center if you have one, emergency food supplies (including canned milk and pet food), and so on. (See Part 5, “Crisis Management.”)

Midnight Shift, CSR Off Shift, One Hour before Shift Commencement The best time to test off-peak is the midnight shift. This is because most calls do not come in during the midnight shift and you can expect call volume to be low. However, if your company runs a call center that takes calls from around the globe, this might not be your best time to test. Look at your overall patterns of call volume. If international calls plus local calls show any noticeable peaks and lows, test during the low-volume periods. Another good time to test would be one hour before the next shift commencement. This would give you an opportunity to keep the CSRs that are going off shift and bring in a new

© 1998 by CRC Press LLC

EXHIBIT 19-4. SOLUTIONS TO ACCIDENTAL DISASTERS (ANNOTATED) Disaster

Problems

Solutions

“Monday Effect”

Lack of coordination, grouchiness, fuzzy thinking

More coffee! Advise more sleep, exercise, snack for energy

Fatigue

Slow reaction time, lack of motivation, errors of judgment, irritability, stumbling and dropping things

Give employee break time, advise snack for energy, during crisis: EOC cot

Illness

Danger of infection, fever delirium, effects similar to fatigue, memory loss, collapse

Send to doctor or hospital, if severe, call ambulance

Carelessness caused by fatigue or illness

Filing errors, haste, forgetfulness, “diskette in the microwave” mental processing errors, incomplete notes

Supervisor double-check work, break time, doctor or hospital, in crisis, EOC cot

Carelessness caused by poor attitude

Habitual sloppiness

Counsel, reassign, or transfer away

Carelessness caused by stress

Anxiety and forgetfulness due to family event, new position, crisis

Counsel, supervisor double-check work, meditation or distraction, in crisis, give break or snack if possible

Internal accident

Visiting child or pet, spills, corridor collisions, stairway falls

Keep employees focused, except security, paramedic, or maintenance

External accident

Vehicle crashes through wall, lost truck load blocks exits

Enact plan, partial handover to ICH or RTMO facility

EXHIBIT 19-5. DISASTER TYPE PROBABILITY Disaster Type

Probability, %

Cause

Human error

68

Mondays, illness, lack of sleep, carelessness, accident

Technological failure

25

Infrastructure, equipment

Natural causes

5

Weather, disease

Human intention

2

Malice

group from the next shift. You should try to hit the shift changeover period, so that you would get the maximum number of people there with minimal number of dollars spent.

HOW ARE YOU GOING TO TEST? Notice or No Notice Both notice or no-notice tests are required when you are testing. At first, start with preannounced testing so that company personnel (especially the recovery team) can become familiar with the testing process and with their checklists.

© 1998 by CRC Press LLC

As personnel become more at ease with the idea of plan testing, you and your planning team will probably get more value from no-notice tests. This is because very seldom do real disasters give anyone a lot of notice, unless they are impending wide-area disasters such as hurricanes. Personnel need to learn that the checklists work whether or not the test is a surprise.

Logical Review (Look for Flaws) The logical review is also called a tabletop exercise. Running such a test gives you and your team the opportunity to talk through the plan and checklists, and look for flaws. Some items that seem good when you write them down on paper might look illogical when you reread them at a later date.

Simulated Input Operational Evaluation (Talk-Through) You would do this type of test by taking your recovery team and dividing them into two sets of people. One set would create some simulated input material and the other set of people would take the recovery manuals that you have created and actually talk their way through responses to each input. An “input” is a test circumstance that is given to the participants verbally or on paper. The personnel running the test would give inputs to participants, each supplying only one part of the event scenario. For example, one test “input” might say, “A loud crash is heard in the parking lot.” Another might say, “The lights go off, and the computer screens all go dark.” These two inputs might be part of a combined truck accident/power failure scenario where a truck strikes the utility pole or poles bringing power lines to your facility. Usually, test inputs would be created beforehand and sealed in envelopes. The envelopes would be labeled with a test period date and time. When that time comes around, the team members doing the test would open the envelope. When they saw what kind of problem they had, then they would have to tell the other team (in accordance with their checklists) everything that they would do to handle that problem. Meanwhile, the other team would make notes recording how the participants were handling the situation. Next time, the teams would switch places.

Physical Verification Exercise (Walk-Through) This type of test is similar to the simulated input operational evaluation except that this exercise requires personnel to actually get up and walk through the steps of responding. This is important because checklists sometimes are written that ask people to do things that are impossible. For instance, we once saw a company that had instructions for people to get down into their shelter during a tornado. The problem was that somebody had put up a wall between the operations room and the hallway leading down to the shelter. No one on the planning team had noticed the change. If people had followed the checklist in a real emergency, they would have walked face-first into a wall. They would not have had any clear indication how to get down to the basement shelter. This sort of problem sounds amusing until you envision the confusion and panic that could result during a real tornado. Injury or loss of life could result. As a side note, if the company in question were updating its plan and checklists regularly, that situation would not ever have arisen.

Evacuation Exercise (Backup, Hand Over, and Walk Out) The evacuation exercise is a mainstay of disaster recovery planning. Many of the disasters that come along will require that personnel leave the building. This exercise actually

© 1998 by CRC Press LLC

involves all personnel practicing handing over calls to another facility and getting out. The procedures for an evacuation exercise are to backup all the data in accordance with the checklist, hand the calls over the ICH facility, and get out. You do not have to evaluate your whole call center with this exercise at once. You could take a number of people from one side of the call center and use the other side of the call center as the handover facility. However, to train all personnel, it would be valuable to run an exercise periodically in which everyone has to leave the building, including executives. This would allow personnel to run through the “head count and condition of personnel” checklist and confirm that everyone knows how to find the prearranged meeting place.

Computer Simulation Testing (Heartbeat Drill) There are two types of computer simulation testing that are available. One is the WinCAPS Computer Simulation Testing offered by AT&T. This will simulate the flow of your calls in your own environment and allow you to see what your results are from switching your calls from one facility to another. Another type of computer testing is offered by Interactive Quality Services, Inc. IQ Services provide simulated calls that are generated from a computer. These calls are programmed so that they will go through your IVR and put in the proper inputs. When the calls come to the CSR, they will require the CSR to enter in digits on a timed basis. The digits are randomly generated. The purpose of this activity is to tie up the CSR for the same amount of time as a live call.

Full-Function Tactical Evaluation (from Event to Resumption of Operations) This is what we refer to as a full-body workout. It involves all company personnel, the backup facility, and the critical vendors, local government agencies, security team, and public relations team. You should write this test so that it goes from bad to worse. You would start with a single event such as a hurricane warning. Then you and your planning team would write the exercise, simulating the storm as it goes from a hurricane watch then to a hurricane warning and finally it starts bearing down on your facility. You want to observe how company personnel do the commencement of handing over calls and getting everyone under shelter, and so on. Your test should go through a single incident all the way through to its conclusion. It is a good idea to use simulated calls and to do this late at night when the call volume is low. Another alternative might be a holiday weekend; however, you might have to work hard to convince all participants, especially your vendors.

WHO ARE YOU GOING TO TEST? Recovery Teams These are the first teams that get tested. They are involved in the logical review and the simulated input operational talk-through. Generally they are the only people involved in those exercises. Recovery team members must be involved in every exercise, and they are the first ones involved, since they are the ones who would assess a real disaster and decide how to react. Do not forget to include your planning team on some of these tests, even if you have to divide the team in half for test purposes. Other than the recovery team, they are the members of the company most likely to be familiar with all parts of the plan. You might consider appointing these people to be incident manager alternates or deputies.

© 1998 by CRC Press LLC

Team Leaders Team leaders generally become involved once you have reached the physical verification exercise and the evacuation exercise. They should be involved all the way through the fullbody workout.

Call Center by Department The call center subdepartments such as the help desk, customer service, and sales, plus technical support line CSRs should go through, testing department by department. They would begin testing generally when you start to work on the evacuation exercise, although their input can be helpful at any time during the exercise process.

Facility Managers Facility managers should be aware of the testing and review the results. These people need to know what is going on with their facility. The facility managers themselves will generally become involved in the actual testing process during evacuation exercises or full-function tactical exercises.

WHY ARE YOU TESTING? (RESULT) Training Value The training value of the testing cannot be understated. The first time you do your test, it will seem like a total mess. This is because people are doing something that they have not had experience doing before. The aim is to train all personnel to the point that, if necessary, they would be able to find their exits without having to search for them. The aim is to train them to be able to follow plan action checklists such as the “data backup” or the “Telco prerouted switching,” or “handover of calls to ICH facility” checklists quickly and confidently.

Logical Faults As a general rule, when you are first writing the plan and associated checklists, you are probably going to forget things. There will be steps left out in the checklist that would be disastrous if personnel had to use that checklist in a real disaster. If you should find any steps left out of a checklist during the testing phase, it is a simple matter to amend it. Try not to be discouraged if you do find any such missing steps. As we said before, just be glad that you and your planning team can catch the fault in testing and not in reality.

Prove Plan Functionality Sometimes the senior management does not believe that something will work. Proving the functionality of the plan (such as recovery to an alternate site) by testing is an excellent way to show that the plan will work. Sometimes the best of processes fail to stand up under the light of testing, so you will have to decide how rigorous your test is going to be. You must be prepared to change the plan if it is not functioning properly.

Prove Plan Current (Vendor and Other Contact Lists) Testing can prove that the plan is current. Normally during a test, if you open up a checklist and it says to phone “ABC Company,” and you have not been doing business with ABC Com-

© 1998 by CRC Press LLC

pany, that is going to come to somebody’s attention. The vendor and other contact lists must be current. Testing can also show if the vendor has not been informing your recovery team external liaison representative of personnel changes or address changes in the vendor’s own operation. The vendor might assure your company representative that yes, “Vendor Company” really truly does believe in the necessity of continuity planning. However, at first, getting all your vendors to comply with your requests for confirmation of current contact data might be an uphill battle. We suggest that if your purchasing agent (taking cues from your CFO) has reason to doubt the commitment of any vendor to the continuity project, that your company should find another supplier. During a large-scale or wide-area disaster, the survival of your company might depend upon the ability of your vendors to fulfill their service-level agreements. Just as the banks must require the other banks that they deal with to have workable business resumption plans, so must your company require its vendors to be effective members of your continuity project team.

Instill Confidence (Team!) The testing that your company does will instill the confidence that company personnel will need when the chips are down. We have heard both in television reports and in person how valuable the confidence is that people can derive from their training. “We were trained for this, so therefore we knew how to do it,” say people after an emergency. It is important to instill confidence in the team and let them know that they can get through the test problem. After the test, it is important to have a “postexercise” assessment meeting, followed by a memorandum that lists the valuable skills learned. If the plan was shown to need amendment, the memorandum should discuss that, too. This is important so that personnel who have undergone the test that discovered a plan flaw will not blame themselves for the weakness that was revealed. A word about morale: the attitude of your people is very, very important. If they receive sufficient training to develop a “can-do” attitude, they will use their skills and confidence to triumph over adversity. If they develop confidence in the checklists, and confidence in the leadership of the recovery team, in the event of disaster they will be much more resilient. (Conversely, without a plan, leadership, or training they could just succumb to feelings of helplessness or panic.) So, after a test, feel free to hold a small “Yahoo” gathering or other morale event. Treat them like heroes, and they will become heroes.

© 1998 by CRC Press LLC

20 Types of Testing

I

n a section of the previous chapter (How Are You Going to Test) we touched briefly on the subject matter of this chapter. We discussed the five different types of tests: the simulated input, physical verification, evacuation exercise, computer simulation, and full-function tactical evaluation tests. These various kinds of tests can help you determine how well the plan works, and whether company personnel (and external parties) need more training.

SIMULATED INPUT OPERATIONAL EVALUATION The simulated input operational evaluation is designed to test the overall accuracy of the checklists and other similar items. In the following test you will see that a radio is an integral part of the test. Whether or not the person going through the test has turned on the radio helps to determine if the test was successfully completed.

Paper Test With a paper test, you divide your recovery team into two groups. Have one or two people on that team design an input test similar to the one below, and have the rest of the group respond to the test. This requires planned inputs.

Planned Inputs You can see in Exhibit 20-1 that there is activity dependency. This is the action that must have been done in the previous input, to be able to accept the next input. That is to say, if the person did not turn on the radio, then they could not receive the tornado warning that was broadcast. The purpose of the CSR telling the call center manager was to get him to implement the severe weather warning checklist. During this evaluation, if the radio was not used, then the first indication of a problem was the power failure. The next would have been the sirens. Unfortunately, tornado-warning sirens often do not give enough warning. Typically, these sirens go off only minutes before the tornado hits. If evacuation instruction was given when the radio indicated the tornado had touched down, then no employees were injured and the call center can be recovered according to the disaster plan. If the manager waited until the sirens went off to evacuate personnel, then there were some injuries and possibly some deaths and the call center can still be recovered. But the manager who did not evacuate has a huge number of injured and dead to deal with and the call center is effectively out of business. The manager who evacuated and began the disaster plan right away accomplished the disaster plan objectives in the time frame that the plan called for. However, if the call center manager waited until the sirens sounded, then the plan would have taken longer to implement. The chaos that would be associated with a last-minute dash to a shelter would make it difficult to implement the call volume recovery plan. Unless there was one person working to implement the voice (call) recovery portion of the test and one person working to ensure that all employees were safe, there might be significant delay in implementing the plan. This exercise looks at what were the manager responses. What were they and how could they be improved? A look at your checklists might give you some clues for future training.

© 1998 by CRC Press LLC

EXHIBIT 20-1. PLANNED INPUTS ACTIVITY DEPENDENCY

Activity Dependency

Input Source

Result (Expected Action)

Activity Number

Time

Activity

CSR

1

9:00

Reports severe weather warning heard on her way into work

Must have turned on radio

Radio

2

9:30

Radio alert that a tornado watch has been put into effect

Must have turned on radio

Radio

3

10:00

Radio alert that a tornado watch has been put into effect

(Call center manager should evacuate personnel)

4

10:15

Power failure

(Call center manager should implement power failure procedures) Evacuate personnel

(Call center manager should turn on radio per checklist)

Must have turned on batteryoperated radio

Radio

5

10:23

Tornado is two miles south, southwest of call center

Evacuated when radio warning was given, radio with batteries was on

Radio

6

10:24

All personnel take their checklists and go to the tornado, hurricane shelters

Took “to go” case

In shelter

7

10:28

Begin to activate call center disaster recovery plan by cellular phone

Manager instructed evacuation

Windows blown out of call center

8

10:33

All personnel in shelter when tornado strikes call center

No injuries

Sirens

9

10:30

Evacuate personnel

Sirens did not give enough warning to evacuate all personnel

Windows blown out of call center

10

10:33

Tornado strikes call center, sirens did not give enough warning to evacuate all personnel

5% of personnel are injured or dead

Evacuated with sirens or waited for other warning to evacuate.

© 1998 by CRC Press LLC

EXHIBIT 20-1. PLANNED INPUTS ACTIVITY DEPENDENCY (continued)

Activity Dependency Did not evacuate

Input Source Windows blown out of call center

Took “to go” case into the tornado shelter

Activity Number

Time

11

10:33

Tornado strikes call center

12

10:50

All calls recovered to other locations according to disaster plan

Activity

Result (Expected Action) 63% of personnel are injured, 15% are in critical condition, and 2% are dead

Critical Business Functions Critical business functions for this particular input were dependent on the manager’s responses to input sources. If the radio were turned on, then there would have been time to shelter all personnel and to implement the call volume recovery portions of the disaster plan. If the manager did not have the radio on, then there was little or no time to hand over the call volume and other functions until after the tornado had passed and destroyed the call center.

Not Time Sensitive This exercise is not time sensitive. You can run this in quick time, such as one minute equals two minutes. Or, you could run it even faster, as one minute equals five minutes. You and your planning team should make sure each person completes an activity dependency. If the person (the manager, if the manager is the one whose decisions are the subject of the test) does not turn on the radio, then he or she does not receive any disaster indicators until that activity dependency is completed. Just go on to the next input. You cannot hear radio tornado warnings during a power failure without a radio powered by batteries.

PHYSICAL VERIFICATION EXERCISE Walk-Through with Checklists Exhibit 20-1 works well with the physical verification exercise. With this exercise you can have the inputs come from the actual sources. It is vital that when input is given, the person states that it is an exercise. “For the purposes of Tornado Test #1, I heard on the radio that there is a tornado watch in affect.” This way, real warnings could still be given and not confused with part of an exercise. In the physical verification exercise, your people delivering the inputs should give the information as if it came from a real source or a simulated source. For instance, the person would say, “For the purposes of Tornado Test #1 I am giving a radio warning. A tornado watch is in effect for the greater Buffalo area. In the event of a tornado the local shelters are.…” Your personnel doing the test need to walk through the checklists. If the checklist calls for the person to turn on a radio, then that person should go to the radio and physically turn it on. During a (simulated) power failure, the person should physically unplug the radio and have it run on its batteries.

© 1998 by CRC Press LLC

Simulate all aspects of the checklists without affecting the entire call center. If in a portion of the test there is a (simulated) power failure, the person or the team running the test should not shut off the power to the call center in reality. They should simply test the equipment that would be affected or required in such a situation, such as a portable electrical generator, to make sure that the equipment is in working order.

Where Are the Emergency Exits The person, or people, should go to the emergency exits and the emergency shelter that is on the checklists. When the order is given to evacuate, the team administering the test should go through the emergency exits and walk with the people taking the test. This means physically going down to the shelter and completing the simulated evacuation. When testing a portion of the plan, you might want to take a number of people to do the physical verification of this portion of the test. Do not take just one or two people. We recommend that you get at least one person from each part of the call center to simulate the (physical) evacuation. This is so that you will be able to time the whole procedure. Then you will know how long it would take to get the first person from each location into the shelter from all areas of your call center. This also applies to testing the response of people in other departments such as accounting, shipping, sales, and so on. You should also ensure that once personnel have arrived at the shelter they follow their checklists and do a head count. Is anyone missing? In a real disaster, it would be necessary to know if anyone might be trapped in debris outside the shelter.

Where Are the Backup Disks If the call center or other department needs backup disks or other pieces of equipment to function, that should be noted in the checklists. When testing that portion of the plan, the people going through the test should physically go and pick up necessary items such as the “to go” box or case. The people should physically make sure that the disks or equipment are in working order and depart carrying them.

What Equipment is Used for Handover of Calls The people doing the test should go through step by step on the checklist, and physically make sure that all of the equipment that is required to hand over the calls and to shut down the call center is there. It is important to make sure that company personnel know where the equipment is, that the equipment is operational, and that the checklists are accurate when it comes to the proper use of that equipment.

Time Sensitive Normally the physical verification exercise should be time sensitive. The people running the test should give inputs in real time, and they should stick to the scripted times. This will help you and your planning team to know if the plan time frames are realistic.

EVACUATION EXERCISES Evacuation exercises are similar to the physical verification exercise except that a larger portion of company employees is involved. The purpose of this exercise is to get all employees familiar with the disaster recovery procedures, so that all evacuation and other disaster recovery functions become automatic to them (see Exhibit 20-2).

© 1998 by CRC Press LLC

EXHIBIT 20-2. OTHER EVACUATION EXERCISES Disaster Type

Evacuation Action

Tornado

Secure shelter without glass

Earthquake

Shelter under desk, then exit building

Bomb threat

Exit building immediately

Fire Drills (out of Building) In many buildings, a fire alarm has two distinctly different alarms. The first, usually two short rings repeated, indicates that a fire is suspected, and will warn personnel that an alarm has been sounded and that personnel are checking out the area. The second alarm, usually one long ring, indicates that a fire has been detected, and is to tell people to leave the building. Most fire departments recommend that people do not leave the building during the first (double-ring) fire alarm, but when the second (single-ring) alarm is sounded, then people should evacuate the building in a calm and orderly fashion. However, you should check with the fire department to find out what the standard is for your building and area. Fire drills should use one half to one quarter of your staff during a nonpeak period. You and your team who would be running the test should first tell all employees that they would be conducting a fire drill. Ask the personnel assigned to be involved in the drill to act in accordance with their checklists.

Handover of Calls and Head Counts All personnel who are involved in the drill should go through the procedures in their checklists, terminate their (simulated) calls, leave the building via their checklist fire routes, and proceed with the rest of their checklist. All types of drills should be handled in the same fashion. All evacuations must include handover of calls and include head counts, and the checklists should reflect this. These are two essential actions that should be part of every evacuation, no matter what the scenario. If (simulated) handover of calls should occur after the (simulated) head count, okay. As long as the “to go” case contains a satellite or cellular phone, (simulated) handover of calls could be done from a shelter or predetermined meeting place. All evacuations are time sensitive. People should complete their checklist actions in a safe and timely fashion. It is worthwhile to time each section on its performance.

COMPUTER SIMULATION TESTING AT&T WinCAPS There is very little computer simulation testing software available on the market to date. Earlier in the book, we mentioned AT&T WinCAPS software. If you are an AT&T customer, call the company to find out if this service is available for your call center.

© 1998 by CRC Press LLC

IQ Services Another service that is available is from a company called Interactive Quality Services, Inc. The following is from some of its literature: IQ Services is a unique professional services organization that does not sell hardware or software. We offer … a complete package for the testing of disaster recovery plans. Test calls are launched from our location with a powerful automated testing system that provides detailed reports and customized calls to meet your specific requirements. We can respond to your immediate need to design and run test drills in pursuit of confirming that your disaster recovery plans work as designed. … StressTest™ Drills When you develop and deploy new solutions, StressTest™ Drills are designed to load-test voice processing systems, call center systems and associated networks before you turn them loose to your customers. … HeartBeat™ Drills When your applications are in production, HeartBeat™ Drills are designed to inexpensively fulfill your needs for documented verification of up-time and rapid notification in the event of any issues. HeartBeat™ Drill makes regular telephone calls into your voice processing systems and exercise their services just like a regular customer caller. Along with an immediate notification of any issues that occur, a detailed report of the past week HeartBeat™ Drill activity will be e-mailed or faxed to the beginning of the following week. … A HeartBeat™ Drill has two primary goals in testing a target system: 1. The voice response or other target system is up and working with key functions. 2. Interfaces and integrations are functioning normally.

The verification process involves placing a specified number of calls for each application being tested. Greetings, prompts, response times and access to information are typically verified. If the call or calls are not handled as expected, then a notification telephone call is generated. … Voice response systems, PBX systems, and call centers are evolving into heavily integrated environments. Each user organization has specific business goals, existing information technology, and management styles that drive unique solutions. Verifying correct and robust capabilities of these unique arrangements is a challenge to traditional testing methods. … Test drills are designed and performed to exercise systems with telephone calls that closely replicate real customer calls…. Disaster recovery procedures may require periodic Drills to verify procedures and facilities are ready every day. … The test plan for each situation is different, but there are some common elements. • Because the test calls use the public network and private systems to gain access to the testing target, a Drill may test several things at once. • Network Call prompting, call allocation, call distribution, call redirection and other capabilities can be verified for functions and capacities. A variety of traffic patterns and volumes can explore the effectiveness of call handling mechanisms.

© 1998 by CRC Press LLC

• Automatic Call Distribution DNIS routing, ANI routing, time/day routing, call prompting, management reports, and other capabilities can be verified for functions and capacities. Interaction with voice response systems is a prime area of interest for verification. • Interactive Voice Response ACD interaction, call prompting, information input, information access, database access, legacy system interaction, transfer to live ACD agents, and other capabilities can be verified for functions, capacities and response times. The custom nature of these applications make them prime targets for verification under high traffic conditions. • Computer Telephony Integrations ACD and other interfaces with data systems can be exercised for call routing, agent expertise, screen pop, outbound preview dialing, data network impact, and related services in terms of functions, capacities, and response times. The custom nature of these applications make them prime targets for verification under various traffic conditions. • Outbound Predictive Dialing Outbound calling campaigns can be launched to the IQ Services location to verify call process tone detection, answering machine detection, call handling after answer, nuisance rates and so forth. • Management Reporting In addition to the management reports provided by individual systems, custom management reporting arrangements that consolidate information from multiple sources or multiple call segments can be exercised to establish their performance and accuracy. • PBX & Voice Messaging A call to each Direct Inward Dial telephone number can verify proper telco assignment, correct PBX administration, voice message coverage, fax machines, modems, and more. A rigorous Drill run during the hectic cutover time can quickly and economically validate the quality of the situation. • Other Capabilities Callback messaging arrangements, major component upgrades, intermittent problem conditions are candidates for Drills to verify functions or stress specific capabilities.

As you can see, the IQ Services testing package is very comprehensive.

FULL-FUNCTION TACTICAL EVALUATION Yes, It Is a Big Deal The different tests that the company has been working on up to this point have been preparing the personnel for the full function tactical evaluation. This is a big step because of all of the planning that has gone into the disaster recovery plan to this point, to cover a range of circumstances. In running a full-function tactical evaluation, you should add in as many different circumstances as possible, because you will not get to do a full function test very often. The full-body workout is an awfully big deal. Why would senior management agree to disrupt company operations to such a degree? Why would vendors agree to participate? Why would local emergency departments like the police, the fire department, and ambulance services agree to take part? Why would government agencies like FEMA or groups like the American Red Cross ever want to join your company in such an enterprise? The answer to these questions is, a full-function tactical evaluation is a big deal to everybody. These other groups have the same need to do serious disaster training as your company has. And why? To exercise their plans, to train their people, the same as your company must.

© 1998 by CRC Press LLC

Depending on the size of your company, you might have many vendors to contact, or only a handful. We suggest that you send out a written invitation to participate in joint training, and coordinate with the various groups to come up with a date that is agreeable to the majority of possible participants. A side note: We caution you that you must have the approval of senior management to run this kind of test. In case that seems too restrictive, please remember that the meltdown at Chernobyl was the result of an unauthorized test.

Real Emergencies Come at Worst Time Do not be afraid to push the limits on this test. Real emergencies come at the worst time, come in a variety of forms, and are generally uncooperative with humans and their ideas of comfort or convenience. Real emergencies often also come along in the worst season, time of day, and time of week. Frequently, they will occur when key personnel are away at conventions, are sick at home, or went away for the weekend.

Obtaining Cooperation of Emergency Organizations The fire departments and other emergency organizations are generally very happy to see companies do a variety of these types of tests and are usually willing to help in any way possible. The better prepared your company can become, the more likelihood there is that if a real disaster should strike, your company personnel would be well trained. Emergency personnel would far rather arrive at your site and find the building evacuated and the employees gathered at the predetermined meeting place, than the alternative. Please remember that if a real emergency should take place elsewhere at the same time as your test, the emergency organizations would have to cancel their participation in your test. You and your planning team should be prepared to simulate their presence with written inputs if necessary.

Obtaining Cooperation of Your Critical Vendors The best way to obtain the cooperation of your critical vendors is to inform them well in advance of the exercise. Then when the exercise comes along, make sure that it includes inputs exercising the participation of these vendors. This will allow them to test their response times with regard to their service-level agreements. Ask the vendor representatives to give you documents on their test evaluations, so that you can ensure that service-level agreements could be met. This includes computer hardware and software vendors, electrical generator suppliers, electricians and plumbers and heating/cooling contractors, among others. Other venders who should participate are your interim call-handling facility and your real time minimum call overflow facility. During the course of a full-body workout, you will be sending your calls to these companies. (Run these tests at nonpeak call volume times if at all possible.) The test might entail sending your entire call volume or a portion of that call volume to these companies. If you should know in advance that some part of the call volume management system has limited capacity, try to work out the difficulties before the full function test. For instance, you might alter the plan to include shunting low-priority calls to IVR. The test would then exercise that plan alteration and prove whether it works. Try not to schedule the test during any prearranged sales campaigns.

© 1998 by CRC Press LLC

Obtaining Cooperation of Company Personnel Obtain cooperation of company personnel. It is vital that every employee be involved in this type of test. Each department should test its plan and see how these plans affect the plans of other departments. When it comes time to test this plan, then anyone in the organization that is in the facility that is being tested would have to act according to what is going on in the call center. You might want to get some personnel to simulate injuries (major and minor) and see how participants deal with these injuries.

Public Relations “Public relations” can be both internal and external. Your full-function tactical evaluation should exercise both. Therefore, your exercise plan should include inputs for the public relations team, to show whether or not its departmental plan has any flaws. Employee Communications: IVR Situation Line. The passive IVR hotline should be used, just as if it were a real disaster, for inbound and outbound messages. This should prove whether employee checklists include the toll-free hotline number, whether personnel could call into leave situation reports (especially the recovery team personnel), and whether personnel could receive situation updates or instructions concerning emergency shift schedules and shelters, etc. The training value of the IVR input is the practice that employees would receive. All personnel (including senior management) need to become familiar with calling in and registering their locations. (For instance, if some go to simulated shelters and some go to visit simulated relatives outside the local area.) If you simulate an evacuation, you should send a number of people to pay phones to test your IVR communications system. Have these employees register singly and as groups. You should also test for call blocking, and test the IVR unit outbound capability for calling the employee contact list. This should be done by having a number of calls go out to cellular telephones or to employees homes. Media Relations: Will You Be Inviting or Simulating? Please remember, when you design this exercise, that you do have a choice of whether to invite the news media, or simulate their actions with written inputs. However, in a real disaster, you would probably not have a choice. In such (real) events, your public relations team would provide the incident manager with prewritten statements that present the facts truthfully, do not assign blame, do not cover up, but do describe the situation with sensitivity and a minimum of emotion. The newspapers, television, and radio stations love to take part in this kind of event. It offers some drama without the risks involved in daring real hurricane-force winds. Participating in such an exercise would be a good opportunity for media representatives to show that they are being responsible corporate citizens. There are some drawbacks, however. Often things go wrong in such an exercise, and that is why your company and all those emergency organizations are testing. If the plan should have flaws, you have to identify them. What if a journalist should catch wind of the problem? How do you explain things to the press? This is where your public relations team keeps control of the situation by issuing a prewritten, prescheduled press release. Such a press release would NOT say, “We are simulating the possible deaths of fifty employees due to a management error of judgment.” The public relations team should prepare a press release that might instead say something like the following.

© 1998 by CRC Press LLC

In the [Name] Company full-function disaster exercise, we simulate a [type of disaster, e.g., weather event] that strikes the [Name] city core in the early morning hours of a (day of week). Our company simulates the possible toll on our employees and the property and equipment damage that might occur during the [type of storm]. This is what might happen if a real disaster came upon the city without warning. We are testing our disaster recovery plan in conjunction with local emergency organizations so that we can improve the safeguards for the safety of our employees. Our company simulates what might happen to any company in this area if such an event took place. Liaison with Senior Management. Your public relations team should also simulate

preparing situation updates for the incident manager to provide for the CEO and other senior management. (Please look back to Chapter 8 concerning the incident manager and senior management.) Again, your public relations team should prepare prescripted report templates for use during simulated disasters — and in case of real ones, too. Real-Life Disruptions and Casualties. All of the live situation reports that would be completed as the exercise progresses, and afterward, should be submitted to the recovery team representative assigned to receive them. However, there has to be a method of monitoring the “live” side of company operations. Manual or automatic recording procedures for the live environment must be maintained. If the live environment were to suffer disruption, you and your planning team would have to enact exercise termination procedures. (In the military, a special code word is used to indicate that a real casualty or real injury has occurred. Anyone who suffers a real injury is pulled out of the exercise, and the person stays out until further notice.) An example of this in the real-world experience might of your company if your interim call-handling facility reporting a disaster of its own. The company automatic call direction and management for the live environment must be maintained. You need to be able to go back to your live environment at any time.

Handing Over Calls: Keeping Customers Out of It We recommend that you use simulated calls during your full-function test and that you keep customers uninvolved. The simulated calls could be generated by IQ Services (as discussed above) or you could generate your own calls. If you should chose to generate your own simulated calls, you might decide to hire people from your community on a temporary basis. Alternatively, you could hire students from a nearby community college (especially if they are in a call center skills program) to call in with scripted messages to test your call handling. If that were not satisfactory, you might set up some inbound calls from one of your other call center facilities. You should either set up a toll-free number for this purpose, or use some other special inbound test number. To keep the test separate from your customers, you should ensure that all customer lines and most of your administration lines are switched over to your interim call-handling facility before the test begins.

Security Control of Access to the Facility (Gates, Doors, Furniture, Signage). Security is a big issue during these exercises. You and your planning team need to make sure that you can maintain security of the facility. To that end, you should divide your security personnel into two groups (if possible) — those helping to plan the exercise and those who will take part in the exercise.

© 1998 by CRC Press LLC

During the exercise, there will be many people going in and out of the building in a short period. You need to do this in the same fashion that you would if it were a real disaster. Will a large number of people be entering and departing your call center during the exercise? Control of access to the facility could be compromised. Prevent Unauthorized Property Removal (Looting). You might want to place security at vital areas of the building to ensure that only personnel who are supposed to go into those particular sections of the building are allowed to do so. If you feel that this is necessary, you might want the security personnel to keep a log of who goes in and out of the areas that they are watching. You might need extra security people to ensure that people do not steal from the call center during this type of exercise. This would be especially important after evacuation of the building. If your security setup is already rigorous, this type of exercise should not present too many changes. In that case, you would already control access, and you would already maintain logs of those entering and leaving. You would probably already have names and photographs of vendor representatives. Interact with Police and Emergency Services. Security personnel should be on site to interact with police and emergency services and offer needed assistance to these emergency personnel as requested. They should be prepared to assist them in locating offices where exercise-simulated fire or injuries have taken place. When Security Personnel Should Evacuate Themselves, Too. You and your planning team (including the planning team security representative) need to plan procedures for when the security personnel should evacuate the call center as well. Is everyone else evacuating into the basement because of a tornado or hurricane? Is everyone else leaving the building because of fire, smoke, or poisonous fumes? Let us consider that if the building is too dangerous for company employees, it is probably too dangerous for looters, too. However, during an exercise the dangers are only simulated, so entrances to the facility must be protected even if the security test subjects are gathering with the rest of the simulated evacuees at the predetermined meeting place.

Time Sensitive The entire exercise is time sensitive. Suppose that at 10:27 P.M. a specific exercise input or response is needed. You and your planning team should note at that time whether or not the response took place, how long it took to perform, and whether or not the response would have been successful when followed by other timed inputs. You can see in Exhibit 20-3 how the various activity dependencies work together. In this case, the facility would be having a multiple problem disaster. Exhibit 20-3 shows how one missed action could lead to dire consequences. However, no single error of judgment could be entirely catastrophic. However, the team undergoing the test could recover a partial error by taking correct actions later. We hope that this and the other examples will help you when you start drafting your company exercise plans.

© 1998 by CRC Press LLC

EXHIBIT 20-3. MULTIPLE PROBLEM EXERCISE DEPENDENCIES Activity Dependency

Result

Activity Number

Time

Activity

(expected action)

Fire alarm system

1

12:45

Two short rings for your fire alarm warning

(Get checklist and send security with a two-way radio to find location of fire on your fire alarm system)

If they checked fire alarm system

Cafeteria alarm

2

12:48

Fire in cafeteria

(Send security personnel with a two-way radio to find out if there is a fire in that location; make sure you have what you need to leave the building)

If they checked fire alarm system

Security personnel

3

12:54

Toaster fire small

(Used fire extinguisher, fire is put out)

If they checked fire alarm system and security has two way radio

Message from security over two-way radio

4

1:05

Fire is out; fire department called and notified; all clear.

If did not check fire alarm and security has a two-way radio

Radio in from security over two-way radio

5

1:05

Toaster fire in kitchen but an employee tried to use a pot of water to put it out; fire is spreading; one injured employee with electrical and fire burns

Phone call to security desk

6

1:08

Call says, “By now you have detected a fire in your kitchen, so you know I am serious. I have placed a bomb in your building. If anyone leaves, it goes off. If my demands are not met, it goes off. I want $100,000.…”

© 1998 by CRC Press LLC

Input Source

(Write all information down. Keep caller on the phone as long as possible; start evacuation)

EXHIBIT 20-3. MULTIPLE PROBLEM EXERCISE DEPENDENCIES (continued) Activity Dependency

Input Source

Result

Activity Number

Time

Activity

(expected action)

Kept caller on the line

Bomb threat call ends

12

1:10

Caller

Did not evacuate or did not pick up the phone call

Boom

7

1:12

You are all dead or injured

If did not check fire alarms and security does not have two-way radio

Security is leaving with injured person

8

1:15

Fire detected in kitchen, one person seriously injured

Call 911 for fire department and an ambulance for the injured worker, evacuation

If no two-way radio but checked fire alarm system

Security person

9

1:20

Fire is out at this time; no injuries reported.

Call fire department and tell them the situation

Fire suppression system activated in kitchen, sirens sound

10

1:22

Fire is out; fire alarm in full effect

People are evacuating on their own

If did not try to keep caller on the line and fire alarm went off and people evacuated

Bomb threat call ends; but a quick call back is received

11

1:25

“I said no one is to leave the building.” BOOM!! Some people get out, most are dead or injured including you

Kept caller on line, evacuation ordered, started call volume DR plan

BOOM!

13

1:40

Everyone is safe, call center is destroyed, calls are being recovered

14

2:55

No injuries; all calls recovered to other locations according to disaster plan

Fire was put out, kept caller on line, evacuation ordered, started call volume DR plan

© 1998 by CRC Press LLC

You have written down the information; ordered evacuation and start of switching call volume in accordance with DR plan; (some one else) by passing a note to security with some details

21 Test Objectives

T

his is a set of possible test objectives for a full-function tactical evaluation. Does the plan as a whole function? Is there continuity of business processes? Do backup plans work? Different parts of the evaluation check out different aspects of the call center business continuity plan, but, taken together, they familiarize your personnel with emergency procedures in a way that just reading a manual never could. You could use different aspects of these test objectives to evaluate the other tests as well. However, in a full-function test the following objectives should be met.

PROVE FUNCTIONALITY OF THE PLAN The basic objective of all tests is to prove the functionality of the plan. This is not an easy thing to do at first, because a new plan is likely to be rough around the edges. You should probably run the logical assessment as the first test, because this should show the holes in logic of your plan. Does the plan call for people to do things that are impossible? Has any step been left out of the basic processes? For example, does the plan say that the “to go” case should be used by the incident manager, but does not talk about preparing the case first? How about sending personnel to a shelter without evacuating them from the building as a necessary first step? Do not worry if you find a flaw at this point, just fix it. Every version of the plan will get better. The next step is the tabletop test. We practically guarantee that you will find more problems with the plan. Try to get some enthusiastic people to talk through responding to a specific disaster by following what you and your planning personnel have provided in the plan. Some people will take particular joy in finding problems in your plan. Remember to thank them! The idea is to test the plan and find problems before anyone has to put the plan into practice.

Do Not Be Distressed If Tests Seem Like a Mess You and your planning team should not feel distressed if the tests feel like a mess. The first one or two will seem disorganized, and people might not do their jobs exactly the way that they are supposed to. How could such first tests not be a mess? You and your team would probably have written a perfect plan (which would take a very long time) and have perfect personnel (all with 50 years experience with someone else’s disaster plan) — and this is not very likely. A messy test will help you find more flaws in the plan, and that is a good thing.

Finding the Leak in the Fire Engine Tire Try to think of your testing as finding a leak in a fire engine tire. You would not throw away the whole fire engine because of one tire. You would just patch up or replace the tire and carry on with the truck according to plan. If you, the planning team, and your enthusiastic test subjects should find flaws in the plan, fix the part with the problem. In your next test, maybe you will find a problem with another part of the plan. So you and your team will fix that, too.

© 1998 by CRC Press LLC

Test More, Not Less If you are finding problems with your disaster plan, test more, not less. There are some companies that are terrified of testing, and that is not a good thing. Maybe the people in these companies conclude that if their plan were not good enough to succeed in a test, it certainly would not be good enough in a real disaster. So they give up testing, and stick the plan binder on a shelf in the office of the CEO to gather dust. These companies work very hard to deny that there are disasters. But if some unavoidable event were to come along and disrupt their business, what could they do? They would have to walk away from their crippled company, and that plan binder would end up in a dumpster with the rest of their files. You could compare the process of writing and testing a plan to that of a baby learning to walk. Any infant falls down over and over (and collects bumps and scrapes) until finally the day comes when the wee one succeeds in mastering the basic technique. Toddlers do not give up. Maybe there is an instinct at work that equates mobility with survival. You look away for a day and suddenly the child has turned into a baseball player or gymnast. You and your team should not give up either. The first faltering steps toward running a functioning plan enactment might be shaky. Keep practicing! Your disaster plan will become better with every revision. Your planning and recovery teams will start to be able to find the weakness in a part of the plan and suggest worthwhile changes. Soon after that, the plan tests will start to focus less on finding flaws, and more on training personnel to develop their skill and confidence.

TEST CONTINUITY OF BUSINESS PROCESSES Backing Up Data Some of the test objectives that you should look at pertain to testing the continuity of the business process. For instance, consider backing up data. The backup procedure should be tested to make sure that the data that ends up at the other site is useful. We came upon a company that told us that they had been sending their backup tapes faithfully off to the backup facility on a regular basis. Yet something was happening along the way. (To this day they still do not know what it was.) Whatever was happening, it was demagnetizing many of their backup tapes. When they ran their first test and checked out the backup data, they discovered that the data was corrupted. Only one out of every five tapes was usable. It was fortunate that they found this out in advance and not in the middle of an emergency. When you test the backup of your data, take it to where it is supposed to go. Go through its entire route and test the data at the destination. Can the database be restored? If not, test the backup procedure along every point of the process until the problem can be found. This applies to restoring data at your assisting call center or your hot or warm site, after storage at your local, off-site storage location, and after storage at your distant, off-site storage facility.

HANDING OVER CALLS TO AN ALTERNATE SITE Test to make sure that your call center rerouting actually takes place. Ensure that the call volume does indeed get handed over and that the calls actually arrive at the other site. When company calls get switched to the alternate facility, monitor the call handling to ensure that the hardware, software, and personnel are working together, as they should. Note that this applies whether outsourcer personnel or your company personnel at a branch site, virtual facility, hot site, or cold/warm site receive the calls.

© 1998 by CRC Press LLC

If You Plan for the Alternate Site to Make Sales Test to make sure the alternate site can do sales. An RTMO facility will be doing work for your company all the time. Its personnel should have a fairly high skill level, so you can probably be reasonably comfortable that they can handle sales calls. You should ensure, however, that your ICH facility gets tested. The CSRs there would only handle calls for you if your call center were experiencing a disruption. Since they would not see your calls on a regular basis, they could not have experience with them. If the scripts that look good to you seem like total gibberish to the ICH call-taker that is going to have to take the calls, there could be a problem. Run some tests where the ICH CSRs take some sales calls (either real or simulated) to test how well they do with them. The ICH should be able to handle sales calls or other calls that you might send there during an emergency. So the objective of the test is to establish whether the script needs to be rewritten, or whether that type of call should be sent to the RTMO facility instead. If your company personnel are operating at a hot or warm site, ensure that all necessary equipment, application software, and data have been installed.

If You Plan for the Alternate Site to Receive Payments If you have planned to send calls to an alternate site and your company is doing sales by telephone, many sales orders might be paid by credit card. Check to make sure that the alternate facility can process credit card payments or other types of receivables. If calls were being handled by an ICH facility and your company takes payment on account by credit card, you will want to make sure that your ICH facility can do direct deposit (preferably electronic) so that you could receive payments from them. At the very least, you would need to ensure that they could provide you with spreadsheet information about payments, and forward the funds to your company by some sort of emergency banking arrangement, even during a wide-area disaster. What if your customer payment options are out of the ordinary? Do your customers sometimes pay by debit card? What about personal checks? Does your company receive payment on customer revolving credit accounts? Are customer accounts subject to discounts for prepayment or interest and late charges? In such a case, the ICH facility probably could not handle that. You would need the accounts receivable functions to be performed by someone who was connected to your system regularly and had access to your accounting department records and data backups. That would probably have to be either an accounting service bureau or your company personnel at a hot or warm site. This test objective could be assessed either with live sales calls or with simulated calls and token monetary amounts. We recommend that simulated testing should be done before you allow real customers to pay the assisting call center their money. As discussed below, your plan should include procedures for forwarding sales information to the fulfillment outsourcer that would ship the goods to the customer.

If You Plan for the Alternate Site to Provide Fulfillment If you have set up alternate site fulfillment, test to see if it can actually fulfill orders in the event of a disaster. For this test, you would need to provide sales information (customer address, product ordered, whether prepaid or COD, whether shipping prepaid or collect) for simulated sales and some of your products. However, as a test you should ship actual products from the alternate fulfillment facility warehouse, to show that the whole process works. If your team is planning on using an outsourcer, the planning team shipping representative, together with the team accounts receivable representative, should establish some

© 1998 by CRC Press LLC

guidelines for monitoring the service level of the fulfillment outsourcer. How fast a turnaround should there be, from receipt of the sales order at the ICH facility? Should the fulfillment outsourcer have discretion about which courier or freight line to use to ship or should the outsourcer use your usual company method? (This could make a difference if your company had obtained a preferential rate or volume discounts.) What mechanism should be arranged in case of damaged or returned goods? If your company is planning to use an alternate shipping facility with company personnel, your team should consult with the various shipping companies to ensure that their drivers would know where the location of the alternate site. Once you have done the first simulated tests, your team representatives should look at the company shipping database. Some basic questions should be considered, and action taken either by your shipping department or together with the outsourcer. What quantity of inventory should be stored at the alternate fulfillment warehouse? How often should that inventory be restocked? How long would that inventory last at current sales levels? Could the facility be restocked with inventory if the disaster were of long duration? If deciding to use an outsourcer, some decisions would need to be made with regard to the outsource contract. How long would the outsourcer continue working together with the ICH or company hot or warm facility — for a week, a month, or for a long-term disaster of unknown duration? Your team might choose to draw up a testing schedule, sending occasional real packages via the fulfillment outsourcer every month or two. Alternatively, your company shipping department might work with the outsourcer in somewhat the same way as your call center works with the RTMO facility. This would develop outsourcer personnel familiarity with your company products and shipping methods. If choosing to maintain a “warm” fulfillment site with company shipping personnel, your team would have to make some decisions about maintenance and testing. Would the fulfillment warm site be used only during monthly tests, or would it operate continuously as a satellite facility? (Who would staff it? Such a site should not be in the local area.) If operating only during testing periods, would inventory have to be rotated back to the main facility to prevent aging of the stock? What about providing security for the inventory? Would the warm site for fulfillment be colocated with the call center warm site? Note that inventory for fulfillment could not be toted along by evacuating personnel in a “to go” box. Providing for a fulfillment warm site would require provision of leased or purchased premises prestocked with inventory, manual or computer record-keeping systems, forms such as labels and bills of lading, and contact and tariff information concerning shipping companies. If your company ships across national borders, customs/excise information and forms would be required also. The fulfillment “to go” case should contain shipping company contact lists, information concerning the primary and alternate facility inventory, a cellular phone and charging equipment, and backup information (on paper or a laptop computer) relating to current sales orders awaiting shipping. Your tests (whether full-body workout or verbal or walkthrough simulations) should check to see that all these items are packed, and that evacuating personnel remembered to pick up the case on the way out.

If You Plan for the Alternate Site to Process Payables You and your planning team should check to see if the alternate site(s) could process payables. This business function is generally way down on the company lists of things to recover, but it should not be overlooked. Many companies would not survive if they could not pay their payroll or accounts payable. Do not leave this area out of your testing. If you are going to set up an alternate site, you need to do this test (with simulated inputs) to show that it could be done. If employing an outsource facility, you would need to ensure that all required information would be provided

© 1998 by CRC Press LLC

for the outsourcer personnel. However, if your company plan is to move personnel to a cold or warm site, you should test your accounting people’s ability to recreate their accounting system at another facility. Payroll. Payroll must usually be paid either every two weeks or semimonthly. Depending on when the payday falls in relation to the disaster, this might be a pressing concern, or there might be a week or so to prepare. If a service bureau is doing payroll, you should test getting backup payroll and banking information to it. That means ensuring that the accounting department recovery plan includes this information in the accounting “to go” case. (You might want to inquire about the continuity plan of the service bureau.) Could the hot site facility do payroll? You would probably want to ensure that privacy concerns were taken care of first. With your accounting recovery team representatives doing the payroll, privacy would be protected. However, if the disaster happened the day before payday, the recovery time objective for this function would become much shorter. That would make maintaining the accounting department “to go” case (with current payroll data backups) a semiweekly task. A question to discuss and test: Could accounting personnel work in a rented hotel room or boardroom if hot or warm site facilities were not available right away? During a widearea disaster, could they work in a tent or trailer? If local banks were crippled by the same disaster, could the company pay employees in cash brought from outside the local area? What security arrangements would be necessary? Payroll is an essential task. Companies that do not pay their employees tend to shrink rapidly. Accounts Payable. In a longer-term recovery (45 days or more) your company would have to ensure that there was a mechanism to handle accounts payable. Either a service bureau with the power to prepare payment or accounting department personnel would need to have access to the accounting database and banking information. Note that the accounting department backup data should be duplicated in secure local and distant storage facilities in the same way that other essential information is safeguarded. We discuss emergency vendor agreements, credit arrangements, blanket purchase orders, and the company continuity repair-and-replacement fund below in the section relating to the test objective, “Prove Logistical Plans.”

If You Plan for the Alternate Site to Handle Business Communications Most companies that do business with your company would understand if there were delays in pursuing contract negotiations, preparing advertising, and following up other business communications during a disaster. This is not the kind of thing that could be handed over to a service bureau. Company options would be to • Suspend business communications until all operations were restored • Follow up matters only of the highest urgency, especially those related to recovery • Send company personnel to work from home as telecommuters • Hire consultants on temporary contract • Work from the call center hot site, and transfer to a warm site at the 45-day mark

Possibly, management personnel could get their letters, faxing, and copying done by a photocopy shop or printing establishment. During a wide-area disaster, however, such services might be scarce or expensive.

© 1998 by CRC Press LLC

Your planning/testing team should check that department heads and other senior management members each maintain a “to go” case, including copies of their checklists, an upto-date contact database, data storage media, and duplicate current correspondence, perhaps in reduction copy. Optionally (depending on the urgency) these personnel could bring a cellular phone (and charging equipment), the smallest possible laptop computer, and basic fax machine or fax modem for the laptop.

If You Plan for the Alternate Site to Handle Customer Service and Public Relations Whether or not company personnel or consultants should practice doing these tasks as a critical function to be restored after disaster would depend entirely on the business needs of your organization. If these would be tasks that the company would have to perform within a short period of time after a disaster, then your planning/testing team should by all means set up testing procedures. To some degree, whether or not customer service, help desk functions, and public relations would be required would depend upon the size of your company and the amount of lost business or reputation that could occur without them. Customer service and help desk functions could be done by an outsourcer, by providing your company RTMO facility with the relevant training information, databases, and anything else required to give superior service to the customer. Alternatively, company CSRs could take these calls at a hot or warm site. Testing these functions could be done with simulated calls or live calls during a full-function evaluation. Public relations activities were discussed in Chapter 8, relative to the prewritten press releases that the incident manager should issue in case of interest from the media in the company situation. Basically, such releases to the press should be truthful and unemotional and should describe the event factually but with sensitivity. Whether such reports should be written by the public relations member of the recovery team or by a public relations consultant would depend on what your company normally does when not under the stress of a disaster. If public relations were a regular function of the company marketing department, then these activities could be done from a hot or warm site. If the disaster were so severe as to attract throngs of journalists, then the recovery team public relations member should be present in the incident manager’s EOC.

If You Plan for the Alternate Site to Handle Purchasing Purchasing during a disaster would only be done in the most basic sense of buying equipment necessary for the recovery of critical company functions. Therefore, most purchasing would be done by means of prearranged vendor service-level agreements and blanket purchase orders. (Please read the section below, “Prove Logistical Plans.”) As such it would probably be done by the accounting or purchasing recovery team member, from the hot or warm site, and would be essential if the company were ramping up a cold site. Such tasks could probably not be done by an outsourcer, unless a third-party crisis management company were on retainer. Testing this function would probably involve simulated use of the preprepared “to go” case, simulating telephone calls to vendors, and (simulated) retention of recovery-related purchase receipts.

Testing Function and Timeliness The two components of these tests are to ensure that (1) the personnel can do the job and (2) the job can be done in a reasonably short period of time. The time period is defined by the RTO for that particular function.

© 1998 by CRC Press LLC

TEST BACKUP PLANS You should test to make sure that, in the event of an emergency, the current backup from the call center (and any other department planned with regard to this function) could actually be received by the backup facility. If you were sending the data electronically (electronic vaulting), you would want to ensure that the system was configured for a “fire and forget” backup procedure. Company personnel must be able to start the backup procedure and quickly evacuate the building. If the plan called for delivery of backup data by courier, it would be important to ensure that company personnel could contact the courier at any time of the day or night. You might consider putting procedures in place whereby the person who is working at the facility at the time of the disaster takes the backup tape along to the hot site facility, to bring up the data on the backup tape or disk without delay. This could be tested by simulating such a procedure, or by live testing during the full-function evaluation. The last backup data would be used by the outsource call center, hot site, or virtual call center, only if the current back up could not be made (because of computer failure or evacuation) or if the current backup were damaged or corrupted. The “last backup” data could be obtained from the primary or outsource facility magnetic media storage vault or cabinet, or in duplicate from the local off-site storage facility. It would be a good idea (once call volume was handed over to the alternate facility) to use a last backup to ensure that the integrity of the current data is good. If corrupted data is found, the last backup data from the facility, or from off-site storage, should be used to compare the corrupted portions with the prior backup. Testing the “corrupted data” scenario should not be simulated. This test must be done in a hands-on fashion. In testing the backup function, you should make the test time sensitive.

TEST CALL FLOW PATTERNS You should ensure that your alternate call-handling facilities could handle the calls during the peak periods. Also, you and your planning/testing team should check to see whether or not it would be profitable to handle those calls during the low-call-volume periods. It would not be costeffective to activate an ICH or RTMO facility on a midnight shift if the call center might only receive a couple of calls. It would be better to let those few calls go unanswered than to recover calls with an immediate RTO. Your RTO might change depending on the time of night. During the daytime you might have a 15-minute RTO, because of the higher call volume and revenues at stake. At night you might lengthen the RTO to two or three hours. This decision could only be made if you were regularly testing your call flow patterns, and the current or last information was available in the call center “to go” case.

Peaks and Lows; Orders and Customer Service; Intuition and Graphs Your team could test to see whether or not a short RTO would be economically viable to have in the evening period. The call center manager or alternate could look at the number of sales orders or the number of customer service calls that were processed. Check your call flow patterns. You and your team might find that the evening calls are almost all customer service calls, with hardly any sales orders. This would help you to decide what kind of RTO is appropriate. You might decide that your customer service calls (if received in the middle of the night) could wait until the next day. Or, you might want to provide an IVR message (via the ICH facility) inviting customers to leave voice mail giving a phone number and preferred time for a callback. Then again, if customer service is considered

© 1998 by CRC Press LLC

paramount and could not wait until the next day, you and your team might want to set up call answering by CSRs at the RTMO facility. The best way to assess the call patterns would be with graphs; however, a good deal of intuition would be needed to interpret them. Depending on seasonal and time-of-day peaks, graphs might not always be dependable unless they were maintained for every shift, every day, over a long period of time. All of the above options could be tested with either live or simulated calls — depending, of course, on the call patterns your company.

IVR Programming Loops (“IVR Hell”) One thing that could go a little crazy during a disaster is the IVR unit. The caller could get caught in what we call “IVR hell” during a disaster. This could occur if the caller is transferred from one IVR, to another IVR, to another IVR with no way of zeroing out and speaking with a live agent. Check to make sure that your IVR programming avoids endless loops. Ensure that if the customer decided to get out of the IVR to speak with a CSR that it would be possible. Either that, or make the IVR message very short and simple. For example, “Thank you for calling [your company]. We are experiencing technical difficulties and we would be pleased to assist you tomorrow between the hours of 8 A.M. and 5 P.M.” This could be tested by a company, such as IQ Services, or by company or RTMO facility CSRs. Your IS recovery team member should provide a checklist for either the hot site team or the RTMO facility, describing methods and passwords for reprogramming the IVR with an updated greeting. During Ice Storm ‘98, we had occasion to access automated messages from companies whose personnel thought they would be able to return to their offices in only one day’s time. It is possible that, as the wide-area infrastructure disaster progressed, these companies could have updated their messages to something sounding less sad and futile. As part of your logical and talk-through testing, you should ensure that the IVR line (and especially the passive IVR crisis hotline) is included in the rerouting of lines to the alternate call-handling facility.

PROVE LOGISTICAL PLANS All logistical tests should be recorded on checklists, dated, and initialed. If a flaw in the checklist (or in the plan itself) is found, your planning team should correct it as soon as possible. You can find some necessary checklists in Appendix A.

Delivering Calls to Alternate Site(s) You have to ensure that the calls can get to the other site(s). We talked about route diversity before and after ensuring that there would be no single point of failure. You have to make sure that you sit down with the telephone carrier and plan how the calls would be diverted to one or more other call-handling facility. You and your recovery team telephony member should work with Telco to set up a number of preprogrammed call-routing plans. It would be best if the company would allow you to preselect at least six different preprogrammed call-routing plans. You should doublecheck to be sure that when you switch to your central office (CO), there is an alternate CO available. Then your company in conjunction with Telco should test to see if those calls could get through to the alternate site via each CO, and every one of the call-routing plans.

© 1998 by CRC Press LLC

You need to plan for switching to each outsourcer call center you use (such as ICH and RTMO), for your hot site (in case of needing it for 45 days), and your cold/warm site once you (might) start ramping it up. You use these tests to prove connectivity, and no simulated customer messages need be sent.

Delivering Backup Data to Alternate Site(s) Sometimes it can be hard getting the data to the actual site. This is one of the things that you have to test for. You should test to see that the data could be sent to the alternate site electronically, either by e-mail or by computer-to-computer faxing. You should determine whether or not the alternate site could do database sampling (for single transactions) — but this might require that the primary site computers were online, or that a third-party maintained your database online at a distant site. You should ensure that some method could be used to get your backup disks or tapes quickly from your primary site to the alternate site and from the alternate site to your recovery site. You should check to see that there are airports nearby each facility so that data backups could be carried quickly by air from one facility to another.

Receiving Replacements of Equipment In the vendor agreements that we have seen, vendors agree to deliver x pieces of replacement equipment within 24 hours and x more pieces of equipment within 7 days. At the time of negotiating such agreements, your recovery team purchasing department member should establish an exact time period within which your company would receive the equipment. The vendor firm should give some assurance that it has a continuity plan, too. We recommend that your company deal with large national firms, rather than local dealers. This reduces vulnerability to wide-area disaster, because, if necessary, the company could deliver equipment from a distant facility. The relationship with the local dealer should be for dayto-day convenience, but your team member should ensure that replacement or repair is supported by the national organization. To test this function, ask the vendor to tell you where the equipment provided for in the agreement is located at the date and time of the test, and how the vendor would get that equipment to your company. You should test delivery to both the primary site and the cold/warm site. You could run simulated tests about once every three months. The full-function tactical evaluation (once per year) should include actual shipment of a box from the vendor. The box need not contain the actual piece of equipment (unless your CFO agreed to a short-term rental), but should contain technical specifications for the equipment and a list of the cables, manuals, and software that would be bundled with it. If the vendor company did not have its own continuity plan, or was uncooperative with regard to testing, we recommend that you find another vendor supplying comparable equipment that (in case of disaster) would actually be delivered.

Receiving Electrical Generators You and your planning team should find a source of electrical generators that would not be crippled in case of a large or wide-area disaster in your region. Such a company should have a fleet of big trucks to haul immense generators from one facility to another. As with all emergency vendors, the service-level agreement should provide for delivery of a prearranged generator model within x hours of the incident manager’s request.

© 1998 by CRC Press LLC

Your test should show whether or not there are alternate routes that the vendor truck could take in case of closure of a road or closure of several roads. You also have to check to see whether or not they could distribute the maps that you have provided to their driver(s), and get the equipment out to you. It is probable that a generator company like this would agree to do a tabletop or scripted input exercise to ensure that things would proceed smoothly in case of a disaster. Your team should ensure that the generators would have fuel and know who would provide it and who would run the generators. Your team should also check whether this or another vendor would be able to supply numbers of smaller electrical generators for household use, as is described below. Testing could be by simulated input or actual delivery of a box empty except for manuals and specifications. As above, your test should ensure that fuel for the generators would be available, and could be delivered.

Emergency Supplies for Employee Residences Your senior management should make a policy decision about personnel needs during a wide-area disaster. We have heard of many instances of companies that have been left without personnel because the employees were more interested in taking care of their families than in reporting to work. Your company could overcome this by having an emergency plan set up for the employees or by educating employees that they can make life easier during a disaster if they are prepared (this is also cheaper to implement). This would include working with the company bank to have a provision for emergency banking so that employees could get money from their own bank or the company — either their regular pay or a personal needs allowance if they had lost their homes. The recovery team human relations member could work with employees to ensure that they had water, food, and possibly even emergency shelter. If senior management makes this corporate decision, your planning/testing team should ensure that the supplies would be readily available. Normally, if there is a wide-area disaster it is difficult to get supplies locally. Therefore, your team would need to find national vendors that would have their own continuity plans, and be cooperative about testing. The service-level agreements should include delivery times, types of food or equipment, alternate delivery arrangements in case of blocked roads, and quantities to be shipped. Testing could be by simulated input (verbal or by fax) or by the “empty box” test. What is the empty box test?

Delivering Personnel to Hot Site This is one of the toughest things you could try to test, because it would be difficult to simulate the conditions under which your team would have to deliver company personnel to a hot site. We have heard hot site stories about companies that found all the highways and roads to be either congested or impassable. Getting people to the hot site became an arduous trek for these companies that took days instead of the hours that they had planned. The other thing that you should test is when and how to send the people. If you are sending people by airplane, you would need to figure out how many people could to get into the aircraft. During a wide-area disaster, traveling is not as safe as it is normally, so your hot site recovery team should travel on several separate airplanes. Because of the call volume being handled by the ICH and RTMO, you would have the ability to cope with flight delays, or hearing that an airport was crippled and passengers were being transported by bus.

Testing Use of the Continuity Fund The continuity fund is an amount of cash and prearranged bank checking and credit accounts intended solely and only for recovery in the event of disaster. As such, it should be

© 1998 by CRC Press LLC

EXHIBIT 21-1. TESTING THE CONTINUITY FUND Financial Instrument

In “To Go” Case

Checklists/Contact Lists

Letters of credit Signing checks Company credit cards Company debit cards Cash and security

under the control of the recovery team accounting member(s) and designated management representatives on the recovery team. See Exhibit 21-1 and those exhibits provided on this subject in Appendix A. Check Exhibit 21-1 and ensure that the personnel who need to get to these items would be able to reach them in a crisis situation. Are they packed in a “to go” case? Would there be stored copies of this information near a hot site? How would members of the recovery team get to the “to go” case if the personnel on shift at the time of the disaster were injured? The other thing that you would need to check is how to keep the “to go” case secure during normal company operations. You should make sure that the credit and debit cards, lines of credit, and so on are available “to go” for an evacuation. But you would want to keep them out of the hands of theives during nonemergency times, who might run up charges on your company credit cards. We recommend that testing of this important (and financially valuable) matter should be done by tabletop, talk-through, and walk-through tests. The recovery team members who would be entrusted with the “to go” case should be bonded. If possible, your CFO should informally gather anecdotal information from other companies to find out how much cash might be required during a wide-area disaster. Stories that we have heard suggest that during a wide-area disaster “cash is king.” Prearranged vendor service-level agreements should include credit arrangements, retainer fees, and/or monthly contingency payments. All these items should be tested for inclusion on the “to go” case checklist, and inclusion in your company local and distant off-site document storage should be tested also. Your CFO might not be comfortable with building up a cash balance with vendors, in case the planned-against disaster never happens. An alternative might be putting contingency funds into money market funds with checking privileges, or posting bonds against the possibility of disaster. Testing these functions would require high-level involvement of the CFO of your company and the fund or bond-issuing corporation and should, therefore, be done only about every six months. “Why not just call the insurance agent?” We refer you back to the anecdotes about widearea disaster. “Cash is king.”

Testing Vendor Response Then test objectives shown in Exhibit 21-2 could be simulated by telephone or fax simulated input. Your testing team should ensure that all simulated inputs name the company, test type, and test name. Depending on the test, your team could make the response of the vendor time sensitive. You should do this test with all vendors, starting with vendors of emergency equipment and supplies first. Additional tables relating to this test objective (and actual disaster recovery) are in Appendix A.

© 1998 by CRC Press LLC

EXHIBIT 21-2. TESTING VENDOR RESPONSE Test Objective

Response (Good or Needs Upgrading)

Contact information is current Contact outside of office hours Contact if vendor is forewarned (e.g. Rising floodwaters) Contact if vendor is surprised (e.g. Earthquake) Checking credit arrangements

EXHIBIT 21-3. TESTING INSURANCE Test Objective

Result (Good or Needs Upgrading)

Broker contact information is current Policy provisions, riders, and chattels inventory is current Would broker front company emergency money? “smoking hole” test: would broker pay replacement value of premises How adaptable is insurer to unusual situations? (like testing!) Recovery team accounting representatives liaison Documentation backups are current and safe off-site, local and distant

Testing Insurance Arrangements You and your planning/testing recovery team accounting members should making make sure that the company would get its insurance settlement from the insurance company in a timely fashion. Many companies have difficulty with this issue. As a practical matter, the insurance company is extremely hesitant to give out the money. Insurance companies are in the business of making a profit off the money that you pay them. Your accounting department recovery team planning and testing members should talk through the company insurance policies with the insurance broker. Determine exactly what documentation would be required to settle an insurance claim, and ensure that the documents, photographs of equipment, floor plans, and copies of the policy are duplicated and stored in both local and distant off-site locations. Find out what the continuity plan of the insurance brokerage provides and what provisions are in place in case of a wide-area disaster. Your team should include your broker in simulated vendor testing. Try contacting your broker late at night and find out what could be done. How much does the broker know right off the top of his or her head? Exhibit 21-3 displays areas in which you should test your insurance company. Additional checklists are available in Appendix A.

© 1998 by CRC Press LLC

Wide-Area Disaster: Dealing with Distant Vendor Representatives In a wide-area disaster, local suppliers might be unable to help your company, so your planning and testing teams should ensure that alternate contact lists have been prepared for retailers outside the local area. Your team should make sure that suppliers are far enough away from the local area to be safe from a wide-area disaster. The applicable recovery team representatives (emergency response, not planning) should get familiar with the distant vendor representatives and form a relationship with them. Your team needs to make sure that the distant vendor representatives have the same kind of commitment toward delivering emergency equipment and supplies as your local vendor representative does. What kind of continuity plan does the distant vendor have, in case it might have a disaster at the same time as yours? Try to ensure that your company uses a vendor that is enthusiastic about testing. The tests should usually be done by telephone or fax simulation. The empty box test should be reserved for the full-function evaluation once per year.

Walk-Through and Time-Sensitive Tests One of the best ways to check out all of the above test objectives is to perform tabletop or walk-through tests. In these, your testing team would go through each item according to scripted inputs and people would say what they are going to do. For the continuity fund items, your testing team should ask the recovery team members to check if they could actually open the safe in a rushed fashion (if the “to go” case is contained in the safe). Make the tests time sensitive.

TEST RESTORATION PROCEDURES Many of the companies that help with restoration of the items described below are listed in the Disaster Recovery Yellow Pages, of Newton, MA (Telephone: 617-332-3496, Fax: 617-332-4358)

Restoration of Building and Systems Obviously, it would be difficult to simulate destruction of your primary site. The most practical way to test for restoration of building and systems is to ensure that the companies that you would do business with would be readily available to help. Your company must have vendor service-level agreements set up with contractors and building product suppliers in the event of a disaster. The agreements would depend on what your company had to restore. For instance, if the company required restoration of computer systems, it would need to enter into an agreement with a firm specializing in that work. If you and your team arrange a quick-ship system from another company, ensure that your original vendor would not have a problem with it. When negotiating service-level agreements with such companies, your team should ensure that the vendors would agree to participate in telephone or fax simulated testing, and the “empty box” test for the full-function evaluation once per year. A note about construction firms: During a wide-area disaster, local contractors would likely be in the same condition as your company. You need to find contractors outside the local area that are experienced in traveling to remote job sites.

Data Reconciliation What if the ICH facility creates its own database of the calls that have been received, and your hot site team starts creating its own database? On recovery of operations to the primary site

© 1998 by CRC Press LLC

or to the warm site, your recovery team IS technicians would have to test the data restoration and reconciliation. The training tests for this should require personnel actually to bring groups of divergent test data together and turn them into one coherent database again. Your testing team should write several tests on an increasing scale of difficulty, for instance, a tabletop (verbal) simulation, a walk-through, a hands-on physical procedure using simulated data, and a hands-on test using old backup data.

Restoration of LAN The best way to test restoration at your local area network (LAN) is to have your recovery team IS people run through a checklist of what they would do to restore the LAN. Your planning/testing team IS member should double-check what they were doing to ensure that the procedure would result in recovery of the LAN. A tabletop or walk-through test of this would serve two purposes. First, it would alert people in your IS department to all the things that could go wrong. Second, it would test to make sure that the procedures set out in the checklists would work. If the hardware or software were defective in any way, a vendor listed in the Disaster Recovery Yellow Pages might be able to help, if a service-level agreement had been prearranged.

Restoration of Call Volume Restoration of call volume is difficult to test. In the early stages of plan testing, the ICH, RTMO, and hot site recovery teams should do this as a tabletop exercise. This would help to find any flaws in the checklists so that they could be corrected. When running the full-function tactical evacuation, the test for this (with simulated calls) should be tacked on at the end. Your recovery team would probably feel that they had triumphed when handover to the hot site, the ICH facility, and the RTMO facility was complete. Your testing team should not let them stop the test there. The test should continue running until the call volume was handed back to your primary facility. The test team should make sure that the calls could be dealt with by your CSRs and that all systems had been restored to the normal configuration.

Restoration of Critical Business Functions Your planning team has identified critical business functions (other than those handled by the call center) such as payroll, fulfillment, and so on. Your team should look at them to see what the methods of restoration would be. Then the recovery team representatives who would perform those functions should go through a logical review and a tabletop test. Would the critical business functions work the way they would need to? To restore these function upon return to the primary site, or identification of the warm site as a long-term or permanent site, what checklist tasks would have to be added or corrected? Your planning team should note that one such function would be plan update and maintenance. New evacuation routes and other checklist items would have to be drafted to fit the postdisaster circumstances of the company. All vendors would have to be informed of the new company address. The company advertising and public relations templates might have to be updated. The emergency forces, government agencies, and even the tax department would all require new contact information for your company.

© 1998 by CRC Press LLC

EXHIBIT 21-4. TESTING CONNECTIVITY Test Objective

Result (Good or Needs Upgrading)

Cold site (Telco trunk line) ICH facility RTMO facility Hot site (assisting call center) Virtual call center (telecommuters) Your office coordinates telecommuter calls Assisting call center coordinates telecommuter calls Cellular telephone and radiotelephone Fax and e-mail Video- and audioconferencing Toll-free contact with assisting call center Toll-free contact with third-party crisis manager Toll-free contact with IVR situation hotline

PROVE NETWORK CONNECTIVITY To test your network connectivity, all your testing team would need to do would be to ensure that the alternate facilities had the capability of receiving telephone calls. Your team should periodically test these different sites according to the plan and prove them capable of receiving telephone calls. Refer again to the section on IQ Services testing options. That company can perform a number of different tests and drills. Your planning/testing team should investigate contracting with this company to do periodic testing. IQ Services could do heartbeat drills, which would send test traffic to all sites on a regular basis and ensure that the network connectivity has not been impaired in any way. With a cold site, it is a little more difficult because there is actually no one there to answer the phone call, nor in many cases is there even a telephone line or trunk connected. You would have to rely on your telephone company to test the lines in the locality of the cold site to make sure that they were actually functional. Fortunately, that is one of the functions that Telcos do as part of their day-to-day operations. To prove your network connectivity you need to test, or arrange testing of, the items listed in Exhibit 21-4. Some companies might have the capability to perform a high-tech assessment of damage. This kind of testing would probably require teamwork between IS, telephony, and security personnel, plus your Telco representative. Once your planning/testing team have developed checklists, they should run the recovery team members through the logical assessment, then do tabletop and walk-through tests. During the full function evaluation, these tests could be run hands-on with live data or simulated inputs.

© 1998 by CRC Press LLC

FAMILIARIZE PERSONNEL WITH EMERGENCY PROCEDURES Virtues of a Messy First Test As we have previously stated, the purpose of the test is to train your personnel as much as it is to test the plan itself. Do not be afraid if your first test is a bit messy. It is expected to be. Your second and third tests should be much better.

Breaking the Plan into Bite-Sized Chunks Do not try to test the entire plan all at once at the beginning. Even the military does not test their entire war plan all at once. The most it does is to deploy various divisions in an exercise. The entire military force is not tested all at the same time unless there is a war. You and your planning/testing team should break your plan down into chunks. Do not be ashamed if you are testing little chunks of the plan. Just make sure that your team tests regularly, and avoids testing the same parts repeatedly. You (as the planner) should review test results about once per quarter to ensure that no plan elements have been forgotten.

Five-Minute Walk-Through Test A five-minute walk-through test could be a very interesting exercise and not consume a lot of personnel time. You could schedule these five-minute walk-through tests both announced and unannounced. If you announced that you were going to do a five-minute walk-through test as a tornado drill at 4 P.M. it would give people a chance to look at their checklists. Perhaps the next time you might want to do it unannounced, and do the tornado test the way such an event might really occur, with hardly any warning. Important: Make sure that senior management approves all tests in advance. Ensure that immediate supervisors on the call center shift know that the test is taking place, perhaps about a half-hour in advance. As we have mentioned previously, Chernobyl (the most serious nuclear accident in the world to date) was caused by three workers doing a test that was not approved by management. If you do unapproved tests, a lot of things could go wrong. You never know, the CEO might be bringing foreign investors or journalists on a tour the day that you and your team schedule a test. Your test schedule could probably slip a day without harm to the company reputation. Disasters usually come along unannounced, but that is why you are writing the plan. 2.Teaching “Second Nature” (Learning to Live by Checklist) We emphasize use of checklists strongly in this book. Your company personnel need to live by their checklists. The first thing they need to do when something unexpected happens is to grab their checklists and refer to them. People just running with the checklists in hand, not reading them, would do no good at all. We can tell you from personal experience that checklists in disaster situations work well. If personnel are trained to grab their checklists and begin following the procedures, it can calm them down even in the worst of situations. Your testing team should promote training that makes using the checklist second nature to the employees. During an actual activation of the disaster plan, that training would pay off by making recovery procedures smooth and efficient.

Explaining Plan Updates There might be times when you and your team update the plan to the point it changes an established procedure. You do not want to make these things seem arbitrary. The people who are going to act out the plans need to understand why the plan updates have been done that way.

© 1998 by CRC Press LLC

Sometimes the plan updates might be worse than the original plan. This is the reason the logical assessment should not be reserved for just the opening days of plan testing. If you and your team go through all parts of the plan, you should be able to spot any logical flaws in your plan updates.

Make Testing Part of Normal Routine If you and your team run the five-minute walk-through tests regularly, company personnel should begin to become familiar with the testing ritual. Then, if an actual disaster should happen, they would be more likely to have the same attitude that they had during the testing. They would have the confidence to carry on throughout the emergency. We heard of a case where company personnel were so used to being tested in fire drills that when an actual fire alarm went off, they thought it was a drill. The evacuation went very smoothly, because no one was apprehensive. It was not until they all got out of the building that the people realized there was an actual fire.

Announcing Test by Name In a large-scale test where you and your team would be receiving inputs from CSRs, it would be important that the tests were each given a name. This would ensure that the inputs would be delivered in a standardized fashion. “This is a simulated input for CSR Test 001. A tornado has been spotted on the outskirts of town,” might be the way the testing team announced the beginning of a test. By giving the test a name, you would ensure that you separated fantasy from reality. What if somebody did see a tornado and ran into the call center saying “There’s a tornado, there is a tornado!” There would not be any doubt that it was a real tornado. Your call center manager would not think that it was part of the script. Make sure that all scripted inputs are announced by name, and that people understand that every time there is a simulated input that it is a simulated input.

After-Test Morale Boost (T-Shirts, Congratulatory E-Mail, Perspective) A lot of times after a test people are likely to be embarrassed or upset that things did not go as well as they could have. Try to avoid hammering people after a test with all the things that they did wrong. That might cause them to develop an active dislike for the testing process. Try to point out the positive aspects of the test to personnel. Congratulate those who “got out alive.” Part of the testing team’s work should be helping the employees deal with the whole planning and testing process. If the employees could greet the tests somewhat like a “yahoo” event, they would not stress out over news that their department was being simulated as having burned up or flooded. We do not advocate holding a party every time you run a fiveminute walk-through test. But sending an appreciative piece of e-mail, or giving department members a cupcake after a major test could help instill a positive, “can-do” attitude in all personnel.

RUNNING YOUR TEST Use Responses Laid Out in Checklists and Manuals Use prescripted inputs to make sure that the test is fair. As you saw in the previous chapters, we have developed some examples of scripted inputs for you. Make sure that you stick

© 1998 by CRC Press LLC

to your scripts because otherwise your testing team might go overboard and start exercising totally unrealistic scenarios. (Such as, “Godzilla Visits Peoria.”) This kind of extreme test scenario would just discourage people from participating. When people respond to the scripts, they should use responses that are in their checklists and manuals. The testing team should not let them simulate using equipment that your company does not have, or bring in vendors that your company does not have contracts with.

Note What People Really Do in (Simulated) Crisis If somebody needs to deviate from his or her checklist or manual to solve the test problem, your testing team should make sure that it gets noted. Then the planning/testing team would be able to make changes afterward so that the manuals and checklists would reflect what people would really have to do in a crisis.

Flag Deviations from the Plan in Your Test Notes What if the people doing a test did everything on a checklist, but not in the sequence that was given on the checklist? If the checklist needed to be amended, your team should try rewriting it according to the way the person performed the procedure. Or your team should script other alternatives, given the type of disaster. Your team should use judgment here. If the person doing the test were a trainee, getting the steps of the checklist wrong would indicate inexperience. In that case, the checklist should be tested with the trainee’s supervisor to see if a person with more experience would act differently.

Use Only Resources That Are Available at the Time of the Test When you and your team run a test, you should use only the resources that are available at the time of the test. Your team should not give personnel extra resources to work with. If you were running a test and some people had called in sick, your testing team should not pretend that those CSRs are there. What if Bob the IS technician was on vacation when your team ran a test, and Bob was responsible for a major part of the plan? No one should call Bob in from the beach. The testing team should just run the test without missing personnel to see what would happen.

Act as If the Event Were Real (for Training Value) We have seen numerous tests where people taking the test acted very lackadaisically and were giggling throughout. If you have noticed on television “real TV” shows, when the military and police do training, they take their simulations very seriously. You and your test team should ensure that company personnel take testing the same way; otherwise, they might not remember what they are supposed to be doing. The training value of a half-hearted semblance of a test would be much reduced.

Use Monitoring Staff from Departments Not Being Tested Although the planning/training team would be drawn from all departments, you should use test team members from departments not being tested to run the simulations. Try not to use people from the same department because they could tend to be either too easy on those undergoing the test, or too demanding. If the test should reveal plan flaws or lack of confidence on the part of employees, the planning/testing team members should be the ones working to fix the plan.

© 1998 by CRC Press LLC

Be Prepared to Terminate Test If Someone Suffers Panic (and Reschedule) Be prepared to terminate the test if someone suffers panic or something else goes awry, such as discovering a major plan flaw. Your testing team should have a code word or phrase that could be used to terminate the test if it gets too intense. For instance, in the military they use the phrase “no duff” if they need to get somebody out of a situation. Once they have reevaluated test material, your team might decide to rewrite either that part of the plan or the test inputs. Your company personnel need to learn that if they get to a part of the checklist and an essential step is missing, they are not at fault; the checklist is.

Identify Each Incident with Its Own Test Name We noted above that each test should have a special name. Each incident that is part of a test should have a formal name, as well. “For Exercise Physical Verification 98-01, This Is A Power Outage,” might be test input given to personnel in several different departments at once. When your team checks its test notes against the checklist later on, it should identify what went wrong (for future planning) with the exercise name and the flaw that was found. Your team should identify all checklists and notes with the exercise name and all parts of a large test with the exercise name and incident name. All test notes should also be dated and initialed by the test team.

Plan Test Objectives in Advance Your team should plan what the expected response from the individuals involved in the test would be, to ensure that people were doing what they were supposed to be doing. For instance, suppose that your test team gave CSRs a scripted input that smoke from a fire was detected. The expected response would be the CSRs opening up their fire checklists. The following items should be included in the objectives of the tests depending on what the disaster type might be. • Phases of Crisis, Crisis Management (see Chapter 23) • Call Volume Management (see Part 3) • Internal and External Crisis Communications (see Chapter 22)

The testing team should try to instill confidence in all personnel, in all tests.

Plan Test Assumptions in Advance What are the operating conditions for the test run? Simulated, Accelerated or Real Time? Would the test be run in real time, which means that every minute of the clock is a minute in the exercise? Or would it be accelerated, where every thirty minutes equals an hour to try to test a slow-motion disaster more quickly than it would actually occur. Or would all test actions be simulated, so that time would mean nothing at all? In such a test, once a person had completed one task, then the next input would be given. Time of Week and Day. What time of the day or week would the test event happen? You could take a midnight shift and say that it was the middle of the day for the purpose of the test. You do not have to stick with the actual time.

© 1998 by CRC Press LLC

Weather. Your team should set what the weather would be for the test in advance. Would your area be in the middle of a blizzard? Would there be severe storm warnings at the time? Call Volume. What would the call volume be at the time of the test? Would the call center be in the middle of its peak time, or would you schedule the test in the middle of the night when (theoretically) the call volume would not be a problem. Management On Site. The managers that would be in the facility during the test should be there to manage the test from the standpoint of being a test participant. (No one in the company should be allowed to “opt out” of testing.) Other managers should monitor how this is going. Any managers on the planning/testing team that take part in such testing should (as in other tests) not be from the same department as the manager participating in the test. As far as scheduling tests with regard to decisions of management on site, your testing team should determine who is going to be available during the test. Some managers travel a lot, and if they were away, how could they participate in the test? Disruption or Disaster. Your testing team should plan what kind of disaster or disruption the test would simulate, in advance. Your test team should keep it realistic. If the test would include multiple disruptions, they would have to be related to each other. For instance, a power failure normally would not happen during the middle of a fire unless the fire and the power outage were related to each other. Your team should also decide in advance whether the event scenario would be a minor disruption or a major disaster. For instance, consider the minor problem of one IVR unit (of two) malfunctioning. Then compare that to the major disaster of a fire in the Telco room knocking out communications, causing fire sprinklers to turn on and destroy all computer equipment, and then the UPS exploding because of the water. The UPS explosion could cause injuries and a fire that might destroy much of the building before the fire department could bring it under control. The predetermined level of the test scenario would determine how extensive the testing would be, and how long a time should be allocated to the testing. Note that if the test were expected to take longer than five minutes, provision should be made to provide alternate personnel to handle real-life call center and business functions during the test.

TEST EVALUATION All Participants Attend After-Test Assessment Meeting All participants of the test must attend an after-test assessment meeting. People’s immediate comments (before they forget) or any deviations from the checklists that they might have noticed should be recorded in the test notes.

Distant Participants Attend by Telephone or Videoconference The distant participants should attend by videoconference or telephone conference so that they can assess how it went, as well. For instance, suppose that a distant vendor had participated in the empty box test, and the vendor’s driver could not find the cold site. Discovering the reasons for that difficulty and correcting them would be important for correcting the plan. A list of questions to ask at such conferences is shown in Exhibit 21-5.

© 1998 by CRC Press LLC

EXHIBIT 21-5. POST-TEST CONFERENCE Questions to Assess

Yes/No

Remarks

Does the continuity plan function as a whole? Did the part of plan tested work? Did critical business functions suffer? Did logistics work as planned? Was backup data usable? Did handover of call volume take place as planned? Did cold/warm/hot site work as planned? Did virtual call center work as planned? Did vendor arrangements work as planned? Were personnel needs supplied as planned? Did company meet contractual, legal, or audit requirements? Did people react confidently or show confusion?

Record Proposed Plan Amendments After the post-test conference, your test team should make sure that they recorded the items that were important. A post-test report should be issued, and the team should amend the plan as needed. Company personnel should not have to make the same mistake over and over again. That would not do anyone any good. Your team should report the proposed plan amendments.

Update All Plan Books and Checklists Urgent changes to essential procedures, especially those preserving human life, should be made right away. However, your testing team should collect about five regular amendments before issuing plan changes. The testing team should ensure that all copies of the plan, even those in offices of personnel away sick or on vacation, be amended at once. At that time, all relevant checklists should be amended as well. All copies of a checklist (e.g., CSR Fire Checklist) should be amended at the same time.

© 1998 by CRC Press LLC

Part 5

Crisis Management

T

his section discusses the phases of a crisis. From the precipitating event, the giving of the first alarm, through escalation of the plan from “disruption” to “emergency,” it all must be covered by the continuity plan. We talk about recovering calls to an assisting call center, and the conclusion of the crisis, when your call center restores all systems and calls and returns to normal operations. We discuss the authority of the crisis leader or manager, and talk about the “internal” aspects of public relations during the disaster. We talk about external “command and control” by a remote command center. We examine the role the government is likely to play during wide-area disasters such as extreme weather or hazardous material incidents, or when responding to terrorism. This section includes an in-depth examination of personnel issues. Can you bus your call center staff to an alternative call center? How much would that cost? Who pays for meals? We discuss the effects of postdisaster trauma on the company employees. How long will it take for your employees to deal with the wreckage at home (in case of wide-area disaster) and return to their predisaster efficiency? How do you deal with “anniversary trauma”?

© 1998 by CRC Press LLC

22 Crisis Leader, Internal Command, and Control AUTHORITY TO DEAL WITH ALL ASPECTS OF THE CRISIS Like the captain of a ship, the crisis leader must be known to all participants in the plan and have authority to deal with the situation as it develops. The crisis leader and the delegation of authority has to be established clearly in advance. At any given time all call center personnel have to know who is going to be the designated crisis leader in the event of a disaster. Normally, it would be the call center manager until the recovery team leader (called the incident manager or crisis manager) reported into deal with the disaster. We have talked previously about the three different departments of the recovery team: the disaster site recovery team, hot site recovery team, and the ICH and RCMO Team. The crisis leader must have the authority to deal with the crisis as it occurs. The crisis leader must have the authority to allocate any resource of the company as is necessary to protect the company from further loss during a disaster. This means that the crisis leader must be reasonably high in the chain of command so that the trust of the company is behind the crisis leader in the event of a disaster. The crisis leader might in fact be the president (CEO), in which case the president would have to submit to many hours of designing the plan and even more hours of testing. The process of testing was described in Chapter 21.

CHAIN OF DELEGATED AUTHORITY There has to be a chain of delegated authority so that if the person who is the crisis leader is not available there is an alternate crisis leader. Likewise, there should be an alternate to take the place of every leader of the disaster recovery team and its parts. A chain of delegated authority must be devised, so that there is always someone who can fill in at given position until the usual person comes into fill that position. You need to be confident that there will always be someone able to be the crisis leader, even if the call center manager or other person is away sick or on vacation.

DISCUSSION OF CRISIS COMMUNICATIONS Different people involved in the disaster recovery plan would need different levels of crisis communications according to the role that they were performing at the time of the disaster. The crisis leader must be able to communicate with everybody at any given time. The recovery team leaders need to communicate with the people they are responsible for, the areas they are responsible for, and, of course, the crisis team leader. When you get down to their individual roles, these people need to be able to communicate with the people who are designated on their checklists. When you are testing the crisis communications, make sure that everyone has the communication capability that would be required in an actual disaster.

© 1998 by CRC Press LLC

CSR basic requirements in a crisis would be to get directions from their supervisors. Suppose that those personnel needed to evacuate the building or area because of a disaster. They would need a number that they could call to find out what they should do and where they should go to regroup. If their evacuation destination were also affected by the disaster, they would need to get directions to an alternate area to regroup.

INTERNAL COMMAND AND CONTROL PROCEDURES Public Relations Coordination Public relations and communication procedures are not something that can be ignored until a crisis develops, and then be done “by seat of pants” improvisation. The position of public relations coordinator should be assigned to one person, with delegates. All checklist books should contain a paragraph that tells all employees to whom inquiries should be referred. This is required so that in the event of a disaster, misinformation is not spread. There should be only one source of information from the company to the press for the duration of the disaster. That would be important to prevent CSRs or other employees getting their “15 minutes of fame” by speculating about management responses to the disaster when questioned by local media representatives. The name and telephone number of the public relations coordinator must be in all contact lists. There should be public relations representatives assigned to each of the three teams. Journalists might come out to all of the locations where the company disaster recovery teams might be. Your company needs to control any damage that the press could create while maintaining your positive company image. Let us emphasize that it is a necessity and virtue to tell the truth. The press would quickly know if you were not giving the whole truth about the situation. Your public relations coordinator and delegates must give the press either the truth or “no comment.” For example, suppose that a disaster just occurred. At that point the disaster recovery team would still be dealing with the situation. It would be too early to tell what the impact on company operations might be ultimately. Your public relations coordinator should make sure that there were no statements made to the press that could look untrue in the future. The tape recorder or videocamera never forgets what you say. Statements made “off the cuff” without proper preparation could surface later at the worst possible time. Avoid speculation about the problem. Both the public relations coordinator and the incident manager (crisis leader) should avoid talking to journalists informally. Until they know exactly what the situation is, and have been authorized to release information to the press, they should not make any comment. It is also important not to assign blame. Words that were said at the heat of the moment could come back in a lawsuit. If any spokesperson should say anything about the cause of the disaster (which we recommend not to do anyway), the person should just state when and how the disaster started, without saying who started it. Never cover up a situation. If a person made a mistake, your spokesperson should not give some other cause. What if your company was in a situation where things were severely fouled up? Your spokesperson should first avoid the topic, then give a “no comment” to it if pressed for an answer. If there is senior management approval, your public relations coordinator or incident manager should state the facts only, without making accusations. Your company should never assign blame to anyone or any part of the company. Before admitting that something might have gone wrong, the spokesperson should speak with the company lawyer first. It is necessity to show sensitivity when dealing with disasters. The spokesperson should be sensitive to the needs of company employees and their families. Company personnel

© 1998 by CRC Press LLC

should not get angry or hostile with the journalists no matter how rude or impatient they might get. In regard to scandals and accidents, company representatives should always tell the truth! (This could be called the basic rule of public relations.) Public trust and forgiveness (which are the basis of your company reputation) depend upon your company showing that it is a good corporate citizen. Your company representatives should strive to maintain the highest moral character for themselves and the company.

Authorized Release of Information Communication Procedures. Information should only be released to the media in a formal press release that the management and its lawyers have had an opportunity to discuss and approve. The public relations coordinator or incident manager (as the case might be) should get the signature of a senior manager and the written approval of legal counsel before releasing any statement to the press. This is to protect both the company and individual from liability. Press Releases, Press Conferences, Retain Releases in Public Relations File “Plan”. Press releases regarding emergencies should give the following information.

Remember never to assign blame. • Who is dealing with the situation? (Fire, police, ambulance, etc.) • Who was involved in the incident? (Without naming people or companies, for example, “The fire appears to have been set by an arsonist”) • What happened? (Do not go into great detail) • When did it happen? (Date, time of day) • Where did it happen? (Company facility at its street address) • Why did it happen? (Do not speculate; if the cause is unknown, say, “Authorities are investigating”) • How did it happen? (Give a description that is not dramatic or lurid; for example, “The fire spread quickly” or “Smoke alarms alerted the personnel, and they evacuated the building”)

If certain dangers to the company are perceived in advance, then the company public relations team should prepare event-specific press release templates covering specific situations. Alternatively, the public relations team could create a press release template with blanks provided for filling in the details of a situation, as they become available. Treat local and national media representatives equally. The last thing that your company would want to do is tick off the media. They would be around a long time after the disaster was forgotten, and if company personnel harass them, then they would send that back tenfold in bad publicity. Your company should never pick a fight with anyone who buys newsprint by the ton or can send video clips around the world.

Designated Meeting Place Designated meeting areas are very important in a variety of disasters. Company personnel could require different meeting places and backup meeting places, depending on the type of disaster. Each meeting place should be in all disaster recovery plan and checklist books. An infrastructure disaster recovery plan would require a meeting place outside of the area of the disaster. This is also true for wide-area disasters. A good meeting place might be the selected hot site facility, or a hotel conference room.

© 1998 by CRC Press LLC

When preparing against a disaster that might destroy the facility, your plan must include an evacuation plan. Your company would need to set up a meeting place that takes into account the severity of the disaster. For instance, if there were a fire in your building, the meeting place could be across the street from the building (out of the way of fire engines, etc.). A back up meeting place could be designated as the concourse of a nearby shopping mall. The designated meeting place should be in an assembly area where it would be easy to see who is there by doing a simple head count. In a wide-area disaster (if possible) your team should select an evacuation destination unaffected by the disaster. Depending on the type of disaster, your company should ensure the safety of your employees by meeting in a safe location such as a basement shelter (severe weather) or parking lot (earthquake). Depending on the disaster, you might have several meeting places. Your three recovery teams might have different designated gathering points to fulfill their disaster recovery plan objectives. The company personnel in general would require a place that would support a crowd, so that a head count could be made. A rescue and first aid station might be needed to deal with urgent problems. This would help emergency departments, too, if, when they arrived, there was already a first aid station set up. If your facility has one or more disabled workers, they might not necessarily make it to the meeting spot without help. Your checklists should set out a buddy system. It would make it easier to find out who is missing, if anyone. Partners for disabled workers would help to prevent those employees from being left behind during an evacuation.

© 1998 by CRC Press LLC

23 Basic Crisis Management

H

ow serious is a given event, and what level of response is required? Your call center business continuity plan should include levels of action, depending on whether a situation is a minor disruption or a major catastrophe.

EVENT Each event might be unique, but disasters can be generally categorized by risk type. Your incident manager’s assessment of the risk type and subsequent instructions would determine the recovery team’s enactment of a specific part of the recovery plan. In Chapter 12, we discussed the different types of disaster as classified by their speed of onset. They were as follows: • The sudden-onset disasters such as a fire, tornado, flash flood, or mud slide. These are the most dangerous disasters and could account for the largest loss of life. • The next one we described was the long-warning disaster. These are events like hurricanes or floods that people know are expected to crest next week. These are the most damaging disasters and could devastate a large area. • The slowest disaster type was the slow-motion disaster. These are events that people do not even recognize as disasters until suddenly they are in the middle of them. (We gave the examples of tar fumes filling a ventilation system, and carpet-cleaning fumes caused by an unexpected chemical reaction.) These events seem like annoyances, until their effect on personnel or call center operations becomes severe.

ALARM The alarm happens in a large variety of fashions. The crisis (or incident) manager should be a little paranoid, because it is important to pay attention to what is an alarm and what is not. In one of our test scenarios in a previous chapter, the action expected was that the call center manager should turn on a radio after the CSR told him about the heavy weather coming. If the manager did not turn on the radio, no one would hear the warning (alarm) about the tornado until it was almost too late. It would be helpful if all the people in the call center knew how to raise the alarm. Perhaps you, the reader, are now thinking that that statement is a little ridiculous. Consider the following story we heard at a conference. A secretary had been told by her boss that his important meeting was not to be disturbed, “no matter what” happened. A bomb threat was phoned into her. Not wanting to disobey her boss’s order, and without a clear idea of what else to do, the secretary simply picked up her purse and left. The bomb threat turned out to be real. One of the valuable things about checklists is that everybody, if the checklist is followed, will know how to raise the alarm. If the person who first encounters the disaster does not know how to proceed, the person has the checklist for reference.

© 1998 by CRC Press LLC

ASSESSMENT In Chapter 7, we discussed the types of disasters by risk types such as fire, tornado, flood, bombs, aircraft crashes, sabotage, chemical spills, computer virus, epidemic, earthquake, hurricane, cut cable, labor dispute, and so on. Assessment is a phase of a disaster that might cause company personnel (and especially the supervisors and managers) some doubt. For instance, consider the slow-motion disaster. It is easy to assess that there is a disaster on its way when a person can look out the window and see that a tornado is bearing down on the call center. However, it is a great deal more difficult to assess whether or not a disaster is developing when the “alarm” consists of someone’s casual remark that there is a leak in the roof or a bunch of people feeling sick after lunch. If in doubt, the crisis manager (whoever that might be) should “go to yellow alert” (to use an old Star Trek saying) and take precautions. That individual should alert the recovery team and the ICH or RTMO that the call center might be having, or about to have, a disaster. The plan should be set up to accommodate such a warning. In addition, the crisis manager should have clear guidelines for action that are derived from classifying the disaster by the course of action necessary. If the call center RTO was 30 minutes and the Crisis Manager could see that the disruption could put the call center out of service for more than 30 minutes then the crisis manager would have a disaster to tackle — no matter whether the event looked like a disaster or not. We also discussed classifying disasters by the course of action that they might force the incident manager and the recovery team to take: • A short-duration outage would only put the call center out of communication for a short time, and possibly no recovery of call volume elsewhere might be required. • A “short evacuation required” type of event would be a minor disaster, and only a limited amount of recovery might be required. Recovery actions would probably be limited to handover of call volume to the ICH or RTMO facility. • A “moderate damage to call center or infrastructure” type of disaster would be a major disaster. Full recovery of critical call center and business functions would be required, but the call center could be operational again within 45 days. • A “severe damage to call center or Infrastructure” type of disaster would be a severe disaster. Full recovery of critical call center and business functions would be required. The call center probably could not be operational again within 45 days. The company might have to set up a new call center at an alternate location.

When considering the topic of crisis management, it would be helpful to consider all the groups described above. If assessing the approach of a funnel cloud, your crisis manager would have guidelines for decision. The guidelines would show that the event would be a tornado, that tornadoes are a sudden-onset type of disaster, and it would only force personnel to evacuate for a short duration — provided it did not hit the building. The crisis manager would already have a pretty clear idea of what to do to recover from that crisis. Best of all, if it were a holiday weekend and the CEO, call center manager, or other executive were not available to perform the crisis manager function, junior personnel would have the same guidelines to help them with their decisions. These groups fit well into the phases of a crisis that we covered in Chapter 12.

NOTIFICATION Notification of all parties involved in managing a crisis would fortunately be covered in the relevant checklist. If you were the manager hauled out of bed at 3 A.M. to deal with a crisis, you would not have to remember all of the people that would need to be contacted.

© 1998 by CRC Press LLC

The planning team should try to keep the contact list as short as reasonable. The recovery team should not be required to call ten people during a crisis. It would get in the way of the crisis manager (and helpers) effectively doing the job of recovering call center functions. However, the plan should provide a contact list for calling enough people to perform an effective recovery. The notification contact list should provide for enough personnel to reroute calls, back up data, and take the call center “to go” case out if evacuation were required. As stated above, your company should provide in its plan the ability to put the recovery team and backup facilities on standby (yellow alert). It might seem embarrassing at times; however, the one time a precautionary alert saves the company, it would be worth it.

PHASES OF A CRISIS Exhibit 23-1 will show how to respond to the following stages of a crisis: 1. Declaration 2. Escalation of Plan a. Initial Escalation Goals i. Activate Response Teams ii. Gather Information b. Determine If Further Escalation Is Required c. Identify Areas of Impact 3. Recovery of Calls 4. Restoration of Systems 5. Restoration of Calls 6. Return to Normal Operations 7. Plan Evaluation and Updating

Exhibit 23-1 could be turned into a chart that escalates for other types of risks, as seen in Exhibit 23-2. Note that both exhibits deal with sudden-onset type disasters. If you and your planning team were to set up decision guidelines like the tables in Exhibits 23-1 and 23-2, the basic crisis management technique would make the crisis manager’s decision much easier. The use of such tables removes the doubt from decisions of junior personnel who might be called upon to substitute for senior managers. (What if a call center shift supervisor is the only one who did not eat the Salmonella Noodles at lunch?) It would also be a help to senior personnel in case of smoke or fumes that might impair their judgment.

© 1998 by CRC Press LLC

EXHIBIT 23-1. PHASES OF A CRISIS: DURATION AND DAMAGE

Short-Duration Outage

Short Evacuation Required

Moderate Damage to Call Center or Infrastructure

Severe Damage to Call Center or Infrastructure

Event

Fire

Fire

Fire

Fire

Alarm

Fire alarm

Fire alarm

Fire alarm

Fire alarm

Assessment

Checklist states begin evacuation

Checklist states begin evacuation

Checklist states begin evacuation

Checklist states begin evacuation

Notification

911 Notify backup call center of fire alarm

911 Notify backup call center of fire alarm

911 Notify backup call center of fire alarm

911 Notify backup call center of fire alarm

Declaration

Advise staff of evacuation

Advise staff of evacuation

Advise staff of evacuation

Advise staff of evacuation

Evacuation

Take “to go” case and evacuate

Take “to go” case and evacuate

Take “to go” case and evacuate

Take “to go” case and evacuate

Meet in designated area

Meet in designated area

Meet in designated area

Meet in designated area

Do a head count

Do a head count

Do a head count

Do a head count

Assessment

Determine if fire is real

Determine if fire is real

Determine if fire is real

Determine if fire is real

Escalation

No

Yes

Yes

Yes

Recovery of calls

No recovery plan activation required

Call hand-over to ICH or RTMO facilities

Call handover to ICH or RTMO facilities

Call handover to ICH or RTMO facilities

Assessment

Determine size of fire

Determine size of fire

Determine size of fire

Escalation

Small fire, no damage to call center

If moderate to large fire,

If moderate to large fire,

Recovery Plan

Return to call center

Activate hot site

Activate hot site

Activate all three recovery teams

Activate all three recovery teams

Notify ICH or RTMO facilities of situation

Notify ICH or RTMO facilities of situation

Coordinate with local fire authorities

Coordinate with local fire authorities

© 1998 by CRC Press LLC

EXHIBIT 23-1. PHASES OF A CRISIS: DURATION AND DAMAGE (continued) Moderate Damage to Call Center or Infrastructure

Severe Damage to Call Center or Infrastructure

Assessment

Assess damage after fire is extinguished

Assess damage after fire is extinguished

Escalation

Damage is moderate; will be able to return in 45 days or less

Large amount of damage; will not be able to return for more than 45 days

Recovery Plan

Continue operations in hot site

Activate cold site or alternate location

Complete repairs as necessary

Activate drop-ship provisions

Short-Duration Outage

Restoration of system

Short Evacuation Required

Begin clean up

Set up new facility Restoration of calls

Return calls

Return calls

Return calls to new site

Return to normal operations

Return to normal operations

Return to normal Operations

Return to normal operations

Return to normal Operations

Plan evaluation and update

Evaluate evacuation and update plan if necessary

Evaluate evacuation and update plan if necessary

Evaluate fire recovery and update plan if necessary

Evaluate fire recovery and update plan if necessary

© 1998 by CRC Press LLC

EXHIBIT 23-2. PHASES OF A CRISIS: SUDDEN ONSET Event

Fire

Tornado

1. Alarm

a. Fire alarm b. Smoke c. You hear “FIRE, FIRE, FIRE”

a. Radio reports sighting nearby b. Tornado sirens

2. Assessment

Checklist states call 911 and recovery team and begin evacuation

Checklist states call recovery team and begin evacuation

3. Notification

a. Call 911 b. Notify recovery team of fire alarm

Notify recovery team of tornado

4. Declaration

Advise staff of evacuation

Advise staff of evacuation

5. Evacuation

Take “to go” case and evacuate a. Meet in designated area b. Do a head count

Take “to go” case and evacuate a. Meet in designated shelter b. Do a head count

6. Assessment

Determine if fire is real

Determine if the tornado is going to affect the call center

7. Escalation

If Yes, go to next/ If No go to 18

If Yes, go to next/ If No, go to 18

8. Recovery of calls

Call handover to ICH or RTMO facilities

Call handover to ICH or RTMO facilities

9. Assessment

Determine size of fire

Determine extent of damage

10. Escalation

a. If moderate to large fire, go to next/ b. If small fire with little or no damage to call center, go to 16a or 18

a. If power failure, go to power failure escalation chart b. If tornado hits, call center go next c. If tornado does little damage, go to 16

11. Recovery plan

Activate hot site

Activate hot site

Activate all three recovery teams

Activate all three recovery teams

Notify ICH or RTMO facilities of situation

Notify ICH or RTMO facilities of situation

Coordinate with local fire authorities

Coordinate with local fire authorities

12. Assessment

Assess damage after fire is extinguished

Assess damage after fire is extinguished

13. Escalation

a. Damage is moderate, will be able to return in 45 days or less; go to 15a b. Large amount of damage, will not be able to return for more than 45 days; go to 15 b, c, and d

a. Damage is moderate, will be able to return in 45 days or lessl go to 15a b. Large amount of damage, will not be able to return for more than 45 days; go to 15 b, c, and d

© 1998 by CRC Press LLC

EXHIBIT 23-2. PHASES OF A CRISIS: SUDDEN ONSET (continued) Event

Fire

Tornado

14. Recover Plan

a. Continue operations in hot site or b. Activate cold site or alternate location c. Activate drop-ship provisions d. Set up new facility

a. b. c. d.

15. Restoration of system

a. Begin cleanup b. Complete repairs as necessary

a. Begin cleanup b. Complete repairs as necessary

16. Restoration of calls

Return calls

Return calls

17. Return to normal

Return to normal operations

Return to normal operations

18. Plan evaluation and update

Evaluate evacuation and update

Evaluate evacuation and update

© 1998 by CRC Press LLC

Continue operations in hot site or Activate cold site or alternate location Activate drop-ship provisions Set up new facility

24 Personnel Issues

W

e have discovered, during our various conferences and interactions with companies that have experienced wide-area disasters, that personnel issues can cause serious problems. Many disaster recovery books would not help a company to plan for personnel issues beyond the scope of the basic disaster planning. That is why we have dedicated this portion of the book to this topic.

LOCATION Wide-Area Disaster Makes Commuting Arduous Without the personnel to run the call center, the call center would not work, obviously. So what could be done to assist company personnel? We have mentioned that if your company did not have assistance for its personnel outside of work, during a disaster that affects their homes as well, then they might not come into work. Families come before work for most people, and we support that. Most of the employees would be home with their families after a wide-area disaster. If they did not have home still standing, then they would likely be camping out in an emergency shelter. Their focus would be on the basics of living rather than trying to help your company. According to Maslow’s “Hierarchy of Needs,” food and shelter are two of the primary needs of all people. Your company would need to ensure that the employees’ families had these needs met, to have an effective disaster recovery plan. As former members of the Armed Forces, we have seen that during disaster this reassessment of priorities does happen, even to the most dedicated people. The basic idea of a disaster recovery plan is to get company personnel to a recovery facility in an unaffected area. Wide-area disasters make travel arduous at best. It would be difficult for your employees even to get to their families. If the plan called for an evacuation because of a wide-area disaster, then the company would need to assist employees’ families as well. Keeping these facts in mind, there are some important points to keep in mind when making a decision on moving to hot site and/or cold site facilities.

ALTERNATIVE CALL CENTER Cost Your plan to open an alternative facility should include the cost of meals and accommodation for company personnel and accommodations for their family members. If the company could not help the employees’ family members, then you could bet that the employees would not help the company. Single people are rather handy to have on staff, but in some places it is illegal to keep track of marital status. Besides, they would not always remain single. The cost of being away from home is generally higher than being at home. However, if this were a disaster situation where homes were destroyed, perhaps you and your planning team could work out a plan with the employees. What if the company were to pay for a percentage of the accommodation and food costs for the employees and their families?

© 1998 by CRC Press LLC

We heard of a company that had a hurricane bearing down on its call center. It decided to relocate to its hot site. It took much longer to get there than anyone ever expected. When people got there, the hurricane had passed through their area and had not affected their call center. One of the problems that this company had was that one of the CSRs staying at the hotel discovered the mini bar. This person called the spouse at home, and said that the refrigerator was just chock-full of goodies. The person’s “other half” said, “Well, bring some snacks home for the kids.” So the employee loaded up a duffel bag and the next day the minibar was full again. Naturally, the person did it again; it seemed like a magic pantry. This went on for every day that that employee was working at the hot site. By the end of the stay, the bar bill for the company employees was around $5,000. The disaster plan should make clear that the company covers a daily allowance for personal items. The employees should be clear on what the company would pay for and what it would not pay for. The company should publish guidelines on what the hot site recovery team would be allowed as a maximum. Would long-distance telephone calls be paid for, and how many minutes? If your company allowed or encouraged employees to bring their families to the hot site location, would child care be provided? Would there be a facility to board pets? These are all kinds of issues that your plan must address when bringing company personnel to a hot site. These are some of the reasons why we recommend ICH and RTMO facilities being used as long as possible before you activate the outsourcer hot site or company cold site. How long is the disruption forecasted to last? If it were going to be a significant disruption, you would have to make allowances for time off and payroll. Often during a wide-area disaster, it is difficult for your employees to get money from their accounts. The previous pay system, used at the primary (disaster) site, might not be appropriate at the hot site. Automatic deposit and checks as a pay system do not work all that well during a disaster.

Capacity How many shifts would be scheduled at the hot site? Would you ramp up to full capabilities or run shifts that would strip the call center down to its essential components? We recommend that you strip the call center down by leaving as many calls as possible at the ICH and RTMO facilities and retaining a minimum staff at the hot site facility. This would reduce costs and complications at the hot site. If you were running the hot site with a minimal staff, then you would need to deal with stress. Not only would there be stress from the disaster, and from people living away from home, but there would also be stress because of the increase in call volume. Stress counseling would be an aspect that would have to be dealt with at the hot site, and when people return to the primary (or alternate) site. How much productivity could the company expect of people in a disaster situation before they began to burn out? (Trauma counseling is one of the services offered in the Disaster Recovery Yellow Pages.)

Coordination of Transportation Cost and means of transportation to the hot site facility would be determined by the number of shifts that would be run, the number of people (families included), and how far away the hot site facility is from the company local area. You and your planning team should as sure that the company does not send the recovery team on the same transport. The team should split up into smaller groups, so that if anything happened to part of the team, then the disaster recovery plan could still continue. For instance, there could be few things so frustrating as a bus that breaks down and strands the disaster recovery team that was manage the hot site. Meanwhile, what if the rest of the employees arrived, but did not know what to do?

© 1998 by CRC Press LLC

POSTDISASTER TRAUMA Willingness to Return to Work Our discussions with people whose companies had experienced a major disaster showed that in such situations companies might suffer as much as a 60% turnover rate in personnel. For example, after the Northridge earthquake in California many employees had little willingness to return to work. After a disaster, people are traumatized, in shock, and have trouble with basic daily activities. This would, of course, be dependent on how the disaster affected them directly. Your company should investigate crisis counseling assistance as a possible postincident expense. Crisis counseling deals with post-traumatic stress syndrome, which could affect a percentage of company personnel after a disaster, especially if there were injuries or fatalities. When these people returned to work, the company could not get full productivity from them. In a disaster that we witnessed, there were people wandering around for several days afterward who were in no shape to fulfill their basic needs. It takes quite some time for people to regain their composure. Even weeks after a disaster, people can suddenly lose all composure and need crisis counseling, or a quiet moment alone. People who had had their homes ripped from the ground would need time to put things back together. They would want to spend more time working at cleaning up their homes than working for your company. (And if you and the rest of the recovery team went through the same disaster, so would you.) So, to get personnel to come to work, your company would need incentives to help employees get back to normal. One tactic to get people back to work would be to provide reduced or flexible work hours. This would allow them more time to work on their homes and be with their families. After a disaster, people would likely be financially devastated with immediate repair bills. A salary increase during the period that it takes the company to recover would assist the employees and increase their loyalty to your company. Perhaps the company could purchase insurance to cover this aspect of the disaster recovery plan. Salary continuity for employees who could not make it to the hot site could be a big issue. Take the case of a company that suffered a fire that burned down its building. While it was rebuilding, the salaries for its employees were paid by a settlement from its insurance policy. As a result, nearly all of the employees returned to the company. This reduced the training costs of this company for replacement employees. Your senior management would need to weigh what it cost to train new employees to the level of the current employees, or how much it would cost simply to pay them as if they were at work. If the business insurance would cover it, then it would be an excellent way to ensure the return of the employees. Your recovery team would need to give situation updates to all employees, particularly to those who would be away from their homes. People would tend to hang on longer if they knew when the company could return to its normal operation and when they could return to their homes. Regular updates would also help to reduce stress. During the ice storm that devastated much of the northeast United States, Quebec, and a portion of Ontario in January 1998, some teams involved in the disaster recovery in the affected areas of Canada handed out news flyers door to door to people affected by the storm. The sheets were updated every two to three days, and this gave people an idea of what was happening in their communities. Where these flyers were available, they also gave homeowners a list of what services, food items, and equipment were needed and where these items could be found. As a result of these flyers, fewer calls were made to disaster recovery HQ and more was accomplished in a shorter period of time than was originally expected. The company right provide employees help with legal and insurance issues. For example, help obtaining repairs for their homes, government grants for low-income earners, and

© 1998 by CRC Press LLC

loans to higher-salaried personnel to replace personal belongings and rebuild their dwellings. Other things the company could help with might be assistance in disaster aid by helping employees with their insurance claims and getting them needed assistance. If the company had a group insurance plan, then it would be able to make sure that all employees were taken care of as well as the company.

Family Care Issues Family care is another issue that should be thoroughly addressed in the disaster recovery plan, so that the company would be able to ensure that most people would return to work. There is a list of family care issues that the plan should provide. • Waiting for the building inspector after wide-area disaster • Lineups for water and food, shelter, and cots • Electrical power at home • Care packages • Food, water • Medicines, vitamins • Personal care items • Child and elder care • Providing an on-site childcare center, professional or volunteer (as dependents would be without school to go to, kids might be at the work site; retired staff might volunteer to furnish daycare; older kids could help with younger children). • Child care supplies: toys, books, diapers, blankets, baby food • Elder care supplies: playing cards, crafts, blankets, prescription medicines • Provisions for adult and teen family members • Disaster unemployment assistance • Temporary housing assistance • Provisions for pets • Inclusion of family in continuity plan training • Anniversary trauma (crisis counseling) • Depression (crisis counseling) • Media retrospectives (crisis counseling) • Memorial services and wreaths (crisis counseling) • Monitoring of CSR efficiency (anniversary week, crisis counseling) • “Diana” effect — pilgrimage crowds (crisis counseling) • Postdisaster stress (crisis counseling) • Alarms and bangs (crisis counseling) • Phobias will fade (provide comfort) (crisis counseling) • Irritability (crisis counseling) • Bingeing of various kinds (crisis counseling) • Mutual caring and tolerance by all personnel (crisis counseling)

© 1998 by CRC Press LLC

REPLACEMENT WORKERS Replacement workers might be needed to carry the company over the “trauma” period after a major or wide-area disaster. The disaster recovery plan should provide guidance concerning whether or not the company would seek out-of-area hires. The senior management would have to decide (in advance) whether the company would offer such personnel a temporary housing assistance allowance. The speed of training would determine whether the call center at the warm or the primary site would maintain its former quality levels. In considering the time to train such a replacement worker up to expert skill, the person’s previous experience would count for a lot. As with all new hires, replacement workers would require training for continuity plan familiarization.

© 1998 by CRC Press LLC

Part 6

Conclusion

T

© 1998 by CRC Press LLC

his section puts the book into the perspective of a successful recovery.

25 Continuity Planning for Your Call Center THE WELL-EQUIPPED FIRE ENGINE Brief Summary We have compared the well-written business continuity plan for a call center (and the rest of its company) to a well-equipped fire engine. Both require the combination of technology, personnel, planning, and especially training. We have examined the potentially damaging illusions of recoverability that could fool company management into believing that its (preliminary) disaster plan would work. We have discussed putting together the business impact analysis and risk assessment. We have outlined the processes involved in planning, team selection, finding alternate sites, and plan maintenance. We have described the heart of call center continuity planning, the call volume management technique, for both emergency and real-time overflow. We have talked about call center outsourcing, hot; warm, and cold sites; checklists; testing the plan; and crisis management including personnel issues.

Discussion: Continuity Might Be of Essentials Only It is important to note that the continuity plan should provide for essentials only. The call center that provides so much of the company revenue is an essential. Accounts receivable and payable functions are essentials. Fulfillment is an essential. Look back at the critical business functions that are discussed in this book. As much as we might try, disaster recovery is not business as usual. Regrettable as it might be, nonessentials cannot be protected in a major disaster. The stamina of your recovery team and your continuity fund resources are finite. Those plans that someone might have had to attend a sales seminar (or other noncritical project) would just have to be put aside if the company were battling for its life. We heard anecdotes from Ice Storm ‘98, about some banks and other companies lighting their offices with candles. Because of the fragility of the power grid in Quebec, some companies there had enough electricity (from electrical generators) to power their computers. But they did not have enough spare power to light their offices. Everything except critical business functions pretty much came to a halt during the disaster. Note that those anecdotes were about companies that had done some continuity planning and preparation. Companies that had no plan just stopped operating until their power could be restored. Many of their employees were living in shelters, because their houses were without heat. We have since tried to contact some of those companies without success — no one was answering their telephones.

© 1998 by CRC Press LLC

Just Reading This Book Is Not Enough! Your company should not put this book on inventory, gathering dust on a shelf. That would not help you or your company in the event of a real-life crisis. Nor would it prove “due diligence” in case of a liability suit by a customer or client, or of an investigation by a regulatory agency. Your company, which is dependent on continued call center operations, would benefit only if you took this book and used it to write and test a workable continuity plan. You must continue testing regularly. You must forge those contingency relationships with vendors, outsourcer call centers, real estate firms and so on before the inevitable crisis. (And it is inevitable.) Your company should assign personnel to write the plan, prepare the arrangements that should be put into place, and test the plan. Furthermore, the continuity-planning attitude should become second nature in all departments. Is a software company trying to sell a product in “beta” version? (Using that kind of program could precipitate a disaster.) Does a vendor seem cool to being included in continuity plan testing? (How dependable would that supplier be during a wide-area disaster?) Are new policies or facilities in the planning stages? What plan amendments would be required if a disaster were to occur in the middle of ramping up a new facility? All of these questions should be considered in the light of continuity planning.

SUCCESSFUL RECOVERY What is the goal of call center continuity planning? You and your team should aim to keep the call center operating at such a level that a customer telephoning from a distant region could not tell that a crisis was even happening!

Testing Continuity We heard of a company that had two voice centers and one data center. The centers backed each other up and were operating on a 24/7 schedule. The operation was literally never closed. In order to test their center, components of their equipment were taken offline. Live recovery testing, by purposefully disabling equipment such as an IVR unit, was the only way they could practice their disaster plan. At the time we heard of them, they had never suffered a large disaster.

Planning for Continuity in Banking During our research for this book, we happened upon a number of stories about how banks plan for continuity. Some of them call this planning “business continuity,” while others call it planning for “business resumption.” We heard of a bank that has an aggressive growth strategy (in today’s financial marketplace, many banks do) that counts the cost of planning and providing redundancy of infrastructure as a normal cost of doing business. This bank has a standard continuity plan for all its branches, and maintains what it calls “support teams” to help its managers plan and test for continuity. We heard of a bank that maintains an emergency recovery call center where the CSRs are allowed to bring in their children and pets. This facility has electrical generators, food and water, and redundancy of equipment and telephony. The most important part of this facility is that it has been fortified to withstand severe weather, and it has no glass. Because of this, even in major disasters the personnel for this facility tend to show up for work rather than evacuating the area.

© 1998 by CRC Press LLC

We heard of a bank that keeps a firefighter paramedic on staff as a liaison officer. This person has the experience to talk with emergency departments on a level that other managers might not. This person is also able to give a higher quality of first aid than is normally available. Not every bank has an “excellent” disaster plan. Some banks are still developing their plans; some have plans that have not been tested. Although some find that they need to make an assessment of volunteers’ resolve (would they actually show up), many bankers had touching stories to tell about the dedication of their tellers and other personnel. We have heard of banks staying open (before predicted severe weather or flooding) until the last moment to provide for customers’ needs for emergency cash. As noted above, we have heard of banks operating by candlelight. A number of banks operated emergency trailers or mobile homes that carry their own power source and heat, so that in a disaster area they could put currency into the hands of their customers. There is a greater-than-normal demand for cash when there is no electricity for credit card transactions. That was one thing that hurricane, earthquake, and ice storm anecdotes had in common. Tellers at bank branches have had to remove cash from the bank machines (which required electricity) to dispense it by hand. The banks in disaster areas had a reduced demand for loans and other banking services, and a rise in customer service calls asking where branches that were open were located. A long-term disaster exacts a toll upon personnel, and some banks rotated their staff to keep them fresh. We heard varying estimates of what it might cost a bank to be out of business for 24 hours. For general service banking or credit transactions, the cost might be $8 million or more. For international currency exchange, the numbers might climb into the billions of dollars, not counting the penalties for delay in settlements. Clearly, there is a reason banks have been among the leaders in continuity planning.

Planning for Continuity for Utilities Twenty First Century Communications, Inc., is a company that provides high-volume call answer service to utility companies. As described previously (in Chapters 4 and 15) they assisted Central Maine Power (CMP) in responding to the January ice storm that caused so much damage in the Northeast. The CMP general office communications center is open 24 hours, 365 days per year. Ordinarily it utilizes 46 trunk lines. If call volume gets ahead of the ability of CSRs to answer, during ordinary operation the company can route calls to a 48-port IVR unit. For ordinary storm situations, they have the capability to route calls to their other call centers at Portland and Lewiston (Maine). This increases their number of trunk lines to 117. Their system has the capability of distinguishing what locality calls are coming from by coding by the first three numbers of the calling telephone number, or phone exchange number. Twenty First Century Communications (CMP calls them TFCC) can assist by taking high-volume outage-related calls on a guaranteed minimum of 2,000 trunk lines. TFCC can expand that number to 4,000 trunks depending on its other call volume. Switching from the CMP multiple call center network to TFCC is done with not one but two password-protected mechanisms. The long-distance telecommunications provider is AT&T. CMP has a team of managers and supervisors that regularly reviews weather forecasts. If severe weather seems likely, the team can start precautionary preparations in advance. It takes about 15 minutes to start routing outage calls to TFCC. About 3% of calls during the first ice storm (January 7 to 18, 1998) related to life-threatening emergencies, and were rerouted back to CMP. About 35 to 40% of callers during that period used the TFCC outage application without dialing zero to speak with a customer

© 1998 by CRC Press LLC

representative. As the outage wore on, about 25% of callers felt they needed to speak with a person, and some customers called long distance to the CMP business office to do so. The utility company put a recording on its business calls line stating that business calls could not be handled during the storm restoration effort. During the outage, TFCC received 596,328 calls. The calls received by both TFCC and CMP during the outage totaled 645,760. Amendments to Plan. During the first storm (of two) of this disaster of infrastructure, the work orders for electrical service restoration to CMP subscribers had to be amended. For the first time ever, work orders had to include a new code, invented for the occasion, to give repair priority to customers who were “Trapped” by downed lines and poles blocking their doors and driveways. Changing these procedures bore fruit within the same month, when a second ice storm arrived. Because plans and procedures were updated during the first storm and recovery of January 7 to January 18, 1998, everything was ready for the second storm of January 24 and recovery through to January 27, 1998. The CMP CSRs began providing shelter locations to subscribers who had no power or heat and were in distress. Supervisors at CMP created a spreadsheet to track employee in and out times, and meal entitlements. This was necessary for weekly payroll review and approval, and to track union regulations and shifts. Business Communications Come after Customer Communications. New orders, cancellation of service, estimated bills, and commercial clients (except for restoration of power) were all put on hold during restoration efforts. Mostly everyone understood. Problems. Some customers found ways to circumvent the HVCA system. Some claimed a life-threatening emergency when they only needed to ask about power restoration time. Some called the General (business) Office and were then transferred to the CMP call center. Some callers called the CMP call center numbers intended for emergency contact with fire departments, police and municipal authorities. After the ice storm, Central Maine Power’s invoices were mailed with an insert noting that customer bills for billing periods falling during the ice storm were estimated. Some customers had to be coded (in the billing system) to receive a minimum bill rather than an estimate, because of being seasonal accounts, such as summer cottages or resorts. After the storm there was a backlog of service work orders, especially for people moving in or out of residences during the outage period. CMP had to make billing adjustments to accommodate these customers. Cost of outsourcer vs. in-house expansion. CMP noted in its postincident report that it pays a monthly retainer to TFCC, and uses the outsourcer about six times per year. The outage for Ice Storm ‘98 totaled 1,322,824 billable call minutes. The approximate cost, including CMP in-house CSRs, for the disaster period was about $1 million. CMP management estimated trying to answer all calls without the assistance of TFCC would have cost $1.9 million, with annual maintenance costs of $61 million. It estimated that purchasing the technology would cost about $3.4 million, and expanding its building(s) and work stations would cost $11 to $18 million. These figures include assumptions with regard to additional numbers of CSRs at current CMPs wage rates. CMP compared the ice storm with Hurricanes Gloria in 1985 and Bob in 1991. In the previous storms, CMP had no way of knowing how many subscribers called and received a busy signal. In 1998, it knew exactly how many customers called, and had a lower-than-expected

© 1998 by CRC Press LLC

complaint ratio. We were very impressed by what we read about both the utility and the assisting call center, Twenty First Century.

Discussion: Not Just Like It Was The fate of nonessential departments (depending on the size of the disaster) might be to be forgotten for a day, for a month, or forever. It could surprise everyone in your company how some items might seem unimportant after “the team” has been laboring for more than a month to get the alternate site ready before the end of the hot site contract. The continuity project, as a planning tool, will probably tend to focus the minds of managers on what is critical, possible, and affordable. The CFO might choose to use the breakeven point between risk and preparedness as a financial indicator when deciding whether or not to approve departmental budgets. Your call center manager will probably start planning for data and telephony redundancy, to reduce vulnerability to equipment failure. Your MIS will probably start planning to acquire “off-the-shelf ” equipment and programs, to allow for speed of replacement and portability of applications to other facilities. For example, purchases of office supplies would be simplified if all the printers in your data center used easily available toner cartridges. The effect upon your company is likely to be a positive one. Instead of viewing small disruptions as irritations, your company personnel might view them as merely another test of the disaster plan. As noted previously, it is possible that the employees might become so used to testing the plan that a real disaster might seem no more threatening to them than a larger test. A wide-area disaster could not be ignored by anybody, but your company would have a much better chance to survive one with a well-tested plan and confident personnel.

Call to Action: Plan and Test All employees and all vendors should know about your continuity plan. It should be an important part of orientation for all new employees. Those new personnel should receive their disaster checklists on their first day with the company. To get to the point where management has anticipated most possible disasters, you (and your team) must write the plan. To get to the point where all company personnel (from CEO to CSR) follow their checklists as second nature, you (and your team) must test the plan.

Call to Action: Keep Plans Up-to-Date Failure to test and update the continuity plan is the most likely cause for the demise of a company that had a plan. Testing and updating will keep your plan in tune with new company procedures, upgrades of equipment, and expansion. Call center continuity planning requires a commitment from company personnel at every level. Putting a well-made, well-tested plan into effect should help all personnel greet even unexpected events with skill and confidence.

Contact Us We want to hear your plans and stories. If you are doing the continuity plan for your company and you do not understand something in this book, please send us e-mail at [email protected]. Depending on the number of questions and the detail required to answer, we will either send the answer at no charge, or advise you if there is a fee.

© 1998 by CRC Press LLC

Part 7

Appendices

T

© 1998 by CRC Press LLC

his section contains tables, sources, a case study, chapter notes, and the index.

Appendix A Tables EXHIBIT A-1. PLAN FLOW (FROM EXHIBIT 5-1) Step One: From Exhibit 7.2 (part of the risk assessment in Chapter 7), we take information on the risk of internal fire. This category is divided into 1 through 6 levels of risk. Internal fire

Fire drill False alarm

Small fire Quickly extinguished

Small fire Fire department Called Required to extinguish

Small to moderate fire

Moderate fire

Large fire

Step Two: This flows into Exhibt 7.6, which is a risk result table. This table is divided into the following categories: Shortduration outage

No recovery plan activation required

Return to normal operations

Short evacuation required

Evacuation of call center

Call handover

Return calls

Return to normal operations

Moderate damage to call center or infrastructure

Evacuation of call center

Call handover

Activate hot site

Return calls

Return to normal operations

Severe damage to call center or infrastructure

Evacuation of call center

Call handover

Activate hot site

Activate cold site

Return calls to new site

Return to normal operations

Step Three: In Chapter 9, you and your planning team find out how a plan is created “in general,” to deal with each set of required actions, and how they are accomplished. Step Four: In Chapter 20, you and your planning team see how checklists are created for each position, risk, and set of required actions. Step Five: In Chapters 23 through 25, we ask you to test these checklists, to find any problems with the checklists, planning, or discover risks that were overlooked. Step Six: When, not if — at least the first time around, you and your planning team find a problem, you must begin the process over again to deal with the new risks, or with problems that you had not considered.

© 1998 by CRC Press LLC

EXHIBIT A-2. TASK ASSIGNMENT SHEET (FROM EXHIBIT 5–2)

Recovery Team Selection Team structure (see Exhibit 8-1) Business Impact Analysis Identify business functions Business impact summary Identify company applications Critical application summary Organizational impacts from disasters Develop recovery time frames Risk Analysis Risk analysis matrix Insurance broker consultation Plan Stages Mission statement Definition of disaster Time frames Escalation plan Critical functions Critical applications Command center Team Responsibilities Plan update requirements Predisaster responsibilities Disaster responsibilities Recall/call lists Contact lists (including government authorities) Equipment inventory (and emergency replacement)

© 1998 by CRC Press LLC

How will tasks get done?

When will tasks get done?

Who will be responsible for what stages?

What is the time commitment cost?

EXHIBIT A-2. TASK ASSIGNMENT SHEET (FROM EXHIBIT 5–2) (continued)

Recovery Team Selection Backup Procedures IS backups Hotsite backup Call routing according to site outage and volume Emergency Procedures Life preservation Assessment (of whether or not to declare a disaster). Declaration (of disaster and type) Disaster checklists Emergency call handling Emergency communication procedure Escalation chart Alternate site recovery Personnel restoration Equipment restoration Crisis Management Identify crisis leader External command and control procedures Checklist procedures Return to Normal Operations Hot site Cold Site Alternate locations for permanent/temporary relocation Restoration procedures a.) System restoration b.) LAN restoration

© 1998 by CRC Press LLC

How will tasks get done?

When will tasks get done?

Who will be responsible for what stages?

What is the time commitment cost?

EXHIBIT A-2. TASK ASSIGNMENT SHEET (FROM EXHIBIT 5–2) (continued)

Recovery Team Selection Plan Testing Test objectives/ locations Announced and unannounced testing Tactical exercises Plan maintenance procedures Operational changes (see also team responsibilities) Personnel changes Hardware changes

© 1998 by CRC Press LLC

How will tasks get done?

When will tasks get done?

Who will be responsible for what stages?

What is the time commitment cost?

EXHIBIT A-3. CALL TYPE CHART (ANNOTATED) (FROM EXHIBIT 6-1) Type of Call

Type of Call

Order taking Yes

Telemarketing outbound Yes

Reservations Yes

Answering inquiries for general information Yes

Product location inquiries Yes

Sales Yes

Channeling customer complaints to others for resolution Sometimes (to sales manager)

Collections

Solving customers problems

Outage reporting

Answering inquiries about product/service use Yes

Scheduling repair/warranty service

Contest lines

Advertising Sometimes

Advising senior management on customer issues Sometimes (to sales manager)

Market research Sometimes

Public relations

Quality management

Crisis handling Rarely (transfer to sales manager)

Billing

Defining customer service/Policy/strategy

Specialized certification/Licensing

© 1998 by CRC Press LLC

EXHIBIT A-4. CALL TYPE CRITICAL IMPACT (ANNOTATED) (FROM EXHIBIT 6-2) Call type Location of Calls Priority of call type Income per call or cost of operations per call Average duration of call Number of stations capable or dedicated to receiving call type Average number of calls covered per hour per station/total Peak number of calls per hour, total Estimated loss of business if down 30 minutes during peak Estimated loss of business if down 1 hour during peak Estimated loss of business if down for 4 hours at peak based on average call volume Estimated loss of business if down for 1 day based on average call volume Estimated loss of business if down for 2 days based on average call volume

© 1998 by CRC Press LLC

EXHIBIT A-5. CALL FLOW BY CALL TYPE (FROM EXHIBIT 6-3) Call Type Originating facility (home, office, both, call center) Routing Path 1 Routing Path 2 Routing Path 3 Receiving Path 1 Receiving Path 2 Receiving Path 3 Processing Path 1 Processing Path 2 Processing Path 3 Postprocessing Path 1a Postprocessing Path 1b Postprocessing Path 2 Postprocessing Path 3 Postprocessing Path 4

© 1998 by CRC Press LLC

Primary

Subcategory

Sub-Subcategory

EXHIBIT A-6. CALL CENTER DATA REQUIREMENTS (FROM EXHBIIT 6-4)

Call Types*

*

Critical Applications

Data Processing Location

Group all types that have similar data requirements together.)

© 1998 by CRC Press LLC

Time to Restore, Include Backlog

Restoration Cost

EXHIBIT A-7. COMMUNICATIONS INTERDEPENDENCIES (FROM EXHIBIT 6-5)

Department: Telephones Facsimile machines Computer stations Mail Couriers Internal e-mail — internal reporting External e-mail Other

© 1998 by CRC Press LLC

Number of Units

Estimated Daily Usage

Estimated Loss If Unable to Use

Additional Information

EXHIBIT A-8. UTILITIES (FROM EXHIBIT 6-6) Question How many power grids service the building? Can you switch from one power grid to another effectively? How long would the UPS last without power? Is there a generator in the building or reserved off site for your company? Has the generator been tested and maintained on a regular basis? How long can the company carry on operations without power?

© 1998 by CRC Press LLC

Answer

Downtime

Cost

EXHIBIT A-9. LOST SALES (FROM EXHIBIT 6-7) Question In the event of a disaster would your customers purchase from one of your competitors? If those customers purchase from other companies, would those sales be lost permanently? Do you provide an exclusive service? Would you lose customers (accounts) permanently during a disaster? Would your loss of customers be only temporary?

© 1998 by CRC Press LLC

Answer

Cost

EXHIBIT A-10. NEGATIVE CONSEQUENCES (FROM EXHIBIT 6-8) Question Additional operating costs Cost to replace equipment, buildings Public relations/advertising to repair reputation Cleaning, repair, decontamination (mitigation) Inadequate service to customer What are the effects of loss of business?

© 1998 by CRC Press LLC

Answer

Cost

EXHIBIT A-11. LEGISLATION AND PENALTIES (FROM EXHIBIT 6-9) Question Are you governed by the Foreign Corrupt Practices Act? What penalties could your company and management suffer under law if they were ill prepared for a disaster? What legislation is applicable in your state, or industry? For example, is your business governed by the Environmental Protection Act?

© 1998 by CRC Press LLC

Answer

Cost

EXHIBIT A-12. LICENSING (FROM EXHIBIT 6-10) Question Can the company lose its license(s)? Is your company in insurance, stock brokerage, engineering, utilities or in the field of medicine? What kind of insurance will you need? Can the company be sued for damages for not answering its telephones? What are the reporting requirements under legislation? What are the consequences of not reporting?

© 1998 by CRC Press LLC

Answer

Cost

EXHIBIT A-13. SAFETY LEGISLATION (FROM EXHIBIT 6-11) Question What legislation governs the safety of workers in your state or industry? If workers are killed or injured in a disaster, what are the penalties? What are the standards for preparedness?

© 1998 by CRC Press LLC

Answer

Cost

EXHIBIT A-14. CUSTOMER LOSSES (FROM EXHIBIT 6-12) Question What are the possible losses to customers in the first hour? In the fourth hour? In the first day? After the first week of a disaster? Would customers seek a refund, or would they take your company to court? Would a judge understand your company difficulties or award a large settlement? How many customer or client accounts might be affected by a disruption?

© 1998 by CRC Press LLC

Answer

Cost

EXHIBIT A-15. LOST TRUST (FROM EXHIBIT 6-13) Question Would your company permanently lose customers during a disruption? How difficult would it be, and what would be the costs associated with, gaining back customers lost when they could not contact your company? What would the company have to do to assure customers that this type of disaster would not affect them again in the future? Would customers be more likely to be demanding in the future?

© 1998 by CRC Press LLC

Answer

Cost

EXHIBIT A-16. LOST TRUST (FROM EXHIBIT 6-14) Question

Answer

Cost

Who at your company would be liable for customer damages? Would directors have to pay personally, or would the company only bear the liability? Who could be named in a lawsuit? Will breaches of contract affect a lawsuit outcome? Does the company have service-level agreements or specific performance levels that must be met? What is the risk of customer death or injury? What is the liability of the company in the event of death or injury? How much does liability increase if the company has no plan? What are the reasonable requirements according to what other companies have? What would the company be expected to plan for?* *Companies would be expected to have planned for an ice storm in northern Quebec but probably would not be in southern Florida. Similarly, a heat wave in northern Canada might be unexpected, but would be normal in the southern United States.

© 1998 by CRC Press LLC

EXHIBIT A-17. LOST MARKET SHARE (FROM EXHIBIT 6-15) Question How many competitors do you have? How likely are competitors to be affected by the same disaster at the same time as you are? How quickly and effectively would your competitor take advantage of the disaster? Does your competitor have a continuity plan? If so, could they carry on business when you were not able to?

© 1998 by CRC Press LLC

Answer

Cost

EXHIBIT A-18. LOST GOOD WILL (FROM EXHIBIT 6-16) Question How would local news media react? Would you be accused of lack of sensitivity, or poor planning? How well would your employees deal with customers who were rude, upset at the outage, or irate due to delays? How would your customers be affected by your disaster? Would your disaster in turn create a disaster for your customers? Would you have to issue public relations announcements? What would happen with your technical service or customer service lines?

© 1998 by CRC Press LLC

Answer

Cost

EXHIBIT A-19. LOST PRODUCTIVITY (FROM EXHIBIT 6-17) Question Would the simple threat of a disaster affect your employee productivity? After the disaster, would your employees be mentally and physically able to return to work at their normal productivity? How long would this process take? Do you have a plan to get psychological therapy for employees who are traumatized by the disaster so that they could return to work? Would employees’ families be affected by the disaster? How would this affect the employees? Would employees feel that management would support them through the disaster? Would employees be able to get to work or would the disaster make travel impossible? What help would employees expect for long-term Trauma, and Injury? What help would employees expect for dependents, and damage to home? Would it seem like management cares? What about casualties at the workplace? How would you replace and train workers? What are the costs of replacing a worker? What are the costs associated with disability compensation for workers? What are the costs of dealing with survivor trauma? How would you get employees back to work?

© 1998 by CRC Press LLC

Answer

Cost

EXHIBIT A-20. LOST INVESTORS (FROM EXHIBIT 6-18) Question What would be the effective investor confidence after a disaster? What would be the risk to capital? Are investors aware of your current lack of a business continuity plan? Would investors compare business continuity vs. Risk to assets? Would there be a massive sell-off of company shares after disaster? What is the chance of the company surviving investor loss of confidence and sell-off of shares? What is the chance that investors pulling out might affect the ability of the company to meet its current financial obligations? What would be the additional operating costs to replace buildings and equipment? What would you need to spend in advertising to repair the company reputation? Would prior agreements be met? What litigation would be forced upon the company because of the inability to complete agreements? What would be the effect on future IPOs?

© 1998 by CRC Press LLC

Answer

Cost

EXHIBIT A-21. CALCULATED LOSS CHART (FROM EXHIBIT 6-19) 30 Min Peak

1 Hour Peak

4 Hours Avg.

Exhibit 6-4

N/A

N/A

N/A

All other Exhibits in Chapter 6

N/A

N/A

N/A

1 Day Avg. 2 Days Avg.

Exhibit 6-2

© 1998 by CRC Press LLC

N/A

EXHIBIT A-22. EXPECTED LOSS DUE TO DOWNTIME CHART (FROM EXHIBIT 6-20)

Number of Disaster Risks in Category

Ratio of Expected Events

Very high

1:4

High

1:6

Moderate

1:12

Low

1:36

Very low

1:64

Number of Risks Expected to Occur

Average Hours of Downtime for These Risks

Expected Dollar Cost

To use this table: 1. Take the total number of risks in the category. 2. Divide by the ratio. If you have a 90% probability of disaster it is extremely probable that 1:4 of them will strike. This equals the Number of Risks Expected to Occur. 3. Multiply by the number of hours, and estimate how many working hours or working days the disaster would consume. 4. Multiply by the appropriate time box on Exhibit 6-19 to arrive at your expected dollar cost.

© 1998 by CRC Press LLC

EXHIBIT A-23. INDUSTRY-BASED RTO (FROM EXHIBIT 6-21) Time Frame

RTO before Call Centers

Industry Type

RTO Essential to Company Survival

Immediate

Formerly 0 to 24 hours

“Instant contact” industries, e.g., sales, utilities, “just-in-time” supply system

Recovery 0 to 30 minutes requires prerouted switching, hot site*

Intermediate

Formerly 1 to 7 days

“Return call” industries, e.g., professionals, consultants

Recovery 30 minutes to 2 days requires warm site or hot site

Long-term

Formerly 7 days or more

“Return letter” industries, e.g., manufacturing, scheduled projects

Recovery 2 to 5 days requires warm site or ramped-up cold site

*Prerouted switching would be to an interim call handling or real-time minimum overflow facility, as discussed in Parts 3, 4, and 5.

© 1998 by CRC Press LLC

EXHIBIT A-24. COST-EFFECTIVE RTO (FROM EXHIBIT 6-22)

Time Period Peak Average daytime Average evening Average midnight

© 1998 by CRC Press LLC

Acceptable Loss

(Divided by) Cost Per Hour Of Outage

(Equals) RTO

© 1998 by CRC Press LLC

Tornado warning

Weather watch

Medical alert notice

Personnel delays at shift-time

NONE

Flood watch

Roof leak

Flood watch

Notice of possible quake

Heavy thunder storms/wind damage

Epidemic

Snow and ice storms

Tidal wave

Wide-area flooding

Internal flooding

External flooding

Earthquak or seismic damage

1

Tornado

Hurricane or typhoon

Natural Disaster

Disaster Type

Personnel disruption in affected area

Affecting some people getting to work

Roof leak near ACD

Flood warning

Minor power outages

Some personnel affected

Hurricane warning

2

Major equipment failure Major damage; 3–5 days

Major damage; RTO 1–3 days Minor tremor; some damage; RTO 1–3 hours

Building flooded; major repairs required

Building under water more than a week

Power out less than five days; major damage

Building or personnel quarantine

Power outage more than a day

Strike within six blocks

Hurricane strike within 30 min

5

Water leak in building; some damage

Major water damage to equipment

Major flood damage

Power out 1 day; property damage

More than 50% affected

Damage to power lines, e.g., broken branches

Strike within city

Hurricane watch

4

Building surrounded by water

Water main to upper floor burst

Minor flood damage

Widespread power outages

30% personnel absent

Weather warning

Tornado watch

3

Building destroyed; 1–4 weeks

Building evacuated; equipment replacement required

Repairs required 2–4 weeks

Building and equipment destroyed

Power out more than five days.

Power out more than a week

Direct strike to facility

Hurricane strike

6

Note: To save on typing time, we have included an annotated chart for this appendix. Please delete anything that does not fit with your plan.

EXHIBIT A-25. RISK ASSESSMENT MATRIX (FROM EXHIBIT 7-2)

© 1998 by CRC Press LLC

EXHIBIT A-25. RISK ASSESSMENT MATRIX (FROM EXHIBIT 7-2) (continued) Disaster Type

1

Volcanic eruption

Possible eruption notice

Electromagnetic interference, solar flares, etc.

Brief interruption; solar flares 1–3 min

2

3

4

5

6

Affecting personnel on route to work

Fallen ash causing breathing problems

Evacuation of nonessential personnel

Lava flow; complete evacuation

Building and equipment destroyed

Infrastructure Disaster Hardware failure

Minor glitch affecting only a few PCS

Power failure to some equipment 1–3 hours

Major hardware breakdown; RTO 1–5 hours

Main cable cut; repairs require 2–12 hours

Major damage to cable; RTO 2–5 days

Equipment replacement required 1–3 weeks

Software failure

Virus alert

Virus detected; 30–60 min to clear

Minor failure of main program RTO 1–3 hour

Major program failure; RTO 2–8 hours

Complete software failure; 1–3 days

Replacement required of all software; RTO 1–3 weeks

Power failure

Some lights affected

Minor disruption to equipment; 30–60 min

Major disruption; 1–4 hours

Complete power failure; RTO 4–12 hours

Major transformer/circuit damage; RTO 1–3 days

Major equipment damage; RTO 1–3 weeks

Disruption to local service only

800 service lost; RTO 1–3 hours

Major disruption of service; RTO 1–6 hours

Complete loss of landlines; RTO 6–24 hours

Major repairs required; 1–3 days

Heating, ventilation, or air-conditioning failure Water

NO COFFEE

Gas pipeline break Gas leak Telco local CO

© 1998 by CRC Press LLC

Telco ixc/lec or 1-800 service

Occasional disruption of local service

Lost of 800 lines; RTO 30–60 min

Minor interruption of IXC/LEC; 1–3 hours

Major failure of communications RTO 1–4 hours

Complete communications Failure; RTO 1–12 hours

Major equipment damage; RTO 1–3 days

Chemical exposure

News release on highway chemical spill

Some personnel delayed on route to work

Minor exposure; some evacuation required

Major exposure; full evacuation of staff

Minor equipment damage due to corrosion

Major corrosion damage; RTO 1–5 days

Internal fire

Fire drill False alarm

Small fire, quickly extinguished

Small fire, fire department called, required to extinguish

Small to moderate fire

Moderate fire

Large fire

External fire

Grass fire affecting personnel in route to work

Adjacent building on fire under control

Building threatened; evacuation of people 1–4 hours

Some external equipment damaged; RTO 4–12 hours

Major damage to building and equipment; 1–5 days

Building and equipment destroyed

Hits on the phone line

Minor interruptions to local telephones

Failure of internal communications 1–3 hours

Major failure to internal/external communications 1–4 hours

Major repairs required; 1–3 days

Major cable replacement; 1–3 weeks

Man-Made Disaster

External criminal activity Facility structure Failure of system software Explosion Loss of physical access to resources Malfunction or failure or CPU Telecommunications failure Communications Failure internal/external

© 1998 by CRC Press LLC

EXHIBIT A-25. RISK ASSESSMENT MATRIX (FROM EXHIBIT 7-2) (continued) Disaster Type

1

2

3

4

5

6

Human error

Minor mistakes corrected in minutes

Accidental program deletion; 1–3 hours

Allow access to restricted area

Damage caused by unauthorized personnel

Disgruntle employee

Complains continually — family problems

Mental anguish — disrupts other workers

Threat to other personnel and equipment

Distraught — destroys equipment and causes a calamity

Shoots supervisor, wounds co-worker

Kills himself or herself

Strikes/picketing

Negotiations

Noon pickets

Strike declared

Major picket line

No workers will cross; work stops

Plant closes

Minor embarrassment; no damage

Some codes compromised

Some data must be replaced; 1–3 hours

Major compromise of sensitive data

Data obtained by competitor

Data used by competitor

Data entry error Improper handling of sensitive data Unauthorized physical access Malicious damage or destruction or software or data Unauthorized access to data or theft of data Unauthorized or accidental modification of software or hardware Robbery

© 1998 by CRC Press LLC

Theft

Employees taking pens and pencils

Software disks taken

Keys and codes to restricted areas taken

Sabotage caused by theft

Major hardware stolen

Company secrets sold to competitor

Bomb threats

Bomb threat; briefings to personnel

All threats to be taken seriously

All personnel evacuated area searched

Bomb squad called; bomb located; no damage

Small explosion; some damage

Building and equipment destroyed

Civil disorder

News reports

Strikers interfering with workers

Protesters disrupting operations, tying up equipment

Destruction of property

Sabotage of equipment

Operations forced to shut down

External site affects personnel reporting to work

Internal spill; minor clean up 1–10 min

Major spill; some evacuation required

Major spill causes full evacuation 1–12 hours

Corrosive spill; some damage to equipment; 1–5 days

Major equip damage; 1–3 weeks to replace

Possibility of aircraft accident

Road closure due to aircraft accident

Aircraft crash near site; personnel evacuation

Aircraft crashes into site; fire damage

Major damage; 1–5 days

Complete loss of building

Burglary

Nuclear attack Chemical spill

Kidnapping Vandalism Sabotage War Biological contamination Toxic Waste Vehicle crash Airport proximity

Work stoppage — internal/external Computer crime

EXHIBIT A-26. TEAM SELECTION BY CRITICAL COMPONENTS (FROM EXHIBIT 8-1) Company Primary Planning Manager Crisis or Incident Manager Senior Management Representative Accounting Department Representative “Disaster Site” Recovery Team ICH Facility Team Hot Site Recovery Team

© 1998 by CRC Press LLC

Company Secondary

Location 1

Location 2

EXHIBIT A-27. ADDITIONAL POSSIBLE PERSONNEL (FROM EXHIBIT 8-1) Company Primary PBX, ACD, IVR logs Vectoring Customer Call #’s (800- and 888#’s, Call Coordinators, ‘Must answer’ lines redirected Agents to work problems (skills, training, accommodations) Work stations/tools (includes data scopes for IE) Applications/Systems- Lotus Notes, CSS, Web tools/servers, IP Admin, etc. Critical Customer Support (list) — XNS Corp, IPS Inc., Banking, etc. Internet Support as backup Communications (Pagers, cellular phones) Customer Billing Visa, MC, remote authorization Customer Fulfillment (shipping) Physical Connectivity LAN WAN -verify network plan Command Centers Access to main office lines redirected IE Support Outage<2 days require support Alt. Call center > 2 days outage

© 1998 by CRC Press LLC

Company Secondary

Location 1

Location 2

EXHIBIT A-28. SCALABLE LEVELS OF RESPONSE (FROM EXHIBIT 9-3) Category

Decision

Shortduration outage 5 determine if fire is real

Short evacuation required

8 determine size of fire

Moderate damage to call center or infrastructure 11 assess damage after fire is extinguished

Severe damage to call center or ir infrastructure

© 1998 by CRC Press LLC

Actions 1 Fire alarm

2 Take “to go” case and evacuate

3 Meet in designated area 1

6 If no, then

7 No recovery plan activation required

8 Return to normal operations

6 If yes, then

7 Call handover to ICH or RTMO facilities

9 If small fire with no damage to call center, then

10 Return to call center; begin clean up

9 If moderate to large fire

10 Activate hot site

12 If damage is moderate and will be able to return in 45 days or less, then 12 Activate cold site or alternate location

4 do a head count

11 Return calls

12 return to normal operations

13 Complete repairs as necessary

14 Return calls

15 return to normal operations

13 Activate drop-ship provisions from vendors

14 Set up new facility

15 return calls to new site

16 return to normal operations

EXHIBIT A-29. CONTENTS OF INCIDENT MANAGER’S “READY TO GO” CASE (FROM EXHIBIT 9-4) Checked One satellite or cellular telephone, with battery charging equipment Maps to hot site(s) or other emergency facilities Checklists (risk type, team tasks, call/recall, inventories, etc.) Emergency communication contact list The complete disaster plan Flashlight Advanced first aid kit Summary of emergency vendor agreements Company VISA card with minimum $10,000 limit Cash minimum $500 Optional laptop computer with battery and power cord and fax modem and software Note: All batteries should be tested periodically.

© 1998 by CRC Press LLC

Date

EXHIBIT A-30. CRITICAL FUNCTIONS (FROM EXHIBIT 9-5) Call Type Critical Functions

© 1998 by CRC Press LLC

Notes

EXHIBIT A-31. CRITICAL APPLICATION REQUIREMENTS (FROM EXHIBIT 9-6) Critical Application

© 1998 by CRC Press LLC

Requirement 1

Requirement 2

Alternate Location

Expected Result

EXHIBIT A-32. OTHER CRITICAL EQUIPMENT (FROM EXHIBIT 9-7)

Tasks

© 1998 by CRC Press LLC

Frequency

Positions

Equipment, Forms, and Supplies

Software and Documents

Facilities

EXHIBIT A-33. RECOVERY SITE INVENTORY (FROM EXHIBIT 9-8) Date _____ Time

Recovery Site Inventory Checklists — hot site recovery procedures Power and lights Heat and plumbing Telephone lines operational Computer data lines operational Computers checked and functioning Backup data tapes arrive on site Supplies checked Personnel contact/recall lists Critical business functions (applications) Interim call center online Assisting call center online (overflow, expert calls, telecommuters) Communications established with disaster site Vital records recovered from storage (e.g., service-level agreements) Vendors notified Food, water, sleeping accommodation arranged Additional items

© 1998 by CRC Press LLC

Problems

Initials

EXHIBIT A-34. BACKUP AND RECOVERY ACTIONS (FROM EXHIBIT 9-9) Action

Consequence

Routine backups

Can recover data to last backup period

Special backups on declaration of emergency

Can recover data up to minute of declaration

Alerting assisting call center(s)

Ich facility stands ready for prerouted call switching

Prerouted switching of calls to assisting call center and/or telecommuters

Telco initiates call switching on receipt of coded password

Sending current backup information to hot site

Csrs standing by get data current up to minute of backup

By data line

Data arrives in seconds or minutes

By courier

Data arrives in hours

When emergency prevents current backup

Must use last backup data

Opening last backup at warm/hot site

Customer and other data known only to last backup period

Last backup with the verbal comments of your csr

Last transactions not backed up might be replicable from remembered details

© 1998 by CRC Press LLC

EXHIBIT A-35. ALTERNATE POSITION LISTING (FROM EXHIBIT 9-11)

Position

© 1998 by CRC Press LLC

Primary Person

Telephone

Alternate Person

Telephone

Alternate Person

Telephone

EXHIBIT A-36. EXTERNAL CONTACT LIST (FROM EXHIBIT 9-12)

Name of Organization FEMA/EPA

Police Department — 911

Ambulance — 911

Fire Department — 911

Hospitals

National Guard

Post Office

Disaster Service Centers*

Public Relations Consultants

© 1998 by CRC Press LLC

Contact Person/Position

Telephone Numbers (Main/Cellular/Pager)

EXHIBIT A-36. EXTERNAL CONTACT LIST (FROM EXHIBIT 9-12) (continued)

Name of Organization

Contact Person/Position

Television

Legal Assistance

Personnel Agencies

Utility Companies

Vendors

Security

Major Account Customers

* For example, see IBM BRS at end of Chapter 9.

© 1998 by CRC Press LLC

Telephone Numbers (Main/Cellular/Pager)

EXHIBIT A-37. CRITICAL DOCUMENTATION INVENTORY (FROM EXHIBIT 9-13) Storage Locations Service-Level Agreements

Contracts

Recovery Plan Copies

Computer Manual(s)

Operating Systems Software Manual(s)

Software Manual(s)

© 1998 by CRC Press LLC

Method of Replacement

EXHIBIT A-37. CRITICAL DOCUMENTATION INVENTORY (FROM EXHIBIT 9-13) Storage Locations Application Software Manual(s)

Disaster Plan Standards Documentation

Accounting, ATM Cards

Accounting Procedures

Policy/Procedure Manuals

Recovery Plan Copies

Other Documentation

© 1998 by CRC Press LLC

Method of Replacement

EXHIBIT A-38. EQUIPMENT INVENTORY (FROM EXHIBIT 9-14) Type of Assistance Electrical power Telephony Information systems instrumentation Computer hardware Computer software First aid equipment Emergency food Emergency shelter Portable water Personal care items Waste control Heating/cooling Secure document storage local Secure document storage distant Magnetic media replacement Backups storage off-site local Backups storage off-site distant Backups storage on-site Banking

© 1998 by CRC Press LLC

Vendor

Description

Replacement Time

Model

EXHIBIT A-39. INVENTORY OF FORMS (FROM EXHIBIT 9-15)

Form Number

© 1998 by CRC Press LLC

Type

Supplier(s)/ Telephone

Replacement Time

Off-Site Storage

Notes

EXHIBIT A-40. DISTRIBUTION REGISTER (FROM EXHIBIT 9-16) Department

______________________

Plan Version #

______________________

Update Number ______________________

Station

© 1998 by CRC Press LLC

Checklist Number

Date

Name of Person Inserting Update

Initials

© 1998 by CRC Press LLC

Determine size of fire If small fire with no damage to call center

Escalation

Call handover to ICH or RTMCA facilities

Yes

Determine if fire is real

Advise staff of evacuation

911-notify backup call center of fire alarm

Checklist states begin evacuation

Assessment

No recovery plan activation required

Do a head count

Recovery of calls

Meet in designated area

No

Take to-go case and evacuate

Evacuation

Escalation

Do a head count

Advise staff of evacuation

Declaration

Determine if fire is real

Meet in designated area

911-Notify backup call center of fire alarm

Notification

Assessment

Take to-go case and evacuate

Checklist states begin evacuation

Assessment

Fire alarm

Fire alarm

Alarm

Fire

Fire

Event

Short-Duration Outage

Short Evacuation Required

EXHIBIT A-41. STAGES OF DISASTER (FROM EXHIBIT 12-7)

If moderate to large fire

Determine size of fire

Call handover to ICH or RTMCA facilities

Yes

Determine if fire is real

Do a head count

Meet in designated area

Take to-go Case and evacuate

Advise staff of evacuation

911-Notify backup call center of fire alarm

Checklist states begin evacuation

Fire alarm

Fire

Moderate Damage to Call Center or Infrastructure

If moderate to large fire

Determine size of fire

Call handover to ICH or RTMCA facilities

Yes

Determine if fire is real

Do a head count

Meet in designated area

Take to-go Case and evacuate

Advise staff of evacuation

911-notify backup call center of fire alarm

Checklist states begin evacuation

Fire alarm

Fire

Severe Damage to Call Center or Infrastructure

© 1998 by CRC Press LLC

EXHIBIT A-41. STAGES OF DISASTER (FROM EXHIBIT 12-7) (continued)

Short-Duration Outage Recovery Plan

Short Evacuation Required Return to call center

Moderate Damage to Call Center or Infrastructure

Severe Damage to Call Center or Infrastructure

Activate hot site

Activate hot site

Activate all three recovery teams

Activate all three recovery teams

Notify to ICH or RTMCA facilities of situation

Notify ICH or RTMCA facilities of situation

Coordinate with local fire authorities

Coordinate with local fire authorities

Assessment

Assess damage after fire is extinguished

Assess damage after fire is extinguished

Escalation

Damage is moderate, will be able to return in 45 days or less

Large amount of damage, will not be able to return for more than 45 days

Recovery plan

Continue operations in hot site

Activate cold site or alternate location

Complete repairs as necessary

Activate drop-ship provisions

Restoration of system

Begin clean up

Set up new facility Restoration of calls

Return calls

Return calls

Return calls to new site

Return to normal operations

Return to normal operations

Return to normal operations

Return to normal operations

Return to Normal Operations

Plan evaluation and update

Evaluate evacuation and update

Evaluate evacuation and update plan if necessary

Evaluate fire recovery and update plan if necessary.

Evaluate fire recovery and update plan if necessary

EXHIBIT A-42. ESCALATION CHART WITH OTHER TYPES OF RISKS (FROM EXHIBIT 12-8) Event

Fire

Tornado

1. Alarm

a. Fire alarm b. Smoke c. You hear “FIRE, FIRE, FIRE”

a. Radio reports sighting nearby b. Tornado sirens

2. Assessment

Checklist states call 911, recovery team, and begin evacuation

Checklist states call recovery team and begin evacuation

3. Notification

a. 911 b. Notify recovery team of fire alarm

Notify recovery team of tornado

4. Declaration

Advise staff of evacuation

Advise staff of evacuation

5. Evacuation

Take to-go case and evacuate a. Meet in designated area b. Do a head count

Take to-go case and evacuate a. Meet in designated shelter b. Do a head count

6. Assessment

Determine if fire is real

Determine if the tornado is going to effect the call center

7. Escalation

If Yes, go to next/ If No, go to 18

If Yes, go to next/ If No, go to 18

8. Recovery of calls

Call handover to ICH or RTMCA facilities.

Call handover to ICH or RTMCA facilities

9. Assessment

Determine size of fire

Determine extent of damage

10. Escalation

a. If moderate to large fire, go to next/ b. If small fire with little or no damage to call center, go to 16a or 18

a. If power failure, go to power failure escalation chart b. If tornado hits call center, go next c. If tornado does little damage, go to 16

11. Recovery plan

Activate hot site

Activate hot site

Activate all three recovery teams

Activate all three recovery teams

Notify ICH or RTMCA facilities of situation

Notify ICH or RTMCA facilities of situation

Coordinate with local fire authorities

Coordinate with local fire authorities

Assess damage after fire is extinguished

Assess damage after tornado has passed

12. Assessment

© 1998 by CRC Press LLC

EXHIBIT A-42. ESCALATION CHART WITH OTHER TYPES OF RISKS (FROM EXHIBIT 12-8) (continued) Event

Fire

Tornado

13. Escalation

a. If damage is moderate, will be able to return in 45 days or less, go to 15a b. If large amount of damage, will not be able to return for more than 45 days, go to 15 b, c, and d

a. If damage is moderate, will be able to return in 45 days or less, go to 15a b. If large amount of damage, will not be able to return for more than 45 days, go to 15 b, c, and d

14. Recover Plan

a. Continue operations in hot site or b. Activate cold site or alternate location c. Activate drop-ship provisions d. Set up new facility

a. Continue operations in hot site or b. Activate cold site or alternate location c. Activate drop-ship provisions d. Set up new facility

15. Restoration of system

a. Begin cleanup b. Complete repairs as necessary or

a. Begin cleanup b. Complete repairs as necessary or

16. Restoration of calls

Return calls

Return calls

17. Return to normal

Return to normal operations

Return to normal operations

18. Plan evaluation and update

Evaluate evacuation and update

Evaluate evacuation and update

© 1998 by CRC Press LLC

EXHIBIT A-43. TYPES OF RESTORATION SERVICES (FROM EXHIBIT 13-1) Type of Service* Actuaries Air Ambulance Charter & Evacuation Appraisers Architectural/Engineering Services Art Restoration Audio Tape & Record Restoration & Preservation Ballast Recyclers (Lighting) Banking Equipment Repair Building Materials, Bulk Quantity Brokers Burglar & Fire Alarm, Consultation & Design Cleaning Companies, Data Processing Site Specialists Cleaning Companies, Electronic Equipment PCs on Site & off Site Cleaning Companies, Insurance/Disaster Specialists Cleaning Companies, Medical Cleanup: Post-Crime/Post Cleanup & Restoration Computer Repair Specialists Concrete & Masonry Repair Conservation: Art, Library, & Museum Construction, Emergency Cooling Services Crisis Management Services Data & Records Recovery Data Backup/Recovery Software Data Distribution Service Data Recovery

© 1998 by CRC Press LLC

Company Name

Address

Phone No.

EXHIBIT A-43. TYPES OF RESTORATION SERVICES (FROM EXHIBIT 13-1) (continued) Type of Service* Data Recovery from Broken Hard Disks Data Recovery from Damaged Media Misc. Data Recovery from Damaged Tape Data Recovery from Optical Disks Dec Equipment Repair Decontamination Equipment, Portable Detective Agencies Disaster Specialists Disk Access Control Dispatch Software Disposal, Electrical Equipment Disposal, Hazardous Waste Dry Cleaning & Laundry: Specialty & Emergency Drying & Dehumidification Education, Training & Awareness Electric Heaters, Dehumidifiers Electronic Vaulting Emergency Information Location Emergency Management Software Emergency Response Logistics Emergency Staffing Software Environmental Cleanup Erased & Damaged File Recovery Utilities Facilities Preparation for Data Processing Equipment Fire & Water Damage Restoration Fire Protection Flood Damage Restoration

© 1998 by CRC Press LLC

Company Name

Address

Phone No.

EXHIBIT A-43. TYPES OF RESTORATION SERVICES (FROM EXHIBIT 13-1) (continued) Type of Service* Food, Lodging, & Sanitary Services, Emergency Food, Lodging, & Sanitary Services — Mobile Forensic Accountants Forensic Engineers Full-Service General Contractors: Restoration & Reconstruction Furniture Restoration Glass Board-Up Government Services Guard Services Hard Disk Repair Hazardous Material Management Software Hazardous Materials Heating Services Housing, Temporary IBM PCs & Compatibles — Repair Incident Management & Planning Software Information Integrity Verification Software Information Management & Protection Instrument Repair Insurance Insurance Adjusters, Independent Insurance Adjusters, Public Insurance, Special Situations Item Processing Jewelry, Fire Damage Repair Jewelry, Replacement Laser Printer Repair

© 1998 by CRC Press LLC

Company Name

Address

Phone No.

EXHIBIT A-43. TYPES OF RESTORATION SERVICES (FROM EXHIBIT 13-1) (continued) Type of Service* Legal Issues, Environmental Legal Issues, Insurance Life Support Services & Supplies Lighting, Emergency Lighting, Solar Generated Liquidators/Salvage Log-Off, Automatic Loss Prevention & Investigation Macintosh — Repair Mailing, Printing & Inserting Services Maintenance Services, Computer/Networks Microfilm Drying & Re-Wash Microfilm Drying & Re-Wash Service Microfilming & Record Restoration Mirror-Imaging Software/Hardware Misc. Peripherals Repair Missing Source-Code, Recovery Mobile Administrative Offices & Command Centers Mobile Bank Branches Mobile Buildings & Teller Facilities Mobile Cargo Storage Space Mobile Classrooms Mobile Communication Centers Mobile Computer Rooms & Data Centers Mobile Computers Client-Specified Configurations Mobile Computers Hewlett Packard Mobile Computers IBM — Midrange

© 1998 by CRC Press LLC

Company Name

Address

Phone No.

EXHIBIT A-43. TYPES OF RESTORATION SERVICES (FROM EXHIBIT 13-1) (continued) Type of Service* Mobile Computers Wang Mobile Computers, Delivered On Site Mobile Disaster Response Trailers Mobile Homes Mobile Satellite Communication Dishes, Emergency Museum Services Network Backup/Recovery Software Network Census & Inventory Software Network Problem Analysis & Management Network Recovery & Recovery Planning Notification & Dispatch Services, Phone, Fax, Pagers Office Equipment Repair Office Furniture Restoration Off-Site Record & Hard-Copy Storage Off-Site Tape, Film, Optical, etc. Data Storage Online Software Backup/Recovery Outside Audit of Disaster Plan Adequacy Outside Evaluation Audit of Disaster Plan Adequacy Overspray Removal Services Password Management & Recovery Utilities Personal Computer Repair Power Protection & Continuity Power-Loss-Detect-&-Shutdown Utility Software Printer Repair — General Printing, Emergency — Business & Bank Forms

© 1998 by CRC Press LLC

Company Name

Address

Phone No.

EXHIBIT A-43. TYPES OF RESTORATION SERVICES (FROM EXHIBIT 13-1) (continued) Type of Service* Psychological Services Crisis Management Psychological Services Human Factors Psychological Services Trauma Counseling Public Relations Records Management Recover Water-Damaged Books & Documents Referral Service, Disaster Cleanup Remote-Operation Software Restoration: Audio Materials Restoration: Video Materials Salvage & Surplus Buyers Satellite Dishes, Mobile Scrap Computer & Banking Equipment — Buyers Scrap Fluorescent Lamps & Ballast — Buyers Scrap Hard Drives — Buyers Search & Rescue Secretarial Services, Temporary Security — General Security & Access Control Security Systems — Repair & Replacement Site Remediation Smoke & Odor Counteracting Products Smoke & Odor Counteracting Services Snow Removal Plows, Trucks & Equipment Steam Cleaners & Pressure Washers Stolen Computer Equipment Registry Subrogation Services

© 1998 by CRC Press LLC

Company Name

Address

Phone No.

EXHIBIT A-43. TYPES OF RESTORATION SERVICES (FROM EXHIBIT 13-1) (continued) Type of Service* Sun Equipment Repair Surge Protectors Surplus Computer Equipment Buyers Surplus Computer Equipment: Charitable Donations Switches, Ruggedized & Specialty Tanks, Storage Tape (Cartridge) Testing Systems Tape Backup Units Tape Backup Units — Portable Tape Drive Repair Tape Library Management Systems Telecommunications Telecommunications, Specialty and Nationwide Coverage Telecommunications: Lost-Power Bypass Units Telecommunications: Wireless & Fiber Optics Telephone Answering Telephone Dialing Equipment Automated Telephone Supply Transformers, Electric Transportation, Specialized Trucking Turnkey Data Processing Site & Service Restoration Underground Rescue UPS — Uninterruptable Power Supplies Videotape Restoration Virus Detection/Prevention/Recovery Voice/Call Center Recovery

© 1998 by CRC Press LLC

Company Name

Address

Phone No.

EXHIBIT A-43. TYPES OF RESTORATION SERVICES (FROM EXHIBIT 13-1) (continued) Type of Service*

Company Name

Warning Systems Water Damage Restoration Water Testing Equipment Weather Forecasting Weather-Tracking Software, PC-Based Wireless Telecommunications Backup Workplace Violence Wrapping, Protective: Buildings & Structures Year 2000 Software Problems: Analysis & Solutions *These categories were selected from Disaster Recovery Yellow Pages.

© 1998 by CRC Press LLC

Address

Phone No.

EXHIBIT A-44. EXAMPLE INCIDENT REPORT (FROM EXHIBIT 14-1) Date:

Time:

Live:

Questions

Test Name:

Answers

Action

What was the incident? (disaster type) What position were you working at during the incident? How much warning did you have? How much damage was there? Which systems were affected? Did the checklist accurately provide all the information needed to deal with the incident? If no, which areas were missed? Were the affected areas recovered within the recovery time objective? Did you have to evacuate? If so, did the checklist accurately guide you to your shelter or designated area? List the recovery options used: Did you have any switching problems? Did you have any database recovery issues? Did you have any interim call handling problems? What control measures could be used in the future to ensure this does not happen again? Comments:

© 1998 by CRC Press LLC

RTMO:

ICH:

Dist. Arch:

Hot Site:

Cold Site:

Other:

© 1998 by CRC Press LLC

• Use own staff • Equipment already setup

• No

• Use own staff

• No

• No

• No • No

Emergency communications for staff (high-volume inbound-outbound IVR)

Call center planning assistance

Crisis management

Full call center testing

• Yes

• Yes

• Yes

• Yes

• Yes

• Yes

• Yes

• Uses best route according to RTO • Very short RTO • Variety of options available • Agents trained on customer system • Easier to implement

• Requires that a small number of calls be handled by an external agency (expert calls)

Note: Where we say “yes” on the chart means that we expect the companies who wish to perform this service should have this capability.

• No

• Yes

• Yes

• Yes

• Reasonably short RTO • Excellent for large volume of simple calls

• Untrained agents on customer calls • Heavy reliance on scripting • Only able to deal with basic call types

• No CTI • Manual systems • Must relocate staff

Advantages

No CTI Manual systems Must relocate staff Hard to test Very long recovery time

• • • • •

Disadvantages

15 to 30 minutes

• RTMO continuing overflow system that facilitates the availability of trained agents off site

• A call center and/or a virtual call center setup • Scripting in place, waiting to receive calls

30 to 45 minutes

Real-Time Minimum Call Overflow

Interim Call Handling

1 to 6 hours

6 hours to 2 days

Recovery Time

• Site with T1s • Equipment set up with computer equipment and basic phone systems • Also includes mobile sites

Hot Site

• Site with T1 lines • No equipment setup

Cold Site

Description of service

Types of Call Center Recovery

EXHIBIT A-45. TYPES OF CALL CENTER RECOVERY (FROM EXHIBIT 15-1)

EXHIBIT A-46. CALL TYPE CHART WITH DIFFICULTY FACTORS (FROM EXHIBIT 15-2) Duration (min) Peak No. of Calls

Type of Call

Difficulty Factors

Order taking Reservations Answering sales calls for manufacturing Answering sales calls for insurance or professional services Answering sales calls for retail Product location inquiries Channeling customer complaints to others for resolution Solving customers’ problems Telemarketing outbound Answering inquires for general information Answering inquires about product service Answering inquires about product use Advising senior management on customer issues Defining customer service Defining customer policy/strategy Public relations Crisis management Market research Quality management Billing Collections Repair/warranty service Outage reporting Contest lines (advertising) Difficulty Levels: (L) Low; (M) Medium; (H) High. Difficulty Factors: 1. 2. 3. 4. 5. 6. 7.

© 1998 by CRC Press LLC

No complexity Services knowledge Features/benefits knowledge Product/services changes Knowledge base Product knowledge Experienced personnel will give seamless service to customers

8. Technology to support 9. Tracking and delivery of information for fulfillment 10. Required understanding and close working relationships with management and company 11. Effective problem solving skills 12. Sales skills/closing techniques 13. IVR or RAN message 14. Other

EXHIBIT A-47. PLANNING YOUR CALL CENTER BUSINESS CONTINUITY SERVICE (FROM EXHIBIT 16-1) Call Center Information Company Information Company Name: ____________________________________________ Address: ___________________________________________________ Address: ___________________________________________________ City: ___________________________________

Province/State: _________________________________

Country: _______________________________

Postal/Zip: _____________________________________

Contact Name: __________________________

Title: __________________________________________

Telephone: _____________________________

Fax: ___________________________________________

Toll Free: _______________________________

E-Mail: ________________________________________

Description of Services the Call Center Can Offer Full call center recovery billed at a ______ rate (service level 80/20) Overflow support billed at a per minute rate service level as available) Call center business impact analysis assistance Call center disaster recovery planning consultation Crisis management assistance Crisis communication (a predictive dialer and IVR capable of contacting or receiving calls from all employees) Basic inbound call handling — see definition of primary call types Single ACD queue associated with a single inbound telephone number with a queue size of 5 Two or more delay announcements associated with wait times in queue Basic data collection based on script developed jointly by ICH and customer Customer call backs Single form database with information gathering, i.e., name, address, etc.

© 1998 by CRC Press LLC

We Are Currently Performing This Service

We Have the Capability to Do This Service

We Cannot Provide This Service

EXHIBIT A-47. PLANNING YOUR CALL CENTER BUSINESS CONTINUITY SERVICE (FROM EXHIBIT 16-1) (continued)

Description of Services the Call Center Can Offer Scheduled delivery of data to customer within predefined time frames in industry standard electronic format Voice-mail/automated attendant Custom delay announcements Enhanced/custom/detailed reporting Skills-based routing Queue size larger than 5 Multilingual capabilities Multiple inbound lines Specialized/custom training packages Multiform or relational database Credit card authorizations (please state which ones) Fulfillment (please state location of warehouse) Real-time interaction with existing customer information systems Addition of routing technology, dedicated circuits (frame relay, point to point, etc.) Originating call center may elect to locate its own dedicated equipment at support call center Dedicated equipment at support call center (own by the support call center) to support specific requirements Shared specialty equipment at support call center to support multiple customers Interconnectivity into existing customer network infrastructure, security issues and advanced network management issues Distributed architecture — facilities in selected agents’ homes with incoming calls prescreened and routed to best agent Direct CSR link — allows for specialized recovery, even from pay or cellular phones Secondary office as call center (do you have backup for your call center?)

© 1998 by CRC Press LLC

We Are Currently Performing This Service

We Have the Capability to Do This Service

We Cannot Provide This Service

EXHIBIT A-48. CALL CENTER GENERAL INFORMATION (FROM EXHIBIT 16-2) Industry Sectors (Please indicate all sectors in which your company provides service.) Credit and Collections

Technical Response

Financial Services

Telemarketing

Government

Telecommunications

Health Care

Transportation

Hospitality

Product Support/Service

Insurance

Utilities

Manufacturing

Wholesale

Retail

Other

Services

Other

Number of Sites: ____________

Hours of Operation: ____________________

Time Zones: ________________

Geographic area covered: _______________

Call Center Profile Calls per year:

% Inbound: ___________ %Outbound: ___________ %IVR: ___________

Calls per day:

% Inbound: ___________ %Outbound: ___________ %IVR: ___________

Peaks:

Seasons: ______________________________________________________ Day of week: __________________________________________________ Time of day: __________________________________________________

Average length of call: ______________ mm:ss

Average handle time: ______________ mm:ss

Objectives: Telephone service factor: _______________________________________ Average speed of answer: _______________________________________ Abandon rate: _________________________________________________ Transfer rate: _________________________________________________ What languages do you offer support for?

______ English ______ French ______ Spanish ______ German

Others ________________________________________________________________________________________

© 1998 by CRC Press LLC

EXHIBIT A-48. CALL CENTER GENERAL INFORMATION (FROM EXHIBIT 16-2) (continued) Recovery Time Capability Recovery Time Capability

Recovery Time Capability — Time of Day

15 minutes

8:00 A.M. to 8:00 P.M.

30 minutes

8:00 P.M. to 8:00 A.M.

60 minutes 2 hours 5 hours

© 1998 by CRC Press LLC

EXHIBIT A-49. LAN NETWORK WORKSHEET (FROM EXHIBIT 16-3) Call Center Systems Do you currently offer: ❑ Skills-based routing? ❑ Zero outs to live agents from IVR?

❑ CTI? ❑ Access to live agents in call center via Internet? ❑ Access to call center via Internet e-mail? ❑ Electronic call recording?

Dynamic?

❑ Multiple queue login? ❑ Interactive voice response applications?

Server Platform

Agent Desktop

❑ Windows NT

PCs or NCs with ❑ Windows 3x

❑ AS400

❑ Windows 95 ❑ Windows NT 4.0 WS

❑ RISC

❑ Emulation Software ❑ UNIX

❑ UNIX

❑ Other

❑ Custom Programs Terminals with ❑ Emulation ❑ Custom Applications

Please include the following if applicable: # Users on network: __________ Applications:

_______________________________________________

Router platform:

_______________________________________________

Network is:

______ LAN

______ WAN

Please attach network diagram if available. Telephony Long Distance Networks AT&T ______ 800 numbers

MCI ______

Sprint ______

Bell ______

Other ______

___________ ___________ ___________

# Local trunks

___________ # ACD queues required __________________________

# DIDs

___________ Custom delay announcements (RAN) _____________ Detailed custom reporting ________________________

© 1998 by CRC Press LLC

Continuous?

EXHIBIT A-50. CSR BOMB THREAT CHECKLIST (FROM EXHIBIT 18-5) 1. Note: • Keep your cool • Keep caller talking • Listen carefully • Log all information 2. Ask: • Bomb location? • When will it explode? • What does it look like? • What is the explosive? • Why was it placed? • Who are you? • Where are you? • Request more information to save lives. 3. Note characteristics of caller: • Sex • Estimated age • Accent • Voice (loud, soft, pitch) • Speech (fast, slow) • Diction — good, nasal, lisp, etc. • Manner — calm, excited, vulgar • Background noise • Does the caller seem familiar with the area? 4. Immediately after caller hangs up, notify: • Immediate supervisor • Site security • Co-workers • Police and other units 5. Note: Supervisor may have personnel conduct search in their immediate area • Sound alarm • Vacate building • If you must evacuate: a. Exit the call using emergency scripting and exit screen b. Report to your predesignated meeting point c. Report to your immediate supervisor that you are present d. Your assembly spot is the coffee shop across the street unless otherwise directed

© 1998 by CRC Press LLC

EXHIBIT A-51. CSR TORNADO WARNING CHECKLIST (FROM EXHIBIT 18-6) 1. Note: • Keep cool • Get location • Listen carefully • Log all information 2. If telephoned in, ask: • Present location • Direction of travel • Speed of travel • Wind speed • Associated weather hazards • Additional information 3. Notify: • Immediate supervisor • Site security • Alert shelter staff (if applicable) • Other sections (computer, operators, etc.) 4. If seen, notify: • Authorities (911) • Immediate supervisor • Shelter staff • Other personnel 5. Seek Cover! Your shelter location in the red door in the first level of the basement If you must evacuate: • Exit the call using emergency scripting and exit screen • Report to your predesignated meeting point • Report to your immediate supervisor that you are present

© 1998 by CRC Press LLC

EXHIBIT A-52. CSR COMMUNICATIONS OUTAGE CHECKLIST (FROM EXHIBIT 18-7) 1. If by telephone, ask: • Problem • Equipment effected • Estimated RTO (if available) • Effect on operations (minor or major) • Log all information, note time received 2. Notify: • Immediate supervisor • Maintenance section • Log information 3. If you detect a problem: • Write down all problems • Note equipment affected • Notify immediate supervisor • Check for any safety concerns • Log all information, including names of personnel notified Remember: Safety first!

© 1998 by CRC Press LLC

EXHIBIT A-53. CSR FIRE CHECKLIST (FROM EXHIBIT 18-8) 1. If you detect the fire: a. sound the alarm by i.

Shouting “Fire, Fire, Fire”

ii. Pulling nearest fire alarm b. Ensure immediate supervisor is aware of the situation c. Report on all details of the alarm to your supervisor 2. Upon hearing a fire alarm: a. Determine if it is an evacuation alarm (in some buildings two short rings indicate not to evacuate but to stand by for further instructions) 3. If you must evacuate: a. Exit the call using emergency scripting and exit screen b. Exit building using nearest stairwell (do not use elevators) c. Report to your predesignated meeting point d. Report to your immediate supervisor that you are present NOTE: Remember: In any fire situation, remain calm and do not panic. 4. Upon returning to your station (only when directed) a. Determine if your station has been damaged b. If no damage is apparent use startup procedure #3 when directed 5. If sent to a hot site, secure a hot site CSR checklist and follow the instructions 6. If sent home, report to phone number 1-888-555-help and await a return call

© 1998 by CRC Press LLC

EXHIBIT A-54. SUPERVISOR FIRE CHECKLIST (FROM EXHIBIT 18-9) 1. If you detect the fire: a. Sound the alarm by i.

Shouting “Fire, Fire, Fire”

ii. Pulling nearest fire alarm a. Ensure immediate supervisor is aware of the situation b. Report on all details of the alarm to your supervisor 2. Upon hearing a Fire alarm: a. Determine if it is an evacuation alarm (in some buildings two short rings indicate not to evacuate but to stand by for further instructions) 3. Upon receiving a fire alarm or report: a. Confirm from call center manager (if unavailable, then make decision) if evacuation is required, if so: i.

Ensure all calls are exited

ii. All your personnel have left the area iii. That the doors are not locked 4. If you must evacuate: a. If you are on a call, exit the call using emergency scripting and exit screen b. Exit building using nearest stairwell (do not use elevators.) c. Report to your predesignated meeting point d. Upon arriving at the predesignated meeting point i.

Ensure all your personnel are accounted for

ii. Dispatch someone to direct the emergency crews to the fire area iii. Report to your call center manager that you are present 5. Ensure all pertinent information has been logged. 6. Report to the Call Center Manager.

© 1998 by CRC Press LLC

EXHIBIT A-55. CALL CENTER MANAGER FIRE CHECKLIST (FROM EXHIBIT 18-10) 1. If you detect the fire: a. Sound the alarm by i.

Shouting “Fire, Fire, Fire”

ii. Pulling nearest fire alarm b. Ensure crisis manager is aware of the situation 2. Upon hearing a fire alarm: a. Determine if it is an evacuation alarm (in some buildings two short rings indicate not to evacuate but to stand by for further instructions) 3. Upon receiving a fire report: a. Determine if evacuation is required b. Initiate interim call handling c. Advise recovery facility to initiate fire checklist d. Take “to go” case e. As you exit, ensure: i.

Doors are not locked

ii. All personnel have left the building f.

If you must evacuate: i.

Exit building using nearest stairwell (do not use elevators)

ii. Report to your predesignated meeting point iii. Report to crisis manager that you are present iv. Ensure all supervisors are present v. Take report from supervisors to ensure everyone is accounted for vii. Ensure all required steps have been taken g. Collect data and logs from supervisors for postincident briefing; complete incident report h. Contact interim call handling company to ensure its checklist is complete Note: Ensure some form of communication is available to talk to the interim call handling facility.

© 1998 by CRC Press LLC

EXHIBIT A-56. INCIDENT REPORT (FROM EXHIBIT 18-11) Date:

Time:

Live:

Questions

Test Name:

Answers

Action

What was the incident? (disaster type) What position were you working at during the incident? How much warning did you have? How much damage was there? Which systems were affected? Did the checklist accurately provide all the information needed to deal with the incident? If no, which areas were missed? Were the affected areas recovered within the recovery time objective? Did you have to evacuate? If so, did the checklist accurately guide you to your shelter or designated area? List the recovery options used: Did you have any switching problems? Did you have any database recovery issues? Did you have any interim call handling problems? What control measures could be used in the future to ensure this does not happen again? Comments:

© 1998 by CRC Press LLC

RTMO:

ICH:

Dist. Arch:

Hot Site:

Cold Site:

Other:

EXHIBIT A-57. IS TECHNICIAN FIRE CHECKLIST (FROM EXHIBIT 18-12) 1. If you detect the fire: a. Alert personnel by i.

Shouting “Fire, Fire, Fire”

ii. Pulling the nearest fire alarm 2. Upon hearing a fire alarm: a. determine from call center manager or supervisor if evacuation is required b. If so, before leaving ensure: i.

Haylon dump is turned to “manual on”

ii. Start emergency data backup to internal system iii. Set doors to unlock c. Exit building by nearest stairwell (Do not use elevators) d. Report to the predesignated meeting point e. Log all pertinent information f.

Report to immediate supervisor or to the call center manager.

© 1998 by CRC Press LLC

EXHIBIT A-58. INTERIM CALL-HANDLING COMPANY FIRE CHECKLIST (FROM EXHIBIT 18-13) 1. Upon receiv-ing a fire call from the call center manager: a. Begin line transfer procedure b. Call AT&T at 1-800-xxx-xxxx c. Identify uses code word xxxx d. Request preprogrammed route # 1 2. Activate call stations 3. Confirm actions to the call center manager at 1-905-xxx-xxxx that transfer has taken place 4. Ensure all pertinent information has been logged for use at the postincident debriefing

© 1998 by CRC Press LLC

EXHIBIT A-59. SOLUTIONS TO NATURAL DISASTER (ANNOTATED) (FROM EXHIBIT 19-1) Disaster type

Problems

Solutions

Hurricane or typhoon

Evacuation, damage to buildings

Fortified shelters, glass-free eoc and emergency call center, passive IVR hotline

Internal flooding

Electrical, pbx, computers, comfort

Tarps to cover electrical equipment, vendor service-level agreements including furniture

Wind damage

Roof, fallen trees, injury from debris

Fortified shelter, tarps until roof repaired, chainsaws or contractor, advanced first aid kit

External flooding

Damage to roads, destruction of furniture and paper files

Buses transport employees to workplace, contractors repair or replace furniture and dry of paper files

Epidemic

Illness or death of key personnel, nonfunctional workers

Cross training, replacement workers, distant ich facility

Earthquake, seismic damage

Collapse or condemnation of buildings

Distant ich and hot site facility, real estate buyer’s agent finds alternate site

Snow and ice storms

Electricity and telephony, impassable roads

Distant ICH, snow removal, check employees have heat and electricity

Volcanic eruption

Evacuation, ash, fumes, fire and lava, earthquake, thrill seekers

Evacuation!!! ICH, relocate to distant alternate site, salvage equipment if undamaged, crowd control, safety of families

Tsunami (tidal wave)

Evacuation, warned or unwarned. Files, building damage

Treat as aftermath of earthquake, evacuate inland to high ground, ICH, alternate site

Electromagnetic interference

Power bumps, corrupted data, solar flares

Multiple backups, distant off-site storage, protective magnetic storage media

© 1998 by CRC Press LLC

EXHIBIT A-60. SOLUTIONS TO INFRASTRUCTURE DISASTERS (ANNOTATED) (FROM EXHIBIT 19-2) Disaster

Problems

Solutions

Hardware failure

Stoppage of data processing or communications

Redundant equipment, ICH facility

Internal fire

Evacuation, smoke and water damage

Evacuation, ICH facility, vendor repair and replacement

Software failure

Confusion, probably cannot backup current data

ICH facility, uses “last backup” data

External fire

Road blocks, evacuation, smoke

Evacuation, passive IVR hotline, ICH facility

Power failure

Equipment unusable, no lights

ICH facility, get power restoration estimate from electrical utility

Facility structural failure

Personnel to rescue, denial of access

Heating, ventilation, or airconditioning failure

Discomfort, failure of equipment

Explosion, water service interruption

No cooling, no fire sprinklers, denial of access by firefighters

Loss of physical access to resources

Closure of building or area by emergency or government officials, doors locked during dispute

Gas interruption

Lack of heating

Malfunction or failure of CPU

Server cannot drive ACD or IVR

Gas leaks

Illness, evacuation, risk of explosion

Telecommunications failure

Wide-area calls cease, cellular phone to back-up

Activate ICH by cellular phone

Chemical exposure

Illness, rescue and evacuation, denial of access

Evacuation, decontamination, ICH facility, hot site, restoration of primary site possibly lengthy

Communications failure internal/external

PBX, trunk line

Transportation blockage or failure

Relieving shift delayed

© 1998 by CRC Press LLC

Find alternate routes, ICH facility to allow leaving shift to depart

EXHIBIT A-61. SOLUTIONS TO MAN-MADE DISASTERS (ANNOTATED) (FROM EXHIBIT 19-3) Disaster

Problems

Solutions

External criminal activity

Extortion, theft, blackmail, police investigation

Security team on alert, cooperation with authorities, inventory of equipment, cash, and company credit cards

Civil disorder/riot

Assault, vandalism, looting

Switch to ICH facility, lock file cabinets, and safes, evacuate personnel

Disgruntled employee

Vandalism, theft, violence

Security team on alert, call police, deny access, if necessary switch to ICH and evacuate

Nuclear attack

Evacuation, denial of access, electromagnetic pulse

Distant ICH, if time seek distant evacuation site, otherwise use protected EOC or basements

Strikes

Picketing, violence, intimidation

RTMO or ICH facilities, security team on alert, escorts for arriving/departing shifts, call police

Chemical spill/toxic waste spill

Illness, evacuation, denial of access, EPA investigation

Data entry error

Sabotage of information

Switch to ICH facility “last backup” data

Kidnapping/death of key personnel

Confusion, delegation, trauma, police or FBI investigation*

Deputy and alternate personnel take over duties, alert security team, report any ransom demands, cooperate with police

Improper handling of sensitive data

Loss or compromise, lawsuit

Consult lawyer, public relations team, do not cover up

Vandalism/sabotage

Damage to equipment or files

Copy from off-site duplicate files, insurance claim, replace equipment

Unauthorized physical access

Theft, violence, sabotage

In progress: enact plan, after fact: repair or replace

War

Evacuation, failure of infrastructure, violence, flight of personnel, imposition of martial law

Plan for multiple sites, safety of personnel, relocation to alternate sites, hardened shelters

Malicious damage or destruction of software or data

Confusion, hacker, virus

Multiple backups, antivirus software, off-site storage of “last backup,” hardware and software vendor service-level agreements

© 1998 by CRC Press LLC

EXHIBIT A-61. SOLUTIONS TO MAN-MADE DISASTERS (ANNOTATED) (continued) (FROM EXHIBIT 19-3) Disaster

Problems

Solutions

Biological contamination

Disease, quarantine, evacuation, denial of access, WHO or Centers for Disease Control

Distant ICH facility, replacement workers from out of area, emergency banking for evacuated personnel

Unauthorized access to data or theft of data

Commercial Espionage

Radiation contamination

Corruption of data, quarantine, evacuation, illness, denial of access, government investigation

Robbery, theft, burglary

Replacement, police investigation, insurance claim

Vehicle crash

Road rage, car theft, injuries or trauma to personnel

Deputy and alternate personnel take over duties

Bomb threats

Hoax or real? Embarrassment, damage, injury, evacuation, police investigation

Treat as real, alert security and police, evacuation and ICH

Airport proximity

Plane Crash due to Hijacking

Danger of fire, enact plan, rescue employees or passengers

* RCMP in Canada.

© 1998 by CRC Press LLC

EXHIBIT A-62. SOLUTIONS TO ACCIDENTAL DISASTERS (ANNOTATED) (FROM EXHIBIT 19-4) Disaster

Problems

Solutions

Monday effect

Lack of coordination, grouchiness, fuzzy thinking

More coffee! Advise more sleep, exercise, snack for energy

Fatigue

Slow reaction time, lack of motivation, errors of judgment, irritability, stumbling and dropping things

Give employee break time, advise snack for energy, during crisis: EOC cot

Illness

Danger of infection, fever delirium, effects similar to fatigue, memory loss, collapse

Send to doctor or hospital, if severe, call ambulance

Carelessness caused by fatigue or illness

Filing errors, haste, forgetfulness, “diskette in the microwave” mental processing errors, incomplete notes

Supervisor double-check work, break time, doctor, or hospital, in crisis, EOC cot

Carelessness caused by poor attitude

Habitual sloppiness

Counsel, reassign, or transfer away

Carelessness caused by stress

Anxiety and forgetfulness due to family event, new position, crisis

Counsel, supervisor double-check work, meditation or distraction, in crisis, give break or snack if possible

Internal accident

Visiting child or pet, spills, corridor collisions, stairway falls

Keep employees focused, except security, paramedic, or maintenance

External accident

Vehicle crashes through wall, lost truck load blocks exits

Enact plan, partial handover to ICH or RTMO facility

© 1998 by CRC Press LLC

EXHIBIT A-63. PLANNED INPUTS ACTIVITY DEPENDENCY (FROM EXHIBIT 20-1)

Activity Dependency

Input Source

Result (Expected Action)

Activity Number

Time

CSR

1

9:00

Reports severe weather warning heard on her way into work

Must have turned on radio

Radio

2

9:30

Radio alert that a tornado watch has been put into effect

Must have turned on radio

Radio

3

10:00

Radio alert that a tornado watch has been put into effect

(Call center manager should evacuate personnel)

4

10:15

Power failure

(Call center manager should implement power failure procedures) Evacuate personnel

Activity

(Call center manager should turn on radio per checklist)

Must have turned on radio with batteries

Radio

5

10:23

Tornado is two miles south, southwest of call center

Evacuated when radio warning was given, radio with batteries was on

Radio

6

10:24

All personnel take their checklists and go to the tornado, hurricane shelters

Took “to go” case

In shelter

7

10:28

Begin to activate call center disaster recovery plan by cellular phone

Manager instructed evacuation

Windows blown out of call center

8

10:33

All personnel in shelter when tornado strikes call center

No injuries

Sirens

9

10:30

Evacuate personnel

Sirens did not give enough warning to evacuate all personnel

Windows blown out of call center

10

10:33

Tornado strikes call center, sirens did not give enough warning to evacuate all personnel

5% of personnel are injured or dead

Evacuated with sirens or waited for other warning to evacuate.

© 1998 by CRC Press LLC

EXHIBIT A-63. PLANNED INPUTS ACTIVITY DEPENDENCY (continued) (FROM EXHIBIT 20-1)

Activity Dependency Did not evacuate

Took “to go” case into the tornado shelter

© 1998 by CRC Press LLC

Input Source Windows blown out of call center

Activity Number

Time

11

10:33

Tornado strikes call center

12

10:50

All calls recovered to other locations according to disaster plan

Activity

Result (Expected Action) 63% of personnel are injured, 15% are in critical condition, and 2% are dead

EXHIBIT A-64. MULTIPLE PROBLEM EXERCISE DEPENDENCIES (FROM EXHIBIT 20-3) Activity Dependency

Input Source

Activity Number

Time

Activity

Result

Fire alarm system

1

12:45

Two short rings for your fire alarm warning

(Get checklist and send security with a two-way radio to find location of fire on your fire alarm system)

If they checked fire alarm system

Cafeteria alarm

2

12:48

Fire in cafeteria

(Send security personnel with a two-way radio to find out if there is a fire in that location; make sure you have what you need to leave the building)

If they checked fire alarm system

Security personnel

3

12:54

Toaster fire small

(Used fire extinguisher, fire is put out)

If they checked fire alarm system and security has two way radio

Message from security over two-way radio

4

1:05

Fire is out; fire department called and notified; all clear.

If did not check fire alarm and security has a two-way radio

Radio in from security over two-way radio

5

1:05

Toaster fire in kitchen but an employee tried to use a pot of water to put it out; fire is spreading; one injured employee with electrical and fire burns

Phone call to security desk

6

1:08

Call says, “By now you have detected a fire in your kitchen, so you know I am serious. I have placed a bomb in your building. If anyone leaves, it goes off. If my demands are not met, it goes off. I want $100,000.…”

(Expected Action)

© 1998 by CRC Press LLC

(Write all information down. Keep caller on the phone as long as possible; start evacuation)

EXHIBIT A-64. MULTIPLE PROBLEM EXERCISE DEPENDENCIES (continued) (FROM EXHIBIT 20-3) Activity Dependency

Input Source

Activity Number

Time

Activity

Result

Kept caller on the line

Bomb threat call ends

12

1:10

Caller

Did not evacuate or did not pick up the phone call

Boom

7

1:12

You are all dead or injured

If did not check fire alarms and security does not have two-way radio

Security is leaving with injured person

8

1:15

Fire detected in kitchen, one person seriously injured

Call 911 for fire department and an ambulance for the injured worker, evacuation

If no two-way radio but checked fire alarm system

Security person

9

1:20

Fire is out at this time; no injuries reported.

Call fire department and tell them the situation

Fire suppression system activated in kitchen, sirens sound

10

1:22

Fire is out; fire alarm in full effect

People are evacuating on their own

If did not try to keep caller on the line and fire alarm went off and people evacuated

Bomb threat call ends; but a quick call back is received

11

1:25

“I said no one is to leave the building.” BOOM!! Some people get out, most are dead or injured including you

Kept caller on line, evacuation ordered, started call volume DR plan

Boom

13

1:40

Everyone is safe, call center is destroyed, calls are being recovered

© 1998 by CRC Press LLC

You have written down the information; ordered evacuation and start of switching call volume in accordance with DR plan; (some one else) by passing a note to security with some details

EXHIBIT A-64. MULTIPLE PROBLEM EXERCISE DEPENDENCIES (continued) (FROM EXHIBIT 20-3) Activity Dependency Fire was put out, kept caller on line, evacuation ordered, started call volume DR plan

© 1998 by CRC Press LLC

Input Source

Activity Number

Time

14

2:55

Activity No injuries; all calls recovered to other locations according to disaster plan

Result

EXHIBIT A-65. TESTING THE CONTINUITY FUND (FROM EXHIBIT 21-1) Financial Instrument Letters of credit Signing checks Company credit cards Company debit cards Cash and security

© 1998 by CRC Press LLC

In “To Go” Case

Checklists/Contact Lists

EXHIBIT A-66. TESTING VENDOR RESPONSE (FROM EXHIBIT 21-2) Test Objective Contact information is current Contact outside of office hours Contact if vendor is forewarned (e.g. Rising floodwaters) Contact if vendor is surprised (e.g. Earthquake) Checking credit arrangements

© 1998 by CRC Press LLC

Response (Good or Needs Upgrading)

EXHIBIT A-67. TESTING INSURANCE (FROM EXHIBIT 21-3) Test Objective Broker contact information is current Policy provisions, riders, and chattels inventory is current Would broker front company emergency money? “smoking hole” test: would broker pay replacement value of premises How adaptable is insurer to unusual situations? (like testing!) Recovery team accounting representatives liaison Documentation backups are current and safe off-site, local and distant

© 1998 by CRC Press LLC

Result (Good or Needs Upgrading)

EXHIBIT A-68. TESTING CONNECTIVITY (FROM EXHIBIT 21-4) Test Objective Cold site (Telco trunk line) ICH facility RTMO facility Hot site (assisting call center) Virtual call center (telecommuters) Your office coordinates telecommuter calls Assisting call center coordinates telecommuter calls Cellular telephone and radiotelephone Fax and e-mail Video- and audioconferencing Toll-free contact with assisting call center Toll-free contact with third-party crisis manager Toll-free contact with IVR situation hotline

© 1998 by CRC Press LLC

Result (Good or Needs Upgrading)

EXHIBITA-69. POST-TEST CONFERENCE (FROM EXHIBIT 21-5) Questions to Assess Does the continuity plan function as a whole? Did the part of plan tested work? Did critical business functions suffer? Did logistics work as planned? Was backup data usable? Did handover of call volume take place as planned? Did cold/warm/hot site work as planned? Did virtual call center work as planned? Did vendor arrangements work as planned? Were personnel needs supplied as planned? Did company meet contractual, legal, or audit requirements? Did people react confidently or show confusion?

© 1998 by CRC Press LLC

Yes/No

Remarks

Appendix C A Wake-Up Call for the Call Center Industry THE STORM The approach of Hurricane Andrew was not unexpected. Hurricane Andrew formed from a tropical wave that crossed from the west coast of Africa to the tropical North Atlantic Ocean on August 14, 1992, moving westward. As early as Friday, August 21, 1992, Florida and Gulf Coast businesses knew that the still distant storm was heading toward Miami. Although many companies backed up their computer databases, it was pretty much a normal business day. On Saturday, August 22, Florida tour operators were taking telephone reservations in their call centers as normal. As the hurricane came nearer, many businesses brought out their “hurricane disaster” plans. (The primary disaster concern in Florida is a hurricane.) Some businesses found that their plans were only rudimentary “plans for plans” which had been started at some earlier date, but not completed. Some companies found that these rudimentary plans consisted of separate memos kept in binders in various departmental offices. A hurricane watch was issued at 9 P.M. on August 22, an “official” indication that a hurricane could occur, for the Florida east coast from Titusville (east of Orlando), southward through the Florida Keys including the Dry Tortugas, a group of islands 68 nautical miles west of Key West. A Hurricane Warning was in effect for the Northwest Bahamas, from Andros and Eleuthera Islands northward through Grand Bahama and Great Abaco. At 00:01 A.M., on Sunday, August 23, 1992, the winds of Hurricane Andrew swirled at approximately 110 miles (225 kilometers) per hour at twenty-five degrees north latitude. This location was about 360 miles or 600 km north of the Dominican Republic, and about 550 miles or 900 kilometers east of the Bahamas. At noon a hurricane warning was issued, for the Florida east coast from Vero Beach southward through the Florida Keys to the Dry Tortugas including Florida Bay. This was an indication that the hurricane was expected to hit Florida within 24 hours. During the day, various businesses started to implement their disaster plans, but were hampered by the fact that it was a Sunday. Some companies that leased their premises had trouble getting into their buildings to secure important documents, power down computer systems, and so on. Some companies were informed by their landlords that they would not have access to their premises after a specified hour, probably a sign that the landlords had their own disaster plans to implement. Also at noon on August 23, a Tropical Storm Warning was issued for the Florida east coast, north of Vero Beach to Titusville, and a Hurricane Watch was issued for the Florida west coast, south of Bayport, including the greater Tampa area to north of Flamingo. Floridians experienced with “hurricane season” prepared their homes for the storm. Batteries for emergency radios were checked, canned food and containers of water were gathered. Windows were reinforced with masking tape or plywood. At 6 P.M. on August 23, a Hurricane Warning was issued for the Florida west coast south of Venice and Lake Okeechobee. A Tropical Storm Warning was issued for the west coast of Florida, north of Venice to Bayport.

© 1998 by CRC Press LLC

Many pondered the question, to evacuate or not. Evacuation routes were studied, car fuel tanks were filled, and large amounts of cash were withdrawn from bank ATMs. Those hoping to flee were better off if they left early, and they phoned hotels as far away as Georgia for reservations. If they had not left as soon as the hurricane watch was issued, the chances were that they would be trapped in gridlock. Road traffic might move as slowly as five miles per hour (about 10 kilometers per hour), fleeing a storm that might advance as fast as 25 miles per hour, or 40 kilometers per hour. Those without the resources to flee far away, immediately, had to review the locations of local Red Cross shelters. Those making a personal evacuation decision might or might not have known that hurricane evacuations end up as false alarms in five out of six trips. Also, there was the possibility that anyone fleeing might drive into the path of the storm, rather than out of it. And, what if the storm veered away from the forecast path? Officials ordered massive evacuations in Florida and Louisiana as the likelihood of Andrew making landfall in those regions increased. About 55,000 people left the Florida Keys. Evacuations were ordered for 517,000 people in Dade County, 300,000 in Broward County, 315,000 in Palm Beach County, and 15,000 in St. Lucie County. For counties farther west in Florida, Collier (population 25,000), Glades (4,000), and Lee (2,500) also evacuated more than 1,000 people each. Andrew was a category 4 hurricane (on the Saffir/Simpson Hurricane Scale) when its eye passed over northern Eleuthera Island in the Bahamas late on Sunday. It passed over the Bahamian southern Berry Islands early on Monday, August 24. Between midnight and 1 A.M. on Monday, August 24, Hurricane Andrew was carrying winds of approximately 140 miles per hour. Meteorologists estimated that the air pressure at the center of the storm was 922 mb (millibars of mercury) when it made landfall near Homestead Air Force Base, Florida, at 5:05 A.M. EDT. Homestead and Florida City are located 19 miles (31 kilometers) south of downtown Miami. At that time, the winds roared at 140 miles per hour (225 kilometers per hour), with gusts up to 168 miles per hour or 270 kilometers. The winds might even have intensified once Andrew was over land instead of water. Some records estimate that the wind speed ranged from 145 miles per hour to 175 miles per hour. Differences in measurements taken at various localities inland were because of variations in the airflow due to trees, buildings, and other obstacles. In some places there was drag that reduced the wind speed. Elsewhere, there were brief, local accelerations of the wind immediately next to structures such as buildings. The wind speed around a house in the eye of the hurricane can vary around the structure. The end result, no matter what the wind speed: very little sleep!!! Some homes were completely destroyed; others were seriously damaged. Even at the height of the storm, both professional and amateur weather watchers continued to record wind speed and barometric pressure. This information would subsequently show wind speed highs and air pressure lows of awe-inspiring magnitude. Two amateurs, a sister and brother who lived about a quarter mile apart, recorded minimum barometric pressure at their homes of 921 and 923 mb, respectively. (After the hurricane, these barometers were tested by officials and found to be reliable.) The sister’s home had been built in 1945 with 22-inch-thick concrete and coral rock walls. Some of the windows broke, but the hurricane-proof walls held steady. The strongest gust was reported from the northern eye wall of the hurricane, a little more than a mile from the shoreline. A homeowner recorded a gust of 184 knots (about 211 miles per hour) moments before parts of a windward wall of his house failed. The hurricane also destroyed the anemometer that he had been using. Damage at that location was much less than the damage to similar buildings only two miles south of his neighborhood, meaning that apparently even stronger winds blew than those that were recorded.

© 1998 by CRC Press LLC

During the morning hours of Monday, August 24, Hurricane Andrew generated “storm surge” along shorelines of southern Florida. On the southeast Florida coast, the peak of the storm surge arrived at about the time of high astronomical tide. The height of the storm tide (the sum of the storm surge and astronomical tide, relative to mean sea level) ranged from four to six feet (up to two meters) over the normal level of high tide at the northern and southern sides of Miami’s Biscayne Bay. The watery surge increased to a maximum of 16.9 feet (about 5.6 meters) at the Burger King International Headquarters, located on the western shoreline in the center of the bay, in the southern end of metropolitan Miami. The storm tide near Flamingo, on the tip of the Florida southwest coast, was about four to five feet, a little less than two meters. The storm surge rose an extra six to seven foot (about two meters) near Goodland, which is situated about 20 miles south of Naples, Florida. Andrew moved nearly due westward when over land and crossed the extreme southern portion of the Florida peninsula in about four hours. During that time, the hurricane damaged roads, power lines, water and sewer services, and severely damaged communication systems. The greatest damage occurred in South Dade County, and Homestead Air Force Base was completely destroyed. At 1 P.M. on Monday, August 24, the Hurricane Warning for the western Bahamas and Florida, except for Lake Okeechobee and the west coast south of Venice to Flamingo, was discontinued. The Hurricane Watch and Tropical Storm Warning for the Florida east and west coasts were discontinued. At that time, a Hurricane Watch was issued for portions of the northern Gulf Coast from Mobile, Alabama to Sabine Pass, Texas. At 6 P.M. the Hurricane Warning for the rest of Florida was discontinued. Andrew reached the north-central Gulf of Mexico and turned toward the northwest. The hurricane struck a sparsely populated section of the south-central Louisiana coast with category 3 intensity on Wednesday, August 26. Storm tides in Louisiana were at least eight feet (about 2.6 meters) and caused flooding. In Louisiana, a tornado occurred in the city of Laplace, several hours prior to Andrew’s landfall, which killed two people and injured thirty-two others. Other tornadoes were seen but reportedly did not cause casualties. Andrew weakened rapidly after landfall, to tropical storm strength in about 10 hours and to depression status 12 hours later. The cyclone moved northward and then turned northeastward, producing heavy rain of more than 10 inches near its track. Several damaging tornadoes in Georgia late on Thursday, August 27, were part of Andrew’s remnant. Rainfall totals in excess of seven inches were recorded in southeast Florida, Louisiana, and Mississippi. Twenty-six deaths in Florida were directly attributed to Hurricane Andrew.

Returning to Work: Anecdotes Anyone living and working in Dade County, Florida, faced unprecedented challenges in the aftermath of Hurricane Andrew. There was the emotional battering from a night of howling wind. Twenty-six people were dead. Thousands of people were homeless, seeking the necessities of life from the Red Cross. Those whose homes were not destroyed had significant damage to repair. On Tuesday, August 25, Andrew left the Florida landmass and headed out into the Gulf toward Louisiana. Wet weather continued for several days. Offices even to the northwest of Miami (not hit by the maximum effect of the storm) were without electrical power until Wednesday, August 26. Lack of electricity made the computers and STMs of the banks inoperative. Numerous buildings were destroyed or damaged. Various roads were destroyed or blocked by fallen trees. Some office buildings had electricity, but no water, because of water mains destroyed by the roots of storm-felled trees. This meant that not only was there no water for drinking or for washroom flushing, but there was no water for air-conditioning systems using water,

© 1998 by CRC Press LLC

water-based coolant systems for computers, or building fire-sprinkler systems. Many buildings were closed by the fire department because (since the mains were broken) large areas had no usable fire hydrants. Lack of water also meant that restaurants could not cook, and grocery stores and other retailers could not clean. Many employees of large companies who were able to get into work found the corporate offices in a shambles. Windows were blown out; interior walls were destroyed. Whole desks had been pushed out through the walls of the building, filing cabinets were destroyed, and wet papers from various files were scattered like autumn leaves over as much as half a mile of the nearby city. Any recent information that might have been kept on desktops had been blown away. Even those companies so fortunate as to have intact buildings suffered serious absenteeism of employees. The population of the whole area had suffered emotional trauma. Some roads were impassable. Many people could not leave their residences unattended (even those that were only partially damaged) until they had been inspected and declared safe for habitation. Some people could not leave their children, since their normal babysitters or day care centers were evacuated, or destroyed, or out of communication. Even pets were a concern: if the people were homeless, so were their animals. By Thursday, August 27, relief efforts were in full swing. Whole tent cities had been created to house the homeless. The military (XVIII Airborne Corps from Fort Bragg, and even some volunteers from the Canadian Armed Forces) arrived. Water was trucked into the area to supplement contaminated supplies. Debris from felled trees and destroyed buildings was removed. The soldiers repaired school buildings and ran a relief supply distribution center. They also “rendered aid to the local government” by providing assistance to local uniformed personnel such as police and firefighters, who even with this help were working many hours of overtime. The military disaster relief effort would not wind down until October 21, 1992, a little less than two months after the hurricane struck. Multilingual information kiosks were erected throughout Dade County. Installed in high-traffic areas, the kiosks displayed information provided in English and Spanish by the local newspapers, the Red Cross, the Federal Emergency Management Administration (FEMA), and local government agencies. Hurricane victims could find out about emergency shelters, contacting their insurance agents, finding reputable contractors, and other important details. Each computerized kiosk had a printer to provide hard copies of the information. Some businesses, because to luck (location on a part of the coast not hit by the storm) or design (prior arrangement of electrical power generators and backup equipment), were able to return to seminormal call center operations on Tuesday, August 25. The saying, “to the swift go the spoils,” would certainly apply in this case. If their competitors were unable to open or operate their call centers, these companies stood to scoop millions of dollars worth of orders from them. These extra “disaster” orders were in addition to the level of business that they normally would transact. Other companies endured call center outages lasting for five days or more. The millions of dollars of orders lost to competitors with better business continuity plans (“disaster recovery”) was in addition to losses due to destruction of their buildings, equipment, files, and computers. In some cases, telecommunication lines had been seriously damaged by the storm. Some firms were handicapped by the inability of technical personnel to travel from their homes in devastated areas. This left the powering-up or repair of their hardware to other employees who were not as experienced. If the less-experienced technicians accidentally damaged computer equipment in the course of powering it up, then IBM or other technology service assistance was required — all of which would cost both time and money. Some companies received insurance settlements promptly, but many had to wait while the overloaded insurance adjusters worked through their very long list of claims. Some

© 1998 by CRC Press LLC

firms were compelled to take their insurance firms to arbitration to seek settlement of their claims. Many companies had to deal with employee turnover of as much as 60% in the aftermath of the hurricane. Those construction companies still able to operate had a great deal of work to do. Other out-of-area companies came into handle the overload. Due to Hurricane Andrew and also Hurricane Iniki in Hawaii, nine insurance companies went bankrupt in 1992. Homeowners, many of whom were uninsured, sued at least one condominium builder.

Why Planning Is Essential: The Costs Associated with Wide-Area Disaster Hurricane Andrew, which swept through Florida and the Gulf Coast in 1992, was a pointed reminder that humanity’s dominance of this planet is imperfect. Although some disaster recovery plans had been made, the scope of the disaster rendered some of those plans useless, and demonstrated serious weaknesses in many others. If Hurricane Andrew had been a slightly larger system, or one making landfall just a few nautical miles farther to the north, the situation would have been catastrophic for heavily populated, highly commercialized, and similarly vulnerable areas to the north. Only slightly up the coast, downtown Miami, Miami Beach, Key Biscayne, and Fort Lauderdale, if Andrew had devastated them, would probably have had a much higher death toll and damage cost. Fortunately Andrew also swung through a sparsely populated area of Louisiana, and left the New Orleans region more or less untouched. Floridians filed more than 725,000 insurance claims related to Andrew. Included were business “time-element” losses, living expenses, and “business interruption” costs. FEMA estimated that $0.06 billion was spent for school repairs, in a February 1993 report. Subsequent accounting of direct and indirect deaths, and cost of damage in billions of dollars, would show the following: Florida as a whole suffered fifteen deaths caused directly by the hurricane, and twenty-nine indirect, and incurred $25 billion (U.S.) of damage. Dade County bore the brunt of this, suffering all fifteen of the direct hurricane deaths, twenty-five of the indirect, and most of the damage. Broward County suffered three of the indirect deaths, and Monroe County suffered one indirect death. Together they each incurred about $100,000 million damage. Collier County was estimated to have incurred about $30,000 million in damage. Ultimately more than one-half of the fatalities would prove to be indirect. Additional indirect loss of life would bring the death toll to sixty-five. Many of the indirect deaths occurred during the “recovery phase” immediately following Andrew’s passage, when the stress load from destruction of homes and business premises was be at its peak. From the Kendall district southward through Homestead and Florida City to near Key Largo, Andrew reportedly destroyed 25,524 homes and damaged 101,241 others. It was reported that 90% of all mobile homes in south Dade County were totally destroyed. In Homestead, more than 99% (1167 of 1176) of all mobile homes were completely destroyed. Homestead Air Force Base was so completely devastated that at first it was condemned to be abandoned. Subsequently, it was refurbished and reopened as an aid to the local economy. The damage to Louisiana was estimated at $1 billion. Damage in the Bahamas was estimated at $0.25 billion (U.S.). The devastation was not limited to land-based businesses. Damage in the Gulf of Mexico was estimated at $0.5 billion.

© 1998 by CRC Press LLC

“Wake-Up” Hurricane Andrew is said to have issued a “wake-up call” to business and government planners. This was because previous to this event, disaster recovery plans had focused mainly on data recovery. The impact of a wide-area disaster upon communications (voice recovery) and also upon the safety and morale of employees had not been taken into account. The number of businesses that failed because of Hurricane Andrew is not known. It is, however, known that inoperative call centers of tour operators, banks, and other industries lost millions of dollars worth of calls — per day. Not every disaster that strikes a company, city, or region will be a hurricane. But every disaster unplanned for has the potential to disrupt and even destroy both lives and livelihoods.

© 1998 by CRC Press LLC

Appendix B

T

he contact information that we have given here is not an exhaustive list. As noted previously, because of concerns for their privacy we disguised the details of the locations and industries of some companies. In the “Information Age,” we live in a sea of data. It is possible that some individuals or companies are not named here, and expected that they would be. If we have forgotten to include a source of information, or if we cited an anecdote or observation in the text but have forgotten where we heard it, we apologize. We have listed companies and individuals alphabetically.

MAJOR SOURCES David A. Johnson, FBCI, CBCP Canadian Center for Emergency Preparedness P.O. Box 2911 Hamilton, Ontario L8N 3R5 Telephone: Facsimile: E-mail: Web site:

800-965-4608 or 905-520-4116 905-546-2340 [email protected] http://www.nas.net/ccep

Jay G. Bender, CBCP Executive Director (or Karl Muller, Co-Executive Director, DRI Canada) Disaster Recovery Institute International 1810 Craig Road, Suite 125 St. Louis, MO 63146 Telephone: 314-434-2272, in Canada 905-546-2867 Facsimile: 314-434-1260, in Canada 905-546-2340 • DRI International offers certification courses for contingency planning professionals throughout North America.

Steve Lewis Disaster Recovery Yellow Pages The Systems Audit Group, Inc. 25 Ellison Road Newton, MA 02159 Telephone: 617-332-3496 Facsimile: 617-332-4358

© 1998 by CRC Press LLC

FEMA, Federal Emergency Management Agency 500 C Street, SW Washington, D.C. 20472 Telephone: 202-646-2500 Speech: “Federal Response Planning: Maintaining Communications in a Disaster” by Glenn Garcelon, Manager of the Chief National Teleregistration Center, Denton, TX on Tuesday May 20, 1997 at the IQPC Conference “Disaster Recovery Planning for Call Centers in Atlanta, GA” Publication: Emergency Management Guide for Business & Industry (FEMA 141, Oct. 1993) Web site: http://www.fema.gov • Information about IQPC is located below.

Mary Kay Morgan Product Manager — Voice Business Recovery Services Integrated Systems Solutions Corporation (An IBM Subsidiary) 300 Long Meadow Road Sterling Forest, NY 10979 Telephone: 914-759-4700 Facsimile: 914-759-4710 • Ms. Morgan speaks frequently at IQPC, IBM and other conferences.

Gregg Williams Interactive Quality Services, Inc. (IQS) 1200 Main Street Hopkins, MN 55343-7521 Telephone: Facsimile: E-mail: Web site:

612-932-0211 612-932-0214 [email protected] http://www.iq-services.com

• Mr. Williams speaks frequently at IQPC and other conferences.

Gary W. Via Senior Consultant Lucent Technologies, Inc. (Bell Labs Innovations) 4850 D. Thales Road Winston-Salem, NC 27104 Telephone: 910-760-2242 Facsimile: 910-760-0437 E-mail: [email protected] • Mr. Via speaks frequently at IQPC and other conferences.

© 1998 by CRC Press LLC

Tim Ging PNC Bank USX Tower 36th Floor 600 Grant Street Pittsburgh, PA 15219 Telephone: 800-762-2265 or 412-762-2614 Facsimile: 412-762-0726 • Mr. Ging has spoken at conferences such as the IBM BRS Summit of 1997, and is the originator of the “Six Principles of Emergency Communications.”

Jeff Fried Director of Product Management Teloquent Communications 4 Federal Street Billerica, MA 01821-3594 Telephone: Facsimile: E-mail: Web site:

800-468-6434 or 508-436-2644 508-663-7543 [email protected] http://www.teloquent.com

• Mr. Fried speaks frequently at IQPC and other conferences.

James L. Kennedy CEO Twenty First Century Communications, Inc. 760 Northlawn Dr., Suite 200 Columbus, OH 43214 Telephone: 1-800-382-8356 or 614-442-1215 Facsimile: 614-442-1180 • Our thanks again to Jim Kennedy for allowing us to use information relating to TFCC assistance to Central Maine Power during Ice Storm ‘98.

MINOR SOURCES Devlin, Edward S., Emerson, Cole H., and Wrobel, Leo A., Jr., Business Resumption Planning, Auerbach Publications, New York, 1996/97. Numerous articles, Disaster Recovery Journal, P.O. Box 510110 St. Louis, MO 63151 Telephone: Facsimile: E-mail: Web site:

© 1998 by CRC Press LLC

314-894-0276 314-894-7474 [email protected] http://www.drj.com

Ken Scala Conference Director International Quality & Productivity Center 150 Clove Road P.O. Box 401 Little Falls, NJ 07424-0401 Telephone: Facsimile: E-mail: Web site:

800-303-9160 or 800-882-8684 973-256-4370 [email protected] http://www.iqpc.com

• IQPC is not a direct source, but has sponsored many of the seminars and workshops where we met representatives of various companies. Their conferences are well produced and are held throughout North America. The speakers are expert and informative. We found the anecdotes from attendees to be very informative (to us), as well.

John Klassen Vice President OEB International 10 Lower Spadina Avenue, Suite 500 Toronto, Ontario, Canada M5V 2Z2 Telephone: 416-260-6000 Facsimile: 416-260-2708 E-mail: [email protected] • OEB International has offices around the world. The company has been providing public relations counsel since 1946.

© 1998 by CRC Press LLC

Appendix D Glossary Agents ACD BIA Business continuity Cold site CSRs CTI CVM Demarcation line DR plan Drop-ship EOC Fulfillment Hot site HQ ICH Inbound calls Infrastructure failure IPO IVR LAN Outbound calls Outsourcer PBX Predictive dialer R/A Reciprocal agreement RTML RTO Service level T1 UPS Vendors Virtual call centers Voice recovery WAN Wide-area disaster

© 1998 by CRC Press LLC

“Live” as opposed to automated Automatic call distribution (newer equipment replaces PBX) Business impact analysis Process of ensuring a business will continue operations Vacant alternate call center facility Customer service representatives Computer telephony integration Call volume management Point to which the Telco provides telephone service Disaster recovery plan Quick ship programs Emergency operations center Shipment of products ordered by customers who call Alternate call center facility having power and telephony online Headquarters (head office) Interim call handling Calls being received from parties calling in Failure of electrical power, telephone, water/sewer, or road grids Initial public offering of company shares Interactive voice response Local area network (computer system) Calls being made by personnel calling out Call center providing call handling as supplier under contract Private branch exchange (older equipment replaced by ACD) Outbound application hardware and software Risk analysis Mutual aid agreement with neighboring company Real-time minimum call overflow Return time operational Agreement with vendor specifying time limit for emergency delivery Fiber optics telecommunications line with twenty-eight voice lines Uninterruptable power source Companies supplying your company with products or services Distributed architecture means CSRs telecommute from anywhere Handling of voice communcations to ensure continuity Wide-area network Disaster that affects an entire city or region, usually caused by extreme weather, earthquake, or explosion