Business Challenge
A diversified financial services company wanted to validate and improve its existing disaster recovery plan and increase awareness of the issues surrounding the execution of the plan during a crisis. Like most financial organizations, the company performed regular, live IT systems testing of its data restoration operations. The company's senior IT management engaged Forsythe, as an independent business continuity expert, to design and facilitate a "table-top" exercise in which their 30-person disaster recovery team would talk through a detailed response to a hypothetical disaster scenario.
Solution
Forsythe researched the company's current and historical disaster recovery plans--collecting and reviewing documentation of processes, procedures, and tests--and examined its primary data center's IT infrastructure. Based on input from the disaster recovery team, Forsythe defined the scope of the exercise, pinpointed key areas of concern, and constructed a purposeful disaster scenario.
Forsythe created a one-day workshop that would engage the entire disaster recovery team, comprised of IT managers from three geographies, in a virtual disaster response. In the workshop, Forsythe presented the disaster scenario and led a general group discussion of key areas and specific questions Forsythe had identified as requiring critical attention.
The participants, who represented the IT elements supported by the primary data center--database group, application support group, development group, and network, server, and storage support groups--addressed the questions in separate breakout sessions by geographic location, each facilitated by a session leader. This was not a drill--the goal was not to grade the team's performance, or even to answer all the questions that day, but rather to flush out hidden discrepancies and uncertainties so they could be resolved by the appropriate parties throughout the organization. Neither was it conducted in real-time; 72 hours' worth of scenario was considered in a day.
Issues discussed spanned from logistical problems to service provider concerns. The logistics required to restore the company's operations at a different facility were examined in detail, including traveling off-site, obtaining tapes and restoring data, sending in damage assessment teams, receiving status reports from the field, establishing alternate carriers for network connectivity, considering licensing issues for back-up instances of applications, and more. The needs for communication, documentation, redundancy of personnel assignments, and back-up communication procedures were recurring themes. The need for extensive network bandwidth to support recovery activities was also raised. At the end of the day, more than 50 open issues (gaps) had been uncovered.
Results
The exercise produced three important results: a comprehensive list of specific gaps in their plan; a set of concrete action items for the disaster recovery team--namely the assignment of the open issues to appropriate personnel to resolve within established deadlines; and a greater overall awareness of what disaster recovery planning requires from everyone within the organization.
The list transformed the issue of disaster recovery from a lurking concern in the back of everyone's mind into a series of finite steps that could be logically assigned to individuals or small groups to resolve. As a result, the company is now in the process of fleshing out their disaster recovery procedures to greatly increase the likelihood of success, should they ever need to be implemented.
Even before the exercise was conducted, word of it raised the profile of disaster recovery planning throughout the company, dramatically increasing the number of participants from 30 to 70, including high-level IT executives, and business unit and risk managers not directly associated with IT. Post-workshop survey responses indicated that, beyond the specific outcomes of the exercise, the simple act of bringing so many key members of the organization together to consider and discuss disaster recovery was tremendously valuable.
