Eliminate Nightmare Scenarios with a Disaster Recovery Test Plan

by Jul 13, 2020Business Continuity

A disaster recovery test plan provides a framework for testing IT systems and protocols to identify problems that may impede recovery efforts during a real disaster.

By routinely testing your systems and processes, you help to eliminate unexpected challenges during a disruption and ensure that the business is able to quickly recover in a real-world event.

The following guide provides an overview of how disaster recovery testing works, why it’s so important and what should be included in the plan.

 

What is a disaster recovery test plan?

A disaster recovery test plan is an organization’s documented outline of various readiness and recovery tests, each comprised of its series of steps and procedures. The document serves to guide businesses on how and when to test their preparedness for a disaster.

A test plan is not the same as a disaster recovery plan (DRP), but it can be an important component of one. Whereas a disaster recovery plan outlines a business’s entire approach to disaster recovery, the test plan focuses specifically on testing procedures.

 

Areas of focus

For many businesses, a disaster recovery test plan focuses largely on IT system tests, such as backup restore tests or network stress tests. But the tests can also apply to drills carried about by employees to test their response to an emergency.

Disaster recovery test plan examples:

  • Testing business continuity & disaster recovery (BC/DR) solutions to ensure data backups are viable and can be restored without issue.
  • Mock recovery tests that guide recovery personnel through the steps to restoring backups or resolving other IT system failures.

 

  • Emergency drills that test the procedures outlined in a business’s DRP, such as steps for replacing failed hardware or even employee evacuation procedures, i.e. in a fire.

 

There are no hard-and-fast rules about what should be included in a test plan. However, any system or procedure designed to support recovery in a disaster should be tested and outlined in the test plan accordingly.

 

Why test?

When do businesses usually find out that a backup can’t be restored? At the worst possible time – after data loss has already occurred.

Routine testing helps eliminate those nightmarish 3 a.m. wakeup calls. By running tests, you ensure that systems can be recovered and that the recovery process itself is as fast and efficient as it should be.

Disaster recovery testing can help to identify:

  • Data corruption in the backup that would result in a failed restore.
  • Delays that prolong the recovery time and make it impossible to achieve RTO objectives.
  • Inefficiencies in the recovery process that would have disastrous consequences during a real event.
  • Mistakes made by recovery personnel that can be eliminated with process improvement and further testing.

 

Here’s a simple analogy …

Fire drills.

Most jurisdictions require businesses to run routine fire drills. Because without them, people don’t know what to do. They may know where the exits are (or they might not), but they won’t always know the fastest and safest routes out of the building. So, when a fire alarm goes off—without drills having been performed—people go in all different directions. In a real fire, this would be extremely dangerous.

A disaster recovery test plan provides guidance much like the procedures for fire drills, but applied specifically to recovery systems and protocols.

 

DR test plan scenarios

We’ve mentioned some examples of tests that can be included within your DR planning. But let’s look at some more specific scenarios within those tests.

It’s the responsibility of your DR planning team to consider the various scenarios that might arise during a disaster and what the response should be. Those scenarios will guide the necessity of conducting various tests.

Within BC/DR, these scenarios could include:

  • Backup verification / validation
  • Test booting virtualized backup
  • Testing restores on local devices and in the cloud
  • File- and folder-level restore tests
  • Testing bare metal restores or hypervisor recoveries
  • Procedures for responding to failed restores or errors during any of these tests

 

Other types of recovery tests

A comprehensive test plan extends beyond data backup and IT to ensure the business can respond to a wide range of disaster scenarios. If your organization already has a business continuity plan (BCP) or DR plan in place, then you’ve hopefully already identified the unique risks that threaten your business.

Each of those scenarios should have a plan for recovery. And each plan should be tested.

Some non-IT examples:

  • Physical destruction of infrastructure or business location
  • Limited/restricted access to the building (i.e. after a fire or natural disaster)
  • Sudden work stoppages or workforce shortages, i.e. caused by a pandemic, transportation breakdowns, worker strikes and so on
  • Unexpected, forced operational shifts, such as how COVID-19 resulted in the closure of some businesses

 

How can you test for some of these scenarios?

Let’s take the limited building access, for example. Let’s pretend an earthquake has compromised the structural integrity of your building and it’s been condemned by the city. A good test for this situation would be to have recovery teams do a mock drill, running through the list of procedures for restoring operations in another location.

You would be testing the ability to quickly secure another location or move to a decentralized work environment. You’d also have to test IT systems to ensure they could support this shift to remote work. And assuming that the on-site IT infrastructure would not be available, you need to test all your contingency systems, such as offsite server capabilities, virtual environments, cloud and so on.

Remember, it’s not just the systems you’re testing. It’s also the processes for using those systems during a recovery.

 

Disaster Recovery Test Plan Template

Every test plan is different and should be unique to the business’s objectives and risks. However, the underlying foundation for most plans will be similar: identify the tests that must be performed; why; how; and when.

Here is a basic example of a disaster recovery test plan template:

 

1) Testing Objectives

In this section, you’ll outline the objective of the plan and testing, addressing what they aim to accomplish and how they fit into your overall disaster recovery planning.

  1. Like the DRP itself, all tests need to have a purpose.
  2. Include test objectives at the start of your DR test plan.
  3. Objectives help to provide a clearer understanding of the overall goals and purpose for the tests, which helps to educate stakeholders and guide those who conduct the tests.

 

2) Testing Approach

The approach section outlines the process for testing, addressing how it should be administered, how often and what actions need to be completed to facilitate the testing.

  1. Identify the necessary tasks surrounding the overall testing process.
  2. These tasks are NOT the individual protocols for each test, but instead the responsibilities of those who coordinate the testing process.
  3. Examples include: identifying testing schedules, meetings, stakeholder signoffs, gap analysis, incorporation of findings & updates into the DRP, etc.
  4. Schedules should also incorporate timeframes (i.e. 1 day, 1 week, etc.) for completing the tests.

 

3) Testing Responsibilities

  1. Identify which personnel or recovery teams are responsible for conducting the steps.
  2. Include any specific tasks or responsibilities related to the tests, such as logs, reporting requirements, managers to communicate with, and so on.

 

4) Testing Scenarios & Systems

Now, we’re getting to the real meat of the DR test plan. In this section, you’ll list the tests that must be performed and the steps that each test is comprised of.

  1. Outline the actual systems and scenarios to be tested.
  2. Break down each test into specific instructions.
    1. Include clear steps for how the test should be performed.
    2. Use clear scenarios, i.e. “If A occurs, proceed to B.”
  3. When feasible, consider the use of visuals such as flow charts for a clearer guide of testing scenarios.

 

5) Operational DR Testing

Remember that disaster recovery testing should not be limited to IT. Organizations should test their preparedness for every type of operational disruption. Examples can include, but are not limited to:

  1. Loss of equipment or business assets.
  2. Loss of workforce.
  3. Loss of suppliers.
  4. Loss of electricity, gas or other utilities.

 

6) Post-Recovery Testing

Testing isn’t just a preventative measure. Following a disaster, restored systems should be tested further to ensure they are stable and functioning properly. This section outlines the steps for conducting those post-recovery tests.

  1. Identify the systems that need immediate testing after (or even during) a disaster scenario.
  2. These tests are particularly important during a partial recovery; for example, when you’re transitioning from backup systems (or systems from an alternate site) back to the original site.

 

7) Documentation and Updates

  1. Testing should be documented according to the guidelines in this section and/or the Testing Approach listed above.
  2. Include instructions for evaluating the documentation for further assessment of the disaster recovery planning.
  3. Include directions for updating the test plan based on the outcomes of tests (i.e. if process improvements can be made)

 

Request a Free Demo

For more information on disaster recovery testing or data backup solutions, request a free demo or contact our BC/DR experts at Invenio IT. Call us at (646) 395-1170 or email success@invenioIT.com.

New call-to-action

Dale Shulmistra is a Business Continuity Specialist at Invenio IT, responsible for shaping the company’s technology initiatives -- selecting, designing, implementing & supporting business continuity solutions to bolster client operational efficiencies and eliminate downtime.