12 Disaster Recovery Mistakes that Could End Your Business
On a Saturday morning in March, attackers infiltrated the county’s networks, locking staff out of nearly every government system. 9-1-1 dispatchers’ screens went dark. Email went down. County prison doors couldn’t be opened. Police-cruiser computers were practically unusable.
County personnel described the incident as “traumatic” as 9-1-1 dispatchers were forced to use paper logs and maps, worried that they were sending emergency responders to the wrong locations.
A last resort
With limited data-backup capabilities, the county had no choice but to pay the attackers what they wanted: $400,000 in cryptocurrency. It was a gamble, but thankfully the attackers delivered the decryption keys, allowing the county to restore its data.
The incident underscored what can happen when disaster recovery goes wrong: prolonged downtime, skyrocketing costs, even an increased risk to the safety of employees.
We’re sympathetic to the situation that Jackson County faced. No organization is completely immune from a ransomware attack. That said, there are several common disaster recovery mistakes that businesses can avoid with proper planning. Here are 12 of them.
1) There’s no disaster recovery plan.
Recovering from a disaster is extremely challenging when there’s no plan in place. Recovery teams simply won’t know what to do and will instead rely on last-minute guesswork. This nearly guarantees that mistakes will be made, resulting in a longer, more clunky, more costly recovery.
Creating a comprehensive disaster recovery plan (DRP) is essential for every organization.
A good DRP helps a business better understand the scenarios that threaten operations and how to combat them. It lays out specific instructions for recovery, so that teams know exactly what to do when disaster strikes. It also outlines the technology deployments that support the recovery process and provides guidance on how they should be used.
2) You didn’t test the plan.
How do you know the procedures in your DRP are any good? Well, without testing them, you don’t.
Some organizations make the mistake of creating a plan and never touching it again. Maybe some protocols are implemented or some IT deployments are made. But nobody ever puts the plan to the test. So when a disruption actually occurs, no one knows if the planning will actually work until it’s too late to do anything about it.
Organizations must run mock recovery tests to determine if the proposed procedures are effective. Every step and every scenario should be tested. And when weaknesses are identified, the DRP should be updated accordingly.
3) The plan is outdated.
Just like your business continuity plan, your disaster recovery plan needs to be reviewed and updated on a regular basis. This ensures that all the information is still accurate, up to date and effective.
We’ve already addressed the importance of updating your DRP after testing. But that’s only part of it. As the months go by, a lot of the information may no longer be applicable. Employees identified as recovery personnel may have left the company. IT systems may have been replaced or upgraded. New threats may have emerged.
If disaster strikes and the procedures in the DRP can’t actually help the recovery process, then the plan is useless.
4) Not enough disasters were considered.
Risk assessments are a crucial part of disaster recovery planning. Businesses must be prepared for the wide range of disruptions that can affect their operations. Assessing risks enables you to determine which threats are possible, how probable they are, what impact they have and what steps are needed for recovery.
We know from the Jackson County ransomware attack that officials had not adequately prepared for such a disaster. But unfortunately, this is common among government agencies and businesses, especially as newer cybersecurity threats have emerged.
In order for businesses to quickly recover from a disaster, every threat needs to be considered and prepared for.
5) You never set an RTO.
RTO – short for Recovery Time Objective – dictates how quickly a recovery should happen to avoid more undesirable consequences. It can be applied to specific systems, such as networking, or operations as a whole. For example, if your business needs to restore critical operations within 6 hours to avert a catastrophic meltdown, then your RTO would be a max of 6 hours.
When no RTOs are in place, then recovery teams have no reference guide for how quickly the recovery needs to happen. This can mean that recovery takes far longer than it needs to.
Businesses should use an Impact Analysis to calculate the impact of each disaster scenario. In turn, this allows for setting accurate recovery point objectives that ensure operations are restored as quickly and efficiently as possible.
6) There’s no communication.
Even the greatest plans will fall apart when there’s no communication during recovery. Recovery teams absolutely need to communicate with each other and with key stakeholders to ensure that everyone is on the same page during these critical moments. Otherwise, confusion and chaos will ensue.
What does “communication” mean when it comes to disaster recovery?
It means having working communication channels for your mission-critical teams, even if primary communication methods are down. It means having a communication structure, outlining who should be contacted about updates and how. It means having a communication plan for keeping all personnel in the loop, whether via text messages, personal email, or other notification systems.
Better communication will almost always translate into a smoother, faster recovery.
7) No backups.
Now we’re entering truly dangerous territory.
Data backups are a fundamental component of disaster recovery, ensuring that businesses always have a fallback when data loss occurs. Not having a backup is one of the greatest mistakes a business can make.
Data loss is extremely common at businesses of all sizes. Employees accidentally delete files. Folders go missing without explanation. Data gets overwritten. Hard drives go bad.
And then there’s ransomware, which can lock up all your servers in a matter of minutes and essentially destroy your data (unless you can decrypt it or restore a backup).
Syncing files to the cloud (ala Google Drive or Dropbox) is not enough. If businesses want to survive a data catastrophe, they must deploy a robust data backup and business continuity solution.
8) You’re putting all your backups in one basket.
In other words, your backups are being stored in a single location, making them vulnerable to a wide range of on-site disasters.
Yes, having a local backup is better than having no backup at all. But if a fire destroys your entire on-site infrastructure, then you’ll be left with nothing.
Today’s best disaster recovery solutions employ the concept of hybrid backups, which combine an on-site backup with replicated backups in the cloud. This provides an extra layer of protection in case the local backup is destroyed or inaccessible.
9) The backups aren’t being tested.
Imagine a critical server has failed. No reason to freak out yet, because there’s a backup! And it’s only 30 minutes old!
But just as you’re breathing a sigh of relief, something else goes wrong. The backup can’t be restored. The data is corrupted.
This is an extremely common scenario, especially at businesses that are relying on older backup technology, where errors occur in the backup chain, making each new backup unusable.
One solution to this is deploying newer BC/DR technology in which each new backup is not dependent on the chain (i.e. Datto’s Inverse Chain Technology). But regardless of which solution you use, you absolutely must test your backups. Automated backup verification and testing can verify that backups are viable and can be restored without problem.
10) You never set a RPO.
Don’t conflate RPO with RTO – they are equally important objectives for disaster recovery, but they mean two different things.
While RTO refers to the speed of a recovery, RPO refers specifically to backups. Short for Recovery Point Objective, RPO sets a goal for how recent the last backup should be in the event of a data-loss event. Anything beyond that point would mean an unacceptable amount of data loss.
Setting an RPO ensures two things: 1) that you’re deploying backup technology that can accommodate your desired backup, and 2) that your IT teams are properly configuring these systems.
Here’s another way to look at the mistake of not setting an RPO …
11) The last backup is too old.
Without an RPO, you run the risk of having backups that are too old to be useful when data loss occurs.
Let’s say that ransomware has completely locked up your servers. You have backups, but they’re from last weekend – nearly 5 days old. By restoring those backups, you would lose all the data that was newly created or modified in the past week. For many businesses, that kind of loss would be devastating.
Use your DR planning to determine the appropriate RPO and backup frequency for your needs, and then make sure your backup technology can accommodate it.
12) You choose the wrong recovery method.
Even when your backups are viable and recent, you can still make big mistakes when restoring them. That’s because not all recovery methods are appropriate for the same situations. And if you choose the wrong one, you could be unnecessarily prolonging the recovery process or losing data.
For example, you probably don’t need to restore a full backup if you only need to recover a few files or folders. Similarly, in cases of ransomware attacks, you may be able to roll back only the data that has been encrypted, rather than everything. Or, maybe a server is no longer booting and it makes more sense to upload the backup to a hypervisor, rather than doing a bare metal restore.
It all depends on the capabilities of your BC/DR system. But selecting the appropriate restore option is critical to ensuring quick and seamless recovery.
Are your recovery systems adequate?
Learn more about protecting your organization with today’s best disaster recovery solutions. Request a free demo or contact our business continuity experts at Invenio IT. Call us at (646) 395-1170 or email success@invenioIT.com.