3 Real-world Disaster Recovery Scenarios

When a midsize manufacturing company got hit with a ransomware attack, the IT Director had the unenviable task of breaking the news to the CEO.  The firm’s ERP database had been compromised, and a well-organized group of hackers was demanding tens of thousands of dollars to release the data.

Manufacturing operations were heavily reliant on the company’s IT systems. Inventory, production processes, customer orders, and shipments were all managed within a single, highly integrated system. Without its ERP system, the company was brought to a near standstill.

Even if the company paid the ransom, top executives weren’t sure that would mean an end to their problems. Recent backups were also compromised, so the only options were to pay up or try to rebuild from an older copy of the company’s database.  It took two months to recover from the damage.

Why You Need Robust Disaster Recovery

We have all heard about the crippling ransomware attacks on Colonial Pipeline and other large companies.  Yet most ransomware attacks never show up in the headlines. Many are aimed at small and midsize organizations, and they’re increasing in frequency.

Ransomware isn’t the only reason you need a robust disaster recovery (DR) strategy, of course. Natural disasters, localized hazards, and acts of sabotage can also be devastating.

Experts estimate the average cost of downtime at around $9,000 per minute. According to cybersecurity company Malwarebytes, 1 in 5 businesses that experience cyber-attacks are compelled to completely cease operations until they can resolve the crisis. And those costs don’t include the cost of recovering data that has been compromised or destroyed.  Insurance carrier Hiscox estimates that the average cost of recovery is $200,000.

This article shares disaster recovery plan examples in reaction to three real-world scenarios. We’ll also explain what a successful recovery process looks like.

Disaster Recovery Plan Examples:

Let’s look at a few situations involving the restoration of data availability. The lessons below apply generally to any type of disaster recovery scenario.

Example 1: A DDoS Attack

Imagine that a group of malicious hackers executes a Distributed-Denial-of-Service (DDoS) attack against your company. The DDoS attack focuses on overwhelming your network with illegitimate requests so that legitimate data cannot get through.

As a result, your business can no longer connect to databases that it accesses via the network. In today’s age of cloud-native everything, that probably includes some of your company’s most important databases.

In this scenario, disaster recovery means being able to restore data availability even as the DDoS attack is underway. (Ending the DDoS attack would be helpful, too, but anti-DDoS strategies are beyond the scope of this article. Moreover, the reality is that your ability to stop DDoS attacks once they are in progress is often limited.) Having backup copies of your data is critical in this situation.

What may be less obvious, however, is the importance of having a plan in place for making the backup data available by bringing new servers online to host it. You could do this by simply keeping backup data servers running all the time, ready to switch into production mode at a moment’s notice. That can be costly, however, because it means keeping backup servers running at full capacity all the time.

A more efficient approach is to keep backup data server images on hand, spinning up new virtual servers in the cloud when you need them. Although not instantaneous, this process can take just a few minutes, provided that you have the images and data in place and ready to go.

An air-gapped backup copy of data is a physical copy of data that is stored offline and disconnected from any network or internet connection. This physical copy of the data is typically stored in a secure location, such as a vault or a fireproof safe. Air-gapped backups are considered to be the most secure type of backup because they are not vulnerable to cyberattacks. Air-gapped backups are physically isolated from the internet and other networks, making them immune to cyberattacks such as ransomware and malware. This is because cyberattacks typically rely on network connectivity to spread and infect systems. By keeping backups offline, you ensure that they cannot be compromised by these threats

Example 2: Data Center Destruction

One of the worst-case scenarios that a modern business can face is a disaster that destroys part or all of its data center – including the servers and disks inside it.

While such a situation is rare, it can happen.  It’s not limited to large-scale disasters, either, such as an earthquake or hurricane. Localized events like electrical surges, burst pipes, or even rodent infestations can cause permanent data center damage.

The best way to prepare your business for recovery from this type of disaster is to ensure that you have offsite copies of your data. If your production data lives on-premise in one of your data centers, this would mean keeping backups of the data at another data center site or in the cloud. If your data is hosted in the cloud, you could back it up to local storage, another cloud, or a different region of the same cloud.

You should also make sure you have a way of restoring the backup data to the new infrastructure quickly. Moving large amounts of data from one site to another over the Internet can take a long time, so it’s not always a wise strategy to rely on this approach. In some cases, it might be faster to move physical copies of disks from one site to another. Alternatively, it might prove quicker and easier to set up new servers in the data center where your backup data lives, then connect them to the backup data and turn them into your production servers.

The bottom line: Restoring data after data center destruction requires having offsite copies of the data available, as well as a plan for moving that data quickly to wherever it needs to go following the disaster to keep your business running.

Example 3: Data Sabotage

A third type of data disaster that might befall your business is one in which someone – such as a disgruntled employee – deliberately sabotages data. The employee might insert inaccurate or bogus data into your databases, for example, to lower data quality and make the data unusable for your business. He or she might even insert malicious code into your data to spread malware to your systems.

The critical step in preparing for this type of disaster is to ensure that you have backup copies of your data that go back far enough in time to allow you to recover using a version of the data that you know to be safe. If the only copy of your data that you have available was taken a day ago, but the damage occurred three days ago, the backup won’t necessarily help.

This is why it’s a good idea, when possible, to have multiple backups of your data on hand, each taken at a different time increment. Instead of deleting the last data backup when you make a new one, keep older backups on hand so that you can use them for disaster recovery if necessary. If you make a backup daily, and keep seven backups on hand, then you know that you can restore data from as long as a week ago if newer backups contain damaged information.

There is a trade-off when using an older backup for disaster recovery, of course. Any data that was added or modified since the last good backup will be lost. In certain disaster recovery situations, however, this is a relatively small price to pay.

Keep in mind, too, that if you can identify which parts of your data were sabotaged, you can leave that data intact, and recover only the damaged data to minimize data loss.

The time to plan your disaster recovery strategy is now, before you encounter a problem. Get started by reading our free white paper The One Essential Guide to Disaster Recovery. This paper provides a basic understanding of the building blocks of IT and business continuity – from understanding the concepts of disaster recovery and information availability to calculating the business impact of downtime and selecting the right software solution.