Ninety percent of IT leaders say they are confident they could recover from a ransomware attack. In practice, according to a Veeam survey of more than 900 IT and security leaders published in April 2026, only 28% of ransomware victims fully recovered their data. The average victim got back 72% of it.
That gap between the confidence and the outcome is what disaster recovery testing exists to close.
A second problem lies beneath it. A lot of the published advice about how often you are required to test is simply wrong. Articles state that HIPAA requires annual testing. It does not. That SOC 2 requires it. It does not. That ISO 27001 control A.17 covers it. That control has not existed since 2022.
This guide covers how to run a test that actually tells you something, what goes wrong when organisations skip it, and what the frameworks genuinely say — with clause numbers so you can check.
The short version
- Five kinds of DR test exist, and they answer completely different questions. A tabletop and a full failover are not substitutes.
- Only two of nine major frameworks put a number in writing. PCI DSS says 12 months. DORA says yearly. The rest say “periodic,” “regularly,” or “at planned intervals.”
- The requirement most people miss is the change trigger, not the calendar.
- What fails is rarely the technology. It is ownership, undocumented dependencies, and assumptions nobody checked.
- The test report is the deliverable — a dated artifact with objectives, measured results and tracked fixes.
What disaster recovery testing actually is
A DR test is a controlled rehearsal of your recovery plan, run to find out whether you can restore service inside your stated targets before something forces you to find out for real.
Two numbers decide whether a test passed:
- RTO (Recovery Time Objective) — how long recovery is allowed to take.
- RPO (Recovery Point Objective) — how much data you can afford to lose, measured in time.
A test without those written down beforehand is not a test, it is an activity. You can only fail against a target you set in advance, and a test that cannot fail tells you nothing.
Google’s SRE book puts the underlying point more bluntly than most vendors will:
“The most important principle in this layer is that backups don’t matter; what matters is recovery. The factors supporting successful recovery should drive your backup decisions, not the other way around.” — Site Reliability Engineering, Chapter 26
The five types of DR tests
They cost wildly different amounts and prove wildly different things.
| Type | What happens | Effort | What it proves | What it misses |
|---|---|---|---|---|
| Plan review | Someone reads the document and checks it is current | An hour | The plan exists and names the right people | Nothing about whether it works |
| Tabletop | The team talks through a scenario in a room | Half a day | People know their roles; gaps in the decision chain | Everything technical |
| Simulation | Recovery steps performed in an isolated environment | Days | The runbook is followed and the steps are in the right order | Real dependencies and real load |
| Parallel test | Systems recovered to standby and run alongside production | Days to weeks | Recovery works and produces correct output | Whether the cutover itself works |
| Full interruption | Production is failed over for real | Weeks to plan | Everything, including the part everyone is afraid of | Nothing — but it carries real risk |
A sensible progression from a standing start: tabletop first to find the obvious gaps cheaply, then a simulation of your most critical system, then a parallel test, then a full failover once the earlier three stop producing surprises. Going straight to a full interruption test on an untested plan is how organisations cause the outage they were trying to prevent.
Both AWS and Google argue the same thing about frequency, from opposite ends of the industry. AWS’s Well-Architected Framework rates “never exercise failovers in production” as a high-risk anti-pattern, and states:
“Our experience has shown that the only error recovery that works is the path you test frequently.” — AWS Well-Architected Framework, Reliability Pillar, REL13-BP03
Google’s version:
“If recovery tests are a manual, staged event, testing becomes an unwelcome bit of drudgery that isn’t performed either deeply or frequently enough to deserve your confidence. Therefore, automate these tests whenever possible and then run them continuously.”
What actually goes wrong
The failures are consistent across organisations, and they are almost never the thing people plan for. Each of these is a documented case, not a hypothetical.
The backups existed, and none of them worked
In January 2017, a GitLab engineer running a routine replication fix wiped roughly 300 GB from the production database instead of the replica. Then they went looking for the backups.
“Unfortunately the process of both finding and using backups failed completely.”
The nightly pg_dump had been failing silently for months, because the tooling was version 9.2 running against a 9.6 database. The failure notifications existed — and were being rejected before anyone saw them:
“While notifications are enabled for any cronjobs that error, these notifications are sent by email… Unfortunately DMARC was not enabled for the cronjob emails, resulting in them being rejected by the receiver. This means we were never aware of the backups failing, until it was too late.”
The S3 bucket was empty. Azure disk snapshots had not been enabled, and GitLab’s stated reason is one most teams would recognise in themselves:
“We assumed our other backup procedures were sufficient.”
They recovered from a staging snapshot taken six hours earlier and lost the difference. The copy alone took about 18 hours, because the disks were throttled to around 60 Mbps.
GitLab’s own answer to why none of this was caught is the single most useful line in the whole postmortem:
“Why was the backup procedure not tested on a regular basis? – Because there was no ownership, as a result nobody was responsible for testing this procedure.” — GitLab postmortem, 10 February 2017
The backup was on the same system as the thing it was backing up
In August 2023, Toyota halted production at its domestic plants for three days. Not a cyberattack — a maintenance job that ran out of disk space.
“Since these servers were running on the same system, a similar failure occurred in the backup function, and a switchover could not be made. This led to the suspension of domestic plant operations.” — Toyota statement, 6 September 2023
A DR test would have caught this in an afternoon, because it is the first thing a test does: fail the primary and see whether the secondary takes over.
The backups were perfect. There was nothing to restore them onto.
The British Library’s October 2023 ransomware attack produced one of the most candid incident reviews any organisation has published, and it describes a failure mode most DR plans do not consider at all:
“while we have secure copies of all our digital collections – both born-digital and digitised content, and the metadata that describes it – we have been hampered by the lack of viable infrastructure on which to restore it.”
The attackers had deliberately destroyed servers to inhibit recovery. The Library’s own diagnosis of why recovery took months rather than days:
“our reliance on legacy infrastructure is the primary contributor to the length of time that the Library will require to recover from the attack.”
Their own stated lesson:
“Business continuity plans for the total outage of all systems need to be practised regularly, in addition to those relating to individual systems and services.” — British Library Cyber Incident Review, 8 March 2024
Redundancy across two regions, deleted by something above both of them
In May 2024, UniSuper — an Australian pension fund managing tens of billions — had its entire Google Cloud private cloud subscription deleted. Google’s own writeup explains why:
“one input parameter was left blank when using an internal tool to provision the customer’s Private Cloud. As a result of the blank parameter, the system assigned a then unknown default fixed 1 year term value for this parameter.”
A year later, it expired and auto-deleted. UniSuper had duplication across two geographies, and it did not help:
“when the deletion of UniSuper’s Private Cloud subscription occurred, it caused deletion across both of these geographies.”
What saved them was a backup held with a completely different provider:
“UniSuper had backups in place with an additional service provider. These backups have minimised data loss, and significantly improved the ability of UniSuper and Google Cloud to complete the restoration.” — Joint statement, UniSuper and Google Cloud, 8 May 2024
Geographic redundancy protects against a region failing. It does nothing about a failure at the account layer sitting above every region you use. That is worth testing for specifically.
The same lesson destroyed Code Spaces in 2014. An attacker reached the AWS control panel; when the company tried to lock them out, the attacker deleted the EBS snapshots, S3 buckets, and machine instances together. The company had advertised full redundancy — but production and backups shared one control plane, so a single credential compromise took both. Code Spaces never reopened.
DNS took out the tools needed to fix DNS
Meta’s six-hour global outage in October 2021 is the standard example of a dependency nobody had mapped:
“first, it was not possible to access our data centers through our normal means because their networks were down, and second, the total loss of DNS broke many of the internal tools we’d normally use to investigate and resolve outages like this.”
Meta already ran a mature drill programme — “we simulate a major system failure by taking a service, data center, or entire region offline.” It still had a blind spot, which they acknowledged:
“while we’ve never previously run a storm that simulated our global backbone being taken offline, we’ll certainly be looking for ways to simulate events like this moving forward.” — Meta Engineering, 5 October 2021
Even a good testing programme only covers the scenarios someone thought of.
The pattern across all of them
None of these were caused by a technology that failed to work as designed. They were caused by an assumption nobody had checked. Uptime Institute’s 2026 outage analysis reaches the same conclusion from the data:
“For 2026, failures to follow established procedures remain the leading driver of human error-related outages”
The same report found 57% of respondents said their most recent major outage cost more than $100,000, and for the second year running, one in five said it exceeded $1 million. — Uptime Institute Annual Outage Analysis 2026, published 13 May 2026
How often you actually have to test
This is where most published advice goes wrong, so here is what the frameworks say, with references.
| Framework | Testing required? | Interval stated? | Reference |
|---|---|---|---|
| PCI DSS 4.0.1 | Yes — of the incident response plan | Yes: at least every 12 months | Req. 12.10.2 |
| DORA (EU financial) | Yes | Yes: at least yearly | Art. 11(6), Art. 24(6) |
| HIPAA Security Rule | Yes — but Addressable, not Required | No. “Periodic” only | 45 CFR §164.308(a)(7)(ii)(D) |
| SOC 2 | Yes, if Availability is in scope | No — you set it | TSC A1.3 |
| ISO 27001:2022 | Yes | No interval anywhere | A.5.30, A.8.13 |
| ISO 22301:2019 | Yes | No — “at planned intervals” | Cl. 8.5, 8.6 |
| NIST SP 800-34 Rev. 1 | Yes (guidance) | No — “organisation-defined” | §3.5, via SP 800-53 CP-4 |
| FFIEC (US banking) | Yes | No — risk-based | BCM booklet 2019, §VII |
| NIS2 | Yes, in substance | No — “at planned intervals” | CIR (EU) 2024/2690, Annex 4.1.4 |
Five claims you will see that are not true
“HIPAA requires annual DR testing.” Wrong twice. §164.308(a)(7)(ii)(D) says “periodic,” with no interval, and it is marked Addressable rather than Required. What is Required is the data backup plan, the disaster recovery plan and the emergency mode operation plan. Addressable is not optional, though — you either implement it or document a reasoned assessment of why it is not reasonable and appropriate, and what you did instead.
“SOC 2 requires annual DR testing.” It requires nothing of the kind. Criterion A1.3 names no interval, and the entire Availability category is optional — a Security-only SOC 2 has no DR testing criterion at all. The annual figure comes from your policy, which then becomes the standard the auditor holds you to.
“ISO 27001 control A.17 covers DR testing.” A.17 has not existed since the 2022 revision. The relevant controls are A.5.30 (ICT readiness for business continuity) and A.8.13 (backup, “regularly tested”). Neither states an interval. Articles citing A.17.1.3 are working from the 2013 edition.
“PCI DSS requires an annual DR test.” PCI DSS has no business continuity or disaster recovery requirement — the word “continuity” appears exactly once in the standard. Requirement 12.10.2 mandates an annual test of the incident response plan, which under 12.10.1 must include business recovery and continuity procedures. Narrower than the claim, and it still means you test.
“NIST recommends annual testing.” SP 800-34’s guidance says “organisation-defined frequency” and defers to SP 800-53 control CP-4, which is a fill-in-the-blank. “Annually” appears in its sample policy sidebar and template boilerplate. The federal annual expectation comes from FedRAMP and agency baselines filling in that blank — a different and much narrower claim.
So what should you do?
Annually is a well-founded default. Most organisations adopt it, most auditors expect it, and two frameworks genuinely require it.
But two things matter more than the number.
Write the frequency into your policy, then hit it. For SOC 2, ISO 27001 and FFIEC, your own policy is the standard you are measured against. The compliance failure is almost never “you tested every 18 months.” It is “your policy said annually, and you cannot produce a test.”
Retest after significant changes. This appears in PCI DSS, DORA, ISO 22301, the NIS2 implementing regulation and FFIEC guidance — and almost never in articles about DR testing. FFIEC puts it plainly: significant changes can render existing test plans obsolete, so plans should be retested soon after the change. A migration, a new provider or a major re-architecture resets the clock, whatever the calendar says.
If your RTO is genuinely short — payments, clearing and settlement, critical clinical systems — annual testing is a documentation exercise rather than a control.
Running a test without taking production down
The most common reason DR tests do not happen is a reasonable one: nobody wants to be the person who caused an outage during a drill.
- Test in an isolated network. Recover into a separate VPC or subscription with no route to production. You lose some realism and gain the ability to run the test at all.
- Pick a quiet window, and tell people. Surprise tests sound rigorous and mostly produce panic and bad data.
- Start with one system, not the estate. A full-estate test is a project. A single critical system is a Tuesday.
- Agree an abort condition in writing. What would make you stop, who can call it, how you roll back.
- Restore to a point in time, not just “latest.” Restoring the newest backup proves a backup exists. Restoring to a chosen moment proves you can meet your RPO.
- Check the recovery environment can actually hold it. Google’s recovery checklist asks whether you have “sufficient machine resources to run all of the setup, restore, and post-processing tasks.” The British Library’s answer to that question, discovered the hard way, was no.
What the test report should contain
The report is the deliverable, and auditors across every framework want broadly the same things.
- Date, scope and participants.
- The objectives set in advance — RTO and RPO as numbers.
- The scenario you ran, and why that one.
- Measured results against those objectives. Actual recovery time, actual data loss. Numbers, not adjectives.
- Evidence generated during the test — restore logs, system output, screenshots, attendance for a tabletop. Contemporaneous beats reconstructed.
- What went wrong, plainly, including anything that only worked because someone improvised.
- A tracked issue list with named owners and target dates.
- Evidence the plan was updated. Several frameworks pair testing with revision; the update is half the requirement.
- Anything you decided not to fix, with a documented risk acceptance.
A test that produces a clean report and no findings usually means the test was too easy.
A practical first test
- Pick one system — the one whose failure would hurt most.
- Write down its RTO and RPO before anything else. If nobody knows them, that is your first finding.
- Run a tabletop. An hour with the people who would actually be involved. Expect to learn that two of them did not know they were in the plan.
- Fix what that surfaces, then schedule a technical restore into an isolated environment.
- Time it. Compare against the RTO.
- Assign an owner to the testing process itself. GitLab’s finding — that nobody was responsible — is the most repeatable failure on this page.
- Write the report, assign the fixes, and put the next test date in the calendar before you close it.
Common questions
How long does a DR test take?
A tabletop, one to three hours. A single-system technical restore, a day to run and a few days to prepare. A parallel test, one to three weeks including planning. A full interruption test is a project with a change-approval process attached.
How often should we test?
Annually is the reasonable default and the one most auditors expect. More often if your RTO is short or your environment changes constantly — and always after a significant change, rather than waiting for the anniversary.
What is the difference between a DR test and a backup test?
A backup test proves data can be restored. A DR test proves the service can be brought back and used. Passing the first and failing the second is extremely common. As Google’s SRE book observes, quoting a saying of unknown origin: “No one really wants to make backups; what people really want are restores.”
Do we need a separate DR test for cloud workloads?
Yes. Cloud providers guarantee their infrastructure, not your recovery. UniSuper had duplication across two Google Cloud regions and lost both, because the deletion happened at the subscription layer above them. Region failover, cross-region replication, IAM in the recovery account, and whether your infrastructure-as-code can genuinely rebuild the environment all need testing.
Who should be involved?
The people who would genuinely be involved in a real incident — usually more people than the plan lists. Infrastructure, application owners, networking, security, and someone who can make a business decision about when to declare a disaster.
What if the test fails?
Then it worked. A failed test in a controlled window costs a report and some fixes. The same failure found during a real outage cost GitLab six hours of data and Toyota three days of production.
