Your backup is a hypothesis until you restore it

Every organisation that has lost data had backups. That is not a joke, it is the pattern. The backup job was green. The retention was configured. Nobody had ever run the restore.

The green tick measures the wrong thing

A successful backup job proves that bytes were written somewhere. It does not prove that:

• those bytes are readable on the destination media
• the archive is structurally valid
• the encryption key is still available and correct
• the restore procedure works on current infrastructure
• anyone still employed knows how to run it
• the restore finishes inside the window the business assumes

Each of those has taken an organisation down. None is detected by monitoring that only watches whether the job exited zero.

3-2-1, and the parts people drop

The familiar rule is three copies, on two different media, with one off site. The extended version adds one offline or immutable copy and zero verification errors, which exists because ransomware happily encrypts a network-mounted backup target, and because unverified copies are not copies.

The offline requirement is the one most often quietly dropped, and the verification requirement is the one most often never implemented at all.

Running an actual drill

A restore drill does not need to be elaborate to be worth doing.

1. Pick the restore, not the backup. Choose a real scenario: last night’s database, a file deleted three weeks ago, an entire host.
2. Restore somewhere else. Never over the live system. The point is to measure, not to gamble.
3. Time it. Compare the result to the recovery time objective the business believes it has. This is usually where the surprise happens.
4. Verify the content, not the exit code. Row counts, checksums against a known-good manifest, an application that starts and serves a real request.
5. Have someone else do it. If only one person can restore, you do not have a restore procedure, you have a dependency.

Where a deliberately damaged file earns its place

The drill above tests the happy path. The interesting failures live on the other one, and you can construct that deliberately.

Take a database dump and corrupt it lightly, so most statements survive and it fails partway through. Then run the restore and answer:

• Does the import run inside a transaction, so a mid-file failure rolls back cleanly rather than leaving half a schema?
• Is the failure loud, or does a cron job swallow the exit code?
• Does the integrity check run before the restore begins, or does a corrupt archive get the chance to overwrite a good one?
• If the restore aborts, is the previous copy still intact?

That last question has ended companies. A restore that deletes the target before verifying the source is a single-step path to total loss.

A partially damaged SQL dump is the fixture for that test, and Light level is the setting that produces it: enough of the file survives to start, not enough to finish. You can build a broken restore fixture in a few seconds and keep it in the runbook repository.

The broader argument for testing error paths deliberately is made in testing the error path nobody tests, which covers the application side of the same discipline.

Verification that actually verifies

Two rules make the difference.

Check the copy, not the source. Hashing the file you just wrote while it is still in page cache verifies that memory matches memory. Read it back from the destination, ideally later, ideally on the machine that would perform the restore.

Store the checksum somewhere the failure cannot reach. A manifest that lives inside the archive it describes is decorative. The same principle explains why filesystem scrubs keep checksums separate from the blocks they cover.

The uncomfortable summary

An untested backup is a belief. A tested restore is a fact. The gap between them is discovered at precisely the moment you can least afford it, and closing it costs an afternoon.