Storage

Silent data corruption, and the filesystems that catch it

Most filesystems return whatever the disk hands back. ZFS and Btrfs checksum every block and verify on read, which is the difference between noticing bit rot and backing it up.

Files Corrupter ·

Silent data corruption, and the filesystems that catch it

There is a failure mode that produces no error, no log line and no alert. The disk returns data, the application accepts it, and the data is wrong. It is usually called silent data corruption, or bit rot when the cause is decay.

Why it is silent

A conventional filesystem such as ext4 or NTFS does not checksum file contents. It trusts the storage layer. Drives do carry per-sector error correction and will report an unrecoverable read error when they cannot fix a sector, and that case is loud and handled.

The quiet case is when something returns data that is wrong but plausible: a misdirected write that lands in the wrong place, a firmware bug, a marginal SATA cable flipping a bit in transit, a memory error corrupting the buffer before it is ever written. The drive reports success because as far as it knows, everything worked.

The file is now wrong, and nothing in the stack has any way to know.

What ZFS and Btrfs do differently

Both write a checksum with every block, data and metadata alike, and store it separately from the block itself. On every read the checksum is recomputed and compared. A mismatch is not a warning; it is a hard error the filesystem acts on.

ZFS uses SHA-256 by default, with Blake3 available in OpenZFS 2.3 and later. Btrfs uses CRC32C, which is considerably faster and weaker, but entirely adequate for the accidental corruption it is designed to catch. The trade-offs between those choices are the subject of CRC32, MD5 and SHA-256, which one catches what.

Scrubbing

Verification on read only helps for blocks you actually read. Data you have not touched in three years is exactly the data most likely to have decayed, and it is the data nothing has checked.

A scrub solves that by walking every allocated block in the pool, recalculating checksums and comparing them against what was recorded at write time. Unlike a traditional fsck, a scrub runs while the filesystem is online and mounted, so it is a scheduled background job rather than an outage.

Where there is redundancy, a mirror or a RAIDZ vdev, ZFS does more than report the mismatch. It reads the good copy, returns that to the application, and rewrites the bad block. The corruption is repaired during the scrub, and the counter in zpool status is the only sign it happened.

Without redundancy, a single-disk pool detects corruption and cannot fix it. That is still far better than not knowing, because you learn which file is damaged while you still have a backup that predates it.

The scrub schedule people skip

Monthly is a reasonable default for consumer drives; enterprise deployments often go weekly. The specific interval matters less than the fact that it happens at all and that someone reads the result. A scrub that runs faithfully and reports to nobody has the same practical value as no scrub.

ECC memory, the caveat

Checksums protect data on its way to and from the disk. They do not protect it while it sits in RAM. If a memory error corrupts a buffer before the checksum is computed, the filesystem faithfully stores corrupt data along with a perfectly valid checksum for it, and every subsequent verification passes.

This is why ECC memory belongs in anything storing data you care about. It is not a ZFS requirement in the sense that ZFS refuses to run without it, but the combination is what actually closes the loop.

Why any of this connects to deliberate corruption

Two reasons.

First, it explains why file size is such a poor health indicator. Silent corruption never changes the size. A 40 MB file with a rotted block is still 40 MB, and it copies and syncs like any other file, an idea explored further in what file corruption actually is.

Second, it means the recovery path in your software will eventually be exercised, whether or not you have tested it. If you want to know how your importer behaves against a file with damage in the middle rather than at the front, you do not have to wait years for a drive to decay. You can reproduce that failure on demand and find out this afternoon.