Magic bytes, and why a file extension is not a file type
The name after the dot is a hint, not a fact. Signatures, libmagic and content sniffing decide what a file really is, and the gap between them is a security problem.
Rename invoice.pdf to invoice.png and nothing about the file changes. Not
one byte. The extension is a convention for humans and for the operating
system’s file association table, and it carries no authority at all about what
the file contains.
What actually identifies a file is the bytes at the front of it.
Signatures
Most formats begin with a fixed sequence, usually called magic bytes or a file signature:
| Format | Signature | Notes |
|---|---|---|
25 50 44 46 | %PDF | |
| ZIP, DOCX, XLSX, APK, JAR | 50 4B 03 04 | PK, after Phil Katz |
| PNG | 89 50 4E 47 0D 0A 1A 0A | Deliberately includes bytes that break naive text transfer |
| JPEG | FF D8 FF | Start of image marker |
| GIF | 47 49 46 38 | GIF8 |
| Windows PE | 4D 5A | MZ, Mark Zbikowski |
| ELF | 7F 45 4C 46 | .ELF |
| RIFF (WAV, AVI, WebP) | 52 49 46 46 | Format named again at offset 8 |
Not every format cooperates. Some put their signature at an offset rather than
at byte zero: ISO 9660 puts CD001 at 0x8001, after a 32 KB system area. Some
have no fixed signature at all, which is true of CSV, plain text and most source
code. And some are containers, so PK tells you it is a ZIP but not whether it
is a spreadsheet, an e-book or an Android package.
The PNG signature deserves a mention for how much thought went into it. The
high-bit byte catches connections that strip the eighth bit, the \r\n
detects line-ending translation, and a Ctrl+Z stops the file printing endlessly
if someone types it on DOS. It is eight bytes of defensive engineering.
What tools actually use
The Unix file command reads a magic database, historically /etc/magic, now
usually libmagic. It checks signatures, offsets and structural rules, and
returns a best guess with a confidence ordering. It is right most of the time
and is not a security boundary.
Browsers do something similar called MIME sniffing. If a server sends a
response with a suspicious or missing Content-Type, the browser may inspect the
bytes and decide for itself. That behaviour is convenient and has caused a long
tail of vulnerabilities, which is why X-Content-Type-Options: nosniff exists
and why you should send it.
Polyglots
Now the interesting part. Because different parsers look in different places, a single file can satisfy several of them at once. These are polyglots.
A classic example puts a GIF header at the front, which satisfies an image validator checking magic bytes, while the remainder is a valid ZIP or a script that some other component happily executes. The file is genuinely both things. Neither parser is wrong; they are simply asking different questions.
This is why “we check the magic bytes” is a weaker guarantee than it sounds. It proves the first few bytes look right. It does not prove the rest of the file is what you expect, and it does not prove that no other component will interpret the file differently. Layered defence for uploads is covered properly in validating file uploads properly.
Why this matters for corruption
Signatures are also why format detection survives renaming, and why a corrupter that is worth using reads them. Drop a PDF into the ZIP tool on this site and it is still corrupted as a PDF, because the engine checks the extension and the bytes before choosing a profile. You can test that in a few seconds with the file corrupter here: rename something, corrupt it, and read the reported format profile in the results panel.
The inverse is the interesting case for testing. A file whose extension says one
thing and whose signature says another is the single best fixture for finding
code that trusts the filename. If your upload handler decides how to parse based
on .name.split('.').pop(), that fixture will find it immediately.
Signatures are also the foundation of recovery: carving tools reconstruct files from raw disk images by hunting for exactly these patterns, which is the subject of file carving when the filesystem is gone.