Backups you can actually restore
A backup that has never been restored is a guess. Here is the drill we run every month, and how to run your own.
By ByteRoast Engineering · 4 min read
Most backup conversations are about schedules and retention. The only question that matters is whether the restore works. Everything else is bookkeeping.
The four-minute drill
- Pick a snapshot from the previous night. Not a special one — a normal one.
- Restore it to a new server, not over production.
- Boot the server and run a checksum or a smoke test against the data.
- Destroy the copy and write down how long it took.
If step three fails, you do not have a backup. You have a scheduled job with optimistic output.
What breaks restores
- Secrets. The application boots, but the environment file was excluded or the key is wrong.
- External state. The database restored, but the object storage bucket did not.
- Time. The restore works, but it takes six hours and nobody has budgeted for it.
- Permissions. Files restored with the wrong owner, so the service starts and immediately fails.
How we run it
Every month we restore a sample of production snapshots into an isolated network and verify that the machines boot and serve traffic. We record the result. If a drill fails, it is an incident with a cause and a fix, not a footnote.
Do this on your own servers
Put the drill in the onboarding checklist. Run it before a big release, not after. The goal is not a perfect backup system; it is a team that knows exactly what happens when the disk dies.
Get early access
Servers, DNS and backups. Riyadh region live.