Backups: The Boring Discipline That Saves Your Homelab
Everyone agrees backups matter and almost nobody tests them. That gap — between believing you are protected and actually being protected — is where homelabs go to die. I know, because I lived on the wrong side of it for longer than I would like to admit, running nightly backups I had never once restored and feeling safe for reasons that turned out to be entirely imaginary.
A backup you have never restored is not a backup. It is a hope with a cron job attached. The day you find out whether it works should not be the day you need it.
Nobody wants backups. What people want is restores. The backup is just the unpleasant subscription you pay to have one available.
The night it mattered
The lesson landed properly when a routine update to a stateful container ate its own database. Not corrupted-but-recoverable — gone, the volume mangled in a way that no amount of staring at it was going to fix. The old me would have lost days of work and a chunk of dignity. The slightly-less-old me, who had finally started testing restores, pulled the previous night’s snapshot, restored it into a fresh volume, and was back up in under twenty minutes.
The difference between those two outcomes was not the backup. Both versions of me had a backup. The difference was that one version had proven the restore worked and the other was running on faith. That night converted me permanently. Backups stopped being a box I ticked and became a discipline I actually practise.
The rule that survives contact with reality
The 3-2-1 rule is old, unfashionable, and correct. Three copies of anything you care about. On two different kinds of media or storage. With one copy off-site. It is not clever and that is precisely why it works — it survives the failure modes that actually happen rather than the ones that sound dramatic.
Three copies, because one is no copies and two is one bad coincidence away from none. Two media, because a single failure — a dead disk, a bad filesystem, a fat-fingered rm — should not be able to take every copy at once. One off-site, because fire, theft and flood do not care how good your RAID array is. The whole rule is just a refusal to let any single event reach all of your copies.
What I actually run
The implementation is deliberately boring, because boring survives. Stateful data — databases, container volumes, the Markdown that is the real asset — gets backed up nightly to a local NAS, then replicated encrypted to off-site storage. I use restic because it is a single binary, does deduplication and encryption properly, and its snapshots are content-addressed so I can keep a deep history without the size exploding.
# Nightly: snapshot the stateful volumes, deduplicated and encrypted
restic backup /srv/docker/data \
--exclude-file /etc/restic/excludes.txt \
--tag nightly
# Keep a sensible history without hoarding forever
restic forget --prune \
--keep-daily 7 --keep-weekly 4 --keep-monthly 6
# The line that actually matters — prove a restore works
restic restore latest --target /tmp/restore-test --include /srv/docker/data/atlas
That last command is the whole point. The first two lines are what most people stop at. The third line — restoring into a throwaway directory and checking the data is real — is the line that converts a hope into a backup.
Stateless versus stateful, and why it changes everything
The thing that made backups tractable was separating my world into two kinds of thing. Stateless services hold no unique data — they are defined entirely by their configuration, and that configuration lives in Git. If a stateless container vanishes, I rebuild it from the compose file and lose nothing. I do not back these up at all; I back up the recipe, which is already in version control.
Stateful services hold data that exists nowhere else — the database, the uploaded files, the index. These are the only things that genuinely need backing up, and ruthlessly separating them from the stateless majority shrank my backup problem to a fraction of its original size. Most of my homelab does not need a backup. It needs a git clone and a docker compose up. Only the small, precious core of actual state needs the 3-2-1 treatment, and once you see the line clearly, the whole thing stops being overwhelming.
This is the same instinct as keeping configuration declarative across the homelab: the running machine is a rebuildable artefact, and the only irreplaceable thing is the data it holds.
What it costs, honestly
The cost of backups is not the storage, which is cheap. It is the discipline of testing, which is tedious and easy to skip because nothing bad happens when you skip it — right up until everything bad happens at once. I put a monthly restore test on the calendar precisely because I knew that “I’ll test it when I get round to it” means never. Twenty minutes a month, restoring a random snapshot into a scratch directory and confirming it is real. It is the most boring recurring task I have and the one I would defend most fiercely.
The other honest cost is that backups feel like pure overhead until the exact moment they do not, and that moment is unpredictable. You pay every night for an insurance policy you hope never pays out. The temptation to let it lapse is constant. Resisting that temptation is the entire discipline.
The boring conclusion
There is no clever ending here, because backups are not a clever problem. They are a discipline problem wearing a technical costume. Three copies, two media, one off-site, and a restore you have actually performed with your own hands. Do that and your homelab moves from fragile to recoverable — from a thing that one bad update can destroy to a thing that shrugs off disasters because every disaster is just an excuse to prove the restore still works.
I learned this the way most people do, which is nearly the hard way. I got the warning shot instead of the fatal one, and only because I had started testing restores a few weeks before I needed one. Do not wait for your warning shot. Test the restore this week, while it is boring, so that it is boring the day you actually need it.