❯ A Backup You Don't Restore Is Just Hope. Test Your Backups.
by Jonas Reyes — the builder's desk, Side Quest Studios
October 31, 2026
Alright, settle in. We’re talking backups. Not the "set it and forget it" brochureware you get pitched at every conference, but the kind that actually saves your ass at 3 AM. Because let's be blunt: a backup you haven’t restored is not a backup. It’s a hope. And hope, my friends, is not a disaster recovery plan.
I’ve seen enough systems go sideways to know the difference. The hard drive that spun itself into a metallic shriek. The accidental rm -rf / on a production box (don't ask). The database corruption that looked like cosmic rays had taken a personal interest in our data integrity. In every single one of those moments, the only thing that mattered was: could we get the data back? And more importantly, could we get it back fast and correctly?
The industry is currently buzzing about AI agents that might share your banking password Bank of America or the staggering list of risks Anthropic is filing for their IPO, including "weapons, blackmail, and rogue agents" Anthropic Risks. All very interesting, all very future-facing. But you know what’s a far more immediate and common risk than your AI agent blackmailing you? Your primary data store disappearing. And you know what saves you from that? A verified backup.
Stop Thinking, Start Testing: The Only Real Backup Strategy
Forget the vendor pitch decks. Forget the "industry best practices" that sound suspiciously like "buy our expensive solution." We’re talking about what works when the chips are down. And what works is simple: restore your backups, regularly.
You wouldn't buy a fire extinguisher and never check its pressure gauge, would you? You wouldn't install an alarm system and never test if it actually calls the police. So why do people treat backups like a mystical incantation that just works? It doesn't. You have to prove it.
This isn't just about the data. It's about the process. When you’re under pressure, adrenaline pumping, the last thing you want to be doing is reading a poorly documented restore procedure written three years ago by someone who quit to join a crypto startup. You want muscle memory. You want a script that runs. You want confidence.
The Linux Way: rsync and tar Are Your Friends
For most of what we do, the core tools are old, reliable, and battle-tested. You don't need some proprietary agent daemon eating 20% of your CPU.
File-level backups: For simple filesystems, rsync is your absolute best friend. It’s fast, efficient, and handles incremental backups like a pro.
# Full backup to a remote server
rsync -avz --delete /source/path/ user@backup.server:/dest/path/full/
# Incremental backup (using hard links for efficiency)
# This creates a full snapshot each time, but only uses space for changed files.
# Requires a previous full backup to exist at /dest/path/current/
rsync -av --link-dest=/dest/path/current/ /source/path/ /dest/path/snapshots/$(date +%Y-%m-%d_%H-%M-%S)/
rm -f /dest/path/current
ln -s /dest/path/snapshots/$(date +%Y-%m-%d_%H-%M-%S)/ /dest/path/current
The --link-dest trick is pure gold. It makes every snapshot look like a full backup, but only changed files consume new disk space. You can roll back to any snapshot instantly.
For archives, tar is the undisputed champion. Compress it with gzip or zstd (for speed) and encrypt it with gpg.
# Encrypt and compress a directory
tar -czf - /path/to/data | gpg --encrypt --recipient "your_key_id" > backup_$(date +%Y%m%d).tar.gz.gpg
Database backups: This is where it gets trickier, but the principles are the same: get a consistent snapshot.
- PostgreSQL:
pg_dumpallfor global objects,pg_dumpfor individual databases. Use the custom format (-Fc) for flexibility.bash pg_dump -Fc -Z 9 my_database > my_database_$(date +%Y%m%d).bakRestore withpg_restore. - MySQL/MariaDB:
mysqldump. Use--single-transactionfor InnoDB tables to get a consistent snapshot without locking.bash mysqldump --single-transaction --all-databases > all_databases_$(date +%Y%m%d).sqlRestore by piping the SQL file back intomysql.
Always, always, always pipe to gzip or zstd to save space and reduce network transfer times.
The Restore Environment: No Shortcuts Here
This is where most people fail. They have a backup, but no place to restore it to, or no clear path for how to restore it.
Rule 1: Dedicated Restore Machine/VM. You need a clean, isolated environment to test your restores. This could be a staging server, a dev VM, or even a Docker container setup specifically for this. It cannot be your production system. Seriously. Don't even think about it.
Rule 2: Automated Restore Script. Write a script that automates the entire restore process. This script should: 1. Pull the latest backup. 2. Decompress/decrypt it. 3. Perform the restore (e.g., pg_restore, mysql -u... < file.sql, rsync). 4. Run validation checks.
Example (simplified for a file backup):
#!/bin/bash
BACKUP_DIR="/mnt/backups/latest/"
RESTORE_TARGET="/tmp/restore_test/"
echo "Starting restore test for $(date)"
mkdir -p $RESTORE_TARGET
# Simulate pulling the latest backup
# In a real scenario, this might involve s3cmd, rclone, or scp
cp -r $BACKUP_DIR/* $RESTORE_TARGET/
# Validate a specific file exists and has expected content
if grep -q "important_string" $RESTORE_TARGET/path/to/important_file; then
echo "Validation successful: important_file contains expected content."
else
echo "Validation FAILED: important_file missing or incorrect content!"
exit 1
fi
# Clean up
rm -rf $RESTORE_TARGET
echo "Restore test completed successfully."
Rule 3: Validation, Validation, Validation. Restoring the files is only half the battle. You need to verify that the restored data is correct and usable. * For databases: Run a few SELECT COUNT(*) queries on critical tables. Try logging into your application against the restored database. * For files: Check file counts, sizes, and crucial content (e.g., specific configuration values). Use diff -r if you have a known good state.
This isn't rocket science, but it requires discipline. Set up a cron job to run your restore test weekly, or at least monthly. Alert on failures. Treat a failed restore test with the same urgency as a production outage. Because that’s what it is: a precursor to a production outage.
Offsite Storage: Because Data Centers Burn Down
I’ve personally seen a data center hit by a small, localized fire. It wasn’t catastrophic, but it was enough to take out a rack of servers. If your backups are in the same physical location as your primary data, you're not doing it right.
Use an object storage solution like S3-compatible storage (MinIO for on-prem, AWS S3, Google Cloud Storage, Backblaze B2, etc.). rclone is your universal adapter here.
# After configuring rclone (rclone config)
rclone copy /local/backup/dir remote:bucket-name/backups/
Encrypt everything before it leaves your machine. Your private keys stay with you. This isn't just about compliance; it's about not trusting anyone else with your raw data.
Don't Fall For The Hype
We see a lot of "AI-powered data management solutions" and "intelligent backup agents" these days. They promise to simplify your life, optimize storage, and predict failures. My experience? They mostly add complexity, introduce black boxes, and cost a fortune.
Stick to the basics. Linux, Python, open source. Tools you understand, tools you can inspect, tools you can script. Your backup strategy should be transparent, auditable, and something you can explain to a junior engineer in five minutes. If it involves a vendor-specific GUI or a "magic" restore button, be very, very skeptical.
The real world is messy. Disks fail, networks drop, humans make mistakes. The only thing standing between you and utter catastrophe is a reliable, tested, and restored backup. Stop hoping. Start proving.
by Terminal King — the terminal, Side Quest Studios