❯ Your Backups Are Hope, Not a Plan: Test Them Now.

by Jonas Reyes — the builder's desk, Side Quest Studios
November 15, 2026

Look, I get it. You’re building something cool. Maybe it’s an AI agent that buys shoes 1, or a whole platform like LangChain 2. You're shipping, iterating, pushing code, probably working 16-hour days. Backups? Yeah, you set up a cron job, a rsync script, maybe even pay some vendor for a “fully managed, enterprise-grade, zero-RTO, blockchain-secured, AI-powered backup solution.” Great. Now tell me: when was the last time you actually restored from one of those? Not a drill, not a simulated tabletop exercise. I mean a full, bare-metal, “oh shit, production is down and I need to get it back NOW” restoration. If the answer isn't "last month," you don't have backups. You have wishes. And wishes don't keep your business running when the cyberattack hits 3.

We’ve all seen the graveyard of AI projects 4. Don't let yours be another headstone because you skipped the boring but critical stuff. A backup you haven't restored is a hope, not a backup. Let's make it real.

Backups Are a Process, Not a Command

You typed tar czf /mnt/backup/data.tar.gz /var/www/html once. Good start. But that’s not a backup system. A system involves: 1. What to back up: all your data, configuration, code, and OS state. 2. Where to store it: multiple locations, off-site, immutable. 3. How often: frequency based on data change rate and recovery objectives. 4. How to restore: the actual, documented, tested procedure. 5. Verification: ensuring the backup is actually valid and readable.

Most people stop at #1 and #2, maybe #3. That's a rookie mistake. The value isn't in making the copy; it's in retrieving it.

Your Data: The Crown Jewels

Forget your git repo for a second. That's code. Important, yes. But your data is unique. It's your database, user uploads, logs, trained models, vector embeddings. If you lose that, you're dead.

For databases like PostgreSQL or MySQL, don't just dump the files. Use the native tools. * PostgreSQL: pg_dump -Fc mydb > mydb.dump for a custom format, or pg_dumpall > all_dbs.sql for everything. For large databases, consider pg_basebackup for continuous archiving (WAL shipping). * MySQL: mysqldump -u user -p mydb > mydb.sql. For point-in-time recovery, you need binary logging enabled.

For filesystems, rsync is your friend. It’s fast, incremental, and reliable. rsync -avzh --delete /path/to/source/ user@backup-server:/path/to/destination/ The --delete flag is crucial for keeping the destination a mirror of the source, removing files that no longer exist. Be careful with it. Test it first.

For block-level backups, LVM snapshots are gold on Linux. lvcreate --size 10G --snapshot --name my_snapshot /dev/vg/my_lv Then you can tar or rsync from the snapshot, ensuring data consistency, and lvremove when done.

Configuration: The Secret Sauce

You spent hours tweaking Nginx, configuring your Python environment, setting up cron jobs. These aren't just files; they're the soul of your operation. Don't forget them. * /etc/: Everything here. rsync -avzh /etc/ user@backup-server:/etc_backup/ is a good start. * /var/www/ or your application's root. * ~/.ssh/, ~/.bashrc, ~/.profile for your critical user accounts (e.g., your deployment user). * Your Dockerfiles, docker-compose.ymls, Kubernetes manifests. These are configuration.

My shortcut for configs: etckeeper. It puts /etc under version control (Git, by default). Makes tracking changes and restoring individual files trivial. Install it, run etckeeper init, and it’ll commit changes automatically. Lifesaver.

Off-Site, Immutable, and Isolated

Your backup server shouldn't be in the same rack, or even the same data center, as your production. If your data center floods, or a ransomware gang encrypts your primary storage, you need a physically separate copy.

Storage options: * Another server you own: Simple rsync target. Make sure it's secure. SSH keys only, no passwords. * Cloud storage: S3, Backblaze B2, whatever. Use rclone to sync your tar files or pg_dump outputs. rclone sync /path/to/local/ s3:your-bucket/backups/ --immutable (check your cloud provider's immutable storage options). Immutable storage is key. It prevents ransomware from deleting or modifying your backups.

Isolation is non-negotiable. Your backup user on the backup server should only have write access, not delete. If your primary server is compromised, you don't want the attacker to wipe your backups. Reverse rsync (rsync -avzh user@prod-server:/path/to/source/ /path/to/backup/destination/) is one way to achieve this pull model. Or, if pushing from prod, make sure the target permissions prevent deletion.

The Restore Test: This Is Where the Rubber Meets the Road

This is the non-negotiable part. Schedule it. Treat it like a fire drill. * Regularity: Once a month, at least. If you change your data schema or infrastructure significantly, do it sooner. * Environment: Don't restore to production. Spin up a new VM, a new Docker container, a new cloud instance. Make it a clean slate. * Documentation: Write down every single step. What commands did you run? What files did you edit? What dependencies did you install? This becomes your disaster recovery runbook. * Time it: How long does it actually take? Your Recovery Time Objective (RTO) depends on this. If it takes 8 hours to restore, and your business can only tolerate 4 hours of downtime, you have a problem. * Verify: Did the application start? Can you log in? Is the data consistent? Run your tests.

My standard restore drill: 1. Provision a fresh Linux VM (e.g., Ubuntu LTS). 2. Install all OS-level dependencies (Python, PostgreSQL, Nginx, etc.). 3. Pull latest configuration backups (/etc, application configs). 4. Restore database dump (psql -f mydb.sql or mysql -u user -p mydb < mydb.sql). 5. Restore application code (from git). 6. Restore user-generated content/file uploads (rsync). 7. Start application. 8. Run sanity checks and smoke tests.

If you can't get your application fully functional from a fresh install and your backups, then your backups are broken. Period.

Pitfalls I've Seen (and Fixed) at 3 AM

  • Permissions: Restoring files but forgetting chown -R www-data:www-data /var/www/html. Application fails silently.
  • Encoding issues: pg_dump in UTF-8, but the new database was created with LATIN1. Data corruption on restore. Always specify encoding when creating new DBs.
  • Missing dependencies: The new VM doesn't have python3-psycopg2 installed, or a critical system library. Application won't even start. Your restore script needs to handle this.
  • Disk space: Not enough space on the restore target for the database dump or file backups. Always check free space.
  • Broken tar archives: A tar file that was corrupted during transfer. Always verify integrity with tar tvf archive.tar.gz > /dev/null. If it errors, it's bad.
  • "Cloud vendor does it for us": Your cloud provider snapshots are great, but are they your backups? Can you restore only your application data and configs, or do you have to restore a whole VM image? What if their service is down? Don't put all your eggs in one basket. Have an independent backup strategy.

This isn't theory. This is the stuff that saves your business when the shit hits the fan. You're building an empire, not a sandcastle. Build it to last. Go test your backups. Now.

by Terminal King — the terminal, Side Quest Studios

References

💬 Discuss this article in the community — 0 replies →
Comments live on community.sqs.chat — one thread per article.