❯ Disk Full Panic: The Rsync Habit That Prevents It
by Jonas Reyes — the builder's desk, Side Quest Studios
November 06, 2026
Let's cut the crap. You’ve been there. The pager screams at 3 AM. Disk full. df -h shows 100%. The app is down. Customers are screaming. Your AI model training job just crapped out because the /tmp directory filled up with garbage. You're scrubbing logs, deleting old backups, sweating over every megabyte. It's not a fun party, and frankly, it's preventable.
Everyone’s talking about AI agents planning trips Airbnb, or video-chatting with their digital muse Meta. Great. But none of that matters if your foundational infrastructure is crumbling under its own data weight. You want to ship. You want to iterate. You can’t do that if you’re constantly fighting fires because you neglected the fundamentals.
The single best habit to prevent disk full panics isn't some fancy observability platform or a vendor pitch deck about "AI-driven storage optimization." It’s a simple, consistent, automated rsync to a secondary location. A mirror. A copy. A sanity check. It’s boring, it’s old school, and it works. Every damn time.
Why Your Disk Fills Up (It's Not Always What You Think)
Your disks fill up for a few core reasons, and they rarely involve a benevolent growth in actual, valuable data.
- Runaway Logs: The silent killer. A bug, an misconfigured service, or just verbose debugging left on in production. Suddenly,
/var/logis consuming gigabytes per hour. You wake up to a system that can’t even write to/tmpfor basic operations. - Forgotten Temp Files: Ever run a complex data processing job that dumps intermediate files? Or a Python script that creates massive CSVs for analysis? People forget to clean up. Cron jobs fail.
/tmpbecomes a graveyard of unloved data. - Failed Backups/Snapshots: Irony, isn't it? Your backup solution itself can fill your disk. If your local backup destination isn't pruned, or snapshots aren't rotated, you're just moving the problem around.
- Developer Cruft: Devs pulling down massive datasets for local testing, forgetting to clean up. Large Docker images, build artifacts, npm caches. It accumulates.
The root cause? Lack of a consistent, automated cleanup and offload strategy. You need a second place for critical data, not just for recovery, but for sanity.
Rsync: Your Best Friend for Disk Sanity
Forget complex replication schemes for a moment. We're talking about a simple, reliable way to ensure that if one disk goes belly-up, or even just gets full, you have an identical, recent copy elsewhere. And you can prune that copy without impacting your live system.
rsync is a goddamn marvel. It copies files. It syncs directories. It's efficient, only transferring changes. It handles permissions, timestamps, symlinks. It's been around forever because it's rock-solid. And it's on every Linux box worth its salt.
The Rsync Habit: A Two-Pronged Attack
Here’s the habit. It’s two parts:
- Critical Data Mirror: A daily, or even hourly,
rsyncof your most critical application data, configuration files, and even key log directories to a separate volume, or better yet, a separate machine. - Offload and Prune: A weekly (or more frequent)
rsyncof everything else – old logs, archived data, completed job outputs – to cheap object storage (S3, MinIO, whatever) or another dedicated backup server. Then, and this is key, you delete the originals from your primary disk.
Let’s get concrete.
Part 1: The Critical Data Mirror
You have an application running. It writes to /var/www/app/data, /etc/app/config, and /var/log/app. These directories must be available. If your primary disk fills up, these are the first things that break.
The Script (let’s call it mirror_critical_data.sh):
#!/bin/bash
SOURCE_DIRS="/var/www/app/data /etc/app/config /var/log/app"
DEST_HOST="backup-server.yourdomain.com"
DEST_PATH="/mnt/backup/critical_app_data"
LOG_FILE="/var/log/rsync_critical_data.log"
echo "$(date): Starting critical data rsync..." >> "$LOG_FILE"
for DIR in $SOURCE_DIRS; do
# -a: archive mode (preserves permissions, timestamps, etc.)
# -h: human-readable output
# -v: verbose
# --delete-after: delete extraneous files from dest AFTER transfer completes (safer)
# --exclude: common exclusions you might have. Adjust as needed.
# --log-file: keep a separate log for this rsync run
rsync -ahv --delete-after \
--log-file="$LOG_FILE" \
--exclude='*.tmp' \
--exclude='cache/' \
"$DIR/" "$DEST_HOST:$DEST_PATH$(basename "$DIR")" >> "$LOG_FILE" 2>&1
if [ $? -ne 0 ]; then
echo "$(date): ERROR: rsync failed for $DIR!" >> "$LOG_FILE"
# Add your alerting mechanism here: email, Slack, PagerDuty
# For example: sendmail -t <<EOF
# Subject: CRITICAL RSYNC FAILED
# To: ops@yourdomain.com
#
# Rsync failed for directory $DIR on $(hostname). Check logs in $LOG_FILE.
# EOF
fi
done
echo "$(date): Critical data rsync finished." >> "$LOG_FILE"
What’s happening here: * We’re syncing specific directories. Don’t just rsync /. That’s overkill and dangerous. * --delete-after: This is crucial. If a file is deleted on the source, it will be deleted on the destination after the sync. This keeps your mirror clean. * --exclude: Don’t copy temporary files or caches. You don’t need them on your mirror, and they just waste space. * $DIR/ (note the trailing slash): This means "sync the contents of this directory," not the directory itself. If you omit it, rsync will create $DEST_PATH/$DIR on the remote, which might not be what you want. * Error handling: Yes, you must have it. If rsync fails, you need to know.
Cron It: Put this in your /etc/cron.d/rsync_critical_data or crontab -e:
0 */4 * * * root /usr/local/bin/mirror_critical_data.sh > /dev/null 2>&1
This runs every four hours. Adjust frequency based on your data change rate. For some systems, hourly or even more frequent might be necessary.
Part 2: The Offload and Prune Strategy
This is where you prevent the insidious slow creep of disk usage. Identify directories that accumulate data but don’t need to live on your primary, fast storage forever. Think old logs, completed analysis outputs, staging artifacts.
The Script (let’s call it offload_and_prune.sh):
#!/bin/bash
# Directories to offload and then clean up
SOURCE_DIRS_TO_OFFLOAD="/var/log/old_app_logs /home/user/completed_jobs /opt/app/archive"
OFFLOAD_DEST="s3://your-s3-bucket/offloads/" # Or another remote server: user@remote:/path/
OFFLOAD_TOOL="aws s3 sync" # or `mc mirror` for MinIO, or plain rsync to another server
LOG_FILE="/var/log/offload_and_prune.log"
echo "$(date): Starting offload and prune..." >> "$LOG_FILE"
for DIR in $SOURCE_DIRS_TO_OFFLOAD; do
echo "$(date): Offloading $DIR..." >> "$LOG_FILE"
# Use the appropriate tool for your offload destination
# For S3:
$OFFLOAD_TOOL "$DIR" "$OFFLOAD_DEST$(basename "$DIR")/" >> "$LOG_FILE" 2>&1
# For rsync to another server:
# rsync -ahv --remove-source-files "$DIR/" "$DEST_HOST:$DEST_PATH$(basename "$DIR")" >> "$LOG_FILE" 2>&1
if [ $? -ne 0 ]; then
echo "$(date): ERROR: Offload failed for $DIR!" >> "$LOG_FILE"
# Add your alerting here
else
echo "$(date): Offload of $DIR successful. Proceeding to prune local files..." >> "$LOG_FILE"
# CRITICAL STEP: DELETE OLD FILES LOCALLY AFTER SUCCESSFUL OFFLOAD
# Adjust 'mtime' to how old you want files to be before deletion.
# Here: files older than 7 days.
find "$DIR" -type f -mtime +7 -delete >> "$LOG_FILE" 2>&1
if [ $? -ne 0 ]; then
echo "$(date): WARNING: Pruning of old files in $DIR failed or encountered issues." >> "$LOG_FILE"
# Add alerting for pruning failures if needed
else
echo "$(date): Successfully pruned old files in $DIR." >> "$LOG_FILE"
fi
fi
done
echo "$(date): Offload and prune finished." >> "$LOG_FILE"
Key elements: * $OFFLOAD_TOOL: This could be aws s3 sync, mc mirror (for MinIO), or another rsync command to a separate backup server. Pick your poison. * find "$DIR" -type f -mtime +7 -delete: This is your golden ticket for cleanup. It finds files in $DIR that are older than 7 days (+7) and deletes them. Adjust +7 to your retention policy. Be careful with this. Test it on a non-production directory first. * --remove-source-files (for rsync offload): If you’re using rsync for the offload, this flag will delete the source files after a successful transfer. Use with extreme caution and ensure your destination is reliable. My preference is separate find -delete after the sync, as shown, because it gives more control and separation of concerns.
Cron It:
0 2 * * 5 root /usr/local/bin/offload_and_prune.sh > /dev/null 2>&1
This runs every Friday at 2 AM. A weekly schedule is often good for offloading and pruning, but daily might be