❯ The lock file that outlived the process

A customer paid us $129 for a server. The server came up. The DNS resolved. And then the order just sat there, "provisioning", for twenty minutes, completely stuck — with no error a human would notice and nothing in the logs that said stop.

The cause was a lock file. Specifically, a lock file that was telling the truth about a process that no longer existed.

What a lock is for

You put a lock file in place so two copies of a job don't run at once — a simple, correct instinct. The job starts, writes its PID to /tmp/provisioner.lock, does its work, deletes the lock on exit. The next run checks the file first: if a live process owns it, stand down; if not, take over. Textbook.

The problem is the middle of that sentence. "If a live process owns it" is doing a lot of work.

How it lies

Our job ran in a container. The container got recreated — a deploy, a crash, doesn't matter. In the new container, PID 10 was now a completely different process. The lock file still said 10. So the check asked "is PID 10 alive?" — yes, it was — and concluded the job was already running. It wasn't. The real job had died with the old container; the lock outlived the process and started blocking every retry.

Every two minutes the cron fired, saw the "running" lock, and quietly exited. A stuck job that never errors is the worst kind: it looks like patience, and it is actually a deadlock.

The fix is a heartbeat, not a smarter PID check

You can make the PID check cleverer — compare the process command line, check the start time — and it will still be wrong eventually. The durable fix is simpler: the lock has to prove it's in use, not just that some number is alive.

So the loop now touches the lock file's timestamp on every iteration. The lock has a heartbeat. If the heartbeat goes quiet past a threshold, the lock is stale and gets stolen. A dead process can't touch a file. A live one does. That's the whole test.

while True:
    touch(lock)          # heartbeat — I am still working
    do_the_batch()
    sleep(30)

Now a hung job can't block retries forever, a long legitimate batch isn't mistaken for a dead one (it keeps beating), and a fresh worker takes over cleanly.

The general lesson

Any lock, lease, or "is it running?" flag that has no heartbeat is a trap waiting for the first ungraceful exit. Processes die without cleaning up — that's not a bug in your code, it's physics. If your coordination depends on a process cleaning up after itself, your coordination is a hypothesis, not a guarantee.

Add the heartbeat. Make staleness a first-class state you can detect and recover from. And while you're at it: assert the outcome. Don't let a stuck job look like a working one.

References

• Docker — pid namespaces & container lifecycle
• Wikipedia — Lock (computer science)
• Google SRE Book — distributed coordination & failure modes

by Jonas Reyes — the builder's desk, Side Quest Studios
AI-assisted, curated for Side Quest Studios.