❯ No Alerts Is Not Healthy: How to Find the Silent Failures

by Jonas Reyes — the builder's desk, Side Quest Studios
November 14, 2026

You've got a dashboard glowing green. No PagerDuty alerts. Your Slack channel is quiet. "Everything's fine," you think. "My agents are agenting, my LangChain is chaining, my AI is not killing me in my sleep yet." That's your first mistake. A quiet system isn't a healthy system; it's a silent system. And silent systems are where the real bombs are ticking. I’ve seen enough 3 AM calls to know that “no alerts” often means “no instrumentation where it matters.” This isn't about some hypothetical AI doomsday scenario OpenAI board member makes a terrifying prediction; it's about the everyday, quiet failures that cost you real money, real customers, and real sleep.

The Illusion of Green Dashboards

Remember the last time a vendor promised you "observability" out of the box? I do. It always translates to a pretty graph of CPU utilization and network I/O, maybe some HTTP 200 counts. Great for confirming your server is still plugged in. Useless for telling you if your business logic is actually working.

Take a common scenario: you have a backend service that processes user uploads. Your metrics show: * cpu_utilization: steady 20% * memory_usage: steady 10GB * network_rx_bytes: steady, low * http_requests_total: thousands per minute, all 200 OK * http_requests_errors_total: zero

Looks great, right? Your service is humming. But what if the upload processing itself silently failed for 10% of users because a dependency choked, or a new file format wasn't handled, or a database transaction hung and timed out without bubbling up an HTTP 500? Your metrics wouldn't show it. Your users would. Eventually.

This is the quiet failure. It's not a crash; it's a slow bleed. It's the reason why LangChain, with its 7,000 customers and $3B valuation AI agent startup LangChain hits $3B valuation, still needs robust monitoring beyond just "is the service up?"

Beyond Infrastructure: Business Logic Metrics

Your systems exist to achieve business outcomes. Your monitoring must reflect that. Stop looking at server health and start looking at transactional success.

What to do: 1. Identify Key Transactions: What are the 3-5 most critical user flows or background jobs in your system? For an e-commerce site, it might be checkout_complete, item_added_to_cart, payment_processed. For an AI agent system, agent_task_completed, model_inference_successful, data_ingestion_success. 2. Instrument the Success: Add explicit metrics at the end of each successful transaction. Not just "request received," but "order processed" or "agent task finalized." * In Python, using prometheus_client is a no-brainer. ```python from prometheus_client import Counter, Gauge, generate_latest

    # Define a counter for successful orders
    orders_processed_total = Counter(
        'orders_processed_total', 'Total number of successful orders'
    )

    # Inside your order processing logic:
    try:
        # ... do payment, database update, etc. ...
        orders_processed_total.inc() # Increment ONLY on success
    except Exception as e:
        # ... handle error, maybe another counter for failed orders ...
        pass
    ```
  1. Instrument the Failure (Explicitly): For every success metric, have an explicit failure metric. orders_failed_total, agent_inference_errors_total. Don't just rely on HTTP 500s. A database deadlock that rolls back a transaction might still return an HTTP 200 to the user but result in a failed order.
  2. Define SLOs (Service Level Objectives): Based on these business metrics, set targets. "99.9% of orders must be processed successfully within 5 seconds." Alert when you drop below that.

This is where the magic happens. Your server can be green, but if orders_processed_total flatlines or orders_failed_total spikes, you know you have a problem that matters.

The "Zombie Process" Problem

Another classic quiet failure: the process that's "running" but not actually doing anything useful. It's using CPU, it's holding memory, but it's stuck. Maybe it lost its connection to a database, or a queue, or an external API, and it's in a retry loop that's never going to succeed.

Your standard systemd unit file might show it as active (running). ps aux | grep my_service shows it there. But it’s a zombie, consuming resources and generating no value.

What to do: 1. Heartbeat Files: Have your critical long-running processes periodically touch a file. ```python import time import os

HEARTBEAT_FILE = "/var/run/my_service.heartbeat"
while True:
    with open(HEARTBEAT_FILE, "w") as f:
        f.write(str(time.time()))
    time.sleep(60) # Update every minute
```
Then, a simple cron job can check this file's age:
```bash
# /etc/cron.d/check_my_service_heartbeat
*/5 * * * * root find /var/run/my_service.heartbeat -mmin +5 -exec systemctl restart my_service \;
```
This restarts your service if the heartbeat file hasn't been updated in 5 minutes. Blunt, effective, and it’s saved me more times than I can count.
  1. Internal Health Checks: Build a /healthz or /metrics endpoint into every service. This endpoint shouldn't just return 200 OK. It should check connections to its critical dependencies: database, message queues, external APIs, local caches.

A simple Python Flask example: ```python from flask import Flask, jsonify import redis import psycopg2app = Flask(name) redis_client = redis.Redis(host='localhost', port=6379) db_conn_string = "dbname=mydb user=myuser password=mypass host=localhost"@app.route('/healthz') def health_check(): status = {"redis": False, "database": False} try: redis_client.ping() status["redis"] = True except Exception: pass # Log the error, but don't fail the whole health check yet

try:
    conn = psycopg2.connect(db_conn_string, connect_timeout=1)
    conn.close()
    status["database"] = True
except Exception:
    pass

if all(status.values()):
    return jsonify({"status": "healthy", **status}), 200
else:
    return jsonify({"status": "unhealthy", **status}), 503

if name == 'main': app.run(host='0.0.0.0', port=8080) `` Then, your load balancer or a separate monitoring agent (likePrometheuswithblackbox_exporter) hits this/healthz` endpoint. If it returns anything but 200, or if a critical dependency within it fails, then you alert.

Log Analysis: The Underrated Detective

Logs are gold. Every problem eventually shows up in a log file. The challenge is sifting through the noise. Don't just dump logs to /dev/null or expect a human to read terabytes of JSON.

What to do: 1. Structured Logging: Start using structured logs. JSON is your friend. ```python import logging import json

class JsonFormatter(logging.Formatter):
    def format(self, record):
        log_record = {
            "timestamp": self.formatTime(record, self.datefmt),
            "level": record.levelname,
            "message": record.getMessage(),
            "service": "my_app",
            "component": record.name,
            "pid": record.process,
            # Add custom fields like user_id, transaction_id here
        }
        if record.exc_info:
            log_record["exception"] = self.formatException(record.exc_info)
        return json.dumps(log_record)

handler = logging.StreamHandler()
handler.setFormatter(JsonFormatter())
logger = logging.getLogger(__name__)
logger.addHandler(handler)
logger.setLevel(logging.INFO)

logger.info("User requested QR code generation", extra={"user_id": "abc-123", "qr_type": "privacy"})
logger.error("Failed to connect to QR API", extra={"user_id": "abc-123", "api_error": "timeout"})
```
This makes logs machine-readable and searchable. Forget `grep -r "ERROR"` across 100 servers.
  1. Centralized Logging: Ship all your logs to a central system: Elasticsearch, Loki, Splunk, whatever. Just get them out of /var/log on individual machines.
  2. Anomaly Detection on Logs: Once centralized, set up alerts for patterns in logs.
    • A sudden spike in ERROR level logs from a specific component.
    • A significant drop in INFO level logs (e.g., "processed X items") from a worker that's supposed to be busy. This signals a silent stall.
    • Specific keywords that indicate a problem your metrics might miss (e.g., "connection reset by peer," "disk full," "out of memory," "invalid data format").

This means you’re not just looking for ERROR strings, but for deviations from normal log behavior. A process that stops logging is often just as bad as one that starts spewing errors.

Trust Your Gut, But Verify with Data

My systems have failed in ways no vendor pitch deck ever prepared me for. They fail because of bad data, race conditions, unexpected external API changes, and silent resource exhaustion. They fail quietly, sometimes for hours or days, before a customer complains or a revenue report looks off.

Don't let a green dashboard lull you into a false sense of security. Dig deeper. Instrument your business logic. Build explicit health checks. Watch your logs like a hawk. The only good quiet system is one you know is quiet because it’s working as expected, not one that’s quiet because it’s silently dying.

Ship it.

by Terminal King — the terminal, Side Quest Studios

References

💬 Discuss this article in the community — 0 replies →
Comments live on community.sqs.chat — one thread per article.