❯ Your Monitoring Sucks: What It's Really Telling You (and Missing)
by Jonas Reyes — the builder's desk, Side Quest Studios
November 12, 2026
Alright, settle down. You're probably running some kind of "observability platform" or "AI-powered anomaly detection" and thinking you've got a handle on your systems. Let me save you some time: you don't. Your fancy dashboards are showing you what worked when you set them up, and they're missing everything that's about to break your weekend. We're going to talk about what monitoring actually means, what those blinking lights are lying about, and how to get real signals from the noise. Because the only thing worse than no monitoring is bad monitoring that gives you a false sense of security.
The Vendor Dream vs. Your Reality
Every vendor pitch deck has a slide about "proactive anomaly detection" and "AI-driven insights." They show you pretty graphs where a line suddenly spikes, and their AI "magically" flags it. Cute. In the real world, your systems don't behave like perfect sine waves. They have Tuesday afternoon slumps, end-of-month spikes, and that one cron job that takes 4 hours on the third Sunday of every month. Your "AI" sees a spike and screams. You see Tuesday. This isn't trust; this is a boy crying wolf with a GPU.
The core problem is that most monitoring is set up for known unknowns. You know a disk can fill up, so you alert on df -h /dev/sda1 hitting 90%. You know CPU can spike, so you alert on sar 1 5 showing 95% us or sy for five minutes straight. This is essential, foundational stuff. If you're not doing this, go back to basics. Seriously, node_exporter and Prometheus are your friends here. Get top output into a graph. Know your baselines.
But what about the unknown unknowns? The weird interaction between two microservices that only happens when a specific user segment hits a specific API endpoint with a specific payload on a leap year? Your dashboards don't have a panel for that because you didn't know to build one. That's where most monitoring falls down. It's a rearview mirror, showing you where you've been, not where you're going or what's coming up behind you.
What Your Metrics Are Lying About
"My CPU utilization is 20%, everything's fine!" No, it's not. That 20% could be 18% idle and 2% system, meaning your kernel is thrashing, waiting on I/O, or swapping like crazy. Or it could be 20% user spread across 100 processes, each barely sipping. Or it could be 20% user on one single-threaded process that's completely CPU-bound while 79 other cores are twiddling their thumbs. Metrics without context are dangerous.
Here's what you need to actually look at:
- Load Average, Not Just CPU %:
uptimegives you load average. A high load average (e.g., 5-minute average > number of cores) often indicates processes waiting for something. It could be CPU, disk I/O, network, or even memory. It's a better indicator of overall system stress than a raw CPU percentage. Checkcat /proc/loadavg. - I/O Wait (iowait): If your
iostat -x 1shows highiowait, your processes are sitting around waiting for disk. This isn't a CPU problem; it's a storage problem. Your shiny NVMe might be getting hammered, or your network storage is lagging. This is silent killer number one. - Context Switches:
vmstat 1will show youcs(context switches per second). High context switches mean your kernel is spending a lot of time switching between tasks. This can be normal under heavy load, but an unusually high number relative to your baseline could indicate a poorly tuned application, excessive thread creation, or contention. - Memory Swapping:
vmstat 1also showssiandso(swap in/out). Any sustained non-zero swap activity on a production server means you're out of memory or have a memory leak. Period. Full stop. Fix it. This kills performance far faster than high CPU. - Network Queue Lengths: Your
netstat -sorss -soutput can reveal dropped packets or full queues. Hightxqueuelenonifconfigoutput might indicate a bottleneck at the network card or driver level. This is often ignored until users complain about slow responses.
You want to scale AI models? You want "trustworthy AI"? aitrends.com: Advance trustworthy AI and ML and identify best practices for scaling AI Great. If your underlying infrastructure is lying to you, your "trustworthy AI" is built on quicksand. You need to understand the why behind your system's behavior, not just the what.
The Missing Piece: Your Application's State
Your infrastructure metrics are crucial, but they're a blunt instrument. A healthy server doesn't mean a healthy application. Your Python script could be consuming 100% CPU on one core, but if it's stuck in an infinite loop, it's doing nothing useful. Your database server might have low CPU, but if it's deadlocked, your users are seeing errors.
This is where application-level monitoring comes in. Not just "is the process running?" but "is the process working?"
- Custom Metrics from Your Code: Instrument your Python code. Don't rely on black-box magic. Use
prometheus_clientto expose custom metrics:request_duration_seconds_bucketfor API response times.items_processed_totalfor batch job progress.cache_hit_ratiofor your Redis or Memcached usage.database_query_errors_totalfor explicit database failures. These are the true indicators of your application's health. Build them in from day one. If you're using FastAPI or Flask, it's a few lines of code to expose these endpoints.
- Log Analysis: Your logs are a treasure trove. Stop just archiving them. Get them into a system like ELK (Elasticsearch, Logstash, Kibana) or Grafana Loki. Don't just grep for "ERROR." Look for patterns.
- Rate of specific log messages: A sudden surge in "User authentication failed" could indicate a credential problem or a DDoS attempt.
- Latency in log messages: If your
Processing request X took Nmsmessages suddenly show N spiking, you've got a performance issue before your HTTP endpoint metrics even catch it. - Unusual values in structured logs: If you're logging JSON, look for unexpected values in key fields. This is harder, but powerful for detecting subtle data corruption or bad inputs.
- Distributed Tracing: For complex microservice architectures, tracing is non-negotiable. Jaeger or OpenTelemetry. See exactly which service took how long, where the bottlenecks are, and how requests flow through your system. This is how you debug "the system is slow" from "is the network slow?" to "is
service-alphacallingservice-beta100 times instead of once?"
Don't just collect data. Analyze it. Look for deviations from normal behavior, not just hard thresholds. Normal behavior for your system changes.
The Problem with "AI Ethics" and Hiring: It's All About Data Bias
Speaking of AI, there's a lot of noise about "AI ethics" and guarding against bias, especially in hiring aitrends.com: Promise and perils of using AI for hiring guard against data bias. They talk about how engineers see things in black and white aitrends.com: Getting government AI engineers to tune into AI ethics seen as challenge. Here's my take: most of this "ethics" talk is just a fancy way of saying "your training data is garbage."
If your AI model for hiring is biased, it's because the historical data you fed it was biased. It learned to discriminate from the existing human-driven discrimination embedded in past hiring decisions. The AI didn't invent bias; it merely reflected it.
The same principle applies to your monitoring. If you're only monitoring what has been a problem, your "AI-powered anomaly detection" will only find variations of past problems. It won't find the truly novel failures, the ones that blindside you, because it hasn't seen them before. The "novelty" in Go, as some AI folks like to discuss aitrends.com: Novelty in the game of go provides bright insights for AI and autonomous vehicles, is what breaks systems.
You need to build monitoring that can surface unexpected patterns, not just deviations from a static baseline. This means:
- Entropy in logs: Look for sudden increases in the number of unique log messages or unique values within specific log fields. A new error message you've never seen before is a critical signal.
- Correlation across disparate metrics: If network latency on one host suddenly spikes and database connections drop and your Python workers start erroring, those three signals together are far more powerful than any one in isolation. This is where a decent time-series database and query language (Prometheus's PromQL, for instance) truly shines.
- Outlier detection on distributions, not just averages: The average response time might look fine, but if the 99th percentile jumps from 200ms to 2 seconds, your users are suffering. Don't just look at means; look at tails.
Stop Dithering, Start Shipping
You want to build resilient systems? You want to avoid the 3 AM pager call? Then stop relying on whatever "observability solution" your sales rep pushed on you. Get your hands dirty.
- Know your system's guts: Log into those servers. Run
dstat,atop,strace,lsof. Understand what those commands mean. - Instrument your code: Add those custom metrics. Print meaningful structured logs.
- Build simple, robust alerts: Start with
df -h,free -h,uptime,iostat. Then layer on your application-specific metrics. - Practice incident response: Your monitoring is only as good as your response to it. When an alert fires, do you know what to do? Have you written runbooks?
This isn't rocket science. It's disciplined engineering. The tools are out there: Linux, Python, Prometheus, Grafana, Loki. They're open source, they work, and they don't lie about "AI-powered magic." They just show you the data. Your job is to understand it, and then fix the damn problem. Now go ship something useful.
by Terminal King — the terminal, Side Quest Studios
References
- aitrends.com: Advance trustworthy AI and ML and identify best practices for scaling AI
- aitrends.com: Promise and perils of using AI for hiring guard against data bias
- aitrends.com: Getting government AI engineers to tune into AI ethics seen as challenge
- aitrends.com: Novelty in the game of go provides bright insights for AI and autonomous vehicles