Observability: Metrics, Logs & Traces (The Three Pillars)

The 3am question monitoring can't answer

It's 3am. Your dashboard is red: error rate is up, latency is climbing. Monitoring did its job, it told you *something* is wrong. Then comes the question monitoring cannot answer: why? Which service? Which dependency? Which one customer's request set off the cascade? You start SSHing into boxes and grepping logs by hand, and the incident drags on for an hour that should have been five minutes.

That gap, between *knowing something broke* and *understanding why*, is exactly the gap observability fills. It's one of the clearest dividing lines between an engineer who can keep a system alive and one who can only watch it die. This article builds the mental model from zero: what observability actually means, the three pillars it rests on, what to measure, and how to set targets you can defend.

Monitoring vs observability, the one distinction that matters

Monitoring answers questions you already knew to ask. Observability lets you ask new questions of your system without shipping new code to answer them.

Monitoring is dashboards and alerts for known failure modes: CPU over 80%, disk almost full, error rate above threshold. You decided in advance what to watch. That's necessary, but production fails in ways nobody predicted. Observability is the property that, when something *unexpected* happens, you can explore the system's actual behaviour and figure it out from the data you're already emitting.

The practical upshot: you build observability so that the *next* incident, the one you can't imagine yet, is debuggable with data you're already collecting. You don't get a second chance to instrument an outage that already happened.

How telemetry actually flows

Before the three pillars, see the shape of the whole system. Your application emits three kinds of telemetry. They travel through a collector, land in backends suited to each data type, and surface as dashboards and alerts. Follow the flow:

Notice the division of labour: metrics drive alerts because they're cheap to aggregate, logs and traces are where you investigate once an alert fires. A mature setup links them, an alert on a metric jumps you to the relevant traces, which link to the exact log lines. That stitched-together path is what turns a one-hour incident into a five-minute one.

The three pillars, side by side

Metrics, logs, and traces are not competing choices, they answer different questions, and you need all three. The fastest way to internalise them is to ask what each one is *good at*:

Metrics, cheap, aggregated, alert-friendly

A metric is a number measured over time: request count, latency, CPU, queue depth. Because they're pre-aggregated, metrics are cheap to store and fast to query, which makes them ideal for dashboards and alerts. Their weakness is that aggregation throws away detail, a metric tells you 3% of requests failed, never *which* ones or *why*.

Logs, high detail, the per-event truth

A log line is a record of one event. Logs carry the detail metrics lose. The single biggest upgrade you can make is structured logging, emit JSON, not free text, so you can filter and aggregate. {"level":"error","user":8842,"reason":"card_declined"} is queryable; payment failed for user is a needle in a haystack.

{
  "ts": "2026-06-05T03:11:42Z",
  "level": "error",
  "service": "checkout",
  "trace_id": "a1b2c3d4",
  "user_id": 8842,
  "event": "payment_failed",
  "reason": "card_declined",
  "latency_ms": 1340
}

Traces, following one request through everything

In a system with many services, one user action triggers a chain of internal calls. A trace records that whole journey as a tree of spans, each span is one operation, with timing. Traces are how you answer "the checkout is slow, *which* of the eight services it touches is the culprit?" The trace_id in that log line above is what ties the pillars together: from a slow trace you jump straight to its logs.

What to actually measure: the four golden signals

You could measure a thousand things. Google's SRE book distilled what matters for any user-facing service down to four. If you only instrument these, you'll catch the overwhelming majority of real problems:

• Latency, how long requests take. Track p50, p95, p99, and crucially split *successful* from *failed* requests (a fast failure can hide a slow success).
• Traffic, how much demand the system is under: requests per second, transactions per minute.
• Errors, the rate of failed requests. Include both explicit failures (HTTP 500s) and implicit ones (wrong answers, policy violations).
• Saturation, how "full" the system is: CPU, memory, queue depth, connection pool usage. The leading indicator of impending failure.

SLI, SLO, SLA, measuring 'good enough' on purpose

Metrics tell you what's happening. SLOs tell you whether what's happening is acceptable, and they turn "is the site okay?" from a gut feeling into a number. The three terms get mixed up constantly, so here they are in plain language:

The genius idea that falls out of SLOs is the error budget. A 99.9% monthly SLO means you're *allowed* to be bad 0.1% of the time, roughly 43 minutes a month. That budget is a tool: if you have budget left, ship features fast. If you've burned it, freeze risky changes and spend on reliability. It ends the eternal feature-vs-stability argument with math instead of opinions.

Common mistakes that cost hours

1. Alerting on causes instead of symptoms. An alert on "CPU > 80%" pages you for something users may never feel. Alert on what users experience, error rate, latency SLO burn, and use causes for investigation, not paging.
2. Logging unstructured text. log.info("user " + id + " failed") can't be filtered or aggregated. Emit structured JSON with consistent field names from day one; retrofitting it across a codebase is miserable.
3. High-cardinality metric labels. Putting user IDs or request IDs in metric labels explodes storage and cost. Keep labels low-cardinality; push the detail to logs and traces.
4. No trace IDs linking the pillars. Without a shared trace_id propagated across services and into logs, your three pillars are three islands and every investigation starts from scratch.
5. Alert fatigue. Hundreds of noisy, non-actionable alerts train people to ignore the pager. Every alert should be actionable and tied to user impact, or it shouldn't page. (More on this in the on-call article below.)

Where to go next

Observability is the sense organ of a reliable system. Once you can see clearly, the next step is building systems that fail gracefully, and running the humans who respond when they do:

• Make systems survive the failures you can now *see*: Reliability & Resilience: Designing for Failure.
• See observability in the context of running real clusters: Kubernetes in Production.
• Turn signals into a humane response process: Incident Management & On-Call.
• Get hands-on inspecting a live system: the kubectl Lab lets you query pod health and logs the way you would during an incident.

Instrument one service with the four golden signals this week. The next time it misbehaves, you'll be reading data instead of guessing.