Observability: Metrics, Logs & Traces (The Three Pillars)

The 2 a.m. page nobody can answer

It's 2 a.m. Your phone buzzes: "Checkout latency p99 above 3s." You open the dashboard. Yep, latency is high, the graph agrees with the alert. Now what? *Which* service is slow? Is it the database, a downstream payment API, a single bad host, or one specific customer hammering an endpoint? The dashboard tells you that something is wrong. It can't tell you why. So you start grepping logs across six services, eyeballing timestamps, and guessing.

That gap, between knowing something broke and understanding what broke, is exactly what observability exists to close. And it rests on three kinds of data your systems can emit: metrics, logs, and traces. Get the mental model for these three and most monitoring tools suddenly make sense.

What is observability?

Observability is the ability to understand the internal state of a system purely from the data it emits, so you can answer questions you never thought to ask in advance.

The key phrase is "questions you never thought to ask." Traditional monitoring is built around dashboards and alerts you set up ahead of time, you predict what might go wrong and watch for it. Observability is about having rich enough data that when something *new* goes wrong, you can interrogate the system and find the answer without shipping new code first. The three pillars are simply the three shapes that emitted data comes in.

Vitals tell you *that* the patient is unwell and how unwell. Notes tell you *what specifically* happened and when. The diagnostic journey shows you *where in the process* time was lost. You need all three to actually treat the patient, and you reach for them in that order.

The picture: how telemetry flows

Here's the shape of a typical observability setup. A single service emits all three signal types; they flow to a backend that stores and indexes them; and you consume them through dashboards, alerts, and ad-hoc queries.

1. 1

   Instrument the service

   Add a library (or an agent / OpenTelemetry SDK) that records numbers, writes structured events, and tags each request with a trace ID. This is the one-time setup cost.

2. 2

   Emit three signal types

   On every request the service bumps counters (metrics), writes a JSON log line, and records spans for each operation it performs.

3. 3

   Ship to a backend

   Telemetry is exported, usually over the network, to a backend that ingests, stores, and indexes it. Each signal type is often stored differently (time-series DB for metrics, log store for logs, trace store for spans).

4. 4

   Alert on metrics

   Cheap numeric thresholds fire pages: "error rate > 1%". Metrics are what you alert on because they're fast and cheap to evaluate continuously.

5. 5

   Investigate with logs and traces

   Once an alert fires, you pivot: open the trace for a slow request to find the slow hop, then read the logs from that service at that moment for the why.

The three pillars side by side

The single most useful thing to internalize is what each pillar is *good at* and what it *costs*. The big lever is cardinality, how many distinct label combinations a signal carries. Metrics must stay low-cardinality to stay cheap; logs and traces can be high-cardinality because you only fetch the ones you need.

What they look like in code

Concretely, here's one request handler emitting all three signals. A metric counts and times the request, a structured log records what happened, and a trace span captures the timing of a sub-operation. Note how the same trace_id ties everything together, that's the glue.

import time, logging, json
from opentelemetry import trace, metrics

logger = logging.getLogger("checkout")
tracer = trace.get_tracer("checkout")
meter = metrics.get_meter("checkout")

# --- METRIC: a counter and a latency histogram (low cardinality) ---
requests = meter.create_counter("checkout_requests_total")
latency = meter.create_histogram("checkout_latency_ms")

def handle_checkout(order):
    start = time.time()
    # --- TRACE: a span wraps the work; child spans nest inside ---
    with tracer.start_as_current_span("handle_checkout") as span:
        span.set_attribute("order.id", order.id)          # high-cardinality is OK here
        ctx = span.get_span_context()
        trace_id = format(ctx.trace_id, "032x")

        with tracer.start_as_current_span("charge_payment"):
            charge(order)                                  # timed sub-operation

        elapsed = (time.time() - start) * 1000
        requests.add(1, {"status": "ok"})                 # bounded label only
        latency.record(elapsed, {"status": "ok"})

        # --- LOG: a structured event, correlated by trace_id ---
        logger.info(json.dumps({
            "event": "checkout_completed",
            "order_id": order.id,
            "amount": order.amount,
            "latency_ms": round(elapsed, 1),
            "trace_id": trace_id,
        }))

Read the three differently. The metric lines (requests.add, latency.record) feed dashboards and alerts, they only carry the bounded status label. The log line is a single JSON object you can grep, filter, and parse on any field. The span records how long charge_payment took *inside* the overall request. When you put trace_id on the log too, you can jump from a slow trace straight to its log lines, or vice versa.

Monitoring vs observability

These words get used interchangeably, but the distinction is real and worth holding onto. Monitoring answers known questions. You decide in advance what matters, error rate, CPU, queue depth, build a dashboard and an alert, and watch. It's excellent for the failures you can anticipate.

Observability answers new questions. When something breaks in a way you *didn't* predict, can you slice the data to find the cause without first deploying new instrumentation? That's the test. If the only way to debug an incident is to add logging and redeploy, you have monitoring, not observability.

They're not rivals, observability is the superset. You still build dashboards and alerts (that's the monitoring layer, almost always on metrics). But you back them with logs and traces detailed enough to investigate the unknown. A practical way to scope your alerts is the The Four Golden Signals: latency, traffic, errors, and saturation.

Common mistakes that cost hours

1. Alerting on logs instead of metrics. Scanning log volume to fire alerts is slow and expensive. Alert on cheap numeric metrics; use logs to investigate *after* the page.
2. High-cardinality labels on metrics. Putting user_id, order_id, or request_id on a metric explodes your time-series count and your bill. Those fields belong on logs and traces.
3. Unstructured logs. Free-text log lines can't be queried or aggregated. Emit JSON with consistent field names from day one.
4. No correlation IDs. Without a shared trace_id across services and on your logs, you can't connect a slow trace to the log line that explains it. Propagate it everywhere.
5. Tracing nothing or everything. No traces means no cross-service latency breakdown; 100% sampling at scale is ruinously expensive. Sample intelligently (e.g. keep all errors, sample the rest).
6. Treating dashboards as the whole job. A wall of green graphs feels safe but only covers the questions you already thought of. Real incidents are usually the ones you didn't.

Takeaways

Where to go next

You now have the data model. The next steps are learning the standard way to emit these signals, going deep on the pillar that closes the most incidents, and scoping what to alert on.

• OpenTelemetry in Practice, the vendor-neutral standard for emitting metrics, logs, and traces from any language.
• Distributed Tracing, how spans propagate across services so you can follow one request end to end.
• The Four Golden Signals, the four metrics worth alerting on for any user-facing service.
• Follow the full SRE career path to build observability, alerting, and reliability skills in order.