Observability — metrics, logs, traces and alerts

A design that cannot be debugged is not finished. The interviewer asks "how do you know it is working?" and the answer cannot be "we would look at the logs".

Three signals answer three different questions. Metrics tell you something is wrong. Traces tell you where. Logs tell you why. People collect all three and can still answer none of them, because they collected the wrong things.

Metrics are numbers over time, cheap to store and always on. The four worth naming for any request-serving system are request rate, error rate, latency percentiles and saturation (how full the thing is). Quote p50 and p99, never the average, for the reason the latency page gives: nobody experiences the average. For a queue, the equivalents are consumer lag, oldest message age and dead letter arrival rate.

The rule that keeps metrics affordable is no unbounded labels. A metric tagged with user id or request id creates a separate time series per value, and that is how a monitoring bill ends up larger than the hosting bill. Tag by endpoint, status code and region: things with a small fixed set of values.

Traces follow one request across every service it touches, with a timing for each hop. This is the signal that answers "was it one slow call or two hundred fast ones", which is usually the question when something is slow. It needs a trace id created at the edge and passed through every call, including into queue messages, or the trace stops at the first async boundary, where many of the hard failures are. Because tracing every request is too expensive, you sample, and the right approach is to always keep traces for errors and slow requests, and sample the boring ones.

Logs are the detail, and they are the most expensive per byte. Make them structured (key-value, not prose) so they can be searched and aggregated instead of grepped. Every line should carry the trace id, so a log and a trace can be joined. And logs are where personal data leaks by accident, so decide explicitly what may not be written.

SLIs and SLOs are how you turn all of this into a decision. An SLI is the thing you measure: say, the share of requests that succeed in under 300 ms. An SLO is the target you commit to, say 99.9% over 30 days. The useful part is what falls out of it: an error budget. If the target allows 0.1% failure and you have used a quarter of it this month, you can keep shipping. If you have burned it, you stop shipping features and fix reliability. That converts an argument about "is this stable enough" into arithmetic, and mentioning it is a strong senior signal.

Alerting has one rule that matters: alert on symptoms, not causes. Page someone when users are affected: error rate up, latency past the SLO, the queue's oldest message getting older. Do not page them because CPU hit 90%, because CPU at 90% with everything working fine is not a problem, and it teaches people to ignore the pager. Every alert should be actionable, should say what to do, and should be reviewed when it fires and turns out not to matter.

Two more that come up. Health checks should distinguish liveness (is this process alive; cheap and dependency-free) from readiness, which may check dependencies. Confusing them takes down whole fleets, as the load balancing page describes. And dashboards are not monitoring: a dashboard nobody is looking at at 3am detects nothing. The alert is the detection; the dashboard is for the investigation afterwards.