Golden Signals Monitoring

Definition

Golden signals monitoring is the discipline, codified in Google's SRE book, of instrumenting every user-facing service with four core metrics: latency (how long requests take), traffic (how many requests arrive), errors (how many fail), and saturation (how full the underlying resources are). Together, these four tell a small team the health of a service without drowning them in dashboards. The concept is intentionally minimal — not because monitoring should be minimal, but because these are the questions that always need answering first.

Why It Matters

Modern observability platforms can generate hundreds of thousands of metrics per service. The temptation is to instrument everything and dashboard everything. What actually happens is that on-call engineers, staring at a wall of green squares during an incident, cannot tell which of the four things they need to know. Golden signals restore focus: whichever of the four is moving is where the investigation starts. Every additional metric answers a follow-up question, not the first one.

The Four Signals in Detail

• Latency — the time to respond to a request. Measure success and failure latency separately; a fast failure and a slow success both need attention. Use percentiles (p50, p95, p99, p99.9), not averages.
• Traffic — the demand on the system, in the unit that matters. Requests per second for a web API, transactions per second for a database, sessions per minute for a streaming service.
• Errors — the rate of requests that fail, explicitly (5xx, timeouts) or implicitly (wrong content, business-level failures). The last category is the hardest to measure and the most common source of "everything looks green but customers are complaining."
• Saturation — how close the system is to its capacity limit. CPU, memory, connection pools, queue depths. For latency-sensitive services, saturation matters at 70–80% because latency degrades non-linearly before hitting the ceiling.

Real-World Example

A logistics company had 640 dashboards across 90 services. During a Black Friday incident, the on-call spent 25 minutes locating the right dashboard while error rates for a partner integration climbed. The retrospective led to a mandated top-of-page four-tile "golden signals" panel for every service. Nine months later, a similar incident occurred; the on-call opened the service page, saw latency p99 climbing and saturation at 92% on the outbound connection pool, and had a fix deployed in eleven minutes. The wider dashboards still existed for deep dives. The change was not what was measured but what was seen first.

How to Implement Them

1. Identify user-facing services and instrument them first. Internal services can follow.
2. Choose the right unit for each signal. Requests per second may be wrong for a batch service; jobs per hour may be right.
3. Use percentiles, not averages, for latency. An average latency hides the tail experience.
4. Separate success and failure latency. A ten-second timeout in the average makes p95 unreadable.
5. Instrument business errors, not just HTTP errors. A 200 OK response with an empty basket page is still a failure.
6. Define saturation for each resource the service depends on — CPU, memory, connections, queues, downstream rate limits.
7. Put all four on the top of the service dashboard and never let anyone below the level of team lead move them.

Practical Lessons Learned

• p50 lies, p99 tells the truth. Median latency looks fine while a quarter of your users are having a bad time; the 99th percentile catches it.
• Traffic drops are as suspicious as traffic spikes. A sudden fall usually means an upstream is failing before you see the errors.
• Saturation is the leading indicator. The connection pool fills before the errors start; watch it and act early.
• Errors need context. A rise in 4xx may be a bot campaign, not a service issue. Segment errors by source before triaging.
• Same-shape dashboards across services mean on-call engineers can jump between systems without relearning layouts.

Expert Tips

• Overlay the previous week's data on the same chart — a spike is easier to judge against last Wednesday than against nothing.
• Tie golden signals to SLOs on the same dashboard — the on-call can see budget burn in the same glance.
• Alert on multi-window burn rates rather than raw thresholds. Static thresholds either page too often or too late.
• Include a downstream saturation panel — many services fail because a dependency is saturated, not because they themselves are.
• Review the golden signals panel with the team monthly. If a signal has been flat for a year, question whether the unit is right.

Common Mistakes

• Measuring what is easy to measure — CPU on every host — rather than what matters to the user.
• Average latency instead of percentiles; the tail experience is invisible.
• No business-error instrumentation; the service reports 200 OK while returning broken responses.
• Saturation measured only at 100% CPU; latency has already collapsed by then.
• Different dashboard layouts per team, so on-call engineers waste incident minutes navigating.

Key Takeaways

• Latency, traffic, errors, saturation — the four questions to answer first.
• Percentiles for latency; segmented sources for errors; leading indicators for saturation.
• Consistent layout across services matters more than clever visualisation.
• Pair the signals with SLO burn on the same page.
• Deeper dashboards exist for the follow-up question, not the first one.

Related Concepts

Pairs with Service Level Indicator, Service Level Objective, Error Budget Policy, and Alert Fatigue Management.