Observability

Deep dive

Signals and where to keep them

Decision table

Code

Instrument a Rust service with metrics and structured logs (Prometheus format):

use prometheus::{register_histogram_vec, register_int_counter_vec, Encoder, HistogramVec, IntCounterVec, TextEncoder};
use std::sync::LazyLock;
use std::time::Instant;

static REQS: LazyLock<IntCounterVec> = LazyLock::new(|| register_int_counter_vec!(
    "http_requests_total", "Requests", &["route", "status"]).unwrap());
static LAT: LazyLock<HistogramVec> = LazyLock::new(|| register_histogram_vec!(
    "http_request_duration_seconds", "Latency", &["route"],
    vec![0.001, 0.0025, 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5]).unwrap());

pub fn observe<T>(route: &'static str, f: impl FnOnce() -> (u16, T)) -> T {
    let t = Instant::now();
    let (status, out) = f();
    LAT.with_label_values(&[route]).observe(t.elapsed().as_secs_f64());
    // route is a template ("/users/:id"), never the raw path: bounded cardinality
    REQS.with_label_values(&[route, &status.to_string()]).inc();
    tracing::info!(route, status, ms = t.elapsed().as_millis() as u64, "request");
    out
}

pub fn metrics_text() -> String {
    let mut buf = Vec::new();
    TextEncoder::new().encode(&prometheus::gather(), &mut buf).unwrap();
    String::from_utf8(buf).unwrap()
}

Multi-window burn-rate alert for a 99.9 % availability SLO (Prometheus rule):

groups:
- name: slo
  rules:
  - alert: ErrorBudgetBurnFast
    expr: |
      ( sum(rate(http_requests_total{status=~"5.."}[1h])) / sum(rate(http_requests_total[1h])) > 14.4 * 0.001 )
      and
      ( sum(rate(http_requests_total{status=~"5.."}[5m])) / sum(rate(http_requests_total[5m])) > 14.4 * 0.001 )
    for: 2m
    labels: { severity: page }
    annotations: { summary: "Burning 2% of monthly error budget per hour" }

OpenTelemetry Collector tail sampling (keep errors and slow traces, 5 % of the rest):

processors:
  tail_sampling:
    decision_wait: 10s
    policies:
      - { name: errors, type: status_code, status_code: { status_codes: [ERROR] } }
      - { name: slow,   type: latency,     latency: { threshold_ms: 500 } }
      - { name: rest,   type: probabilistic, probabilistic: { sampling_percentage: 5 } }

Efficiency

• Cost: Prometheus stores an average of 1-2 bytes per sample (Prometheus storage docs), so disk = retention seconds × samples per second × bytes per sample. Logs cost per GB ingested and traces per span stored. Tail sampling buffers each trace for decision_wait (default 30 s) before it decides (OpenTelemetry tail sampling processor). Bound: active series and ingest volume, not query speed.
• Levers:
  • bounded label values (route templates) — series count stays small (Code above) — every metric
  • longer scrape interval — disk is proportional to samples per second (Prometheus formula) — slow-moving gauges
  • tail sampling: keep errors and slow traces, a percentage of the rest — stores a fraction of traces (OpenTelemetry tail sampling processor) — tracing
  • short hot retention, archive to object storage — log volume stays cheap (Key facts, order of) — logs
  • drop or sample debug logs — fewer GB ingested (this page) — production
• Naive → standard → state of the art:
  • everything at debug to a per-GB SaaS, user ID as a label · bill grows with traffic and cardinality
  • bounded labels, JSON logs with trace ID, tail sampling, 15 days hot · 1-2 bytes per sample, a fraction of traces
  • wide events in a columnar store plus SLO burn-rate alerts · pay for queried events, few pages

Implementation notes

• RED for services (Rate, Errors, Duration), USE for resources (Utilisation, Saturation, Errors).
• Histograms, not averages; report p50/p99/p99.9. Native/exponential histograms cut series count.
• Log as JSON with a trace ID in every line; drop debug logs in production or sample them.
• Keep the monitoring stack outside the system it monitors (different cluster, region or provider), plus an external uptime probe.
• Budget: observability commonly lands at 5-15 % of infrastructure spend; above that, cut volume.

Pitfalls

• Unbounded labels (user ID, request path with IDs, error message text) explode series count and cost.
• Logging request/response bodies: cost, and personal data in logs (GDPR).
• Alerts nobody acts on train people to ignore pages.
• Head sampling at 1 % drops the rare error trace you needed; use tail sampling.
• Scrape intervals below 10 s multiply cost for little insight.

Implementations

Sources

• Betsy Beyer et al., The Site Reliability Workbook, O'Reilly, 2018 (chapter on alerting on SLOs). chapter
• Benjamin H. Sigelman et al., Dapper, a Large-Scale Distributed Systems Tracing Infrastructure, Google Technical Report, 2010. report
• Tuomas Pelkonen et al., Gorilla: A Fast, Scalable, In-Memory Time Series Database, VLDB 2015. paper
• OpenTelemetry, Collector documentation (opentelemetry.io). docs · Prometheus, Storage. docs