Alerting on Spatial Validation SLOs with Prometheus

A validation pipeline that fails silently is worse than one that fails loudly, because everyone downstream keeps trusting data nobody checked. Prometheus is a good fit for catching that, provided the instrumentation is designed rather than accumulated: a small metric set with bounded cardinality, objectives that separate “did the pipeline run” from “is the data good”, and burn-rate alerts that page for genuine budget consumption instead of for every transient blip. This guide sets that up, implementing the observability signals described in Observability and Lineage for Validation.

Prerequisites

  • Prometheus 2.45+ with Alertmanager, and either a Pushgateway or a metrics endpoint the scraper can reach. Batch validation jobs are short-lived, so a Pushgateway is the usual answer.
  • prometheus_client 0.20+ in the validation job.
  • A layer inventory with a validation cadence per layer — daily, hourly, weekly. The freshness objective is meaningless without it.
  • Defect-rate baselines, derived from history as described in Spatial Data Quality Metrics and Reporting.

Step-by-Step Procedure

Step 1 — Export a small, bounded metric set

# observability/metrics.py
import time

from prometheus_client import CollectorRegistry, Counter, Gauge, Histogram, push_to_gateway

REGISTRY = CollectorRegistry()

RUNS = Counter(
    "spatial_validation_runs_total", "Validation runs by outcome",
    ["layer", "outcome"], registry=REGISTRY)                 # outcome: success|failure

DURATION = Histogram(
    "spatial_validation_duration_seconds", "Wall-clock duration of a validation run",
    ["layer"], buckets=(10, 30, 60, 300, 900, 1800, 3600), registry=REGISTRY)

FEATURES = Gauge(
    "spatial_validation_features", "Features validated in the last run",
    ["layer"], registry=REGISTRY)

FINDINGS = Counter(
    "spatial_validation_findings_total", "Findings by severity",
    ["layer", "severity"], registry=REGISTRY)                # severity: 3 values only

LAST_SUCCESS = Gauge(
    "spatial_validation_last_success_timestamp_seconds",
    "Unix time of the last successful run", ["layer"], registry=REGISTRY)

RULESET = Gauge(
    "spatial_validation_ruleset_info", "Rule-set version currently deployed",
    ["layer", "version"], registry=REGISTRY)


def publish(layer: str, ok: bool, seconds: float, features: int,
            findings: dict[str, int], ruleset_version: str, gateway: str) -> None:
    RUNS.labels(layer=layer, outcome="success" if ok else "failure").inc()
    DURATION.labels(layer=layer).observe(seconds)
    FEATURES.labels(layer=layer).set(features)
    for severity in ("blocker", "warning", "informational"):
        FINDINGS.labels(layer=layer, severity=severity).inc(findings.get(severity, 0))
    if ok:
        LAST_SUCCESS.labels(layer=layer).set(time.time())
    RULESET.labels(layer=layer, version=ruleset_version).set(1)
    push_to_gateway(gateway, job=f"spatial-validation/{layer}", registry=REGISTRY)

Verification: count the label combinations. Layers times severities is bounded and small; adding a rule label would multiply it by the rule count and adding feature_id would make it unbounded — the classic way to take down a Prometheus server. Per-rule detail belongs in the findings table.

The metric set, and why each one earns its cardinalityGrid of six exported metrics with the label set of each and the question it answers, showing that every label combination is bounded.LabelsAnswersruns_totallayer, outcomedid the run happen and succeed?duration_secondslayeris it slowing down?featureslayerthe denominator for every ratefindings_totallayer, severityis the data getting worse?last_success_timestamplayerhas anything run at all?ruleset_infolayer, versionwhich rules produced this?Layers times severities is bounded and small. A rule label would multiply it by the rule count; a feature label would make it unbounded and take the server down.
Six metrics, every label set bounded — cardinality is the constraint that shapes the whole design.

Step 2 — State the objectives

# slo/objectives.yaml
objectives:
  - name: freshness_critical_layers
    description: "Every critical layer validated successfully within its cadence"
    target: 0.99                 # 99% of evaluation intervals
    window: 30d
    applies_to: [parcels, road_network, water_mains]
    cadence_seconds: 86400

  - name: blocker_rate_within_threshold
    description: "Blocker findings stay below the agreed rate per 1,000 features"
    target: 0.99                 # 99% of runs
    window: 30d
    threshold_per_1k: 0.5
    applies_to: [parcels, road_network]

  - name: run_duration_within_budget
    description: "Validation completes inside the batch window"
    target: 0.95
    window: 30d
    budget_seconds: 1800

Verification: three objectives, each with a target, a window and an explicit scope. An objective without a window is a wish; one without a scope will be argued about the first time it is breached on a layer nobody thought it covered.

Step 3 — Recording rules so the alerts stay cheap

# prometheus/recording-rules.yaml
groups:
  - name: spatial-validation-derived
    interval: 1m
    rules:
      - record: spatial_validation:last_success_age_seconds
        expr: time() - spatial_validation_last_success_timestamp_seconds

      - record: spatial_validation:blocker_rate_per_1k
        expr: |
          1000 *
          (increase(spatial_validation_findings_total{severity="blocker"}[1d])
           / clamp_min(spatial_validation_features, 1))

      - record: spatial_validation:run_failure_ratio_1h
        expr: |
          sum by (layer) (rate(spatial_validation_runs_total{outcome="failure"}[1h]))
          / clamp_min(sum by (layer) (rate(spatial_validation_runs_total[1h])), 1e-9)

      - record: spatial_validation:run_failure_ratio_6h
        expr: |
          sum by (layer) (rate(spatial_validation_runs_total{outcome="failure"}[6h]))
          / clamp_min(sum by (layer) (rate(spatial_validation_runs_total[6h])), 1e-9)

Verification: recording rules give every alert and dashboard the same definition of a rate. Two alerts computing “blocker rate” slightly differently is a real and common failure — one fires, the other does not, and nobody can say which is right.

Step 4 — Burn-rate alerts

# prometheus/alerting-rules.yaml
groups:
  - name: spatial-validation-slo
    rules:
      # Freshness: absence of a successful run. Counters cannot express this.
      - alert: SpatialValidationStale
        expr: |
          spatial_validation:last_success_age_seconds{layer=~"parcels|road_network|water_mains"}
            > 1.5 * 86400
        for: 10m
        labels: {severity: page, team: geospatial-platform}
        annotations:
          summary: "{{ $labels.layer }} has no successful validation for {{ $value | humanizeDuration }}"
          runbook_url: "https://runbooks.internal/spatial/stale-validation"

      # Fast burn: the 30-day budget will be gone within hours.
      - alert: SpatialValidationFailuresBurningFast
        expr: |
          spatial_validation:run_failure_ratio_1h > 14.4 * 0.01
          and
          spatial_validation:run_failure_ratio_6h > 6 * 0.01
        for: 5m
        labels: {severity: page, team: geospatial-platform}
        annotations:
          summary: "{{ $labels.layer }} validation failing fast ({{ $value | humanizePercentage }} of runs)"
          runbook_url: "https://runbooks.internal/spatial/run-failures"

      # Slow burn: a ticket, not a page.
      - alert: SpatialValidationFailuresBurningSlow
        expr: spatial_validation:run_failure_ratio_6h > 3 * 0.01
        for: 2h
        labels: {severity: ticket, team: geospatial-platform}
        annotations:
          summary: "{{ $labels.layer }} validation failures above the slow-burn threshold"

      # Data health, distinct from pipeline health.
      - alert: SpatialBlockerRateAboveThreshold
        expr: spatial_validation:blocker_rate_per_1k > 0.5
        for: 30m
        labels: {severity: ticket, team: data-stewards}
        annotations:
          summary: >-
            {{ $labels.layer }} blocker rate {{ $value | printf "%.2f" }} per 1k
            exceeds the agreed 0.5
          runbook_url: "https://runbooks.internal/spatial/blocker-rate"

Verification: the two burn-rate multipliers (14.4 over one hour, 6 over six hours) are the standard fast and slow pairing for a 30-day window at a 99% target. Requiring both conditions for the page suppresses single-scrape spikes while still catching a genuine outage within minutes.

Step 5 — Route by severity and attach a runbook

# alertmanager/routes.yaml
route:
  receiver: default
  group_by: [alertname, layer]
  group_wait: 30s
  group_interval: 5m
  repeat_interval: 4h
  routes:
    - matchers: [severity="page"]
      receiver: oncall-pager
      continue: false
    - matchers: [severity="ticket", team="data-stewards"]
      receiver: steward-queue
    - matchers: [severity="ticket"]
      receiver: platform-backlog

receivers:
  - name: oncall-pager
    pagerduty_configs:
      - routing_key_file: /etc/alertmanager/pd_key
  - name: steward-queue
    webhook_configs:
      - url: http://ticketing.internal/hooks/data-quality
  - name: platform-backlog
    webhook_configs:
      - url: http://ticketing.internal/hooks/platform

Verification: the routing sends data-health alerts to the stewards and pipeline-health alerts to the platform team. Sending both to one channel is the fastest way to make a quality alert invisible — the audience that can act on a rising blocker rate is not the audience that restarts a failed job.

Interpreting Results

Alert combination Diagnosis First action
Stale only Scheduler or upstream data missing Check the orchestrator, not the data
Fast burn only Job crashing repeatedly Read the job logs; likely a dependency
Blocker rate only Data regression Steward investigation; the pipeline is fine
Stale + blocker rate The last run was bad and nothing has run since Fix the run first, then the data
Blocker rate rising with features falling Partial run inflating the rate Not a data event; check the input scope
Duration alert only Data volume growth or a slow rule Profile before scaling

The features gauge is what makes the fifth row diagnosable. A defect rate is only comparable when the denominator is stable, and having the denominator as its own series means the dashboard can show them together — the same discipline described in Publishing Validation Dashboards with Grafana.

Fast burn pages, slow burn ticketsTwo error-budget consumption curves over thirty days. The fast-burn curve rises steeply and would exhaust the budget within hours, triggering a page. The slow-burn curve rises gently and exhausts it near the end of the window, opening a ticket instead.100%0%time through the 30-day windowbudget usedfast burn — budget gone in hours → pageslow burn — trending over → ticket
The same objective, two windows: one wakes somebody, the other files work for the morning.

Gotchas & Edge Cases

Batch jobs and Prometheus’s pull model do not naturally fit. A job that runs for ten minutes each night is not scrapeable. Push to a Pushgateway and remember that Pushgateway metrics persist until deleted — a decommissioned layer keeps reporting its last values forever unless the group is deleted explicitly.

Which alert fired, and what it tells you firstGrid of five alert combinations with the diagnosis and the first action, distinguishing pipeline problems from data problems.DiagnosisFirst actionStale onlyscheduler or upstream data missingcheck the orchestratorFast burn onlyjob crashing repeatedlyread the job logsBlocker rate onlydata regressionsteward, not platformStale + blocker ratebad run, then nothing ranfix the run firstRate up, features downpartial run inflating the ratenot a data eventThe last row is why the features gauge is exported at all: without the denominator, a partial run and a data regression look identical.
Two alerts produce five diagnoses, because the combination carries more information than either alone.

Counters reset when the job restarts. That is expected and increase() handles it, but a gauge set from a counter-like value will jump. Use counters for counts and gauges for states, not interchangeably.

Cardinality creeps. Every new label multiplies the series count. A rule label on findings looks harmless until the rule set reaches two hundred; combined with layers and severities that is a five-figure series count for one metric.

clamp_min guards against division by zero. Without it, a layer with zero features validated produces +Inf and alerts fire in ways nobody expects.

A for duration shorter than the scrape interval is meaningless. With a 60-second scrape, for: 30s evaluates on a single sample and behaves like no for at all.

Alert on the absence of data, not just on its values. absent() and the last-success timestamp are the only ways to detect a job that stopped existing; every value-based alert goes quiet when the metric disappears, which is exactly backwards.

When to Escalate

  • An alert firing repeatedly with no action taken should be deleted or fixed within a sprint. A permanently firing alert degrades every other alert in the same channel.
  • A blocker-rate alert that the stewards cannot action means the threshold was set without agreement. Take it back to the policy, per Defining Spatial Data Quality Policies.
  • Cardinality growth affecting Prometheus stability is a platform incident, not a tuning exercise — drop the offending label immediately and redesign afterwards.
  • Freshness breaches caused by upstream data arriving late are an upstream service-level conversation. The validation pipeline cannot validate data it has not received, and alerting harder does not change that.

Frequently Asked Questions

Which metrics should a validation pipeline export?

Six are usually enough: run started and completed counters, run duration, features validated, findings by severity, and the timestamp of the last successful run per layer. Everything else can be derived. Resist exporting per-feature or per-rule-instance metrics — cardinality is what kills a Prometheus deployment, and the detail belongs in the findings table.

What is a sensible SLO for a validation pipeline?

Two, in different families. A freshness objective — every critical layer has a successful validation within its cadence window — and a quality objective — the blocker rate stays below its threshold. They fail for different reasons and need different responses, which is exactly why they should not be combined into one number.

Why use burn-rate alerts rather than simple thresholds?

A simple threshold fires on every transient breach and trains people to ignore it. A burn-rate alert asks how fast the error budget is being consumed: a fast burn over a short window pages because the objective will be missed within hours, while a slow burn over a long window opens a ticket. The result is far fewer pages for the same coverage.

How is a missing run detected in Prometheus?

With a last-success timestamp gauge and an alert on the age of that timestamp: time() minus the gauge, compared against the layer's cadence. Counters cannot express absence — a job that never ran increments nothing — which is why the timestamp gauge is the single most important metric in the set.

Related

Back to Observability and Lineage for Validation