Spatial Data Quality Metrics and Reporting

A validation pipeline produces findings; nobody acts on findings. What people act on is a small number of figures they trust, tracked over time, attached to a consequence. Getting from one to the other is a design problem with a few reliable failure modes: counts that move with dataset size, denominators that shift silently, scores nobody can decompose, and reports written for the wrong reader. This topic sets out how to define the metric set — stable denominators, rates, severity weighting, history-based baselines — and how to publish it in the three shapes that different audiences need, extending the policy work in Defining Spatial Data Quality Policies.

Prerequisites

• A findings table with the shared result contract — feature identifier, rule identifier, severity, run identifier — retained over time. Metrics need history; a pipeline that overwrites its last run has no trend.
• A per-run record of what was validated: feature count, area, partitions, rule-set version. This is the denominator source, and it is the part most often missing.
• A rule catalogue mapping rules to quality dimensions — completeness, logical consistency, positional accuracy, temporal quality, thematic accuracy — as defined in Aligning Spatial Metadata with ISO 19157.
• Named owners per layer, so a metric that moves has somebody to move it back, per Data Stewardship Roles and Responsibilities.

Core Concepts & Architecture

The denominator is the metric. Almost every argument about a quality figure turns out to be an argument about what it was divided by. Defects per feature, per thousand features, per square kilometre and per validated record give different answers and each is right for a different question. Choose one per rule family, write it down, and change it only with a version bump — because a silently changed denominator produces a step in the trend that looks like a data event.

Rates beat counts, and rates need a floor. A rate over a tiny denominator is noise: one defect in forty features is 2.5%, and it means nothing. Publish the denominator alongside every rate and suppress rates below a minimum sample, the same discipline that applies to the stratified samples in Measuring Geocoding Accuracy Against Reference Points.

Severity weighting encodes consequence. A thousand precision warnings are not equivalent to one overlapping parcel, and a metric that sums findings says they are. Weight by severity — a common scheme is blocker 100, warning 10, informational 1 — and the composite tracks impact rather than volume. The weights are a policy decision and belong in the same versioned file as the rules.

Baselines come from history, not from ambition. A threshold set at a round number is either always breached or never breached. A threshold set at the 95th percentile of the last thirty runs fires when something genuinely changed, which is the only alert anyone keeps paying attention to.

Three audiences, one metric set. The operational view answers “did tonight’s run work and is anything on fire”. The steward view answers “which layers are getting worse and what is the backlog”. The executive view answers “is the programme improving and where should the next investment go”. All three derive from the same numbers; publishing three different metric sets is how organisations end up arguing about which dashboard is right.

Designing for Scale

Compute metrics from the findings table, not from the raw data. A metrics job that re-reads the datasets is a second validation pass with a different failure surface; one that aggregates the findings table is a few seconds of SQL regardless of how large the underlying layers are.

Materialise a daily aggregate rather than querying the raw findings for every dashboard load. Findings tables grow quickly — a million findings a night is unremarkable on a large estate — and a dashboard that scans them live becomes both slow and expensive. A metrics_daily table keyed by run date, layer and rule is small, fast and stable.

Keep the aggregate append-only and versioned by rule-set version. When rules change, the metric changes for reasons that have nothing to do with the data, and the only way to explain a step in a chart six months later is to have the rule-set version stored beside the number.

Retain raw findings for a bounded window and the aggregate indefinitely. Nobody needs individual findings from two years ago; everybody eventually wants the trend.

Comparability is the property everything else depends on. A metric that cannot be compared against last month, against another layer, or against the figure a colleague quoted is a number rather than a measurement. Three things make a spatial quality metric comparable: a denominator that did not move, a rule set whose version is recorded, and a scope statement saying which features were in the run. Any one of the three changing invalidates a comparison, so all three travel with every figure. This is the same reproducibility argument made for validation runs in observability and lineage for validation, applied one layer up: a metric is a claim about a run, and a claim about a run needs the run’s identity attached.

Aggregate once, read many times. Metrics should be computed in a single scheduled job that writes a daily aggregate, and every consumer — the operational dashboard, the steward scorecard, the executive report, the catalogue conformance statement — should read that one table. The alternative, where each audience computes its own figures from the findings, produces four numbers for the same question and an argument about which is right. The aggregate is small, cheap to query and easy to retain for years, which also makes it the natural home for the trend analysis that turns a snapshot into a signal.

Rule Evaluation Strategies

The metric set that covers most programmes is small:

# metrics/compute.py — daily metrics from a findings table
import pandas as pd

SEVERITY_WEIGHT = {"blocker": 100, "warning": 10, "informational": 1}
MIN_DENOMINATOR = 500          # below this, rates are noise


def daily_metrics(findings: pd.DataFrame, runs: pd.DataFrame) -> pd.DataFrame:
    """One row per run per layer: rates, weighted score and the denominator behind them."""
    counts = (findings.groupby(["run_id", "layer_id", "severity"])
              .size().unstack(fill_value=0).reset_index())
    for level in SEVERITY_WEIGHT:
        if level not in counts:
            counts[level] = 0

    merged = counts.merge(
        runs[["run_id", "layer_id", "features_validated", "rule_set_version", "run_date"]],
        on=["run_id", "layer_id"], how="left")

    n = merged["features_validated"].clip(lower=1)
    merged["blocker_rate_per_1k"] = 1000 * merged["blocker"] / n
    merged["warning_rate_per_1k"] = 1000 * merged["warning"] / n
    merged["weighted_defect_score"] = (
        sum(merged[level] * weight for level, weight in SEVERITY_WEIGHT.items()) / n)
    merged["clean_share"] = 1 - (
        findings.groupby(["run_id", "layer_id"])["feature_id"].nunique()
        .reindex(pd.MultiIndex.from_frame(merged[["run_id", "layer_id"]]))
        .fillna(0).to_numpy() / n)
    merged["rate_reportable"] = merged["features_validated"] >= MIN_DENOMINATOR
    return merged

Four figures per layer per run — blocker rate, warning rate, weighted score, clean share — plus the denominator and the rule-set version. That is enough to answer every question the three audiences ask, and small enough that people remember what the numbers mean.

clean_share deserves a note: it counts distinct features with no finding at all, which is the figure non-specialists find most intuitive. “97.2% of parcels passed every check” communicates better than any rate, and it is the natural headline for the executive view.

Error Handling & Remediation

Metrics have their own failure modes, and they are quieter than pipeline failures.

A missing run produces a gap, not a zero. If the nightly job did not run, the correct chart shows nothing for that day. Filling the gap with zero defects reads as a perfect night and hides an outage — the single most common way a quality dashboard misleads.

A denominator change produces a step. When the scope of a layer changes — a new district, a retired sub-layer — every rate shifts. Record scope changes as annotations on the trend so the step has a caption rather than a theory.

Rule-set changes produce steps too. A tightened rule increases defects with no change in the data. That is why the rule-set version travels with the metric, and why the report should show it.

Suppressed rates need to be visibly suppressed. A layer below the minimum denominator should render as “insufficient sample”, not as zero or as a blank. Blank cells get interpreted as good news.

Observability, Lineage and Compliance

The metrics layer is also the compliance artefact. An ISO 19157 conformance statement is, in practice, a measure with a threshold and a result — exactly what the metric set produces. Generating conformance metadata from the same table that feeds the dashboard removes the gap between what the dashboard says and what the catalogue claims, which is otherwise a reliable source of audit findings.

Retain the mapping from metric to rule to policy clause. When an auditor asks why a layer is described as conformant, the answer should traverse from the published figure to the rules that produced it to the obligation those rules implement, without anybody reconstructing it from memory.

Publish the metric definitions themselves. A dashboard whose figures cannot be reproduced by a reader with access to the findings table is a dashboard that will eventually be disbelieved.

Retention differs by artefact. Raw findings are large and lose value quickly — a defect list from two years ago tells you nothing that the aggregate does not. The daily aggregate is small and gains value with age, because trend is the whole point. A common arrangement keeps ninety days of raw findings for investigation and the aggregate indefinitely, with the rule-set version stored alongside each row so historical figures remain interpretable after the rules change.

Best Practices & Anti-Patterns

• Do publish the denominator next to every rate, always.
• Do derive thresholds from the recent distribution, and re-derive them periodically.
• Do version the severity weights alongside the rules.
• Do annotate scope and rule changes on the trend charts.
• Do give every metric an owner who can act on it.
• Don’t report raw counts as a headline; they track dataset growth.
• Don’t fill missing runs with zeros.
• Don’t publish a composite score without its components.
• Don’t set a target because it is a round number.
• Don’t build a separate metric set per audience — build one set and three views.

Frequently Asked Questions

Related

• Building a Spatial Data Quality Scorecard — the per-layer scorecard and its weighting
• Publishing Validation Dashboards with Grafana — the operational view, panels and alert rules
• Reporting Data Quality to Non-Technical Stakeholders — the executive view without the jargon
• Defining Spatial Data Quality Policies — where thresholds and severities are decided
• Observability and Lineage for Validation — the run signals these metrics are computed from

Back to Spatial Data Governance & Compliance Basics