Building a Spatial Data Quality Scorecard

A scorecard is a summary that has to survive being argued with. Somebody will ask why the parcel layer scored 82 when the roads layer scored 91, whether last month’s 76 is comparable, and what would need to change to reach 90 — and if the answers are not immediately derivable from the published numbers, the scorecard stops being used. This guide builds one that answers those questions: rules mapped to quality dimensions, each dimension normalised against its own history, weights that reflect what the layer is for, a grade capped by blocking defects, and a drill-down that names the rules doing the damage. It implements the metric design set out in Spatial Data Quality Metrics and Reporting.

Prerequisites

  • A findings table with run history — at least thirty runs per layer, or the normalisation has no distribution to work against.
  • A per-run denominator: features validated, per layer. Without it the sub-scores are counts wearing a percentage sign.
  • A rule catalogue where every rule has exactly one quality dimension and one severity. Rules mapped to two dimensions make the sub-scores overlap and the weights meaningless.
  • A declared purpose per layer, which is where the weights come from.
  • pandas 2.x and somewhere to persist the scorecard table — the same warehouse as the findings is easiest.

Step-by-Step Procedure

Step 1 — Map every rule to exactly one dimension

# scorecard/step1_catalogue.py
RULE_DIMENSION = {
    "ATTR_NULL_001":      "completeness",
    "ATTR_DOMAIN_001":    "thematic_accuracy",
    "ATTR_RANGE_001":     "thematic_accuracy",
    "GEOM_VALID_001":     "logical_consistency",
    "GEOM_TYPE_001":      "logical_consistency",
    "TOPO_OVERLAP_001":   "logical_consistency",
    "TOPO_GAP_001":       "completeness",
    "CRS_001":            "positional_accuracy",
    "PREC_DRIFT_001":     "positional_accuracy",
    "TEMPORAL_STALE_001": "temporal_quality",
}

DIMENSIONS = ["completeness", "logical_consistency", "positional_accuracy",
              "temporal_quality", "thematic_accuracy"]


def unmapped_rules(findings) -> set:
    """Any rule without a dimension silently disappears from the score — find those first."""
    return set(findings["rule_id"].unique()) - set(RULE_DIMENSION)

Verification: run unmapped_rules before every scorecard build and fail the job if it returns anything. An unmapped rule is worse than a missing rule: the defect is detected, reported and then excluded from the summary that people read.

Dimension weights follow what the layer is forGrid of three layers against five quality dimensions showing the weight each layer assigns: parcels weight logical consistency and completeness, the road network weights positional accuracy, and land use weights thematic accuracy.completeconsistentpositionaltemporalthematicparcels0.300.350.200.050.10road network0.250.300.300.100.05land use0.200.150.100.150.40Weights sum to 1.0 per layer and are versioned with the rules.Weights invented in a meeting and never revisited are the norm. Tie them to the layer’s declared purpose and review them when that purpose changes.
The same defect matters differently per layer, which is the whole justification for weighting.

Step 2 — Normalise each dimension against its own history

# scorecard/step2_normalise.py
import numpy as np
import pandas as pd


def dimension_rates(findings: pd.DataFrame, runs: pd.DataFrame, mapping: dict) -> pd.DataFrame:
    """Defects per 1,000 features, per dimension, per run."""
    f = findings.assign(dimension=findings["rule_id"].map(mapping))
    counts = (f.groupby(["run_id", "layer_id", "dimension"])
              .size().rename("defects").reset_index())
    merged = counts.merge(runs[["run_id", "layer_id", "features_validated"]],
                          on=["run_id", "layer_id"], how="right")
    merged["defects"] = merged["defects"].fillna(0)
    merged["rate_per_1k"] = 1000 * merged["defects"] / merged["features_validated"].clip(lower=1)
    return merged


def sub_scores(rates: pd.DataFrame, history_runs: int = 30) -> pd.DataFrame:
    """0-100 per dimension: 100 at the historical best decile, 0 at three times the median."""
    out = []
    for (layer, dim), g in rates.groupby(["layer_id", "dimension"]):
        g = g.sort_values("run_id")
        hist = g["rate_per_1k"].tail(history_runs)
        best = float(np.percentile(hist, 10))
        worst = max(float(hist.median()) * 3, best + 1e-6)
        score = 100 * (worst - g["rate_per_1k"]) / (worst - best)
        out.append(g.assign(sub_score=score.clip(0, 100).round(1),
                            scale_best=round(best, 3), scale_worst=round(worst, 3)))
    return pd.concat(out, ignore_index=True)

Verification: a run at the historical median should score around 67, a run at the best decile 100, and a run three times worse than the median 0. Sanity-check those three points on real data before publishing anything — a normalisation that puts every run at 99 is measuring nothing.

Step 3 — Weight by what the layer is for

# scorecard/step3_weights.py
LAYER_WEIGHTS = {
    "parcels": {           # cadastral: geometry and completeness dominate
        "completeness": 0.30, "logical_consistency": 0.35,
        "positional_accuracy": 0.20, "temporal_quality": 0.05,
        "thematic_accuracy": 0.10,
    },
    "road_network": {      # routing: position and consistency matter most
        "completeness": 0.25, "logical_consistency": 0.30,
        "positional_accuracy": 0.30, "temporal_quality": 0.10,
        "thematic_accuracy": 0.05,
    },
    "land_use": {          # classification: thematic accuracy dominates
        "completeness": 0.20, "logical_consistency": 0.15,
        "positional_accuracy": 0.10, "temporal_quality": 0.15,
        "thematic_accuracy": 0.40,
    },
}


def composite(sub: "pd.DataFrame") -> "pd.DataFrame":
    rows = []
    for (run_id, layer), g in sub.groupby(["run_id", "layer_id"]):
        weights = LAYER_WEIGHTS.get(layer)
        if weights is None:
            continue                       # an unweighted layer is not scored, not scored as zero
        applicable = {d: w for d, w in weights.items() if d in set(g["dimension"])}
        total_w = sum(applicable.values()) or 1
        score = sum(
            float(g.loc[g["dimension"] == d, "sub_score"].iloc[0]) * w
            for d, w in applicable.items()) / total_w
        rows.append({"run_id": run_id, "layer_id": layer,
                     "composite": round(score, 1),
                     "dimensions_scored": sorted(applicable),
                     "weight_coverage": round(total_w, 2)})
    import pandas as pd
    return pd.DataFrame(rows)

Verification: weight_coverage below 1.0 means some dimensions had no applicable rules and the weights were renormalised. Publish that figure — a layer scored on 60% of its intended weight is a partially measured layer, and the scorecard should say so rather than implying full coverage.

Step 4 — Grade, and cap on blockers

# scorecard/step4_grade.py
GRADE_BANDS = [(90, "A"), (80, "B"), (70, "C"), (60, "D"), (0, "F")]
BLOCKER_CAP = "C"


def grade(composite_score: float, blocker_count: int) -> dict:
    letter = next(g for threshold, g in GRADE_BANDS if composite_score >= threshold)
    capped = False
    if blocker_count > 0 and letter in ("A", "B"):
        letter, capped = BLOCKER_CAP, True
    return {
        "composite": round(composite_score, 1),
        "grade": letter,
        "capped_by_blockers": capped,
        "blocker_count": int(blocker_count),
        "explanation": (
            f"capped at {BLOCKER_CAP}: {blocker_count} blocker-severity finding(s)"
            if capped else "grade from the weighted composite"),
    }

Verification: construct a case with an excellent composite and one blocker, and confirm it grades C with the explanation attached. This single rule prevents the most damaging scorecard failure — a green light on a layer that a downstream system will reject.

Step 5 — Publish with the drill-down attached

# scorecard/step5_publish.py
import pandas as pd


def scorecard_row(run_id: str, layer: str, graded: dict, sub: pd.DataFrame,
                  findings: pd.DataFrame, rule_set_version: str, features: int) -> dict:
    top_rules = (findings[(findings["run_id"] == run_id) & (findings["layer_id"] == layer)]
                 .groupby("rule_id").size().sort_values(ascending=False).head(3))
    return {
        "run_id": run_id,
        "layer_id": layer,
        "rule_set_version": rule_set_version,
        "features_validated": features,
        **graded,
        "sub_scores": {
            r.dimension: {"score": r.sub_score, "rate_per_1k": round(r.rate_per_1k, 3)}
            for r in sub.itertuples()
        },
        "top_contributing_rules": [
            {"rule_id": rule, "findings": int(n)} for rule, n in top_rules.items()
        ],
    }

Verification: the published row should let a reader reconstruct the composite by hand from the sub-scores and the weights. If they cannot, the scorecard has become a black box, and black-box scores get ignored the first time somebody disagrees with one.

Interpreting Results

Pattern What it means Response
Composite falls, one sub-score falls A specific dimension regressed Read top_contributing_rules; usually one rule
Composite falls, all sub-scores fall The denominator shrank, or a scope change Check features_validated before investigating data
Grade capped repeatedly at C Persistent blockers nobody is clearing A backlog problem, not a measurement problem
Score improves after a rule change Rules were relaxed, not data improved Compare rule_set_version; annotate the chart
weight_coverage below 1 Some dimensions have no applicable rules Either add rules or reduce the claimed coverage
Score stable at 99 for months Normalisation scale is too generous Re-derive best and worst from recent history

The most common misreading is treating a composite move of two or three points as meaningful. Normalised scores are noisy at that scale; look at the sub-scores and the underlying rate before acting. A useful convention is to publish the composite with a band — “82 ± 3 over the last five runs” — which stops people reacting to noise while still surfacing genuine movement.

One layer, decomposed: why the composite movedBar chart of sub-scores for the parcel layer this month against last month: completeness fell from 91 to 88, logical consistency from 94 to 71, positional accuracy held at 96, temporal quality at 99 and thematic accuracy at 93.logical consistency71 — was 94, the causecompleteness88 — was 91thematic accuracy93 — unchangedpositional accuracy96 — unchangedtemporal quality99 — unchangedThe composite fell from 93 to 84. One sub-score explains almost all of it, which is why the decomposition ships with the headline number.
A composite is only useful if a fall can be attributed — publish the components beside it.

Gotchas & Edge Cases

A dimension with no rules is not a perfect dimension. If a point layer has no topology rules, logical_consistency should be excluded from the weighting, not scored 100. The weight_coverage figure exists precisely to make that exclusion visible.

Four ways a scorecard misleads, and the guard for eachGrid of four scorecard failure modes with what they hide and the guard that prevents each: averaging over a blocker, scoring an unmeasured dimension, comparing unlike layers and rewarding rule removal.HidesGuardAverage over a blockerone blocking defect in a 98 scorecap the grade on blockersUnmeasured dimensionno rules scored as perfectreport weight coverageLeague tablefewer rules scores highercompare like with like onlyRule removaldeleting a rule raises the scoretrack applicable rule countThree of the four are invisible in the headline number, which is why each one needs an explicit guard rather than good intentions.
Every one of these has been shipped by somebody — the guards exist because the failure modes are predictable.

Historical normalisation embeds historical mediocrity. Scoring against your own past means a layer that has always been poor can score well while remaining poor in absolute terms. Publish the raw rate alongside the sub-score so absolute quality stays visible, and re-anchor the scale when a step change in quality is achieved.

Rule additions look like regressions. A new rule finds defects that were always there, and the score drops. This is correct behaviour and must be annotated, or the team learns that adding rules makes them look worse — which is how rule sets stop growing.

Weights must sum to something you can explain. Weights invented in a meeting and never revisited are the norm. Tie them to the layer’s stated purpose and re-review when that purpose changes, as part of the stewardship cycle in Data Stewardship Roles and Responsibilities.

Small layers produce wild scores. A layer with 200 features moves several points on a single finding. Suppress the score below a minimum denominator and show the raw counts instead.

One score per layer, not per file. Scoring individual files or partitions produces a distribution nobody can act on. Aggregate to the layer, and use the drill-down for locality.

When to Escalate

  • A layer capped at C for several consecutive runs is a backlog escalation, not a measurement question — the blockers have an owner and are not being cleared.
  • A composite that cannot be reconstructed from published components means the pipeline and the dashboard have diverged. Fix that before anyone makes a decision on the number.
  • Weights that nobody will own indicate the layer has no clear declared purpose. That is a governance gap worth closing before the scorecard is published at all.
  • Persistent disagreement about a dimension’s normalisation usually means two teams are measuring different denominators. Settle the denominator first; the scale argument disappears with it.

Frequently Asked Questions

How do I choose the dimension weights?

From what the layer is used for. An emergency routing network weights positional accuracy and logical consistency heavily and thematic accuracy lightly; a land-use layer does the reverse. Write the weights next to the layer's declared purpose, review them when the purpose changes, and version them — a scorecard whose weights drift silently cannot be compared over time.

Why cap the grade when a blocker fires?

Because averages hide blockers. A layer with one overlapping parcel and otherwise perfect metrics scores 98 and looks publishable, when in fact it has a defect that stops a downstream process. Capping the grade whenever any blocker is present keeps the composite honest without abandoning the average.

Should the scorecard compare layers against each other?

Cautiously. Layers differ in what is measurable — a point layer has no topology dimension — so a league table rewards layers with fewer applicable rules. Compare a layer against its own history first, and against peers only within the same layer type and rule coverage.

How do I stop the score becoming a target people game?

Publish the components and the raw rates beside the score, and review rule coverage as part of the scorecard. The easiest way to raise a score is to remove a rule, so any drop in the number of applicable rules should be as visible as a change in the score itself.

Related

Back to Spatial Data Quality Metrics and Reporting