Snapshot Testing Validation Reports in Pytest

Unit tests prove that a rule fires. They cannot prove that the report a consumer parses still has the fields it expects, that findings are still grouped by layer, that severities still roll up the way the dashboard assumes, or that a refactor did not quietly reorder everything. A snapshot test does: it runs the whole validator over a fixed fixture and compares the output against a golden copy, so any change in the report’s shape shows up as a reviewable diff. The catch is churn — snapshots that change for uninteresting reasons get regenerated without reading, and then they protect nothing. This guide builds one that stays stable, completing the tiering described in Testing Spatial Validation Code.

Prerequisites

• pytest 8.x and Python 3.10+. No snapshot plugin is required; the thirty lines below are usually preferable to a dependency.
• A deterministic validator — same input, same output. If two runs over one fixture differ, fix that before snapshotting, because the snapshot will simply record the non-determinism.
• The fixture catalogue from Generating Synthetic Invalid Geometries for Tests, so the snapshot fixture is small and its contents are understood.
• A report with a stable result contract. Snapshotting an ad-hoc dictionary just freezes the ad-hoc-ness.

Step-by-Step Procedure

Step 1 — Canonicalise before you store anything

# tests/snapshot/canonical.py
import re
from typing import Any

VOLATILE_KEYS = {"run_id", "started_at", "finished_at", "duration_s", "hostname", "pid"}
PATH_RE = re.compile(r"(/[\w.\-]+)+/")


def canonicalise(obj: Any, float_places: int = 6) -> Any:
    """Strip everything that legitimately varies between two identical runs."""
    if isinstance(obj, dict):
        return {
            k: canonicalise(v, float_places)
            for k, v in sorted(obj.items())
            if k not in VOLATILE_KEYS
        }
    if isinstance(obj, list):
        items = [canonicalise(v, float_places) for v in obj]
        # Sort findings deterministically; report order must not depend on dict iteration.
        if items and isinstance(items[0], dict) and "rule" in items[0]:
            items.sort(key=lambda d: (d.get("feature_id", ""), d.get("rule", "")))
        return items
    if isinstance(obj, float):
        return round(obj, float_places)
    if isinstance(obj, str):
        return PATH_RE.sub("<path>/", obj)
    return obj

Verification: run the validator twice over the same fixture, canonicalise both, and assert equality. If that fails, the canonicaliser is incomplete — find the differing key and decide whether it is volatile (strip it) or genuinely non-deterministic (fix the validator).

Step 2 — Store and compare the golden copy

# tests/snapshot/plugin.py
import json
import os
from pathlib import Path

import pytest

SNAPSHOT_DIR = Path(__file__).parent / "__snapshots__"
UPDATE = os.getenv("UPDATE_SNAPSHOTS") == "1"


@pytest.fixture
def snapshot(request):
    def _compare(payload, name: str | None = None):
        SNAPSHOT_DIR.mkdir(exist_ok=True)
        filename = f"{name or request.node.name}.json"
        path = SNAPSHOT_DIR / filename
        rendered = json.dumps(payload, indent=2, sort_keys=True, ensure_ascii=False)

        if UPDATE or not path.exists():
            path.write_text(rendered + "\n", encoding="utf-8")
            if not UPDATE:
                pytest.skip(f"snapshot created: {path.name} — review and commit it")
            return

        expected = json.loads(path.read_text(encoding="utf-8"))
        assert payload == expected, (
            f"report differs from {path.name}. "
            f"Review the change; if intended, re-run with UPDATE_SNAPSHOTS=1."
        )
    return _compare

Verification: delete a snapshot file and run the suite. The test should create it and skip with a message telling you to review it — never silently pass, which would let a wrong first snapshot become the baseline.

Step 3 — Diff structurally so failures are readable

# tests/snapshot/diff.py
def structural_diff(actual, expected, path: str = "$") -> list[str]:
    """A short list of field-level differences, not a wall of text."""
    diffs: list[str] = []
    if type(actual) is not type(expected):
        return [f"{path}: type {type(expected).__name__} -> {type(actual).__name__}"]

    if isinstance(expected, dict):
        for key in sorted(set(expected) | set(actual)):
            if key not in actual:
                diffs.append(f"{path}.{key}: removed")
            elif key not in expected:
                diffs.append(f"{path}.{key}: added ({actual[key]!r})")
            else:
                diffs.extend(structural_diff(actual[key], expected[key], f"{path}.{key}"))
    elif isinstance(expected, list):
        if len(actual) != len(expected):
            diffs.append(f"{path}: length {len(expected)} -> {len(actual)}")
        for i, (a, e) in enumerate(zip(actual, expected)):
            diffs.extend(structural_diff(a, e, f"{path}[{i}]"))
    elif actual != expected:
        diffs.append(f"{path}: {expected!r} -> {actual!r}")
    return diffs[:25]

Verification: change one severity value in a snapshot and check the reported diff names the exact path — something like $.findings[3].severity: 'warning' -> 'blocker'. A snapshot test whose failure message is a thousand-line text diff will be regenerated rather than read.

Step 4 — Snapshot a report, not a dataset

# tests/test_report_snapshot.py
from rules.engine import validate_layer
from tests.fixtures.catalogue import DEFECTS
from tests.snapshot.canonical import canonicalise


def build_mixed_fixture():
    """A small layer containing one instance of each defect class, in a fixed order."""
    return [
        {"feature_id": f"F{i:03d}", "geometry": generator()}
        for i, (_name, (generator, _sev)) in enumerate(sorted(DEFECTS.items()))
    ]


def test_full_report_shape(snapshot):
    report = validate_layer(build_mixed_fixture(), layer_id="fixture.parcels")
    snapshot(canonicalise(report), name="mixed_defects_report")


def test_summary_only_for_large_input(snapshot):
    """Where the full report is too large to review, snapshot the summary view."""
    report = validate_layer(build_mixed_fixture() * 50, layer_id="fixture.parcels")
    summary = {
        "layer_id": report["layer_id"],
        "counts_by_rule": report["counts_by_rule"],
        "counts_by_severity": report["counts_by_severity"],
        "first_findings": report["findings"][:3],
    }
    snapshot(canonicalise(summary), name="large_input_summary")

Verification: open the committed snapshot and read it. If you cannot tell from the file what the validator is claiming about the fixture, the snapshot is too large or the report is poorly shaped — both worth fixing before the test is trusted.

Step 5 — Make updates deliberate

# Regenerating snapshots is explicit and shows up in the diff.
UPDATE_SNAPSHOTS=1 pytest tests/test_report_snapshot.py
git diff tests/snapshot/__snapshots__/     # this diff is the review artefact

# tests/test_snapshot_guard.py — stop an unreviewed regeneration slipping through
import os
import subprocess


def test_snapshots_are_committed():
    """Fails if snapshots were regenerated but not staged — catches a stray local update."""
    changed = subprocess.run(
        ["git", "status", "--porcelain", "tests/snapshot/__snapshots__"],
        capture_output=True, text=True, check=True).stdout.strip()
    assert not changed, f"uncommitted snapshot changes:\n{changed}"

Verification: regenerate a snapshot without committing it and confirm the guard fails locally. In continuous integration the working tree is clean, so the guard is inert there and costs nothing.

Interpreting Results

The distinction to hold onto: a snapshot diff is a question, not a failure. Most of the time the answer is “yes, that was the point of this change”, and the value of the test is that the change was visible rather than silent.

Gotchas & Edge Cases

Snapshots drift into being the specification. Once a golden file exists, people reason from it rather than from the contract. Keep a short written description of the result contract alongside the snapshots, and treat the snapshot as evidence rather than as the definition.

Set and dictionary ordering is not stable across Python versions. The canonicaliser sorts dictionaries and finding lists for this reason. Sets serialised directly to JSON are a common source of order churn — convert to sorted lists in the report itself.

Absolute paths and hostnames leak in constantly. They appear in error messages, in file references and in lineage fields. The path substitution in Step 1 catches most; watch for new ones when a snapshot fails on a colleague’s machine but not yours.

Geometry in a snapshot needs rounding. WKT of a repaired polygon differs in the last digits across GEOS builds. Round coordinates before writing, or snapshot a geometry hash rather than the geometry itself.

A snapshot per rule is the wrong granularity. That is what unit tests are for, and it produces dozens of files nobody reads. One or two whole-report snapshots over a well-chosen fixture do the job.

Deleting a snapshot to “fix” a failing test removes the only record of the expected output. If the intent is to accept the change, regenerate and commit the diff so it appears in review.

When to Escalate

• A snapshot diff nobody can explain means someone changed report structure without knowing it. Find the commit before accepting the new baseline; unexplained structural changes break consumers.
• A contract narrowing — a removed or renamed field — should go through the consumer teams before the snapshot is updated. The snapshot caught a breaking change; updating it silently defeats that.
• Persistent churn that resists canonicalisation usually means non-determinism in the validator itself: dictionary iteration, parallel result collection, or an unsorted spatial index result. That is a bug worth fixing on its own merits, because it also makes production runs unreproducible, contrary to the lineage goals in Observability and Lineage for Validation.
• Snapshots growing past a few hundred lines — switch to a summary view before the team stops reading them.

Frequently Asked Questions

Related

• Testing Spatial Validation Code — where snapshot tests sit among the other tiers
• Generating Synthetic Invalid Geometries for Tests — the fixture the snapshot runs over
• Categorizing and Prioritizing Spatial Errors — the roll-up structure a snapshot protects

Back to Testing Spatial Validation Code