Property-Based Testing of Spatial Rules with Hypothesis

Example-based tests check the cases you thought of. Geometry bugs live in the cases you did not: three collinear vertices, a ring whose last coordinate is a hair off the first, a polygon with an area of 1e-14, coordinates near the precision limit of a projected system. Property-based testing inverts the process — you state what must always be true, and a generator spends its time trying to prove you wrong. This guide builds geometry strategies, writes the invariants worth asserting for validation code, uses shrinking to turn a failure into a minimal reproduction, and keeps the suite fast and deterministic enough for continuous integration. It extends the suite structure in Testing Spatial Validation Code.

Prerequisites

  • hypothesis 6.x, pytest 8.x, shapely 2.0+.
  • A rule implementation with a stable signature — properties are stated about a function, so a rule that reaches into global state cannot be property-tested meaningfully.
  • The deterministic fixture catalogue from Generating Synthetic Invalid Geometries for Tests. Property tests complement example tests; they do not replace them.
  • Somewhere to persist the example database — a directory in the repository, or a CI cache. Without it, every run starts its search from scratch and known failures are not replayed.

Step-by-Step Procedure

Step 1 — Build geometry strategies

# tests/strategies.py
from hypothesis import strategies as st
from shapely.geometry import Polygon, LineString, Point

COORD = st.floats(min_value=-1e5, max_value=1e5,
                  allow_nan=False, allow_infinity=False, width=64)


@st.composite
def points(draw):
    return Point(draw(COORD), draw(COORD))


@st.composite
def rings(draw, min_vertices: int = 3, max_vertices: int = 12):
    """A closed ring of arbitrary shape — frequently self-intersecting, by design."""
    n = draw(st.integers(min_vertices, max_vertices))
    coords = draw(st.lists(st.tuples(COORD, COORD), min_size=n, max_size=n))
    return Polygon(coords)


@st.composite
def simple_polygons(draw, radius_range=(1.0, 1000.0), vertices=(4, 10)):
    """A star-shaped polygon around a centre — always simple, useful for 'must not fire' tests."""
    import math
    cx, cy = draw(COORD), draw(COORD)
    n = draw(st.integers(*vertices))
    radii = draw(st.lists(st.floats(*radius_range, allow_nan=False, allow_infinity=False),
                          min_size=n, max_size=n))
    coords = [(cx + r * math.cos(2 * math.pi * i / n),
               cy + r * math.sin(2 * math.pi * i / n))
              for i, r in enumerate(radii)]
    return Polygon(coords)


@st.composite
def linestrings(draw, min_vertices: int = 2, max_vertices: int = 10):
    n = draw(st.integers(min_vertices, max_vertices))
    return LineString(draw(st.lists(st.tuples(COORD, COORD), min_size=n, max_size=n)))

Verification: draw a few hundred examples from each strategy and check the mix. rings() should produce a healthy share of invalid polygons — that is its job — while simple_polygons() should produce almost none. Two strategies with different validity profiles let you write both “must handle anything” and “must not fire on good data” properties.

Properties worth asserting about a spatial ruleGrid of five invariants with what each one catches and the failure it has caught in practice: output validity, idempotence, area bounds, symmetry and empty-input behaviour.CatchesTypical real failureOutput always validrepair returning broken geometrycollapsed collection on collinear inputIdempotencenon-deterministic repairsecond pass shifts a vertexArea never growsrepair inventing areabuffer used as a repairSymmetryargument-order dependencepair reported once in one order onlyEmpty inputcrash on an empty frameindex error in the aggregatorAll five are stated without re-implementing the rule, which is what keeps a property test from testing two versions of the same misunderstanding.
Five invariants cover most of what example tests miss, and none of them requires knowing the right answer.

Step 2 — State invariants, not re-implementations

# tests/test_repair_properties.py
from hypothesis import given, settings, strategies as st
from shapely import normalize

from rules.repair import repair_geometry
from tests.strategies import rings, simple_polygons

TOL = 1e-9


@given(rings())
@settings(max_examples=200, deadline=None)
def test_output_is_always_valid(poly):
    """Whatever goes in, what comes out must satisfy the OGC rules or be empty."""
    result = repair_geometry(poly)
    assert result.is_valid or result.is_empty


@given(rings())
@settings(max_examples=200, deadline=None)
def test_repair_is_idempotent(poly):
    once = repair_geometry(poly)
    twice = repair_geometry(once)
    assert normalize(once).equals_exact(normalize(twice), TOL)


@given(simple_polygons())
@settings(max_examples=200, deadline=None)
def test_valid_input_is_returned_unchanged(poly):
    """A repair that edits already-valid geometry is a data-corruption bug."""
    result = repair_geometry(poly)
    assert normalize(result).equals_exact(normalize(poly), TOL)


@given(rings())
@settings(max_examples=200, deadline=None)
def test_repair_never_invents_area(poly):
    reference = poly.buffer(0)
    assert repair_geometry(poly).area <= reference.area + 1e-6

Verification: temporarily break the repair function — for instance, apply a small buffer — and confirm the third property fails. A property suite that passes against a deliberately broken implementation is asserting nothing.

Step 3 — Let shrinking do the diagnosis

# tests/test_predicate_properties.py
from hypothesis import given, settings, example
from shapely.geometry import Polygon

from rules.topology import must_not_overlap
from tests.strategies import simple_polygons


@given(simple_polygons(), simple_polygons())
@settings(max_examples=300, deadline=None)
def test_overlap_is_symmetric(a, b):
    """Order of arguments must not change the answer."""
    forward = must_not_overlap([a, b])
    backward = must_not_overlap([b, a])
    assert len(forward) == len(backward)


@given(simple_polygons())
@settings(max_examples=200, deadline=None)
@example(Polygon([(0, 0), (1, 0), (1, 1), (0, 1)]))   # a promoted counterexample
def test_self_comparison_reports_once(poly):
    assert len(must_not_overlap([poly, poly])) <= 1

Verification: when a property fails, Hypothesis reduces the input before reporting it — coordinates collapse toward zero, vertex counts fall to the minimum that still reproduces. Read the shrunk case first; it is usually an obvious degenerate shape that explains the bug immediately, whereas the original random input looks like noise.

Step 4 — Control determinism in CI

# tests/conftest.py (property profiles)
import os

from hypothesis import HealthCheck, Phase, settings

settings.register_profile(
    "dev", max_examples=50, deadline=None,
    suppress_health_check=[HealthCheck.too_slow],
)
settings.register_profile(
    "ci", max_examples=100, deadline=None, derandomize=True,
    print_blob=True,                       # a reproducible blob in the failure output
)
settings.register_profile(
    "nightly", max_examples=5_000, deadline=None,
    phases=[Phase.explicit, Phase.reuse, Phase.generate, Phase.target, Phase.shrink],
)
settings.load_profile(os.getenv("HYPOTHESIS_PROFILE", "dev"))
# .github/workflows/tests.yml (fragment)
- name: Restore the Hypothesis example database
  uses: actions/cache@v4
  with:
    path: .hypothesis/examples
    key: hypothesis-${{ github.ref }}-${{ github.sha }}
    restore-keys: hypothesis-${{ github.ref }}-

Verification: run the ci profile twice on an unchanged tree and confirm identical output. derandomize=True makes the generated examples a deterministic function of the test, which turns “it failed once in CI” into a reproducible case.

Step 5 — Promote every counterexample

# tests/test_regressions.py
"""Named regressions found by property tests. One test per real bug."""
from shapely.geometry import Polygon

from rules.repair import repair_geometry
from rules.topology import must_not_overlap


def test_collinear_triangle_does_not_crash_repair():
    """Found 2026-05-14 by test_output_is_always_valid; GEOS returned an empty collection."""
    degenerate = Polygon([(0, 0), (1, 1), (2, 2)])
    result = repair_geometry(degenerate)
    assert result.is_empty or result.is_valid


def test_zero_area_pair_is_not_an_overlap():
    """Found 2026-06-02 by test_overlap_is_symmetric; zero-area inputs matched both ways."""
    flat = Polygon([(0, 0), (10, 0), (5, 0)])
    assert must_not_overlap([flat, flat]) == []

Verification: each promoted test should fail against the code as it was before the fix. If it passes on the old code, the property test found something else and the reproduction was not captured correctly.

Interpreting Results

Signal Meaning Action
A property fails on a shrunk degenerate input The rule has an unhandled edge case Fix, then promote to a named regression
A property fails only at high example counts A rare interaction, often floating point Investigate; do not “fix” by lowering the count
Many examples filtered out The strategy is too loose and assume() is doing the work Reshape the strategy
Property passes but example tests fail The property is too weak to constrain the behaviour Strengthen the invariant
Shrinking takes minutes The failure depends on large input Bound the strategy tighter
Nightly finds failures the CI profile misses Working as intended Promote them; the CI profile will replay them thereafter

The most valuable property in practice is idempotence, because it catches an entire class of pipeline bugs: a repair that changes its output on a second pass makes reruns non-reproducible, breaks incremental processing, and makes two runs over the same data disagree. It is also cheap to state and rarely written without a property framework prompting for it.

How shrinking turns a random failure into a readable oneFunnel showing a failing case reducing from a 12-vertex polygon with coordinates in the tens of thousands, to 8 vertices, to 5, to a 3-vertex collinear triangle at the origin — the minimal case that still reproduces the failure.Original failing case12 vertices, coordinates near 90,000First reduction8 vertices, coordinates near 1,000Second reduction5 vertices, small integersMinimal case3 collinear points at the originRead the shrunk case, never the original. The original looks like noise; the minimal case usually explains the bug on sight.
Shrinking is the diagnosis step — it is why property tests are readable at all.

Gotchas & Edge Cases

Floating-point tolerance must be scaled to coordinate magnitude. An absolute tolerance of 1e-9 is meaningless at coordinates near 1e5, where the representable spacing is already around 1e-11 and error accumulates well beyond the tolerance. Use a relative comparison, or bound the strategy’s coordinate range so the absolute tolerance is defensible.

Profiles: example counts against where they runGrid of three Hypothesis profiles against three properties — example count, determinism and where the profile runs — for the development, continuous integration and nightly settings.examplesdeterministicruns ondev50noevery saveci100yesevery pushnightly5,000noscheduleThe example database replays every previously found counterexample in all three.Determinism matters most in continuous integration, where "it failed once" has to be reproducible before anyone can act on it.
Three profiles, three jobs — the shared example database is what lets the fast one stay fast.

assume() is a performance trap. Discarding 90% of generated examples means the search explores a tenth as much for the same runtime, and Hypothesis will eventually complain about it. Express the constraint in the strategy instead.

Deadlines fail on the first, slow example. GEOS operations on the first call include library initialisation, which can exceed the default deadline and produce a confusing flaky failure. Set deadline=None for geometry properties.

Shrinking can change which bug you are looking at. Occasionally a shrunk input triggers a different failure from the original. When a shrunk case seems unrelated to the property that failed, re-run with print_blob=True and reproduce the original before concluding.

Strategies that build valid polygons are surprisingly hard. The star-shaped construction in Step 1 is one of the few simple approaches that reliably yields simple polygons. Random vertex lists are almost always self-intersecting once past five or six vertices — which is useful for “must handle anything” properties and useless for “must not fire” properties.

The example database is not a test artefact to discard. Losing it means losing the replay of every counterexample ever found. Cache it in CI and consider committing it for small suites.

When to Escalate

  • A property that cannot be satisfied is a specification problem rather than a code problem. If repair genuinely cannot be idempotent for some input class, the team needs to decide and document why, not weaken the test quietly.
  • Failures that only appear with a specific GEOS version belong upstream. Pin the version, record the behaviour difference, and raise it with the library maintainers if the behaviour looks incorrect.
  • A rule that fails symmetry may be correct and asymmetric by design — a “must be covered by” rule is directional. Adjust the property to match the rule’s semantics rather than forcing symmetry.
  • Nightly runs finding new failures for weeks suggests the rule surface is genuinely under-specified. That is worth a design conversation about the rule set, along the lines set out in Defining Spatial Data Quality Policies.

Frequently Asked Questions

What makes a good property for a spatial rule?

Something that must be true for every input, expressed without reimplementing the rule. Good examples: repair output is always valid, repair is idempotent, repair never increases area, a symmetric predicate gives the same answer in both argument orders, and a rule over an empty input returns no findings. Bad example: re-deriving the expected result with a second implementation, which just tests two versions of the same misunderstanding.

How do I stop the generator producing degenerate garbage?

Constrain the strategy rather than filtering the output. Bound coordinate magnitudes, exclude NaN and infinity, and set a minimum vertex count. Heavy use of assume() throws away most generated examples and slows the search dramatically; a well-shaped strategy produces useful inputs by construction.

Are property tests too slow for continuous integration?

Not with a profile. Run a small example count on every commit — fifty is enough to catch regressions once the example database holds previous failures — and a much larger count nightly. Hypothesis replays known counterexamples first, so the fast profile still covers everything that has ever failed.

Should a found counterexample stay as a property test only?

No. Copy it into an explicit example-based test with a descriptive name. The property test proves the class of bug is gone; the named test documents the specific case for the next person, survives changes to the strategy, and fails with an immediately readable message.

Related

Back to Testing Spatial Validation Code