Generating Synthetic Invalid Geometries for Tests

Every geometry rule needs a geometry that breaks it, and waiting for production to supply one is not a plan. This guide builds a catalogue of deliberately invalid geometries — one minimal generator per defect class, each asserted against the GEOS reason string it should produce — plus the valid look-alikes that catch over-broad rules. The catalogue then does double duty: it is the fixture source for individual rule tests, and it is the coverage check that stops a defect class being added without a rule to detect it. It supplies the fixtures assumed by Testing Spatial Validation Code.

Prerequisites

• shapely 2.0+ and pytest 8.x. GEOS 3.10 or newer is assumed; reason strings differ slightly across older versions.
• A list of the defect classes your rules claim to detect. The catalogue is derived from that list, not the other way round — if a rule exists with no fixture, that gap is the first finding.
• Familiarity with the OGC validity definitions described in Geometry Validity Checks for Vector Data, because the fixture names should match the vocabulary the rules use.

Step-by-Step Procedure

Step 1 — One minimal generator per defect class

# tests/fixtures/invalid.py
import json

from shapely.geometry import Polygon, MultiPolygon, LineString, shape
from shapely import wkt


def bowtie() -> Polygon:
    """Exterior ring crosses itself once, at (5, 5)."""
    return Polygon([(0, 0), (10, 10), (10, 0), (0, 10)])


def hole_outside_shell() -> Polygon:
    """Interior ring lies entirely beyond the exterior."""
    shell = [(0, 0), (10, 0), (10, 10), (0, 10)]
    hole = [(12, 2), (16, 2), (16, 6), (12, 6)]
    return Polygon(shell, [hole])


def nested_holes() -> Polygon:
    """One interior ring contained inside another."""
    shell = [(0, 0), (20, 0), (20, 20), (0, 20)]
    outer_hole = [(4, 4), (16, 4), (16, 16), (4, 16)]
    inner_hole = [(7, 7), (12, 7), (12, 12), (7, 12)]
    return Polygon(shell, [outer_hole, inner_hole])


def duplicate_rings() -> MultiPolygon:
    """Two identical parts in one multipolygon."""
    part = Polygon([(0, 0), (5, 0), (5, 5), (0, 5)])
    return MultiPolygon([part, part])


def degenerate_ring() -> Polygon:
    """Three collinear points: zero area, no interior."""
    return Polygon([(0, 0), (5, 5), (10, 10)])


def unclosed_ring() -> Polygon:
    """Shapely closes rings, so this defect must be built at the text level."""
    return wkt.loads("POLYGON ((0 0, 10 0, 10 10, 0 10, 0 5))")


def self_touching_line() -> LineString:
    """Non-simple linestring: revisits a vertex."""
    return LineString([(0, 0), (10, 0), (5, 5), (5, -5)])


def near_duplicate_vertices(epsilon: float = 1e-9) -> Polygon:
    """Two vertices closer together than the working precision."""
    return Polygon([(0, 0), (10, 0), (10 + epsilon, 1e-12), (10, 10), (0, 10)])


def geojson_with_reversed_winding() -> dict:
    """Structurally fine, but exterior ring is clockwise (RFC 7946 wants counter-clockwise)."""
    return json.loads(json.dumps({
        "type": "Polygon",
        "coordinates": [[[0, 0], [0, 10], [10, 10], [10, 0], [0, 0]]],
    }))

Verification: each function returns in a line or two and its coordinates can be sketched on paper. A generator that needs a loop is usually testing the generator rather than the rule.

Step 2 — Pin each fixture to its reason string

# tests/test_fixture_integrity.py
import pytest
import shapely

from tests.fixtures import invalid

EXPECTED_REASON = {
    "bowtie": "Self-intersection",
    "hole_outside_shell": "Hole lies outside shell",
    "nested_holes": "Holes are nested",
    "duplicate_rings": "Duplicate Rings",
    "degenerate_ring": "Too few points in geometry component",
}


@pytest.mark.parametrize("name,fragment", EXPECTED_REASON.items())
def test_fixture_produces_expected_reason(name, fragment):
    geom = getattr(invalid, name)()
    assert not shapely.is_valid(geom), f"{name} should be invalid"
    reason = shapely.is_valid_reason(geom)
    assert fragment.lower() in reason.lower(), (
        f"{name} drifted: expected {fragment!r}, GEOS says {reason!r}")

Verification: change a coordinate in bowtie so the rings no longer cross and confirm this test fails. That is the whole purpose — a fixture that quietly becomes a different defect makes its rule test meaningless while still passing.

Step 3 — Add the valid look-alikes

# tests/fixtures/valid_lookalikes.py
from shapely.geometry import Polygon, MultiPolygon, LineString


def figure_of_eight_but_valid() -> MultiPolygon:
    """Looks like a bowtie; is two disjoint valid parts."""
    return MultiPolygon([
        Polygon([(0, 0), (5, 5), (0, 10)]),
        Polygon([(10, 0), (5, 5), (10, 10)]),
    ])


def legitimate_hole() -> Polygon:
    """A donut — valid, and easily confused with hole_outside_shell by a sloppy rule."""
    return Polygon([(0, 0), (20, 0), (20, 20), (0, 20)],
                   [[(6, 6), (14, 6), (14, 14), (6, 14)]])


def very_thin_but_legitimate() -> Polygon:
    """A 200 m rail corridor, 3 m wide: thin, not a sliver."""
    return Polygon([(0, 0), (200, 0), (200, 3), (0, 3)])


def touching_at_a_point() -> MultiPolygon:
    """Two parts meeting at one vertex — valid, and a classic false-positive source."""
    return MultiPolygon([
        Polygon([(0, 0), (5, 0), (5, 5), (0, 5)]),
        Polygon([(5, 5), (10, 5), (10, 10), (5, 10)]),
    ])


def closed_but_not_simple_ring() -> LineString:
    """Closed ring that touches itself — is_ring is False even though it closes."""
    return LineString([(0, 0), (10, 0), (5, 5), (10, 10), (0, 10), (5, 5), (0, 0)])

Verification: run the entire rule set over the look-alikes and assert zero findings. This suite catches over-broad rules, which are far more damaging than missing ones — a sliver rule that deletes rail corridors destroys data, while a missing rule merely fails to find something.

Step 4 — Expose a catalogue

# tests/fixtures/catalogue.py
from tests.fixtures import invalid, valid_lookalikes

DEFECTS = {
    "self_intersection": (invalid.bowtie, "blocker"),
    "hole_outside_shell": (invalid.hole_outside_shell, "blocker"),
    "nested_holes": (invalid.nested_holes, "blocker"),
    "duplicate_rings": (invalid.duplicate_rings, "warning"),
    "degenerate_ring": (invalid.degenerate_ring, "blocker"),
    "unclosed_ring": (invalid.unclosed_ring, "blocker"),
    "non_simple_line": (invalid.self_touching_line, "warning"),
    "near_duplicate_vertex": (invalid.near_duplicate_vertices, "informational"),
}

LOOKALIKES = {
    "two_disjoint_triangles": valid_lookalikes.figure_of_eight_but_valid,
    "donut": valid_lookalikes.legitimate_hole,
    "rail_corridor": valid_lookalikes.very_thin_but_legitimate,
    "point_touching_parts": valid_lookalikes.touching_at_a_point,
}

Verification: the catalogue keys should read as the vocabulary your reports use. If a report says “self-intersection” and the catalogue says “bowtie”, one of the two is wrong and the mismatch will surface in every conversation about a finding.

Step 5 — Use the catalogue as a coverage gate

# tests/test_rule_coverage.py
import pytest

from rules.engine import evaluate_all
from tests.fixtures.catalogue import DEFECTS, LOOKALIKES


@pytest.mark.parametrize("name", sorted(DEFECTS))
def test_every_defect_is_detected_by_something(name):
    generator, expected_severity = DEFECTS[name]
    findings = evaluate_all([generator()])
    assert findings, f"no rule reports {name} — the defect class is invisible"
    assert any(f["severity"] == expected_severity for f in findings), (
        f"{name} detected, but not at severity {expected_severity}: "
        f"{[f['severity'] for f in findings]}")


@pytest.mark.parametrize("name", sorted(LOOKALIKES))
def test_no_rule_fires_on_a_valid_lookalike(name):
    findings = evaluate_all([LOOKALIKES[name]()])
    assert findings == [], f"false positive on {name}: {findings}"

Verification: add a new entry to DEFECTS before writing its rule. The coverage test fails immediately with a message naming the invisible defect class, which is exactly the workflow you want — the fixture comes first, the rule follows.

Interpreting Results

The last row is the ongoing work. Every time production surfaces a defect the suite did not anticipate, the fix is two commits: add the fixture, then add or amend the rule. Over a year that turns the catalogue into an accurate description of what your data actually does wrong, which is far more valuable than a generic list of OGC violations.

Gotchas & Edge Cases

Shapely repairs some defects on construction. Rings are closed automatically, and some constructors reorder coordinates. Any defect that lives in the serialisation — an unclosed ring, a malformed coordinate array, a mixed-dimension geometry — must be built as text or JSON and parsed, not constructed in memory.

GEOS reason strings are version-dependent. The wording has changed between releases, and matching an exact string makes the suite brittle across environments. Match a distinctive fragment, and pin the GEOS version in the test environment so a change is a deliberate upgrade rather than a surprise.

buffer(0) is not a neutral operation in a fixture. It is a repair, and using it to normalise a fixture can silently turn an invalid geometry valid before the rule ever sees it. Keep repairs out of fixture code entirely.

Very small epsilons stop being representable. A near-duplicate vertex at 1e-15 of a coordinate near 400,000 does not exist in double precision — the two vertices are literally the same point. Scale the epsilon to the coordinate magnitude, or the fixture tests floating-point arithmetic rather than your rule.

Multi-defect fixtures are hard to reason about. A polygon that is both self-intersecting and has a hole outside its shell will report only the first defect GEOS encounters. Keep one defect per fixture and compose them only in a deliberate “multiple defects” case.

When to Escalate

• A defect class you cannot construct synthetically usually means it is not a geometry defect but a data or process defect — a wrong CRS, a missing attribute, an ordering assumption. Those belong to different rule families and different fixtures.
• A rule that cannot be made to pass the look-alike suite is a rule with an unresolved definition. Take it back to the data steward and pin down the boundary case explicitly, as described in Defining Spatial Data Quality Policies.
• Reason strings differing between the development environment and production means the two run different GEOS builds. That difference will also change validation results, so resolve it before trusting either.

Frequently Asked Questions

Related

• Testing Spatial Validation Code — the surrounding suite structure and tiering
• Property-Based Testing of Spatial Rules with Hypothesis — where randomised generation does belong
• Geometry Validity Checks for Vector Data — the defect vocabulary these fixtures are named after

Back to Testing Spatial Validation Code