Streaming Spatial Data Validation

Batch validation gets to see the whole dataset. A streaming validator sees one event at a time, in an order it does not control, with no promise that the next event is not from three minutes ago. That constraint decides everything: which rules are possible, how much state each one costs, what “a violation” even means when the data it depends on has not arrived yet. This topic sets out how to classify spatial rules by their state requirements, how to run the ones that fit, how to window the ones that need neighbours, and how to fail without stalling — the moving-data counterpart to the batch design in Batch Processing Large Spatial Datasets.

Prerequisites

• A stream processor with event-time semantics — Kafka Streams, Apache Flink, or a Python consumer built on confluent-kafka with your own windowing. Event-time support matters because sensor data arrives out of order and processing-time windows silently mix intervals.
• Python 3.10+ with shapely 2.0+ for per-event geometry work, or the equivalent JTS calls if you run on the JVM.
• A declared event schema with a CRS field, a timestamp field and an entity identifier. Streaming validation with an implicit schema is guesswork; register the schema and version it as described in Attribute Schema Mapping for Spatial Datasets.
• A dead-letter topic and a policy for what goes into it. Every streaming design needs somewhere for bad events to go that is not “the log”.
• A latency budget. Streaming validation exists because someone needs an answer quickly; the number they need it in decides how much windowing you can afford.

Core Concepts & Architecture

The organising question is what state does this rule need, and there are only three answers.

Stateless per-event rules need nothing but the event. Does the payload match the schema? Is the CRS declared and canonical? Is the geometry valid per the OGC rules? Are the coordinates inside the plausible extent? Do the attributes satisfy their domains? These run at wire speed, they parallelise perfectly across partitions, and they should run first so that everything downstream can assume a well-formed event.

Per-entity sequence rules need a small, bounded memory of the same entity’s past. Speed between consecutive positions, implausible jumps, heading reversals, duplicate timestamps — all of these need exactly one prior record per tracked entity. The state is small (a coordinate, a timestamp, perhaps a heading) but it is unbounded in the number of entities, so it needs an eviction policy: a vehicle that stopped reporting three days ago should not still occupy state.

Cross-entity rules need other entities, and on a stream that means a window. Overlap between concurrently reported footprints, proximity alerts, coverage within an area — these are evaluated over the events that fell inside a time window, which is a different question from the batch equivalent. A batch overlap check asks “do these two polygons overlap”; the streaming version asks “did these two polygons overlap among the events seen in this window”, and the answer can change when a late event arrives.

Some rules simply do not fit. Coverage completeness — the “must not have gaps” family described in Understanding OGC Topology Rules — needs every feature at once. Attempting it on a stream produces a rolling approximation that is wrong in a way nobody can characterise. Run it in the batch pass and let the stream handle what streams can.

Designing for Scale

Partitioning decides both correctness and throughput. Partition by entity identifier and every sequence rule becomes local: all events for one vehicle land on one consumer, so the prior-position lookup is an in-memory dictionary rather than a distributed query. Partition by geography — an H3 cell or a tile identifier — and cross-entity rules become local instead, at the cost of entities crossing partition boundaries as they move.

You cannot have both, which is why mature designs run two stages: a per-entity stage keyed by identifier for sequence checks, then a re-keyed stage by spatial cell for proximity and overlap checks. The re-key costs a shuffle, so it should carry only what the second stage needs.

Reference data is the other scaling trap. A rule such as “the event must fall inside the operating area” is trivial in batch and lethal in a stream if it is implemented as a database query per event. Load the reference geometry into the operator, build an index once, and refresh on a schedule or via a broadcast topic. A million-event-per-hour stream cannot afford a network round trip per event; it can comfortably afford an in-memory STRtree query per event.

State size follows entity count, not event rate. Ten thousand vehicles reporting every second cost the same state as ten thousand vehicles reporting every minute; only the throughput differs. Size the state store from the entity population and set the eviction TTL from how long an entity may plausibly go quiet.

Rule Evaluation Strategies

Order the per-event checks cheapest-first and short-circuit. Schema parsing fails fastest, then coordinate range, then CRS, then geometry validity, then attribute domains. There is no value in running a validity check on a payload that failed to parse, and short-circuiting keeps the median event cost close to the cost of the first check.

For sequence rules, the pattern is a fold over the entity’s history with a bounded accumulator:

# streaming/sequence.py — bounded per-entity state for sequence checks
from dataclasses import dataclass
from math import radians, sin, cos, asin, sqrt


@dataclass
class LastSeen:
    lon: float
    lat: float
    ts: float          # epoch seconds


def haversine_m(a: LastSeen, lon: float, lat: float) -> float:
    r = 6371008.8
    dlon, dlat = radians(lon - a.lon), radians(lat - a.lat)
    h = sin(dlat / 2) ** 2 + cos(radians(a.lat)) * cos(radians(lat)) * sin(dlon / 2) ** 2
    return 2 * r * asin(sqrt(h))


def check_sequence(state: dict[str, LastSeen], entity: str, lon: float, lat: float,
                   ts: float, max_speed_ms: float = 55.0) -> list[dict]:
    """Speed and ordering checks against the entity's previous event only."""
    findings = []
    prev = state.get(entity)
    if prev is not None:
        dt = ts - prev.ts
        if dt < 0:
            findings.append({"rule": "STR_ORDER_001", "severity": "warning",
                             "message": f"event {dt:.1f}s older than the last seen position"})
        elif dt == 0:
            findings.append({"rule": "STR_DUP_001", "severity": "warning",
                             "message": "duplicate timestamp for this entity"})
        else:
            speed = haversine_m(prev, lon, lat) / dt
            if speed > max_speed_ms:
                findings.append({"rule": "STR_SPEED_001", "severity": "blocker",
                                 "message": f"implied speed {speed:.1f} m/s exceeds {max_speed_ms}"})
    # Advance state only for events that are not older than what we hold.
    if prev is None or ts >= prev.ts:
        state[entity] = LastSeen(lon, lat, ts)
    return findings

The final two lines matter more than the arithmetic above them: updating state from an out-of-order event corrupts every subsequent comparison for that entity, and the corruption is silent.

Windowed cross-entity rules follow the standard event-time pattern — a window length, an allowed lateness, and a side output for events later than that. The spatial part is unremarkable once the windowing is right: build an index over the window’s contents and evaluate predicates exactly as a batch job would.

Error Handling & Remediation

A streaming validator must never block on a bad event. The three destinations are: pass to the validated topic, flag and pass with an annotation, or reject to the dead-letter topic. The severity model is the same one used in the batch pipeline, described in Categorizing and Prioritizing Spatial Errors, and keeping it identical is what lets a single dashboard cover both.

Dead-letter events must carry enough context to be reprocessed: the original payload, the rule identifier that rejected it, the consumer version, the offset and the timestamp. A dead-letter topic of bare payloads is a graveyard; one with rule identifiers is a work queue.

Retries deserve care. Transient failures — a broker hiccup, a temporary state-store error — should retry with backoff. Data failures should not: an invalid geometry will be invalid on the third attempt too, and retrying it consumes capacity while delaying every event behind it. This is the same distinction drawn in Asynchronous Validation Workflows, and it is even more consequential on a stream because the queue is shared.

Backpressure is the failure mode that surprises teams new to streaming spatial work. A geometry repair or a reference join that is slightly too slow does not fail — it accumulates lag, and lag becomes hours before anyone notices. Alert on consumer lag as a first-class signal, not on error rate alone.

Observability, Lineage & Compliance

Three signals describe a streaming validator’s health, and they must be separated. Consumer lag says whether the pipeline is keeping up. Rejection rate says whether the data is healthy. Late-event rate says whether the windowing assumptions still hold. A rising rejection rate with flat lag is a data problem; rising lag with a flat rejection rate is a capacity problem; a rising late-event rate means the allowed lateness needs revisiting or an upstream system has changed its buffering.

Lineage on a stream is per window rather than per run. Emit an event when a window closes, carrying the window bounds, the counts by severity, the rule-set version and the number of late events admitted. That gives auditors the same reproducibility story that batch runs get from the events described in Tracking Spatial Data Lineage with OpenLineage.

Retention is a compliance question as much as a technical one. Dead-letter events contain the raw payload, which for tracking data is often personal data. Set a retention period deliberately and document it, rather than letting a topic default to infinite.

Best Practices & Anti-Patterns

• Do run the stateless checks first and short-circuit. Median event cost should be close to the cheapest check, not the average of all of them.
• Do partition by entity for sequence rules and re-key for spatial rules; trying to serve both from one partitioning produces either a distributed lookup or a wrong answer.
• Do publish the allowed lateness as part of the pipeline’s contract — consumers need to know when a window’s result is final.
• Do keep the severity model and result contract identical to the batch pipeline.
• Don’t query a database per event. Cache the reference layer in the operator and refresh it deliberately.
• Don’t run ST_MakeValid inline; it is slow enough to cause lag and can change the geometry type mid-stream.
• Don’t update per-entity state from out-of-order events.
• Don’t attempt coverage or gap detection on a stream — the answer is undefined without the complete set.
• Don’t let the dead-letter topic become write-only. If nobody consumes it, the pipeline is discarding data with extra steps.

Frequently Asked Questions

Related

• Validating GPS Tracks for Speed and Teleport Outliers — the sequence rules in full, with thresholds
• Running Geometry Checks in a Kafka Consumer — a working consumer with dead-letter routing
• Handling Late and Out-of-Order Spatial Events — watermarks, allowed lateness and window correctness
• Asynchronous Validation Workflows — the queue-based sibling of this design
• Batch Processing Large Spatial Datasets — where the rules that need the whole dataset live

Back to Validation Pipeline Architecture