Scoping Spatial Audits for State DOT Networks

You are managing GIS validation for a state Department of Transportation (DOT) network — thousands of centerline miles, bridge asset records, crash locations, and federally mandated submissions — and need to define exactly which datasets get audited, to what standard, and with what automated checks. This page walks through the full scoping and implementation process, from tier classification through pipeline code to CI/CD integration, as a focused extension of the broader audit scoping for municipal GIS assets approach.

Prerequisites

Before running any automated audit, verify the following environment and data conditions:

• Python 3.11+ with geopandas 0.14+, shapely 2.0+, pyproj 3.6+, networkx 3.2+, and pandas 2.0+ installed in an isolated virtual environment.
• GDAL (Geospatial Data Abstraction Library) 3.6+ available on the system PATH for reading file geodatabases (.gdb) and GeoPackages.
• CRS metadata present on every layer. Any file missing a .prj or embedded coordinate reference system (CRS) definition must be rejected at ingestion — do not silently assume a CRS.
• A data dictionary listing every layer, its owner, its update frequency, and its target CRS (EPSG code). This is the reference document the validation rules are written against.
• Access to the FHWA HPMS Field Manual schema (or your state DOT’s equivalent attribute template) for Tier 1 layers.

Gotcha upfront: geopandas.read_file silently succeeds on .gdb files even when the CRS is missing — always call .crs on the loaded GeoDataFrame and assert it is not None before any spatial operation.

Step-by-Step Procedure

Step 1 — Classify Layers into Compliance Tiers

State DOT datasets span radically different compliance obligations. Classify every layer before writing a single validation rule, or you will apply blocking checks to non-critical data and miss mandatory checks on federal submissions.

Document this classification in your data dictionary. The spatial data quality policies your organization defines should map directly to these tiers so that audit rules have a governance anchor, not just a technical one.

Verification check: Run ogr2ogr -al -so /dev/null input.gdb and compare the listed layer names against your data dictionary. Any layer present in the file but absent from the dictionary is unclassified — treat it as Tier 3 until reviewed.

Step 2 — Build the Core Validation Pipeline

Manual spatial audits cannot scale across multi-county DOT networks. The following function implements the geometry, CRS, and attribute phases as a reusable, testable unit:

import geopandas as gpd
import pandas as pd
from shapely.validation import make_valid
from pyproj import CRS
import warnings

warnings.filterwarnings("ignore", category=UserWarning)

EXPECTED_CRS = CRS.from_epsg(4269)  # NAD83 geographic — reproject to State Plane after load


def audit_dot_centerlines(
    gdb_path: str,
    layer_name: str,
    required_attrs: list[str],
    target_epsg: int = 26917,  # NAD83 / UTM zone 17N — adjust per state
) -> dict:
    """
    Validates a DOT centerline layer for geometry validity, CRS conformity,
    and attribute completeness. Returns structured audit metrics.
    """
    gdf = gpd.read_file(gdb_path, layer=layer_name)

    # Assert CRS is present before any operation
    if gdf.crs is None:
        raise ValueError(f"Layer '{layer_name}' has no CRS — ingest rejected.")

    # Reproject to target projected CRS for metric distance operations
    gdf = gdf.to_crs(epsg=target_epsg)

    # 1. Geometry validity and repair
    invalid_mask = ~gdf.geometry.is_valid
    invalid_count = int(invalid_mask.sum())
    if invalid_count > 0:
        gdf.loc[invalid_mask, "geometry"] = (
            gdf.loc[invalid_mask, "geometry"].apply(make_valid)
        )

    # 2. Attribute completeness
    missing_attrs = {
        col: int(gdf[col].isnull().sum())
        for col in required_attrs
        if col in gdf.columns
    }
    missing_cols = [col for col in required_attrs if col not in gdf.columns]
    incomplete_ids = gdf[gdf[required_attrs].isnull().any(axis=1)].index.tolist()

    # 3. Self-intersecting geometries (linestrings that cross themselves)
    self_intersecting_ids = gdf.index[~gdf.geometry.is_simple].tolist()

    # 4. Duplicate geometries
    duplicate_count = int(gdf.duplicated(subset=["geometry"], keep=False).sum())

    return {
        "total_features": len(gdf),
        "invalid_geometries_repaired": invalid_count,
        "missing_required_columns": missing_cols,
        "null_counts_per_attr": missing_attrs,
        "incomplete_record_ids": incomplete_ids,
        "self_intersecting_ids": self_intersecting_ids,
        "duplicate_geometry_count": duplicate_count,
    }

Verification check: Call the function on a known-good layer and assert result["invalid_geometries_repaired"] == 0 and result["self_intersecting_ids"] == []. If either fails on a reference layer, your source data has upstream issues that must be documented before audit results are trusted.

The rule logic here mirrors the approach in building rule engines with GeoPandas — the same predicate-per-feature pattern applies, just targeted at DOT-specific schemas.

Step 3 — Enforce LRS Calibration and Network Connectivity Rules

Linear Referencing System (LRS) calibration is the check most specific to DOT networks and most frequently overlooked. A route can pass geometry validity yet have milepost values that break event location queries entirely.

import networkx as nx
import numpy as np


def audit_lrs_calibration(gdf: gpd.GeoDataFrame) -> dict:
    """
    Checks LRS calibration integrity: monotonic mileposts, no gaps > 0.01 mi,
    no overlaps, and route continuity via graph connectivity.

    Expects columns: route_id, begin_mp, end_mp
    """
    issues = {"monotonicity_errors": [], "gaps": [], "overlaps": [], "disconnected_routes": []}

    for route_id, group in gdf.groupby("route_id"):
        seg = group.sort_values("begin_mp").reset_index(drop=True)

        # 1. Monotonicity: end_mp must be greater than begin_mp on every segment
        bad_mono = seg[seg["end_mp"] <= seg["begin_mp"]]
        if not bad_mono.empty:
            issues["monotonicity_errors"].append(
                {"route_id": route_id, "segment_indices": bad_mono.index.tolist()}
            )

        # 2. Gaps between consecutive segments (tolerance: 0.01 mi)
        gap_mask = (seg["begin_mp"].iloc[1:].values - seg["end_mp"].iloc[:-1].values) > 0.01
        gap_positions = np.where(gap_mask)[0].tolist()
        if gap_positions:
            issues["gaps"].append({"route_id": route_id, "after_segment_index": gap_positions})

        # 3. Overlaps: next begin_mp is less than current end_mp
        overlap_mask = (seg["begin_mp"].iloc[1:].values - seg["end_mp"].iloc[:-1].values) < -0.001
        overlap_positions = np.where(overlap_mask)[0].tolist()
        if overlap_positions:
            issues["overlaps"].append(
                {"route_id": route_id, "at_segment_index": overlap_positions}
            )

    # 4. Graph connectivity: each route should form a single connected component
    G = nx.Graph()
    for _, row in gdf.iterrows():
        coords = list(row.geometry.coords)
        start = (round(coords[0][0], 4), round(coords[0][1], 4))
        end = (round(coords[-1][0], 4), round(coords[-1][1], 4))
        G.add_edge(start, end, route_id=row["route_id"])

    components = list(nx.connected_components(G))
    if len(components) > 1:
        issues["disconnected_routes"] = [
            {"component_index": i, "node_count": len(c)}
            for i, c in enumerate(components)
            if len(c) > 0
        ]

    return issues

Verification check: On a clean export from your DOT’s authoritative LRS, all four issue lists should be empty. If gaps is non-empty on the reference dataset, that is a source-data defect — file it before auditing contractor submissions against the same standard.

Step 4 — Integrate Compliance Thresholds and Generate Per-District Reports

Audit results only drive action when mapped to specific regulatory requirements. For Tier 1 layers, the FHWA HPMS Field Manual specifies attribute codes, spatial accuracy thresholds, and submission timelines. Build a reporting step that cross-references audit metrics with these thresholds:

def generate_compliance_scorecard(
    audit_result: dict,
    lrs_result: dict,
    district_id: str,
    tier: int,
) -> dict:
    """
    Produces a pass/fail scorecard for a single district's Tier 1 submission.
    """
    blocking_failures = []

    if audit_result["invalid_geometries_repaired"] > 0:
        blocking_failures.append(
            f"{audit_result['invalid_geometries_repaired']} invalid geometries detected and repaired — source must be corrected."
        )
    if audit_result["missing_required_columns"]:
        blocking_failures.append(
            f"Missing HPMS schema columns: {audit_result['missing_required_columns']}"
        )
    if audit_result["incomplete_record_ids"]:
        pct = len(audit_result["incomplete_record_ids"]) / audit_result["total_features"] * 100
        blocking_failures.append(
            f"{pct:.1f}% of records have null required attributes (Tier 1 requires 100% completeness)."
        )
    if lrs_result["monotonicity_errors"] or lrs_result["gaps"] or lrs_result["overlaps"]:
        blocking_failures.append("LRS calibration errors detected — see lrs_result for detail.")

    return {
        "district_id": district_id,
        "tier": tier,
        "status": "FAIL" if blocking_failures else "PASS",
        "blocking_failures": blocking_failures,
        "total_features_audited": audit_result["total_features"],
    }

For teams whose compliance framework alignment spans multiple regulatory schemas — such as state DOT requirements alongside FHWA mandates — parameterize the required columns list per schema rather than hardcoding HPMS field names.

Step 5 — Operationalize with Version Control and Immutable Audit Trails

Automated checks are only defensible when embedded in a controlled lifecycle:

• Git + DVC (Data Version Control): Store validation scripts in Git and track large .gdb or .parquet files with DVC. Every audit run must be reproducible from a specific commit hash and data version.
• Pre-promotion quality gates: In your CI/CD system, invoke the audit pipeline and parse the scorecard JSON. If status == "FAIL" for any Tier 1 layer, block the merge to main.
• Immutable audit logs: Append each scorecard to a centralized log table (PostgreSQL JSONB column or a partitioned Parquet file) with run_timestamp, script_git_sha, dataset_dvc_hash, and operator_id. This log is the compliance record for state and federal reviews.

For teams building the async promotion queue that sits between validation and production, the patterns in designing async validation queues with Celery apply directly — the audit pipeline becomes a Celery task, and the scorecard JSON is the task result.

Interpreting Results

When explain_validity from Shapely returns strings like "Self-intersection[x y]" or "Ring Self-intersection[x y]", the coordinates in brackets are the location of the defect. Pass them to your editing team with the layer’s object ID so the correction targets the exact geometry.

Gotchas & Edge Cases

1. Silent CRS inheritance in geopandas. Calling gdf.to_crs() without first asserting gdf.crs is not None will raise CRSError: Cannot transform naive geometries — catch this at ingest, not mid-pipeline.

2. is_simple returns False for valid multipart lines. A MultiLineString where two component lines share an endpoint will fail is_simple. Filter by geometry type first: apply the simple check only to LineString and LinearRing geometries, not MultiLineString.

3. County boundary seams produce false LRS gaps. Routes that cross county lines are often digitized as separate segments by each county GIS team, creating a legitimate 0.001-mile gap at the seam. Build a tolerance-exempt list keyed on known seam coordinates to suppress these false positives.

4. networkx node matching requires consistent rounding. Floating-point coordinates from different segments of the same intersection will not match unless rounded to a consistent decimal place before adding graph nodes. Four decimal degrees (approximately 11-meter resolution) is safe for road networks; use fewer decimals for low-precision datasets.

5. File geodatabases with multiple LRS event tables. A single .gdb may contain both LRSN_Calibration_Point and LRSN_Route tables. Validate each independently and cross-reference: a route present in one table but absent in the other is a calibration orphan.

When to Escalate

Move beyond this Python-based scoping approach when:

• Feature count exceeds 500,000 centerline segments. At that volume, per-row Python iteration in audit_lrs_calibration becomes a bottleneck. Port the gap and overlap checks to PostGIS using window functions (LAG/LEAD over route partitions), which execute the same logic as a single SQL pass.
• Concurrent multi-district submissions require parallel processing. The single-threaded pipeline above processes one layer at a time. When five districts submit simultaneously, wrap each audit_dot_centerlines call in a Celery task and fan out across workers — see the categorizing and prioritizing spatial errors approach for severity-based task routing.
• Contractor submissions arrive in unpredictable formats. When you regularly receive Shapefiles, GeoPackages, and CAD DXF exports from different contractors, add a schema normalization stage before validation — the geometry validity checks for vector data patterns handle mixed-format ingestion before feeding the audit pipeline.
• State auditors require a signed audit log. Append a HMAC signature over the scorecard JSON before inserting it into the audit log table, using a key stored in your secrets manager. This makes the log tamper-evident without requiring a blockchain.

Related:

• Audit Scoping for Municipal GIS Assets — cross-jurisdictional scoping patterns that complement the DOT-specific tier model above
• Defining Spatial Data Quality Policies — establishing the governance rules that validation tiers enforce
• Geometry Validity Checks for Vector Data — OGC-standard validity predicates that underpin the geometry audit step
• Categorizing and Prioritizing Spatial Errors — severity classification for routing audit failures to the right remediation queue

Back to Audit Scoping for Municipal GIS Assets