Validating Spatial File Formats

The container a dataset arrives in decides what its schema can even express. A field name that is legal in one format is silently truncated in another; a NULL that is preserved in one becomes a zero in the next; a strongly typed timestamp survives one round-trip and degrades to a string on the following export. For GIS analysts, QA engineers, data stewards, and platform teams, format-level validation is the first gate in any quality pipeline — it catches structural loss that no downstream geometry or attribute rule can recover, because the information is already gone by the time a rule engine sees it. This guide is part of Spatial Validation Tooling and Framework Selection, and it focuses on how Shapefile, GeoPackage, GeoParquet, and FlatGeobuf differ in what they can enforce, and how to inspect and assert each with the Geospatial Data Abstraction Library (GDAL) and its OGR vector component, and with Fiona.

Prerequisites

Confirm every item before you rely on format-level checks. Driver behaviour varies between GDAL builds, and a mismatched version will report constraints your production host does not actually enforce.

1. GDAL 3.8+ with the OGR Python bindings. Verify with python -c "from osgeo import gdal; print(gdal.__version__)". GDAL 3.8 stabilised the GeoParquet and Arrow drivers and improved GeoPackage constraint handling. Older builds (3.4 and earlier) lack the Parquet driver entirely.

2. Fiona 1.9+ and pyogrio 0.7+. Fiona exposes OGR schemas as plain Python dictionaries; pyogrio provides a faster vectorised path used by GeoPandas 0.14+. Pin both: fiona==1.9.*, pyogrio==0.7.*.

3. PyArrow 15+ for GeoParquet. GeoParquet validation reads Parquet metadata and Arrow schema through PyArrow rather than OGR when you need column-level and row-group detail. Pin pyarrow==15.* or newer.

4. A version-controlled schema contract. Format validation is only meaningful against an expected schema: the authoritative field names, types, nullability, and geometry type. If your attribute contracts are not yet formalised, attribute schema mapping for spatial datasets covers how to define one before this stage consumes it.

5. A SQLite client for GeoPackage internals. GeoPackage is SQLite, so sqlite3 (bundled with Python) lets you inspect gpkg_* metadata tables, triggers, and CHECK constraints that OGR abstracts away.

Conceptual Foundation

A spatial file format is not a neutral container. It is a schema-enforcement system with a specific, fixed expressive ceiling, and validation begins by knowing that ceiling for each format you accept.

Shapefile is really a bundle — .shp geometry, .shx index, .dbf attributes, .prj projection — glued together by shared basename. Its attribute store is the dBASE .dbf table from the 1980s, and that lineage dictates hard limits: field names are capped at ten characters and are silently truncated on write; the type system is limited to string, a single numeric type with width and decimal precision, date, and a weak boolean stored as a single character; there is no native NULL for numeric columns, so a missing number becomes zero or an empty-padded string depending on the driver; and each .shp or .dbf component cannot exceed 2 gigabytes. The .prj file uses a well-known-text CRS description that many writers omit entirely, leaving the coordinate reference system undefined. Every one of these limits is a place where a valid source schema can degrade without an error being raised.

GeoPackage (.gpkg) is an Open Geospatial Consortium standard built on SQLite. Because the payload is a real relational database, it inherits SQLite’s constraint machinery: NOT NULL, UNIQUE, CHECK, DEFAULT, and foreign keys, plus triggers that can enforce arbitrary rules at write time. It stores multiple vector and raster layers in one file, records each layer’s geometry type and CRS in the gpkg_geometry_columns and gpkg_spatial_ref_sys tables, and keeps an R-tree spatial index. For validation this is a step change: constraints live in the file itself, so an out-of-range value or a missing mandatory attribute can be rejected by the storage engine rather than caught later in Python.

GeoParquet is a columnar, analytics-oriented format built on Apache Parquet. Every column carries an explicit Apache Arrow type, so integers, floats, booleans, timestamps, and strings are all distinctly and unambiguously typed — there is no coercion of a boolean into a character. Geometry is stored in a dedicated column and described by a geo metadata key in the Parquet file footer, which records the geometry column name, encoding (well-known binary, or the newer GeoArrow native encoding), and per-column CRS as PROJJSON. Parquet’s row-group statistics (min, max, null count per column) let you validate ranges and nullability without scanning every value.

FlatGeobuf (.fgb) is a single-file, streamable binary format with a FlatBuffers header that declares the schema and CRS up front and an optional packed Hilbert R-tree spatial index. It is designed for fast sequential and bounding-box reads over cloud storage. Its schema model is typed like GeoParquet’s but its role in a QC pipeline is usually as a transport and streaming format rather than a constraint-bearing store of record.

The practical rule that follows from these differences: keep weakly-typed formats at the edges and convert to a constraint-bearing format as early as possible. Accept a Shapefile, validate what it can express, then migrate it to GeoPackage or GeoParquet so every downstream stage inherits enforceable typing and nullability. The two detailed walk-throughs in this section handle the two migrations that matter most — Shapefile vs GeoPackage schema enforcement for the relational path, and validating GeoParquet schemas with PyArrow for the columnar path.

Step-by-Step Implementation

Step 1: Read the Declared Schema with OGR

Before reading a single feature, extract what the file claims its schema is. OGR exposes this from the layer definition without scanning the data.

from osgeo import ogr

def read_ogr_schema(path: str, layer: int | str = 0) -> dict:
    """
    Return the declared schema of a vector layer:
    field names, OGR type names, widths, and geometry type.
    Works uniformly across Shapefile, GeoPackage, GeoParquet, FlatGeobuf.
    """
    ds = ogr.Open(path)
    if ds is None:
        raise IOError(f"OGR could not open {path}")
    lyr = ds.GetLayer(layer)
    defn = lyr.GetLayerDefn()

    fields = []
    for i in range(defn.GetFieldCount()):
        fd = defn.GetFieldDefn(i)
        fields.append({
            "name": fd.GetName(),
            "type": fd.GetTypeName(),          # e.g. "Integer64", "Real", "String"
            "width": fd.GetWidth(),
            "precision": fd.GetPrecision(),
            "nullable": bool(fd.IsNullable()),
        })

    return {
        "driver": ds.GetDriver().GetName(),
        "geometry_type": ogr.GeometryTypeToName(defn.GetGeomType()),
        "feature_count": lyr.GetFeatureCount(),
        "crs": lyr.GetSpatialRef().GetAuthorityCode(None) if lyr.GetSpatialRef() else None,
        "fields": fields,
    }

Verification: run against a known GeoPackage and confirm nullable is False for any column you created with NOT NULL. If every field reports nullable=True on a Shapefile, that is expected — the format has no nullability concept to report.

schema = read_ogr_schema("data/parcels.gpkg", layer="parcels")
print(schema["driver"], schema["geometry_type"], schema["crs"])
for f in schema["fields"]:
    print(f["name"], f["type"], f["nullable"])

Step 2: Assert the Schema Against a Contract

A declared schema is only useful when compared to an expected one. Fail fast on drift rather than letting a renamed or retyped column surface as a downstream KeyError.

def assert_schema(actual: dict, expected: dict) -> list[str]:
    """
    Compare an OGR schema (from read_ogr_schema) against a contract.
    Returns a list of human-readable violations; empty list means conformant.
    'expected' shape:
      {"geometry_type": "Polygon",
       "crs": "4326",
       "fields": {"parcel_id": "String", "assessed_value": "Real"}}
    """
    problems: list[str] = []

    if expected.get("geometry_type") and \
       expected["geometry_type"].lower() not in actual["geometry_type"].lower():
        problems.append(
            f"geometry type: expected {expected['geometry_type']}, got {actual['geometry_type']}"
        )

    if expected.get("crs") and str(actual["crs"]) != str(expected["crs"]):
        problems.append(f"CRS: expected EPSG:{expected['crs']}, got EPSG:{actual['crs']}")

    actual_fields = {f["name"]: f["type"] for f in actual["fields"]}
    for name, exp_type in expected["fields"].items():
        if name not in actual_fields:
            problems.append(f"missing field: {name}")
        elif actual_fields[name] != exp_type:
            problems.append(
                f"type drift on {name}: expected {exp_type}, got {actual_fields[name]}"
            )

    extra = set(actual_fields) - set(expected["fields"])
    for name in sorted(extra):
        problems.append(f"unexpected field: {name}")

    return problems

Verification: feed the function a contract that deliberately misnames one field and confirm it reports exactly one missing field violation plus one unexpected field for the real column name.

Step 3: Check Format-Specific Constraints

The generic schema check misses limits that only apply to one format. Add a format-aware pass keyed on the driver.

def check_shapefile_limits(schema: dict) -> list[str]:
    """Flag Shapefile-specific schema risks the generic check cannot see."""
    warnings: list[str] = []
    for f in schema["fields"]:
        if len(f["name"]) == 10:
            warnings.append(f"field '{f['name']}' is exactly 10 chars — possible truncation")
        if f["name"][-2:-1] == "_" and f["name"][-1].isdigit():
            warnings.append(f"field '{f['name']}' looks like a truncation-collision suffix")
    if schema["crs"] is None:
        warnings.append("no CRS defined (.prj missing) — coordinates are ambiguous")
    return warnings

Verification: run this against a Shapefile you produced by exporting long-named columns; the function should surface every ten-character name and any _1/_2 collision suffix that OGR generated during truncation.

Step 4: Inspect GeoPackage Constraints Directly

OGR reports nullability but not CHECK constraints or triggers. Read those from the SQLite catalog to know what the file actually enforces on write.

import sqlite3

def geopackage_constraints(path: str, table: str) -> dict:
    """Extract CHECK constraints and triggers enforced by a GeoPackage table."""
    con = sqlite3.connect(path)
    cur = con.cursor()

    cur.execute("SELECT sql FROM sqlite_master WHERE type='table' AND name=?", (table,))
    row = cur.fetchone()
    table_sql = row[0] if row else ""

    cur.execute(
        "SELECT name, sql FROM sqlite_master WHERE type='trigger' AND tbl_name=?",
        (table,),
    )
    triggers = {name: sql for name, sql in cur.fetchall()}

    con.close()
    return {
        "has_check": "CHECK" in table_sql.upper(),
        "table_ddl": table_sql,
        "triggers": triggers,
    }

Verification: create a GeoPackage column with CHECK (assessed_value >= 0), then confirm has_check is True and the DDL string contains the predicate. A file with no constraints returns has_check=False, which is itself an actionable finding.

Step 5: Convert to a Constraint-Bearing Canonical Format

Once a source passes format-level checks, migrate it so downstream stages inherit enforceable structure. The ogr2ogr command line remaps types and names; the SQL layer adds constraints the source could not express.

import subprocess

def shapefile_to_geopackage(src_shp: str, dst_gpkg: str, layer: str) -> None:
    """
    Convert a Shapefile to a GeoPackage layer, normalising CRS to EPSG:4326
    and promoting to a typed, constraint-ready relational store.
    """
    cmd = [
        "ogr2ogr",
        "-f", "GPKG",
        "-t_srs", "EPSG:4326",
        "-nln", layer,
        "-nlt", "PROMOTE_TO_MULTI",   # avoid mixed Polygon/MultiPolygon failures
        dst_gpkg,
        src_shp,
    ]
    subprocess.run(cmd, check=True)

Verification: re-run read_ogr_schema on the output GeoPackage and confirm the driver is now GPKG, the CRS resolves to 4326, and full-length field names (renamed during conversion) are present. The relational path is covered end to end, including CHECK constraints and triggers, in Shapefile vs GeoPackage schema enforcement.

Common Failure Modes & Fixes

Performance & Scale Considerations

Schema checks are cheap; feature checks are not. Reading a layer definition touches only the header or catalog, so Steps 1–4 run in milliseconds regardless of feature count. Reserve full scans for the checks that genuinely require per-feature inspection, and push those down into the format wherever possible.

Exploit each format’s index and statistics. GeoPackage carries an R-tree that OGR uses for bounding-box filters, so a spatial extent check need not read every geometry. GeoParquet’s row-group statistics expose per-column min, max, and null counts in the footer — a range or nullability assertion can be answered from metadata alone, without decoding a single value. This is one reason columnar formats scale so well for validation, and it is developed further in validating GeoParquet schemas with PyArrow.

Prefer pyogrio over the classic Fiona path for bulk reads. When a check does require materialising features, pyogrio.read_dataframe is typically several times faster than record-by-record Fiona iteration because it reads through OGR’s Arrow interface in a single vectorised call. Use Fiona when you need lazy, memory-bounded streaming; use pyogrio when the layer fits in memory and throughput matters.

Choose the format to match the read pattern. FlatGeobuf’s packed R-tree and streamable layout make it ideal when validation only ever touches a bounding-box subset over cloud storage. GeoParquet’s column pruning wins when checks touch a handful of attribute columns across billions of rows. Match the container to how the validator reads, not only to how the data is produced.

For validation workloads that exceed a single host — millions of features partitioned across workers — format choice interacts directly with the batch engine, covered in batch processing large spatial datasets.

Integration with the Validation Pipeline

Format-level validation is the ingestion boundary of the wider quality pipeline. It runs before any geometry or attribute rule engine, and its contract with downstream stages is explicit: emit only data in a canonical, constraint-bearing format with a known CRS, a stable typed schema, and no format-induced loss.

Upstream contract. Accept any supported source format, but read and assert its declared schema (Steps 1–2) and its format-specific limits (Step 3) before a single feature is trusted. A source that fails schema-contract assertion is routed to a rejection log with the specific violations, never silently coerced.

Attribute-constraint handoff. Once a dataset is in GeoPackage or GeoParquet, attribute-level constraints — value ranges, enumerations, referential integrity — become enforceable in the store itself. The mapping from a logical attribute contract to concrete constraints is covered in mapping attribute constraints to GeoJSON schemas, and the same contract drives both the GeoJSON validation path and the CHECK constraints you add in GeoPackage.

Observability. Log the driver, feature count, CRS, and every schema violation per file as a structured record. A sudden appearance of ten-character field names, or a spike in crs=None, is an early signal that an upstream export process changed or that a new data supplier is emitting Shapefiles where GeoPackages were expected — cheaper to catch here than three stages downstream.

Frequently Asked Questions

Related

• Shapefile vs GeoPackage Schema Enforcement — the relational migration path, with CHECK constraints, triggers, and an ogr2ogr plus SQL workflow
• Validating GeoParquet Schemas with PyArrow — reading geo metadata, Arrow column types, geometry encoding, and row-group checks
• Attribute Schema Mapping for Spatial Datasets — how to define the attribute contract that format validation asserts against
• Mapping Attribute Constraints to GeoJSON Schemas — translating logical constraints into enforceable schema rules
• Batch Processing Large Spatial Datasets — how format choice interacts with distributed validation at scale

Back to Spatial Validation Tooling and Framework Selection