Validating GeoParquet Schemas with PyArrow

You are ingesting GeoParquet — perhaps from a data lake, a partner feed, or the output of a Spark job — and you need to confirm it is structurally sound before it reaches a geometry rule engine: the geo metadata present and well-formed, the coordinate reference system declared, the geometry encoding known, and every attribute column carrying the type your contract expects. PyArrow is the correct tool for this because it reads the Parquet footer and Arrow schema without decoding a single geometry, so structural validation is fast even on files with millions of rows. This page walks through that validation end to end, and it sits under validating spatial file formats, which frames why columnar formats enforce so much more than a Shapefile can.

Prerequisites

• PyArrow 15+. GeoParquet validation reads the Parquet footer and Arrow schema through PyArrow. Confirm with python -c "import pyarrow; print(pyarrow.__version__)". Row-group statistics access is stable from version 12, but 15+ tracks the current GeoParquet 1.1 metadata shape.
• GeoPandas 0.14+ and Shapely 2.0+ for the follow-on geometry checks referenced at the end — not for the structural validation itself, which is pure PyArrow.
• A typed schema contract. GeoParquet’s strength is that every column has an explicit Apache Arrow type; validation compares the file’s types against your expected types. Define that contract via attribute schema mapping for spatial datasets if you have not already.
• Gotcha to flag now: the geo metadata is stored as a raw bytes value under a bytes key in the Parquet key-value metadata. You must decode both the key and the value from UTF-8 and parse the value as JSON — indexing the metadata dictionary with a plain str key returns None and makes a valid file look non-conformant.

Step-by-Step Procedure

Step 1 — Read footer metadata without decoding data

import json
import pyarrow.parquet as pq

def read_geo_metadata(path: str) -> dict:
    """
    Return the parsed GeoParquet 'geo' metadata block from the Parquet footer.
    Reads only the footer — no geometry or attribute values are decoded.
    """
    pf = pq.ParquetFile(path)
    kv = pf.schema_arrow.metadata or {}     # keys and values are bytes

    geo_bytes = kv.get(b"geo")
    if geo_bytes is None:
        raise ValueError(f"{path} has no 'geo' metadata key — not a valid GeoParquet file")

    return json.loads(geo_bytes.decode("utf-8"))

geo = read_geo_metadata("data/parcels.parquet")
print("version:", geo.get("version"))
print("primary_column:", geo.get("primary_column"))
print("columns:", list(geo.get("columns", {})))

Verification: the call returns a dict with version, primary_column, and a non-empty columns map. A ValueError here is itself a valid result — it means the file is plain Parquet, not GeoParquet, and must be rejected or rewritten.

Step 2 — Validate the geo metadata block

def validate_geo_block(geo: dict) -> list[str]:
    """Assert the geo metadata is structurally complete. Returns violations."""
    problems: list[str] = []

    if "version" not in geo:
        problems.append("geo.version missing")
    primary = geo.get("primary_column")
    if not primary:
        problems.append("geo.primary_column missing")

    columns = geo.get("columns", {})
    if primary and primary not in columns:
        problems.append(f"primary_column '{primary}' not described in geo.columns")

    for name, spec in columns.items():
        if "encoding" not in spec:
            problems.append(f"column '{name}': encoding missing")
        if "crs" not in spec:
            # crs may be null (meaning CRS84) but the key should be present
            problems.append(f"column '{name}': crs key absent (expected null for CRS84)")
        geom_types = spec.get("geometry_types")
        if geom_types is not None and not isinstance(geom_types, list):
            problems.append(f"column '{name}': geometry_types must be a list")

    return problems

violations = validate_geo_block(geo)
print("geo block violations:", violations)

Verification: violations is empty for a conformant file. Deliberately delete primary_column from a copy and confirm the function reports exactly that one violation.

Step 3 — Assert Arrow column types against a contract

import pyarrow as pa
import pyarrow.parquet as pq

def validate_column_types(path: str, expected: dict[str, pa.DataType]) -> list[str]:
    """
    Compare the Arrow schema of a GeoParquet file against an expected
    {column_name: pyarrow type} contract. Returns type-drift violations.
    """
    schema = pq.ParquetFile(path).schema_arrow
    problems: list[str] = []

    for name, exp_type in expected.items():
        if name not in schema.names:
            problems.append(f"missing column: {name}")
            continue
        actual = schema.field(name).type
        if not actual.equals(exp_type):
            problems.append(f"type drift on {name}: expected {exp_type}, got {actual}")

    return problems

EXPECTED = {
    "parcel_identifier": pa.string(),
    "assessed_value":    pa.float64(),
    "is_exempt":         pa.bool_(),
    "zoning_code":       pa.string(),
}

print(validate_column_types("data/parcels.parquet", EXPECTED))

Verification: the returned list is empty when types match. Unlike a Shapefile, GeoParquet distinguishes pa.bool_() from pa.string(), so is_exempt is a real boolean and any drift to string is reported explicitly.

Step 4 — Confirm geometry encoding matches the declared metadata

import pyarrow as pa
import pyarrow.parquet as pq

def validate_encoding(path: str, geo: dict) -> list[str]:
    """
    Cross-check the geometry column's declared encoding (WKB or geoarrow)
    against its actual Arrow column type.
    """
    problems: list[str] = []
    schema = pq.ParquetFile(path).schema_arrow
    primary = geo["primary_column"]
    declared = geo["columns"][primary].get("encoding", "").lower()

    field = schema.field(primary)
    actual_type = field.type

    if declared == "wkb":
        if not (pa.types.is_binary(actual_type) or pa.types.is_large_binary(actual_type)):
            problems.append(
                f"encoding=WKB but column '{primary}' is {actual_type}, expected binary"
            )
    elif declared in ("point", "linestring", "polygon", "multipolygon", "geoarrow"):
        # GeoArrow native: nested list/struct of coordinates
        if not (pa.types.is_list(actual_type) or pa.types.is_struct(actual_type)
                or pa.types.is_fixed_size_list(actual_type)):
            problems.append(
                f"encoding={declared} (GeoArrow) but column '{primary}' is {actual_type}"
            )
    else:
        problems.append(f"unknown encoding '{declared}' for column '{primary}'")

    return problems

print(validate_encoding("data/parcels.parquet", geo))

Verification: an empty list confirms the declared encoding matches the physical column type. A WKB declaration over a struct column, or a GeoArrow declaration over a binary column, is a malformed-writer signature and is reported here.

Step 5 — Validate ranges and nullability from row-group statistics

import pyarrow.parquet as pq

def check_row_group_stats(path: str, column: str,
                          min_allowed=None, max_allowed=None,
                          allow_null=True) -> list[str]:
    """
    Validate a numeric column's range and nullability using per-row-group
    statistics from the footer — no full scan.
    """
    problems: list[str] = []
    md = pq.ParquetFile(path).metadata
    col_index = md.schema.names.index(column)

    for rg in range(md.num_row_groups):
        stats = md.row_group(rg).column(col_index).statistics
        if stats is None:
            problems.append(f"row group {rg}: no statistics for '{column}'")
            continue
        if min_allowed is not None and stats.min is not None and stats.min < min_allowed:
            problems.append(f"row group {rg}: min {stats.min} < allowed {min_allowed}")
        if max_allowed is not None and stats.max is not None and stats.max > max_allowed:
            problems.append(f"row group {rg}: max {stats.max} > allowed {max_allowed}")
        if not allow_null and stats.null_count and stats.null_count > 0:
            problems.append(f"row group {rg}: {stats.null_count} nulls in non-null column")

    return problems

print(check_row_group_stats("data/parcels.parquet", "assessed_value",
                            min_allowed=0, allow_null=True))

Verification: the function inspects every row group and returns an empty list when all statistics fall within bounds. Because it reads only footer statistics, it completes in milliseconds even on a multi-million-row file — confirm by timing it against a large file and observing that runtime is independent of row count.

Interpreting Results

The five checks form a strict-to-lenient gate. Steps 1 and 2 are pass/fail on structure: no geo key, or a malformed geo block, means the file is not usable as GeoParquet and there is nothing to salvage without a rewrite. Step 3 catches type drift — the columnar equivalent of the Shapefile coercion problems described in Shapefile vs GeoPackage schema enforcement — but here every type is explicit, so drift is unambiguous rather than inferred. Step 4 verifies internal consistency between what the metadata claims and what the column physically holds. Step 5 answers attribute-quality questions from statistics alone.

A file that clears all five is structurally trustworthy: correct geometry column, known CRS, known encoding, contracted types, and in-range values. What these checks do not prove is geometry validity — a WKB blob can be well-typed and in-bounds yet encode a self-intersecting polygon. That final layer belongs to a geometry rule engine, and the clean hand-off is: validate structure with PyArrow, then decode with GeoPandas for predicate checks.

Gotchas & Edge Cases

The geo metadata key and value are both bytes. Parquet key-value metadata is a dict of bytes -> bytes. Look up b"geo", not "geo", and decode the value before json.loads. A str lookup silently returns None and misclassifies a valid file as non-conformant.

A null CRS means CRS84, not missing CRS. In GeoParquet, a crs value of null explicitly means longitude/latitude on WGS84 (OGC:CRS84). Do not treat crs: null as an error; treat an absent crs key as the error. The check in Step 2 distinguishes the two deliberately.

Row-group statistics can be absent. A writer may omit statistics for a column, in which case statistics is None and you cannot range-check that row group from the footer. Treat missing statistics as a finding that forces a full scan for that column, not as a silent pass.

GeoArrow encoding names are geometry-type-specific. GeoParquet 1.1 GeoArrow encodings appear as "point", "linestring", "polygon", "multipolygon", and so on, rather than a single literal "geoarrow". Match the set of native encodings, as Step 4 does, rather than looking for one keyword, or valid native-encoded files will be rejected.

Dictionary-encoded columns report a different Arrow type. A string column written with Parquet dictionary encoding surfaces as dictionary<values=string> rather than string. Normalise with pa.types.is_dictionary(t) and inspect t.value_type before comparing against a pa.string() contract, or add the dictionary form to the expected types.

When to Escalate

PyArrow structural validation is the right first gate, but escalate when:

• You need geometry-level validity, not just structure. Once the schema passes, decode with GeoPandas read_parquet and run Shapely predicate checks for validity, self-intersection, and bounds. Structural conformance says nothing about whether the polygons are topologically sound.
• The dataset is partitioned across many files or exceeds a single host. A directory of GeoParquet parts validated per-file needs coordinated, distributed checking; push validation into the batch layer described in batch processing large spatial datasets, which handles partitioned GeoParquet natively and parallelises both the metadata and geometry passes.
• You need enforced constraints rather than after-the-fact checks. GeoParquet metadata is descriptive, not enforcing — nothing stops a future writer from emitting out-of-range values. When you need writes to be rejected at ingestion, land the data in a constraint-bearing store such as a GeoPackage, as covered in Shapefile vs GeoPackage schema enforcement, or a PostGIS table.

Related:

• Validating Spatial File Formats — the parent overview of format-level schema enforcement across four formats
• Shapefile vs GeoPackage Schema Enforcement — the enforced-constraint alternative when descriptive metadata is not enough
• Batch Processing Large Spatial Datasets — validating partitioned GeoParquet at scale across distributed workers

Back to Validating Spatial File Formats