Enforcing Codelists and Domains in GIS Attributes

Coded attributes are where spatial data quietly goes wrong. A zoning code, a surface type, a status flag or a classification value looks fine in any viewer, passes every geometry check, and is simply not one of the values the schema allows — or is one that was retired three years ago, or is right in isolation and impossible in combination with the field next to it. Enforcing codelists is not hard; enforcing them in a way that survives a codelist update, an archive validation and a supplier who capitalises differently is the actual work. This guide covers that: versioned codelists, normalisation before comparison, deprecation windows, cross-field domains, and quarantine instead of defaulting — the coded-value half of Attribute Schema Mapping for Spatial Datasets.

Prerequisites

  • Python 3.10+ with pandas 2.x and geopandas 0.14+; PostgreSQL 14+ for the database enforcement section.
  • An authoritative codelist per coded field, with a status and validity dates per entry. If what you have is a list of strings in a document, converting it into that structure is the first task.
  • A record date per feature — the codelist validity window is meaningless without knowing when the record was authored.
  • A named owner for each codelist, because the review queue in Step 5 needs somebody to work it, per Data Stewardship Roles and Responsibilities.

Step-by-Step Procedure

Step 1 — Give the codelist a structure and a version

# codelists/zoning.yaml — versioned, with validity windows
version: "2026.03"
field: zoning_code
authority: "City Planning Department"
codes:
  - code: R1
    label: "Residential — low density"
    status: active
    valid_from: "2011-01-01"
  - code: R2
    label: "Residential — medium density"
    status: active
    valid_from: "2011-01-01"
  - code: C1
    label: "Commercial — neighbourhood"
    status: active
    valid_from: "2011-01-01"
  - code: M1
    label: "Light industrial"
    status: active
    valid_from: "2011-01-01"
  - code: MX
    label: "Mixed use"
    status: active
    valid_from: "2023-04-01"
  - code: I2
    label: "Heavy industrial (superseded by M2)"
    status: deprecated
    valid_from: "2011-01-01"
    valid_to: "2023-03-31"
    superseded_by: M2
# codelist/load.py
from dataclasses import dataclass
from datetime import date

import yaml


@dataclass(frozen=True)
class Codelist:
    field: str
    version: str
    entries: dict          # code -> entry dict

    def status_on(self, code: str, when: date) -> str:
        e = self.entries.get(code)
        if e is None:
            return "unknown"
        start = date.fromisoformat(e["valid_from"])
        end = date.fromisoformat(e["valid_to"]) if e.get("valid_to") else None
        if when < start:
            return "not_yet_valid"
        if end and when > end:
            return "retired"
        return e["status"]


def load(path: str) -> Codelist:
    doc = yaml.safe_load(open(path, encoding="utf-8"))
    return Codelist(field=doc["field"], version=doc["version"],
                    entries={c["code"]: c for c in doc["codes"]})

Verification: status_on("I2", date(2019, 6, 1)) must return active and status_on("I2", date(2026, 1, 1)) must return retired. That single behaviour is what makes archive validation possible without a wall of false findings.

A coded value from raw input to accepted recordChain of five stages: the raw value is normalised, looked up in the versioned codelist, checked for validity on the record date, tested against cross-field rules and either accepted or quarantined with the raw value preserved.Raw value"r-1 "Normalisetrim, fold, strip→ R1Codelist lookupon record dateactive / retired/ unknownCross-field rulesplausible combination?Accept or quarantineraw value kept
Five stages, and the record survives all of them — quarantine keeps the row and flags the value.

Step 2 — Normalise before comparing

# codelist/normalise.py
import re
import unicodedata

SEPARATORS = re.compile(r"[\s\-_./]+")


def normalise_code(raw) -> str | None:
    """Fold the differences that are formatting, not meaning."""
    if raw is None:
        return None
    text = unicodedata.normalize("NFKC", str(raw)).strip()
    if not text or text.lower() in {"n/a", "na", "none", "null", "-", "unknown"}:
        return None
    text = SEPARATORS.sub("", text)
    return text.upper()

Verification: "r-1 ", "R1" and "r_1" must all normalise to R1, while "R11" must not. Count how many records change under normalisation: a high proportion means the upstream system has no input control, which is worth reporting on its own.

Step 3 — Validate against active, deprecated and unknown

# codelist/validate.py
import pandas as pd

from codelist.load import Codelist
from codelist.normalise import normalise_code


def validate_codes(df: pd.DataFrame, codelist: Codelist,
                   value_col: str, date_col: str) -> list[dict]:
    findings = []
    for idx, row in df.iterrows():
        raw = row[value_col]
        code = normalise_code(raw)
        when = pd.to_datetime(row[date_col]).date()

        if code is None:
            findings.append({"feature_id": str(idx), "rule": "CODE_NULL_001",
                             "severity": "blocker",
                             "message": f"{codelist.field} is empty or a null placeholder "
                                        f"({raw!r})"})
            continue

        status = codelist.status_on(code, when)
        if status == "unknown":
            findings.append({"feature_id": str(idx), "rule": "CODE_UNKNOWN_001",
                             "severity": "blocker",
                             "message": f"{code!r} is not in codelist {codelist.version}"})
        elif status == "retired":
            entry = codelist.entries[code]
            findings.append({"feature_id": str(idx), "rule": "CODE_RETIRED_001",
                             "severity": "warning",
                             "message": (f"{code} retired on {entry['valid_to']}; "
                                         f"use {entry.get('superseded_by', 'a current code')}")})
        elif status == "not_yet_valid":
            findings.append({"feature_id": str(idx), "rule": "CODE_FUTURE_001",
                             "severity": "warning",
                             "message": f"{code} was not valid on {when}"})
        elif str(raw) != code:
            findings.append({"feature_id": str(idx), "rule": "CODE_FORMAT_001",
                             "severity": "informational",
                             "message": f"{raw!r} normalised to {code}"})
    return findings

Verification: four distinct rule identifiers for four distinct problems. Collapsing them into one “invalid code” rule is the most common mistake here, and it makes the findings unactionable — an unknown code needs a steward, a retired code needs a bulk update, and a formatting variance needs an upstream fix.

Step 4 — Cross-field domain rules

# codelist/cross_field.py
FORBIDDEN_COMBINATIONS = [
    # (field_a, value_a, field_b, forbidden_b_values, explanation)
    ("zoning_code", "R1", "building_class",
     {"BLAST_FURNACE", "REFINERY", "CHEMICAL_PLANT"},
     "heavy industry cannot sit in low-density residential zoning"),
    ("zoning_code", "C1", "permit_status", {None, ""},
     "commercial parcels require a permit status"),
    ("surface_type", "WATER", "building_count",
     None,      # handled numerically below
     "water surfaces cannot carry buildings"),
]


def cross_field_findings(df) -> list[dict]:
    out = []
    for idx, row in df.iterrows():
        if row.get("zoning_code") == "R1" and row.get("building_class") in {
                "BLAST_FURNACE", "REFINERY", "CHEMICAL_PLANT"}:
            out.append({"feature_id": str(idx), "rule": "CODE_COMBO_001",
                        "severity": "blocker",
                        "message": (f"zoning R1 with building_class "
                                    f"{row['building_class']} — heavy industry in "
                                    f"low-density residential zoning")})
        if row.get("surface_type") == "WATER" and (row.get("building_count") or 0) > 0:
            out.append({"feature_id": str(idx), "rule": "CODE_COMBO_002",
                        "severity": "warning",
                        "message": (f"surface_type WATER with "
                                    f"{row['building_count']} building(s)")})
    return out

Verification: domain experts should write these, not engineers. The most valuable cross-field rules encode knowledge that is obvious to a planner and invisible in a schema, and eliciting them is a conversation rather than a code review.

Step 5 — Enforce at the database, quarantine the unknowns

-- Codelist as a table, so the constraint and the validation share one source.
CREATE TABLE ref.zoning_codes (
  code           text PRIMARY KEY,
  label          text NOT NULL,
  status         text NOT NULL CHECK (status IN ('active', 'deprecated')),
  valid_from     date NOT NULL,
  valid_to       date,
  superseded_by  text REFERENCES ref.zoning_codes(code)
);

-- New records must use a currently active code; historical records are untouched.
ALTER TABLE parcels ADD CONSTRAINT parcels_zoning_fk
  FOREIGN KEY (zoning_code) REFERENCES ref.zoning_codes(code) NOT VALID;

CREATE OR REPLACE FUNCTION check_code_currency() RETURNS trigger AS $$
BEGIN
  IF NOT EXISTS (
    SELECT 1 FROM ref.zoning_codes c
    WHERE c.code = NEW.zoning_code
      AND c.status = 'active'
      AND NEW.record_date BETWEEN c.valid_from AND COALESCE(c.valid_to, 'infinity'::date)
  ) THEN
    INSERT INTO qa.code_quarantine (table_name, feature_id, field, raw_value, reason, seen_at)
    VALUES ('parcels', NEW.parcel_id, 'zoning_code', NEW.zoning_code,
            'code not active on record_date', now());
  END IF;
  RETURN NEW;      -- the record is kept; the finding is recorded
END;
$$ LANGUAGE plpgsql;

CREATE TRIGGER parcels_code_currency
  BEFORE INSERT OR UPDATE OF zoning_code ON parcels
  FOR EACH ROW EXECUTE FUNCTION check_code_currency();

Verification: insert a record with a retired code and confirm it lands in both parcels and qa.code_quarantine. The record is kept deliberately — rejecting it loses data that is probably mostly correct, while the quarantine row ensures somebody sees it.

Interpreting Results

Finding pattern Likely cause Response
Many CODE_FORMAT_001 from one supplier No input control upstream Normalise on ingest; raise with the supplier
A single unknown code on many records A new code the authority introduced Add to the codelist, bump the version
Many distinct unknown codes Free text in a coded field The field is not actually coded — fix the schema
CODE_RETIRED_001 spike after a codelist update Expected — the update retired a code Bulk-update using superseded_by
CODE_FUTURE_001 Record dates wrong, or codes used before approval Check the date field before the code field
Cross-field failures clustered spatially A bad batch, or a mis-mapped source layer Investigate the batch, not the individual records

The most informative signal is the ratio of distinct unknown codes to affected records. One unknown code across 4,000 records is a codelist that needs updating; 400 unknown codes across 4,000 records means the field is being used as free text and no codelist will fix it.

Four code outcomes, four different responsesGrid of four code outcomes against three response properties — severity, whether the record loads, and who acts — for unknown codes, retired codes, not-yet-valid codes and formatting variance.severityrecord loads?who actsunknown codeblockeryes, flaggedstewardretired codewarningyesbulk updatenot yet validwarningyescheck datesformatting varianceinfoyesupstream fixFour outcomes that a single "invalid code" rule would collapse into one.Collapsing them is the most common mistake here: it makes the finding unactionable because the response differs in every row.
One rule per outcome, because each one goes to a different person with a different fix.

Gotchas & Edge Cases

Normalising too aggressively merges distinct codes. Stripping separators turns R-1 into R1, which is usually right, and would also turn R-11 into R11 — fine — but a scheme where R1 and R-1 genuinely differ would break. Check the codelist for collisions after normalisation before deploying the rule.

A code through its lifecycle, and what validation does at each pointTimeline of a codelist entry: introduced and active, superseded and deprecated, retired with a validity end date, and finally historical — with the validation behaviour at each stage.Introducedactive — accepted on new records2011Supersededstill active, successor published2023-04Retiredvalid_to set — warning on new records2023-04Historicalstill correct on pre-2023 records2026A retired code is not an invalid code. Without validity dates, validating an archive produces thousands of false findings.
Validity dates are what let one rule serve both current data and the archive.

Null and “unknown” are different. A missing value and an explicitly recorded “unknown” carry different information, and collapsing them loses the distinction between “nobody filled this in” and “somebody looked and could not tell”. Keep a real code for genuinely unknown values if the domain needs one.

Deprecation without a supersession mapping creates orphans. A retired code with no superseded_by leaves every affected record with no correct value. Insist on the mapping when the codelist is updated.

Foreign keys break archive loads. A strict foreign key onto the current codelist rejects historical records using retired codes. The NOT VALID constraint plus a trigger, as in Step 5, keeps new data clean without blocking archive ingestion.

Codelist versions must travel with the findings. A record that was valid under version 2025.11 and invalid under 2026.03 is a rule change, not a data change, and the run report needs the version to explain it — the same discipline applied to rule-set versions in Observability and Lineage for Validation.

Multilingual labels are not codes. Validating against the label rather than the code breaks the moment somebody translates the interface. Always compare codes; treat labels as display only.

When to Escalate

  • A code the authority uses that is not in your list is a codelist maintenance failure, not a data defect. Escalate to the codelist owner with the frequency and the affected layers.
  • A field with hundreds of distinct unknown values needs a schema decision — either it is free text and should be typed as such, or it needs a controlled list that does not yet exist.
  • Cross-field rules that domain experts disagree about should be resolved before implementation. A rule shipped without agreement produces findings nobody accepts, and it will be disabled rather than debated.
  • Retirements with no supersession should be sent back to the authority. Without a mapping, the correct value for thousands of records is genuinely unknown, and guessing it in a pipeline is worse than reporting it.

Frequently Asked Questions

Should an unknown code fail the load or be defaulted?

Neither — quarantine it. Failing the load blocks good data for one bad value; defaulting invents information that later looks authoritative. Load the record with the raw value preserved and the code flagged as unresolved, and put it in a review queue where a steward can either add the code to the list or correct the record.

How should retired codes be handled?

As valid for historical records and invalid for new ones. A codelist entry needs a status and a validity window, so a value retired in 2023 is correct on a 2019 record and a finding on a 2026 one. Treating retired codes as simply invalid produces thousands of false findings the first time you validate an archive.

Where should the codelist live?

In version control, next to the rules, and loaded from there by every consumer. A codelist embedded in a validation script diverges from the one in the database, which diverges from the one in the documentation. One versioned file, referenced by its version in every run, is the only arrangement that stays consistent.

What is a cross-field domain rule?

A constraint on a combination of values that are each individually legal. A zoning code of "residential" with a building class of "blast furnace" passes both single-field checks and is obviously wrong. These rules catch the errors that single-column validation cannot, and they are usually the ones domain experts care about most.

Related

Back to Attribute Schema Mapping for Spatial Datasets