Validating Address Normalization Before Geocoding

When a geocoding batch comes back with an 84% match rate, there are two possible explanations and no way to tell them apart: the reference data does not contain those addresses, or the strings sent were not addresses in a form anything could match. Normalisation separates the two. It parses each input into components, standardises the parts that have controlled forms, applies structural rules that catch nonsense before it costs an API call, and gates the records that cannot be parsed at all. What comes out the other side is a match rate that means something — the prerequisite for every measurement described in Geocoding and Address Data Validation.

Prerequisites

  • Python 3.10+ with a locale-appropriate parser: usaddress for US-style addresses, pyap for several English-speaking locales, or postal (libpostal bindings) for international data. Libpostal is much heavier to install and much better at messy multi-country input.
  • A controlled list of street types and directionals for your locale. The examples use a short illustrative map; a production system uses the official postal authority list.
  • A postal-code format rule per country you handle. These are stable, well documented and cheap to enforce.
  • A decision about what is not an address. Care-of lines, delivery instructions, building nicknames and “as above” placeholders are common in operational data and must be stripped or rejected explicitly rather than sent to a geocoder.

Step-by-Step Procedure

Step 1 — Clean the raw string without losing it

# normalise/step1_clean.py
import re

NOISE_PATTERNS = [
    r"\bc/o\b.*", r"\bcare of\b.*", r"\battn?:?\b.*",
    r"\bas above\b", r"\bsame as .*", r"\bn/?a\b",
    r"\bleave (with|at)\b.*", r"\bring bell\b.*",
]
NOISE_RE = re.compile("|".join(NOISE_PATTERNS), re.I)


def clean(raw: str) -> dict:
    """Strip non-address noise and normalise whitespace, keeping the original."""
    if raw is None:
        return {"raw": None, "cleaned": None, "rejected": "null input"}

    text = raw.replace("\n", ", ").replace("\t", " ")
    removed = bool(NOISE_RE.search(text))
    text = NOISE_RE.sub(" ", text)
    text = re.sub(r"[;|]+", ",", text)
    text = re.sub(r"\s*,\s*", ", ", text)
    text = re.sub(r"\s+", " ", text).strip(" ,")

    if len(text) < 5 or not re.search(r"[A-Za-z]", text):
        return {"raw": raw, "cleaned": text, "rejected": "too short to be an address"}
    return {"raw": raw, "cleaned": text, "noise_removed": removed, "rejected": None}

Verification: run over a thousand raw records and inspect everything with rejected set. The rejects should be genuinely unusable — empty strings, “unknown”, pure phone numbers. Anything real appearing there means the cleaning rules are too aggressive.

Where a batch of addresses ends up before it reaches the geocoderFunnel from 100,000 raw address records: 98,400 survive cleaning, 94,100 parse into components, 91,600 have all required components, and 88,900 pass every structural rule and are classed as clean.Raw records100,000 — as receivedCleaned98,400 — noise and placeholders removedParsed94,100 — tagged into componentsComplete91,600 — number and street presentClean88,900 — all structural rules passThe match rate should be quoted over the clean bucket. Quoted over the raw count it blames the geocoder for the input pipeline.
Five buckets, and only the last one is a fair denominator for a match rate.

Step 2 — Parse into components

# normalise/step2_parse.py
import usaddress

REQUIRED = {"AddressNumber", "StreetName"}


def parse(cleaned: str) -> dict:
    try:
        tagged, kind = usaddress.tag(cleaned)
    except usaddress.RepeatedLabelError as exc:
        return {"parsed": False, "reason": f"ambiguous components: {exc.original_string[:60]}"}

    components = dict(tagged)
    missing = REQUIRED - components.keys()
    return {
        "parsed": True,
        "kind": kind,                       # 'Street Address', 'PO Box', 'Intersection'
        "components": components,
        "missing_required": sorted(missing),
        "complete": not missing,
    }

Verification: group the outputs by kind. A batch that is supposed to be street addresses but contains 8% PO boxes has a data-collection problem: a PO box has no building to geocode, and forcing one through produces a locality centroid every time.

Step 3 — Expand abbreviations against a controlled list

# normalise/step3_expand.py
STREET_TYPES = {
    "st": "Street", "str": "Street", "rd": "Road", "ave": "Avenue", "av": "Avenue",
    "dr": "Drive", "cl": "Close", "ln": "Lane", "ct": "Court", "cres": "Crescent",
    "blvd": "Boulevard", "pl": "Place", "sq": "Square", "ter": "Terrace",
}
DIRECTIONALS = {"n": "North", "s": "South", "e": "East", "w": "West",
                "ne": "Northeast", "nw": "Northwest", "se": "Southeast", "sw": "Southwest"}


def expand(components: dict) -> dict:
    out = dict(components)
    for field, table in (("StreetNamePostType", STREET_TYPES),
                         ("StreetNamePreType", STREET_TYPES),
                         ("StreetNamePreDirectional", DIRECTIONALS),
                         ("StreetNamePostDirectional", DIRECTIONALS)):
        value = out.get(field)
        if value:
            out[field] = table.get(value.lower().strip("."), value.title())
    if "StreetName" in out:
        out["StreetName"] = out["StreetName"].title()
    return out


def canonical_line(components: dict) -> str:
    order = ("AddressNumber", "StreetNamePreDirectional", "StreetNamePreType", "StreetName",
             "StreetNamePostType", "StreetNamePostDirectional", "OccupancyType",
             "OccupancyIdentifier", "PlaceName", "StateName", "ZipCode")
    return " ".join(components[k] for k in order if components.get(k))

Verification: count distinct canonical street names before and after expansion. A well-formed expansion collapses “MAIN ST”, “Main Street” and “main st.” into one value; if the count barely moves, the abbreviation table does not match the conventions in your data.

Step 4 — Apply structural rules

# normalise/step4_rules.py
import re

POSTCODE_RE = {
    "GB": re.compile(r"^[A-Z]{1,2}\d[A-Z\d]?\s*\d[A-Z]{2}$", re.I),
    "US": re.compile(r"^\d{5}(-\d{4})?$"),
    "CA": re.compile(r"^[A-Z]\d[A-Z]\s*\d[A-Z]\d$", re.I),
}


def structural_findings(components: dict, country: str = "US") -> list[str]:
    findings = []

    number = components.get("AddressNumber", "")
    if number and not re.match(r"^\d+[A-Za-z]?(-\d+[A-Za-z]?)?$", number):
        findings.append(f"implausible house number {number!r}")
    if number.isdigit() and int(number) > 99999:
        findings.append(f"house number {number} out of plausible range")

    pc = (components.get("ZipCode") or "").strip()
    rule = POSTCODE_RE.get(country)
    if pc and rule and not rule.match(pc):
        findings.append(f"postal code {pc!r} does not match the {country} format")
    if not pc:
        findings.append("no postal code — match will rely on locality alone")

    street = components.get("StreetName", "")
    if street and len(street) < 2:
        findings.append(f"street name {street!r} is implausibly short")
    return findings

Verification: the postal-code rule is the highest-value check here because it is objective and cheap. Records failing it are frequently transposed digits or a locality typed into the postcode field, both of which are fixable at source and neither of which any geocoder will resolve correctly.

Step 5 — Gate on parse quality

# normalise/step5_gate.py
def gate(record: dict) -> dict:
    """Decide whether a record is fit to send to the geocoder."""
    if record.get("rejected"):
        return {"send": False, "bucket": "unusable", "reason": record["rejected"]}
    if not record.get("parsed"):
        return {"send": False, "bucket": "unparsed", "reason": record.get("reason")}
    if record.get("kind") == "PO Box":
        return {"send": False, "bucket": "po_box",
                "reason": "no physical premises to geocode"}
    if not record.get("complete"):
        return {"send": True, "bucket": "partial",
                "reason": f"missing {record['missing_required']}",
                "expect": "locality or postcode level at best"}
    if record.get("structural"):
        return {"send": True, "bucket": "suspect", "reason": "; ".join(record["structural"])}
    return {"send": True, "bucket": "clean", "reason": None}

Verification: report the bucket counts before every geocoding run. Those five numbers turn “the match rate dropped” into a diagnosable statement — a fall in the clean bucket is an input-quality regression, while a stable clean bucket with a falling match rate points at the reference data.

Interpreting Results

Bucket What it means What to do
clean Fully parsed, all required components, no structural findings Geocode; this is the population your match rate should be quoted over
suspect Parsed but a rule fired (bad postcode, odd number) Geocode and compare; a suspect record that matches at rooftop is fine, one that falls back is evidence
partial Missing street or number Geocode with reduced expectations; never count these as rooftop failures
po_box No physical premises Do not geocode; report separately
unparsed Parser could not tag the string Route to review; these are usually a handful of recurring patterns worth adding rules for
unusable Not an address Fix at source; the record has no spatial content

The single most useful output of this stage is the pairing of bucket with eventual match level. Once you can say “94% of clean records match at rooftop, 41% of suspect records fall back to postcode”, the normalisation rules pay for themselves — the buckets predict outcomes, so a batch can be assessed before it is geocoded at all.

Which parse bucket produces which match levelGrid of four parse buckets against three match outcomes — rooftop or parcel, interpolated or street, and fallback or failure — showing that clean records overwhelmingly match at building level while partial and suspect records mostly fall back.rooftop/parcelstreet levelfallback/failclean94%5%1%suspect61%27%12%partial8%34%58%unparsed2%11%87%Share of each bucket by eventual match outcome, one batch of 100,000 records.Once this table exists, a batch can be assessed before it is geocoded at all — the bucket counts predict the outcome.
The buckets are predictive, which is what makes normalisation worth its cost.

Gotchas & Edge Cases

Title-casing breaks some real names. “McDonald Street” becomes “Mcdonald Street” under naive title case, and Irish, Dutch and Scottish name particles suffer similarly. Keep an exception list, or preserve the original casing and normalise only for comparison rather than for storage.

Normalisation rules that are safe, and one that is notGrid of four normalisation operations with what each fixes and the risk it carries: whitespace and case folding, separator stripping, abbreviation expansion, and title casing.FixesRiskTrim + case fold"r1 " vs "R1"noneStrip separators"R-1" vs "R1"collides if both are real codesExpand abbreviations"St" vs "Street"locale-specific; wrong table, wrong resultTitle case"MAIN STREET"breaks McDonald, van der, O’BrienNormalise for comparison, and keep the raw string. The last row is why storing the title-cased form as authoritative is a mistake.
Three of these are safe and one quietly damages real names — keep the raw value regardless.

The same abbreviation means different things in different locales. “Cl” is Close in the UK and rarely used in the US; “Ter” is Terrace in both but also appears as part of genuine street names. Locale-specific tables are not optional if you handle more than one country.

Parsers are probabilistic and confident. usaddress tags every token, including ones it is unsure about, and RepeatedLabelError catches only outright contradictions. Sample the parser output regularly against human judgement; a silently mis-tagged locality is worse than a parse failure because it passes the gate.

Unit numbers migrate into the house number field. “12/4 Mill Road” is flat 4 at number 12 in some conventions and number 12–4 in others. Decide which convention your data uses and encode it, because the two produce different geocoding results and neither errors.

Normalisation must be versioned. Change the abbreviation table and the canonical strings change, which changes the cache keys and the deduplication results. Store the normalisation version alongside the record, exactly as the geocoder version is stored, so a difference between two runs can be attributed correctly.

When to Escalate

  • A recurring unparsed pattern above a few percent is a source-system problem, not a parser problem. Take it to the team that captures the data; adding parser rules for a broken input form is a permanent maintenance cost.
  • Postal-code format failures concentrated in one source point at a field mapping error upstream — often a locality or a phone extension landing in the postcode column. Fix the mapping rather than the strings.
  • A parse quality drop with no code change means the input distribution moved: a new supplier, a new intake form, or a system migration. This belongs in the drift monitoring described in Attribute Schema Mapping for Spatial Datasets.
  • PO boxes or non-premises records forming a material share of a dataset intended for spatial analysis is a scoping conversation, not a cleaning task — those records will never have a building position.

Frequently Asked Questions

Does normalisation actually improve match rates?

Substantially, and more importantly it makes the match rate interpretable. Modern geocoders tolerate messy input, so normalisation buys a few points of match rate at best — but it converts an unknown mixture of geocoder limitations and input defects into two separately measurable numbers. Once parse failures are gated out, a drop in match rate is a statement about the reference data rather than about your input pipeline.

Should the normalised or the raw string be sent to the geocoder?

Send structured components where the provider accepts them, and the normalised single line otherwise. Structured input removes the geocoder's own parsing step, which is the largest source of variation between providers. Always keep the raw string: it is the only thing that lets you reproduce a result after the normalisation rules change.

How should unit and sub-premises information be handled?

Parse it out and keep it in its own field, but do not expect it to affect the coordinate. Most reference data positions the building rather than the flat, so units mainly matter for deduplication and for explaining legitimate coordinate coincidence. Sending unit information to a geocoder that cannot use it occasionally degrades a match rather than improving it.

Is a postcode enough on its own?

Only for postcode-level analysis, and only if you record that it is what you did. A postcode-only query returns a postcode centroid by construction, which is legitimate for regional aggregation and unusable at building scale. The important thing is to mark those records so they are not later mistaken for failed rooftop matches.

Related

Back to Geocoding and Address Data Validation