Checking Cloud-Optimized GeoTIFF Structure

A cloud-optimized GeoTIFF (COG) is an ordinary GeoTIFF arranged so that a client reading it over HTTP can fetch just the part it needs. The arrangement is the whole product: internal tiling so a window maps to a few contiguous byte ranges, an internal overview pyramid so a zoomed-out view does not read full resolution, and a header layout that lets the client learn the file’s structure from its first request. Get any of the three wrong and the file still opens perfectly on a local disk while costing a hundred times more to read from object storage. This guide validates all three, plus compression settings, and finishes with the byte-range test that proves it in practice. It is the structural companion to the grid and void checks in Raster and Elevation Data Quality Checks.

Prerequisites

• GDAL 3.4+ (3.8+ preferred, for the COG driver’s improved defaults) and rio-cogeo 5.x for the structural validator: pip install rio-cogeo==5.*.
• Network access to the published location if you intend to run the byte-range test. The structural checks work on local files; only the final proof needs the object store.
• A written product specification covering block size, overview resampling method, minimum pyramid depth and the permitted compression codecs. Structural validation compares the file to that specification, not to taste.
• Familiarity with the alignment rules in Validating Raster Alignment and Resolution with Rasterio — a COG can be perfectly structured and still sit off the product grid.

Step-by-Step Procedure

Step 1 — Run the structural validator first

# cog/step1_validate.py
from rio_cogeo.cogeo import cog_validate


def validate_structure(path: str) -> dict:
    """rio-cogeo's structural check: is this file a valid COG, and why not?"""
    is_valid, errors, warnings = cog_validate(path, strict=True)
    return {
        "path": path,
        "valid_cog": bool(is_valid),
        "errors": list(errors),
        "warnings": list(warnings),
    }


if __name__ == "__main__":
    import json
    import sys
    result = validate_structure(sys.argv[1])
    print(json.dumps(result, indent=2))
    raise SystemExit(0 if result["valid_cog"] else 1)

Verification: a compliant file returns valid_cog: true with an empty error list. Run it against a plain gdal_translate output without COG options and confirm it fails with “The file is greater than 512xH or Wx512, but is not tiled” — that message is the single most common COG defect in the wild.

Step 2 — Inspect tiling and the overview pyramid

The validator answers “is it a COG”; the specification answers “is it our COG”.

# cog/step2_layout.py
import rasterio


def layout_facts(path: str) -> dict:
    with rasterio.open(path) as src:
        block_y, block_x = src.block_shapes[0]
        overviews = src.overviews(1)
        smallest = None
        if overviews:
            factor = overviews[-1]
            smallest = (src.width // factor, src.height // factor)
        return {
            "tiled": src.profile.get("tiled", False),
            "block_size": (block_x, block_y),
            "overview_factors": overviews,
            "overview_count": len(overviews),
            "smallest_overview": smallest,
            "compression": str(src.compression) if src.compression else None,
            "interleave": str(src.interleaving) if src.interleaving else None,
        }


def check_layout(facts: dict, block: int = 512, min_smallest: int = 512) -> list[str]:
    problems = []
    if not facts["tiled"]:
        problems.append("raster is striped, not tiled — range reads will be pathological")
    if facts["block_size"] != (block, block):
        problems.append(f"block size {facts['block_size']} != specified ({block}, {block})")
    if not facts["overview_factors"]:
        problems.append("no internal overviews")
    elif max(facts["smallest_overview"] or (10 ** 9,)) > min_smallest:
        problems.append(
            f"pyramid too shallow: smallest overview is {facts['smallest_overview']}, "
            f"should be under {min_smallest} px on the long side")
    return problems

Verification: the pyramid should continue until the smallest overview fits inside roughly one block. A 20,000-pixel raster therefore needs about six levels; stopping at two means a continental zoom level still reads 5,000-pixel data.

Step 3 — Confirm the header layout

# The COG driver reports the internal structure; the IFD offsets must precede the image data.
gdalinfo --debug on -json elevation_cog.tif 2>&1 | grep -i "ifd\|BLOCK\|OVERVIEW" | head -20

# A quicker structural summary for a whole directory:
for f in *.tif; do
  printf '%s: ' "$f"
  python -c "
from rio_cogeo.cogeo import cog_info
i = cog_info('$f')
p = i.Profile
print(f\"tiled={p.Tiled} blocks={p.Blockxsize}x{p.Blockysize} ovr={len(p.Overviews)} comp={p.Compression}\")
"
done

Verification: cog_info reports the profile without reading pixel data. Every file in a collection should print the same block size and compression; a stray line in that output is a tile that went through a different production path.

Step 4 — Check compression and predictor for the data type

# cog/step4_compression.py
import rasterio

LOSSY = {"jpeg", "webp"}
PREDICTOR_FOR = {
    "int16": 2, "uint16": 2, "int32": 2, "uint32": 2,   # horizontal differencing
    "float32": 3, "float64": 3,                          # floating-point predictor
}


def check_compression(path: str, allowed=("deflate", "lzw", "zstd")) -> list[str]:
    problems = []
    with rasterio.open(path) as src:
        comp = (str(src.compression.value) if src.compression else "none").lower()
        dtype = src.dtypes[0]
        predictor = src.profile.get("predictor")

    if comp in LOSSY:
        problems.append(f"{comp} is lossy — unacceptable for measurement data")
    elif comp not in allowed:
        problems.append(f"compression {comp!r} is not in the permitted set {allowed}")

    expected = PREDICTOR_FOR.get(dtype)
    if expected and predictor not in (expected, str(expected)):
        problems.append(
            f"dtype {dtype} should use predictor {expected}, found {predictor!r} "
            "— files will be substantially larger than necessary")
    return problems

Verification: re-encode a sample tile with and without the predictor and compare sizes. On a 16-bit elevation model the horizontal predictor typically saves 30–50%; if your files are unexpectedly large, this is usually why.

Step 5 — Prove partial reads over HTTP

Structure is a means to an end. This is the end.

# cog/step5_range.py — how many bytes does a small window actually cost?
import os

import rasterio
from rasterio.windows import Window

os.environ["CPL_VSIL_CURL_CHUNK_SIZE"] = "16384"
os.environ["GDAL_DISABLE_READDIR_ON_OPEN"] = "EMPTY_DIR"
os.environ["CPL_CURL_VERBOSE"] = "NO"


def window_read_cost(url: str, col: int = 4096, row: int = 4096, size: int = 512) -> dict:
    """Read one window from a remote COG and report the transferred bytes."""
    from rasterio._env import get_gdal_config, set_gdal_config

    set_gdal_config("CPL_VSIL_CURL_REPORT_STATS", "YES")
    with rasterio.open(f"/vsicurl/{url}") as src:
        data = src.read(1, window=Window(col, row, size, size))
        total_px = src.width * src.height
    return {
        "window_px": int(data.size),
        "raster_px": total_px,
        "fraction_requested": round(data.size / total_px, 6),
        "note": "compare against transferred bytes reported by CPL_VSIL_CURL_REPORT_STATS",
    }

Verification: run the same read against a striped copy of the file and compare the transferred bytes reported by GDAL’s curl statistics (or by your object store’s access log). A correct COG transfers a few hundred kilobytes for a 512-pixel window; a striped file of the same size transfers tens or hundreds of megabytes, because a stripe spans the full raster width.

Interpreting Results

Structural validity is binary and non-negotiable; the specification checks around it are graded. A file that fails cog_validate should never reach the catalogue. A file that passes but uses a 256-pixel block where the specification says 512 is a consistency warning — worth fixing at the next re-processing, not worth blocking a delivery.

Gotchas & Edge Cases

gdal_translate -co TILED=YES is not enough. It produces a tiled TIFF, not a COG: no overviews, and no guarantee about header placement. Use the dedicated COG driver (gdal_translate -of COG) or rio cogeo create, both of which handle the layout requirements together.

Adding overviews after the fact rewrites the file. gdaladdo on an existing COG can produce a file that is tiled and overviewed but no longer COG-compliant, because the new IFDs land after the image data. Re-encode rather than append.

GDAL_DISABLE_READDIR_ON_OPEN changes remote performance more than the file does. Without it, GDAL lists the containing prefix on every open, which on a bucket with a million objects dominates the read. Set it in the client environment before concluding that a COG is badly built.

The web-optimized variant is a different thing. rio cogeo create --web-optimized reprojects to web mercator and aligns to the tiling scheme. That is a valid product decision, but it changes the CRS and the grid — validate it against a web-mercator specification, not against your source grid.

Sparse files pass structural validation and surprise consumers. A COG written with SPARSE_OK=TRUE omits blocks that are entirely nodata. This is legitimate and efficient, but a client that assumes every block exists may misread the absence. Record sparseness in the product metadata.

When to Escalate

• A whole delivery fails cog_validate — this is a producer configuration issue, not a per-file defect. Send the validator output and the exact gdal_translate invocation you expect.
• Files are compliant but reads are slow from your bucket — investigate client configuration and object-store request patterns first; escalate to infrastructure rather than to the data producer.
• Lossy compression on measurement data — this is a specification breach, not a tuning question. Reject and re-source; JPEG artefacts in an elevation model cannot be undone.
• Structural requirements conflict with an internal tiling scheme — decide the product’s primary access pattern once, and write it into the specification rather than settling it per delivery.

Once the structure is trusted, the pixel-level checks become affordable: continue with Detecting Voids and Nodata Gaps in DEMs, which relies on the tiled layout validated here to keep memory bounded.

Frequently Asked Questions

Related

• Raster and Elevation Data Quality Checks — where structural checks sit in the wider raster rule set
• Validating Raster Alignment and Resolution with Rasterio — grid checks that a valid COG can still fail
• Validating GeoParquet Schemas with PyArrow — the same header-first validation idea applied to vector storage

Back to Raster and Elevation Data Quality Checks