Airflow vs Prefect for Spatial Pipeline Orchestration

You have settled on scheduling your spatial validation directed acyclic graph (DAG) with a real orchestrator rather than cron, and the choice has narrowed to Apache Airflow and Prefect. This page builds the same pipeline — validate a set of GeoPackage layers, one per tile, and report invalid geometries — in both tools so you can compare DAG authoring, dynamic task mapping over spatial partitions, retries, alerting, and deployment against code you can run. It assumes single-cluster deployments up to a few thousand tiles per run and sits under orchestration tool selection for spatial pipelines, where the broader scheduling-model trade-offs are laid out. If your inputs are streaming rather than batched, the queue-based approach in designing async validation queues with Celery is the better fit.

Prerequisites

• Python 3.10+ with two isolated environments — Airflow and Prefect pin conflicting versions of pendulum and pydantic, so never install both in one environment.
• Airflow 2.9.x: pip install "apache-airflow==2.9.*" with a PostgreSQL metadata database. The LocalExecutor is fine for evaluation; KubernetesExecutor or CeleryExecutor for production.
• Prefect 2.19.x: pip install "prefect==2.19.*". A local run needs nothing beyond the package; scheduled runs need a Prefect Server or Prefect Cloud workspace and a started worker.
• GeoPandas 0.14+ and Shapely 2.0+ in both environments for the shared validation function.
• A GeoPackage dataset partitioned by a tile_id column. Gotcha: GeoPackage layers are read whole by read_file; use the where filter or convert to GeoParquet if tiles are large, or reads will pull the entire layer per task.

Step-by-Step Procedure

Step 1 — Write the framework-agnostic validation function

Keep spatial logic out of both orchestrators so the comparison is fair and the logic is testable in isolation.

# validation_core.py — imported unchanged by both Airflow and Prefect
import geopandas as gpd


def validate_layer(gpkg_path: str, layer: str, target_epsg: int = 4326) -> dict:
    """Validate one GeoPackage layer; return a small JSON-safe summary."""
    gdf = gpd.read_file(gpkg_path, layer=layer)
    if gdf.crs is None:
        raise ValueError(f"Layer {layer!r} has no CRS")
    if gdf.crs.to_epsg() != target_epsg:
        gdf = gdf.to_crs(epsg=target_epsg)

    valid = gdf.geometry.is_valid
    return {
        "layer": layer,
        "features": int(len(gdf)),
        "invalid": int((~valid).sum()),
        "null_geom": int(gdf.geometry.isna().sum()),
    }

Verification: call validate_layer("tiles.gpkg", "tile_0001") in a plain Python shell and confirm it returns a dict with an invalid count before touching either orchestrator.

Step 2 — Build the Airflow DAG with dynamic task mapping

# dags/spatial_validation.py
from datetime import datetime, timedelta
from airflow.decorators import dag, task
from validation_core import validate_layer


@dag(
    schedule="0 2 * * *",
    start_date=datetime(2026, 1, 1),
    catchup=False,
    default_args={"retries": 3, "retry_delay": timedelta(minutes=1),
                  "retry_exponential_backoff": True},
    tags=["spatial"],
)
def spatial_validation():

    @task
    def list_layers(gpkg_path: str) -> list[str]:
        import fiona
        return sorted(fiona.listlayers(gpkg_path))

    @task
    def validate(gpkg_path: str, layer: str) -> dict:
        return validate_layer(gpkg_path, layer)

    layers = list_layers("/data/tiles.gpkg")
    validate.partial(gpkg_path="/data/tiles.gpkg").expand(layer=layers)


spatial_validation()

Verification: run airflow dags test spatial_validation 2026-07-10 and confirm one mapped validate instance completes per layer, each with its own log.

Step 3 — Build the equivalent Prefect flow

# flow.py
from prefect import flow, task
from prefect.tasks import task_input_hash
from validation_core import validate_layer
import fiona


@task(retries=3, retry_delay_seconds=60, cache_key_fn=task_input_hash)
def validate(gpkg_path: str, layer: str) -> dict:
    return validate_layer(gpkg_path, layer)


@flow(name="spatial-validation")
def spatial_validation(gpkg_path: str = "/data/tiles.gpkg") -> list[dict]:
    layers = sorted(fiona.listlayers(gpkg_path))
    return validate.map(gpkg_path=gpkg_path, layer=layers)


if __name__ == "__main__":
    print(spatial_validation())

Verification: run python flow.py and confirm the flow-run tree shows one validate task run per layer, with the returned list of summary dicts printed.

Step 4 — Wire retries, alerting, and deployment

Airflow attaches alerting through a failure callback; Prefect through a state hook or a Cloud automation.

# Airflow: alert on any task failure
def alert_on_failure(context):
    ti = context["task_instance"]
    # send to Slack / PagerDuty using ti.task_id, ti.log_url
    print(f"ALERT: {ti.task_id} failed at {ti.log_url}")

# add default_args={"on_failure_callback": alert_on_failure, ...} to the @dag

# Prefect: alert via an on-failure state hook
from prefect import flow

def alert(flow, flow_run, state):
    print(f"ALERT: {flow_run.name} entered {state.type}")

@flow(on_failure=[alert])
def spatial_validation(gpkg_path: str = "/data/tiles.gpkg"):
    ...

Verification: force a failure (point at a missing layer) and confirm the alert callback fires exactly once per failed task in Airflow and once per failed flow run in Prefect.

Interpreting Results

The two pipelines produce identical per-tile summaries; the difference is operational, not analytical. In Airflow, open the grid view and read failures column by column — each mapped validate instance is a cell, so a red cell points straight at the failing tile and its log. In Prefect, open the flow run and read the task-run tree — each validate.map child is a node with its own state and logs.

Deployment is where the two diverge most. An Airflow pipeline is deployed by placing the DAG file where the scheduler parses it and ensuring every worker image contains the same dependencies — GeoPandas, GDAL, and the PROJ data files that to_crs needs. Because the scheduler re-parses DAG files continuously, any heavy import or network call at module top level slows the whole deployment, so spatial imports belong inside task bodies. A Prefect pipeline is deployed by building a deployment that points at your flow and binding it to a work pool; a worker in that pool pulls runs and executes them in a process or container you control. The practical consequence for spatial teams is that Prefect makes it trivial to pin a heavyweight geospatial container per deployment, while Airflow expects a more uniform worker fleet. Neither is wrong, but a shop that already runs Kubernetes will find Prefect work pools and the Airflow KubernetesExecutor roughly comparable, whereas a shop on a single VM will find Prefect’s process worker the lighter path.

The table below maps the comparison dimensions to a recommendation.

Gotchas & Edge Cases

GeoPackage layer reads are not lazy. gpd.read_file(path, layer=...) loads the whole layer into memory. If tiles are large, either push a where predicate or convert to GeoParquet so each mapped task reads a bounded partition. This matters more in Airflow’s packed workers than in isolated Prefect Kubernetes jobs.

Dependency conflicts between the two tools are real. Airflow and Prefect pin incompatible pydantic majors in some releases. Never share a virtual environment; use separate images or pipx-managed environments.

XCom and Prefect results are not for geometry. Returning a GeoDataFrame from a task serialises it into the metadata store. Return only the summary dict shown above; persist geometry to object storage and pass the path.

Prefect caching can mask a changed layer. The cache_key_fn=task_input_hash in Step 3 keys on arguments, so if a GeoPackage is overwritten in place without changing its path, a cached run may skip re-validation. Include a content hash or modification time in the cache key when inputs mutate under a stable path.

Airflow top-level DAG code runs on every parse. Calling fiona.listlayers at module import (outside a task) would hit disk on every scheduler heartbeat. Keep all I/O inside @task functions, as shown.

Backfills behave differently. Airflow treats backfill as a first-class command — airflow dags backfill reprocesses a date range using each run’s logical execution date, which is ideal when a validation rule changes and you must re-validate history. Prefect has no logical-date concept baked into flows; you reprocess by triggering deployment runs over a parameter range, passing the tile or date explicitly. If reprocessing historical spatial data on rule changes is a frequent operation for you, Airflow’s model requires less custom plumbing.

Worker memory is your responsibility in both. Neither tool measures the memory a geometry-heavy task will consume. In Airflow, cap concurrency with pools or max_active_tis_per_dag; in Prefect, limit the work pool’s concurrency. Size both from the measured peak memory of validating your largest single layer, not from CPU count, or the Linux out-of-memory killer will terminate workers with an opaque signal rather than a clean, retryable exception.

When to Escalate

• Assets and lineage are the primary concern: if you think in terms of “the validated parcels layer” rather than “the nightly job,” an asset-oriented orchestrator fits better — see scheduling spatial validation with Dagster assets.
• Inputs are streaming or sub-second: neither batch scheduler suits continuous ingest; move to a queue with async validation queues built on Celery.
• Tile counts exceed a few thousand per run: scheduler parsing and mapped-task expansion slow down; batch tiles into groups and cap concurrency, or shift the fan-out into a distributed compute layer rather than the orchestrator.

Frequently Asked Questions

Related

• Orchestration Tool Selection for Spatial Pipelines — the scheduling-model, retry, and backfill trade-offs behind this comparison
• Scheduling Spatial Validation with Dagster Assets — the asset-oriented alternative to both Airflow and Prefect
• Designing Async Validation Queues with Celery — the queue-based path when inputs stream rather than batch

Back to Orchestration Tool Selection for Spatial Pipelines