How Apache Airflow Works Internally: DAG Parsing, Task Scheduling, and the Executor Architecture Behind Production Workflow Orchestration

Most engineers who use Airflow know it as “the thing that runs your pipelines.” They write DAGs, watch the UI, and debug task failures. What they rarely dig into is how the scheduler actually decides when to run a task, why DAG parsing is a constant source of performance problems, and what the executor really does under the hood. That understanding matters the moment you start operating Airflow at scale: slow DAG parsing, scheduler bottlenecks, and misconfigured executors are the three most common failure modes in production, and you cannot fix what you do not understand.

This post covers Airflow’s internals from the ground up: how DAG files become scheduled task instances, how different executor types work and where their tradeoffs bite you, how the metastore schema drives everything, and what the triggerer component adds to the async deferral story. It closes with a comparison of Airflow against Prefect, Dagster, Temporal, and Argo Workflows.

DAG Files Are Python Code, and That Is the First Problem

Airflow discovers workflows by importing Python files from a configured DAGs folder. The scheduler process runs a DagFileProcessor that imports each file, introspects the resulting DAG objects, and persists metadata to the metastore. This happens continuously on a configurable interval (default: 30 seconds per file).

The key point is that DAG files are executed as regular Python modules. Any code at module level runs on every parse cycle. Expensive imports, database calls, API lookups, or large in-memory objects at module level will slow the entire scheduler. This is why you see warnings everywhere about keeping DAG files “lightweight.”

# Bad: module-level database call runs on every parse
from mylib.db import fetch_pipeline_config
config = fetch_pipeline_config()  # executed every 30 seconds by the scheduler

with DAG("my_pipeline", schedule="@daily", ...) as dag:
    ...

# Better: defer to task runtime
from airflow.decorators import task

with DAG("my_pipeline", schedule="@daily", ...) as dag:
    @task
    def fetch_and_process():
        from mylib.db import fetch_pipeline_config
        config = fetch_pipeline_config()
        ...

Starting with Airflow 2.0, the project introduced DAG serialization: after the file processor parses a DAG, it serializes the DAG structure to JSON and stores it in the serialized_dag table in the metastore. The webserver and other components read from this serialized representation instead of re-parsing the Python files themselves. This decouples UI rendering from file parsing and significantly reduces the load on the webserver, but it also means that changes to a DAG file are not reflected immediately in the UI until the next parse cycle completes and serialization updates the database row.

The Scheduler Tight Loop

The Airflow scheduler is a long-running process with a core loop that performs four things repeatedly:

1. File parsing: A pool of DagFileProcessor subprocesses imports DAG files and persists DAG definitions and metadata to the metastore.
2. DAG run creation: For each active DAG, the scheduler checks whether a new DagRun should be created based on the schedule interval and the last successful run. It evaluates catchup behavior, data interval boundaries, and paused state.
3. Task instance scheduling: For each open DagRun, the scheduler evaluates task dependencies. A task instance transitions from none to scheduled when all upstream dependencies are met, sensors pass, and no pool slots are exhausted.
4. Executor submission: Scheduled task instances are submitted to the configured executor, which handles actual execution.

The scheduler writes task instance state changes to the metastore and polls it to detect completion. This poll-based model means every executor type ultimately communicates back through the metastore, not through direct IPC.

The min_file_process_interval and scheduler_heartbeat_sec configuration values control how frequently each phase runs. In large deployments with thousands of DAGs, the file processing phase dominates CPU and can starve the scheduling phase.

# airflow.cfg relevant tunables
[scheduler]
min_file_process_interval = 30      # seconds between re-parsing the same file
scheduler_heartbeat_sec = 5         # loop frequency
parsing_processes = 4               # parallel DagFileProcessor workers
max_dagruns_to_create_per_loop = 10 # cap on DagRun creation per loop iteration
max_tis_per_query = 512             # task instances fetched per DB query

Starting with Airflow 2.4, you can run multiple scheduler replicas in HA mode. Each scheduler instance participates in leader election for DAG run creation using a SchedulerJob lock in the metastore, while task instance scheduling is distributed across all healthy scheduler instances. This dramatically improves throughput but requires a PostgreSQL or MySQL metastore (SQLite does not support the concurrent access patterns involved).

The Metastore Schema

Everything in Airflow is mediated through a relational database. The core tables are:

• dag: one row per discovered DAG, storing schedule, paused state, fileloc, and ownership metadata.
• serialized_dag: the JSON-serialized DAG structure, updated on each successful parse.
• dag_run: one row per execution of a DAG, with run_id, execution_date (renamed logical_date in 2.2+), state (running/success/failed), and run type (scheduled/manual/backfill).
• task_instance: one row per task per dag run. Contains state, start_date, end_date, try_number, hostname, queue, and pool. This table is the primary coordination surface between the scheduler and executors.
• xcom: stores task cross-communication values. Each row has dag_id, task_id, run_id, key, and value (serialized). XCom values are capped at the database column size by default (often 64KB for MEDIUMBLOB in MySQL), which makes XCom inappropriate for large payloads.
• connection: named connection objects storing credentials and endpoint configuration. Referenced by operators via conn_id.
• pool: concurrency groups. Task instances declare a pool and consume pool_slots. The scheduler enforces pool limits before submitting task instances to the executor.
• variable: key-value store for runtime configuration.

The scheduler performs a complex join across dag_run and task_instance on every scheduling loop to compute which tasks are eligible to run. At high task-instance volume (millions of rows), this join is a frequent source of database CPU pressure. Keeping the task_instance table pruned via the [core] min_serialized_dag_fetch_interval and the built-in db clean CLI command matters in production.

Executor Architectures

The executor is responsible for one thing: taking a scheduled task_instance and running it somewhere. The interface is simple, but the implementations vary considerably.

Local Executor

The LocalExecutor runs tasks as subprocesses on the same machine as the scheduler. It uses Python’s multiprocessing module with a configurable parallelism limit.

[core]
executor = LocalExecutor

[local_executor]
parallelism = 32  # max concurrent subprocesses

This is the right choice for development and small deployments. The failure mode is clear: if the scheduler machine goes down, everything stops. There is no worker tier to scale independently.

Celery Executor

The CeleryExecutor publishes task instances to a Celery broker (Redis or RabbitMQ) and Celery worker processes consume and execute them. The scheduler acts as a Celery producer; workers report completion back by writing to the metastore directly.

[core]
executor = CeleryExecutor

[celery]
broker_url = redis://redis:6379/0
result_backend = db+postgresql://airflow:airflow@postgres/airflow
worker_concurrency = 16

Workers are stateful long-running processes. They pull tasks from the broker queue and run them in forked subprocesses (or gevent threads, depending on the Celery worker type). The key operational facts:

• Worker pool size is fixed at startup. Scaling requires restarting workers or running more instances.
• All workers consume from the same default queue unless tasks or workers are configured with named queues.
• The result_backend pointing to the metastore means Celery’s task result tracking is co-located with Airflow’s own state tracking. This is redundant but required for Celery’s internal mechanics.
• Worker logs are local to each worker machine, which complicates centralized log collection.

Named queues give you worker specialization: GPU tasks go to GPU workers, high-memory tasks go to large-memory workers.

@task(queue="gpu")
def train_model():
    ...

Kubernetes Executor

The KubernetesExecutor creates a Kubernetes pod for each task instance. When a task completes, the pod terminates. There are no persistent worker processes.

[core]
executor = KubernetesExecutor

[kubernetes]
namespace = airflow
worker_container_repository = myregistry/airflow-worker
worker_container_tag = 2.9.0
delete_worker_pods = True

The scheduler communicates with the Kubernetes API server to launch pods. Each pod runs a single task and writes its completion state back to the metastore before exiting. The pod spec is built from a configurable template (pod_template_file) and can be overridden per-task via executor_config.

from kubernetes.client import models as k8s

@task(
    executor_config={
        "KubernetesExecutor": {
            "pod_override": k8s.V1Pod(
                spec=k8s.V1PodSpec(
                    containers=[
                        k8s.V1Container(
                            name="base",
                            resources=k8s.V1ResourceRequirements(
                                requests={"memory": "4Gi", "cpu": "2"},
                                limits={"memory": "8Gi", "cpu": "4"},
                            ),
                        )
                    ]
                )
            )
        }
    }
)
def heavy_transform():
    ...

The Kubernetes executor eliminates the idle worker cost of Celery (pods only exist when tasks are running) and gives per-task resource isolation. The tradeoff is pod startup latency (10 to 60+ seconds for large images), higher Kubernetes API server load at scale, and more complex debugging because pod logs are ephemeral.

CeleryKubernetes Executor

The CeleryKubernetesExecutor routes tasks to either Celery workers or Kubernetes pods based on a queue value. Tasks queued to kubernetes get a pod; everything else goes to Celery workers. This is useful when most tasks are lightweight and benefit from fast Celery worker startup, but some tasks need per-task resource isolation.

[core]
executor = CeleryKubernetesExecutor

[celery_kubernetes_executor]
kubernetes_queue = kubernetes

The Triggerer and Async Deferrals

Before Airflow 2.2, sensors and other blocking tasks occupied a worker slot for their entire duration. A sensor polling for an S3 file every 60 seconds held a Celery worker subprocess for hours. At scale, this caused worker starvation: all concurrency slots consumed by sleeping sensors.

The triggerer component solves this. Deferrable operators split their work into two phases:

1. The operator’s execute() method raises a TaskDeferred exception, passing a Trigger object that describes what to wait for. The task instance transitions to deferred state and releases its worker slot.
2. The triggerer process runs an asyncio event loop and polls all active triggers concurrently (thousands in a single process). When a trigger fires, the task instance is re-queued and a worker picks it up to complete execution from the resume() method.

from airflow.sensors.base import BaseSensorOperator
from airflow.triggers.temporal import DateTimeTrigger
from airflow.utils.context import Context
from datetime import timedelta
from airflow.utils import timezone

class DeferrableWaitSensor(BaseSensorOperator):
    def execute(self, context: Context):
        self.defer(
            trigger=DateTimeTrigger(moment=timezone.utcnow() + timedelta(minutes=30)),
            method_name="execute_complete",
        )

    def execute_complete(self, context: Context, event=None):
        return event

The triggerer runs as a separate process and can be horizontally scaled. Each triggerer instance handles a subset of active triggers. The metastore stores trigger state, so triggerer restarts are safe.

Connection and Pool Management

Airflow’s Connection model stores named credentials in the metastore or in environment variables (AIRFLOW_CONN_<CONN_ID>). Operators receive connections via the BaseHook.get_connection() method, which checks environment variables first, then the database.

import os
# Environment-based connection (preferred in containerized deployments)
# AIRFLOW_CONN_MY_POSTGRES=postgresql://user:pass@host:5432/mydb
os.environ["AIRFLOW_CONN_MY_POSTGRES"] = "postgresql://user:pass@host:5432/mydb"

Pools decouple task concurrency from executor parallelism. A task might be scheduled and eligible but blocked because its pool has no available slots. This is useful for rate-limiting connections to external systems.

# Define pool in UI or CLI:
# airflow pools set my_api_pool 10 "Rate-limited external API"

@task(pool="my_api_pool", pool_slots=1)
def call_external_api():
    ...

Production Considerations

Tune DAG parsing aggressively. In deployments with 500+ DAG files, parsing_processes should match available CPU cores on the scheduler machine. Set min_file_process_interval to at least 60 seconds for DAGs that change infrequently. If you have DAGs that are rarely modified, consider grouping them and using .airflowignore patterns to exclude inactive files entirely.

Use HA scheduler in production. A single scheduler is a single point of failure. Two scheduler replicas with PostgreSQL as the metastore is the minimum for production. Three provides better resilience during rolling restarts. Monitor scheduler_heartbeat metrics; a missed heartbeat longer than scheduler_health_check_threshold (default 30 seconds) means scheduling has stalled.

Right-size Celery workers. worker_concurrency defaults to 16 but the right value depends on task workload. CPU-bound tasks: match physical cores. I/O-bound tasks: 2x to 4x cores. Memory-intensive tasks: calculate (total_memory - OS_overhead) / max_task_memory and use that as the ceiling. Running OOM tasks will silently kill workers.

Keep the task_instance table pruned. Airflow accumulates task instance rows indefinitely unless you run airflow db clean. At tens of millions of rows, scheduling loop queries slow noticeably. Automate airflow db clean --clean-before-timestamp on a weekly schedule with a 90-day retention window for most deployments.

XCom is not for data transport. XCom values live in the metastore and are fetched by the downstream task before execution. A 100MB XCom value will be serialized, stored as a database blob, and deserialized on every downstream task. Use object storage (S3, GCS) for any payload above a few kilobytes; store only the reference in XCom.

Monitor executor queue depth. For Celery, the Celery Flower UI or Redis LLEN on the task queue shows backlog depth. A growing queue means workers cannot keep up. For Kubernetes, watch Kubernetes API server rate limits; Airflow schedulers hitting the 429 rate limit will queue tasks but fail to submit pods.

Use connection secrets backends. The default metastore-based connection storage is fine for small teams but becomes a security concern at scale. AWS Secrets Manager, HashiCorp Vault, and GCP Secret Manager are all supported as SecretsBackend implementations. They also allow per-environment connection overrides without changing DAG code.

Tradeoffs: Airflow vs the Alternatives

Dimension	Apache Airflow	Prefect	Dagster	Temporal	Argo Workflows
Architecture	Separate scheduler, worker, webserver, triggerer, metastore	Python-native, hybrid cloud/self-hosted	Asset-centric, integrated UI, plugin model	Durable execution engine, separate worker SDK	Kubernetes-native, CRD-based
Scheduling model	Cron + data interval, DB-polled	Event-driven + cron, push-based	Asset freshness policies + cron	Workflow code drives scheduling via signals and timers	Cron + event triggers via sensors
Executor model	Pluggable (Local, Celery, Kubernetes, Hybrid)	Process-based, Kubernetes via task runners	Process-based, Dask, K8s via launchers	Worker processes consuming task queues	Kubernetes pods per step
State backend	PostgreSQL / MySQL (required at scale)	PostgreSQL or Prefect Cloud	PostgreSQL	PostgreSQL or Cassandra	Kubernetes etcd via CRDs
Dynamic DAGs	Supported via Python, but parsing overhead applies	First-class, flows are Python functions	First-class, assets computed at definition time	First-class, workflows are code	Supported via DAG templates and loops
UI	Mature, graph/gantt/log views, some UX debt	Clean, real-time, cloud-hosted option	Asset catalog, lineage graph, type-checked	Basic built-in, typically augmented by custom tooling	Argo UI, decent for K8s-native teams
Community	Very large (Apache project, 10+ years)	Active, commercial backing (PrefectHQ)	Active, commercial backing (Dagster Labs)	Growing, commercial backing (Temporal Technologies)	Growing, CNCF incubating
Operational complexity	High: multiple processes, metastore tuning, executor config	Medium: simpler deployment, managed option reduces ops	Medium: single process possible, managed option	Medium-High: separate server and worker processes	Medium: requires Kubernetes, simpler for K8s-native shops
Sweet spot	Large data engineering orgs with existing Airflow investment, ETL-heavy workloads	Teams wanting Python-first UX with lower ops burden	Teams prioritizing data asset lineage and type safety	Long-running business process orchestration, microservice sagas	Kubernetes-native shops already operating Kubernetes at scale

Closing

Airflow’s architecture is a product of its history: it was designed for batch ETL on a single machine and grew incrementally into a distributed system. The scheduler’s Python-parsing model, the metastore as the universal coordination layer, and the pluggable executor interface all reflect that evolution. Understanding where those seams are, specifically the parsing overhead, the metastore as a bottleneck, and the executor’s role in resource allocation, is what separates teams that run Airflow reliably from teams that fight it constantly. The alternatives in the table above made different bets on these tradeoffs; which bet is correct depends on what your workloads actually look like.