Home / Data Platform / Airflow
Apache Airflow and orchestration: schedulers, backfills, managed services and tuning. 19 posts.
Replace copy-pasted tasks and sensor chains with dynamic task mapping and dataset (asset) triggered DAGs. Control fan-out with max_active_tis_per_dag and pools, handle partial failures, and avoid the mapping patterns that overload the scheduler.
Write a small, correct Airflow DAG with the TaskFlow API, set a schedule without the classic start_date and catchup surprises, pass data between tasks, add retries, and test it with airflow dags test before deploying.
A release changed shared code, a requirement or a provider version, and now many DAGs show import errors, vanish from the UI or never get scheduled. Find the broken import, roll back or hotfix, catch up missed runs safely, and add the CI checks that stop it recurring.
Tasks using deferrable operators sit in the deferred state for hours. Check the triggerer is running and healthy, find triggers that block the async loop, fix provider packages and scale triggerer capacity.
Teams that upgraded the Airflow metadata database in place have hit different failures in staging and production. Clean the database, run the migration on a restored copy, fix DAGs with ruff's AIR rules, and only then schedule the real cutover with a tested rollback.
In Airflow 3, operators and tasks can't open database sessions to the Airflow metadata DB. Find the code that does, decide what it was for, and replace it with the Task SDK context or the official REST API client with a token, rather than a direct SQL hook.
Some teams upgrading from Airflow 3.3.0 to 3.3.1 or 3.3.2 on Kubernetes report every DAG file timing out in the DAG processor, DAGs being deactivated and runs stuck in queued. The root cause isn't confirmed yet. Here's how to tell whether you're hit, what to collect, and how to roll back cleanly.
Two patch releases in August and September 2026 changed behaviour you can trip over - DataFrame XComs under pandas 3, masked team-scoped secrets, backfill API responses and which credential wins when a request has both a cookie and a token. Check each one before rolling out.
The Airflow CLI commands, environment variable formats and secrets backend settings for managing connections and variables, plus how to move them between environments.
Tasks that sit in scheduled or queued are waiting for something, usually a concurrency limit, a full pool, missing workers or a broken executor. Check each limit in order with the Airflow CLI and metadata.
The orchestrator runs every pipeline, so its choice shapes operations for years. Compare code-first and visual tools, platform-native and cross-platform schedulers, decide by what you orchestrate and who builds pipelines, and avoid running several orchestrators that trigger each other in loops.
Slow task scheduling, workers blocked by sensors and a busy metadata database are usually design and configuration problems. Make DAG files cheap to parse, set concurrency limits deliberately, use pools to protect shared systems, replace blocking sensors with deferrable operators and keep the metadata database lean.
Restore an Airflow deployment where no tasks are scheduled because the scheduler is crashed, hung or blocked by its metadata database. Check heartbeats, database health, connections and locks, clean up old metadata safely and bring the backlog through.
When the morning data is late or missing, find which upstream step stalled, communicate, and backfill the missed intervals in Airflow in the right order without overloading sources or creating duplicates.
Move DAGs off a hand-built Airflow server onto managed Airflow. Clean up DAGs and dependencies, move connections and variables to a secrets backend, test on the target version, migrate history only if needed and switch DAG by DAG.
Go from DAG import errors and schedule logic to scheduler health, pools, concurrency limits and executor workers when Airflow tasks don't start.
Run Airflow without single points of failure using multiple active schedulers, a highly available PostgreSQL metadata database, redundant webservers and workers on Kubernetes, and remote logs.
Read Airflow task logs and worker events to tell code failures from killed workers, zombie tasks and timeouts, and set retries, timeouts and resources so jobs recover on their own.
Run Airflow on a single VM for a small team with LocalExecutor, an external PostgreSQL metadata database, persistent logs, secrets, backups and monitoring, and know when to move to a managed or HA setup.
No posts in this category yet.