Home / Data Platform / Airflow

Data PlatformTroubleshootingIntermediate

Airflow tasks failing randomly: zombies, SIGKILL and retries

Read Airflow task logs and worker events to tell code failures from killed workers, zombie tasks and timeouts, and set retries, timeouts and resources so jobs recover on their own.

Deepak Bhardwaj2 min read

Applies toApache Airflow 2.x (Celery and Kubernetes executors), Astronomer, Amazon MWAA, Cloud Composer

Read the end of the task log first

Log line Meaning
A Python traceback Your code failed. Fix the code or the input data.
Task exited with return code -9 / Negsignal.SIGKILL The process was killed, nearly always out of memory
Detected zombie job / was marked as failed because it stopped heartbeating The worker died or lost contact with the metadata DB
AirflowTaskTimeout execution_timeout reached
Task received SIGTERM signal The worker pod or process was stopped (deploy, scale-in, eviction)
Log missing entirely The task never started on a worker, or remote logging is misconfigured

SIGKILL (-9): memory

# Kubernetes executor / KubernetesPodOperator
kubectl get pods -n airflow --field-selector=status.phase=Failed
kubectl describe pod <task-pod> -n airflow | grep -A3 "Last State"   # Reason: OOMKilled

# Celery workers on VMs
dmesg -T | grep -i "killed process"

Fixes: give the task more memory (executor_config with a pod override, or a dedicated Celery queue on bigger workers), or stop loading the whole dataset into pandas. Push the work down to the warehouse or Spark.

Zombies

A zombie is a task the database thinks is running, but whose worker has stopped heartbeating. Common causes:

  • Worker OOM or node failure (see above).
  • The metadata database is overloaded, so heartbeats time out.
  • Long blocking calls in the task process combined with a very low scheduler_zombie_task_threshold.

Check the database (connections, CPU) and worker restarts before raising the threshold.

SIGTERM during deploys

Rolling worker restarts kill running tasks. Use worker_shutdown/warm shutdown for Celery (Airflow waits for running tasks up to the grace period), and set a long enough terminationGracePeriodSeconds for worker pods.

Make tasks self-healing

default_args = {
    "retries": 3,
    "retry_delay": timedelta(minutes=5),
    "retry_exponential_backoff": True,
    "max_retry_delay": timedelta(minutes=30),
    "execution_timeout": timedelta(hours=1),
}

Retries only help if the task is idempotent: rerunning must overwrite a partition or MERGE, never append duplicates.

Rerun after fixing

airflow tasks clear my_dag -t load_orders -s 2026-10-21 -e 2026-10-21 --downstream --yes

Tip

Add an on_failure_callback that posts the log URL and the exit code to your on-call channel. "Return code -9" in the alert tells the engineer it's memory before they open anything.

Related fixes