Airflow failure alerts checklist: callbacks, notifiers, Slack and email that actually fire

#The goal

When a production DAG fails, the right person hears about it within minutes, with a link to the log. When Airflow itself is down, someone still hears about it. Most "we never got alerted" incidents come from one of five gaps: the callback was set in the wrong place, the provider wasn't installed, the connection was wrong, the callback itself threw an exception, or nothing watched Airflow from the outside.

Work through the checklist below once per environment, then re-run the test section after every Airflow upgrade.

#1. Decide what gets an alert

• Each production DAG has a named owner and a destination (channel or mailing list).
• You picked task-level or DAG-level failure alerts per DAG, not both, to avoid duplicate pages.

#2. Install and configure the notifier

Notifiers (Airflow 2.6+) are reusable callback objects shipped in provider packages. Install the provider in every image that runs tasks and in the DAG processor:

pip install apache-airflow-providers-slack apache-airflow-providers-smtp
airflow providers list | grep -E "slack|smtp"

Create the connections. Keep the secret in your secrets backend rather than in the metadata database where you can:

# Slack API connection (bot token with chat:write)
airflow connections add slack_alerts \
  --conn-type slack \
  --conn-password 'xoxb-REPLACE-ME'

# SMTP connection
airflow connections add smtp_alerts \
  --conn-type smtp \
  --conn-host smtp.example.com --conn-port 587 \
  --conn-login airflow@example.com --conn-password 'REPLACE-ME'

airflow connections get slack_alerts

• Provider installed in scheduler, DAG processor, worker and triggerer images (versions match).
• The Slack bot is invited to the target channel; otherwise posts fail with not_in_channel.
• SMTP host reachable from workers (port 587/465 often blocked by egress rules).

#3. Attach callbacks in the right place

from datetime import datetime, timedelta
from airflow.sdk import DAG, task              # Airflow 2.x: from airflow import DAG; from airflow.decorators import task
from airflow.providers.slack.notifications.slack import SlackNotifier
from airflow.providers.smtp.notifications.smtp import SmtpNotifier

task_failed_slack = SlackNotifier(
    slack_conn_id="slack_alerts",
    channel="#data-oncall",
    text=(
        ":red_circle: Task failed after retries\n"
        "DAG: {{ dag.dag_id }} | Task: {{ ti.task_id }} | Try: {{ ti.try_number }}\n"
        "Run: {{ run_id }}\n"
        "Log: {{ ti.log_url }}"
    ),
)

dag_failed_email = SmtpNotifier(
    smtp_conn_id="smtp_alerts",
    from_email="airflow@example.com",
    to="data-oncall@example.com",
    subject="[Airflow] {{ dag.dag_id }} run failed",
    html_content="Run {{ run_id }} of {{ dag.dag_id }} failed.",
)

with DAG(
    dag_id="orders_daily",
    start_date=datetime(2026, 1, 1),
    schedule="@daily",
    catchup=False,
    default_args={
        "owner": "orders-team",
        "retries": 2,
        "retry_delay": timedelta(minutes=5),
        "on_failure_callback": task_failed_slack,   # per task, after final retry
    },
    on_failure_callback=dag_failed_email,           # once per failed DAG run
):
    @task
    def load():
        ...
    load()

Where callbacks run matters when you debug them:

• Task callbacks set through default_args, so new tasks inherit them.
• DAG callbacks passed to the DAG(...) constructor, not to default_args (a DAG-level key in default_args is applied to tasks instead).
• Callbacks are lists if you need more than one: on_failure_callback=[task_failed_slack, page_pagerduty].
• Templates only use fields present in the context (dag, ti, run_id, logical_date, exception for task callbacks). Field availability varies by version; render one in dev before relying on it.

Warning

An exception inside a callback doesn't fail the task or re-alert. It's logged and lost. A wrong connection ID or a missing provider means silent failure, which is why the test in section 5 is not optional.

#4. Email and SLA changes in Airflow 3

• Replace email_on_failure / email_on_retry with SmtpNotifier. The email parameters are deprecated in Airflow 3 and the legacy email backend settings are deprecated in later 3.x releases. Some 3.x builds send legacy emails by default when an email address is set; set email_on_failure=False explicitly if you have moved to notifiers.
• Remove sla and sla_miss_callback. SLAs were removed in Airflow 3.0. Airflow 3.1 added Deadline Alerts as the replacement (marked experimental in 3.1); follow the "SLA to Deadlines" guide for your version, because the API has changed between releases.
• On Airflow 2.x, keep SLA checks outside Airflow (a freshness query on the target table) rather than relying on sla_miss_callback, which only evaluates in specific scheduler conditions.

#5. Test the alert path end to end

Deploy a test DAG to each environment and trigger it after every upgrade or provider bump:

from datetime import datetime
from airflow.sdk import DAG, task   # 2.x: from airflow import DAG; from airflow.decorators import task

with DAG("zz_alert_test", start_date=datetime(2026, 1, 1), schedule=None, catchup=False,
         default_args={"retries": 0, "on_failure_callback": task_failed_slack},
         on_failure_callback=dag_failed_email, tags=["ops"]):
    @task
    def boom():
        raise RuntimeError("Alert path test - ignore")
    boom()

airflow dags unpause zz_alert_test
airflow dags trigger zz_alert_test
airflow dags list-runs zz_alert_test          # 2.x: airflow dags list-runs -d zz_alert_test

• Slack message arrived with a working log link (check base_url under [api] in 3.x or [webserver] in 2.x if the link points to localhost).
• DAG-level email arrived once.
• If either is missing, read the task log tail and the DAG processor log for the callback traceback.

#6. Alert from outside Airflow

Callbacks can't fire if the scheduler is down, the worker was killed by OOM before reporting, or a DAG never got parsed. Cover those from the outside:

# Health endpoint: 2.x /health, 3.x /api/v2/monitor/health (check your version's API docs)
curl -s https://airflow.example.com/api/v2/monitor/health | jq

# DAG import errors
airflow dags list-import-errors

• An external monitor polls the health endpoint and pages when the scheduler or DAG processor heartbeat is unhealthy.
• StatsD/OpenTelemetry metrics shipped, with alerts on rising ti_failures and on missing scheduler heartbeats.
• A freshness check on key output tables (row count or max timestamp for today) catches DAGs that silently stopped scheduling.
• Import errors are checked after each deploy in CI or by the monitor.

#Prevent it next time

• Keep notifier definitions in one shared module (include/alerts.py) so a connection rename is a one-line change.
• Route by owner: build the channel from the DAG's owner or tags rather than sending everything to one noisy channel nobody reads.
• Add zz_alert_test to your upgrade runbook and to provider-bump PR checklists.
• Review alert volume monthly. If a DAG pages every week for a known flaky source, fix the retries or the source; don't train on-call to ignore pages.