Rotate the Airflow Fernet key without breaking connections and variables

#Goal

Replace the Fernet key that Airflow uses to encrypt connection passwords, connection extras and variables in the metadata database, without a window where tasks fail with cryptography.fernet.InvalidToken. Typical triggers: a security policy that requires periodic rotation, a key that leaked into a Git repo or a ticket, or a staff change.

The key is read from [core] fernet_key, usually set as the environment variable AIRFLOW__CORE__FERNET_KEY. Airflow accepts a comma-separated list of keys. It encrypts with the first and tries all of them to decrypt. That is what makes a rotation without downtime possible.

Caution

If you replace the key outright, every encrypted connection and variable becomes unreadable. Without the old key the values cannot be recovered, so you have to re-enter every credential by hand. Save the current key in your secrets store before you touch anything.

#Before you start

#Find where the key is set

Every component has to run with the same key value: scheduler, API server (webserver in Airflow 2), DAG processor, triggerer and workers. Find out where it comes from:

# On any Airflow container or host
airflow config get-value core fernet_key | cut -c1-6   # print only a prefix
env | grep -c AIRFLOW__CORE__FERNET_KEY

On Kubernetes with the official Helm chart, the key is in a Secret. Check which one:

kubectl -n airflow get secret | grep -i fernet
kubectl -n airflow get secret airflow-fernet-key -o jsonpath='{.data.fernet-key}' | base64 -d | cut -c1-6; echo

Warning

The Secret the Helm chart generates for you is not a good fit for rotation, because a helm upgrade won't change it the way you need. Create your own Secret, point fernetKeySecretName at it in your values file, and roll that out before you rotate. Check the chart docs for your chart version.

#Take a backup of the encrypted tables

You need this backup if the rotation goes wrong. Run it against the metadata database (PostgreSQL shown):

pg_dump -h <db-host> -U airflow -d airflow \
  -t connection -t variable -Fc -f airflow_secrets_$(date +%F).dump

#Record what is encrypted today

SELECT COUNT(*) FILTER (WHERE is_encrypted)       AS enc_passwords,
       COUNT(*) FILTER (WHERE is_extra_encrypted) AS enc_extras,
       COUNT(*)                                   AS total
FROM connection;

SELECT COUNT(*) FILTER (WHERE is_encrypted) AS enc_vars, COUNT(*) AS total
FROM variable;

Note these numbers so you can compare them after the rotation. Connections and variables that live in an external secrets backend (Vault, AWS Secrets Manager, Azure Key Vault) are not encrypted with the Fernet key, and this rotation doesn't affect them.

#Rotation procedure

#Step 1: Generate the new key

python -c "from cryptography.fernet import Fernet; print(Fernet.generate_key().decode())"

Store it in your secrets manager next to the old key. Don't paste it into chat or tickets.

#Step 2: Deploy both keys, new key first

Set the value to NEW_KEY,OLD_KEY. The order matters: Airflow encrypts with the first key.

Docker Compose (.env or the x-airflow-common environment block):

AIRFLOW__CORE__FERNET_KEY=<new_key>,<old_key>

Kubernetes with a self-managed Secret:

kubectl -n airflow create secret generic airflow-fernet-key \
  --from-literal=fernet-key='<new_key>,<old_key>' \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl -n airflow rollout restart deployment
kubectl -n airflow rollout restart statefulset

Restart every component and wait until no old pods or processes are left. Until then, a component still running with only the old key can't read values written with the new one.

kubectl -n airflow get pods -o wide   # all pods should show a recent AGE

#Step 3: Re-encrypt existing values

Run this once, from any container that has the two-key config:

airflow rotate-fernet-key

It decrypts each encrypted connection and variable with whichever key works and re-encrypts it with the first key. You can safely run it more than once.

#Step 4: Verify before removing the old key

Read a few known connections and variables through the CLI. These commands decrypt the values, so they fail if something is wrong:

airflow connections get my_postgres_conn -o json >/dev/null && echo OK
airflow variables get some_encrypted_var >/dev/null && echo OK

Then test that the new key alone can decrypt everything. Run this check in a one-off shell with only the new key set, so no running component changes:

AIRFLOW__CORE__FERNET_KEY=<new_key> python - <<'EOF'
from airflow.models import Connection, Variable
from airflow.utils.session import create_session
bad = []
with create_session() as s:
    for c in s.query(Connection):
        try:
            c.password; c.extra
        except Exception as e:
            bad.append(("conn", c.conn_id, type(e).__name__))
    for v in s.query(Variable):
        try:
            v.val
        except Exception as e:
            bad.append(("var", v.key, type(e).__name__))
print("failures:", bad or "none")
EOF

This script uses the ORM directly, so run it from a scheduler or admin container that can reach the metadata database. In Airflow 3, task code can no longer reach the database, so this check won't work from inside a task. Don't continue until it prints failures: none.

#Step 5: Remove the old key

Set the value to NEW_KEY only, then restart all components again in the same way as step 2. Trigger one DAG that uses a database connection, and one that uses a deferrable operator if you have them, to confirm the change end to end.

Keep the old key in the secrets store, marked as retired, until the next backup cycle has passed. If the old key leaked, it can still decrypt any backup taken before the rotation. Treat those backups as exposed, and rotate the credentials themselves as well as the key.

#When it goes wrong

Tip

If you're unsure whether the old key is right, try it in the one-off shell from step 4 before you put it back in the deployment.

#Prevent it next time

• Keep the Fernet key in a secrets manager and inject it into every component from that one place. Never hard-code it in airflow.cfg or in a Docker image.
• Put this runbook into a scheduled change and test it in a non-production environment first, against a copy of the metadata database.
• Move high-value credentials to a secrets backend so most of them don't depend on the Fernet key.
• Alert on InvalidToken in scheduler, API server and task logs. If it shows up, keys are out of sync.