Airflow task logs missing in the UI: \"Could not read served logs\", 403 Forbidden and remote logging that never uploads

A task failed and you open its log in the UI to find out why. Instead you get *** Could not read served logs, 403 Client Error: FORBIDDEN, Log file does not exist, or an empty page. This is usually a configuration problem in one of three places: how the UI fetches logs from the worker, the shared secret that signs that request, or remote logging that was meant to upload the file but never did.

#How the UI finds a log

Knowing the lookup order tells you which part is broken:

1. If remote logging is on, the UI (the webserver in Airflow 2, the API server in Airflow 3) reads the log from object storage.
2. If it isn't there, or the task is still running, it asks the worker's log server over HTTP (default port 8793, [logging] worker_log_server_port), using a short-lived token signed with the shared secret key.
3. If the worker is gone (a deleted Kubernetes pod, a scaled-in Celery node), the local file went with it.

The message lines starting with *** at the top of the log page show which sources were tried and what each one returned. Read them first.

#Symptom → cause → fix

#Check the configuration each component actually uses

Config is often set in one place (a Helm value, an env var on the scheduler) and missing from another. Run these inside each component: UI/API server, scheduler, and a worker.

airflow version
airflow config get-value logging remote_logging
airflow config get-value logging remote_base_log_folder
airflow config get-value logging remote_log_conn_id
airflow config get-value logging worker_log_server_port
airflow providers list | grep -Ei "amazon|google|azure"

On Kubernetes:

for d in deploy/airflow-api-server deploy/airflow-scheduler sts/airflow-worker; do
  echo "== $d"
  kubectl -n airflow exec $d -- airflow config get-value logging remote_logging
done

Deployment names vary by chart and version (airflow-webserver in Airflow 2 charts, airflow-api-server in Airflow 3, workers are a StatefulSet with Celery); check kubectl -n airflow get deploy,sts.

Tip

Environment variables override airflow.cfg. The pattern is AIRFLOW__<SECTION>__<KEY>, for example AIRFLOW__LOGGING__REMOTE_LOGGING=True. Check kubectl exec ... -- env | grep AIRFLOW__ when the file and the running value disagree.

#Fix 403 Forbidden from the worker log server

The UI signs each log request with the secret key and the worker checks the signature and the token age. Two things break it.

#Secret key mismatch

The key must be identical on the UI and every worker. The section changed between major versions:

• Airflow 2: [webserver] secret_key (AIRFLOW__WEBSERVER__SECRET_KEY)
• Airflow 3: [api] secret_key (AIRFLOW__API__SECRET_KEY)

Names vary by version; check your build's configuration reference. Compare a hash, not the value (these commands are for Airflow 3; use webserver instead of api on Airflow 2), so the secret doesn't land in your terminal history or ticket:

kubectl -n airflow exec deploy/airflow-api-server -- sh -c 'airflow config get-value api secret_key | sha256sum'
kubectl -n airflow exec sts/airflow-worker -- sh -c 'airflow config get-value api secret_key | sha256sum'

A common cause is letting each pod generate a random key at start-up. Store the key in a Kubernetes Secret (the official Helm chart has a values entry for it) and mount it into every component.

Warning

Changing the secret key logs out every UI session, and in Airflow 3 can invalidate tokens that running tasks hold. Change it in a quiet window and roll all components together.

#Clock skew

The token is valid only for a short time. If the worker's clock is ahead of or behind the UI host by more than that, every request is rejected. Check:

date -u; kubectl -n airflow exec sts/airflow-worker -- date -u

Fix time sync on the nodes (chrony or the cloud provider's time service). Raising the token age is a workaround, not a fix.

#Can the UI reach the worker at all?

kubectl -n airflow exec deploy/airflow-api-server -- \
  python -c "import socket; s=socket.create_connection(('airflow-worker-0.airflow-worker.airflow.svc', 8793), 5); print('ok')"

A 403 here is expected from a raw request with no token; "Connection refused" or a timeout means a NetworkPolicy, security group or hostname problem. The hostname the worker records for a task comes from [core] hostname_callable; if it records a name the UI can't resolve, fetching fails.

#Turn on remote logging properly

With KubernetesExecutor or autoscaled workers, remote logging is the only way to keep logs after the worker goes away.

#S3

[logging]
remote_logging = True
remote_base_log_folder = s3://my-airflow-logs/prod
remote_log_conn_id = aws_logs
encrypt_s3_logs = False

The provider apache-airflow-providers-amazon must be installed in every image, including the UI image, because it reads the logs. The role needs s3:PutObject, s3:GetObject and s3:ListBucket on the bucket and prefix.

#Google Cloud Storage

[logging]
remote_logging = True
remote_base_log_folder = gs://my-airflow-logs/prod
remote_log_conn_id = google_cloud_default

#Azure Blob Storage

Azure uses a wasb prefix and a separate container setting; the exact keys have changed across provider releases, so follow the logging page for your apache-airflow-providers-microsoft-azure version.

#Test the connection from a worker

The upload runs on the worker, so test from there with the same connection ID:

kubectl -n airflow exec sts/airflow-worker -- python - <<'EOF'
from airflow.providers.amazon.aws.hooks.s3 import S3Hook
hook = S3Hook(aws_conn_id="aws_logs")
hook.load_string("ok", key="prod/_healthcheck.txt", bucket_name="my-airflow-logs", replace=True)
print(hook.read_key(key="prod/_healthcheck.txt", bucket_name="my-airflow-logs"))
EOF

If this fails, remote logging will fail the same way, and the reason ends up in the worker's own stdout, not in the task log. Check it with kubectl logs.

Then run a task and confirm the object appears:

airflow tasks test example_dag print_date 2026-10-31
aws s3 ls s3://my-airflow-logs/prod/ --recursive | tail -5

airflow tasks test runs the task without recording state, so it is safe for a quick check. Log uploads from it depend on your version and settings; a real triggered run is the definitive test.

Caution

[logging] delete_local_logs = True removes the local file after upload. Turn it on only after you have confirmed uploads work, or you lose the only copy.

#Prevent it next time

• Bake the logging provider and remote logging config into one shared image and Helm values file used by every component.
• Keep the secret key in a managed secret, never generated per pod, and alert if components disagree after a deploy.
• Monitor time sync on all nodes; skew causes 403s that look like auth bugs.
• Add a smoke DAG after each deploy that runs a task and checks the log object exists in the bucket.
• Set a bucket lifecycle rule that matches your audit retention, and document it so "old logs are gone" is an expected answer.
• Make sure the UI's network path to workers on port 8793 is in your NetworkPolicy and security group definitions, not added by hand.