Azure Databricks Model Serving endpoint update failed: build logs, service logs and 429 errors

An endpoint shows Not ready (Update failed) or Ready (Update failed), or clients start getting 4xx and 429 responses. Find out first whether production is affected, then whether the failure happened at build time, at model load or at request time.

#First check: is it serving?

EP=churn-prod

databricks serving-endpoints get "$EP" -o json | jq '{
  id,
  state,
  active:  [.config.served_entities[]?         | {name, entity_name, entity_version, state}],
  pending: [.pending_config.served_entities[]? | {name, entity_name, entity_version, state}]
}'

In the API, state.ready is READY or NOT_READY, and state.config_update can be NOT_UPDATING, IN_PROGRESS, UPDATE_FAILED or UPDATE_CANCELED. Each served entity also has state.deployment and a state.deployment_state_message. Read the message first, because it often names the cause. Some documentation examples show update_state instead of config_update, so the command above prints the whole state object.

#Get the logs

The second argument is the served entity's name from the output above, such as churn-7. It isn't the Unity Catalog model name.

# Container image build: dependency and image problems
databricks serving-endpoints build-logs "$EP" churn-7

# Running container: model load and predict() output
databricks serving-endpoints logs "$EP" churn-7

You can also read both in the Logs tab of the endpoint page. Build logs are kept for up to 30 days. Service logs are short-lived, so save them to the incident ticket before you redeploy.

A successful build shows an event of type SERVED_ENTITY_CONTAINER_EVENT with the message Container image creation finished successfully. If no build event appears after an hour, contact Databricks support.

#Symptom, cause, fix

An MlflowException raised by the model returns a 4xx, and 5xx is reserved for Databricks-side faults. So 4xx points to the model or caller, and 5xx to a support case.

#Roll back when nothing is serving

Point the endpoint at the last good model version. Get the previous version from your deployment record or from Unity Catalog model history.

Warning

update-config replaces the served entities and traffic configuration, so include everything the endpoint needs, including environment_vars. Copy the existing values from serving-endpoints get first. A missing secret reference causes a second failed update.

cat > rollback.json <<'JSON'
{
  "served_entities": [
    {
      "name": "churn-6",
      "entity_name": "main.ml.churn",
      "entity_version": "6",
      "workload_size": "Small",
      "scale_to_zero_enabled": false,
      "environment_vars": {
        "API_KEY": "{{secrets/ml_prod/api_key}}"
      }
    }
  ],
  "traffic_config": {
    "routes": [
      { "served_model_name": "churn-6", "traffic_percentage": 100 }
    ]
  }
}
JSON

databricks serving-endpoints update-config "$EP" --json @rollback.json --no-wait

Without --no-wait, the CLI waits for up to 20 minutes by default.

A stopped endpoint only needs starting:

databricks api post "/api/2.0/serving-endpoints/$EP/config:start" --json '{}'

#Reproduce a failure without another deploy

Test the dependencies and the input in a notebook first:

import mlflow

mlflow.models.predict(
    model_uri="models:/main.ml.churn/7",
    input_data={"tenure": 34.2, "spend": 11.2, "plan": "green"},
    content_type="json",
    env_manager="virtualenv",
)

MLflow rebuilds the model's logged dependencies in an isolated environment, so a package that is missing, conflicts or fails to install shows up here.

#Secrets and permissions

Environment variables only resolve secrets with the exact syntax {{secrets/scope/key}}. Anything else is passed as plain text.

The endpoint creator must have READ on the referenced secret. Check the scope's ACLs:

databricks secrets list-acls ml_prod

If a caller can't query the endpoint, check its permissions. These commands use the endpoint id, not its name:

EP_ID=$(databricks serving-endpoints get "$EP" -o json | jq -r .id)
databricks serving-endpoints get-permissions "$EP_ID"

Route-optimized endpoints use a different URL and accept only Databricks OAuth tokens, not personal access tokens. A client that sends a PAT to one will fail even with the correct permissions.

#Is it capacity or the model?

databricks serving-endpoints export-metrics "$EP" \
  | grep -E '^(request_4xx_count_total|request_5xx_count_total|cpu_usage_percentage|mem_usage_percentage|provisioned_concurrent_requests_total)'

If inference tables are enabled through AI Gateway, query the failing requests. Replace the table name with the one configured for your endpoint:

SELECT request_time, status_code, request, response
FROM main.ml_logs.churn_prod_payload
WHERE status_code != 200
  AND request_time >= current_timestamp() - INTERVAL 1 HOUR
ORDER BY request_time DESC
LIMIT 50;

Note

Requests and responses over 1 MB aren't logged to inference tables. Check the current Model Serving limits page for payload size, request duration and QPS limits before you tell a caller what to change.

#Prevent it next time

• Run mlflow.models.predict with env_manager="virtualenv" in CI before you promote a model version.
• Pin dependencies when logging, and package private code as wheel artifacts.
• Keep the last good served_entities JSON in source control, so rollback is one command.
• Split traffic between old and new versions with traffic_config.routes before moving to 100%.
• Scrape /api/2.0/serving-endpoints/<name>/metrics with Prometheus or Datadog, and alert on 5xx, 429s and queue time.
• Set up persistent logging with AI Gateway inference tables or OpenTelemetry export, because service logs don't last.