Databricks REST API 429 Too Many Requests: find the noisy caller and fix automation at scale

#Symptom

Automation that ran fine at 50 jobs starts failing at 500. Calls to the workspace API return 429 Too Many Requests. Depending on the client, the error shows up as REQUEST_LIMIT_EXCEEDED, RESOURCE_EXHAUSTED or TooManyRequests. Typical victims:

• An orchestrator (ADF, Airflow, a custom scheduler) that polls jobs/runs/get for hundreds of runs every few seconds.
• A Terraform plan or apply across a large workspace that reads every job, cluster and permission.
• A script that calls jobs/runs/submit in a tight loop.
• Inventory or chargeback scripts that page through jobs/runs/list for every job.

Databricks rate limits apply per workspace and per endpoint. Some are fixed and cannot be raised. Check the current table on the Azure Databricks "Resource limits" page (section "API rate limits"); don't hard-code the numbers in your design.

Warning

A 429 on one endpoint doesn't block other endpoints. But a client that retries without backoff makes throttling worse for every other caller of that endpoint in the workspace, including production orchestrators.

#Step 1: confirm it is throttling, not a quota

Capture the response body. The error code in the body separates a rate limit from a quota more reliably than the HTTP status does.

#Step 2: find who is calling

Start with client-side logs. Not every throttled request is guaranteed to show up in audit logs. If you have system tables enabled, this shows which identities and user agents generate the most API traffic:

SELECT
  date_trunc('MINUTE', event_time)        AS minute,
  user_identity.email                     AS identity,
  user_agent,
  service_name,
  action_name,
  count(*)                                AS calls,
  count_if(response.status_code = 429)    AS throttled
FROM system.access.audit
WHERE event_time > now() - INTERVAL 2 HOURS
  AND workspace_id = '<workspace-id>'
GROUP BY ALL
ORDER BY calls DESC
LIMIT 50;

What to look for:

• One service principal with tens of thousands of jobs events per hour is usually a polling loop.
• user_agent values carry client names and versions, such as databricks-sdk-py/..., databricks-tf-provider/... or databricks-cli/.... They tell you which tool to fix.
• A spike every N minutes that lines up with a scheduler cadence points at an external orchestrator.

#Step 3: stop the bleeding

If production orchestration is being starved, slow the noisy client first:

1. Pause or disable the offending script or pipeline. Leave production job schedules running.
2. For Terraform, re-run with lower parallelism: terraform apply -parallelism=4.
3. Leave in-flight job runs alone. They keep running when status polling fails.

#Step 4: fix the client

#Let the SDK retry, and cap its rate

The Databricks SDK for Python retries 429 and 503 responses with backoff until retry_timeout_seconds runs out (300 seconds in the versions we checked). It can also cap its own request rate:

from databricks.sdk import WorkspaceClient
from databricks.sdk.core import Config

cfg = Config(
    profile="prod",              # or DATABRICKS_HOST plus auth environment variables
    rate_limit=10,               # max requests per second from this client
    retry_timeout_seconds=600,   # keep retrying throttled calls for up to 10 minutes
)
w = WorkspaceClient(config=cfg)

The same cap works as an environment variable: DATABRICKS_RATE_LIMIT=10. The Databricks Terraform provider has a rate_limit provider argument. Check the provider docs for its default.

Tip

With several workers, a per-client limit multiplies. Ten pods at rate_limit=10 can send 100 requests per second. Budget the workspace total, then divide it across workers.

#Raw HTTP clients: honour Retry-After and add jitter

import random, time, requests

def call(session, method, url, **kw):
    delay = 1.0
    for attempt in range(8):
        r = session.request(method, url, timeout=60, **kw)
        if r.status_code not in (429, 503):
            r.raise_for_status()
            return r.json()
        retry_after = r.headers.get("Retry-After")
        wait = float(retry_after) if retry_after else delay
        time.sleep(wait + random.uniform(0, wait / 2))   # jitter avoids synchronized retries
        delay = min(delay * 2, 60)
    r.raise_for_status()

Only retry calls that are safe to repeat. A timed-out runs/submit may already have created a run. Pass an idempotency_token on runs/submit and run-now so a retry returns the existing run instead of starting a second one.

#Step 5: cut the call volume

Retries handle short bursts. A design that sits above the limit all the time needs fewer calls.

A run-status report that never touches the REST API:

SELECT job_id, run_id, result_state, termination_code,
       min(period_start_time) AS started,
       max(period_end_time)   AS ended
FROM system.lakeflow.job_run_timeline
WHERE period_start_time > now() - INTERVAL 1 DAY
GROUP BY ALL
ORDER BY started DESC;

System tables lag real time, so keep API polling for the few runs where seconds matter, such as a gate in a deployment pipeline.

#Polling with the SDK without hammering the API

run_now() returns a waiter. .result() polls with backoff, so you don't need a hand-written loop:

from datetime import timedelta
run = w.jobs.run_now(job_id=123, job_parameters={"date": "2026-11-09"}).result(
    timeout=timedelta(hours=2)
)
print(run.state.result_state)

#Verify the fix

• Re-run the audit query an hour after the change. Calls per minute from the identity should drop, and the throttled column should be zero.
• Run the automation at its peak concurrency in a non-production workspace before you roll it out.
• Check that production orchestrators (ADF, Airflow) no longer log 429 retries.

#Prevent it next time

• Give every automation its own service principal. Identity-level audit queries then find the culprit in seconds.
• Set rate_limit explicitly in SDK, CLI and Terraform configurations that run at scale.
• Review new integrations for polling intervals and "list everything" calls before they reach production.
• Alert on 429s: schedule the audit query and notify when throttled > 0 for a production identity.
• Re-read the resource limits page when you plan a big increase in job count. Some endpoint limits are fixed, so the fix may have to be a design change rather than a support ticket.