Applies toOpenAI API, Azure OpenAI in Azure AI Foundry, Amazon Bedrock Runtime (InvokeModel and Converse); official Python SDKs
Retry or not?
| Retry with backoff | Don't retry; fix the request or config |
|---|---|
| 429 rate limit / ThrottlingException | 400 bad request / ValidationException |
| 500, 502, 503, 504 / InternalServerException, ServiceUnavailableException | 401 / 403 authentication and authorisation |
| Timeouts and connection resets | 404 model or deployment not found |
| ModelNotReadyException (Bedrock) | Context length exceeded |
| Content filter / guardrail blocks (unless you change the input) |
Use exponential backoff with jitter, honour any Retry-After header, and cap the attempts. The official SDKs retry some errors automatically (max_retries in the OpenAI SDKs, retry config in boto3), so don't stack your own retry loop on top without checking.
OpenAI API
| Status | Typical message / type | Cause | Fix |
|---|---|---|---|
| 400 | invalid_request_error, context_length_exceeded | Bad parameter, or prompt plus max output too long | Validate the parameters; trim history; lower the output limit |
| 401 | invalid_api_key | Wrong, revoked or missing key | Check the key and the environment variable in the running process |
| 403 | Permission / region not supported | Project lacks model access, or the request comes from an unsupported region | Check the project's model access and the organisation settings |
| 404 | model_not_found | Wrong model name, or no access to it | Check the exact model ID |
| 429 | rate_limit_exceeded | RPM/TPM limit for the model | Back off; reduce concurrency; request a higher tier |
| 429 | insufficient_quota | Out of credit or billing limit reached | Fix billing. Retrying won't help |
| 500 / 503 | server_error, overloaded | Provider-side issue | Retry with backoff; fall back to another model |
Azure OpenAI
| Status | Clue | Cause | Fix |
|---|---|---|---|
| 400 | content_filter, ResponsibleAIPolicyViolation | Prompt or completion blocked by content filters | Check which category; adjust the input or the filter configuration (with approval) |
| 400 | context_length_exceeded | Too many tokens | Trim the input; lower max tokens |
| 400 | OperationNotSupported / unsupported parameter | Parameter not supported by this model or API version | Check the API version and model; some reasoning models need max_completion_tokens |
| 401 | Access denied due to invalid subscription key or wrong API endpoint | Wrong key or endpoint (resource) | Match the key to the resource; prefer Entra ID auth |
| 401 / 403 with Entra ID | PermissionDenied | Identity lacks the role | Assign Cognitive Services OpenAI User on the resource |
| 403 | Traffic blocked | Network rules or private endpoint only, called from a public network | Call from an allowed network; check private DNS |
| 404 | DeploymentNotFound | Deployment name wrong, still being created, or deleted | Use the deployment name, not the model name |
| 429 | Requests to the ... have exceeded token rate limit | TPM/RPM of the deployment | Back off using retry-after-ms / retry-after; raise the quota; spread across deployments or regions |
| 500 / 503 | Internal error | Service-side | Retry; fail over to another region's deployment |
Check the deployment and its capacity:
az cognitiveservices account deployment list -g rg-ai -n aoai-prod \
--query "[].{name:name, model:properties.model.name, ver:properties.model.version, sku:sku.name, cap:sku.capacity}" -o table
Amazon Bedrock
Errors come back as exception names:
| Exception | Cause | Fix |
|---|---|---|
AccessDeniedException | IAM policy lacks bedrock:InvokeModel for that model ARN, or model access not granted | Fix the IAM policy; enable model access in the console |
ValidationException | Wrong request body for the model, unsupported parameter, input too long | Use the Converse API for a uniform shape; check the model's limits |
ResourceNotFoundException | Wrong model ID, or the model isn't in this region | Check the ID and region; use an inference profile for cross-region |
ThrottlingException | Requests or tokens per minute exceeded | Back off; request a quota increase; use cross-region inference profiles |
ModelTimeoutException | Model took too long | Shorten the input or output; use streaming |
ModelNotReadyException | Model not ready to serve | Retry after a short wait |
ServiceUnavailableException / InternalServerException | Service-side | Retry with backoff |
ServiceQuotaExceededException | Account quota | Request an increase in Service Quotas |
import boto3, botocore
brt = boto3.client("bedrock-runtime", region_name="eu-west-1",
config=botocore.config.Config(retries={"max_attempts": 5, "mode": "adaptive"}))
try:
brt.converse(modelId="<model-id-or-inference-profile>",
messages=[{"role": "user", "content": [{"text": "ping"}]}])
except brt.exceptions.ThrottlingException:
...
except botocore.exceptions.ClientError as e:
print(e.response["Error"]["Code"], e.response["Error"]["Message"])
What to log for every call
| Field | Why |
|---|---|
| Provider, endpoint/region, model/deployment | Which backend failed |
Request ID (x-request-id, apim-request-id, x-amzn-RequestId) | Needed for a provider support case |
| Status and error code | Grouping |
| Prompt and completion token counts | Throttling and cost analysis |
| Latency (time to first token, total) | Performance |
| Retry count | Hidden throttling |
Don't log full prompts by default if they may contain personal data. Log hashes or redacted text.
Quick incident checks
- Is it one deployment/model, or all of them? (one region or provider issue vs your code)
- Did anything change: key rotation, deployment rename, API version, model retirement?
- Is it 429 across the board? Check the token usage metrics against the quota.
- Check the provider status page.
Checklist
- Retry only retryable errors, with jitter and a cap
- SDK retries and your retries not doubled up
- Request IDs and token counts logged
- Alerts on 429 rate and 5xx rate per deployment
- Fallback model or region for 5xx and sustained 429