Applies toAzure Machine Learning managed online endpoints, Amazon SageMaker real-time and serverless inference endpoints; similar logic for KServe
Symptoms
- First requests after a quiet period fail or take minutes.
- Errors during scale-out:
upstream request timeout, 408/429/503/504 from the endpoint. - Container logs show the model still loading (downloading weights, moving to GPU) when requests arrive.
Step 1: measure the load time
How long does one instance take from start to "ready"?
- Azure ML: deployment logs,
az ml online-deployment get-logs -e <endpoint> -n <deployment> --lines 200; look at the time between container start and the init/loadfunction finishing. - SageMaker: CloudWatch logs for the endpoint (
/aws/sagemaker/Endpoints/<name>), andModelSetupTime/ModelLatencymetrics.
Large LLMs often take several minutes (weight download + GPU load).
Step 2: match the cause
| Cause | Sign | Fix |
|---|---|---|
| Scale to zero / serverless cold start | Failures only after idle | Keep minimum instances ≥ 1 for production (or use provisioned concurrency for serverless) |
| Traffic sent before model ready | Errors right after scale-out | Readiness/liveness probe that only passes after the model is loaded; larger initial delay |
| Request timeout too short for long generations | Timeouts on long prompts only | Raise endpoint request timeout (within platform limits); stream responses |
| Autoscale too slow | Errors during spikes, recovery later | Scale on concurrency/queue metrics, earlier thresholds, more minimum instances |
| Max concurrency per instance too high | Timeouts under load, high latency | Lower concurrent requests per instance; scale out |
Azure ML settings
# deployment.yml (managed online deployment)
instance_count: 2
request_settings:
request_timeout_ms: 90000
max_concurrent_requests_per_instance: 4
readiness_probe:
initial_delay: 300
period: 10
failure_threshold: 30
liveness_probe:
initial_delay: 600
period: 30
Apply with az ml online-deployment update -f deployment.yml. Check the current maximum for request_timeout_ms in the docs. Autoscale is configured with Azure Monitor autoscale rules on the deployment (e.g. CPU/GPU utilisation or request latency), with a minimum instance count.
SageMaker settings
- Real-time endpoints: set
MinCapacityin Application Auto Scaling ≥ 1 (≥ 2 for production), and scale onSageMakerVariantInvocationsPerInstanceor concurrency-based metrics. ContainerStartupHealthCheckTimeoutInSecondsandModelDataDownloadTimeoutInSecondson the production variant for large models.- Serverless inference: provisioned concurrency to avoid cold starts; check memory and payload limits fit your model.
aws application-autoscaling register-scalable-target --service-namespace sagemaker \
--resource-id endpoint/llm-prod/variant/AllTraffic --scalable-dimension sagemaker:variant:DesiredInstanceCount \
--min-capacity 2 --max-capacity 8
Speed up loading
- Keep model weights close: in the image, on attached storage, or in the same region's blob/S3.
- Use formats that load fast (safetensors) and pre-quantized weights.
- Warm-up request in the startup script so CUDA kernels are compiled before the readiness probe passes.
Client side
- Retries with backoff on 429/503 (not on 400s).
- Client timeouts longer than the endpoint's, so you get a proper error instead of cutting off.
- Streaming for chat UIs, so users see tokens early.
Tip
Record "time to ready" for each model version. When a new version is twice the size, the autoscale rules and probes need to change with it.
Prevent it next time
- Minimum capacity ≥ 2 for production endpoints, across zones where supported.
- Probes tied to model readiness; measured load times in the runbook.
- Autoscale on load indicators with headroom; scheduled scale-up before known peaks.