What you gain and give up
| Hand-built VM | Managed endpoint |
|---|---|
| You patch OS, drivers, CUDA | Platform-managed images (you still own the container if custom) |
| Manual restarts, no health checks | Liveness/readiness probes, automatic replacement |
| Scaling = buy another VM | Autoscale rules, multiple instances |
| Risky in-place updates | Blue/green deployments with traffic split |
| Full control of everything | Constraints on instance types, regions, quotas |
1. Inventory each VM-hosted model
Model and version, framework, GPU type and memory used, request rate and latency, input/output schema, dependencies (tokenizers, files), and who calls it.
2. Choose the target
| Need | Good target |
|---|---|
| Open-weight LLM from a catalogue, no custom code | Serverless / pay-per-token model offerings (Azure AI Foundry, Bedrock) where the model is available |
| Your fine-tuned weights, supported architecture | Bedrock Custom Model Import, Foundry/Azure ML managed compute |
| Custom container (vLLM, Triton, own pre/post-processing) | Azure ML managed online endpoint, SageMaker endpoint, OCI model deployment |
3. Package
Prefer a container with the serving stack baked in (e.g. vLLM or Triton) and the model pulled from a registry or storage at start-up. Expose health and scoring routes the platform expects.
4. Deploy (Azure ML example)
# endpoint.yml
$schema: https://azuremlschemas.azureedge.net/latest/managedOnlineEndpoint.schema.json
name: ticket-classifier
auth_mode: aad_token
public_network_access: disabled
# blue.yml
$schema: https://azuremlschemas.azureedge.net/latest/managedOnlineDeployment.schema.json
name: blue
endpoint_name: ticket-classifier
model: azureml:ticket-classifier:7
environment: azureml:vllm-serving:3
instance_type: Standard_NC24ads_A100_v4
instance_count: 2
request_settings:
request_timeout_ms: 90000
max_concurrent_requests_per_instance: 16
liveness_probe:
initial_delay: 600 # large models take minutes to load
readiness_probe:
initial_delay: 600
az ml online-endpoint create -f endpoint.yml -g rg-ml -w mlw-prod
az ml online-deployment create -f blue.yml -g rg-ml -w mlw-prod --all-traffic
az ml online-endpoint invoke -n ticket-classifier --request-file sample.json -g rg-ml -w mlw-prod
The SageMaker equivalent is model → endpoint configuration (instance type, variants) → endpoint, with an LMI/vLLM or custom container. Check GPU instance quota in the target region before the migration window; it's the most common blocker.
5. Test
- Same requests to VM and endpoint; compare outputs (should match for deterministic settings).
- Load test to your peak with headroom; tune concurrency per instance.
- Failure test: delete an instance or force a restart and watch recovery time.
- Cold start: new instances of large models can take many minutes; set autoscale minimums accordingly.
6. Shift traffic
- Point the gateway (or client config) at the endpoint for 10% of requests.
- Compare latency, errors and quality metrics for a few days.
- Move to 100%; keep VMs stopped (not deleted) for the rollback window.
- Future model versions: deploy as
green, mirror or split traffic, then swap.
7. Retire the VMs
Delete VMs, disks, public IPs and old DNS records; revoke their credentials; update the runbook and CMDB.
Tip
Set autoscale minimum to at least two instances for production endpoints. With a single instance, every platform maintenance event or node failure is an outage while a large model reloads.
Checklist
- Models inventoried with resources and callers
- Target chosen per model; GPU quota confirmed
- Container with health routes; probes allow for load time
- Output parity, load and failure tests passed
- Gradual traffic shift; VMs retired and cleaned up