Amazon SageMaker real-time endpoints CLI cheat sheet: status, test calls, variants, autoscaling, logs and metrics

#How the pieces fit

A SageMaker real-time endpoint is made of three objects. When an endpoint misbehaves, you usually have to check all three:

Endpoint configurations can't be changed after creation. To change the instance type or model, you create a new configuration and point the endpoint at it.

Set these once per shell:

export AWS_REGION=eu-west-1
EP=llm-prod

#Status and failure reasons

# All endpoints and their state
aws sagemaker list-endpoints \
  --query 'Endpoints[].[EndpointName,EndpointStatus,LastModifiedTime]' --output table

# Only the broken ones
aws sagemaker list-endpoints --status-equals Failed --query 'Endpoints[].EndpointName'

# One endpoint: status, failure reason, current config, per-variant capacity
aws sagemaker describe-endpoint --endpoint-name $EP \
  --query '{status:EndpointStatus,reason:FailureReason,config:EndpointConfigName,variants:ProductionVariants[].{name:VariantName,current:CurrentInstanceCount,desired:DesiredInstanceCount,weight:CurrentWeight}}'

# What the config says
CFG=$(aws sagemaker describe-endpoint --endpoint-name $EP --query EndpointConfigName --output text)
aws sagemaker describe-endpoint-config --endpoint-config-name $CFG \
  --query 'ProductionVariants[].{variant:VariantName,model:ModelName,type:InstanceType,count:InitialInstanceCount,weight:InitialVariantWeight}'

# What the model is
aws sagemaker describe-model --model-name <model-name> \
  --query '{image:PrimaryContainer.Image,data:PrimaryContainer.ModelDataUrl,role:ExecutionRoleArn,env:PrimaryContainer.Environment}'

# Wait for a create/update to finish (returns non-zero if it fails)
aws sagemaker wait endpoint-in-service --endpoint-name $EP

Endpoint statuses: Creating, Updating, SystemUpdating, RollingBack, InService, OutOfService, Deleting, Failed, UpdateRollbackFailed.

Tip

Large LLM containers can take longer to start than the default health check allows. Production variants support ContainerStartupHealthCheckTimeoutInSeconds and ModelDataDownloadTimeoutInSeconds in the endpoint configuration. Raise them rather than retrying the same failing deploy.

#Test calls

# JSON request; the body format depends on your container
aws sagemaker-runtime invoke-endpoint --endpoint-name $EP \
  --content-type application/json \
  --body '{"inputs":"Say OK","parameters":{"max_new_tokens":5}}' \
  --cli-binary-format raw-in-base64-out \
  out.json && cat out.json

# Send to one variant directly (bypasses weights)
aws sagemaker-runtime invoke-endpoint --endpoint-name $EP \
  --target-variant candidate \
  --content-type application/json --body fileb://request.json out.json

--cli-binary-format raw-in-base64-out lets you pass a raw JSON string with CLI v2. Without it, v2 expects the inline --body value to be base64. fileb:// sends a file as-is.

Errors you'll see from invoke-endpoint:

For token streaming, use invoke-endpoint-with-response-stream from an SDK. The CLI is fine for a quick check, but its output is awkward to read.

#Traffic between variants

# Shift weights without redeploying (canary 10%)
aws sagemaker update-endpoint-weights-and-capacities --endpoint-name $EP \
  --desired-weights-and-capacities VariantName=current,DesiredWeight=90 VariantName=candidate,DesiredWeight=10

# Change instance count on one variant
aws sagemaker update-endpoint-weights-and-capacities --endpoint-name $EP \
  --desired-weights-and-capacities VariantName=current,DesiredInstanceCount=3

# Swap the whole endpoint to a new configuration
aws sagemaker update-endpoint --endpoint-name $EP --endpoint-config-name llm-prod-cfg-20261109
aws sagemaker wait endpoint-in-service --endpoint-name $EP

Warning

update-endpoint with --retain-all-variant-properties keeps the current weights and instance counts. Without it, the values from the new configuration apply. Scaling down a busy endpoint by accident is a common way to cause 429s during a deploy. Also keep the previous endpoint configuration; it's your rollback target.

#Autoscaling

Autoscaling uses Application Auto Scaling. The resource ID format is endpoint/<endpoint>/variant/<variant>.

RID=endpoint/$EP/variant/current

aws application-autoscaling register-scalable-target --service-namespace sagemaker \
  --resource-id $RID --scalable-dimension sagemaker:variant:DesiredInstanceCount \
  --min-capacity 2 --max-capacity 6

aws application-autoscaling put-scaling-policy --service-namespace sagemaker \
  --resource-id $RID --scalable-dimension sagemaker:variant:DesiredInstanceCount \
  --policy-name invocations-per-instance --policy-type TargetTrackingScaling \
  --target-tracking-scaling-policy-configuration '{
    "TargetValue": 20,
    "PredefinedMetricSpecification": {"PredefinedMetricType": "SageMakerVariantInvocationsPerInstance"},
    "ScaleInCooldown": 600, "ScaleOutCooldown": 120 }'

# What's configured and what it did recently
aws application-autoscaling describe-scalable-targets --service-namespace sagemaker --resource-ids $RID
aws application-autoscaling describe-scaling-activities --service-namespace sagemaker --resource-id $RID \
  --query 'ScalingActivities[:10].[StartTime,StatusCode,Description]' --output table

SageMakerVariantInvocationsPerInstance counts invocations per minute per instance. For LLMs, requests vary a lot in length, so pick the target from a load test, not from a guess.

Caution

Deregister the scalable target before you change instance counts by hand, or autoscaling will reset them. Before deleting an endpoint, remember that delete-endpoint removes the serving endpoint at once; there's no soft delete. Confirm the name and Region first.

#Logs

Container stdout and stderr go to CloudWatch Logs under /aws/sagemaker/Endpoints/<endpoint-name>, one stream per variant instance.

aws logs describe-log-streams --log-group-name /aws/sagemaker/Endpoints/$EP \
  --order-by LastEventTime --descending --max-items 5 --query 'logStreams[].logStreamName'

aws logs tail /aws/sagemaker/Endpoints/$EP --since 30m --follow
aws logs tail /aws/sagemaker/Endpoints/$EP --since 2h --filter-pattern '?ERROR ?Traceback ?OutOfMemory'

#Metrics

Invocation metrics are in the AWS/SageMaker namespace with dimensions EndpointName and VariantName. Instance metrics (CPU, memory, GPU, disk) are in /aws/sagemaker/Endpoints.

aws cloudwatch get-metric-statistics --namespace AWS/SageMaker --metric-name ModelLatency \
  --dimensions Name=EndpointName,Value=$EP Name=VariantName,Value=current \
  --start-time $(date -u -d '-1 hour' +%FT%TZ) --end-time $(date -u +%FT%TZ) \
  --period 300 --statistics Average Maximum

aws cloudwatch get-metric-statistics --namespace /aws/sagemaker/Endpoints --metric-name GPUMemoryUtilization \
  --dimensions Name=EndpointName,Value=$EP Name=VariantName,Value=current \
  --start-time $(date -u -d '-1 hour' +%FT%TZ) --end-time $(date -u +%FT%TZ) \
  --period 300 --statistics Maximum

#Prevent it next time

• Alarm on Invocation5XXErrors, on p99 ModelLatency and on GPUMemoryUtilization for every production variant.
• Keep min-capacity at 2 or more for production, so one instance replacement doesn't take the endpoint down.
• Version endpoint configurations by date and keep the last good one for rollback.
• Check the instance quota for endpoint usage before a launch or a scale-out event, not during it.