EC2 instance won't launch: capacity, quota and RunInstances errors decoded

#Symptom

You run aws ec2 run-instances, a Terraform apply or an Auto Scaling scale-out, and nothing comes up. You might see one of three things:

1. The API call fails at once with an error code.
2. The instance appears, sits in pending and then goes to terminated.
3. An Auto Scaling group stays below desired capacity with no obvious error.

Each one has a different set of causes. Find the exact error text first, then use the table below.

#Step 1: Get the real error

For a direct API call, the error code is in the CLI output. For instances that disappeared, read the state reason:

aws ec2 describe-instances --instance-ids i-0abc123 \
  --query "Reservations[].Instances[].{state:State.Name,code:StateReason.Code,msg:StateReason.Message}" \
  --output table

Terminated instances stay visible for about an hour, so check quickly.

For Auto Scaling, the error is in the scaling activities:

aws autoscaling describe-scaling-activities \
  --auto-scaling-group-name web-asg --max-items 5 \
  --query "Activities[].{time:StartTime,status:StatusCode,cause:StatusMessage}" --output table

If the call came from a pipeline and you only have the message, search CloudTrail for the failed RunInstances call:

aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=RunInstances \
  --max-results 20 --output json \
  | jq -r '.Events[].CloudTrailEvent | fromjson | select(.errorCode) | [.eventTime, .errorCode, .errorMessage] | @tsv'

#Step 2: Match the error to a fix

Tip

--dry-run only checks whether you have permission to make the request. It doesn't check capacity, quotas or subnet addresses, so a successful dry run doesn't mean the launch will work.

#Step 3: Quota errors (VcpuLimitExceeded)

On-Demand quotas are counted in vCPUs per instance family group (Standard, G and VT, P, X, and so on), not in number of instances. Find the quota that applies:

aws service-quotas list-service-quotas --service-code ec2 \
  --query "Quotas[?contains(QuotaName,'On-Demand') || contains(QuotaName,'Spot')].[QuotaName,QuotaCode,Value]" \
  --output table

Count how many vCPUs are running now in that family group. The Service Quotas console shows utilization for most EC2 quotas. Then request more:

aws service-quotas request-service-quota-increase \
  --service-code ec2 --quota-code L-1216C47A --desired-value 256

aws service-quotas list-requested-service-quota-change-history \
  --service-code ec2 --query "RequestedQuotas[].{quota:QuotaName,status:Status,value:DesiredValue}" --output table

L-1216C47A is usually the "Running On-Demand Standard instances" quota. Confirm the code against the list output before you submit. Quotas apply per region, so raise them in your DR region too.

#Step 4: Capacity errors (InsufficientInstanceCapacity)

This error is about AWS, not your account. You can't fix it with a quota request. Options, from quickest to most durable:

1. Retry or change AZ. Leaving the AZ out lets EC2 choose one, if your subnet design allows it.
2. Use a different instance type. Check which types each AZ offers:

aws ec2 describe-instance-type-offerings --location-type availability-zone \
  --filters Name=instance-type,Values=m6i.xlarge,m7i.xlarge,m6a.xlarge \
  --query "InstanceTypeOfferings[].[InstanceType,Location]" --output table

3. Give Auto Scaling several instance types and AZs. Use a mixed instances policy so the group has a fallback when one pool is short:

{
  "LaunchTemplate": {
    "LaunchTemplateSpecification": { "LaunchTemplateName": "web-lt", "Version": "$Latest" },
    "Overrides": [
      { "InstanceType": "m7i.xlarge" },
      { "InstanceType": "m6i.xlarge" },
      { "InstanceType": "m6a.xlarge" }
    ]
  },
  "InstancesDistribution": { "OnDemandAllocationStrategy": "prioritized" }
}

aws autoscaling update-auto-scaling-group --auto-scaling-group-name web-asg \
  --mixed-instances-policy file://mixed.json \
  --vpc-zone-identifier "subnet-aaa,subnet-bbb,subnet-ccc"

4. Reserve capacity for launches that must succeed, such as DR failover or a big release, with an On-Demand Capacity Reservation:

aws ec2 create-capacity-reservation --instance-type m6i.xlarge \
  --instance-platform Linux/UNIX --availability-zone eu-west-1a \
  --instance-count 4 --end-date-type unlimited

Warning

You pay for a Capacity Reservation whether or not instances run in it. Use a set end date for one-off events, and cancel reservations you no longer need.

#Step 5: Subnet out of IP addresses

aws ec2 describe-subnets --subnet-ids subnet-aaa subnet-bbb \
  --query "Subnets[].{id:SubnetId,az:AvailabilityZone,cidr:CidrBlock,free:AvailableIpAddressCount}" --output table

Common culprits are EKS pods using VPC CNI IPs, Lambda functions in a VPC, and network interfaces left behind (status=available):

aws ec2 describe-network-interfaces \
  --filters Name=subnet-id,Values=subnet-aaa Name=status,Values=available \
  --query "NetworkInterfaces[].{eni:NetworkInterfaceId,desc:Description}" --output table

Caution

Check the description before you delete an unattached ENI. Some belong to managed services that reattach them, and deleting them can break those services. For lasting headroom, add subnets or a secondary CIDR to the VPC.

#Step 6: Permission and encryption failures

UnauthorizedOperation messages come with an encoded blob. Decode it to see the denied action and resource:

aws sts decode-authorization-message --encoded-message "<blob>" \
  --query DecodedMessage --output text | jq .

The caller needs sts:DecodeAuthorizationMessage. Look for missing iam:PassRole on the instance profile role. That's a common one.

If an instance goes from pending to terminated with Client.InternalError and the volumes are encrypted with a customer managed key, the principal launching it needs kms:CreateGrant, kms:Decrypt and kms:GenerateDataKey* on the key. For Auto Scaling, that principal is the AWSServiceRoleForAutoScaling service-linked role. Check that the key is enabled:

aws kms describe-key --key-id <key-arn> --query "KeyMetadata.{state:KeyState,enabled:Enabled}"

#Prevent it next time

• Set CloudWatch alarms on quota utilization in Service Quotas for the EC2 vCPU quotas, at around 80%.
• Raise quotas in your DR region ahead of time, to match production.
• Give every Auto Scaling group at least two instance types and subnets in at least two AZs.
• Use Capacity Reservations for launches that can't fail, like DR or planned peaks, and test them during DR drills.
• Size subnets for EKS, Lambda and ENI growth, not just today's instance count.
• Watch for RunInstances errors in CloudTrail with an EventBridge rule, so a failing scale-out reaches someone before the next traffic peak.