Placement groups are a trade between two things EC2 normally balances for you: physical proximity, which gives you bandwidth and latency, and physical separation, which gives you failure isolation. Choose the wrong strategy and you get one of two outcomes. Launches fail with InsufficientInstanceCapacity at the worst moment, or a single rack failure takes out every replica of a stateful service. This post covers picking the strategy, building it with the CLI, and running it day to day.
Pick the strategy by failure mode, not by name
| Strategy | What EC2 does | Gains | Costs / limits |
|---|---|---|---|
| cluster | Packs instances close together in one AZ | Lowest latency, highest per-flow bandwidth, needed for EFA/HPC tight coupling | Single AZ; launches are all-or-nothing and more likely to hit capacity errors; correlated hardware failure |
| spread (rack level) | Each instance on distinct hardware racks | Strong isolation for a few critical instances | Max 7 running instances per AZ per group |
| partition | Divides the group into logical partitions; no two partitions share racks | Rack-aware isolation at scale; topology visible to the app | Up to 7 partitions per AZ; instances within one partition can share racks |
Rules of thumb:
- Tightly coupled compute (MPI, distributed training, low-latency trading): cluster. Accept the single-AZ risk and design the job to checkpoint.
- Small sets of irreplaceable nodes (three ZooKeeper or etcd members, a primary and standby pair in one AZ): spread.
- Large replicated storage (Kafka, HDFS, Cassandra, ClickHouse): partition, and map partitions to the application's rack awareness.
- Stateless web tiers: usually none. Spreading across AZs with an Auto Scaling group already gives you better isolation than any placement group.
Tip
Placement groups don't replace multi-AZ design. Spread and partition protect against rack-level failure inside one AZ. Use one group per AZ, or one multi-AZ spread/partition group, and still spread replicas across AZs.
Create the groups
aws ec2 create-placement-group --group-name hpc-a --strategy cluster \
--tag-specifications 'ResourceType=placement-group,Tags=[{Key=owner,Value=hpc}]'
aws ec2 create-placement-group --group-name zk-spread --strategy spread --spread-level rack
aws ec2 create-placement-group --group-name kafka-part --strategy partition --partition-count 3
aws ec2 describe-placement-groups \
--query 'PlacementGroups[].{name:GroupName,strategy:Strategy,partitions:PartitionCount,state:State,arn:GroupArn}' \
--output table
Launch into a specific partition when you want deterministic layout, for example broker N in partition N:
aws ec2 run-instances --launch-template LaunchTemplateName=kafka-broker,Version='$Latest' \
--subnet-id subnet-0aaa1111 \
--placement 'GroupName=kafka-part,PartitionNumber=1' \
--count 1
Leave PartitionNumber out and EC2 spreads instances evenly across partitions.
Make the application rack-aware
Partition placement only pays off if replicas actually land in different partitions. Each instance can read its partition from instance metadata:
TOKEN=$(curl -sX PUT http://169.254.169.254/latest/api/token -H 'X-aws-ec2-metadata-token-ttl-seconds: 300')
AZ=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/placement/availability-zone)
PART=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/placement/partition-number)
echo "broker.rack=${AZ}-p${PART}" >> /etc/kafka/server.properties
Kafka then places replicas of each topic partition on brokers with different broker.rack values. HDFS and Cassandra have equivalent topology scripts and snitches. From outside, check the layout for the whole group:
aws ec2 describe-instances \
--filters Name=placement-group-name,Values=kafka-part Name=instance-state-name,Values=running \
--query 'Reservations[].Instances[].{id:InstanceId,az:Placement.AvailabilityZone,part:Placement.PartitionNumber,name:Tags[?Key==`Name`]|[0].Value}' \
--output table
If two brokers that hold replicas of the same data share a partition, your rack awareness is wrong and one rack failure can cost you both.
Move existing instances in or out
You can only change placement on a stopped instance.
aws ec2 stop-instances --instance-ids i-0abc123
aws ec2 wait instance-stopped --instance-ids i-0abc123
aws ec2 modify-instance-placement --instance-id i-0abc123 --group-name zk-spread
aws ec2 start-instances --instance-ids i-0abc123
To remove an instance from a group, use --group-name "" while it's stopped. The start can fail with InsufficientInstanceCapacity if the group's constraints can't be met. Plan this as a maintenance action, one member at a time, with quorum checked between each.
Warning
Stopping an instance loses instance store data and, without an Elastic IP, its public IPv4 address. For stateful members (ZooKeeper, etcd, brokers), confirm the rest of the cluster is healthy and the member is out of rotation before you stop it.
Cluster placement groups and capacity errors
Cluster groups fail differently from normal launches. EC2 must find enough capacity in one tightly packed location, so:
- Launch the whole fleet in one
run-instancescall, using a single instance type. Adding instances to a running cluster group later is more likely to fail. - If a launch fails, stop and start all instances in the group (or terminate and relaunch together). That gives EC2 a chance to re-place them on hardware that fits the full count.
- Reserve the capacity when the job is scheduled. On-Demand Capacity Reservations can be created in a cluster placement group:
aws ec2 create-capacity-reservation \
--instance-type c6in.16xlarge --instance-platform Linux/UNIX \
--availability-zone us-east-1a --instance-count 16 \
--placement-group-arn arn:aws:ec2:us-east-1:111122223333:placement-group/hpc-a \
--instance-match-criteria targeted --end-date-type limited --end-date 2026-11-20T00:00:00Z
Check which instance types support cluster groups and reservations in placement groups in the current documentation. Older generations and some burstable types don't.
Auto Scaling groups in placement groups
An ASG can target a placement group (--placement-group), but combine that carefully:
- A cluster group forces the ASG into one AZ and one subnet. You lose AZ rebalancing, and scale-out events can fail on capacity. Use it only for batch fleets that can tolerate a failed scale-out.
- A spread group caps the ASG at 7 running instances per AZ. Scale-out beyond that fails. Set
MaxSizeto match. - A partition group works well with ASGs, but the ASG can't choose a partition, so you get an even spread rather than deterministic placement.
aws autoscaling describe-scaling-activities --auto-scaling-group-name batch-cluster --max-items 5 \
--query 'Activities[].{status:StatusCode,msg:StatusMessage}' --output table
Networking notes for cluster groups
Instances in a cluster group get higher single-flow bandwidth than flows that leave the group. Outside a cluster placement group, a single TCP flow is capped well below the instance's aggregate bandwidth. Benchmarks taken inside the group don't predict cross-AZ or cross-group throughput. To use the bandwidth:
- Choose network-optimised instance types, and EFA-capable types for MPI or NCCL.
- Use jumbo frames (MTU 9001) inside the VPC. Traffic leaving the VPC falls back to 1500.
- Measure with several parallel streams (
iperf3 -P 8) before blaming the placement.
Prevent it next time
- Record the placement strategy and partition-to-rack mapping in the service's runbook, and check the
describe-instanceslayout after every replacement. - Put capacity reservations behind any cluster group that runs scheduled or deadline-bound work.
- Alarm on ASG scaling activities with
Failedstatus for groups that use placement groups. - Re-evaluate spread groups as clusters grow. When you need an eighth member per AZ, move to a partition group rather than dropping isolation.
- Don't leave empty cluster groups around in IaC. They're free, but stale names confuse the next person designing capacity.