Home / AWS

AWSTroubleshootingIntermediate

Sev1: Session Manager and Run Command stopped working across the fleet — SSM agent, instance profile and endpoints

Instances show as offline in Systems Manager, so no shell access and no patching during an incident. Fix the instance profile, network path to SSM endpoints, the agent and Default Host Management.

Deepak Bhardwaj3 min read

Applies toAWS Systems Manager (Session Manager, Run Command, Fleet Manager), SSM Agent 3.x, AWS CLI v2

Symptoms

  • aws ssm start-session returns TargetNotConnected.
  • Instances missing from Fleet Manager or showing ConnectionLost.
  • No SSH key access because the fleet relies on Session Manager.

1. Scope the problem

aws ssm describe-instance-information \
  --query "InstanceInformationList[].{id:InstanceId, ping:PingStatus, agent:AgentVersion, last:LastPingDateTime}" \
  --output table
Pattern Likely cause
Every instance in one VPC offline Network path: endpoints, NAT, DNS
Every instance offline in the account IAM: instance profile policy or DHMC role changed
Only new instances missing Launch template lost the instance profile, or AMI without agent
Random instances Agent crashed or disk full

2. IAM: can the instance talk to SSM?

Each instance needs credentials with AmazonSSMManagedInstanceCore permissions, either through its instance profile or through Default Host Management Configuration (DHMC).

aws ec2 describe-instances --instance-ids i-0abc \
  --query "Reservations[].Instances[].IamInstanceProfile.Arn"
aws iam list-attached-role-policies --role-name ec2-app-role

If a policy cleanup detached it:

aws iam attach-role-policy --role-name ec2-app-role \
  --policy-arn arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore

Instances without a profile:

aws ec2 associate-iam-instance-profile --instance-id i-0abc --iam-instance-profile Name=ec2-app-profile

Check DHMC if you rely on it:

aws ssm get-service-setting --setting-id arn:aws:ssm:eu-west-1:111122223333:servicesetting/ssm/managed-instance/default-ec2-instance-management-role

The agent picks up credential changes within minutes; restarting the agent speeds it up.

3. Network: can it reach the endpoints?

The agent needs HTTPS (443) outbound to ssm, ssmmessages and ec2messages endpoints in the region, via NAT or interface VPC endpoints.

aws ec2 describe-vpc-endpoints --filters Name=vpc-id,Values=vpc-0abc \
  --query "VpcEndpoints[].{svc:ServiceName, state:State, dns:PrivateDnsEnabled, sgs:Groups[].GroupId}" --output table
  • Endpoints must have private DNS enabled, and the endpoint security group must allow 443 from the instances.
  • If the VPC uses custom DNS servers, they must forward to the VPC resolver so the private DNS names resolve.
  • Session Manager logging to S3/CloudWatch also needs those endpoints (or NAT), and if KMS encryption is on, kms:Decrypt for the session key.

4. The agent itself

If you still have another way in (EC2 Serial Console, or SSH):

sudo systemctl status amazon-ssm-agent
sudo tail -50 /var/log/amazon/ssm/amazon-ssm-agent.log
sudo systemctl restart amazon-ssm-agent

Windows:

Get-Service AmazonSSMAgent
Get-Content "C:\ProgramData\Amazon\SSM\Logs\amazon-ssm-agent.log" -Tail 50
Restart-Service AmazonSSMAgent

Look for AccessDeniedException (IAM), dial tcp ... timeout (network) and no space left on device.

5. Getting in without SSM

  • EC2 Serial Console (Nitro instances, must be enabled at account level and the OS needs a password user):
aws ec2 enable-serial-console-access
  • EC2 Instance Connect for Linux with a public or endpoint path.
  • Last resort: stop the instance, attach its root volume to a rescue instance and fix the agent config.

Warning

Stopping an instance with instance store volumes loses that data. Check the instance type before stopping.

Prevent it next time

  • Manage SSM IAM through one well-protected role or DHMC, and guard it with SCPs.
  • Deploy SSM interface endpoints in every private VPC through IaC.
  • Alarm on the count of managed instances with PingStatus not Online.
  • Keep the agent updated with a State Manager association.

Related fixes