ECS Exec cheat sheet: shell into Fargate and EC2 tasks and fix execute-command errors

#What you need

ECS Exec lets you run a shell or a one-off command inside a running container without SSH or a bastion. It uses the SSM agent that ECS injects into the task, so it needs four things to line up:

1. The Session Manager plugin on your machine, alongside AWS CLI v2.
2. The task started with execute command enabled. You can't turn it on for a task that is already running.
3. A task role (not the execution role) with the ssmmessages permissions.
4. A network path from the task to the ssmmessages endpoint: NAT, public IP, or a VPC interface endpoint.

Check the plugin first:

aws --version
session-manager-plugin --version

#Enable it

#On a service

aws ecs update-service \
  --cluster prod \
  --service orders-api \
  --enable-execute-command \
  --force-new-deployment

--force-new-deployment replaces the running tasks, because only new tasks pick up the setting. Do this in a quiet period for services without spare capacity.

#On a standalone task

aws ecs run-task \
  --cluster prod \
  --task-definition debug-tools:3 \
  --launch-type FARGATE \
  --enable-execute-command \
  --network-configuration "awsvpcConfiguration={subnets=[subnet-0abc],securityGroups=[sg-0abc],assignPublicIp=DISABLED}"

#Task role policy

Attach this to the role in the task definition's taskRoleArn:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ssmmessages:CreateControlChannel",
        "ssmmessages:CreateDataChannel",
        "ssmmessages:OpenControlChannel",
        "ssmmessages:OpenDataChannel"
      ],
      "Resource": "*"
    }
  ]
}

If the task definition has no task role, add one and register a new revision. The execution role does not count.

#Connect

# Find a running task in the service
TASK=$(aws ecs list-tasks --cluster prod --service-name orders-api \
  --desired-status RUNNING --query 'taskArns[0]' --output text)

# Open a shell
aws ecs execute-command \
  --cluster prod \
  --task "$TASK" \
  --container app \
  --interactive \
  --command "/bin/sh"

# Run one command and exit
aws ecs execute-command --cluster prod --task "$TASK" --container app \
  --interactive --command "cat /etc/os-release"

--interactive is required even for a single command. --container is needed when the task has more than one container.

Warning

You are in a production container, usually as root. Anything you change there (files, processes, config) affects live traffic and is lost when the task is replaced. Read, don't fix in place; ship the fix through a new image or task definition.

#Check a task before you connect

aws ecs describe-tasks --cluster prod --tasks "$TASK" \
  --query "tasks[0].{exec:enableExecuteCommand,platform:platformVersion,containers:containers[].{name:name,agent:managedAgents[?name=='ExecuteCommandAgent'].lastStatus|[0]}}"

You want exec: true and the ExecuteCommandAgent in RUNNING for the container you target. PENDING right after start is normal for a short time; STOPPED means the agent couldn't start or connect.

AWS publishes a checker script that tests most prerequisites in one go (needs jq):

bash <(curl -Ls https://raw.githubusercontent.com/aws-containers/amazon-ecs-exec-checker/main/check-ecs-exec.sh) prod "$TASK"

Read the script before running it in a restricted environment.

#Error → cause → fix

Wording varies by CLI version; match on the key phrase.

#Caller permissions

The person running the command needs ecs:ExecuteCommand and ecs:DescribeTasks. You can restrict which container they can enter:

{
  "Effect": "Allow",
  "Action": "ecs:ExecuteCommand",
  "Resource": [
    "arn:aws:ecs:eu-west-1:111122223333:cluster/prod",
    "arn:aws:ecs:eu-west-1:111122223333:task/prod/*"
  ],
  "Condition": { "StringEquals": { "ecs:container-name": "app" } }
}

#Private subnets without NAT

Create an interface endpoint for ssmmessages in the task VPC and allow HTTPS (443) from the task security group to the endpoint security group:

aws ec2 create-vpc-endpoint \
  --vpc-id vpc-0abc \
  --vpc-endpoint-type Interface \
  --service-name com.amazonaws.eu-west-1.ssmmessages \
  --subnet-ids subnet-0abc subnet-0def \
  --security-group-ids sg-0endpoint \
  --private-dns-enabled

If you encrypt sessions with a customer managed KMS key, the task role also needs kms:Decrypt on that key, and private subnets need a kms endpoint too. Logging to CloudWatch Logs or S3 needs those endpoints (or NAT) as well.

#Read-only root filesystem

ECS Exec needs readonlyRootFilesystem set to false because the agent writes files inside the container. If your security baseline requires read-only roots, keep ECS Exec off for those task definitions and use a separate debug task definition when you need a shell.

#Audit logging

Record who ran what by configuring logging on the cluster:

aws ecs update-cluster --cluster prod \
  --configuration '{"executeCommandConfiguration":{"logging":"OVERRIDE","logConfiguration":{"cloudWatchLogGroupName":"/ecs/exec-audit","cloudWatchEncryptionEnabled":true}}}'

The task role then needs logs:CreateLogStream, logs:DescribeLogStreams, logs:DescribeLogGroups and logs:PutLogEvents on that log group (and s3:PutObject plus s3:GetEncryptionConfiguration if you log to S3). Session output is only captured when the container image includes the script and cat utilities. Every call to ExecuteCommand is also recorded in CloudTrail, with the user identity and container name.

Tip

Find recent sessions quickly: aws cloudtrail lookup-events --lookup-attributes AttributeKey=EventName,AttributeValue=ExecuteCommand --max-results 20.

#Prevent it next time

• Put enableExecuteCommand and the task role policy in your IaC module, so new services are debuggable from day one.
• Add the ssmmessages endpoint to the standard VPC build for private workloads.
• Limit ecs:ExecuteCommand to on-call roles and to named containers with the ecs:container-name condition.
• Turn on cluster audit logging before an incident, not during one.
• Run the checker script as part of your service onboarding so problems show up before you need a shell at 3 a.m.