Home / AWS

AWSRunbookIntermediate

Sev1: locked out of a production EC2 instance — lost key pair, broken sshd or forgotten Windows password

Get back into an EC2 instance when the SSH key is lost, sshd is misconfigured or the Windows Administrator password is unknown. Try Session Manager, EC2 Instance Connect and the serial console first, then the rescue-volume method and SSM automation runbooks.

Deepak Bhardwaj2 min read

Applies toAmazon EC2 Linux and Windows instances, AWS Systems Manager, EC2 Serial Console, AWS CLI v2

Try the no-downtime options first

Option Needs Downtime
SSM Session Manager SSM agent running, instance profile with AmazonSSMManagedInstanceCore, route to SSM endpoints None
EC2 Instance Connect Supported AMI with the package, port 22 reachable from the service range or an endpoint None
EC2 Serial Console Nitro instance, serial access enabled for the account, a local password user None
Rescue volume Instance can be stopped; root volume detachable Yes (stop/start)
AWSSupport-ResetAccess runbook Instance can be stopped; SSM automation Yes

1. Session Manager

aws ssm describe-instance-information --filters Key=InstanceIds,Values=i-0abc123 \
  --query "InstanceInformationList[].{id:InstanceId, ping:PingStatus, agent:AgentVersion}"
aws ssm start-session --target i-0abc123

Once in, fix ~/.ssh/authorized_keys or /etc/ssh/sshd_config, then sudo sshd -t && sudo systemctl restart sshd.

2. EC2 Instance Connect

ssh-keygen -t ed25519 -f /tmp/tempkey -N ""
aws ec2-instance-connect send-ssh-public-key --instance-id i-0abc123 \
  --instance-os-user ec2-user --ssh-public-key file:///tmp/tempkey.pub
ssh -i /tmp/tempkey ec2-user@<ip>      # within 60 seconds

3. Serial console

aws ec2 enable-serial-console-access

Connect from the console (EC2 → instance → Connect → EC2 serial console). You need a user with a password set beforehand; otherwise use the GRUB single-user approach at boot if your distribution allows it.

4. Rescue volume (Linux)

  1. Snapshot the root volume first.
  2. Stop the instance (not terminate). Note the root device name.
  3. Detach the root volume and attach it to a rescue instance in the same AZ as /dev/sdf.
  4. On the rescue instance:
lsblk
sudo mkdir /mnt/rescue && sudo mount -o nouuid /dev/nvme1n1p1 /mnt/rescue    # nouuid for XFS clones
cat ~/.ssh/new_key.pub | sudo tee -a /mnt/rescue/home/ec2-user/.ssh/authorized_keys
sudo chroot /mnt/rescue sshd -t      # validate sshd_config if you edited it
sudo umount /mnt/rescue
  1. Detach, re-attach to the original instance as the root device, start it.

Warning

Stopping an instance loses data on instance store volumes and may change the public IP (unless it is an Elastic IP). Check before you stop.

5. Windows: reset the Administrator password

Run the AWSSupport-ResetAccess automation, which uses EC2Rescue to reset the password and stores the new one encrypted in Parameter Store:

aws ssm start-automation-execution --document-name "AWSSupport-ResetAccess" \
  --parameters "InstanceId=i-0win123,SubnetId=CreateNewVPC"

If SSM agent works, simpler: aws ssm start-session then net user Administrator <new-password>.

Prevent it next time

  • Make Session Manager the standard access path; close port 22/3389 to the internet.
  • Bake the SSM agent and instance profile into every AMI and launch template.
  • Store key pairs and break-glass passwords in Secrets Manager with access logging.

Related fixes