EC2 clock drift: Kerberos, SigV4 and TLS errors fixed with Amazon Time Sync Service

#Symptoms

Clock drift rarely shows up as "the clock is wrong". It shows up as authentication and certificate errors on one instance or a group of them:

If only one instance or one subnet is affected, it is almost always that instance's time configuration, not AWS.

#How time works on EC2

Every EC2 instance can reach the Amazon Time Sync Service at the link-local address 169.254.169.123 (IPv4), and Nitro instances can also use fd00:ec2::123 (IPv6). It needs no internet access, NAT gateway or VPC endpoint. Recent Amazon Linux and AWS Windows AMIs are preconfigured to use it.

Drift happens when that default was replaced:

• The NTP config points to a public pool (pool.ntp.org, time.windows.com) and the instance is in a private subnet with no egress.
• A hardened image or config management overwrote chrony.conf.
• Two time daemons fight (chronyd plus ntpd or systemd-timesyncd).
• A host firewall blocks outbound UDP 123.
• Domain-joined Windows servers follow a domain controller that is itself drifting.

#Check 1: measure the offset on Linux

timedatectl
chronyc tracking
chronyc sources -v

What to read in chronyc tracking:

• Reference ID should show A9FEA97B (169.254.169.123), or the IPv6 address.
• System time is the current offset. A few microseconds to milliseconds is normal; seconds or more is your problem.
• Leap status : Not synchronised means chrony has no usable source.

In chronyc sources, the line marked ^* is the selected source. ^? on every line means no source is reachable.

Confirm only one daemon runs:

systemctl is-active chronyd chrony ntpd systemd-timesyncd 2>/dev/null

Service names vary by distribution (chronyd on Amazon Linux and RHEL, chrony on Ubuntu).

#Check 2: measure the offset on Windows

w32tm /query /status
w32tm /query /source
w32tm /query /configuration
w32tm /stripchart /computer:169.254.169.123 /samples:5 /dataonly

/query /source tells you where the clock comes from:

• 169.254.169.123,0x9 or similar: Amazon Time Sync, as expected.
• A domain controller name: domain hierarchy (Type: NT5DS). The fix is on the DC that holds the PDC emulator role.
• Local CMOS Clock or Free-running System Clock: no source at all.

/stripchart shows the live offset against Amazon Time Sync, which proves the service is reachable from this instance.

#Check 3: find every drifting instance at once

Use Run Command instead of logging in one by one. For Linux:

aws ssm send-command \
  --document-name "AWS-RunShellScript" \
  --targets "Key=tag:Environment,Values=prod" \
  --parameters 'commands=["chronyc tracking | grep -E \"Reference ID|System time|Leap status\""]' \
  --max-concurrency "20%" \
  --comment "time drift check"

For Windows, use AWS-RunPowerShellScript with w32tm /query /status. Read results:

aws ssm list-command-invocations --command-id <command-id> --details \
  --query "CommandInvocations[].{Id:InstanceId,Out:CommandPlugins[0].Output}" --output text

#Fix on Linux (chrony)

Add Amazon Time Sync as the preferred source. The file is /etc/chrony.conf on Amazon Linux and RHEL, /etc/chrony/chrony.conf on Ubuntu (Ubuntu also reads /etc/chrony/sources.d/).

server 169.254.169.123 prefer iburst minpoll 4 maxpoll 4
makestep 1.0 3

Comment out unreachable pool or server lines. Then:

sudo systemctl disable --now ntpd systemd-timesyncd 2>/dev/null
sudo systemctl restart chronyd        # "chrony" on Ubuntu
sudo chronyc makestep
chronyc tracking

makestep 1.0 3 lets chrony step the clock if it is more than one second off during the first three updates after start. Without it, chrony slews a large offset slowly, which can take hours. chronyc makestep forces an immediate step.

Warning

Stepping the clock backwards can confuse databases, schedulers and anything that compares timestamps. On database hosts, step during a maintenance window or stop the service first.

If the instance has a host firewall, allow outbound UDP 123 to 169.254.169.123:

sudo iptables -C OUTPUT -p udp -d 169.254.169.123 --dport 123 -j ACCEPT 2>/dev/null \
  || sudo iptables -I OUTPUT -p udp -d 169.254.169.123 --dport 123 -j ACCEPT

Security groups and network ACLs do not filter traffic to link-local services like this one, so do not spend time there.

#Fix on Windows (w32time)

Standalone or workgroup server:

w32tm /config /manualpeerlist:"169.254.169.123,0x9" /syncfromflags:manual /reliable:no /update
Restart-Service w32time
w32tm /resync /rediscover
w32tm /query /status

Domain-joined members should keep syncing from the domain (NT5DS). Fix the forest root PDC emulator instead, so every DC and member inherits correct time:

# Run on the forest root PDC emulator only
(Get-ADDomain).PDCEmulator
w32tm /config /manualpeerlist:"169.254.169.123,0x9" /syncfromflags:manual /reliable:yes /update
Restart-Service w32time
w32tm /resync /rediscover

If a member was manually pointed somewhere else, put it back on the domain hierarchy:

w32tm /config /syncfromflags:domhier /update
Restart-Service w32time

Caution

If the PDC emulator itself is off by more than the Kerberos tolerance, members may fail to authenticate until it is corrected. Fix the PDC emulator first, then run w32tm /resync on the affected members.

#Confirm the errors are gone

aws sts get-caller-identity      # SigV4 errors should stop
date -u; curl -sI https://aws.amazon.com | grep -i '^date'

Compare the two timestamps; they should match to the second. On Windows, klist purge and a fresh logon confirm Kerberos.

#Prevent it next time

• Bake the chrony or w32time settings above into your golden AMI and config management, and test that a hardening baseline does not overwrite them.
• Publish the offset as a custom metric and alarm on it. A cron job is enough:

OFFSET=$(chronyc tracking | awk '/System time/ {print $4}')
aws cloudwatch put-metric-data --namespace "Custom/Time" \
  --metric-name ClockOffsetSeconds \
  --dimensions InstanceId=$(ec2-metadata -i | cut -d' ' -f2) \
  --value "$OFFSET"

ec2-metadata ships with Amazon Linux; on other distributions read the instance ID from IMDSv2.

• Keep a scheduled Run Command or State Manager association that runs the fleet check and flags any instance not using 169.254.169.123.
• For workloads that need tight bounds (trading, distributed databases), look at the PTP hardware clock on supported instance types and AWS ClockBound; check the EC2 documentation for which instance types support it.