Home / OCI

OCITroubleshootingBeginner

OCI instance clock drift and time sync failures - fixing chrony and Windows Time with the OCI NTP service

Clock skew on an OCI instance breaks API signing, Kerberos and TLS. Check chrony or w32time, point them at the OCI NTP server 169.254.169.254 and keep it that way.

Deepak Bhardwaj6 min read

Applies toOCI Compute, Oracle Linux 7/8/9, other Linux images using chrony, Windows Server on OCI, OCI CLI 3.x

Symptom

One of these shows up, usually on a single instance:

  • OCI CLI or SDK calls from the instance fail with NotAuthenticated and a message about the date or clock skew.
  • Windows: domain logons fail, KRB_AP_ERR_SKEW, or "The time difference between client and server is too large".
  • TLS errors such as "certificate is not yet valid" for a certificate that was issued minutes ago.
  • Log timestamps don't line up across servers, or a cluster (Oracle Clusterware, Kafka, etcd) complains about time.

Most of these break when skew passes a few minutes. Kerberos allows five minutes by default, and OCI request signing also rejects requests whose Date header is too far from the service clock.

How time works on OCI

OCI runs an NTP service on the link-local address 169.254.169.254 in every region, backed by stratum 1 sources. It doesn't need internet access, a NAT gateway or a service gateway.

Oracle-provided Oracle Linux 7, 8 and 9 images ship with chrony already configured for it, and Oracle Windows images are configured for it too. Drift usually means something changed that setup: a custom or imported image, a hardening script, a domain join, or a firewall rule.

Cause Typical sign
Imported or custom image points at public pool servers, instance has no internet egress chronyc sources shows only ? or ^? entries
chronyd / w32time stopped or disabled timedatectl shows System clock synchronized: no
Host firewall blocks outbound UDP 123 Sources unreachable, Reach stays at 0
Large offset after a long stop or restore, chrony only slewing Offset shrinks very slowly
Windows domain member syncing from a DC that drifts Every domain member is off by the same amount
Two time daemons running (ntpd and chronyd) Sync flaps, logs show both

Check it: Linux

timedatectl
# System clock synchronized: yes   NTP service: active

chronyc tracking
# Reference ID, Stratum, System time offset, Leap status

chronyc sources -v

A healthy source line starts with ^* (current sync source) and shows a non-zero Reach value (377 means the last eight polls all worked). ^? means the source isn't reachable.

Check the configuration and the service:

grep -E '^(server|pool)' /etc/chrony.conf     # Debian/Ubuntu: /etc/chrony/chrony.conf
systemctl status chronyd                      # Debian/Ubuntu: chrony
systemctl is-active ntpd 2>/dev/null          # should not also be running

Test reachability of the OCI server directly, without changing the clock:

chronyd -Q 'server 169.254.169.254 iburst'

This prints the offset it measured and exits. If it times out, look at the host firewall next.

Fix it: Linux

1. Point chrony at the OCI NTP service

Edit /etc/chrony.conf and make the OCI server the main source. Comment out pool lines that can't be reached from a private subnet:

server 169.254.169.254 iburst
#pool 2.pool.ntp.org iburst
makestep 1.0 3
rtcsync

makestep 1.0 3 lets chrony step the clock (jump instead of slew) if it's off by more than one second during the first three updates after start. That's what you want after a long stop or a restore.

sudo systemctl enable --now chronyd
sudo systemctl restart chronyd
chronyc sources -v

2. Correct a large offset now

If the clock is minutes off and chrony is slewing slowly:

sudo chronyc makestep
chronyc tracking | grep 'System time'

Warning

Stepping the clock moves time instantly, possibly backwards. Databases, schedulers and anything that writes time-ordered records can misbehave. On a database server or cluster node, stop the application or fail it over first, then step.

3. Open the host firewall if needed

Chrony needs outbound UDP 123 to 169.254.169.254 and the replies back. Oracle Linux firewalld allows outbound traffic by default. If a hardening script added output rules:

sudo iptables -S OUTPUT | head
sudo iptables -L -n -v | grep -E '169.254|123'

On Ubuntu platform images, OCI ships iptables rules (an InstanceServices chain) that allow traffic to 169.254.0.0/16 services. Don't flush iptables or enable ufw with a default-deny policy without keeping those rules; you can lose more than NTP, including iSCSI boot volume access.

4. Remove the second daemon

If both ntpd and chronyd are installed, keep one. On Oracle Linux 8 and 9 that's chrony:

sudo systemctl disable --now ntpd

Check and fix: Windows

w32tm /query /status
w32tm /query /source
w32tm /query /peers
w32tm /stripchart /computer:169.254.169.254 /samples:5 /dataonly

/query /source returning Local CMOS Clock or Free-running System Clock means Windows isn't syncing at all. The stripchart shows the offset to the OCI server and proves UDP 123 works.

Standalone (workgroup) server

Point Windows Time at the OCI NTP service, restart it and resync:

w32tm /config /manualpeerlist:"169.254.169.254,0x9" /syncfromflags:manual /update
Restart-Service w32time
w32tm /resync /force
w32tm /query /source

Oracle's documentation sets the same values through the registry (NtpServer = 169.254.169.254,0x9, Type = NTP under HKLM:\System\CurrentControlSet\Services\W32Time\Parameters). Either method ends in the same configuration.

Domain-joined servers

Members should sync from the domain hierarchy, not from 169.254.169.254 directly:

w32tm /config /syncfromflags:domhier /update
Restart-Service w32time
w32tm /resync /force

Configure the PDC emulator (netdom query fsmo shows which DC holds it) as the only server that uses an external source. If the PDC emulator runs on OCI, give it the manual peer list above plus /reliable:yes. If it runs on premises, make sure it has a working source there; every domain member on OCI will follow its clock.

Tip

If Group Policy sets the time configuration, local w32tm /config changes are overwritten at the next refresh. Check with gpresult /h report.html and change the GPO instead.

Confirm the fix

  • Linux: chronyc tracking shows an offset in milliseconds or less and Leap status: Normal.
  • Windows: w32tm /query /status shows a recent Last Successful Sync Time and the right source.
  • The original failure goes away. For OCI CLI errors, a quick test is:
oci os ns get --auth instance_principal   # or your usual auth method

Prevent it next time

  • Bake the NTP setting into custom images. Before you capture or import an image, set server 169.254.169.254 iburst (or the Windows peer list) and confirm it syncs.
  • Use cloud-init for imported images to write the chrony configuration on first boot.
  • Monitor offset. Ship chronyc tracking output or the Windows Time event log to OCI Logging and alert when the offset passes a threshold you choose (for example 1 second).
  • Keep hardening scripts honest. Add an exception for UDP 123 to 169.254.169.254 in any CIS or in-house firewall baseline.
  • Fix the domain hierarchy once. A correct PDC emulator source fixes every Windows server in the domain at the same time.

Related fixes