NTP and Time Synchronization
NTP, the Network Time Protocol, keeps the clocks across a fleet of machines within a few milliseconds of true time over the open internet, and within tens of microseconds on a quiet LAN. A client exchanges timestamped packets with one or more servers, measures the round-trip delay and the offset between its clock and theirs, and continuously adjusts its own clock to track them. It sounds like plumbing — and it is, right up until a clock drifts far enough to break something user-facing.
Time is infrastructure. A certificate is valid only between two timestamps; a signed token expires at a fixed instant; a distributed database orders writes by clock; a log line is only useful if its timestamp lines up with everyone else's. Every one of those depends on the assumption that machines agree on what time it is, and a clock that has silently drifted is a corruptor of all of them at once. The failures are intermittent and baffling precisely because nothing logs "the clock is wrong" — the symptom always shows up somewhere else.
The Stratum Hierarchy
NTP organizes time sources into a tree measured in strata. Stratum 0 is a reference clock with no network at all — a GPS receiver, a radio clock locked to an atomic standard, or a caesium oscillator. A stratum 1 server is directly attached to a stratum 0 device and is the authoritative network source. Each server that synchronizes to a stratum n server becomes stratum n+1, up to a maximum of 15; stratum 16 means "unsynchronized." The number is not a quality score so much as a distance-from-truth counter.
The hierarchy exists to spread load and limit blast radius, not to make lower strata "worse." A stratum 3 server a few milliseconds from its upstream is more accurate than a misconfigured stratum 1 on a congested link. The practical shape inside an organization is a handful of internal stratum 2 servers pulling from external stratum 1 pools, with every other host syncing to those internal servers — so thousands of machines do not hammer public time servers directly.
Disciplining the Clock: Slew versus Step
A clock is never simply "set." NTP measures the offset on every exchange and corrects it in one of two ways. Slewing speeds up or slows down the clock slightly so it converges on the right time without ever jumping — the system clock stays monotonic, and no application sees time move backward or skip forward. Stepping jumps the clock directly to the correct value. Stepping is fast but violent: any code that assumed time only moves forward can misbehave when it suddenly does not.
The standard policy is to slew small offsets and step only large ones. The classic ntpd threshold is 128 ms — under it the daemon slews, over it it steps — and a step backward is exactly what breaks timers, lease expiry, and anything measuring elapsed wall-clock time. The cure for a host that keeps needing big steps is not bigger steps; it is finding why it drifts (a missing time source, a throttled VM, a flaky link) so it can be disciplined by gentle slewing.
# chrony's view of how well the clock is disciplined chronyc tracking # Reference ID : C0248F01 (time.example.net) # Stratum : 3 # System time : 0.000041 seconds slow of NTP time # Last offset : -0.000023 seconds # RMS offset : 0.000056 seconds <- sub-millisecond, healthy # Leap status : Normal
Why Skew Breaks TLS, Tokens, and Ordering
A TLS certificate carries a notBefore and a notAfter timestamp, and the client checks the current time against both. If a client's clock is days fast, a perfectly valid server certificate looks expired and the handshake fails with a validity error that has nothing to do with the certificate. If it is days slow, a freshly issued certificate looks "not yet valid." The same window logic governs signed tokens: a JWT or Kerberos ticket with a five-minute expiry on a host whose clock is six minutes off is either rejected on arrival or accepted long after it should have died.
Distributed systems add a second failure mode: ordering. Databases and consensus systems that timestamp events to order them assume clocks agree to within a known bound; when skew exceeds that bound, writes can be ordered wrong or a "happened-before" relationship can invert. And across the whole fleet, log correlation falls apart — if two services disagree by 30 seconds, tracing a request through both becomes guesswork. None of these announce a clock problem; they announce a TLS error, an auth failure, a mis-ordered transaction, or an unfollowable trace.
Modern Practice: chrony, Cloud Time, and PTP
On modern Linux, chrony has largely replaced the reference ntpd. Chrony converges faster after boot, copes far better with intermittent connectivity and laptops that suspend, and disciplines the clock well on virtual machines where the hypervisor steals cycles. Most clouds also publish a managed NTP endpoint on a link-local address — a stratum 1-equivalent source a single hop away — and pointing instances at it removes the need to firewall out to public pools and removes the public servers as a dependency.
When milliseconds are not enough, the protocol changes. PTP, the Precision Time Protocol, reaches sub-microsecond accuracy by timestamping packets in NIC hardware and correcting for switch transit delay, but it needs PTP-aware switches and is a LAN technology, not an internet one. Finance uses it because regulations such as MiFID II demand sub-100-microsecond traceability on trade timestamps; telecom uses it to phase-align radio base stations. For everything else, NTP's milliseconds are plenty, and reaching for PTP without the hardware just adds cost and complexity for accuracy nothing in the application can use.
NTP synchronizes clocks in software over ordinary IP, reaching single-digit milliseconds across the internet and tens of microseconds on a LAN. It needs no special hardware and runs everywhere — the right default for servers, fleets, and anything where millisecond agreement is enough.
PTP reaches sub-microsecond accuracy by timestamping in NIC hardware and accounting for per-switch transit delay, but it requires PTP-aware switches and works only on a controlled LAN. Choose it when regulation or physics demands it — trading-floor timestamps under MiFID II, base-station phase alignment — not as a general upgrade over NTP.
- Firewalling UDP port 123 outbound and forgetting it. The host silently falls back to its free-running hardware clock, drifts seconds per day, and the first symptom is a TLS or auth failure weeks later with no obvious link to time.
- Letting the daemon step the clock backward on a production host. Timers fire wrong, elapsed-time measurements go negative, leases expire early, and any code that assumed time only moves forward misbehaves the instant the clock jumps back.
- Relying on a single time source. If that one server is wrong, every client follows it confidently off a cliff; with at least three or four sources, NTP can detect and reject a "falseticker" by majority.
- Ignoring skew until something breaks intermittently. Clock drift never logs itself — it surfaces as flaky TLS handshakes, tokens rejected as expired, or mis-ordered writes, so the time problem gets diagnosed last.
- Running ntpd on a VM that the hypervisor heavily throttles and expecting clean sync. Stolen CPU cycles make the guest clock jump erratically; chrony tolerates this far better, and a hypervisor or cloud time source removes the problem.
- Configure at least three or four time sources on every host so the daemon can outvote a falseticker, and prefer your cloud's managed NTP endpoint or internal stratum 2 servers over hitting public pools directly.
- Run chrony rather than ntpd on modern Linux and VMs — it converges faster after boot, handles suspend and throttled hypervisors, and slews aggressively enough to avoid steps in normal operation.
- Alert on the measured offset, not just on whether the daemon is running. Page when chronyc tracking shows the system more than, say, 100 ms off, so you catch drift before TLS or tokens do.
- Configure the daemon to slew rather than step once a host is in steady state, reserving stepping for boot, so production never sees the wall clock jump backward under a running workload.
- Open outbound UDP 123 (or whatever your time source uses) explicitly in firewall policy and monitor that the path stays up, so a future rule change cannot silently sever the fleet from its clock.
Knowledge Check
A host's clock is 6 minutes fast. Users on it suddenly cannot establish TLS connections to internal services. What is the most likely cause?
- The clock skew pushes the current time outside the cert's validity window, so a valid cert reads as expired
- The fast clock forces a weaker cipher suite that the destination servers reject during the handshake negotiation
- TLS refuses to even begin whenever the client clock differs from the remote server's clock by any amount at all
- The skew corrupts the server's public key so signature verification fails
Why is slewing the clock preferred over stepping it once a host is running production workloads?
- Slewing keeps the clock monotonic, so no app ever sees time jump backward or skip
- Slewing corrects a large offset much faster than a single abrupt step ever could
- Slewing consumes far less network bandwidth on the time exchanges than stepping does
- Slewing promotes the host to a lower stratum number automatically
What does a host's NTP stratum number actually tell you?
- Its distance in network hops from the stratum 0 reference clock, not a direct measure of accuracy
- Its guaranteed accuracy, where a stratum 1 server is always more precise than any stratum 3
- How frequently the host polls its upstream time servers
- Whether the host chooses to slew or step its clock at the moment it applies a correction to its local time
You got correct