Mustafa ERBAY

Posted on Oct 26 Originally published at mustafaerbay.com.tr

My Clock Advertised 1 ms and Stepped Every 41 Nanoseconds

#linux #kernel #time #vdso

I was about to put timestamps in a measurement loop, and first I asked the obvious question: how fine is this machine's clock? The standard way to ask is clock_getres(). The answer came back: 1 millisecond.

I did not believe it. You cannot measure something in microseconds with a millisecond-resolution clock, and I had been seeing microsecond differences on that machine for years. Both things could not be true.

So I took two million samples. The clock stepped every 41 nanoseconds. The advertised number was off by a factor of twenty-four thousand — and the reason is not a bug. It is that my question was never the one being answered.

Two machines, nine clocks

The measurements ran in two places, because a general rule should not come from one machine's quirk.

Machine A: the linuxkit VM behind Docker Desktop on my Mac — aarch64, kernel 6.10.14, clocksource arch_sys_counter. Machine B: VPS3, which hosts this blog — x86_64, kernel 6.8.0-147, clocksource kvm-clock. Different architecture, different hypervisor, different kernel.

I compiled the same C program on both. It walks nine POSIX clock IDs, prints the resolution clock_getres() advertises for each, then reads the clock two million times and finds the smallest non-zero step it observes. That is the experimental answer to "how fine a difference can this clock actually show me".

First, the cost: seven clocks never enter the kernel

Before resolution there was a simpler question: are these calls system calls? I counted 100,000 of them with strace.

Clock ID	A: syscalls	B: syscalls
`CLOCK_REALTIME`	0	0
`CLOCK_MONOTONIC`	0	0
`CLOCK_BOOTTIME`	0	0
`CLOCK_TAI`	0	0
`CLOCK_REALTIME_COARSE`	0	0
`CLOCK_MONOTONIC_COARSE`	0	0
`CLOCK_MONOTONIC_RAW`	0	0
`CLOCK_PROCESS_CPUTIME_ID`	100,000	100,000
`CLOCK_THREAD_CPUTIME_ID`	100,000	100,000

Zero for seven clocks, one per call for two. That is not a coincidence; it is written in the code. In lib/vdso/gettimeofday.c, __cvdso_clock_gettime_common turns the clock ID into a bitmask and tests it against four masks in turn: VDSO_HRES, VDSO_COARSE, VDSO_RAW, and VDSO_AUX, which is reserved for auxiliary clocks. If none matches, the non-matching branch says return false. The fallback is not in this function but in its caller, __cvdso_clock_gettime_data: seeing that false, it calls clock_gettime_fallback() — a real system call.

The masks are defined in include/vdso/datapage.h: VDSO_HRES holds four clocks (CLOCK_REALTIME, CLOCK_MONOTONIC, CLOCK_BOOTTIME, CLOCK_TAI), VDSO_COARSE two, VDSO_RAW one. That is seven of the nine standard IDs here; VDSO_AUX covers a separate range that none of these nine falls into. The CPU-time clocks are in no mask at all.

The source predicts the measurement exactly: the seven clocks in the masks are answered in userspace, and the two left outside go to the kernel on every call.

One caveat belongs here: being in a mask is not enough. Inside, do_hres() also asks vdso_clocksource_ok(), and if the clocksource is not vDSO-capable (clock_mode == VDSO_CLOCKMODE_NONE) all seven clocks fall back to the system call. On my two machines the clocksources — arch_sys_counter and kvm-clock — are capable; on an older VM stuck on acpi_pm or hpet this table would look entirely different.

The vDSO here is a small page of code the kernel maps into every process's address space. I had already seen it on the map while measuring ASLR's four separate regions; this is where it turns out what it is for.

Then, resolution: what the advertised number describes

This is the table that matters. The left column is what clock_getres() says, the right is what I counted. The "changed" column shows how often the value moved across two million reads — as you will see, that is where the real information sits.

Clock	A: advertised	A: observed step	A: changed	B: advertised	B: observed step	B: changed
`CLOCK_REALTIME`	1,000,000 ns	41 ns	35.8%	1 ns	29 ns	100%
`CLOCK_MONOTONIC`	1,000,000 ns	41 ns	35.3%	1 ns	29 ns	100%
`CLOCK_MONOTONIC_RAW`	1,000,000 ns	41 ns	34.4%	1 ns	30 ns	100%
`CLOCK_BOOTTIME`	1,000,000 ns	41 ns	33.2%	1 ns	29 ns	100%
`CLOCK_TAI`	1,000,000 ns	41 ns	33.2%	1 ns	29 ns	100%
`CLOCK_REALTIME_COARSE`	1,000,000 ns	1,000,000 ns	6 times	1,000,000 ns	999,983 ns	20 times
`CLOCK_MONOTONIC_COARSE`	1,000,000 ns	1,000,000 ns	6 times	1,000,000 ns	999,983 ns	21 times
`CLOCK_PROCESS_CPUTIME_ID`	1 ns	83 ns	100%	1 ns	1,031 ns	100%
`CLOCK_THREAD_CPUTIME_ID`	1 ns	83 ns	100%	1 ns	971 ns	100%

First, the floor of my own method

Before publishing that table I put an objection to my own measurement: if I read the clock back to back, the smallest difference I see may not be the clock's step at all but the time between my two reads. If a call takes 22 nanoseconds, then even on a 1-nanosecond clock I cannot see below 22 nanoseconds.

The objection is fair, and the way to separate the two is the "changed" column. If the clock is slower than my reading rate, some consecutive reads return the same value; if it is faster, every read differs.

On machine A the fine clocks move on only one read in three. Two reads out of three see the same value, so the clock really does advance in steps, and that step is 41 nanoseconds. The measurement holds.
On machine B the same clocks change on every single read. There, 29 nanoseconds is not the clock's step but my method's floor. The correct statement is: B's clock is finer than 29 nanoseconds, and this method cannot say how much finer.
The CPU-time clocks change on every read on both machines, so the 83 and 971 nanoseconds there are floors too, not quantisation.

So do not read B's fine clocks, or either machine's CPU-time clocks, as measurements; they are finer than the method can resolve. The five cells in bold are the real measurement — A's five fine clocks independently land on the same 41 nanoseconds.

The two findings that survive

With the objection accounted for, two solid results remain:

On machine A the advertised number is 24,000 times too large. clock_getres says 1 millisecond, the clock advances every 41 nanoseconds, and this one stands up: the change rate confirms stepped behaviour, and the step itself is exactly one hardware tick, as the next section shows.
The same call answers a million times apart on my two machines. Same clock ID, same API; 1,000,000 ns on A, 1 ns on B. Each is the right number for its machine — and neither belongs to the question I asked.

The only place it comes out right is the _COARSE clocks: both machines say 1 millisecond and both really do step once per millisecond. The changed column shows it too — between 6 and 21 movements across two million reads. The 999,983 nanoseconds on B is not noise either: the _COARSE clocks read the last tick value of the NTP-corrected time base, and the correction measured on that machine is −16.077 ppm. 1,000,000 × (1 − 16.077 × 10⁻⁶) = 999,983.9 — the observed number is within a nanosecond. On machine A there is no correction, and the step is exactly 1,000,000.

Where the 1 millisecond comes from

The answer lives in kernel/time/posix-timers.c, in a three-line function. For the fine clocks, clock_getres() calls posix_get_hrtimer_res, and all that function does is this:

tp->tv_nsec = hrtimer_resolution;

hrtimer_resolution is not the clock's resolution. It is the timer's — the answer to "how precisely can I schedule a wakeup". kernel/time/hrtimer.c initialises it like this:

unsigned int hrtimer_resolution __read_mostly = LOW_RES_NSEC;

and changes it when high-resolution mode comes up:

if (hrtimer_resolution != HIGH_RES_NSEC)
    hrtimer_resolution = HIGH_RES_NSEC;

In the same file, #define HIGH_RES_NSEC 1. So a machine reporting 1 nanosecond has high-resolution timers on, and a machine reporting 1 millisecond does not.

Rather than leave that as an inference, I checked it directly on both:

A: # CONFIG_HIGH_RES_TIMERS is not set     (/proc/config.gz)
B: CONFIG_HIGH_RES_TIMERS=y  ·  .hres_active : 1   (/proc/timer_list)

There is a fine but important distinction here: on A the mode is not "switched off", it is absent from the kernel. In that case include/linux/hrtimer.h does not even define the variable:

#else  /* CONFIG_HIGH_RES_TIMERS */
#define hrtimer_resolution  (unsigned int)LOW_RES_NSEC

So the 1,000,000 that clock_getres returns on machine A is not a value computed at runtime; it is a constant substituted in when that kernel was compiled. No circumstance on that machine will ever produce a different number. Neither fact has anything to do with how finely the clock can be read.

The neat part is that the vDSO's own clock_getres path does the same thing: it returns vd->hrtimer_res for the fine clocks and LOW_RES_NSEC for the coarse ones. Whichever road you ask from, you are told about the timer.

This is not a defect. It is a naming accident: clock_getres reads as "the clock's resolution", and what it hands back is the timer grid's step. They are two separate mechanisms in the same kernel.

So what is that 1 millisecond the right answer to?

Up to here I have been calling the advertised number wrong. More precisely: it is a right answer to a different question. And there is a clean way to test that — if the number really describes the timer, then sleep granularity should match it.

On both machines I asked nanosleep() for 1 µs, 10 µs, 100 µs and 1 ms.

My first attempt was wrong, and it belongs in the article: I repeated each request 21 times and took the shortest sleep. But each iteration itself consumes roughly one tick, so the trials lock to the same phase relative to the tick; those 21 samples are not independent, they are one phase offset measured 21 times. I redid it with 101 trials, spreading the phase by busy-waiting a random interval before each one. Medians and ranges below:

Requested sleep	A: shortest – median – longest	B: shortest – median – longest
1 µs	7,792 – 718,416 – 1,334,292 ns	67,455 – 113,652 – 1,609,303 ns
10 µs	39,000 – 760,750 – 1,352,250 ns	49,471 – 128,900 – 820,507 ns
100 µs	250,375 – 747,250 – 1,486,292 ns	117,909 – 196,465 – 767,116 ns
1 ms	1,295,000 – 1,871,833 – 2,612,833 ns	1,056,896 – 1,154,608 – 2,744,415 ns

On machine A the table is a picture of a grid. All three sub-millisecond requests land in the same place: a median of 0.72–0.76 ms with the spread covering zero to 1.35 ms. What decides the outcome is not what you asked for but where your request fell relative to the tick. The 7.8 µs shortest sleep shows the same thing — that is what you get when the request arrives just before a tick. The grid's step is one millisecond, exactly the number clock_getres reported.

My first measurement had led me to a rule: "ask for 1 ms and you land not on the next tick but the one after it." With the phase spread out, the rule disappeared — a 1 ms request lands anywhere between 1.30 and 2.61 ms. There is a real extra-tick mechanism in the kernel: the relevant line in hrtimer.c says "For relative timers we add hrtimer_resolution (i.e. one jiffy) to prevent short timeouts". But it is conditional on CONFIG_TIME_LOW_RES, which is not defined on either of my machines. So the rule had neither data nor mechanism behind it; what I had seen was the shadow of a single phase offset.

What survives is solid: on machine A, whatever you ask for, you hit a one-millisecond grid. That is precisely what clock_getres's 1 millisecond was telling me, and it was telling the truth. It simply was not the question I had asked.

Machine B has no such grid: the sub-millisecond requests have medians of 113–196 µs, far finer than a tick. The floor there is no longer the timer grid but the cost of waking up and being scheduled. Note that this is not the advertised 1 nanosecond either. That number is the grid's step, not an achievable latency — seeing "1 ns resolution" does not buy you a one-nanosecond sleep.

Two machines, two different ways of being misled, one cause: the question I was asking — how finely is this clock read — has no answer in this API at all.

Where the 41 nanoseconds comes from

The observed step comes from the hardware counter. On aarch64 the generic timer's frequency sits in the CNTFRQ_EL0 register, and on this machine it can be read from userspace. I read it with a three-line program:

uint64_t f;
__asm__ volatile("mrs %0, cntfrq_el0" : "=r"(f));

It reports 24,000,000 Hz. That is 1/24,000,000 of a second per tick, or 41.667 nanoseconds. The smallest step across two million samples came out at 41 nanoseconds — exactly one hardware tick.

I should not fall into my own thesis here: CNTFRQ_EL0 is an advertised number too. Firmware fills that register, not hardware, and the kernel's device-tree binding keeps this as a known problem — the clock-frequency property exists "to work around broken firmware which does not configure CNTFRQ on all CPUs to a uniform correct value". The difference is that this number was confirmed by counting: advertised 41.667, observed 41. clock_getres's 1 millisecond matched no observation at all.

The 1,000,000 ns step of the _COARSE clocks has the same kind of explanation: they never read the counter at all. They read the value the kernel wrote to memory at the last tick. Their step is therefore exactly the tick length, which on both of these machines — CONFIG_HZ=1000 — is one millisecond.

So the numbers come from three different places, and clock_getres knows about only one of them:

Number	Where it comes from
Observed 41 ns on A	The hardware counter's frequency (24 MHz)
Observed 29 ns on B	The floor of my method — not the clock's step
`_COARSE` step of 1 ms	The kernel tick (`CONFIG_HZ=1000`); 999,983 on B via NTP
Advertised 1 ms / 1 ns	`hrtimer_resolution` (timer mode)

Cost: three tiers

The cost went on the clock as well — 200,000 calls, three runs each, medians below:

Clock	A: ns/call	B: ns/call
The `_COARSE` ones	2.8 – 3.1	7.7 – 9.4
The other five in the vDSO	15.3 – 22.5	37.1 – 38.5
The CPU-time clocks	180.8 – 188.4	1,460.7 – 1,622.5

Read these as orders of magnitude, not as individual comparisons: differences between clocks within the same tier move from run to run. On machine B one run gave CLOCK_REALTIME at 268 nanoseconds — a sevenfold excursion inside the vDSO tier itself; it is a shared VPS. The differences between tiers, on the other hand, are clearly larger than the noise and point the same way in every run.

There is one more internal consistency check in this table. On machine A the fine clocks changed on 35.8% of reads; with a tick of 41.667 nanoseconds, that puts the time per call at 0.358 × 41.667 ≈ 14.9 nanoseconds. The lower bound of the cost table is 15.3 nanoseconds. Two independent measurements — one counting changes, the other timing calls — arrive at the same number.

The logic of the three tiers fits the table: the coarse clocks read a value from memory, the vDSO clocks read and convert the hardware counter, and the CPU-time clocks enter the kernel and total up the process's accounting. That the last tier costs seven to eight times more on machine B than on A is no surprise either — there, every call is a system call, and that system call happens inside a virtual machine.

What Node does

This blog runs on a Node process, so I put the same question to it. With Node 24.22.1 in the container I made 200,000 Date.now() calls and counted with strace: zero clock_gettime system calls.

Zero results should not be trusted on their own — a tool that is not working also reports zero. I checked with two deliberate positives: the same strace command on the C program with a CPU-time clock (200,000 calls appeared), and process.resourceUsage() inside the same Node process (five getrusage calls appeared). The tool was working; the zero was real.

One gap remains: vDSO calls are by definition invisible to strace, so "zero clock_gettime" does not on its own separate "answered from the vDSO" from "the clock was never read, the value came from a cache". Cost settles it: Date.now() takes 54.0 nanoseconds. A cached value would be on the order of 3 nanoseconds on this machine; 54 says there is a real clock read underneath.

The other two paths sit in the same range: performance.now() at 63.8 ns, process.hrtime.bigint() at 73.0 ns. The gap from raw clock_gettime's 20-odd nanoseconds is the runtime's own layer.

Two monotonic clocks do not run at the same speed

In every table CLOCK_MONOTONIC and CLOCK_MONOTONIC_RAW sat side by side with the same numbers. Their difference does not show up in a single reading; it accumulates with elapsed time. RAW hands you the hardware counter untouched, while the other passes through the frequency correction NTP applies.

I ran them alongside each other for sixty seconds:

Machine	`MONOTONIC`	`MONOTONIC_RAW`	Difference
A (Mac VM)	60,001,705,862 ns	60,001,705,946 ns	−84 ns (−0.001 ppm)
B (VPS3, with NTP)	60,000,177,976 ns	60,001,142,617 ns	−964,641 ns (−16.1 ppm)

On machine A the two clocks are the same for practical purposes: they diverge by 84 nanoseconds in a minute, meaning no frequency correction is being applied there. On machine B, NTP is slowing the counter by 16 ppm; the raw counter counts almost a full millisecond extra per minute.

16 ppm sounds small. It is 58 milliseconds over an hour, 1.4 seconds over a day. In a benchmark run that is usually immaterial — but if you are going to compare your measured duration against a clock outside the machine, say a timer answering "did this job really take 10 minutes", you want the corrected one, CLOCK_MONOTONIC. RAW belongs in the narrow space where you care about the hardware counter's own rate.

Count it on your own machine

The most useful piece of this article is probably the loop below. Instead of reading the advertised number, it makes the clock count: it reads the clock back to back and keeps the smallest non-zero difference it sees.

struct timespec res, p, c;
clock_getres(id, &res);                      /* advertised */
clock_gettime(id, &p);
uint64_t mind = ~0ull;
for (long k = 0; k < 2000000; k++) {
    clock_gettime(id, &c);
    uint64_t a = ns(p), b = ns(c);
    if (b > a) { uint64_t d = b - a; if (d < mind) mind = d; p = c; }
}
/* mind = smallest observed step */

Two million rounds finish in under two seconds on these machines. Do not lower the sample count: the finer the step, the more samples you need to catch the smallest difference — on the _COARSE clocks I saw only between 6 and 21 changes across two million samples, because the clock moves once a millisecond.

Add one more counter to the loop: how many reads actually changed the value. If that number approaches the sample count, what you are measuring is not the clock's step but your own loop's period — the clock is faster than you can read it. That is exactly what happened on my machine B.

One warning: the compiler can optimise this loop away. This one was compiled with -O2 and the loop survived because the result gets printed; print whatever you measure, or what you measure may have been optimised out of existence.

Which clock, when

Measuring a duration? CLOCK_MONOTONIC. The wall clock can step backwards; this is the only one that only moves forward.
Stamping millions of times a second, and a millisecond is enough? CLOCK_MONOTONIC_COARSE. Five times cheaper, and it genuinely delivers the resolution it advertises.
Keep the CPU-time clocks off hot paths. You pay one system call per call; on my machine B that is 1.5 microseconds. Calling it a hundred thousand times a second in a profiler burns 0.15 seconds — a sixth of a core, doing nothing.
Do not decide based on what clock_getres() returns. It is describing the timer. If you want the clock's real step, measure it — but count the change rate as well, or you will publish your own loop's period instead of the clock's step.
Be sure you want _RAW. On an NTP-disciplined server the raw counter drifts from the corrected clock — 16 ppm in my case. For any measurement you will compare against a duration outside the machine, take the corrected one.
Refresh your assumptions on a VM. The same call gave answers a million times apart on my own two machines, and code you move runs on both.

Conclusion

When we ask an API a question, we assume whoever answers heard the question we asked. clock_getres never lies here; it answers its own question with surprising accuracy — the "1 millisecond" grid it reported on machine A is exactly what I found in the sleep measurements. The trouble is that its question was not mine. The name promises the clock's resolution; the contents describe the timer's grid, and on one machine those two sit 24,000 times apart.

Which is why "wrong number" is not quite right either. Closer to it: my question has no answer in this API at all. The way to learn how finely a clock is read is not to ask, but to make it count.

DEV Community