In a disposable virtual machine I added a single entry to the neighbour table. The command succeeded quietly, return value zero. A second later 56 neighbours had been deleted from the table — 56 entries I hadn't added and had nothing to do with.
This isn't a fault; it follows directly from how the kernel manages the ARP and ND caches. The real question is who that table belongs to: there is exactly one of it for every network namespace on your server. On my machine running 59 containers the host namespace showed 24 neighbours while the kernel's own counter said 117; the missing 93 entries were inside the containers, and all of them were eating from the same 1024-entry budget.
The neighbour table — ARP on the IPv4 side, the ND cache on the IPv6 side — is not split per network namespace in Linux. Every namespace shares the same table, the same three thresholds and the same garbage collector. On top of that, none of those three thresholds does quite what its name promises, and the fourth knob, the gc_interval that appears in every tuning guide, has been wired to nothing since 2004.
Measuring who owns the counter
I didn't want to write the claim without measuring it. I created a separate network namespace on the server, put a dummy interface in it and added 50 fake neighbour entries there:
ip netns add nbtest
ip netns exec nbtest ip link add dummy0 type dummy
ip netns exec nbtest ip link set dummy0 up
ip netns exec nbtest ip addr add 10.99.0.1/16 dev dummy0
for i in $(seq 2 51); do
ip netns exec nbtest ip neigh add 10.99.0.$i \
lladdr 02:00:00:00:00:$(printf %02x $i) dev dummy0 nud stale
done
Read from the host namespace, the kernel counter was 117 before the experiment and 167 after it. The new namespace didn't get its own budget; it ate from the main one. The same machine has 53 network namespace files opened by Docker — the numbers don't line up exactly with the 59 containers because some of them share the host's namespace — and all of them land under the same counter.
I read that counter from the first column of /proc/net/stat/arp_cache. This column behaves differently from the rest: the file has one row per CPU, and fields like allocs, lookups and periodic_gc_runs really are per-CPU counters, but the first column repeats the same global value on every row. It's a one-line detail in the kernel source — the first argument of that seq_printf call is atomic_read(&tbl->entries). So if you sum the rows you multiply the table size by the number of CPUs. That's exactly what I did on my first measurement, and on an 18-core machine I got the magnificent number 2106; dividing by 18 gave 117 and I understood the mistake.
To see where the split actually ends, you have to look from inside a container too:
$ docker run --rm alpine ls /proc/sys/net/ipv4/neigh/
erspan0 eth0 gre0 gretap0 ip6_vti0 ip6gre0 ip6tnl0 ip_vti0 lo sit0 tunl0
$ docker run --rm alpine cat /proc/sys/net/ipv4/neigh/default/gc_thresh3
cat: can't open '/proc/sys/net/ipv4/neigh/default/gc_thresh3': No such file or directory
The container's namespace has interface directories but no default directory. gc_thresh1, gc_thresh2 and gc_thresh3 aren't visible there — not readable, not writable. /proc/net/stat/arp_cache isn't there either; that file is only created in the initial namespace. The limit that constrains your container can neither be seen nor changed from inside it.
The kernel source makes the reason plain. The thresholds live in the table itself, with no place in the per-interface neigh_parms struct, and the sysctl entries point straight at those global fields:
} else {
struct neigh_table *tbl = p->tbl;
dev_name_source = "default";
t->neigh_vars[NEIGH_VAR_GC_INTERVAL].data = &tbl->gc_interval;
t->neigh_vars[NEIGH_VAR_GC_THRESH1].data = &tbl->gc_thresh1;
t->neigh_vars[NEIGH_VAR_GC_THRESH2].data = &tbl->gc_thresh2;
t->neigh_vars[NEIGH_VAR_GC_THRESH3].data = &tbl->gc_thresh3;
}
Per-interface directories are terminated before these lines, and the default directory is only set up for the initial namespace. However many containers you run, they all share the 1024-entry default budget of a single structure called arp_tbl. I'd seen the same landscape on the fs.file-max side: the namespace boundary doesn't end where you think it does.
Three thresholds, three misleading names
The documentation describes gc_thresh3 as the "maximum number of non-PERMANENT neighbor entries allowed". Word for word that's correct, but what sticks in the reader's mind — "the table stops at this number" — is wrong. Here's the code on the allocation path:
entries = atomic_inc_return(&tbl->gc_entries) - 1;
gc_thresh3 = READ_ONCE(tbl->gc_thresh3);
if (entries >= gc_thresh3 ||
(entries >= READ_ONCE(tbl->gc_thresh2) &&
time_after(now, READ_ONCE(tbl->last_flush) + 5 * HZ))) {
if (!neigh_forced_gc(tbl) && entries >= gc_thresh3) {
net_info_ratelimited("%s: neighbor table overflow!\n", tbl->id);
What gets counted isn't tbl->entries but tbl->gc_entries. Here's the difference: PERMANENT entries and entries flagged as externally learned never enter gc_entries at all. So the ceiling applies not to the whole table but only to its collectable part.
While we're here: the three thresholds don't share a counter, and I assumed they did until I read the source. gc_thresh2 and gc_thresh3 look at gc_entries, but the gc_thresh1 gate in the periodic collector is measured with atomic_read(&tbl->entries) — the whole table, exempt entries included. That's why picturing the three thresholds as three marks on one ruler misleads you.
I measured this in a disposable virtual machine — I set the thresholds to 50/100/200 and then added 150 PERMANENT entries above the ceiling:
B3: succeeded=150 failed=0 entries=353 (ceiling 200)
B3 state distribution:
150 PERMANENT
194 STALE
The state breakdown only counts the d0 interface; the 9 entries between 344 and 353 belong to the machine's other interfaces. With a ceiling of 200 the table holds 353 entries and not a single insertion was refused. gc_thresh3 reads like a ceiling, but it functions as a trigger.
Five seconds of immunity
Half of what forced garbage collection is willing to take is written in the documentation. Here is the gc_thresh2 entry in full: "Threshold when garbage collector becomes more aggressive about purging entries. Entries older than 5 seconds will be cleared when over this number." It's easy to skip that second sentence, but it's half the story. neigh_forced_gc walks the list from the front, but it only takes an entry if its reference count is down to one and one of these holds: its state is NUD_FAILED or NUD_NOARP, it's a multicast address, or its updated stamp falls outside the last five seconds. Fresh entries are untouchable.
To see what that means, in the same virtual machine I added 300 entries in under five seconds against a ceiling of 200:
B1: succeeded=195 failed=105 entries=204
B1 dmesg:
[357616.510090] neighbour: arp_cache: neighbor table overflow!
105 insertions came back with ENOBUFS. The 204 in that output sitting above the ceiling isn't an inconsistency: that column is tbl->entries, counting everything including exempt entries, while the counter that hits the ceiling is gc_entries. The table was packed, but everything in it was a few hundred milliseconds old, so the collector couldn't lay a hand on any of it. When I waited six seconds and repeated the same load, all 100 insertions went through — the only difference was the age of the entries.
This matters, because in real life a steady load rarely fills a neighbour table. What fills it is a burst: a scan, a service-discovery storm after a restart, a thousand connections opening at once across a decent-sized subnet. For the first five seconds of that burst the collector is a spectator. There's also a time budget: neigh_forced_gc checks the clock every 16 entries and abandons the cleanup if it has exceeded one millisecond. So even if the entire table is stale, there's no guarantee a single call will collect all of it. A cleanup cut short by the budget has a second side effect: the early exit goes through goto unlock and skips the line that updates the last_flush stamp, so the five-second gate isn't reset — the next allocation request can trigger forced collection all over again.
I added one entry and lost 56 neighbours
Now for the measure of that aggression. The documentation says "more aggressive" but never says how aggressive; a single line of code does:
int max_clean = atomic_read(&tbl->gc_entries) - READ_ONCE(tbl->gc_thresh2);
max_clean is computed once and cleanup stops when it reaches that number, so the table doesn't drop below gc_thresh2 — it lands exactly on it. The threshold you crossed is the third one, the target is the second. That makes gc_thresh2 far more of a fallback line than a warning line.
The experiment was easy to set up. Thresholds at 50/100/200 again; I added 150 entries (above the second threshold, below the third), waited six seconds, then added one single entry:
after C1: entries=158 d0=150 forced_gc=109 # 150 fresh entries, GC ran once, took none
after C2: entries=103 d0=95 forced_gc=110 # ONE insertion, 6 s later
This is the measurement behind the scene in the opening. I added one neighbour and 56 neighbours left: the table went from 158 to 159 with my insertion, then down to 103 — right on top of the second threshold. From the system's point of view this is the design itself; but as an operator, if you're thinking "I have a thousand-entry ceiling, I'm at 600, I'm fine", here's what actually happens: from the moment you pass 512, every new neighbour can trigger a bulk eviction of the neighbours older than five seconds.
And the eviction doesn't care about namespaces. I measured that on a container host: with the table at 115 entries I added 204 entries in a separate namespace, and twenty seconds later the table was down to 51. Mine were gone, but so were the stale entries belonging to the host and the other containers. I ran the control too: when I added only 10 entries to the same namespace and stayed under the threshold, nothing was deleted for 35 seconds. What makes the difference isn't my namespace, it's the total count.
The knob that hasn't been connected for twenty-two years
net.ipv4.neigh.default.gc_interval is in every tuning guide. Its default is 30 seconds, it reads as "how often should the garbage collector run", and when tables grow people shrink that number.
The kernel never reads the value. In the source, gc_interval appears in exactly three places: when it's read over netlink, when it's written over netlink, and as the address the sysctl entry points to. When the periodic collector finishes its pass it requeues itself like this:
/* Cycle through all hash buckets every BASE_REACHABLE_TIME/2 ticks. */
queue_delayed_work(system_power_efficient_wq, &tbl->gc_work,
NEIGH_VAR(&tbl->parms, BASE_REACHABLE_TIME) >> 1);
The rhythm is set by base_reachable_time; gc_interval doesn't appear in that line at all. I counted over sixty-second windows in the virtual machine:
| setting | passes in 60 s | per pass |
|---|---|---|
gc_interval=30, base_reachable_time_ms=30000
|
4 | ~15 s |
gc_interval=1, base_reachable_time_ms=30000
|
4 | ~15 s |
gc_interval=1, base_reachable_time_ms=6000
|
17 | ~3.5 s |
Pulling the knob from thirty down to one changed nothing; touching the real knob made the rhythm four times faster (17/4 = 4.25; the theoretical expectation would have been 20 passes). You can do the same arithmetic on a production server, because periodic_gc_runs genuinely is per-CPU and can be summed. On my server, 1,077,802 seconds of uptime map to 70,166 passes: 15.36 seconds per pass. base_reachable_time_ms is 30000, half of which is 15 seconds. I didn't measure where the extra 0.36 seconds come from; the likely source is the work being set up with INIT_DEFERRABLE_WORK — deferrable work doesn't wake an idle core, it rides along with the next wake-up — but timer slack and the duration of the pass itself push the same way. If gc_interval were the real knob I should have seen 35,927 passes in the same period; the measured deviation is a factor of 1.95, which can't be rounding.
There's a subtlety in that count: the table was at 117 entries at the time, below gc_thresh1 (128). The periodic work still gets queued in that state, still bumps the counter and still recomputes reachable_time; it just returns without walking the buckets. For measuring the rhythm that's enough.
The nice part is that I no longer have to be the one claiming this. The gc_interval entry was missing from the kernel's ip-sysctl.rst for a long time; a one-line patch from Gabriel Goller at Proxmox, sent on 25 February 2026 and taken two days later, added it and closed with: "Unused since kernel v2.6.8." That's the summer of 2004. For twenty-two years the knob everyone turned was connected to nothing, and the documentation only admitted it this year. The entry wasn't in 7.0; it landed with 7.1 on 14 June 2026, so on servers still at 7.0 or below the documentation is silent.
What the collector won't touch, and 6.17's new class
The periodic collector's exemption list is short and sits in a single condition in the source:
if ((state & (NUD_PERMANENT | NUD_IN_TIMER)) ||
(n->flags & (NTF_EXT_LEARNED | NTF_EXT_VALIDATED))) {
write_unlock(&n->lock);
continue;
}
The exemption list at allocation time is slightly different: on the ip neigh path PERMANENT, extern_learn and extern_valid are exempt, and for entries the kernel creates itself there's also a loopback exemption (exempt_from_gc = !!(dev->flags & IFF_LOOPBACK)).
NTF_EXT_VALIDATED is the newcomer on that list. Ido Schimmel's patch (03dc03fa0432), sent on 26 June 2025 and taken into the kernel four days later, arrived with 6.17; the iproute2 counterpart is in release 6.17. (In the user-space header the constant is called NTF_EXT_EXT_VALIDATED; the in-kernel flag is NTF_EXT_VALIDATED. If you're looking at the netlink side, the first name is the one to grep for.) Its purpose is EVPN multi-homing: if a user-space control plane decides whether a neighbour is valid, you don't want the kernel deleting or invalidating that entry on its own.
I put all four classes side by side on a machine running kernel 7.0. I threw 200 filler entries into a separate namespace (to push the table above gc_thresh1 — below that the collector doesn't walk the buckets), set the dummy interface's gc_stale_time to 5 seconds and added four marker entries:
--- t=20s GLOBAL=51 fillers=0
10.98.0.3 lladdr 02:00:00:aa:00:03 extern_learn STALE
10.98.0.2 lladdr 02:00:00:aa:00:02 PERMANENT
10.98.0.4 lladdr 02:00:00:aa:00:04 extern_valid STALE
All 200 fillers and the plain STALE marker were gone in twenty seconds; PERMANENT, extern_learn and extern_valid were still there after sixty. That gc_stale_time is per-interface is a useful detail too: the table thresholds are global, but you can tune the staleness timeout for a single interface.
There's also a version reality. I tried the same command on a server running Ubuntu 24.04:
$ ip neigh add 10.96.0.2 lladdr 02:00:00:cc:00:02 dev nbd0 nud stale extern_valid
Error: either "to" is duplicate, or "extern_valid" is a garbage.
iproute2 6.1 doesn't know the flag, and the 6.8 kernel doesn't support it anyway. If you read this and think "that would help our MLAG setup", first make sure the kernel is 6.17 or newer and ip is 6.17 or newer.
So what should you do
Let me start with the bad news: on my server the forced_gc_runs and table_fulls counters are zero. I'm not writing this after an incident, I'm writing it before one. The table sits at 117 entries against a ceiling of 1024; there's plenty of room between me and the wall. But there's a difference between knowing and not knowing that the room is shared by 59 containers, because shared budgets like this are the first thing scale breaks, and when they break it shows up as packet loss, DNS timeouts and a three-hour hunt that starts with "something's weird on the network".
Questions worth asking about your own setup:
-
Where is your table?
printf "%d\n" 0x$(sed -n 2p /proc/net/stat/arp_cache | cut -d' ' -f1)gives you the neighbour count for the whole machine rather than just the host namespace. Do the same forndisc_cacheseparately: the ND cache lives in its own table, with its own thresholds (net.ipv6.neigh.default.*) and its own counter. -
Which counter does your monitoring scrape? The three columns worth graphing are
table_fulls,forced_gc_runsandunres_discards. The momentforced_gc_runsleaves zero, your table is living above the second threshold — even if you aren't seeing errors yet. Don't confuse the order either: an evicted neighbour usually costs you one ARP round trip, not a lost packet; the real loss happens on theENOBUFSpath and when an unresolved entry's queue overflows, which is whatunres_discardsshows. -
If you raise the thresholds, raise both. Lifting only
gc_thresh3doesn't change the behaviour that drags the table back togc_thresh2on every trigger; the wider the gap, the bigger the bulk eviction. The kernel's own ratio is 128/512/1024, that is 1:4:8; keep it as you scale up. The cost is onestruct neighbourper entry plus the driver's private area, and a hash table that grows now and then — under a megabyte for a thousand extra entries, but not free either. -
Delete the
gc_intervalline from your sysctl file. It does nothing, and while it sits there it tells the next person on call the lie that this has been tuned. If you genuinely want to change the rhythm the knob isnet.ipv4.neigh.default.base_reachable_time_ms— the default directory, not the per-interface one, because the pass interval comes from the table's own default parameters. And shrinking it increases neighbour validation traffic; it isn't free. - If your container density is growing, tune the thresholds on the host. These files aren't visible from inside the container, so there's no point putting a sysctl line in your image.
There's an honesty underneath all this that I like: the kernel doesn't manage the neighbour table like a memory pool, it manages it like a landfill. When it fills up it doesn't close the gate; it shovels the old stuff out to make room for something new, and while doing so it doesn't check whose container it came from. Your namespaces are an accounting detail to it, labels under a single counter. Knowing where isolation ends and the shared budget begins is a lot cheaper than assuming everything is separate.
I also used these sources in the body: the kernel's neighbour.c pinned to the 6.17 tag and the v6.8 version of the same file.
Top comments (0)