Mustafa ERBAY

Posted on Sep 17 Originally published at mustafaerbay.com.tr

Green Tea GC: 10 Percent on a Tree, 60 on a Map

#go #garbagecollector #greentea

go version -m /usr/local/bin/kopru-gateway told me something: go1.26.5. This small Go binary that has been running on my server for months (I counted its sockets in the keepalive post) was built on 27 August and has been running with Go's new garbage collector, Green Tea, ever since. Go 1.26 changed the default, and when the CI pipeline that builds it bumped the version, the mechanism changed too; I made no decision. Not noticing is normal, because the promise of the change is to be invisible: the same program, less GC CPU. How much less, in which program, is unknown without measuring; so I measured.

Page-based marking

Go's collector has been a concurrent mark-sweep since 2015: it marks live objects by following pointers from the roots, and sweeps the rest. Michael Knyszek and Austin Clements's blog post from October 2025 states the problem in one sentence: the mark phase jumps around memory irregularly, every pointer chase can be a cache miss, and as memory bandwidth per core shrinks on modern machines, that waiting eats most of the GC cost. Green Tea's idea is to change the unit of marking: the work list holds 8 KiB pages instead of individual objects; objects to be marked accumulate on a page, the page is scanned sequentially, a FIFO queue replaces the depth-first stack. A page's objects are loaded once and processed together; memory access turns into longer and more regular passes. The authors' summary: "work with pages, not objects."

The timeline: experimental in Go 1.25 (August 2025) via GOEXPERIMENT=greenteagc, default in 1.26 (February 2026), switchable off with GOEXPERIMENT=nogreenteagc; the release note says the opt-out is expected to be removed in 1.27. The promised gain is a 10 to 40 percent reduction in GC overhead in GC-heavy real programs, and another 10 percent from vector instructions in small-object scanning on Intel Ice Lake / AMD Zen 4 and newer CPUs.

Two workloads, three versions

Measured in containers: the golang:1.25-alpine, 1.26-alpine and 1.27-alpine images (1.25.11, 1.26.8, 1.27.1), the same source built twice per version, with GOEXPERIMENT=greenteagc and nogreenteagc, GOMAXPROCS=4, GOGC=100. The numbers come from the program's own runtime/metrics counters: /cpu/classes/gc/total:cpu-seconds (CPU spent by the collector, idle marking included) and /gc/heap/live:bytes. The tree load ran five times per configuration, the map load three; I report the median; the server is shared, so I leave the spread in square brackets.

The first workload is from the "binary trees" family: build binary trees of depth 19, sum them, throw them away; with a long-lived tree of depth 21 in the background so every cycle has something to scan. Live heap around 130 MB, objects full of pointers but allocated sequentially, so already adjacent in memory.

version  Green Tea   GC CPU (s)          cycles  wall (s)
1.25     on          5.12 [4.76-5.66]    21      7.46
1.25     off         5.14 [5.00-6.20]    19      7.43
1.26     on          4.45 [4.39-5.01]    22      7.19
1.26     off         4.92 [4.74-5.36]    19      6.85
1.27     on          4.31 [4.08-4.44]    21      6.12
1.27     off         4.55 [4.19-4.80]    19      5.76

1.25's experimental version gained nothing on this load. In 1.26 GC CPU dropped 9.6 percent, in 1.27 5 percent; the typical value the blog post calls "around 10 percent on many workloads". The cycle count went from 19 to 21-22, and wall time is 5-6 percent longer with Green Tea; I do not know why, I did not keep the spread for wall time either, it may be noise on a shared server. The tree itself is the kind Green Tea likes: the tracking issue (golang/go#73581) reports L1/L2 misses halved on the binary-trees benchmark; the bad case is low-fanout, rapidly mutated structures where half the page scans touch a single object. So the modest gain here is not down to the heap's regularity; my candidates are the short mark phase, the weight of idle workers and the absence of AVX-512, none of them measured.

The second workload is closer to what I see on servers: a map[int]*item with 1.5 million entries, each with a string, a slice, a small map[string]int and a pointer to another random entry; then 6 million random accesses, replacing the entry on every third one. Live heap 800-900 MB, pointers scattered across memory, exactly the structure Green Tea targets.

version  Green Tea  GOMAXPROCS  GC CPU (s)            wall (s)
1.26     on          4          7.64 [7.07-8.08]      19.4
1.26     off         4         18.51 [15.43-18.76]    19.5
1.26     on          8          7.08 [5.05-8.27]      19.4
1.26     off         8         17.02 [16.66-17.09]    19.4
1.27     on          4          7.08 [4.77-7.12]      20.6
1.27     off         4         20.77 [17.87-20.81]    20.4
1.27     on          8          8.22 [7.15-8.44]      19.1
1.27     off         8         15.20 [15.01-40.65]    20.1

Here the picture changes: GC CPU drops 59 percent in 1.26, 66 percent in 1.27 on four cores; between 46 and 58 percent on eight. Wall time does not change, because this program is mutator-bound and the collector runs in the background anyway. Same machine, same 1.26, two workloads: 10 percent and 60 percent; the two sides of the release note's "10-40" range. Where your own program lands is decided by the shape of your heap.

/cpu/classes/gc/total has a trap, and the runtime/metrics documentation states it plainly: the total includes the time idle workers (gc/mark/idle) spend; that time is burnt on cores the scheduler could not otherwise use, and "should be subtracted from the total GC CPU time to obtain a measure of compulsory GC CPU time". With a single-goroutine mutator and four Ps, most of the total is exactly that. On 1.26 I read the classes separately over three runs (median):

workload  Green Tea  total    idle    background  assist   compulsory (total − idle)
tree      on          4.64    3.01    1.55        0.05     1.63
tree      off         5.43    3.40    1.93        0.05     2.03
map       on          7.66    5.04    2.56        0.02     2.62
map       off        21.45   14.17    7.13        0.14     7.28

Compulsory GC CPU drops 20 percent on the tree and 64 percent on the map; once the idle workers are removed, the tree's gain is clearer than in the total. The number to read as "CPU left for other work on the machine" is this column; the total column also counts time on cores that were idle anyway.

The 10 percent my server does not get

There is no vector acceleration on this machine. /proc/cpuinfo has avx2, no avx512*; that is the flag set the hypervisor exposes to the virtual AMD EPYC. Green Tea's scanning kernel loads the "seen" and "scanned" bitmaps into 512-bit registers and expands them with VGF2P8AFFINEQB; that path needs an Ice Lake / Zen 4 class CPU. The numbers above are the algorithm change's share alone; the part the release note calls "another 10 percent on newer CPUs" is absent here. For a Go service with high GC load, checking the avx512 flag when choosing a cloud server now has a reason behind it.

Not to be confused with GOAMD64: kopru-gateway was built with GOAMD64=v1, but the vector scan checks the CPU flags at run time, not the build level; internal/runtime/gc/scan/scan_amd64.go looks for seven conditions: AVX512VL, AVX512BW, GFNI, AVX512BITALG, AVX512DQ, AVX512VBMI and POPCNT. A binary built with v1 also uses the fast path on a machine carrying those flags; on my machine there is no such path.

What gctrace shows

For those who prefer GODEBUG=gctrace=1 to the counter, a shrunken version of the same map load (300 thousand entries), 1.26, the same cycle number:

on:  gc 6 @0.458s 7%: 0.080+54+0.094 ms clock, 0.32+0.098/53/103+0.37 ms cpu, 98->105->103 MB, 114 MB goal, 0 MB stacks, 0 MB globals, 4 P
off: gc 6 @0.976s 11%: 0.067+403+0.54 ms clock, 0.27+0.15/402/776+2.1 ms cpu, 144->161->145 MB, 169 MB goal, 0 MB stacks, 0 MB globals, 4 P

The middle number is the concurrent mark time: 54 ms against 403 ms; on the CPU side 0.098/53/103 (assist / background workers / idle workers) against 0.15/402/776. The two lines share a cycle number but not a moment: live heap is 145 MB in the off run and 103 in the on run; a factor of 1.4 of the difference comes from the heap, the rest from the collector. The percentage at the start of the line is the share of CPU that went to GC since the program started: 7 percent against 11. The totals runtime/metrics gives are the accumulation of these lines; gctrace is not left on in production, but for a version comparison putting two lines side by side is enough.

The opt-out is still there in 1.27

The release note says "the opt-out is expected to be removed in 1.27". I tried: with GOEXPERIMENT=nogreenteagc the build succeeds on 1.27.1, runtime.Version() prints go1.27.1-X:nogreenteagc, and the measurement shows the old collector's behaviour (the 1.27 off rows in the tables). The 1.27 release notes say nothing about the GC; in 1.27.1 the switch has still not been removed. If you see a regression, verify it with this flag first, then go to the golang/go repository.

What to do

Every binary built with Go 1.26 or later is already using this; like kopru-gateway, without your knowing. The first thing to do is measure, and with the right counter: the /cpu/classes/gc/ classes of runtime/metrics; Prometheus's client_golang already exports them as go_cpu_classes_gc_*, subtract mark_idle from total and put the before and after of the version upgrade side by side. Wall time did not move at all in my second workload; what to look at is the GC's compulsory CPU share. If your heap is large, scattered and pointer-dense you will look like the second table; a service with a small heap and short mark phases, like the first. And if your GOGC/GOMEMLIMIT settings were calibrated against the old collector's cost, look again: once the cost per cycle drops you can afford a lower GOGC and therefore a smaller heap.

On my own server I will not rebuild kopru-gateway; it is already on Green Tea and its heap is small. But so that something this big does not slip through silently again at the next change that bumps the Go version in CI, I will have the go version -m output written into the deployment log.

Containers golang:1.25-alpine (it gave 1.25.11 that day), 1.26-alpine (1.26.8), 1.27-alpine (1.27.1); host Ubuntu 24.04, 18 vCPU virtual AMD EPYC (avx2 present, avx512 absent), shared; medians of 5 runs on the tree, 3 on the map, the class breakdown from 3 runs on 1.26. The programs report their own GC CPU via runtime/metrics. The 40.65 in the map table's [15.01-40.65] range is a single outlier run.

DEV Community