DEV Community

Cover image for Profile-Guided Optimization in Go 1.26: When the 5% Is Real
Gabriel Anhaia
Gabriel Anhaia

Posted on

Profile-Guided Optimization in Go 1.26: When the 5% Is Real

#go #performance #benchmark

You've heard the pitch for two years. "Free 5% speedup, just collect a profile and rebuild." If it were that easy, every Go service in production would already be running it.

So you run the experiment. Collect a CPU profile from your busiest service for an hour during peak. Rebuild with -pgo=cpu.prof. Deploy to a canary. The p50 barely moves. The p99 moves a percent or so. Nobody on finance notices.

Then you try it on a different service: an internal one that does protobuf parsing on a tight loop, hot path one function deep. The p50 drops mid-single-digits. The p99 drops a bit more. CPU usage on the canary is visibly lower in the dashboard.

Same compiler. Same flag. Two different stories. The 5% number is real. It just averages two very different shapes — and the average is the wrong thing to plan around.

What PGO actually does

The official PGO guide describes the mechanism. The version that matters in practice has two moving parts.

You give the compiler a CPU profile. The compiler reads it, identifies which functions and call sites are hot, and changes two decisions:

  1. Inlining. The compiler inlines hot functions more aggressively than its static heuristics would. Every inline removes a call frame and opens the inlined body up to further constant folding, dead-code elimination, and escape analysis on the caller's locals.
  2. Devirtualization. When the profile shows that an interface call site almost always dispatches to one concrete type, the compiler emits a direct call to that type's method, with a fallback for the rare other case. The direct call inlines. The interface call could not.

The Go team has been honest about expected gains. The PGO docs state: "As of Go 1.22, benchmarks for a representative set of Go programs show that building with PGO improves performance by around 2-14%." The Go 1.26 release notes do not list new PGO features, so the same envelope applies on the current toolchain. PGO sits in maintenance mode now.

Where the 5% is real

The wins concentrate on workloads that have a small, hot inner loop with a clear hot path. The profile points at it. The compiler inlines into it. The result shows up as wall-clock time on your dashboards.

Workloads where this lands:

  • Production-grade serializers. JSON parsers, protobuf encoders, msgpack codecs. These spend their time inside a few generic functions called from many call sites, with branches that are heavily skewed toward the common case (the field is a string, the wire type is varint, the bool is false). PGO loves this shape.
  • Hot interface dispatch. Anywhere your code goes through an interface in a tight loop and the runtime almost always picks the same concrete type. Database drivers are the canonical example: the driver.Conn is the same Postgres conn for the entire request, but every call goes through the interface. PGO devirtualizes the call once it sees the pattern in the profile.
  • Routing, middleware, and request fan-out. Any function that runs once per request, branches a lot, and calls into smaller helpers. If the profile shows the function dominates CPU, PGO will inline its callees and tighten the resulting hot path.

Datadog's writeup on continuous PGO reports about a 3.4% CPU reduction on their metrics-intake service in production after iterating on profile collection, with controlled benchmarks reaching 5.4% on the better profiles. Single-digit, reproducible, boring — that is what a real PGO win looks like.

Where it does not help

PGO is cheap, not free. Some workloads do not earn back the compile-time and binary-size cost.

  • Mostly-cold code. A service that sleeps on a Kafka consumer 95% of the time and does light processing for the other 5% has nowhere for PGO to land its inlines. The profile is dominated by runtime.gopark and the consumer client. The compiler cannot speed up sleeping.
  • Tiny services. A service whose total binary CPU time on a busy day is under 5% of one core has no headroom. You can give PGO the perfect profile and the savings are noise.
  • Dev-only paths. If you build with the same -pgo=cpu.prof for tests and for tooling, the profile from production has no relationship to the code paths your tests exercise. You are paying compile time for nothing.
  • Unrepresentative profiles. This is the failure mode the docs call out specifically. The PGO guide says: "Using an unrepresentative profile is likely to result in a binary with little to no improvement in production." If you collect the profile during a 3am cron run and deploy the binary for daytime traffic, the inlining decisions are tuned for the wrong workload. The binary is sometimes slower than the non-PGO build because the compiler inlined cold code at the cost of binary size and instruction-cache pressure.
  • Microbenchmarks. The same docs warn: "microbenchmarks are usually bad candidates for PGO profiling, as they only exercise a small part of the application, which yields small gains when applied to the whole program." The takeaway: use a real production profile or skip PGO.

If your service falls in this list, move on to a different optimization.

The actual workflow

Three steps. None of them has changed since Go 1.21, and Go 1.26 does not add new ones.

1. Collect a CPU profile from production

You want a profile that reflects the real traffic shape, taken over a long enough window that the rare-but-expensive operations show up. The official guidance is at least 30 seconds during representative load; 5 to 30 minutes during peak is the practical sweet spot — it captures rare-but-expensive paths the 30-second window misses.

If your service exposes net/http/pprof, hit it from outside:

curl -o cpu.prof "http://prod-host:6060/debug/pprof/profile?seconds=300"
Enter fullscreen mode Exit fullscreen mode

If you cannot expose pprof in production, write a small admin handler that opens a profile to a file on disk for a fixed duration and ship that file off-box.

2. Place the profile and build

The cleanest option is to drop the profile next to your main package as default.pgo. The Go toolchain picks it up without flags:

cmd/myservice/
  main.go
  default.pgo
Enter fullscreen mode Exit fullscreen mode 
go build ./cmd/myservice
Enter fullscreen mode Exit fullscreen mode

To keep profiles outside the source tree, point at it explicitly:

go build -pgo=/path/to/cpu.prof ./cmd/myservice
Enter fullscreen mode Exit fullscreen mode

-pgo=off disables it for a build, useful in CI when you want a clean non-PGO baseline.

3. Measure the delta

Most PGO writeups stop at step 2. The delta is where the question actually gets answered.

Pick a benchmark that reflects your real workload. Run it on the non-PGO build and the PGO build. Use benchstat to compare. Eyeballing ns/op numbers from a single go test -bench run will swallow a 5% effect under variance.

package serializer

import (
    "encoding/json"
    "testing"
)

type Order struct {
    ID    string  `json:"id"`
    Items []Item  `json:"items"`
    Total float64 `json:"total"`
}

type Item struct {
    SKU      string `json:"sku"`
    Quantity int    `json:"quantity"`
}

func BenchmarkUnmarshalOrder(b *testing.B) {
    data := []byte(`{"id":"o-1","items":[
        {"sku":"a","quantity":1},
        {"sku":"b","quantity":2}],"total":42.5}`)
    b.ReportAllocs()
    for b.Loop() {
        var o Order
        if err := json.Unmarshal(data, &o); err != nil {
            b.Fatal(err)
        }
    }
}
Enter fullscreen mode Exit fullscreen mode

for b.Loop() is the Go 1.24+ form and is what you want on a 1.26 toolchain; it replaces the older for i := 0; i < b.N; i++ idiom. b.ReportAllocs() plus -benchmem is belt-and-braces — either alone is enough, both together is harmless.

Run it both ways:

go test -bench=BenchmarkUnmarshalOrder \
    -count=10 -benchmem -pgo=off > nopgo.txt
go test -bench=BenchmarkUnmarshalOrder \
    -count=10 -benchmem -pgo=cpu.prof > pgo.txt
benchstat nopgo.txt pgo.txt
Enter fullscreen mode Exit fullscreen mode

Illustrative shape — real benchstat output groups metrics into separate blocks, one per metric:

                   │   nopgo.txt   │           pgo.txt           │
                   │    sec/op     │   sec/op     vs base        │
UnmarshalOrder-10    1.524µ ± 1%    1.452µ ± 1%   -4.72% (p=0.000)

                   │   nopgo.txt   │           pgo.txt           │
                   │     B/op      │     B/op    vs base         │
UnmarshalOrder-10    368.0 ± 0%    368.0 ± 0%       ~ (p=1.000)
Enter fullscreen mode Exit fullscreen mode

Read the vs base percentage; the p= is the t-test confidence. Sub-1% deltas with p > 0.05 are noise.

A 1% delta means PGO is a bad fit for this workload. 5-10% means the profile is good and the inliner is paying. 20%+ on real traffic is suspicious — check whether one synthetic hot loop is dominating the profile before celebrating.

What Go 1.26 changes

Nothing PGO-specific. The Go 1.26 release notes do not call out PGO improvements, which is a useful data point on its own: PGO has stabilized into a quiet, working part of the toolchain.

What 1.26 does change in the same neighborhood shifts the baseline. The Go 1.26 release notes describe the new Green Tea garbage collector as delivering "somewhere between a 10–40% reduction in garbage collection overhead in real-world programs that heavily use the garbage collector," with another 10% on top for newer amd64 CPUs that can use vector instructions for scanning. The compiler also "can now allocate the backing store for slices on the stack in more situations," moving work off the heap entirely.

Always compare PGO-on vs PGO-off on the same Go version. Crossing versions mixes the GC change, the slice-stack change, and the PGO delta into one number you cannot disentangle.

When to turn it on

Turn PGO on when:

  • Your service has a meaningful CPU footprint on a busy day.
  • The profile is dominated by your own hot path, not by sleeping or networking.
  • You can collect a profile from real production traffic, not from a load test.
  • You can measure the delta with benchstat and accept that the answer might be 1%.

Skip PGO when:

  • The service is mostly idle.
  • The hot path is in cgo or a syscall.
  • You cannot get a representative profile.

Run the benchstat. If it lands at 1%, you have your answer. If it lands at 6%, ship it.

If this was useful

PGO leans on receiver style, interface design, and runtime internals — the kind of mechanical-sympathy work the Go books I wrote cover end-to-end. The Complete Guide to Go Programming walks through the runtime, the compiler's escape analysis, and how the inliner reasons about your code. Hexagonal Architecture in Go covers how to structure a Go service so the framework code stays out of your domain hot paths.

If you ship Go in production, both books are written for the engineer who has read the spec and now wants the production-grade version of the answer.

Thinking in Go — the 2-book series on Go programming and hexagonal architecture

Top comments (0)