Columns › 5. Going live and operating › Part 20
Performance Tuning in Go: pprof and Benchmarks
效能調校:pprof 與 benchmark
Measure before you optimise: write reliable benchmarks with testing.B and b.Loop, compare runs with benchstat, profile CPU, heap, goroutines and mutexes with pprof, understand escape analysis and allocations, then tune GOMAXPROCS, GOGC, GOMEMLIMIT and apply PGO.
In this article
- What this article solves
- Step 1: Writing benchmarks with testing.B
- Step 2: Compare with benchstat instead of reading one run
- Step 3: Finding CPU and memory hot spots with pprof
- Step 4: Sampling a running service: net/http/pprof
- Step 5: Escape analysis and memory allocation
- Step 6: GOMAXPROCS, GOGC and GOMEMLIMIT
- Step 7: PGO: letting the compiler optimise for the real workload
- Common pitfalls
- Key takeaways
- Next
In the previous article, “Deploying One Go Service with Docker and Kubernetes”, we packaged the order service into an image and set its resource requests and limits in the cluster. This article answers the questions behind those numbers: where does the service actually spend its CPU and memory, and how do you make it faster and leaner on solid evidence?
What this article solves
The most common way of dealing with performance problems is to change things by intuition: "this uses fmt.Sprintf, it must be slow", "passing a pointer would be cheaper", "let's add a cache". Intuition is sometimes right, but more often you spend ages changing things while the real bottleneck is somewhere else entirely; or, after the change, whether it is actually faster is just a "feeling".
The Go toolchain provides a whole set of built-in tools for performance work, and the principle fits in one sentence: measure first, then change, then measure again. This article covers, in order:
- Writing benchmarks with
testing.Bto measure the time and memory allocations of a single function. - Using
benchstatto judge whether a difference before and after a change is real rather than noise. - Using
pprofto find CPU, memory, goroutine and lock hot spots across the whole program, including sampling a production service safely. - Understanding escape analysis and memory allocation, and knowing which code patterns send values to the heap.
- Tuning the runtime parameters
GOMAXPROCS,GOGCandGOMEMLIMIT, and using PGO to let the compiler optimise based on the real workload.
The examples continue with the order service in internal/order: a function that builds a summary string from an order's contents, and the HTTP service that handles order requests.
Step 1: Writing benchmarks with testing.B
Benchmarks live in the same _test.go files as tests; their names start with Benchmark and they take a *testing.B. Since Go 1.24 the recommended style is for b.Loop() { ... }, replacing the old for i := 0; i < b.N; i++. b.Loop has several advantages: set-up work before the loop is automatically excluded from timing, so you no longer need to call b.ResetTimer; function calls in the loop body, along with their arguments and return values, are not removed by the compiler as dead code, so you no longer need to assign results to a global variable to "fool" the optimiser; and the benchmark function as a whole runs only once, so expensive set-up is not repeated.
Below is a complete test file that runs as is. It compares two ways of building an order summary: one with string concatenation and fmt.Sprintf, the other with a pre-sized strings.Builder and strconv.
package order
import (
"fmt"
"strconv"
"strings"
"testing"
)
type Item struct {
SKU string
Qty int
Price int64 // in cents (minor units)
}
type Order struct {
ID string
UserID string
Items []Item
Total int64
Status string // pending / paid / cancelled
}
// SummaryNaive builds the summary with string concatenation and fmt.Sprintf.
func SummaryNaive(o *Order) string {
s := "order " + o.ID + ":"
for _, it := range o.Items {
s += fmt.Sprintf(" %s x%d", it.SKU, it.Qty)
}
return s
}
// SummaryBuilder estimates the capacity up front, avoiding repeated allocation and fmt's formatting cost.
func SummaryBuilder(o *Order) string {
var b strings.Builder
b.Grow(len("order :") + len(o.ID) + len(o.Items)*16)
b.WriteString("order ")
b.WriteString(o.ID)
b.WriteByte(':')
for _, it := range o.Items {
b.WriteByte(' ')
b.WriteString(it.SKU)
b.WriteString(" x")
b.WriteString(strconv.Itoa(it.Qty))
}
return b.String()
}
func newOrder(n int) *Order {
o := &Order{ID: "A123", UserID: "u-42", Status: "pending"}
for i := range n {
o.Items = append(o.Items, Item{SKU: "SKU-" + strconv.Itoa(i), Qty: i%3 + 1, Price: 1990})
o.Total += int64(i%3+1) * 1990
}
return o
}
// First confirm both versions give the same result; only then is the benchmark comparison meaningful.
func TestSummaryEqual(t *testing.T) {
for _, n := range []int{0, 1, 10, 100} {
o := newOrder(n)
if got, want := SummaryBuilder(o), SummaryNaive(o); got != want {
t.Fatalf("n=%d: got %q, want %q", n, got, want)
}
}
}
func BenchmarkSummary(b *testing.B) {
for _, n := range []int{1, 10, 100} {
o := newOrder(n) // set-up: not timed
b.Run(fmt.Sprintf("naive/items=%d", n), func(b *testing.B) {
b.ReportAllocs()
for b.Loop() {
SummaryNaive(o)
}
})
b.Run(fmt.Sprintf("builder/items=%d", n), func(b *testing.B) {
b.ReportAllocs()
for b.Loop() {
SummaryBuilder(o)
}
})
}
}This example shows a few good habits for writing benchmarks:
- Write a test proving the behaviour is the same first. The worst outcome of an optimisation is "faster, but wrong".
TestSummaryEqualgives the refactoring a safety net, in the same spirit as “Testing in Go: Table-Driven Tests, httptest and Integration Tests”. - Use sub-benchmarks to sweep different sizes.
b.Runlets the same benchmark cover 1, 10 and 100 items. The two versions may be about the same at small sizes and only pull apart at large ones; measuring a single size easily leads to one-sided conclusions. b.ReportAllocs()adds the number of allocations per operation (allocs/op) and bytes allocated (B/op) to the results. In Go, memory allocation is often more expensive than pure computation, because it also adds garbage collection load.
Running go test -bench=Summary -benchmem shows ns/op, B/op and allocs/op for each sub-benchmark. ns/op is the average time per operation, and a number such as -8 after the name is the GOMAXPROCS at run time.
Step 2: Compare with benchstat instead of reading one run
Benchmark results are inherently noisy: CPU frequency scaling, other programs competing for resources and operating system scheduling all make the same code produce different numbers on every run. Running once and declaring "5% faster" may well be pure luck.
The right approach is to run each version several times and compare them statistically with benchstat. Suppose the benchmark keeps only one Summary function: measure once with the string concatenation version, then again after switching to strings.Builder:
# Install benchstat (a tool from golang.org/x/perf)
go install golang.org/x/perf/cmd/benchstat@latest
# Before the change: run 10 times and save to old.txt
go test -run '^$' -bench=Summary -benchmem -count=10 ./internal/order > old.txt
# After changing the code: run again under the same conditions
go test -run '^$' -bench=Summary -benchmem -count=10 ./internal/order > new.txt
# Statistical comparison
benchstat old.txt new.txt-run '^$' means no ordinary tests are run, only benchmarks. benchstat's output looks roughly like this (the numbers are illustrative only; real results depend on the machine):
│ old.txt │ new.txt │
│ sec/op │ sec/op vs base │
Summary/items=1-8 410.0n ± 3% 47.0n ± 2% -88.54% (p=0.000 n=10)
Summary/items=10-8 3.10µ ± 2% 0.43µ ± 1% -86.13% (p=0.000 n=10)
Summary/items=100-8 18.40µ ± 4% 1.12µ ± 2% -93.91% (p=0.000 n=10)
geomean 3.03µ 0.28µ -90.14%Three columns need reading: ± 2% is the variation range, and a large one means the measuring environment is unstable; p= is the p-value of the statistical test, usually considered significant only when below 0.05; ~ means there is no significant difference between the two. The illustrative results above are significant at all three sizes, and the gap widens with the number of items. But remember that this only says that this function got faster: if building the summary is only a small part of the whole request's time, the improvement users notice may be negligible. Function-level gains have to be weighed against the whole, which is why the next step uses profiles to look at the whole program.
Keep the environment as stable as possible while measuring: close other resource-hungry programs, keep the laptop plugged in, and run the old and new versions alternately on the same machine.
Step 3: Finding CPU and memory hot spots with pprof
A benchmark answers "how fast is this function", but it assumes you already know which function to look at. In a real service, the question is usually "why is the whole request slow" or "why does memory keep growing", and that is when you need a profile: periodic sampling while the program runs, recording which functions the time or memory goes to.
Go has several built-in profiles:
| Profile | Question it answers | How it samples |
|---|---|---|
| CPU | Which functions use the most CPU time? | Interrupts about 100 times a second and records the current call stack |
| heap | Who allocated memory? What is currently live on the heap? | Samples by allocation size, with inuse_* and alloc_* views |
| goroutine | Which goroutines exist now, and where is each one stuck? | Snapshots the stacks of all goroutines |
| mutex | Which locks are most contended? | Needs a sampling rate; records time spent waiting for holders to release locks |
| block | Where do goroutines block (channels, locks, select)? | Needs a sampling rate |
The simplest way to get a profile is to produce one while running a benchmark: go test -bench=Summary -cpuprofile=cpu.out -memprofile=mem.out. Then analyse it with go tool pprof. pprof's most useful interface is the web UI: go tool pprof -http=:8081 cpu.out opens a browser with views such as the call graph, the flame graph and annotated source.
Two numbers are key when reading a profile: flat is the time a function spends "itself", and cum (cumulative) is the total time of the function plus everything it calls. A function with high flat is a hot spot that is actually doing the work; a function with high cum but low flat gets its cost from the calls beneath it, so you follow them down.
A flame graph draws this information as one picture: the horizontal axis is the share of samples (not chronological order) and the vertical axis is call depth; the wider the block, the more attention it deserves.
Figure: a flame graph's width represents the share of samples, and wide blocks at the top are hot spots with high flat; in this illustration, formatting and memory allocation take up most of the request handling.
The heap profile's two views serve different purposes: inuse_space shows objects live on the heap now, suited to tracking down memory leaks or excessive resident memory; alloc_space shows how much has been allocated in total, suited to finding places that create lots of short-lived objects and keep the garbage collector busy. In many services' CPU profiles, runtime.mallocgc and GC-related functions take a considerable share, and in that case reducing allocations is often more effective than optimising computation.
Step 4: Sampling a running service: net/http/pprof
Profiles from benchmarks and tests only reflect the test data. The real shape of the load can only be seen in production, and net/http/pprof lets you fetch profiles from a running service over HTTP. Its overhead is low: the CPU profile samples only during the roughly 30 seconds you ask for, and heap sampling is always on at a small cost.
But pprof endpoints leak information about the program's internals, and the CPU profile has a cost, so they must never be exposed publicly. The complete example below mounts pprof on a separate admin port that listens only on localhost, kept apart from the public service port:
package main
import (
"log/slog"
"net/http"
"net/http/pprof"
"os"
"runtime"
"strconv"
"sync"
)
var (
mu sync.Mutex
stock = map[string]int{"SKU-1": 1_000_000}
)
// reserve simulates a stock reservation: one global lock, which becomes contended under high load.
func reserve(sku string, qty int) bool {
mu.Lock()
defer mu.Unlock()
if stock[sku] < qty {
return false
}
stock[sku] -= qty
return true
}
func main() {
// The mutex and block profiles are off by default and need a sampling rate.
runtime.SetMutexProfileFraction(10) // record roughly 1 in 10 contention events
runtime.SetBlockProfileRate(10_000) // sample roughly once per 10µs of blocking
// Admin port: listens on localhost only, accessed through kubectl port-forward.
admin := http.NewServeMux()
admin.HandleFunc("/debug/pprof/", pprof.Index) // handles heap, goroutine, mutex, block and the rest
admin.HandleFunc("/debug/pprof/cmdline", pprof.Cmdline)
admin.HandleFunc("/debug/pprof/profile", pprof.Profile)
admin.HandleFunc("/debug/pprof/symbol", pprof.Symbol)
admin.HandleFunc("/debug/pprof/trace", pprof.Trace)
go func() {
if err := http.ListenAndServe("127.0.0.1:6060", admin); err != nil {
slog.Error("admin server", "err", err)
}
}()
app := http.NewServeMux()
app.HandleFunc("POST /orders", func(w http.ResponseWriter, r *http.Request) {
qty, _ := strconv.Atoi(r.URL.Query().Get("qty"))
if qty < 1 {
qty = 1
}
if !reserve("SKU-1", qty) {
http.Error(w, "out of stock", http.StatusConflict)
return
}
w.WriteHeader(http.StatusCreated)
_, _ = w.Write([]byte("ok"))
})
slog.Info("listening", "app", ":8080", "admin", "127.0.0.1:6060")
if err := http.ListenAndServe(":8080", app); err != nil {
slog.Error("app server", "err", err)
os.Exit(1)
}
}The common idiom import _ "net/http/pprof" registers these paths on http.DefaultServeMux. If the public server happens to use DefaultServeMux too (for example http.ListenAndServe(":8080", nil)), pprof is exposed publicly by accident. The example above explicitly uses its own ServeMux, so the public service has no /debug/pprof/ path at all.
How to fetch profiles:
# Forward the admin port from the cluster to the local machine
kubectl port-forward deploy/order 6060:6060
# CPU: sample for 30 seconds and open the web UI
go tool pprof -http=:8081 'http://localhost:6060/debug/pprof/profile?seconds=30'
# Memory: currently live objects; add -sample_index=alloc_space to see cumulative allocation
go tool pprof -http=:8081 http://localhost:6060/debug/pprof/heap
# goroutine: when the count keeps rising, see which line they are all stuck on
curl -s 'http://localhost:6060/debug/pprof/goroutine?debug=1' | head -50
# Lock contention
go tool pprof -http=:8081 http://localhost:6060/debug/pprof/mutex
# Compare the heap at two points in time to find what has grown
go tool pprof -http=:8081 -diff_base=heap-1.pb.gz heap-2.pb.gz
# Escape analysis: see which variables are moved to the heap
go build -gcflags='-m' ./internal/order 2>&1 | grep -E 'escapes|moved to heap'The goroutine profile is the most direct tool for tracking down leaks: when the go_goroutines metric mentioned in “Observability in Go: Logs, Metrics and Tracing” only ever goes up, look at the aggregated stacks with debug=1 and you will usually find thousands of goroutines stuck on a channel send or receive on the same line; then check against “Goroutines, Channels and Concurrency Patterns in Go” to see who forgot to cancel or close.
The mutex profile points out contention on global locks such as reserve above. The fix is usually to shrink the critical section, shard by key (for example one lock per group of SKUs), or switch to a design that needs no lock. The lock in the example is only illustrative; stock correctness across instances still relies on the database or on the mechanisms discussed in “Idempotency and Distributed Locks”.
Tip: every profile is only a sample; the numbers represent proportions, not exact counts. When comparing two profiles, keep the load conditions as similar as possible, or the difference may simply reflect different traffic.
Step 5: Escape analysis and memory allocation
Go variables can live on the stack or on the heap. Allocating on the stack costs almost nothing and is freed when the function returns; allocating on the heap goes through the memory allocator and afterwards has to be tracked and reclaimed by the garbage collector. The compiler decides where each variable goes through escape analysis: if it cannot prove that a value is no longer used after the function ends, the value "escapes" to the heap.
go build -gcflags='-m' shows the compiler's decisions, with messages such as "&Order{...} escapes to heap", "moved to heap: buf" and "leaking param: o". Common reasons for escaping are:
- Returning a pointer to a local variable. The value must outlive the function, so it can only go on the heap.
- Storing into an interface. Converting a concrete value to
anyor another interface (for example the arguments passed tofmt.Sprintf) causes an allocation in many cases. This is one of the reasonsSummaryNaivein Step 1 is slower. - A closure capturing variables when the closure outlives the current function (for example when it is handed to a goroutine).
- Slices whose size is unknown at compile time, such as
make([]byte, n)wherenis a variable, or whose size exceeds the limit for stack allocation. - Slices or maps whose elements are pointers, where the stored values themselves often escape as well.
Common techniques for reducing allocations:
| Technique | Description |
|---|---|
| Pre-allocate capacity | make([]T, 0, n), strings.Builder.Grow, make(map[K]V, n), avoiding repeated allocation and copying as they grow |
| Reuse buffers | Use sync.Pool on hot paths to reuse temporary objects, such as a bytes.Buffer for encoding; remember to reset before putting it back |
| Avoid unnecessary interface conversions | Use strconv instead of fmt on hot paths, and concrete types instead of any |
| Pass values rather than pointers | Passing a small struct by value is not necessarily slower than by pointer and may avoid an escape; let benchmarks decide |
| Reduce intermediate strings | Conversions between []byte and string often allocate; stick to one type throughout processing where possible |
But remember: escape analysis output is a clue, not a goal. Only when a profile shows that allocation really is a hot spot is it worth sacrificing readability to reduce allocations. Eliminating every escapes to heap is usually just a waste of time.
Step 6: GOMAXPROCS, GOGC and GOMEMLIMIT
Beyond the code, the Go runtime itself has a few parameters that affect performance:
GOMAXPROCS sets the maximum number of operating system threads executing Go code simultaneously, and defaults to the number of available CPUs. In a container, "the number of available CPUs" is the crux: older Go versions look only at the machine's core count, so if the node has 32 cores but the container's CPU limit is only 2, Go starts 32 threads competing for a 2-core quota, resulting in frequent throttling and unstable latency. Since Go 1.25, the default on Linux takes the cgroup CPU limit into account and adjusts automatically when the limit changes; with older versions, the common approach is to use a community package that sets it from the cgroup at startup, or to set it directly with an environment variable.
GOGC controls how often garbage collection runs. Its default is 100, meaning "trigger the next collection when the heap has grown by 100% relative to the live amount after the last collection". Raising it (for example to 200) means fewer GCs and lower CPU usage, but higher memory peaks; lowering it does the opposite. GOGC=off turns off proportion-triggered collection entirely.
GOMEMLIMIT (since Go 1.19) is a soft memory limit. As total memory approaches this value, the GC runs more aggressively to try to stay under it. This solves a problem that is hard to handle with GOGC alone: in a container, what you really care about is "not exceeding the memory limit and getting OOM-killed", not the growth ratio.
Figure: GOGC decides when to collect by ratio; with GOMEMLIMIT set, collection happens earlier as the limit approaches, keeping memory below the container's limit.
Common ways to combine the two:
- Typical services: keep
GOGC=100and setGOMEMLIMITto roughly 80–90% of the container's memory limit (the Deployment in the previous article showed how to inject the limit withresourceFieldRef; in practice you can scale it by a factor). - Services with plenty of memory that want to save CPU: raise
GOGCor set it tooff, leaving control entirely toGOMEMLIMIT, so the GC only runs frequently as the limit approaches.
Watch out for one trap, though: if the live data itself is close to GOMEMLIMIT, the GC runs constantly without reclaiming much, and the CPU is eaten up by the GC; this is called GC thrashing. The Go runtime has a safeguard against this that caps the share of CPU the GC may use, in which case memory may exceed the soft limit and the process will still be OOM-killed in the end. In other words, GOMEMLIMIT cannot replace "giving it enough memory"; when the heap profile shows that live data really is large, the right move is to reduce resident data or raise the limit.
To observe the GC's actual behaviour, set GODEBUG=gctrace=1, which prints a one-line summary to standard error for every collection, including the heap size before and after and the time taken. The implementation of the garbage collector in newer Go versions also keeps improving, so upgrading Go itself may bring measurable gains, which is one reason it is worth rerunning benchmarks when you upgrade.
Step 7: PGO: letting the compiler optimise for the real workload
Profile-Guided Optimization (PGO) has been generally available since Go 1.21: give the compiler a CPU profile representative of the production workload, and it applies more aggressive inlining and other optimisations to hot functions, such as turning interface method calls on hot paths into direct calls. The typical improvement mentioned in the official documentation is a few percent, and the actual effect depends on the program.
Using it is very simple:
- Take a CPU profile from the production service (Step 4's
/debug/pprof/profile?seconds=30), ideally covering a representative load, and merge the profiles of several instances if necessary (go tool pprof -proto a.pprof b.pprof > merged.pprof). - Name the file
default.pgoand put it in the main package's directory, that is,cmd/order/default.pgo. - Run
go buildas usual.-pgo=autois the default and automatically usesdefault.pgoin the main package's directory.
default.pgo should be committed to version control so that builds are reproducible; the multi-stage Dockerfile from “Deploying One Go Service with Docker and Kubernetes” needs no changes, since COPY . . brings it into the build stage too. After that, resample and update it periodically (for example before each major release); building newer code with a slightly older profile is fine, as PGO tolerates a certain amount of code change.
The whole performance work cycle can be drawn like this:
Figure: metrics reveal a problem, a profile locates the hot spot, a benchmark verifies the change, and then you go back to production to confirm the effect; every lap is driven by measurement.
Common pitfalls
- Optimising by intuition. Changing code without a profile very likely changes something that is not the bottleneck, and makes the code harder to read as well.
- Running a benchmark only once. The noise may be bigger than the difference. Run several times with
-countand letbenchstatjudge whether it is significant. - The compiler optimising the benchmark away. In the old style, if the result is not used the whole call may be removed, producing absurdly fast numbers. Using
b.Loop(Go 1.24+) avoids this. - Set-up work counted in the timing. In the old style, forgetting
b.ResetTimermeans the time spent creating test data is averaged into the results.b.Loopautomatically excludes set-up before the loop. - Unrepresentative test data. Measuring only 1 item, or only 10,000 items, can both lead to conclusions that do not match the real load. Sweep different sizes with sub-benchmarks and base them on the data distribution in production.
- Exposing pprof publicly. Using
DefaultServeMuxwhile importingnet/http/pprofamounts to handing internal information, and an endpoint that can burn CPU, to everyone. Mount it on a separate internal port. - Ignoring GOMAXPROCS and the CPU limit in containers. Older Go versions start threads based on the node's core count, causing heavy throttling. Check your version's behaviour or set it explicitly.
- Treating GOMEMLIMIT as a cure-all. When the live data itself is too large, the GC just spins. Use a heap profile first to confirm what is actually taking up the memory.
- Over-optimising.
sync.Pool, manual buffer reuse and avoiding interfaces all add complexity to the code. Use them only on hot paths where a profile proves they are worth it.
Key takeaways
- The principle of performance work is "measure → change → measure again"; every optimisation must be backed by data.
- Write benchmarks with
for b.Loop()(Go 1.24+) andb.ReportAllocs(), sweep different sizes with sub-benchmarks, and first confirm with a test that the old and new behaviour match. - Run several times with
-countand compare withbenchstat, looking at the variation and the p-value, so you are not misled by noise. - pprof provides CPU, heap, goroutine, mutex, block and other profiles; distinguish flat from cum and
inusefromalloc, and look for wide blocks in the flame graph. - Mount
net/http/pprofon a separate internal-only port and sample through port-forward; do not expose the defaultDefaultServeMuxpublicly. - Escape analysis (
-gcflags=-m) explains where allocations come from; reducing allocations on hot paths is often more effective than optimising computation. - In containers, watch how
GOMAXPROCSrelates to the CPU limit; control memory withGOMEMLIMITtogether withGOGC, but it cannot replace having enough memory. - PGO only needs the production CPU profile placed as
default.pgoin the main package's directory for the compiler to optimise for the real workload.
Next
At this point a single service is observable, shuts down cleanly, is well packaged and has had its performance measured; but the real system is api-gateway, order, payment and inventory working together with a database and a message queue. The next part begins with “Deploying a Whole Microservice System with Docker Compose”, getting the whole system fully running on one machine.