Columns › 5. Going live and operating › Part 17
Observability in Go: Logs, Metrics and Tracing
可觀測性:日誌、指標、追蹤
Build observability for Go microservices step by step: structured logs with log/slog and request IDs, RED/USE metrics exposed to Prometheus, OpenTelemetry tracing with context propagation, and correlating all three through a shared trace ID.
In this article
- What this article solves
- Step 1: Structured logs with log/slog
- Step 2: Linking a request's logs with a request ID
- Step 3: Deciding what to measure: RED and USE
- Step 4: Exporting metrics with Prometheus
- Step 5: Basic concepts of distributed tracing
- Step 6: Wiring up OpenTelemetry in Go and propagating context
- Step 7: Cross-referencing the three pillars
- Common pitfalls
- Key takeaways
- Next
In the previous article, “Distributed Transactions: Sagas and the Outbox Pattern”, we used sagas and the outbox to keep the order, payment and inventory services eventually consistent without a global transaction. This article moves on to "going live and operating": once the services are actually running, how do you see what they are doing?
What this article solves
By now the online shop's order service has grown into four services, api-gateway, order, payment and inventory, and we have dealt with timeouts, retries, idempotency, sagas and the outbox. Once the code is written and the tests pass, the real test begins: after the services go live, how do you know whether they are healthy right now? When a user reports that "placing an order is slow", how do you find which part got slower among four services and a dozen or so instances?
In the monolith days, opening a log file and scrolling down usually turned up a clue. After splitting into microservices, a single order passes through api-gateway, the order service, the inventory service and the payment service, and possibly a message queue too; each part's logs are scattered across different machines with no link between them. At this point "printing more logs" is no longer enough. What you need is systematic observability: the ability for the system to answer "what is happening right now, and why" from the outside.
Observability is usually split into three pillars:
| Pillar | Question it answers | Typical tools | Cost profile |
|---|---|---|---|
| Logs | What are the details of one particular event? | log/slog, Loki, Elasticsearch | High volume, proportional to request count |
| Metrics | Is the overall trend healthy? Has a threshold been crossed? | Prometheus, Grafana | Low volume, proportional to label combinations |
| Traces | How much time did one request spend in each service? | OpenTelemetry, Jaeger, Tempo | Moderate, usually sampled |
This article builds all three pillars step by step, and finally ties them together with the same trace ID: from an alerting metric to the matching trace, and from the trace to the logs for that part.
Step 1: Structured logs with log/slog
Since Go 1.21 the standard library has included log/slog, so you no longer need a third-party package for structured logging. Structured means that each log entry is not a sentence for humans to read but a set of "key = value" fields; once it is output as JSON, the logging system can query directly by field, for example "find every entry with order_id=A123" or "count entries with level=ERROR and route=/orders".
The traditional style, such as log.Printf("order %s failed: %v", id, err), is readable to humans, but machines have to pick the string apart with regular expressions, and any change of format breaks the queries. With slog, the same thing is written as logger.Error("create order failed", "order_id", id, "err", err): the message is fixed and everything that varies goes into fields. This habit matters: keep the message text fixed and put variables in fields, so that events of the same kind can be aggregated and counted.
When designing log fields, a few principles are worth settling from the start:
- Field names are uniform across the whole system: for example always
order_id, notorderIdin one service andoidin another. - Levels must mean something:
DEBUGis for chasing details during development,INFOrecords important state changes,WARNis an anomaly that recovers automatically, andERRORis a failure someone needs to look at. Production usually outputsINFOand above only. - Never write passwords, access tokens or full card numbers to the logs. You can implement the
slog.LogValuerinterface so that sensitive types decide for themselves what to output (for example only the last four digits). - Log a given error only once, in the layer that "handles" it. If every layer does
logand thenreturn err, one failure shows up five times in the logs, which makes them harder to read. This was covered in “Error Handling in Go: Wrapping, Sentinels and Custom Types”, and in production the effect becomes even more obvious.
Step 2: Linking a request's logs with a request ID
With structure in place, the next question is: with hundreds of requests running at the same time and their logs interleaved, how do you know which lines belong to the same request? The answer is to give each request an ID as soon as it arrives, and have every log along that request's path carry this field.
In Go, the most natural place for "data specific to this request" is context.Context. As “Go context: Cancellation, Timeouts and Request-Scoped Values” explained, context is suited to values that cross API boundaries and share the request's lifetime. The approach is: a middleware takes the ID from the X-Request-ID header (generating one if the upstream did not provide it) and puts it in the context; then a wrapped slog.Handler pulls the ID from the context and adds it as a field before every log entry is written. Business code then only has to remember to use context-aware methods such as logger.InfoContext(ctx, ...) and never has to pass the ID around itself.
Below is a complete example that uses only the standard library and runs as is. It shows a request ID middleware, an access log that records the status code and duration, and a bare-bones metrics endpoint (Step 4 explains why production should use the Prometheus client package instead):
package main
import (
"context"
"crypto/rand"
"encoding/hex"
"fmt"
"log/slog"
"net/http"
"os"
"sync"
"time"
)
type ctxKey struct{}
// ctxHandler wraps any slog.Handler and adds the request_id from the context before output.
type ctxHandler struct{ slog.Handler }
func (h ctxHandler) Handle(ctx context.Context, r slog.Record) error {
if id, ok := ctx.Value(ctxKey{}).(string); ok {
r.AddAttrs(slog.String("request_id", id))
}
return h.Handler.Handle(ctx, r)
}
func (h ctxHandler) WithAttrs(as []slog.Attr) slog.Handler {
return ctxHandler{h.Handler.WithAttrs(as)}
}
func (h ctxHandler) WithGroup(name string) slog.Handler {
return ctxHandler{h.Handler.WithGroup(name)}
}
func newID() string {
b := make([]byte, 8)
_, _ = rand.Read(b)
return hex.EncodeToString(b)
}
// statusRecorder records the status code actually written.
type statusRecorder struct {
http.ResponseWriter
code int
}
func (s *statusRecorder) WriteHeader(code int) {
s.code = code
s.ResponseWriter.WriteHeader(code)
}
// counters is a minimal request counter keyed by "route status".
type counters struct {
mu sync.Mutex
m map[string]int
}
func (c *counters) inc(key string) {
c.mu.Lock()
c.m[key]++
c.mu.Unlock()
}
func (c *counters) ServeHTTP(w http.ResponseWriter, _ *http.Request) {
c.mu.Lock()
defer c.mu.Unlock()
w.Header().Set("Content-Type", "text/plain; version=0.0.4")
fmt.Fprintln(w, "# TYPE http_requests_total counter")
for k, v := range c.m {
var route, code string
fmt.Sscanf(k, "%s %s", &route, &code)
fmt.Fprintf(w, "http_requests_total{route=%q,code=%q} %d\n", route, code, v)
}
}
func observe(logger *slog.Logger, cnt *counters, next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
id := r.Header.Get("X-Request-ID")
if id == "" {
id = newID()
}
ctx := context.WithValue(r.Context(), ctxKey{}, id)
w.Header().Set("X-Request-ID", id)
rec := &statusRecorder{ResponseWriter: w, code: http.StatusOK}
start := time.Now()
next.ServeHTTP(rec, r.WithContext(ctx))
route := r.Pattern // Go 1.23+: the matched route pattern, not the actual path
if route == "" {
route = "unmatched"
}
cnt.inc(fmt.Sprintf("%s %d", route, rec.code))
logger.InfoContext(ctx, "request done",
"method", r.Method,
"route", route,
"status", rec.code,
"duration_ms", time.Since(start).Milliseconds(),
)
})
}
func main() {
logger := slog.New(ctxHandler{slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{Level: slog.LevelInfo})})
slog.SetDefault(logger)
cnt := &counters{m: map[string]int{}}
mux := http.NewServeMux()
mux.HandleFunc("GET /orders/{id}", func(w http.ResponseWriter, r *http.Request) {
slog.InfoContext(r.Context(), "load order", "order_id", r.PathValue("id"))
fmt.Fprintf(w, `{"id":%q,"status":"paid"}`, r.PathValue("id"))
})
root := http.NewServeMux()
root.Handle("GET /metrics", cnt)
root.Handle("/", observe(logger, cnt, mux))
logger.Info("listening", "addr", ":8080")
if err := http.ListenAndServe(":8080", root); err != nil {
logger.Error("server stopped", "err", err)
os.Exit(1)
}
}There is an easily overlooked detail here: route takes r.Pattern (filled in by ServeMux since Go 1.23, e.g. GET /orders/{id}), not r.URL.Path (e.g. /orders/A123). The former has a bounded set of values; the latter is different for every order. Putting the actual path in logs is fine, but as soon as you use it as a metric label, you get the "label explosion" discussed in Step 4.
A request ID only links things up "within one service". When the order service calls the inventory service, it must also put X-Request-ID into the headers of the outgoing request, so that the inventory service's middleware reuses the same ID. In Step 6, OpenTelemetry's propagation mechanism takes this over in a standard way.
Step 3: Deciding what to measure: RED and USE
Logs are good for looking up details but poor at answering overall questions such as "what was the error rate in the past hour": computing a ratio from millions of log lines in real time is expensive. Questions like that belong to metrics: the program maintains some numbers (counters, gauges, distributions) that are collected periodically and stored as time series, making queries and alerts cheap.
The hardest part of metrics is not how to write them but "what to measure". Two classic methods make good checklists:
| Method | Applies to | Three metrics | Example in the order service |
|---|---|---|---|
| RED | Services that handle requests | Rate (requests/s), Errors (failures), Duration (latency distribution) | POST /orders per second, number of 5xx, P99 latency |
| USE | Resources (CPU, connection pools, queues) | Utilization, Saturation, Errors | Share of the database connection pool in use, number waiting for a connection, acquisition timeouts |
RED looks at a service from the user's point of view: as long as these three numbers are normal, users usually notice nothing. USE looks at bottlenecks from the resource's point of view: when RED shows things slowing down, USE helps you answer "which resource is the bottleneck". Taking the connection pool introduced in “database/sql, Transactions and Connection Pools in Go” as an example, sql.DB.Stats() already provides values such as InUse, WaitCount and WaitDuration; export them periodically as metrics and you have USE out of the box.
In a Go service there are also a few "runtime-level" metrics you almost always want: the number of goroutines, heap memory size and GC pause time. A goroutine count that keeps rising often signals a leak (for example a channel nobody ever reads), the classic problem mentioned in “Goroutines, Channels and Concurrency Patterns in Go”; on a metrics chart it shows up as a line that only ever goes up.
Figure: each pillar answers a different question, and the request ID and trace ID link them into one path from alert to root cause.
Step 4: Exporting metrics with Prometheus
Prometheus is the most common metrics system today. It uses a "pull" model: the service exposes a /metrics endpoint that lists all current metrics in a plain-text format, and Prometheus scrapes it at regular intervals (for example every 15 seconds). The example in Step 2 wrote this format by hand, but in production use the official github.com/prometheus/client_golang, which handles concurrency safety, histogram buckets, runtime metrics and other details.
Prometheus has four metric types: Counter (only goes up, e.g. total requests), Gauge (goes up and down, e.g. current queue length), Histogram (places observations into predefined buckets so percentiles can be computed on the server side) and Summary (computes percentiles on the client side, cannot be merged across instances, generally less used). RED's Duration almost always uses a Histogram, because histograms from several instances can be added together before computing the overall P99.
// Snippet: using github.com/prometheus/client_golang
import (
"net/http"
"strconv"
"time"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promauto"
"github.com/prometheus/client_golang/prometheus/promhttp"
)
var reqDuration = promauto.NewHistogramVec(prometheus.HistogramOpts{
Name: "http_request_duration_seconds",
Help: "HTTP request duration (seconds)",
Buckets: []float64{0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5},
}, []string{"method", "route", "code"})
func metricsMiddleware(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
rec := &statusRecorder{ResponseWriter: w, code: http.StatusOK}
start := time.Now()
next.ServeHTTP(rec, r)
reqDuration.WithLabelValues(r.Method, r.Pattern, strconv.Itoa(rec.code)).
Observe(time.Since(start).Seconds())
})
}
// In main: hand /metrics to promhttp, which by default also outputs runtime metrics such as go_goroutines and go_memstats_*.
// mux.Handle("GET /metrics", promhttp.Handler())The text Prometheus scrapes looks roughly like this; each line is one time series, and the key-value pairs inside {} are the labels:
# HELP http_request_duration_seconds HTTP request duration (seconds)
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{code="201",method="POST",route="POST /orders",le="0.1"} 940
http_request_duration_seconds_bucket{code="201",method="POST",route="POST /orders",le="0.25"} 990
http_request_duration_seconds_bucket{code="201",method="POST",route="POST /orders",le="+Inf"} 1000
http_request_duration_seconds_sum{code="201",method="POST",route="POST /orders"} 61.2
http_request_duration_seconds_count{code="201",method="POST",route="POST /orders"} 1000
go_goroutines 42(The numbers are illustrative only.) With these series you can write RED in PromQL: requests per second is rate(http_request_duration_seconds_count[5m]), the error rate is the rate of 5xx divided by the total rate, and P99 is histogram_quantile(0.99, sum by (le, route) (rate(http_request_duration_seconds_bucket[5m]))).
Tip: every distinct combination of label values is a separate time series. Labels may only hold things with a "bounded set of values", such as the method, route pattern and status code; user IDs, order numbers, full URLs and full error messages should never be labels. High-cardinality data like that belongs in logs or traces.
Alerts should be based on "symptoms" rather than "causes": rather than alerting on "CPU above 80%", alert on "POST /orders error rate above the threshold for five consecutive minutes" or "P99 above the service objective". High CPU that users do not notice is no reason to wake someone at night; conversely, a rising error rate always needs someone to deal with it. This is also the basic idea behind SLOs (service level objectives): first define the objectives users care about, then use metrics to track whether you are drifting from them.
Step 5: Basic concepts of distributed tracing
Metrics tell you "the P99 of POST /orders has got worse", but not where the time is going. A single order might involve: the order service writing to the database, then synchronously calling the inventory service to reserve stock, then calling the payment service for authorisation, and finally sending an event to the message queue. Which part got slower? That is the question distributed tracing answers.
Tracing has a few core terms:
- Trace: one complete journey of a request, with a globally unique trace ID.
- Span: one piece of work within the journey, such as "handle the HTTP request", "run an SQL statement" or "call the inventory service". Each span has its own span ID, start and end times, attributes, and the ID of its parent span.
- Context propagation: when service A calls service B, it passes the trace ID and the current span ID in the request headers, so the spans B creates hang under A's span. The W3C
traceparentheader is the common format today, and looks like00-<32-hex-digit trace id>-<16-hex-digit parent span id>-01.
Lay out all the spans with the same trace ID along a time axis and you get a "waterfall", where the longest part is obvious at a glance:
Figure: the spans under one trace are arranged by parent-child relationship and time; the longest one is the first thing to investigate.
OpenTelemetry (OTel for short) is an open standard from the CNCF that unifies the APIs, SDKs and transport protocol (OTLP) for traces, metrics and logs. Your code depends only on the OTel API; whether the data goes to Jaeger, Tempo or a commercial platform is just a matter of configuration, with no code changes.
Step 6: Wiring up OpenTelemetry in Go and propagating context
Wiring up OTel usually comes in three parts: at startup, create a TracerProvider and configure an exporter; add automatic instrumentation to the HTTP/gRPC servers and clients, so that spans are created and traceparent is injected and parsed automatically; and create spans by hand for key business steps.
// Snippet: using go.opentelemetry.io/otel and contrib's otelhttp
import (
"context"
"net/http"
"go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/attribute"
"go.opentelemetry.io/otel/codes"
"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
"go.opentelemetry.io/otel/propagation"
"go.opentelemetry.io/otel/sdk/resource"
sdktrace "go.opentelemetry.io/otel/sdk/trace"
semconv "go.opentelemetry.io/otel/semconv/v1.26.0"
)
func setupTracing(ctx context.Context) (func(context.Context) error, error) {
exp, err := otlptracegrpc.New(ctx) // endpoint set by OTEL_EXPORTER_OTLP_ENDPOINT
if err != nil {
return nil, err
}
tp := sdktrace.NewTracerProvider(
sdktrace.WithBatcher(exp),
sdktrace.WithSampler(sdktrace.ParentBased(sdktrace.TraceIDRatioBased(0.1))),
sdktrace.WithResource(resource.NewWithAttributes(semconv.SchemaURL,
semconv.ServiceName("order"))),
)
otel.SetTracerProvider(tp)
otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
propagation.TraceContext{}, propagation.Baggage{}))
return tp.Shutdown, nil // must be called on shutdown to flush buffered spans
}
// Server side: create a span for every request automatically and join the upstream trace from the traceparent header.
// handler := otelhttp.NewHandler(mux, "order")
// Client side: write traceparent into the headers automatically when calling the inventory service.
var client = &http.Client{Transport: otelhttp.NewTransport(http.DefaultTransport)}
func reserveStock(ctx context.Context, orderID string) error {
ctx, span := otel.Tracer("order").Start(ctx, "reserve-stock")
defer span.End()
span.SetAttributes(attribute.String("order.id", orderID))
req, err := http.NewRequestWithContext(ctx, http.MethodPost, "http://inventory/reservations", nil)
if err != nil {
return err
}
resp, err := client.Do(req)
if err != nil {
span.RecordError(err)
span.SetStatus(codes.Error, "reserve failed")
return err
}
defer resp.Body.Close()
return nil
}The key to this code is that context is passed all the way down. otelhttp.NewHandler puts the span into the request's context; reserveStock opens a child span from that context; http.NewRequestWithContext carries the child span into the outgoing request, and otelhttp.NewTransport takes it out and writes it into the headers. If any layer in between switches to context.Background(), the trace breaks in two at that point. This is the practical payoff of “Go context: Cancellation, Timeouts and Request-Scoped Values” insisting again and again that "context is passed down as the first argument".
Asynchronous paths must propagate it too. When the order service sends an "order created" event through the message queue, it should write traceparent into the message headers (Kafka headers or AMQP properties); the consumer extracts it with propagator.Extract to restore the context and then opens a span. That way the asynchronous flows in “Message Queues and Event-Driven Design” and “Distributed Transactions: Sagas and the Outbox Pattern” also appear in the same waterfall. If you use an outbox, remember to store traceparent in the outbox table along with the event content, so the relay has it when it publishes.
Sampling is the other important point about tracing. Recording a full trace for every request is expensive in both data volume and cost, so usually only a portion is kept. The ParentBased(TraceIDRatioBased(0.1)) above means: if the upstream has already decided whether to sample, follow its decision (so a trace is never recorded only halfway); when this service is the starting point, keep about one in ten based on the trace ID. A more advanced approach is "tail sampling" in the OTel Collector: wait until the whole trace has finished, then decide to keep every trace that errored or was particularly slow.
Step 7: Cross-referencing the three pillars
Once each pillar is built, the real value lies in being able to jump between them. There are three ways to do this:
- Logs carry the trace ID and span ID. Extend the
ctxHandlerfrom Step 2 slightly: besides request_id, usetrace.SpanContextFromContext(ctx)to takeTraceID()andSpanID()from the context and, when valid, write them astrace_idandspan_idfields. The logging platform can then open the matching trace straight from an error log. You can also use OTel's slog bridge package to send logs directly over OTLP, with these fields added automatically. - Metrics carry exemplars. Prometheus histograms support exemplars: a trace ID of a representative request attached next to a bucket. Grafana draws these points on the latency chart, and clicking one jumps to the trace of that slow request, taking you in one step from "the P99 went up" to "it was this request, and it was slow in this part".
- Spans carry business attributes. Add attributes such as
order.idto spans, and you can find traces the other way round, by business data: when support says "order A123 is stuck", you can search for that attribute directly.
A typical troubleshooting flow therefore becomes: an alert reports that the P99 of POST /orders is over the limit → click an exemplar on the dashboard → the waterfall shows that the inventory: reserve span takes most of the time → jump to the inventory service's logs under this trace ID → see "timed out waiting for a database connection" → go back to the USE metrics and confirm that the connection pool is saturated. From symptom to root cause, every step has a lead to follow, instead of digging through files on several machines one by one.
Figure: starting from a symptom metric, exemplars, traces and logs narrow things down step by step to a specific resource bottleneck.
Common pitfalls
- Label explosion. Use user IDs, order numbers or raw URLs as metric labels and the number of time series grows without limit, with Prometheus's memory ballooning along with it. Labels hold bounded sets only; for routes use patterns such as
r.Pattern. - Broken context. Switching to
context.Background()inside a goroutine, a background job or a queue consumer breaks the trace. When you need to detach from the request's cancellation but keep the tracing information, usecontext.WithoutCancel(ctx), available since Go 1.21. - Forgetting to shut down the TracerProvider. The SDK buffers spans in batches; if
Shutdownis not called before the program exits, the last batch is lost, and it is precisely the most important part, right before things went wrong. The next article, “Configuration and Graceful Shutdown in Go”, puts it into the shutdown sequence. - Runaway log volume. Logging
INFOinside loops or on every retry, or writing whole request and response bodies to the logs, rapidly increases cost and noise. Control it with levels, and sample repeated messages when necessary. - Looking only at averages. Average latency is diluted by large numbers of fast requests and hides the few very slow ones. Always look at the distribution and high percentiles (P95, P99) for latency.
- Recording the health check and metrics endpoints themselves.
/healthzand/metricsare called every few seconds; if they also go into the access log and the RED metrics, the numbers get distorted. They are usually mounted outside the middleware. - Sending sensitive data into the observability system. Logs, span attributes and baggage may be seen by many people and may be sent to third-party platforms. Personal data and secrets must be masked before they are written.
Key takeaways
- The three pillars of observability: logs answer details, metrics answer trends and alerts, and traces answer how time is spread across services. They complement rather than replace each other.
- Use
log/slogto output structured JSON logs, with fixed messages and variables in fields; generate a request ID in a middleware, put it in the context, and add it to every log entry automatically with a custom Handler. - For metrics, use RED for services and USE for resources; use a Histogram for latency, and base alerts mainly on symptoms users can feel.
- Prometheus labels may only hold bounded sets; leave high-cardinality data to logs and traces.
- OpenTelemetry propagates traces between services with
traceparent; in Go this depends on passing context all the way down, including through the message queue. - Sampling should follow the parent's decision to avoid half-recorded traces; call the TracerProvider's
Shutdownbefore the program exits. - Logs carry the trace ID, metrics carry exemplars and spans carry business attributes, so you can jump between the three and follow an alert all the way to its root cause.
Next
Now that the services are visible, two lifecycle-related things are still missing: where configuration comes from and how it is validated, and how to stop during a deployment or scale-down without dropping requests or losing trace data. “Configuration and Graceful Shutdown in Go” also puts the TracerProvider's Shutdown into the shutdown sequence.