This resume is also available in English.

5. Going live and operating1 / 4 · All columns

Columns › 5. Going live and operating › Part 17

Observability in Go: Logs, Metrics and Tracing

可觀測性:日誌、指標、追蹤

Build observability for Go microservices step by step: structured logs with log/slog and request IDs, RED/USE metrics exposed to Prometheus, OpenTelemetry tracing with context propagation, and correlating all three through a shared trace ID.

  • Go
  • Observability
  • OpenTelemetry
  • Prometheus
In this article
  1. What this article solves
  2. Step 1: Structured logs with log/slog
  3. Step 2: Linking a request's logs with a request ID
  4. Step 3: Deciding what to measure: RED and USE
  5. Step 4: Exporting metrics with Prometheus
  6. Step 5: Basic concepts of distributed tracing
  7. Step 6: Wiring up OpenTelemetry in Go and propagating context
  8. Step 7: Cross-referencing the three pillars
  9. Common pitfalls
  10. Key takeaways
  11. Next

In the previous article, “Distributed Transactions: Sagas and the Outbox Pattern”, we used sagas and the outbox to keep the order, payment and inventory services eventually consistent without a global transaction. This article moves on to "going live and operating": once the services are actually running, how do you see what they are doing?

What this article solves

By now the online shop's order service has grown into four services, api-gateway, order, payment and inventory, and we have dealt with timeouts, retries, idempotency, sagas and the outbox. Once the code is written and the tests pass, the real test begins: after the services go live, how do you know whether they are healthy right now? When a user reports that "placing an order is slow", how do you find which part got slower among four services and a dozen or so instances?

In the monolith days, opening a log file and scrolling down usually turned up a clue. After splitting into microservices, a single order passes through api-gateway, the order service, the inventory service and the payment service, and possibly a message queue too; each part's logs are scattered across different machines with no link between them. At this point "printing more logs" is no longer enough. What you need is systematic observability: the ability for the system to answer "what is happening right now, and why" from the outside.

Observability is usually split into three pillars:

PillarQuestion it answersTypical toolsCost profile
LogsWhat are the details of one particular event?log/slog, Loki, ElasticsearchHigh volume, proportional to request count
MetricsIs the overall trend healthy? Has a threshold been crossed?Prometheus, GrafanaLow volume, proportional to label combinations
TracesHow much time did one request spend in each service?OpenTelemetry, Jaeger, TempoModerate, usually sampled

This article builds all three pillars step by step, and finally ties them together with the same trace ID: from an alerting metric to the matching trace, and from the trace to the logs for that part.

Step 1: Structured logs with log/slog

Since Go 1.21 the standard library has included log/slog, so you no longer need a third-party package for structured logging. Structured means that each log entry is not a sentence for humans to read but a set of "key = value" fields; once it is output as JSON, the logging system can query directly by field, for example "find every entry with order_id=A123" or "count entries with level=ERROR and route=/orders".

The traditional style, such as log.Printf("order %s failed: %v", id, err), is readable to humans, but machines have to pick the string apart with regular expressions, and any change of format breaks the queries. With slog, the same thing is written as logger.Error("create order failed", "order_id", id, "err", err): the message is fixed and everything that varies goes into fields. This habit matters: keep the message text fixed and put variables in fields, so that events of the same kind can be aggregated and counted.

When designing log fields, a few principles are worth settling from the start:

  • Field names are uniform across the whole system: for example always order_id, not orderId in one service and oid in another.
  • Levels must mean something: DEBUG is for chasing details during development, INFO records important state changes, WARN is an anomaly that recovers automatically, and ERROR is a failure someone needs to look at. Production usually outputs INFO and above only.
  • Never write passwords, access tokens or full card numbers to the logs. You can implement the slog.LogValuer interface so that sensitive types decide for themselves what to output (for example only the last four digits).
  • Log a given error only once, in the layer that "handles" it. If every layer does log and then return err, one failure shows up five times in the logs, which makes them harder to read. This was covered in “Error Handling in Go: Wrapping, Sentinels and Custom Types”, and in production the effect becomes even more obvious.

Step 2: Linking a request's logs with a request ID

With structure in place, the next question is: with hundreds of requests running at the same time and their logs interleaved, how do you know which lines belong to the same request? The answer is to give each request an ID as soon as it arrives, and have every log along that request's path carry this field.

In Go, the most natural place for "data specific to this request" is context.Context. As “Go context: Cancellation, Timeouts and Request-Scoped Values” explained, context is suited to values that cross API boundaries and share the request's lifetime. The approach is: a middleware takes the ID from the X-Request-ID header (generating one if the upstream did not provide it) and puts it in the context; then a wrapped slog.Handler pulls the ID from the context and adds it as a field before every log entry is written. Business code then only has to remember to use context-aware methods such as logger.InfoContext(ctx, ...) and never has to pass the ID around itself.

Below is a complete example that uses only the standard library and runs as is. It shows a request ID middleware, an access log that records the status code and duration, and a bare-bones metrics endpoint (Step 4 explains why production should use the Prometheus client package instead):

package main

import (
	"context"
	"crypto/rand"
	"encoding/hex"
	"fmt"
	"log/slog"
	"net/http"
	"os"
	"sync"
	"time"
)

type ctxKey struct{}

// ctxHandler wraps any slog.Handler and adds the request_id from the context before output.
type ctxHandler struct{ slog.Handler }

func (h ctxHandler) Handle(ctx context.Context, r slog.Record) error {
	if id, ok := ctx.Value(ctxKey{}).(string); ok {
		r.AddAttrs(slog.String("request_id", id))
	}
	return h.Handler.Handle(ctx, r)
}

func (h ctxHandler) WithAttrs(as []slog.Attr) slog.Handler {
	return ctxHandler{h.Handler.WithAttrs(as)}
}

func (h ctxHandler) WithGroup(name string) slog.Handler {
	return ctxHandler{h.Handler.WithGroup(name)}
}

func newID() string {
	b := make([]byte, 8)
	_, _ = rand.Read(b)
	return hex.EncodeToString(b)
}

// statusRecorder records the status code actually written.
type statusRecorder struct {
	http.ResponseWriter
	code int
}

func (s *statusRecorder) WriteHeader(code int) {
	s.code = code
	s.ResponseWriter.WriteHeader(code)
}

// counters is a minimal request counter keyed by "route status".
type counters struct {
	mu sync.Mutex
	m  map[string]int
}

func (c *counters) inc(key string) {
	c.mu.Lock()
	c.m[key]++
	c.mu.Unlock()
}

func (c *counters) ServeHTTP(w http.ResponseWriter, _ *http.Request) {
	c.mu.Lock()
	defer c.mu.Unlock()
	w.Header().Set("Content-Type", "text/plain; version=0.0.4")
	fmt.Fprintln(w, "# TYPE http_requests_total counter")
	for k, v := range c.m {
		var route, code string
		fmt.Sscanf(k, "%s %s", &route, &code)
		fmt.Fprintf(w, "http_requests_total{route=%q,code=%q} %d\n", route, code, v)
	}
}

func observe(logger *slog.Logger, cnt *counters, next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		id := r.Header.Get("X-Request-ID")
		if id == "" {
			id = newID()
		}
		ctx := context.WithValue(r.Context(), ctxKey{}, id)
		w.Header().Set("X-Request-ID", id)

		rec := &statusRecorder{ResponseWriter: w, code: http.StatusOK}
		start := time.Now()
		next.ServeHTTP(rec, r.WithContext(ctx))

		route := r.Pattern // Go 1.23+: the matched route pattern, not the actual path
		if route == "" {
			route = "unmatched"
		}
		cnt.inc(fmt.Sprintf("%s %d", route, rec.code))
		logger.InfoContext(ctx, "request done",
			"method", r.Method,
			"route", route,
			"status", rec.code,
			"duration_ms", time.Since(start).Milliseconds(),
		)
	})
}

func main() {
	logger := slog.New(ctxHandler{slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{Level: slog.LevelInfo})})
	slog.SetDefault(logger)

	cnt := &counters{m: map[string]int{}}
	mux := http.NewServeMux()
	mux.HandleFunc("GET /orders/{id}", func(w http.ResponseWriter, r *http.Request) {
		slog.InfoContext(r.Context(), "load order", "order_id", r.PathValue("id"))
		fmt.Fprintf(w, `{"id":%q,"status":"paid"}`, r.PathValue("id"))
	})

	root := http.NewServeMux()
	root.Handle("GET /metrics", cnt)
	root.Handle("/", observe(logger, cnt, mux))

	logger.Info("listening", "addr", ":8080")
	if err := http.ListenAndServe(":8080", root); err != nil {
		logger.Error("server stopped", "err", err)
		os.Exit(1)
	}
}

There is an easily overlooked detail here: route takes r.Pattern (filled in by ServeMux since Go 1.23, e.g. GET /orders/{id}), not r.URL.Path (e.g. /orders/A123). The former has a bounded set of values; the latter is different for every order. Putting the actual path in logs is fine, but as soon as you use it as a metric label, you get the "label explosion" discussed in Step 4.

A request ID only links things up "within one service". When the order service calls the inventory service, it must also put X-Request-ID into the headers of the outgoing request, so that the inventory service's middleware reuses the same ID. In Step 6, OpenTelemetry's propagation mechanism takes this over in a standard way.

Step 3: Deciding what to measure: RED and USE

Logs are good for looking up details but poor at answering overall questions such as "what was the error rate in the past hour": computing a ratio from millions of log lines in real time is expensive. Questions like that belong to metrics: the program maintains some numbers (counters, gauges, distributions) that are collected periodically and stored as time series, making queries and alerts cheap.

The hardest part of metrics is not how to write them but "what to measure". Two classic methods make good checklists:

MethodApplies toThree metricsExample in the order service
REDServices that handle requestsRate (requests/s), Errors (failures), Duration (latency distribution)POST /orders per second, number of 5xx, P99 latency
USEResources (CPU, connection pools, queues)Utilization, Saturation, ErrorsShare of the database connection pool in use, number waiting for a connection, acquisition timeouts

RED looks at a service from the user's point of view: as long as these three numbers are normal, users usually notice nothing. USE looks at bottlenecks from the resource's point of view: when RED shows things slowing down, USE helps you answer "which resource is the bottleneck". Taking the connection pool introduced in “database/sql, Transactions and Connection Pools in Go” as an example, sql.DB.Stats() already provides values such as InUse, WaitCount and WaitDuration; export them periodically as metrics and you have USE out of the box.

In a Go service there are also a few "runtime-level" metrics you almost always want: the number of goroutines, heap memory size and GC pause time. A goroutine count that keeps rising often signals a leak (for example a channel nobody ever reads), the classic problem mentioned in “Goroutines, Channels and Concurrency Patterns in Go”; on a metrics chart it shows up as a line that only ever goes up.

Metrics trends and alerts RED / USE, Prometheus Traces cross-service latency spans, OpenTelemetry Logs single-event detail log/slog, JSON trace ID / request ID passed down through context exemplar span attributes trace_id field

Figure: each pillar answers a different question, and the request ID and trace ID link them into one path from alert to root cause.

Step 4: Exporting metrics with Prometheus

Prometheus is the most common metrics system today. It uses a "pull" model: the service exposes a /metrics endpoint that lists all current metrics in a plain-text format, and Prometheus scrapes it at regular intervals (for example every 15 seconds). The example in Step 2 wrote this format by hand, but in production use the official github.com/prometheus/client_golang, which handles concurrency safety, histogram buckets, runtime metrics and other details.

Prometheus has four metric types: Counter (only goes up, e.g. total requests), Gauge (goes up and down, e.g. current queue length), Histogram (places observations into predefined buckets so percentiles can be computed on the server side) and Summary (computes percentiles on the client side, cannot be merged across instances, generally less used). RED's Duration almost always uses a Histogram, because histograms from several instances can be added together before computing the overall P99.

// Snippet: using github.com/prometheus/client_golang
import (
	"net/http"
	"strconv"
	"time"

	"github.com/prometheus/client_golang/prometheus"
	"github.com/prometheus/client_golang/prometheus/promauto"
	"github.com/prometheus/client_golang/prometheus/promhttp"
)

var reqDuration = promauto.NewHistogramVec(prometheus.HistogramOpts{
	Name:    "http_request_duration_seconds",
	Help:    "HTTP request duration (seconds)",
	Buckets: []float64{0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2.5},
}, []string{"method", "route", "code"})

func metricsMiddleware(next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		rec := &statusRecorder{ResponseWriter: w, code: http.StatusOK}
		start := time.Now()
		next.ServeHTTP(rec, r)
		reqDuration.WithLabelValues(r.Method, r.Pattern, strconv.Itoa(rec.code)).
			Observe(time.Since(start).Seconds())
	})
}

// In main: hand /metrics to promhttp, which by default also outputs runtime metrics such as go_goroutines and go_memstats_*.
// mux.Handle("GET /metrics", promhttp.Handler())

The text Prometheus scrapes looks roughly like this; each line is one time series, and the key-value pairs inside {} are the labels:

# HELP http_request_duration_seconds HTTP request duration (seconds)
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{code="201",method="POST",route="POST /orders",le="0.1"} 940
http_request_duration_seconds_bucket{code="201",method="POST",route="POST /orders",le="0.25"} 990
http_request_duration_seconds_bucket{code="201",method="POST",route="POST /orders",le="+Inf"} 1000
http_request_duration_seconds_sum{code="201",method="POST",route="POST /orders"} 61.2
http_request_duration_seconds_count{code="201",method="POST",route="POST /orders"} 1000
go_goroutines 42

(The numbers are illustrative only.) With these series you can write RED in PromQL: requests per second is rate(http_request_duration_seconds_count[5m]), the error rate is the rate of 5xx divided by the total rate, and P99 is histogram_quantile(0.99, sum by (le, route) (rate(http_request_duration_seconds_bucket[5m]))).

Tip: every distinct combination of label values is a separate time series. Labels may only hold things with a "bounded set of values", such as the method, route pattern and status code; user IDs, order numbers, full URLs and full error messages should never be labels. High-cardinality data like that belongs in logs or traces.

Alerts should be based on "symptoms" rather than "causes": rather than alerting on "CPU above 80%", alert on "POST /orders error rate above the threshold for five consecutive minutes" or "P99 above the service objective". High CPU that users do not notice is no reason to wake someone at night; conversely, a rising error rate always needs someone to deal with it. This is also the basic idea behind SLOs (service level objectives): first define the objectives users care about, then use metrics to track whether you are drifting from them.

Step 5: Basic concepts of distributed tracing

Metrics tell you "the P99 of POST /orders has got worse", but not where the time is going. A single order might involve: the order service writing to the database, then synchronously calling the inventory service to reserve stock, then calling the payment service for authorisation, and finally sending an event to the message queue. Which part got slower? That is the question distributed tracing answers.

Tracing has a few core terms:

  • Trace: one complete journey of a request, with a globally unique trace ID.
  • Span: one piece of work within the journey, such as "handle the HTTP request", "run an SQL statement" or "call the inventory service". Each span has its own span ID, start and end times, attributes, and the ID of its parent span.
  • Context propagation: when service A calls service B, it passes the trace ID and the current span ID in the request headers, so the spans B creates hang under A's span. The W3C traceparent header is the common format today, and looks like 00-<32-hex-digit trace id>-<16-hex-digit parent span id>-01.

Lay out all the spans with the same trace ID along a time axis and you get a "waterfall", where the longest part is obvious at a glance:

span 0 ms time → gateway: POST /orders order: create order SQL: insert orders inventory: reserve stock SQL: wait for conn payment: authorise longest span

Figure: the spans under one trace are arranged by parent-child relationship and time; the longest one is the first thing to investigate.

OpenTelemetry (OTel for short) is an open standard from the CNCF that unifies the APIs, SDKs and transport protocol (OTLP) for traces, metrics and logs. Your code depends only on the OTel API; whether the data goes to Jaeger, Tempo or a commercial platform is just a matter of configuration, with no code changes.

Step 6: Wiring up OpenTelemetry in Go and propagating context

Wiring up OTel usually comes in three parts: at startup, create a TracerProvider and configure an exporter; add automatic instrumentation to the HTTP/gRPC servers and clients, so that spans are created and traceparent is injected and parsed automatically; and create spans by hand for key business steps.

// Snippet: using go.opentelemetry.io/otel and contrib's otelhttp
import (
	"context"
	"net/http"

	"go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/attribute"
	"go.opentelemetry.io/otel/codes"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
	"go.opentelemetry.io/otel/propagation"
	"go.opentelemetry.io/otel/sdk/resource"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
	semconv "go.opentelemetry.io/otel/semconv/v1.26.0"
)

func setupTracing(ctx context.Context) (func(context.Context) error, error) {
	exp, err := otlptracegrpc.New(ctx) // endpoint set by OTEL_EXPORTER_OTLP_ENDPOINT
	if err != nil {
		return nil, err
	}
	tp := sdktrace.NewTracerProvider(
		sdktrace.WithBatcher(exp),
		sdktrace.WithSampler(sdktrace.ParentBased(sdktrace.TraceIDRatioBased(0.1))),
		sdktrace.WithResource(resource.NewWithAttributes(semconv.SchemaURL,
			semconv.ServiceName("order"))),
	)
	otel.SetTracerProvider(tp)
	otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(
		propagation.TraceContext{}, propagation.Baggage{}))
	return tp.Shutdown, nil // must be called on shutdown to flush buffered spans
}

// Server side: create a span for every request automatically and join the upstream trace from the traceparent header.
// handler := otelhttp.NewHandler(mux, "order")

// Client side: write traceparent into the headers automatically when calling the inventory service.
var client = &http.Client{Transport: otelhttp.NewTransport(http.DefaultTransport)}

func reserveStock(ctx context.Context, orderID string) error {
	ctx, span := otel.Tracer("order").Start(ctx, "reserve-stock")
	defer span.End()
	span.SetAttributes(attribute.String("order.id", orderID))

	req, err := http.NewRequestWithContext(ctx, http.MethodPost, "http://inventory/reservations", nil)
	if err != nil {
		return err
	}
	resp, err := client.Do(req)
	if err != nil {
		span.RecordError(err)
		span.SetStatus(codes.Error, "reserve failed")
		return err
	}
	defer resp.Body.Close()
	return nil
}

The key to this code is that context is passed all the way down. otelhttp.NewHandler puts the span into the request's context; reserveStock opens a child span from that context; http.NewRequestWithContext carries the child span into the outgoing request, and otelhttp.NewTransport takes it out and writes it into the headers. If any layer in between switches to context.Background(), the trace breaks in two at that point. This is the practical payoff of “Go context: Cancellation, Timeouts and Request-Scoped Values” insisting again and again that "context is passed down as the first argument".

Asynchronous paths must propagate it too. When the order service sends an "order created" event through the message queue, it should write traceparent into the message headers (Kafka headers or AMQP properties); the consumer extracts it with propagator.Extract to restore the context and then opens a span. That way the asynchronous flows in “Message Queues and Event-Driven Design” and “Distributed Transactions: Sagas and the Outbox Pattern” also appear in the same waterfall. If you use an outbox, remember to store traceparent in the outbox table along with the event content, so the relay has it when it publishes.

Sampling is the other important point about tracing. Recording a full trace for every request is expensive in both data volume and cost, so usually only a portion is kept. The ParentBased(TraceIDRatioBased(0.1)) above means: if the upstream has already decided whether to sample, follow its decision (so a trace is never recorded only halfway); when this service is the starting point, keep about one in ten based on the trace ID. A more advanced approach is "tail sampling" in the OTel Collector: wait until the whole trace has finished, then decide to keep every trace that errored or was particularly slow.

Step 7: Cross-referencing the three pillars

Once each pillar is built, the real value lies in being able to jump between them. There are three ways to do this:

  1. Logs carry the trace ID and span ID. Extend the ctxHandler from Step 2 slightly: besides request_id, use trace.SpanContextFromContext(ctx) to take TraceID() and SpanID() from the context and, when valid, write them as trace_id and span_id fields. The logging platform can then open the matching trace straight from an error log. You can also use OTel's slog bridge package to send logs directly over OTLP, with these fields added automatically.
  2. Metrics carry exemplars. Prometheus histograms support exemplars: a trace ID of a representative request attached next to a bucket. Grafana draws these points on the latency chart, and clicking one jumps to the trace of that slow request, taking you in one step from "the P99 went up" to "it was this request, and it was slow in this part".
  3. Spans carry business attributes. Add attributes such as order.id to spans, and you can find traces the other way round, by business data: when support says "order A123 is stuck", you can search for that attribute directly.

A typical troubleshooting flow therefore becomes: an alert reports that the P99 of POST /orders is over the limit → click an exemplar on the dashboard → the waterfall shows that the inventory: reserve span takes most of the time → jump to the inventory service's logs under this trace ID → see "timed out waiting for a database connection" → go back to the USE metrics and confirm that the connection pool is saturated. From symptom to root cause, every step has a lead to follow, instead of digging through files on several machines one by one.

Alert P99 above target Exemplar pick a slow request Trace waterfall find the longest span Logs same trace_id USE metrics confirm the root cause e.g. connection pool saturated, waits rising

Figure: starting from a symptom metric, exemplars, traces and logs narrow things down step by step to a specific resource bottleneck.

Common pitfalls

  • Label explosion. Use user IDs, order numbers or raw URLs as metric labels and the number of time series grows without limit, with Prometheus's memory ballooning along with it. Labels hold bounded sets only; for routes use patterns such as r.Pattern.
  • Broken context. Switching to context.Background() inside a goroutine, a background job or a queue consumer breaks the trace. When you need to detach from the request's cancellation but keep the tracing information, use context.WithoutCancel(ctx), available since Go 1.21.
  • Forgetting to shut down the TracerProvider. The SDK buffers spans in batches; if Shutdown is not called before the program exits, the last batch is lost, and it is precisely the most important part, right before things went wrong. The next article, “Configuration and Graceful Shutdown in Go”, puts it into the shutdown sequence.
  • Runaway log volume. Logging INFO inside loops or on every retry, or writing whole request and response bodies to the logs, rapidly increases cost and noise. Control it with levels, and sample repeated messages when necessary.
  • Looking only at averages. Average latency is diluted by large numbers of fast requests and hides the few very slow ones. Always look at the distribution and high percentiles (P95, P99) for latency.
  • Recording the health check and metrics endpoints themselves. /healthz and /metrics are called every few seconds; if they also go into the access log and the RED metrics, the numbers get distorted. They are usually mounted outside the middleware.
  • Sending sensitive data into the observability system. Logs, span attributes and baggage may be seen by many people and may be sent to third-party platforms. Personal data and secrets must be masked before they are written.

Key takeaways

  • The three pillars of observability: logs answer details, metrics answer trends and alerts, and traces answer how time is spread across services. They complement rather than replace each other.
  • Use log/slog to output structured JSON logs, with fixed messages and variables in fields; generate a request ID in a middleware, put it in the context, and add it to every log entry automatically with a custom Handler.
  • For metrics, use RED for services and USE for resources; use a Histogram for latency, and base alerts mainly on symptoms users can feel.
  • Prometheus labels may only hold bounded sets; leave high-cardinality data to logs and traces.
  • OpenTelemetry propagates traces between services with traceparent; in Go this depends on passing context all the way down, including through the message queue.
  • Sampling should follow the parent's decision to avoid half-recorded traces; call the TracerProvider's Shutdown before the program exits.
  • Logs carry the trace ID, metrics carry exemplars and spans carry business attributes, so you can jump between the three and follow an alert all the way to its root cause.

Next

Now that the services are visible, two lifecycle-related things are still missing: where configuration comes from and how it is validated, and how to stop during a deployment or scale-down without dropping requests or losing trace data. “Configuration and Graceful Shutdown in Go” also puts the TracerProvider's Shutdown into the shutdown sequence.