This resume is also available in English.

5. Going live and operating2 / 4 · All columns

Columns › 5. Going live and operating › Part 18

Configuration and Graceful Shutdown in Go

設定與優雅關閉

Load configuration the 12-factor way from environment variables, flags and files with validation and careful secret handling, then shut Go services down gracefully with signal.NotifyContext, http.Server.Shutdown, consumer draining and a unified lifecycle.

  • Go
  • Configuration
  • Reliability
In this article
  1. What this article solves
  2. Step 1: The 12-factor configuration principles
  3. Step 2: Read into a struct and validate at startup
  4. Step 3: Handling secrets
  5. Step 4: Receiving the shutdown signal with signal.NotifyContext
  6. Step 5: Graceful shutdown for the HTTP server and the consumer
  7. Step 6: Simplifying the lifecycle with errgroup
  8. Step 7: Readiness and liveness during shutdown
  9. Common pitfalls
  10. Key takeaways
  11. Next

In the previous article, “Observability in Go: Logs, Metrics and Tracing”, we made the state of our services visible with logs, metrics and traces. This article deals with the two ends of a service's lifecycle: where its configuration comes from, and how it leaves gracefully when it stops.

What this article solves

This article handles the two ends of a service's lifecycle: getting the right configuration at startup, and leaving gracefully at shutdown.

Both look trivial, yet they are among the most common sources of incidents in production. On the configuration side, the usual stories are: the database address is hard-coded, so changing environment means recompiling; an environment variable is misspelt, the program starts anyway and only fails on the first request because of an empty string; or a password accidentally gets printed to the logs. On the shutdown side, every deployment, every automatic scale-down and every node maintenance stops old instances. If an instance exits as soon as it receives the signal, in-flight order requests are cut off and users see errors; a queue message being processed is left half done, and in the end has to be cleaned up by the mechanisms from “Idempotency and Distributed Locks”.

The goal of this article: configuration has a single source, a clear order of precedence and is validated at startup; at shutdown the service stops taking new work, finishes the work in hand and releases resources, in that order, with a time limit on the whole process. The examples continue with the order service in cmd/order/main.go: it has an HTTP API, a background consumer for "payment completed" events, and a database connection pool.

Step 1: The 12-factor configuration principles

"The Twelve-Factor App" is a widely cited set of principles for cloud applications, and its rule on configuration is: store config in the environment. Configuration means "things that change between deployment environments", such as database addresses, the URLs of external services, secrets and feature flags; things that do not change with the environment (such as the routing table or business rules) belong to the code and are not configuration.

Separating configuration from code has some direct benefits: the same image can be promoted from the test environment all the way to production with only the environment variables changing; you never take the test database's address to production because you forgot to change a file; and secrets never enter version control.

In practice there are four common configuration sources, usually in this order of precedence from highest to lowest:

SourceGood forNotes
Command-line flagsLocal development, temporary overridesVisible in the process list, so no secrets
Environment variablesMost deployment settingsNatively supported by containers and Kubernetes
Config fileLonger, more structured settingsProvided by a ConfigMap or a mount
Defaults in codeSensible defaultsLets local development start with zero configuration
Highest precedence Command-line flag -addr :9090 Environment variable ORDER_ADDR Config file (ConfigMap mount) Defaults in code Lowest precedence Config struct merged Validated at startup exit on error

Figure: higher sources take precedence; everything is merged into one struct and validated in one go at startup.

One principle is worth stressing: read configuration only once at startup, into a strongly typed struct, and pass only that struct around from then on. Do not call os.Getenv all over the place deep inside business code: that makes it impossible to see at a glance which settings the service needs, impossible to validate them all at startup, and harder to write tests. This matches the spirit of “Layered Architecture and Dependency Injection in Go”: configuration is a dependency too, and should be injected downwards from main.

Step 2: Read into a struct and validate at startup

The complete example below uses only the standard library: it sets defaults first, overrides them with environment variables, and then with flags; it then validates, and if anything is invalid it prints all the errors together and exits with a non-zero code.

package main

import (
	"errors"
	"flag"
	"fmt"
	"log/slog"
	"net/url"
	"os"
	"strconv"
	"time"
)

// Secret is masked automatically in logs and fmt output.
type Secret string

func (Secret) String() string          { return "***" }
func (Secret) LogValue() slog.Value    { return slog.StringValue("***") }
func (s Secret) Reveal() string        { return string(s) }

type Config struct {
	Addr            string
	DatabaseURL     Secret
	InventoryURL    string
	ShutdownTimeout time.Duration
	MaxInFlight     int
}

func load(args []string, getenv func(string) string) (Config, error) {
	// 1. Defaults
	cfg := Config{Addr: ":8080", ShutdownTimeout: 20 * time.Second, MaxInFlight: 100}

	// 2. Environment variable overrides
	var errs []error
	if v := getenv("ORDER_ADDR"); v != "" {
		cfg.Addr = v
	}
	cfg.DatabaseURL = Secret(getenv("ORDER_DATABASE_URL"))
	cfg.InventoryURL = getenv("ORDER_INVENTORY_URL")
	if v := getenv("ORDER_SHUTDOWN_TIMEOUT"); v != "" {
		d, err := time.ParseDuration(v)
		if err != nil {
			errs = append(errs, fmt.Errorf("ORDER_SHUTDOWN_TIMEOUT: %w", err))
		}
		cfg.ShutdownTimeout = d
	}
	if v := getenv("ORDER_MAX_IN_FLIGHT"); v != "" {
		n, err := strconv.Atoi(v)
		if err != nil {
			errs = append(errs, fmt.Errorf("ORDER_MAX_IN_FLIGHT: %w", err))
		}
		cfg.MaxInFlight = n
	}

	// 3. Flag overrides (only flags given explicitly take effect)
	fs := flag.NewFlagSet("order", flag.ContinueOnError)
	fs.StringVar(&cfg.Addr, "addr", cfg.Addr, "HTTP listen address")
	fs.DurationVar(&cfg.ShutdownTimeout, "shutdown-timeout", cfg.ShutdownTimeout, "graceful shutdown limit")
	if err := fs.Parse(args); err != nil {
		return cfg, err
	}

	// 4. Validation: collect every problem in one go
	if cfg.DatabaseURL == "" {
		errs = append(errs, errors.New("ORDER_DATABASE_URL is required"))
	}
	if u, err := url.Parse(cfg.InventoryURL); err != nil || u.Scheme == "" || u.Host == "" {
		errs = append(errs, fmt.Errorf("ORDER_INVENTORY_URL must be a full URL, got %q", cfg.InventoryURL))
	}
	if cfg.ShutdownTimeout <= 0 || cfg.ShutdownTimeout > 5*time.Minute {
		errs = append(errs, errors.New("shutdown timeout must be between 0 and 5 minutes"))
	}
	if cfg.MaxInFlight < 1 {
		errs = append(errs, errors.New("ORDER_MAX_IN_FLIGHT must be at least 1"))
	}
	return cfg, errors.Join(errs...)
}

func main() {
	cfg, err := load(os.Args[1:], os.Getenv)
	if err != nil {
		fmt.Fprintln(os.Stderr, "configuration error:\n"+err.Error())
		os.Exit(2)
	}
	// DatabaseURL is printed as *** and never leaks.
	slog.Info("config loaded", "addr", cfg.Addr, "db", cfg.DatabaseURL,
		"inventory", cfg.InventoryURL, "shutdown_timeout", cfg.ShutdownTimeout)
}

A few design points:

  • load takes args and getenv parameters rather than reading os.Args and os.Getenv directly. In tests you can pass a function wrapping a map instead of actually changing the process's environment variables; the table-driven tests introduced in “Testing in Go: Table-Driven Tests, httptest and Integration Tests” are very handy here.
  • Validation collects all the errors and returns them at once with errors.Join, available since Go 1.20. Compared with "fix one, restart, look at the next error", seeing every problem at once saves a lot of time.
  • A configuration error fails fast. The program exits with a non-zero code during startup, Kubernetes sees the Pod restarting over and over, and the rollout stops at this step while the old version keeps serving. That is far safer than going live with a bad configuration and only finding out when traffic arrives.
  • Environment variables carry a service prefix (ORDER_) to avoid clashing with variables injected by other programs or the platform.

If the configuration struct is large, you might consider a community package that maps environment variables to struct field tags automatically, but the principle stays the same: one struct, validated at startup, injected downwards.

Step 3: Handling secrets

Secrets (database passwords, third-party API keys, private keys for signing) are a special kind of configuration. Besides "not hard-coded in the program", a few extra rules apply:

  1. Never in version control. Even example config files hold only placeholder strings. If one is committed by accident, the right response is to revoke and replace that key immediately, not just to delete it from the latest version, because it is still in the history.
  2. Never in the image. A secret put in with ENV or COPY in a Dockerfile is visible to anyone who can get hold of the image; the next article, “Deploying One Go Service with Docker and Kubernetes”, comes back to this.
  3. Never in logs or error messages. The Secret type in the example above implements both fmt.Stringer and slog.LogValuer, so whether it is printed with %v or handed to slog, it only ever shows ***. Only where the plaintext is truly needed (opening the database connection) do you call Reveal(), which makes "using the plaintext" a visible, searchable action in the code.
  4. Prefer file mounts to environment variables. Environment variables are inherited by child processes and may show up in crash debugging information. A Kubernetes Secret can be mounted as a read-only file that the program reads, which also makes rotation without a restart easier. A common convention is to support ORDER_DATABASE_URL_FILE: if a file path is given, read the file; otherwise read the environment variable.
  5. Least privilege and rotation. Each service gets only the secrets it needs; keys must be replaceable regularly, and the program must be able to accept old and new keys side by side for a while.

Tip: a Kubernetes Secret is by default only base64-encoded, not encrypted. Whether it is encrypted in etcd and who can read Secrets are things the cluster administrator has to configure separately; stricter environments use an external secret management service instead.

Step 4: Receiving the shutdown signal with signal.NotifyContext

Now for the other end of the lifecycle. In a container environment, when the platform wants to stop a process it first sends SIGTERM, waits for a grace period (30 seconds by default in Kubernetes), and only if the process has still not exited does it send SIGKILL to force it. SIGKILL cannot be caught, so the program must finish its clean-up by itself within the grace period. In local development, Ctrl+C sends SIGINT.

signal.NotifyContext, available since Go 1.16, turns "a signal was received" into "the context was cancelled", which lets the shutdown signal plug seamlessly into the cancellation mechanism from “Go context: Cancellation, Timeouts and Request-Scoped Values”:

// Snippet
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()

From then on, as soon as either signal arrives, ctx.Done() is closed. Every long-running component (HTTP server, consumer, scheduled job) derives from this ctx, so they all receive the "time to stop" notice together. stop() restores the signals' default behaviour: after the first Ctrl+C triggers a graceful shutdown, if the shutdown hangs, the user can press it again to end the program outright instead of having it intercepted indefinitely.

Step 5: Graceful shutdown for the HTTP server and the consumer

http.Server.Shutdown(ctx) is the standard library's built-in graceful shutdown: it first closes the listening port so no new connections are accepted, then closes idle connections, and then waits for all in-flight requests to finish before returning. The ctx you pass is the upper bound on that wait; on timeout it returns an error, at which point you can call Close() to force the remaining connections closed.

Note that once Shutdown is called, ListenAndServe returns http.ErrServerClosed immediately, which does not mean the requests have finished. So main must not exit as soon as ListenAndServe returns; it must wait for Shutdown itself to return.

The principle for shutting down a background consumer is the same: stop fetching new messages first, then finish and acknowledge (ack) the one in hand, and only then exit. If processing is interrupted halfway, the message is not acknowledged and the queue redelivers it later; as long as the consumer is idempotent, as described in “Idempotency and Distributed Locks”, processing it twice does no harm. But a graceful shutdown keeps this from happening as much as possible.

The complete example below ties the HTTP server, a simulated consumer and resource clean-up into one unified lifecycle. Each component runs in its own goroutine; an error in any of them triggers an overall shutdown, which runs the clean-up steps in order, with a time limit on the whole thing:

package main

import (
	"context"
	"errors"
	"fmt"
	"log/slog"
	"net/http"
	"os"
	"os/signal"
	"sync"
	"sync/atomic"
	"syscall"
	"time"
)

// consume simulates a queue consumer: once ctx is cancelled it fetches no new messages, but finishes the one in hand.
func consume(ctx context.Context, msgs <-chan string) error {
	for {
		select {
		case <-ctx.Done():
			slog.Info("consumer: stop fetching")
			return nil
		case m := <-msgs:
			// Process with a separate context so the shutdown signal does not interrupt a half-processed message.
			work, cancel := context.WithTimeout(context.WithoutCancel(ctx), 5*time.Second)
			err := handle(work, m)
			cancel()
			if err != nil {
				return fmt.Errorf("handle %s: %w", m, err)
			}
		}
	}
}

func handle(ctx context.Context, m string) error {
	select {
	case <-time.After(300 * time.Millisecond):
		slog.Info("consumer: acked", "msg", m)
		return nil
	case <-ctx.Done():
		return ctx.Err()
	}
}

func main() {
	ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
	defer stop()

	var ready atomic.Bool
	mux := http.NewServeMux()
	mux.HandleFunc("GET /healthz", func(w http.ResponseWriter, _ *http.Request) {
		w.WriteHeader(http.StatusOK) // liveness: return 200 as long as the process is alive
	})
	mux.HandleFunc("GET /readyz", func(w http.ResponseWriter, _ *http.Request) {
		if !ready.Load() {
			http.Error(w, "draining", http.StatusServiceUnavailable)
			return
		}
		w.WriteHeader(http.StatusOK)
	})
	mux.HandleFunc("POST /orders", func(w http.ResponseWriter, _ *http.Request) {
		time.Sleep(500 * time.Millisecond) // simulate writing to the database and calling the inventory service
		w.WriteHeader(http.StatusCreated)
	})
	srv := &http.Server{Addr: ":8080", Handler: mux, ReadHeaderTimeout: 5 * time.Second}

	msgs := make(chan string)
	go func() { // simulate the queue continuously delivering "payment completed" events
		for i := 0; ; i++ {
			select {
			case msgs <- fmt.Sprintf("payment-%d", i):
			case <-ctx.Done():
				return
			}
		}
	}()

	// Lifecycle: if any component fails, cancel runCtx so the others clean up too.
	runCtx, cancel := context.WithCancelCause(ctx)
	var wg sync.WaitGroup
	run := func(name string, fn func() error) {
		wg.Add(1)
		go func() {
			defer wg.Done()
			if err := fn(); err != nil {
				cancel(fmt.Errorf("%s: %w", name, err))
			}
		}()
	}

	run("http", func() error {
		if err := srv.ListenAndServe(); !errors.Is(err, http.ErrServerClosed) {
			return err
		}
		return nil
	})
	run("consumer", func() error { return consume(runCtx, msgs) })
	ready.Store(true)
	slog.Info("started", "addr", srv.Addr)

	<-runCtx.Done()
	slog.Info("shutting down", "cause", context.Cause(runCtx))

	// 1. Fail readiness first, giving the load balancer time to take us out of rotation.
	ready.Store(false)
	time.Sleep(2 * time.Second) // in production, tune to the platform's endpoint update delay

	// 2. The overall shutdown has a limit, which must be less than the platform's grace period.
	shutdownCtx, done := context.WithTimeout(context.Background(), 20*time.Second)
	defer done()
	if err := srv.Shutdown(shutdownCtx); err != nil {
		slog.Error("http shutdown", "err", err)
		_ = srv.Close()
	}

	// 3. Wait for all components to finish (the consumer completes the message in hand).
	finished := make(chan struct{})
	go func() { wg.Wait(); close(finished) }()
	select {
	case <-finished:
	case <-shutdownCtx.Done():
		slog.Error("components did not stop in time")
	}

	// 4. Close shared resources last: the database connection pool, TracerProvider, log buffers.
	slog.Info("bye")
	if err := context.Cause(runCtx); err != nil && !errors.Is(err, context.Canceled) {
		os.Exit(1)
	}
}

A few things in this structure are worth noting. When the consumer processes a single message, it wraps context.WithoutCancel(ctx) in a separate context with a timeout: the shutdown signal is used only to stop "fetching new messages" and does not interrupt the one being processed, yet the processing itself still has a limit and cannot hold up the shutdown indefinitely. context.WithCancelCause lets us record in the logs which component triggered the shutdown and why, which is very useful when troubleshooting. Shared resources are closed only in the last step, because the earlier components still need them while they clean up: if you closed the database connection pool first, in-flight requests would fail at the very last moment.

Step 6: Simplifying the lifecycle with errgroup

The hand-written run function above is essentially the core of golang.org/x/sync/errgroup: errgroup.WithContext returns a group and a derived ctx; when any g.Go returns an error, the ctx is cancelled, and g.Wait() returns the first error. Rewritten with it, the main program becomes more concise:

// Snippet: using golang.org/x/sync/errgroup
g, gctx := errgroup.WithContext(ctx)

g.Go(func() error {
	if err := srv.ListenAndServe(); !errors.Is(err, http.ErrServerClosed) {
		return err
	}
	return nil
})
g.Go(func() error { return consume(gctx, msgs) })
g.Go(func() error {
	<-gctx.Done() // signal received or another component failed
	ready.Store(false)
	time.Sleep(2 * time.Second)
	sctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
	defer cancel()
	return srv.Shutdown(sctx)
})

if err := g.Wait(); err != nil {
	slog.Error("exit", "err", err)
	os.Exit(1)
}

Note that the goroutine "responsible for shutting down the HTTP server" is also a member of the group; it waits for gctx.Done() and then does the clean-up. With this pattern of "every component has a run function and a shutdown function, all handed to the same group", adding a component (such as a gRPC server or an outbox relay) just means adding two pieces following the same pattern.

Step 7: Readiness and liveness during shutdown

On Kubernetes, graceful shutdown also involves when the platform switches traffic. When a Pod is deleted, "sending SIGTERM to the container" and "removing the Pod from the Service's endpoint list" start at the same time and proceed independently. The endpoint list update takes a little while to reach every node's kube-proxy or load balancer, and during that window new requests may still be sent to a Pod that has already started shutting down. If the program closes its listening port the moment it receives SIGTERM, those requests get "connection refused".

That is why the example "fails readiness first, then waits a few seconds". The roles of the two probes must be kept distinct:

ProbeQuestion it asksWhat the platform does on failureDuring shutdown it should
livenessIs the process alive? Is it stuck?Restarts the containerKeep returning 200, or it may be restarted midway through clean-up
readinessCan it take new traffic right now?Removes it from the Service endpoints, no restartReturn 503, so traffic moves away early

Another common approach is to set preStop in the Pod spec, so the platform waits a few seconds before sending SIGTERM. The effect is similar, and no code change is needed:

spec:
  terminationGracePeriodSeconds: 30 # must exceed preStop + the in-program shutdown limit
  containers:
    - name: order
      lifecycle:
        preStop:
          sleep:
            seconds: 5 # supported since Kubernetes 1.30; older versions can use exec to run sleep
      readinessProbe:
        httpGet: { path: /readyz, port: 8080 }
        periodSeconds: 5
      livenessProbe:
        httpGet: { path: /healthz, port: 8080 }
        periodSeconds: 10
SIGTERM / Pod deleted grace period ends → SIGKILL readyz returns 503 wait for endpoint removal preStop / sleep Shutdown stop accepting connections consumer stops fetching Finish current work requests, message acks Release resources connection pool, tracing The total of all stages must be less than terminationGracePeriodSeconds

Figure: from receiving SIGTERM to the process exiting, every stage must complete within the platform's grace period, or the process is killed with SIGKILL.

Work out the time budget carefully: the preStop wait + the endpoint removal buffer + the Shutdown limit + the time to clean up resources must add up to less than terminationGracePeriodSeconds. If the longest request may take 30 seconds, widen the grace period accordingly rather than letting requests be killed midway.

Common pitfalls

  • Calling os.Getenv everywhere. Configuration is scattered, cannot be validated in one go, and you cannot tell which variables the service depends on. Read it into a struct in main and inject it.
  • Silently using an empty string when a value is missing. If a required setting is not provided, startup should fail, rather than you finding out on the first request.
  • Secrets appearing in logs or error messages. Printing the whole config struct with %+v is the most common way they leak; mask them with a dedicated type.
  • Treating ListenAndServe returning as shutdown being complete. It returns immediately after Shutdown is called; only once Shutdown returns can you be sure all requests have finished.
  • Calling Shutdown with context.Background() and no limit. If some long-lived connection never ends, the program waits until it is SIGKILLed, and none of the final clean-up happens.
  • Cleaning up with a cancelled ctx. After the signal arrives the ctx is already cancelled; pass it to the database or to Shutdown and the clean-up fails immediately. Clean-up needs a new context with a timeout.
  • Shutting down in the wrong order. Close the database or messaging connections first and then wait for requests to finish, and in-flight requests fail at the last moment. The order should be: stop taking new work → finish the work in hand → release shared resources.
  • A liveness probe that checks external dependencies. When the database briefly drops, every Pod's liveness fails together and they are all restarted together, which makes the problem bigger. The state of external dependencies belongs in readiness, or only in metrics.
  • Ignoring long-lived connections. WebSockets, Server-Sent Events and streaming gRPC do not end automatically because of Shutdown; you need RegisterOnShutdown or your own signal to tell them to wrap up.

Key takeaways

  • Configuration is "values that change between deployment environments" and, following the 12-factor principles, lives in the environment; the usual order of precedence is flags > environment variables > config file > defaults.
  • At startup, read configuration into a single struct, validate every field in one go, exit with a non-zero code on any error, and then inject it downwards.
  • Secrets stay out of version control, out of images and out of logs; handle them with masking types, file mounts and least privilege.
  • signal.NotifyContext turns SIGTERM/SIGINT into context cancellation, so every component shares the same shutdown notice.
  • Shutdown order: fail readiness first → wait for traffic to move away → http.Server.Shutdown and the consumer stops fetching → finish in-flight work → close shared resources such as the database and tracing.
  • Manage component lifecycles together with errgroup (or an equivalent WaitGroup plus cancellation), so that a failure in any component triggers an overall shutdown.
  • The total time of all shutdown steps must be less than terminationGracePeriodSeconds; liveness keeps succeeding during shutdown, while readiness reports 503.

Next

The service can now read its configuration, shut down gracefully and has /healthz and /readyz ready, but it is still just a program running on a local machine. “Deploying One Go Service with Docker and Kubernetes” packages the order service into a small, secure image, and then uses Kubernetes probes and grace periods to actually connect this article's shutdown process to the platform.