Columns › 5. Going live and operating › Part 18
Configuration and Graceful Shutdown in Go
設定與優雅關閉
Load configuration the 12-factor way from environment variables, flags and files with validation and careful secret handling, then shut Go services down gracefully with signal.NotifyContext, http.Server.Shutdown, consumer draining and a unified lifecycle.
In this article
- What this article solves
- Step 1: The 12-factor configuration principles
- Step 2: Read into a struct and validate at startup
- Step 3: Handling secrets
- Step 4: Receiving the shutdown signal with signal.NotifyContext
- Step 5: Graceful shutdown for the HTTP server and the consumer
- Step 6: Simplifying the lifecycle with errgroup
- Step 7: Readiness and liveness during shutdown
- Common pitfalls
- Key takeaways
- Next
In the previous article, “Observability in Go: Logs, Metrics and Tracing”, we made the state of our services visible with logs, metrics and traces. This article deals with the two ends of a service's lifecycle: where its configuration comes from, and how it leaves gracefully when it stops.
What this article solves
This article handles the two ends of a service's lifecycle: getting the right configuration at startup, and leaving gracefully at shutdown.
Both look trivial, yet they are among the most common sources of incidents in production. On the configuration side, the usual stories are: the database address is hard-coded, so changing environment means recompiling; an environment variable is misspelt, the program starts anyway and only fails on the first request because of an empty string; or a password accidentally gets printed to the logs. On the shutdown side, every deployment, every automatic scale-down and every node maintenance stops old instances. If an instance exits as soon as it receives the signal, in-flight order requests are cut off and users see errors; a queue message being processed is left half done, and in the end has to be cleaned up by the mechanisms from “Idempotency and Distributed Locks”.
The goal of this article: configuration has a single source, a clear order of precedence and is validated at startup; at shutdown the service stops taking new work, finishes the work in hand and releases resources, in that order, with a time limit on the whole process. The examples continue with the order service in cmd/order/main.go: it has an HTTP API, a background consumer for "payment completed" events, and a database connection pool.
Step 1: The 12-factor configuration principles
"The Twelve-Factor App" is a widely cited set of principles for cloud applications, and its rule on configuration is: store config in the environment. Configuration means "things that change between deployment environments", such as database addresses, the URLs of external services, secrets and feature flags; things that do not change with the environment (such as the routing table or business rules) belong to the code and are not configuration.
Separating configuration from code has some direct benefits: the same image can be promoted from the test environment all the way to production with only the environment variables changing; you never take the test database's address to production because you forgot to change a file; and secrets never enter version control.
In practice there are four common configuration sources, usually in this order of precedence from highest to lowest:
| Source | Good for | Notes |
|---|---|---|
| Command-line flags | Local development, temporary overrides | Visible in the process list, so no secrets |
| Environment variables | Most deployment settings | Natively supported by containers and Kubernetes |
| Config file | Longer, more structured settings | Provided by a ConfigMap or a mount |
| Defaults in code | Sensible defaults | Lets local development start with zero configuration |
Figure: higher sources take precedence; everything is merged into one struct and validated in one go at startup.
One principle is worth stressing: read configuration only once at startup, into a strongly typed struct, and pass only that struct around from then on. Do not call os.Getenv all over the place deep inside business code: that makes it impossible to see at a glance which settings the service needs, impossible to validate them all at startup, and harder to write tests. This matches the spirit of “Layered Architecture and Dependency Injection in Go”: configuration is a dependency too, and should be injected downwards from main.
Step 2: Read into a struct and validate at startup
The complete example below uses only the standard library: it sets defaults first, overrides them with environment variables, and then with flags; it then validates, and if anything is invalid it prints all the errors together and exits with a non-zero code.
package main
import (
"errors"
"flag"
"fmt"
"log/slog"
"net/url"
"os"
"strconv"
"time"
)
// Secret is masked automatically in logs and fmt output.
type Secret string
func (Secret) String() string { return "***" }
func (Secret) LogValue() slog.Value { return slog.StringValue("***") }
func (s Secret) Reveal() string { return string(s) }
type Config struct {
Addr string
DatabaseURL Secret
InventoryURL string
ShutdownTimeout time.Duration
MaxInFlight int
}
func load(args []string, getenv func(string) string) (Config, error) {
// 1. Defaults
cfg := Config{Addr: ":8080", ShutdownTimeout: 20 * time.Second, MaxInFlight: 100}
// 2. Environment variable overrides
var errs []error
if v := getenv("ORDER_ADDR"); v != "" {
cfg.Addr = v
}
cfg.DatabaseURL = Secret(getenv("ORDER_DATABASE_URL"))
cfg.InventoryURL = getenv("ORDER_INVENTORY_URL")
if v := getenv("ORDER_SHUTDOWN_TIMEOUT"); v != "" {
d, err := time.ParseDuration(v)
if err != nil {
errs = append(errs, fmt.Errorf("ORDER_SHUTDOWN_TIMEOUT: %w", err))
}
cfg.ShutdownTimeout = d
}
if v := getenv("ORDER_MAX_IN_FLIGHT"); v != "" {
n, err := strconv.Atoi(v)
if err != nil {
errs = append(errs, fmt.Errorf("ORDER_MAX_IN_FLIGHT: %w", err))
}
cfg.MaxInFlight = n
}
// 3. Flag overrides (only flags given explicitly take effect)
fs := flag.NewFlagSet("order", flag.ContinueOnError)
fs.StringVar(&cfg.Addr, "addr", cfg.Addr, "HTTP listen address")
fs.DurationVar(&cfg.ShutdownTimeout, "shutdown-timeout", cfg.ShutdownTimeout, "graceful shutdown limit")
if err := fs.Parse(args); err != nil {
return cfg, err
}
// 4. Validation: collect every problem in one go
if cfg.DatabaseURL == "" {
errs = append(errs, errors.New("ORDER_DATABASE_URL is required"))
}
if u, err := url.Parse(cfg.InventoryURL); err != nil || u.Scheme == "" || u.Host == "" {
errs = append(errs, fmt.Errorf("ORDER_INVENTORY_URL must be a full URL, got %q", cfg.InventoryURL))
}
if cfg.ShutdownTimeout <= 0 || cfg.ShutdownTimeout > 5*time.Minute {
errs = append(errs, errors.New("shutdown timeout must be between 0 and 5 minutes"))
}
if cfg.MaxInFlight < 1 {
errs = append(errs, errors.New("ORDER_MAX_IN_FLIGHT must be at least 1"))
}
return cfg, errors.Join(errs...)
}
func main() {
cfg, err := load(os.Args[1:], os.Getenv)
if err != nil {
fmt.Fprintln(os.Stderr, "configuration error:\n"+err.Error())
os.Exit(2)
}
// DatabaseURL is printed as *** and never leaks.
slog.Info("config loaded", "addr", cfg.Addr, "db", cfg.DatabaseURL,
"inventory", cfg.InventoryURL, "shutdown_timeout", cfg.ShutdownTimeout)
}A few design points:
loadtakesargsandgetenvparameters rather than readingos.Argsandos.Getenvdirectly. In tests you can pass a function wrapping a map instead of actually changing the process's environment variables; the table-driven tests introduced in “Testing in Go: Table-Driven Tests, httptest and Integration Tests” are very handy here.- Validation collects all the errors and returns them at once with
errors.Join, available since Go 1.20. Compared with "fix one, restart, look at the next error", seeing every problem at once saves a lot of time. - A configuration error fails fast. The program exits with a non-zero code during startup, Kubernetes sees the Pod restarting over and over, and the rollout stops at this step while the old version keeps serving. That is far safer than going live with a bad configuration and only finding out when traffic arrives.
- Environment variables carry a service prefix (
ORDER_) to avoid clashing with variables injected by other programs or the platform.
If the configuration struct is large, you might consider a community package that maps environment variables to struct field tags automatically, but the principle stays the same: one struct, validated at startup, injected downwards.
Step 3: Handling secrets
Secrets (database passwords, third-party API keys, private keys for signing) are a special kind of configuration. Besides "not hard-coded in the program", a few extra rules apply:
- Never in version control. Even example config files hold only placeholder strings. If one is committed by accident, the right response is to revoke and replace that key immediately, not just to delete it from the latest version, because it is still in the history.
- Never in the image. A secret put in with
ENVorCOPYin a Dockerfile is visible to anyone who can get hold of the image; the next article, “Deploying One Go Service with Docker and Kubernetes”, comes back to this. - Never in logs or error messages. The
Secrettype in the example above implements bothfmt.Stringerandslog.LogValuer, so whether it is printed with%vor handed to slog, it only ever shows***. Only where the plaintext is truly needed (opening the database connection) do you callReveal(), which makes "using the plaintext" a visible, searchable action in the code. - Prefer file mounts to environment variables. Environment variables are inherited by child processes and may show up in crash debugging information. A Kubernetes Secret can be mounted as a read-only file that the program reads, which also makes rotation without a restart easier. A common convention is to support
ORDER_DATABASE_URL_FILE: if a file path is given, read the file; otherwise read the environment variable. - Least privilege and rotation. Each service gets only the secrets it needs; keys must be replaceable regularly, and the program must be able to accept old and new keys side by side for a while.
Tip: a Kubernetes Secret is by default only base64-encoded, not encrypted. Whether it is encrypted in etcd and who can read Secrets are things the cluster administrator has to configure separately; stricter environments use an external secret management service instead.
Step 4: Receiving the shutdown signal with signal.NotifyContext
Now for the other end of the lifecycle. In a container environment, when the platform wants to stop a process it first sends SIGTERM, waits for a grace period (30 seconds by default in Kubernetes), and only if the process has still not exited does it send SIGKILL to force it. SIGKILL cannot be caught, so the program must finish its clean-up by itself within the grace period. In local development, Ctrl+C sends SIGINT.
signal.NotifyContext, available since Go 1.16, turns "a signal was received" into "the context was cancelled", which lets the shutdown signal plug seamlessly into the cancellation mechanism from “Go context: Cancellation, Timeouts and Request-Scoped Values”:
// Snippet
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()From then on, as soon as either signal arrives, ctx.Done() is closed. Every long-running component (HTTP server, consumer, scheduled job) derives from this ctx, so they all receive the "time to stop" notice together. stop() restores the signals' default behaviour: after the first Ctrl+C triggers a graceful shutdown, if the shutdown hangs, the user can press it again to end the program outright instead of having it intercepted indefinitely.
Step 5: Graceful shutdown for the HTTP server and the consumer
http.Server.Shutdown(ctx) is the standard library's built-in graceful shutdown: it first closes the listening port so no new connections are accepted, then closes idle connections, and then waits for all in-flight requests to finish before returning. The ctx you pass is the upper bound on that wait; on timeout it returns an error, at which point you can call Close() to force the remaining connections closed.
Note that once Shutdown is called, ListenAndServe returns http.ErrServerClosed immediately, which does not mean the requests have finished. So main must not exit as soon as ListenAndServe returns; it must wait for Shutdown itself to return.
The principle for shutting down a background consumer is the same: stop fetching new messages first, then finish and acknowledge (ack) the one in hand, and only then exit. If processing is interrupted halfway, the message is not acknowledged and the queue redelivers it later; as long as the consumer is idempotent, as described in “Idempotency and Distributed Locks”, processing it twice does no harm. But a graceful shutdown keeps this from happening as much as possible.
The complete example below ties the HTTP server, a simulated consumer and resource clean-up into one unified lifecycle. Each component runs in its own goroutine; an error in any of them triggers an overall shutdown, which runs the clean-up steps in order, with a time limit on the whole thing:
package main
import (
"context"
"errors"
"fmt"
"log/slog"
"net/http"
"os"
"os/signal"
"sync"
"sync/atomic"
"syscall"
"time"
)
// consume simulates a queue consumer: once ctx is cancelled it fetches no new messages, but finishes the one in hand.
func consume(ctx context.Context, msgs <-chan string) error {
for {
select {
case <-ctx.Done():
slog.Info("consumer: stop fetching")
return nil
case m := <-msgs:
// Process with a separate context so the shutdown signal does not interrupt a half-processed message.
work, cancel := context.WithTimeout(context.WithoutCancel(ctx), 5*time.Second)
err := handle(work, m)
cancel()
if err != nil {
return fmt.Errorf("handle %s: %w", m, err)
}
}
}
}
func handle(ctx context.Context, m string) error {
select {
case <-time.After(300 * time.Millisecond):
slog.Info("consumer: acked", "msg", m)
return nil
case <-ctx.Done():
return ctx.Err()
}
}
func main() {
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()
var ready atomic.Bool
mux := http.NewServeMux()
mux.HandleFunc("GET /healthz", func(w http.ResponseWriter, _ *http.Request) {
w.WriteHeader(http.StatusOK) // liveness: return 200 as long as the process is alive
})
mux.HandleFunc("GET /readyz", func(w http.ResponseWriter, _ *http.Request) {
if !ready.Load() {
http.Error(w, "draining", http.StatusServiceUnavailable)
return
}
w.WriteHeader(http.StatusOK)
})
mux.HandleFunc("POST /orders", func(w http.ResponseWriter, _ *http.Request) {
time.Sleep(500 * time.Millisecond) // simulate writing to the database and calling the inventory service
w.WriteHeader(http.StatusCreated)
})
srv := &http.Server{Addr: ":8080", Handler: mux, ReadHeaderTimeout: 5 * time.Second}
msgs := make(chan string)
go func() { // simulate the queue continuously delivering "payment completed" events
for i := 0; ; i++ {
select {
case msgs <- fmt.Sprintf("payment-%d", i):
case <-ctx.Done():
return
}
}
}()
// Lifecycle: if any component fails, cancel runCtx so the others clean up too.
runCtx, cancel := context.WithCancelCause(ctx)
var wg sync.WaitGroup
run := func(name string, fn func() error) {
wg.Add(1)
go func() {
defer wg.Done()
if err := fn(); err != nil {
cancel(fmt.Errorf("%s: %w", name, err))
}
}()
}
run("http", func() error {
if err := srv.ListenAndServe(); !errors.Is(err, http.ErrServerClosed) {
return err
}
return nil
})
run("consumer", func() error { return consume(runCtx, msgs) })
ready.Store(true)
slog.Info("started", "addr", srv.Addr)
<-runCtx.Done()
slog.Info("shutting down", "cause", context.Cause(runCtx))
// 1. Fail readiness first, giving the load balancer time to take us out of rotation.
ready.Store(false)
time.Sleep(2 * time.Second) // in production, tune to the platform's endpoint update delay
// 2. The overall shutdown has a limit, which must be less than the platform's grace period.
shutdownCtx, done := context.WithTimeout(context.Background(), 20*time.Second)
defer done()
if err := srv.Shutdown(shutdownCtx); err != nil {
slog.Error("http shutdown", "err", err)
_ = srv.Close()
}
// 3. Wait for all components to finish (the consumer completes the message in hand).
finished := make(chan struct{})
go func() { wg.Wait(); close(finished) }()
select {
case <-finished:
case <-shutdownCtx.Done():
slog.Error("components did not stop in time")
}
// 4. Close shared resources last: the database connection pool, TracerProvider, log buffers.
slog.Info("bye")
if err := context.Cause(runCtx); err != nil && !errors.Is(err, context.Canceled) {
os.Exit(1)
}
}A few things in this structure are worth noting. When the consumer processes a single message, it wraps context.WithoutCancel(ctx) in a separate context with a timeout: the shutdown signal is used only to stop "fetching new messages" and does not interrupt the one being processed, yet the processing itself still has a limit and cannot hold up the shutdown indefinitely. context.WithCancelCause lets us record in the logs which component triggered the shutdown and why, which is very useful when troubleshooting. Shared resources are closed only in the last step, because the earlier components still need them while they clean up: if you closed the database connection pool first, in-flight requests would fail at the very last moment.
Step 6: Simplifying the lifecycle with errgroup
The hand-written run function above is essentially the core of golang.org/x/sync/errgroup: errgroup.WithContext returns a group and a derived ctx; when any g.Go returns an error, the ctx is cancelled, and g.Wait() returns the first error. Rewritten with it, the main program becomes more concise:
// Snippet: using golang.org/x/sync/errgroup
g, gctx := errgroup.WithContext(ctx)
g.Go(func() error {
if err := srv.ListenAndServe(); !errors.Is(err, http.ErrServerClosed) {
return err
}
return nil
})
g.Go(func() error { return consume(gctx, msgs) })
g.Go(func() error {
<-gctx.Done() // signal received or another component failed
ready.Store(false)
time.Sleep(2 * time.Second)
sctx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
defer cancel()
return srv.Shutdown(sctx)
})
if err := g.Wait(); err != nil {
slog.Error("exit", "err", err)
os.Exit(1)
}Note that the goroutine "responsible for shutting down the HTTP server" is also a member of the group; it waits for gctx.Done() and then does the clean-up. With this pattern of "every component has a run function and a shutdown function, all handed to the same group", adding a component (such as a gRPC server or an outbox relay) just means adding two pieces following the same pattern.
Step 7: Readiness and liveness during shutdown
On Kubernetes, graceful shutdown also involves when the platform switches traffic. When a Pod is deleted, "sending SIGTERM to the container" and "removing the Pod from the Service's endpoint list" start at the same time and proceed independently. The endpoint list update takes a little while to reach every node's kube-proxy or load balancer, and during that window new requests may still be sent to a Pod that has already started shutting down. If the program closes its listening port the moment it receives SIGTERM, those requests get "connection refused".
That is why the example "fails readiness first, then waits a few seconds". The roles of the two probes must be kept distinct:
| Probe | Question it asks | What the platform does on failure | During shutdown it should |
|---|---|---|---|
| liveness | Is the process alive? Is it stuck? | Restarts the container | Keep returning 200, or it may be restarted midway through clean-up |
| readiness | Can it take new traffic right now? | Removes it from the Service endpoints, no restart | Return 503, so traffic moves away early |
Another common approach is to set preStop in the Pod spec, so the platform waits a few seconds before sending SIGTERM. The effect is similar, and no code change is needed:
spec:
terminationGracePeriodSeconds: 30 # must exceed preStop + the in-program shutdown limit
containers:
- name: order
lifecycle:
preStop:
sleep:
seconds: 5 # supported since Kubernetes 1.30; older versions can use exec to run sleep
readinessProbe:
httpGet: { path: /readyz, port: 8080 }
periodSeconds: 5
livenessProbe:
httpGet: { path: /healthz, port: 8080 }
periodSeconds: 10Figure: from receiving SIGTERM to the process exiting, every stage must complete within the platform's grace period, or the process is killed with SIGKILL.
Work out the time budget carefully: the preStop wait + the endpoint removal buffer + the Shutdown limit + the time to clean up resources must add up to less than terminationGracePeriodSeconds. If the longest request may take 30 seconds, widen the grace period accordingly rather than letting requests be killed midway.
Common pitfalls
- Calling
os.Getenveverywhere. Configuration is scattered, cannot be validated in one go, and you cannot tell which variables the service depends on. Read it into a struct inmainand inject it. - Silently using an empty string when a value is missing. If a required setting is not provided, startup should fail, rather than you finding out on the first request.
- Secrets appearing in logs or error messages. Printing the whole config struct with
%+vis the most common way they leak; mask them with a dedicated type. - Treating
ListenAndServereturning as shutdown being complete. It returns immediately afterShutdownis called; only onceShutdownreturns can you be sure all requests have finished. - Calling
Shutdownwithcontext.Background()and no limit. If some long-lived connection never ends, the program waits until it is SIGKILLed, and none of the final clean-up happens. - Cleaning up with a cancelled ctx. After the signal arrives the ctx is already cancelled; pass it to the database or to
Shutdownand the clean-up fails immediately. Clean-up needs a new context with a timeout. - Shutting down in the wrong order. Close the database or messaging connections first and then wait for requests to finish, and in-flight requests fail at the last moment. The order should be: stop taking new work → finish the work in hand → release shared resources.
- A liveness probe that checks external dependencies. When the database briefly drops, every Pod's liveness fails together and they are all restarted together, which makes the problem bigger. The state of external dependencies belongs in readiness, or only in metrics.
- Ignoring long-lived connections. WebSockets, Server-Sent Events and streaming gRPC do not end automatically because of
Shutdown; you needRegisterOnShutdownor your own signal to tell them to wrap up.
Key takeaways
- Configuration is "values that change between deployment environments" and, following the 12-factor principles, lives in the environment; the usual order of precedence is flags > environment variables > config file > defaults.
- At startup, read configuration into a single struct, validate every field in one go, exit with a non-zero code on any error, and then inject it downwards.
- Secrets stay out of version control, out of images and out of logs; handle them with masking types, file mounts and least privilege.
signal.NotifyContextturns SIGTERM/SIGINT into context cancellation, so every component shares the same shutdown notice.- Shutdown order: fail readiness first → wait for traffic to move away →
http.Server.Shutdownand the consumer stops fetching → finish in-flight work → close shared resources such as the database and tracing. - Manage component lifecycles together with errgroup (or an equivalent WaitGroup plus cancellation), so that a failure in any component triggers an overall shutdown.
- The total time of all shutdown steps must be less than
terminationGracePeriodSeconds; liveness keeps succeeding during shutdown, while readiness reports 503.
Next
The service can now read its configuration, shut down gracefully and has /healthz and /readyz ready, but it is still just a program running on a local machine. “Deploying One Go Service with Docker and Kubernetes” packages the order service into a small, secure image, and then uses Kubernetes probes and grace periods to actually connect this article's shutdown process to the platform.