This resume is also available in English.

6. Deploying the system1 / 5 · All columns

Columns › 6. Deploying the system › Part 21

Deploying a Whole Microservice System with Docker Compose

用 Docker Compose 部署整套微服務

Put the api-gateway, order, payment and inventory services together with PostgreSQL, NATS and Redis into one compose.yaml, covering network isolation, healthcheck-driven startup order, migrations as one-off jobs, secrets and config, and where Compose stops being enough.

  • Docker
  • Docker Compose
  • Microservices
  • Go
In this article
  1. Step 1: Sketch the whole system and the directory layout
  2. Step 2: One Dockerfile shared by four services
  3. Step 3: Write the infrastructure: database, message broker, cache
  4. Step 4: Add the four services and use networks to decide who can see whom
  5. Step 5: Configuration, secrets and env files
  6. Step 6: Startup order and database migrations
  7. Step 7: Day-to-day operations: reading logs, updating a single service
  8. Step 8: Use profiles to switch developer and observability tools on and off
  9. Step 9: How far Compose can go, and when to switch
  10. Common pitfalls
  11. Key takeaways
  12. Next

In the previous article, “Performance Tuning in Go: pprof and Benchmarks”, we found the hot spots in a single service, measured them and made them faster. This article pulls the view back out to the whole system and gets all four services running together.

In “Deploying One Go Service with Docker and Kubernetes” we packaged one Go service into a container image: a multi-stage build, a distroless base, one Deployment plus one Service. But once the order system has been split up, we actually have four services (api-gateway, order, payment, inventory), plus a database, a message queue (NATS acts as the message broker here) and a cache. Each one can be started on its own with docker run, but getting them to run together, start in the right order, find each other, and expose only the entry point to the outside world needs a file that describes "the whole system".

The problems this article solves are concrete:

  • Describe the whole order system in one compose.yaml, so a single command brings it all up on a laptop or a single host.
  • How services find each other by name, and which ports should be exposed and which should not.
  • Services should not race ahead and start before the database is ready or the migrations have finished.
  • Passwords are not written into files; configuration and secrets are managed separately.
  • When only one service changes, how to rebuild and restart just that one.
  • Observability tools (Prometheus, Grafana, Jaeger) and developer tools start only when needed.
  • Finally, an honest look at how far Compose can go in production, and what should be handed over to Kubernetes in the next article.

This article uses the Compose Specification: the file is named compose.yaml, the top of the file does not need, and should no longer have, a version: line, and the command is Docker's built-in docker compose (with a space, not the old docker-compose).

Step 1: Sketch the whole system and the directory layout

Before writing any YAML, list exactly what is being deployed. Continuing the order system from the previous articles, these are the components needed at runtime:

ComponentRoleExposed to
api-gatewayThe single entry point: authentication, rate limiting, routing to backendsExternal (port 8080 on the host)
orderCreates orders, coordinates the Saga, writes the OutboxInternal only
paymentCharges and refunds; needs to call the external payment providerInternal only, but needs outbound access
inventoryReserves and releases stockInternal only
postgresOne separate database per service (in the same instance)Internal only
natsMessage broker (with JetStream enabled for persistence)Internal only
redisCache, rate-limit counters, idempotency keysInternal only

The most important column in this table is the last one. Getting microservices running locally is easy; the hard part is "opening only the doors that should be open". In Compose this is expressed through networks and port publishing, which we will build step by step below.

Next, decide where the files go. Below is a single-repository (monorepo) layout: the four services share one Go module, each has its own main under cmd/, deployment files all live in deploy/, and the Compose ones are collected in deploy/compose/:

shop/                               # module path example.com/shop
├── go.mod
├── cmd/
│   ├── api-gateway/main.go
│   ├── order/main.go
│   ├── payment/main.go
│   └── inventory/main.go
├── internal/                       # domain code (order/, payment/, inventory/…)
├── api/proto/                      # gRPC .proto files
└── deploy/
    ├── docker/
    │   └── Dockerfile              # shared by all four services; the SERVICE arg picks the program under cmd/
    └── compose/
        ├── compose.yaml            # description of the whole system (the star of this article)
        ├── .env                    # for Compose's own variable substitution: TAG, GATEWAY_PORT
        ├── env/                    # non-secret configuration passed into containers
        │   ├── common.env
        │   ├── api-gateway.env
        │   ├── order.env
        │   ├── payment.env
        │   └── inventory.env
        ├── secrets/                # secret files, kept out of version control (.gitignore)
        │   ├── postgres_password.txt
        │   ├── order_db_password.txt
        │   ├── payment_db_password.txt
        │   └── inventory_db_password.txt
        ├── postgres/initdb/10-create-databases.sh
        └── prometheus/prometheus.yml

This layout makes three deliberate choices. First, relative paths in compose.yaml (./env, ./secrets) are all relative to the directory containing compose.yaml, while the build context uses ../.. to point back to the repository root, because a Go build needs the whole module. Second, .env and env/*.env are two different things: the former is used by Compose while parsing compose.yaml to substitute variables such as ${TAG}, while the latter is sent into the container as environment variables. Keeping them in different places avoids the common confusion of "I changed .env but the variable isn't in the container". Third, secrets/ is its own directory and is added to .gitignore as a whole, so "what must never go into version control" is obvious at a glance.

Tip: If your four services live in four separate repositories, the same layout still works; build.context simply points at each one's path (for example ../../../order), or you drop build altogether and use the images CI has pushed to the container registry.

Step 2: One Dockerfile shared by four services

“Deploying One Go Service with Docker and Kubernetes” has already broken down multi-stage builds and distroless in detail, so here is only a trimmed-down version. The key point is using a build arg to decide which service to compile, so the four services don't each have to maintain an almost identical Dockerfile:

# syntax=docker/dockerfile:1
FROM golang:1.27 AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
ARG SERVICE
RUN CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/server ./cmd/${SERVICE}

FROM gcr.io/distroless/static-debian12:nonroot
COPY --from=build /out/server /app/server
USER nonroot:nonroot
ENTRYPOINT ["/app/server"]
CMD ["serve"]

Note two things. First, every service's binary is called /app/server inside the container, so the healthchecks and migration commands later can all take the same shape. Second, a distroless image has no shell, no curl and no wget, so a Compose healthcheck cannot rely on curl localhost/healthz. The fix is to give the program itself an extra subcommand:

package main

import (
	"context"
	"errors"
	"fmt"
	"log"
	"net/http"
	"os"
	"strings"
	"time"
)

// envOrFile first reads the file NAME_FILE points to (Compose secrets, a mounted Kubernetes Secret),
// and only falls back to NAME itself. The same program then runs locally and in the cluster unchanged.
func envOrFile(name string) (string, error) {
	if path := os.Getenv(name + "_FILE"); path != "" {
		b, err := os.ReadFile(path)
		if err != nil {
			return "", fmt.Errorf("read %s: %w", path, err)
		}
		return strings.TrimSpace(string(b)), nil
	}
	if v := os.Getenv(name); v != "" {
		return v, nil
	}
	return "", fmt.Errorf("%s or %s_FILE is required", name, name)
}

// healthcheck is used by the Compose healthcheck: distroless has no curl, so we make the request ourselves.
func healthcheck(addr string) int {
	ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
	defer cancel()
	req, err := http.NewRequestWithContext(ctx, http.MethodGet, "http://"+addr+"/readyz", nil)
	if err != nil {
		return 1
	}
	resp, err := http.DefaultClient.Do(req)
	if err != nil {
		fmt.Fprintln(os.Stderr, "healthcheck:", err)
		return 1
	}
	defer resp.Body.Close()
	if resp.StatusCode != http.StatusOK {
		return 1
	}
	return 0
}

func serve(addr string) error {
	if _, err := envOrFile("DB_PASSWORD"); err != nil {
		return err
	}
	mux := http.NewServeMux()
	mux.HandleFunc("GET /livez", func(w http.ResponseWriter, _ *http.Request) { w.WriteHeader(http.StatusOK) })
	mux.HandleFunc("GET /readyz", func(w http.ResponseWriter, _ *http.Request) {
		// A real service checks here that the database connection pool and message broker connection are ready.
		w.WriteHeader(http.StatusOK)
	})
	srv := &http.Server{Addr: addr, Handler: mux, ReadHeaderTimeout: 5 * time.Second}
	if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
		return err
	}
	return nil
}

func main() {
	cmd := "serve"
	if len(os.Args) > 1 {
		cmd = os.Args[1]
	}
	switch cmd {
	case "healthcheck":
		os.Exit(healthcheck("127.0.0.1:8080"))
	case "serve":
		if err := serve(":8080"); err != nil {
			log.Fatal(err)
		}
	default:
		fmt.Fprintln(os.Stderr, "unknown command:", cmd)
		os.Exit(2)
	}
}

This program only demonstrates the structure; the real serve handles signals and shutdown as in “Configuration and Graceful Shutdown in Go”, and the migrate subcommand calls the migration tool used in “database/sql, Transactions and Connection Pools in Go”. The key piece is envOrFile: it lets the program support both "pass the password directly" and "pass a file path", a convention used by the Compose secrets below and by the Kubernetes Secret in the next article.

Tip: The healthcheck subcommand hits /readyz, not /livez. In Compose, "healthy" decides whether others may start depending on you, which is semantically closer to Kubernetes readiness than to liveness.

Step 3: Write the infrastructure: database, message broker, cache

Start with the components that "depend on nobody"; they are the skeleton of compose.yaml. Pull shared settings out into extension fields right from the start (top-level keys beginning with x-, which Compose ignores and which exist precisely to hold YAML anchors):

name: shop

x-logging: &default-logging
  driver: json-file
  options:
    max-size: '10m'
    max-file: '3'

services:
  postgres:
    image: postgres:17-alpine
    restart: unless-stopped
    logging: *default-logging
    networks: [backend]
    environment:
      POSTGRES_USER: postgres
      POSTGRES_PASSWORD_FILE: /run/secrets/postgres_password
    secrets:
      - postgres_password
      - order_db_password
      - payment_db_password
      - inventory_db_password
    volumes:
      - pgdata:/var/lib/postgresql/data
      - ./postgres/initdb:/docker-entrypoint-initdb.d:ro
    healthcheck:
      test: ['CMD-SHELL', 'pg_isready -U postgres -d postgres']
      interval: 10s
      timeout: 3s
      retries: 5
      start_period: 20s
      start_interval: 1s

  nats:
    image: nats:2.10-alpine
    restart: unless-stopped
    logging: *default-logging
    networks: [backend]
    command: ['-js', '-sd', '/data', '-m', '8222']
    volumes:
      - natsdata:/data
    healthcheck:
      test: ['CMD', 'wget', '-q', '--spider', 'http://127.0.0.1:8222/healthz']
      interval: 10s
      timeout: 3s
      retries: 5
      start_period: 10s
      start_interval: 1s

  redis:
    image: redis:7-alpine
    restart: unless-stopped
    logging: *default-logging
    networks: [backend]
    command: ['redis-server', '--save', '', '--appendonly', 'no']
    healthcheck:
      test: ['CMD', 'redis-cli', 'ping']
      interval: 10s
      timeout: 3s
      retries: 5

A few of the decisions, one by one:

One database per service, but one shared PostgreSQL instance. “When to Split: Service Boundaries and Decomposition Strategies” stressed that "each service owns its data": order may not read inventory's tables directly. There are three ways to apply this principle at deployment time:

ApproachIsolationCostSuits
One PostgreSQL per serviceHighest: versions, resources and failures all separateHighestProduction, very different loads
Same instance, separate databasesMedium: accounts and data isolated, resources sharedLowDevelopment, testing, small single-host deployments
Same database, separate schemasLow: relies on permissions to block cross-schema accessLowestWhen resources are extremely tight

In a Compose setting, the second is a good balance: each service has its own account and database with no permission to touch anyone else's data, yet only one PostgreSQL container runs. The key is that each service uses its own account rather than everyone using the postgres superuser; otherwise "each service owns its data" is only a verbal agreement.

Creating the accounts and databases is left to the official image's initialisation mechanism: scripts in /docker-entrypoint-initdb.d/ run once, when the data directory is first created. Because the passwords live in secrets, a shell script reads them:

#!/bin/sh
# deploy/compose/postgres/initdb/10-create-databases.sh
# Runs only when the pgdata volume is first initialised.
set -eu

for svc in order payment inventory; do
  pw="$(cat "/run/secrets/${svc}_db_password")"
  psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname postgres <<SQL
CREATE ROLE ${svc} LOGIN PASSWORD '${pw}';
CREATE DATABASE ${svc} OWNER ${svc};
REVOKE ALL ON DATABASE ${svc} FROM PUBLIC;
SQL
done

Healthchecks are the foundation of everything that follows. pg_isready confirms PostgreSQL is accepting connections; NATS's monitoring port 8222 serves /healthz, and the alpine image has busybox's wget available; Redis uses redis-cli ping. start_period is a start-up grace period during which failures don't count towards the retries; start_interval makes checks more frequent during the grace period, so a service is marked healthy sooner once it is ready (this needs a fairly recent Docker Engine and Compose; older versions ignore it or report an error, in which case just remove it).

Redis is used only as a cache and counter here, so persistence is turned off and no volume is mounted; losing the data when the container is recreated is expected. If you store idempotency keys in Redis (“Idempotency and Distributed Locks”), assess the impact of losing them, and if necessary enable AOF and mount a volume.

Always set up log rotation. Docker's default json-file driver never deletes old logs, and one noisy service can quietly fill the host's disk. The x-logging anchor lets every service apply the same limits in one line.

Tip: The image versions (postgres:17-alpine, nats:2.10-alpine and so on) are only illustrative in this article. In real use, pin them to versions you have verified, ideally pinning the digest too, so that a re-pull one day doesn't quietly upgrade you.

Step 4: Add the four services and use networks to decide who can see whom

With the infrastructure ready, add the four Go services. They share a lot of settings (restart policy, logging, healthcheck, shared env file), which also go into an anchor:

x-go-service: &go-service
  restart: unless-stopped
  logging: *default-logging
  networks: [backend]
  healthcheck:
    test: ['CMD', '/app/server', 'healthcheck']
    interval: 10s
    timeout: 3s
    retries: 3
    start_period: 15s
    start_interval: 1s

services:
  order:
    <<: *go-service
    build: &order-build
      context: ../..
      dockerfile: deploy/docker/Dockerfile
      args:
        SERVICE: order
    image: shop/order:${TAG:-dev}
    env_file:
      - ./env/common.env
      - ./env/order.env
    environment:
      DB_HOST: postgres
      DB_NAME: order
      DB_USER: order
      DB_PASSWORD_FILE: /run/secrets/order_db_password
      PAYMENT_ADDR: payment:9090
      INVENTORY_ADDR: inventory:9090
      NATS_URL: nats://nats:4222
      REDIS_ADDR: redis:6379
    secrets: [order_db_password]
    depends_on:
      order-migrate:
        condition: service_completed_successfully
      nats:
        condition: service_healthy
      redis:
        condition: service_healthy

  payment:
    <<: *go-service
    build: &payment-build
      context: ../..
      dockerfile: deploy/docker/Dockerfile
      args:
        SERVICE: payment
    image: shop/payment:${TAG:-dev}
    networks: [backend, egress] # needs to reach the external payment provider
    env_file:
      - ./env/common.env
      - ./env/payment.env
    environment:
      DB_HOST: postgres
      DB_NAME: payment
      DB_USER: payment
      DB_PASSWORD_FILE: /run/secrets/payment_db_password
      NATS_URL: nats://nats:4222
    secrets: [payment_db_password]
    depends_on:
      payment-migrate:
        condition: service_completed_successfully
      nats:
        condition: service_healthy

  # inventory has the same shape as payment, just without the egress network; omitted.

  api-gateway:
    <<: *go-service
    build:
      context: ../..
      dockerfile: deploy/docker/Dockerfile
      args:
        SERVICE: api-gateway
    image: shop/api-gateway:${TAG:-dev}
    networks: [edge, backend]
    ports:
      - '${GATEWAY_PORT:-8080}:8080'
    env_file:
      - ./env/common.env
      - ./env/api-gateway.env
    environment:
      ORDER_ADDR: order:9090
      PAYMENT_ADDR: payment:9090
      INVENTORY_ADDR: inventory:9090
      REDIS_ADDR: redis:6379
    depends_on:
      order:
        condition: service_healthy
      payment:
        condition: service_healthy
      inventory:
        condition: service_healthy
      redis:
        condition: service_healthy

networks:
  edge: {}
  backend:
    internal: true
  egress: {}

This step has three core ideas.

The service name is the host name. Compose creates networks for the project, and Docker's built-in DNS server lets containers on the same network resolve each other by service name: order connects to postgres:5432, api-gateway connects to order:9090, with no need to know any IP. This is the simplest form of the "platform-provided service discovery" mentioned in “Service Discovery and Load Balancing for Go Microservices”. If you later run three copies of inventory with docker compose up --scale inventory=3, a DNS query for inventory returns three addresses; a gRPC client using dns:///inventory:9090 with the round_robin policy spreads requests across the three containers. Be aware that an HTTP client reusing long-lived connections may stay stuck to one container, the same situation discussed in that article.

Only api-gateway publishes a port. ports binds a container port to the host, and only api-gateway has that line. order, payment and inventory listen on 9090 and 8080 inside the container network, but cannot be reached from the host or from outside. This is not just a security benefit; it also guarantees that all traffic passes through api-gateway's authentication and rate limiting, so nobody can bypass the entry point and hit a backend directly.

Use networks to carve out trust zones. Three networks are defined here:

  • edge: api-gateway (and later the developer tools) lives here and faces the outside.
  • backend: every service and all infrastructure are here. It is set to internal: true, which means the network has no outbound route, so containers cannot reach the internet.
  • egress: joined only by payment, so it can call the external payment provider's API.

internal: true is a very useful but often overlooked setting: even if some dependency has been compromised with malicious code, inside the backend network it has no way to send data out. The price is that when a service really does need outbound access (payments, sending email, third-party APIs) it must explicitly join another network, and that is exactly what we want: outbound access becomes a visible decision that needs review.

External clients Host port 8080 (the only published port) edge network api-gateway backend (internal: true, no outbound route) order inventory payment Data stores: a database per service, plus a shared message broker and cache postgres nats redis egress network (joined only by payment) External payment API

Figure: The outside world can only reach api-gateway on port 8080; the backend services and data stores sit on the backend network, which has no outbound route, and only payment additionally joins the egress network to reach the external payment provider.

depends_on is deliberately not written to follow the call graph here. At runtime order calls payment and inventory, but its depends_on only lists the database migration, NATS and Redis: the hard dependencies without which it "cannot start at all". Call dependencies between services should be handled at runtime by the mechanisms in “Timeouts, Retries and Circuit Breakers”: if the other side is temporarily absent, retry or return an error, rather than forcing the whole system to start in a particular order. If you copy the full call graph into depends_on, restarting any one service can drag a chain of others to a halt, and it is easy to create circular dependencies. Making api-gateway depend on the health of the three backends is a convenience: in a development environment, by the time api-gateway can accept traffic, the backends are guaranteed to be ready.

Step 5: Configuration, secrets and env files

Configuration comes in layers, each with a different purpose:

LocationRead byContainsIn version control
.envCompose, while parsing the YAMLSubstitution variables such as TAG, GATEWAY_PORTYes
env/*.env (env_file)The program in the containerNon-secret settings such as log level, timeouts, feature flagsYes
environment:The program in the containerTopology-related addresses (order:9090)Yes
secrets/ (secrets)The program in the container (reads a file)Database passwords, API keysNo

The example env files are short, because they hold only values that "may differ per environment but are not secret":

# deploy/compose/env/common.env — shared by all four services
LOG_LEVEL=info
LOG_FORMAT=json
OTEL_EXPORTER_OTLP_ENDPOINT=http://jaeger:4317
SHUTDOWN_TIMEOUT=20s

# deploy/compose/env/order.env — used only by order
ORDER_SAGA_TIMEOUT=30s
OUTBOX_POLL_INTERVAL=500ms

When env_file lists several files, later ones override earlier ones, and environment: in turn overrides env_file. Make use of that order: shared defaults go in common.env, service-specific values go in the service's own file, and addresses tied to the Compose topology go directly in environment:, because they follow the service names in compose.yaml.

Secrets are declared in the top-level secrets: and then listed in each service that needs them:

secrets:
  postgres_password:
    file: ./secrets/postgres_password.txt
  order_db_password:
    file: ./secrets/order_db_password.txt
  payment_db_password:
    file: ./secrets/payment_db_password.txt
  inventory_db_password:
    file: ./secrets/inventory_db_password.txt

Each secret a service lists appears as a file at /run/secrets/<name> in the container. The program reads it through DB_PASSWORD_FILE, that is, the envOrFile from Step 2. Each service can only get its own password: the order container does not have payment's database password, so even if order is compromised, it doesn't leak anyone else's credentials along the way.

Why not just put passwords in environment variables? Because environment variables leak easily: docker inspect shows them, debug output on a crash may print them, and child processes inherit them. File-based secrets at least move them out of those channels. To be honest, though: in single-host Compose, a file: secret is essentially a host file mounted read-only into the container, with no encryption. What it protects against is "being scattered where it shouldn't appear", not "staying safe if the host is compromised". Tighten the permissions on the host's secrets/ directory (for example, readable only by the deployment account).

Tip: docker compose config prints the final configuration after variable substitution, anchor merging and stacking of multiple files. When you wonder "why isn't the value in the container what I expected?", look at that first.

Step 6: Startup order and database migrations

With healthchecks in place, the startup order can be written very precisely. The goal is:

  1. PostgreSQL, NATS and Redis start first, and we wait until they are healthy.
  2. Each service's database migration runs as a one-off job and exits when done.
  3. Only after the migration completes successfully does the corresponding service start, and we wait for it to become healthy.
  4. Only when all three backends are healthy does api-gateway start and begin accepting traffic.

A migration is a "one-off service" that uses the same image with a different command:

services:
  order-migrate:
    build: *order-build
    image: shop/order:${TAG:-dev}
    command: ['migrate', 'up']
    restart: 'no'
    logging: *default-logging
    networks: [backend]
    environment:
      DB_HOST: postgres
      DB_NAME: order
      DB_USER: order
      DB_PASSWORD_FILE: /run/secrets/order_db_password
    secrets: [order_db_password]
    depends_on:
      postgres:
        condition: service_healthy
        restart: true

A few details are worth explaining:

  • build: *order-build reuses order's build settings, so both produce the same image tag. That way docker compose up won't try to pull a shop/order:dev that doesn't exist yet from a remote registry.
  • command: ["migrate", "up"] is appended to the Dockerfile's ENTRYPOINT ["/app/server"], so what actually runs is /app/server migrate up.
  • restart: "no" makes it stop when finished instead of being restarted over and over.
  • order's depends_on uses condition: service_completed_successfully: the migration's exit code must be 0 before order starts. If the migration fails, order does not start and docker compose up reports an error, rather than letting the service run against the old table structure.
  • Where postgres is a dependency, restart: true is added: if postgres is later restarted by hand, Compose also restarts the services that depend on it, avoiding leftover broken connections.

Taking migrations out of the service's startup path, rather than running them automatically at the top of main, has several benefits: multiple replicas don't race to change the schema; when a migration fails, the error is in one place; and in production the same command can be run manually. The Kubernetes Job in the next article is exactly the same idea.

1 Infrastructure 2 Migrations (one-off) 3 Services 4 Entry point postgres nats redis order-migrate payment-migrate inventory-migrate order payment inventory api-gateway service_healthy completed_successfully service_healthy Accepts traffic If any step fails, none of the later containers start

Figure: Migrations run only after the infrastructure is healthy, services start only after their migrations complete successfully, and api-gateway faces the outside only after all three backends are healthy.

All commands in this article are run from the deploy/compose/ directory; when running from the repository root, add -f deploy/compose/compose.yaml. When actually starting up, use --wait so the command doesn't return until every service is healthy (or every one-off job has completed successfully), which is ideal in front of CI integration tests:

# Build and start the whole system; return only once everything is healthy
docker compose up -d --build --wait

# Show each container's state and health
docker compose ps

# Just re-run order's migration by hand
docker compose run --rm order-migrate

Tip: depends_on only governs "the moment of startup". If PostgreSQL restarts after the system is running, the services still need to be able to reconnect. Startup order is a tool for a smooth development experience; it cannot replace the resilience of the program itself.

Step 7: Day-to-day operations: reading logs, updating a single service

Once the system is running, the most common day-to-day tasks are reading logs and changing one particular service.

Logs: each service writes structured JSON to standard output as in “Observability in Go: Logs, Metrics and Tracing”. Compose collects the output of all containers and prefixes each line with the service name:

# Follow the latest logs from order and payment
docker compose logs -f --tail=100 order payment

# Only the last 10 minutes, filtering errors with jq
docker compose logs --since 10m --no-log-prefix order | jq -c 'select(.level == "ERROR")'

Updating just one service: after changing order's code, there is no need to restart the whole system:

# Rebuild order's image and recreate only the order container (leaving its dependencies alone)
docker compose up -d --build --no-deps order

# When the schema changes, run the migration first, then swap the service
docker compose build order
docker compose run --rm order-migrate
docker compose up -d --no-deps order

--no-deps matters. Without it, Compose checks and starts all of order's dependencies; it usually won't recreate containers that are already running, but adding the flag makes the intent explicit: touch only this one. Also be aware that Compose replaces a container by stopping the old one first, then starting the new one, leaving a short gap in between; api-gateway's calls to order fail during that window and have to be ridden out with retries. This is one of the biggest differences between Compose and a Kubernetes rolling update.

During development you can go further with develop.watch, letting Compose watch the source code and rebuild automatically on changes:

services:
  order:
    develop:
      watch:
        - action: rebuild
          path: ./internal/order
        - action: rebuild
          path: ./cmd/order

Then run docker compose watch to start watching. Go is a compiled language, so rebuild is the most direct action; sync, which copies files straight into the container, is more commonly used with interpreted languages.

A few commands to be careful with:

CommandEffectWatch out
docker compose stopStops containers, keeping containers and dataSafe
docker compose downRemoves containers and networks, keeps named volumesSafe, the data is still there
docker compose down -vAlso removes named volumesDatabase data is lost
docker compose restartRestarts containers, without re-reading compose.yamlUse up -d after config changes
docker compose exec order …Runs a command inside a running containerdistroless has no shell

Step 8: Use profiles to switch developer and observability tools on and off

Some containers aren't needed every time: a database admin UI, the NATS command-line tools, Prometheus, Grafana, Jaeger. Put them in the same compose.yaml but add profiles, and the default docker compose up won't start them:

services:
  adminer:
    image: adminer:5
    profiles: [dev]
    networks: [edge, backend]
    ports:
      - '127.0.0.1:8081:8080'

  nats-box:
    image: natsio/nats-box:latest
    profiles: [dev]
    networks: [backend]
    environment:
      NATS_URL: nats://nats:4222

  prometheus:
    image: prom/prometheus:v3.5.0
    profiles: [observability]
    networks: [edge, backend]
    ports:
      - '127.0.0.1:9090:9090'
    volumes:
      - ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
      - promdata:/prometheus

  grafana:
    image: grafana/grafana:12.1.0
    profiles: [observability]
    networks: [edge, backend]
    ports:
      - '127.0.0.1:3000:3000'
    volumes:
      - grafanadata:/var/lib/grafana

  jaeger:
    image: jaegertracing/jaeger:2.9.0
    profiles: [observability]
    networks: [edge, backend]
    ports:
      - '127.0.0.1:16686:16686'

volumes:
  pgdata: {}
  natsdata: {}
  promdata: {}
  grafanadata: {}

The developer tools' ports are all bound to 127.0.0.1, so only a browser on the local machine can reach them; even if the machine has a public IP, nothing is exposed by accident. This is consistent with the "only api-gateway faces the outside" principle: api-gateway is the entry point for users, and the other ports are just windows for developers.

Prometheus's scrape configuration also uses service names directly:

# deploy/compose/prometheus/prometheus.yml
global:
  scrape_interval: 15s
scrape_configs:
  - job_name: shop
    static_configs:
      - targets:
          - api-gateway:8080
          - order:8080
          - payment:8080
          - inventory:8080

There are two ways to enable them:

# On the command line
docker compose --profile dev --profile observability up -d

# Or with an environment variable (it can also go in .env)
COMPOSE_PROFILES=observability docker compose up -d

# Use run for one-off tools; they don't need to stay running
docker compose --profile dev run --rm nats-box nats stream ls
Default (no profile) api-gateway order · payment · inventory Three migration jobs postgres · nats · redis Always started profile: dev adminer nats-box Ports on 127.0.0.1 only profile: observability prometheus grafana jaeger Ports on 127.0.0.1 only docker compose up --profile dev --profile observability

Figure: Services without a profile always start; the dev and observability groups join only when specified.

There is one rule to follow here: services that start by default must not depend on services that have a profile. If order had depends_on: jaeger and jaeger only starts under observability, a normal up makes Compose report an error. Exporting trace data should be designed so that "it's fine if the collector isn't there": when the OpenTelemetry exporter can't connect, it drops the data and logs an error, without stopping the service.

Step 9: How far Compose can go, and when to switch

Putting the previous steps together, one compose.yaml describes the complete order system: four services, three kinds of infrastructure, migrations, network isolation, secrets and optional observability tools. It is very useful in several scenarios:

  • Local development: a new developer clones the repo, prepares secrets/, and has a complete environment with one command.
  • CI integration tests: run end-to-end tests after docker compose up -d --wait, then clean up with docker compose down -v, so every run gets a clean environment.
  • Demo and preview environments: bring up one stack per feature branch, separated by project name (-p).
  • Small single-host production deployments: systems with modest traffic, where brief downtime during updates is acceptable and one host is enough.

But as the system heads into production, the limits of Compose become very clear:

RequirementComposeKubernetes equivalent
Multiple hostsEssentially a single-host toolCluster scheduling
Host failurerestart can save a container, not a hostPods are rescheduled onto other nodes
Zero-downtime updatesStop then start, with a gapRolling updates, gated by readiness
Scaling with load--scale is manual, and services that bind a host port can't run more copiesHPA adjusts the replica count automatically
Secret managementFiles on the host, unencryptedSecrets plus external key management
Service-to-service access controlCoarse network-level separationNetworkPolicy, down to each service
RollbackRemember the previous tag yourself and up againkubectl rollout undo

This doesn't mean Compose can't go to production; it means you must know clearly which limitations you are accepting: a single point of failure, brief interruptions during updates, manual scaling. If all of those are acceptable, one host plus one compose.yaml is a very simple, easy-to-understand architecture; if not, that is what the next article, “Deploying a Microservice System on Kubernetes”, deals with. The good news is that the habits built in this article (one image with several subcommands, healthchecks, migrations as one-off jobs, the _FILE secret convention, finding each other by service name) all carry straight over to Kubernetes.

Common pitfalls

  • Writing depends_on in the short syntax. depends_on: [postgres] only guarantees the container has started, not that PostgreSQL can accept connections. Always use the long syntax with condition: service_healthy, and the dependency must have a healthcheck, otherwise the condition is never met.
  • Writing a curl healthcheck for a distroless image. The image has no curl, so the check always fails and the service is forever "unhealthy". Use the program's built-in healthcheck subcommand instead.
  • Confusing .env with env_file. .env is only for Compose's ${VAR} substitution and does not automatically reach the container; to give a container a variable, it must go through env_file or environment.
  • Assuming a changed initdb script will run again. /docker-entrypoint-initdb.d/ runs only when the data directory is empty. If the volume already exists, handle it with a migration, or in development delete the volume and start over.
  • Hard-coding passwords in compose.yaml. That file goes into version control, and the password goes with it. Use secrets with files excluded by .gitignore.
  • Publishing ports everywhere. Adding ports: ["5432:5432"] to the database for easier debugging and forgetting to remove it binds the database to every interface on the host. If you really need to connect, bind to 127.0.0.1 and put it in the dev profile.
  • Assuming docker compose restart applies new configuration. It only restarts existing containers; after changing compose.yaml or an env file, run docker compose up -d so Compose compares the differences and recreates.
  • Running down -v by accident. This deletes named volumes, and the database data disappears with them. Production data needs backups, not just a volume.
  • Letting Docker logs grow without limit. json-file logs without max-size keep growing until they fill the disk.
  • Putting the whole call graph into depends_on. This causes slow startup, cascading restarts and even circular dependencies. List only hard dependencies, and rely on retries and circuit breakers for the rest.

Key takeaways

  • One compose.yaml (without version:) describes the whole system: four services, PostgreSQL, NATS, Redis, migrations and optional tools.
  • One Dockerfile with a SERVICE build arg is shared by the four services; the binary provides serve, healthcheck and migrate subcommands.
  • Services connect to each other by service name; only api-gateway publishes a port, the backend network is set to internal: true, and services that need outbound access explicitly join an egress network.
  • Each service owns its own database and account; development and single-host environments can share one PostgreSQL instance.
  • Configuration is layered: .env for Compose, env_file for containers, environment for topology addresses, secrets for secrets, read through the _FILE convention.
  • Startup order relies on the long depends_on syntax: infrastructure service_healthy → migrations service_completed_successfully → services service_healthy → api-gateway.
  • depends_on lists only hard dependencies; temporary failures in calls between services are handled by retries, timeouts and circuit breakers.
  • Update a single service with docker compose up -d --build --no-deps <service>; observe and debug with logs -f, ps and config.
  • Developer and observability tools go into profiles, with their ports bound to 127.0.0.1.
  • Compose suits local development, CI, previews and small single-host deployments; when you need multiple hosts, zero-downtime updates, autoscaling and fine-grained access control, hand over to Kubernetes.

Next

Compose makes the whole system run smoothly on one host, but it cannot give you multiple hosts, zero-downtime updates, autoscaling or fine-grained access control between services. The next article, “Deploying a Microservice System on Kubernetes”, moves the same services into a cluster and fills those gaps with Deployments, HPA, NetworkPolicy and Kustomize.