Columns › 6. Deploying the system › Part 21
Deploying a Whole Microservice System with Docker Compose
用 Docker Compose 部署整套微服務
Put the api-gateway, order, payment and inventory services together with PostgreSQL, NATS and Redis into one compose.yaml, covering network isolation, healthcheck-driven startup order, migrations as one-off jobs, secrets and config, and where Compose stops being enough.
In this article
- Step 1: Sketch the whole system and the directory layout
- Step 2: One Dockerfile shared by four services
- Step 3: Write the infrastructure: database, message broker, cache
- Step 4: Add the four services and use networks to decide who can see whom
- Step 5: Configuration, secrets and env files
- Step 6: Startup order and database migrations
- Step 7: Day-to-day operations: reading logs, updating a single service
- Step 8: Use profiles to switch developer and observability tools on and off
- Step 9: How far Compose can go, and when to switch
- Common pitfalls
- Key takeaways
- Next
In the previous article, “Performance Tuning in Go: pprof and Benchmarks”, we found the hot spots in a single service, measured them and made them faster. This article pulls the view back out to the whole system and gets all four services running together.
In “Deploying One Go Service with Docker and Kubernetes” we packaged one Go service into a container image: a multi-stage build, a distroless base, one Deployment plus one Service. But once the order system has been split up, we actually have four services (api-gateway, order, payment, inventory), plus a database, a message queue (NATS acts as the message broker here) and a cache. Each one can be started on its own with docker run, but getting them to run together, start in the right order, find each other, and expose only the entry point to the outside world needs a file that describes "the whole system".
The problems this article solves are concrete:
- Describe the whole order system in one
compose.yaml, so a single command brings it all up on a laptop or a single host. - How services find each other by name, and which ports should be exposed and which should not.
- Services should not race ahead and start before the database is ready or the migrations have finished.
- Passwords are not written into files; configuration and secrets are managed separately.
- When only one service changes, how to rebuild and restart just that one.
- Observability tools (Prometheus, Grafana, Jaeger) and developer tools start only when needed.
- Finally, an honest look at how far Compose can go in production, and what should be handed over to Kubernetes in the next article.
This article uses the Compose Specification: the file is named compose.yaml, the top of the file does not need, and should no longer have, a version: line, and the command is Docker's built-in docker compose (with a space, not the old docker-compose).
Step 1: Sketch the whole system and the directory layout
Before writing any YAML, list exactly what is being deployed. Continuing the order system from the previous articles, these are the components needed at runtime:
| Component | Role | Exposed to |
|---|---|---|
| api-gateway | The single entry point: authentication, rate limiting, routing to backends | External (port 8080 on the host) |
| order | Creates orders, coordinates the Saga, writes the Outbox | Internal only |
| payment | Charges and refunds; needs to call the external payment provider | Internal only, but needs outbound access |
| inventory | Reserves and releases stock | Internal only |
| postgres | One separate database per service (in the same instance) | Internal only |
| nats | Message broker (with JetStream enabled for persistence) | Internal only |
| redis | Cache, rate-limit counters, idempotency keys | Internal only |
The most important column in this table is the last one. Getting microservices running locally is easy; the hard part is "opening only the doors that should be open". In Compose this is expressed through networks and port publishing, which we will build step by step below.
Next, decide where the files go. Below is a single-repository (monorepo) layout: the four services share one Go module, each has its own main under cmd/, deployment files all live in deploy/, and the Compose ones are collected in deploy/compose/:
shop/ # module path example.com/shop
├── go.mod
├── cmd/
│ ├── api-gateway/main.go
│ ├── order/main.go
│ ├── payment/main.go
│ └── inventory/main.go
├── internal/ # domain code (order/, payment/, inventory/…)
├── api/proto/ # gRPC .proto files
└── deploy/
├── docker/
│ └── Dockerfile # shared by all four services; the SERVICE arg picks the program under cmd/
└── compose/
├── compose.yaml # description of the whole system (the star of this article)
├── .env # for Compose's own variable substitution: TAG, GATEWAY_PORT
├── env/ # non-secret configuration passed into containers
│ ├── common.env
│ ├── api-gateway.env
│ ├── order.env
│ ├── payment.env
│ └── inventory.env
├── secrets/ # secret files, kept out of version control (.gitignore)
│ ├── postgres_password.txt
│ ├── order_db_password.txt
│ ├── payment_db_password.txt
│ └── inventory_db_password.txt
├── postgres/initdb/10-create-databases.sh
└── prometheus/prometheus.ymlThis layout makes three deliberate choices. First, relative paths in compose.yaml (./env, ./secrets) are all relative to the directory containing compose.yaml, while the build context uses ../.. to point back to the repository root, because a Go build needs the whole module. Second, .env and env/*.env are two different things: the former is used by Compose while parsing compose.yaml to substitute variables such as ${TAG}, while the latter is sent into the container as environment variables. Keeping them in different places avoids the common confusion of "I changed .env but the variable isn't in the container". Third, secrets/ is its own directory and is added to .gitignore as a whole, so "what must never go into version control" is obvious at a glance.
Tip: If your four services live in four separate repositories, the same layout still works;
build.contextsimply points at each one's path (for example../../../order), or you dropbuildaltogether and use the images CI has pushed to the container registry.
Step 2: One Dockerfile shared by four services
“Deploying One Go Service with Docker and Kubernetes” has already broken down multi-stage builds and distroless in detail, so here is only a trimmed-down version. The key point is using a build arg to decide which service to compile, so the four services don't each have to maintain an almost identical Dockerfile:
# syntax=docker/dockerfile:1
FROM golang:1.27 AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
ARG SERVICE
RUN CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/server ./cmd/${SERVICE}
FROM gcr.io/distroless/static-debian12:nonroot
COPY --from=build /out/server /app/server
USER nonroot:nonroot
ENTRYPOINT ["/app/server"]
CMD ["serve"]Note two things. First, every service's binary is called /app/server inside the container, so the healthchecks and migration commands later can all take the same shape. Second, a distroless image has no shell, no curl and no wget, so a Compose healthcheck cannot rely on curl localhost/healthz. The fix is to give the program itself an extra subcommand:
package main
import (
"context"
"errors"
"fmt"
"log"
"net/http"
"os"
"strings"
"time"
)
// envOrFile first reads the file NAME_FILE points to (Compose secrets, a mounted Kubernetes Secret),
// and only falls back to NAME itself. The same program then runs locally and in the cluster unchanged.
func envOrFile(name string) (string, error) {
if path := os.Getenv(name + "_FILE"); path != "" {
b, err := os.ReadFile(path)
if err != nil {
return "", fmt.Errorf("read %s: %w", path, err)
}
return strings.TrimSpace(string(b)), nil
}
if v := os.Getenv(name); v != "" {
return v, nil
}
return "", fmt.Errorf("%s or %s_FILE is required", name, name)
}
// healthcheck is used by the Compose healthcheck: distroless has no curl, so we make the request ourselves.
func healthcheck(addr string) int {
ctx, cancel := context.WithTimeout(context.Background(), 2*time.Second)
defer cancel()
req, err := http.NewRequestWithContext(ctx, http.MethodGet, "http://"+addr+"/readyz", nil)
if err != nil {
return 1
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
fmt.Fprintln(os.Stderr, "healthcheck:", err)
return 1
}
defer resp.Body.Close()
if resp.StatusCode != http.StatusOK {
return 1
}
return 0
}
func serve(addr string) error {
if _, err := envOrFile("DB_PASSWORD"); err != nil {
return err
}
mux := http.NewServeMux()
mux.HandleFunc("GET /livez", func(w http.ResponseWriter, _ *http.Request) { w.WriteHeader(http.StatusOK) })
mux.HandleFunc("GET /readyz", func(w http.ResponseWriter, _ *http.Request) {
// A real service checks here that the database connection pool and message broker connection are ready.
w.WriteHeader(http.StatusOK)
})
srv := &http.Server{Addr: addr, Handler: mux, ReadHeaderTimeout: 5 * time.Second}
if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
return err
}
return nil
}
func main() {
cmd := "serve"
if len(os.Args) > 1 {
cmd = os.Args[1]
}
switch cmd {
case "healthcheck":
os.Exit(healthcheck("127.0.0.1:8080"))
case "serve":
if err := serve(":8080"); err != nil {
log.Fatal(err)
}
default:
fmt.Fprintln(os.Stderr, "unknown command:", cmd)
os.Exit(2)
}
}This program only demonstrates the structure; the real serve handles signals and shutdown as in “Configuration and Graceful Shutdown in Go”, and the migrate subcommand calls the migration tool used in “database/sql, Transactions and Connection Pools in Go”. The key piece is envOrFile: it lets the program support both "pass the password directly" and "pass a file path", a convention used by the Compose secrets below and by the Kubernetes Secret in the next article.
Tip: The
healthchecksubcommand hits/readyz, not/livez. In Compose, "healthy" decides whether others may start depending on you, which is semantically closer to Kubernetes readiness than to liveness.
Step 3: Write the infrastructure: database, message broker, cache
Start with the components that "depend on nobody"; they are the skeleton of compose.yaml. Pull shared settings out into extension fields right from the start (top-level keys beginning with x-, which Compose ignores and which exist precisely to hold YAML anchors):
name: shop
x-logging: &default-logging
driver: json-file
options:
max-size: '10m'
max-file: '3'
services:
postgres:
image: postgres:17-alpine
restart: unless-stopped
logging: *default-logging
networks: [backend]
environment:
POSTGRES_USER: postgres
POSTGRES_PASSWORD_FILE: /run/secrets/postgres_password
secrets:
- postgres_password
- order_db_password
- payment_db_password
- inventory_db_password
volumes:
- pgdata:/var/lib/postgresql/data
- ./postgres/initdb:/docker-entrypoint-initdb.d:ro
healthcheck:
test: ['CMD-SHELL', 'pg_isready -U postgres -d postgres']
interval: 10s
timeout: 3s
retries: 5
start_period: 20s
start_interval: 1s
nats:
image: nats:2.10-alpine
restart: unless-stopped
logging: *default-logging
networks: [backend]
command: ['-js', '-sd', '/data', '-m', '8222']
volumes:
- natsdata:/data
healthcheck:
test: ['CMD', 'wget', '-q', '--spider', 'http://127.0.0.1:8222/healthz']
interval: 10s
timeout: 3s
retries: 5
start_period: 10s
start_interval: 1s
redis:
image: redis:7-alpine
restart: unless-stopped
logging: *default-logging
networks: [backend]
command: ['redis-server', '--save', '', '--appendonly', 'no']
healthcheck:
test: ['CMD', 'redis-cli', 'ping']
interval: 10s
timeout: 3s
retries: 5A few of the decisions, one by one:
One database per service, but one shared PostgreSQL instance. “When to Split: Service Boundaries and Decomposition Strategies” stressed that "each service owns its data": order may not read inventory's tables directly. There are three ways to apply this principle at deployment time:
| Approach | Isolation | Cost | Suits |
|---|---|---|---|
| One PostgreSQL per service | Highest: versions, resources and failures all separate | Highest | Production, very different loads |
| Same instance, separate databases | Medium: accounts and data isolated, resources shared | Low | Development, testing, small single-host deployments |
| Same database, separate schemas | Low: relies on permissions to block cross-schema access | Lowest | When resources are extremely tight |
In a Compose setting, the second is a good balance: each service has its own account and database with no permission to touch anyone else's data, yet only one PostgreSQL container runs. The key is that each service uses its own account rather than everyone using the postgres superuser; otherwise "each service owns its data" is only a verbal agreement.
Creating the accounts and databases is left to the official image's initialisation mechanism: scripts in /docker-entrypoint-initdb.d/ run once, when the data directory is first created. Because the passwords live in secrets, a shell script reads them:
#!/bin/sh
# deploy/compose/postgres/initdb/10-create-databases.sh
# Runs only when the pgdata volume is first initialised.
set -eu
for svc in order payment inventory; do
pw="$(cat "/run/secrets/${svc}_db_password")"
psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --dbname postgres <<SQL
CREATE ROLE ${svc} LOGIN PASSWORD '${pw}';
CREATE DATABASE ${svc} OWNER ${svc};
REVOKE ALL ON DATABASE ${svc} FROM PUBLIC;
SQL
doneHealthchecks are the foundation of everything that follows. pg_isready confirms PostgreSQL is accepting connections; NATS's monitoring port 8222 serves /healthz, and the alpine image has busybox's wget available; Redis uses redis-cli ping. start_period is a start-up grace period during which failures don't count towards the retries; start_interval makes checks more frequent during the grace period, so a service is marked healthy sooner once it is ready (this needs a fairly recent Docker Engine and Compose; older versions ignore it or report an error, in which case just remove it).
Redis is used only as a cache and counter here, so persistence is turned off and no volume is mounted; losing the data when the container is recreated is expected. If you store idempotency keys in Redis (“Idempotency and Distributed Locks”), assess the impact of losing them, and if necessary enable AOF and mount a volume.
Always set up log rotation. Docker's default json-file driver never deletes old logs, and one noisy service can quietly fill the host's disk. The x-logging anchor lets every service apply the same limits in one line.
Tip: The image versions (
postgres:17-alpine,nats:2.10-alpineand so on) are only illustrative in this article. In real use, pin them to versions you have verified, ideally pinning the digest too, so that a re-pull one day doesn't quietly upgrade you.
Step 4: Add the four services and use networks to decide who can see whom
With the infrastructure ready, add the four Go services. They share a lot of settings (restart policy, logging, healthcheck, shared env file), which also go into an anchor:
x-go-service: &go-service
restart: unless-stopped
logging: *default-logging
networks: [backend]
healthcheck:
test: ['CMD', '/app/server', 'healthcheck']
interval: 10s
timeout: 3s
retries: 3
start_period: 15s
start_interval: 1s
services:
order:
<<: *go-service
build: &order-build
context: ../..
dockerfile: deploy/docker/Dockerfile
args:
SERVICE: order
image: shop/order:${TAG:-dev}
env_file:
- ./env/common.env
- ./env/order.env
environment:
DB_HOST: postgres
DB_NAME: order
DB_USER: order
DB_PASSWORD_FILE: /run/secrets/order_db_password
PAYMENT_ADDR: payment:9090
INVENTORY_ADDR: inventory:9090
NATS_URL: nats://nats:4222
REDIS_ADDR: redis:6379
secrets: [order_db_password]
depends_on:
order-migrate:
condition: service_completed_successfully
nats:
condition: service_healthy
redis:
condition: service_healthy
payment:
<<: *go-service
build: &payment-build
context: ../..
dockerfile: deploy/docker/Dockerfile
args:
SERVICE: payment
image: shop/payment:${TAG:-dev}
networks: [backend, egress] # needs to reach the external payment provider
env_file:
- ./env/common.env
- ./env/payment.env
environment:
DB_HOST: postgres
DB_NAME: payment
DB_USER: payment
DB_PASSWORD_FILE: /run/secrets/payment_db_password
NATS_URL: nats://nats:4222
secrets: [payment_db_password]
depends_on:
payment-migrate:
condition: service_completed_successfully
nats:
condition: service_healthy
# inventory has the same shape as payment, just without the egress network; omitted.
api-gateway:
<<: *go-service
build:
context: ../..
dockerfile: deploy/docker/Dockerfile
args:
SERVICE: api-gateway
image: shop/api-gateway:${TAG:-dev}
networks: [edge, backend]
ports:
- '${GATEWAY_PORT:-8080}:8080'
env_file:
- ./env/common.env
- ./env/api-gateway.env
environment:
ORDER_ADDR: order:9090
PAYMENT_ADDR: payment:9090
INVENTORY_ADDR: inventory:9090
REDIS_ADDR: redis:6379
depends_on:
order:
condition: service_healthy
payment:
condition: service_healthy
inventory:
condition: service_healthy
redis:
condition: service_healthy
networks:
edge: {}
backend:
internal: true
egress: {}This step has three core ideas.
The service name is the host name. Compose creates networks for the project, and Docker's built-in DNS server lets containers on the same network resolve each other by service name: order connects to postgres:5432, api-gateway connects to order:9090, with no need to know any IP. This is the simplest form of the "platform-provided service discovery" mentioned in “Service Discovery and Load Balancing for Go Microservices”. If you later run three copies of inventory with docker compose up --scale inventory=3, a DNS query for inventory returns three addresses; a gRPC client using dns:///inventory:9090 with the round_robin policy spreads requests across the three containers. Be aware that an HTTP client reusing long-lived connections may stay stuck to one container, the same situation discussed in that article.
Only api-gateway publishes a port. ports binds a container port to the host, and only api-gateway has that line. order, payment and inventory listen on 9090 and 8080 inside the container network, but cannot be reached from the host or from outside. This is not just a security benefit; it also guarantees that all traffic passes through api-gateway's authentication and rate limiting, so nobody can bypass the entry point and hit a backend directly.
Use networks to carve out trust zones. Three networks are defined here:
edge: api-gateway (and later the developer tools) lives here and faces the outside.backend: every service and all infrastructure are here. It is set tointernal: true, which means the network has no outbound route, so containers cannot reach the internet.egress: joined only by payment, so it can call the external payment provider's API.
internal: true is a very useful but often overlooked setting: even if some dependency has been compromised with malicious code, inside the backend network it has no way to send data out. The price is that when a service really does need outbound access (payments, sending email, third-party APIs) it must explicitly join another network, and that is exactly what we want: outbound access becomes a visible decision that needs review.
Figure: The outside world can only reach api-gateway on port 8080; the backend services and data stores sit on the backend network, which has no outbound route, and only payment additionally joins the egress network to reach the external payment provider.
depends_on is deliberately not written to follow the call graph here. At runtime order calls payment and inventory, but its depends_on only lists the database migration, NATS and Redis: the hard dependencies without which it "cannot start at all". Call dependencies between services should be handled at runtime by the mechanisms in “Timeouts, Retries and Circuit Breakers”: if the other side is temporarily absent, retry or return an error, rather than forcing the whole system to start in a particular order. If you copy the full call graph into depends_on, restarting any one service can drag a chain of others to a halt, and it is easy to create circular dependencies. Making api-gateway depend on the health of the three backends is a convenience: in a development environment, by the time api-gateway can accept traffic, the backends are guaranteed to be ready.
Step 5: Configuration, secrets and env files
Configuration comes in layers, each with a different purpose:
| Location | Read by | Contains | In version control |
|---|---|---|---|
.env | Compose, while parsing the YAML | Substitution variables such as TAG, GATEWAY_PORT | Yes |
env/*.env (env_file) | The program in the container | Non-secret settings such as log level, timeouts, feature flags | Yes |
environment: | The program in the container | Topology-related addresses (order:9090) | Yes |
secrets/ (secrets) | The program in the container (reads a file) | Database passwords, API keys | No |
The example env files are short, because they hold only values that "may differ per environment but are not secret":
# deploy/compose/env/common.env — shared by all four services
LOG_LEVEL=info
LOG_FORMAT=json
OTEL_EXPORTER_OTLP_ENDPOINT=http://jaeger:4317
SHUTDOWN_TIMEOUT=20s
# deploy/compose/env/order.env — used only by order
ORDER_SAGA_TIMEOUT=30s
OUTBOX_POLL_INTERVAL=500msWhen env_file lists several files, later ones override earlier ones, and environment: in turn overrides env_file. Make use of that order: shared defaults go in common.env, service-specific values go in the service's own file, and addresses tied to the Compose topology go directly in environment:, because they follow the service names in compose.yaml.
Secrets are declared in the top-level secrets: and then listed in each service that needs them:
secrets:
postgres_password:
file: ./secrets/postgres_password.txt
order_db_password:
file: ./secrets/order_db_password.txt
payment_db_password:
file: ./secrets/payment_db_password.txt
inventory_db_password:
file: ./secrets/inventory_db_password.txtEach secret a service lists appears as a file at /run/secrets/<name> in the container. The program reads it through DB_PASSWORD_FILE, that is, the envOrFile from Step 2. Each service can only get its own password: the order container does not have payment's database password, so even if order is compromised, it doesn't leak anyone else's credentials along the way.
Why not just put passwords in environment variables? Because environment variables leak easily: docker inspect shows them, debug output on a crash may print them, and child processes inherit them. File-based secrets at least move them out of those channels. To be honest, though: in single-host Compose, a file: secret is essentially a host file mounted read-only into the container, with no encryption. What it protects against is "being scattered where it shouldn't appear", not "staying safe if the host is compromised". Tighten the permissions on the host's secrets/ directory (for example, readable only by the deployment account).
Tip:
docker compose configprints the final configuration after variable substitution, anchor merging and stacking of multiple files. When you wonder "why isn't the value in the container what I expected?", look at that first.
Step 6: Startup order and database migrations
With healthchecks in place, the startup order can be written very precisely. The goal is:
- PostgreSQL, NATS and Redis start first, and we wait until they are healthy.
- Each service's database migration runs as a one-off job and exits when done.
- Only after the migration completes successfully does the corresponding service start, and we wait for it to become healthy.
- Only when all three backends are healthy does api-gateway start and begin accepting traffic.
A migration is a "one-off service" that uses the same image with a different command:
services:
order-migrate:
build: *order-build
image: shop/order:${TAG:-dev}
command: ['migrate', 'up']
restart: 'no'
logging: *default-logging
networks: [backend]
environment:
DB_HOST: postgres
DB_NAME: order
DB_USER: order
DB_PASSWORD_FILE: /run/secrets/order_db_password
secrets: [order_db_password]
depends_on:
postgres:
condition: service_healthy
restart: trueA few details are worth explaining:
build: *order-buildreuses order's build settings, so both produce the same image tag. That waydocker compose upwon't try to pull ashop/order:devthat doesn't exist yet from a remote registry.command: ["migrate", "up"]is appended to the Dockerfile'sENTRYPOINT ["/app/server"], so what actually runs is/app/server migrate up.restart: "no"makes it stop when finished instead of being restarted over and over.- order's
depends_onusescondition: service_completed_successfully: the migration's exit code must be 0 before order starts. If the migration fails, order does not start anddocker compose upreports an error, rather than letting the service run against the old table structure. - Where postgres is a dependency,
restart: trueis added: if postgres is later restarted by hand, Compose also restarts the services that depend on it, avoiding leftover broken connections.
Taking migrations out of the service's startup path, rather than running them automatically at the top of main, has several benefits: multiple replicas don't race to change the schema; when a migration fails, the error is in one place; and in production the same command can be run manually. The Kubernetes Job in the next article is exactly the same idea.
Figure: Migrations run only after the infrastructure is healthy, services start only after their migrations complete successfully, and api-gateway faces the outside only after all three backends are healthy.
All commands in this article are run from the deploy/compose/ directory; when running from the repository root, add -f deploy/compose/compose.yaml. When actually starting up, use --wait so the command doesn't return until every service is healthy (or every one-off job has completed successfully), which is ideal in front of CI integration tests:
# Build and start the whole system; return only once everything is healthy
docker compose up -d --build --wait
# Show each container's state and health
docker compose ps
# Just re-run order's migration by hand
docker compose run --rm order-migrateTip:
depends_ononly governs "the moment of startup". If PostgreSQL restarts after the system is running, the services still need to be able to reconnect. Startup order is a tool for a smooth development experience; it cannot replace the resilience of the program itself.
Step 7: Day-to-day operations: reading logs, updating a single service
Once the system is running, the most common day-to-day tasks are reading logs and changing one particular service.
Logs: each service writes structured JSON to standard output as in “Observability in Go: Logs, Metrics and Tracing”. Compose collects the output of all containers and prefixes each line with the service name:
# Follow the latest logs from order and payment
docker compose logs -f --tail=100 order payment
# Only the last 10 minutes, filtering errors with jq
docker compose logs --since 10m --no-log-prefix order | jq -c 'select(.level == "ERROR")'Updating just one service: after changing order's code, there is no need to restart the whole system:
# Rebuild order's image and recreate only the order container (leaving its dependencies alone)
docker compose up -d --build --no-deps order
# When the schema changes, run the migration first, then swap the service
docker compose build order
docker compose run --rm order-migrate
docker compose up -d --no-deps order--no-deps matters. Without it, Compose checks and starts all of order's dependencies; it usually won't recreate containers that are already running, but adding the flag makes the intent explicit: touch only this one. Also be aware that Compose replaces a container by stopping the old one first, then starting the new one, leaving a short gap in between; api-gateway's calls to order fail during that window and have to be ridden out with retries. This is one of the biggest differences between Compose and a Kubernetes rolling update.
During development you can go further with develop.watch, letting Compose watch the source code and rebuild automatically on changes:
services:
order:
develop:
watch:
- action: rebuild
path: ./internal/order
- action: rebuild
path: ./cmd/orderThen run docker compose watch to start watching. Go is a compiled language, so rebuild is the most direct action; sync, which copies files straight into the container, is more commonly used with interpreted languages.
A few commands to be careful with:
| Command | Effect | Watch out |
|---|---|---|
docker compose stop | Stops containers, keeping containers and data | Safe |
docker compose down | Removes containers and networks, keeps named volumes | Safe, the data is still there |
docker compose down -v | Also removes named volumes | Database data is lost |
docker compose restart | Restarts containers, without re-reading compose.yaml | Use up -d after config changes |
docker compose exec order … | Runs a command inside a running container | distroless has no shell |
Step 8: Use profiles to switch developer and observability tools on and off
Some containers aren't needed every time: a database admin UI, the NATS command-line tools, Prometheus, Grafana, Jaeger. Put them in the same compose.yaml but add profiles, and the default docker compose up won't start them:
services:
adminer:
image: adminer:5
profiles: [dev]
networks: [edge, backend]
ports:
- '127.0.0.1:8081:8080'
nats-box:
image: natsio/nats-box:latest
profiles: [dev]
networks: [backend]
environment:
NATS_URL: nats://nats:4222
prometheus:
image: prom/prometheus:v3.5.0
profiles: [observability]
networks: [edge, backend]
ports:
- '127.0.0.1:9090:9090'
volumes:
- ./prometheus/prometheus.yml:/etc/prometheus/prometheus.yml:ro
- promdata:/prometheus
grafana:
image: grafana/grafana:12.1.0
profiles: [observability]
networks: [edge, backend]
ports:
- '127.0.0.1:3000:3000'
volumes:
- grafanadata:/var/lib/grafana
jaeger:
image: jaegertracing/jaeger:2.9.0
profiles: [observability]
networks: [edge, backend]
ports:
- '127.0.0.1:16686:16686'
volumes:
pgdata: {}
natsdata: {}
promdata: {}
grafanadata: {}The developer tools' ports are all bound to 127.0.0.1, so only a browser on the local machine can reach them; even if the machine has a public IP, nothing is exposed by accident. This is consistent with the "only api-gateway faces the outside" principle: api-gateway is the entry point for users, and the other ports are just windows for developers.
Prometheus's scrape configuration also uses service names directly:
# deploy/compose/prometheus/prometheus.yml
global:
scrape_interval: 15s
scrape_configs:
- job_name: shop
static_configs:
- targets:
- api-gateway:8080
- order:8080
- payment:8080
- inventory:8080There are two ways to enable them:
# On the command line
docker compose --profile dev --profile observability up -d
# Or with an environment variable (it can also go in .env)
COMPOSE_PROFILES=observability docker compose up -d
# Use run for one-off tools; they don't need to stay running
docker compose --profile dev run --rm nats-box nats stream lsFigure: Services without a profile always start; the dev and observability groups join only when specified.
There is one rule to follow here: services that start by default must not depend on services that have a profile. If order had depends_on: jaeger and jaeger only starts under observability, a normal up makes Compose report an error. Exporting trace data should be designed so that "it's fine if the collector isn't there": when the OpenTelemetry exporter can't connect, it drops the data and logs an error, without stopping the service.
Step 9: How far Compose can go, and when to switch
Putting the previous steps together, one compose.yaml describes the complete order system: four services, three kinds of infrastructure, migrations, network isolation, secrets and optional observability tools. It is very useful in several scenarios:
- Local development: a new developer clones the repo, prepares
secrets/, and has a complete environment with one command. - CI integration tests: run end-to-end tests after
docker compose up -d --wait, then clean up withdocker compose down -v, so every run gets a clean environment. - Demo and preview environments: bring up one stack per feature branch, separated by project name (
-p). - Small single-host production deployments: systems with modest traffic, where brief downtime during updates is acceptable and one host is enough.
But as the system heads into production, the limits of Compose become very clear:
| Requirement | Compose | Kubernetes equivalent |
|---|---|---|
| Multiple hosts | Essentially a single-host tool | Cluster scheduling |
| Host failure | restart can save a container, not a host | Pods are rescheduled onto other nodes |
| Zero-downtime updates | Stop then start, with a gap | Rolling updates, gated by readiness |
| Scaling with load | --scale is manual, and services that bind a host port can't run more copies | HPA adjusts the replica count automatically |
| Secret management | Files on the host, unencrypted | Secrets plus external key management |
| Service-to-service access control | Coarse network-level separation | NetworkPolicy, down to each service |
| Rollback | Remember the previous tag yourself and up again | kubectl rollout undo |
This doesn't mean Compose can't go to production; it means you must know clearly which limitations you are accepting: a single point of failure, brief interruptions during updates, manual scaling. If all of those are acceptable, one host plus one compose.yaml is a very simple, easy-to-understand architecture; if not, that is what the next article, “Deploying a Microservice System on Kubernetes”, deals with. The good news is that the habits built in this article (one image with several subcommands, healthchecks, migrations as one-off jobs, the _FILE secret convention, finding each other by service name) all carry straight over to Kubernetes.
Common pitfalls
- Writing
depends_onin the short syntax.depends_on: [postgres]only guarantees the container has started, not that PostgreSQL can accept connections. Always use the long syntax withcondition: service_healthy, and the dependency must have ahealthcheck, otherwise the condition is never met. - Writing a
curlhealthcheck for a distroless image. The image has no curl, so the check always fails and the service is forever "unhealthy". Use the program's built-inhealthchecksubcommand instead. - Confusing
.envwithenv_file..envis only for Compose's${VAR}substitution and does not automatically reach the container; to give a container a variable, it must go throughenv_fileorenvironment. - Assuming a changed initdb script will run again.
/docker-entrypoint-initdb.d/runs only when the data directory is empty. If the volume already exists, handle it with a migration, or in development delete the volume and start over. - Hard-coding passwords in compose.yaml. That file goes into version control, and the password goes with it. Use
secretswith files excluded by.gitignore. - Publishing ports everywhere. Adding
ports: ["5432:5432"]to the database for easier debugging and forgetting to remove it binds the database to every interface on the host. If you really need to connect, bind to127.0.0.1and put it in thedevprofile. - Assuming
docker compose restartapplies new configuration. It only restarts existing containers; after changing compose.yaml or an env file, rundocker compose up -dso Compose compares the differences and recreates. - Running
down -vby accident. This deletes named volumes, and the database data disappears with them. Production data needs backups, not just a volume. - Letting Docker logs grow without limit.
json-filelogs withoutmax-sizekeep growing until they fill the disk. - Putting the whole call graph into
depends_on. This causes slow startup, cascading restarts and even circular dependencies. List only hard dependencies, and rely on retries and circuit breakers for the rest.
Key takeaways
- One
compose.yaml(withoutversion:) describes the whole system: four services, PostgreSQL, NATS, Redis, migrations and optional tools. - One Dockerfile with a
SERVICEbuild arg is shared by the four services; the binary providesserve,healthcheckandmigratesubcommands. - Services connect to each other by service name; only api-gateway publishes a port, the backend network is set to
internal: true, and services that need outbound access explicitly join an egress network. - Each service owns its own database and account; development and single-host environments can share one PostgreSQL instance.
- Configuration is layered:
.envfor Compose,env_filefor containers,environmentfor topology addresses,secretsfor secrets, read through the_FILEconvention. - Startup order relies on the long
depends_onsyntax: infrastructureservice_healthy→ migrationsservice_completed_successfully→ servicesservice_healthy→ api-gateway. depends_onlists only hard dependencies; temporary failures in calls between services are handled by retries, timeouts and circuit breakers.- Update a single service with
docker compose up -d --build --no-deps <service>; observe and debug withlogs -f,psandconfig. - Developer and observability tools go into
profiles, with their ports bound to127.0.0.1. - Compose suits local development, CI, previews and small single-host deployments; when you need multiple hosts, zero-downtime updates, autoscaling and fine-grained access control, hand over to Kubernetes.
Next
Compose makes the whole system run smoothly on one host, but it cannot give you multiple hosts, zero-downtime updates, autoscaling or fine-grained access control between services. The next article, “Deploying a Microservice System on Kubernetes”, moves the same services into a cluster and fills those gaps with Deployments, HPA, NetworkPolicy and Kustomize.