This resume is also available in English.

6. Deploying the system2 / 5 · All columns

Columns › 6. Deploying the system › Part 22

Deploying a Microservice System on Kubernetes

在 Kubernetes 部署整套微服務

Move the api-gateway, order, payment and inventory services onto Kubernetes with a Deployment and Service per service, Gateway API routing, NetworkPolicy, HPA and PDB, Kustomize overlays for dev and prod, and a release flow covering image tags, migrations, rollouts, rollback and debugging.

  • Kubernetes
  • Kustomize
  • Microservices
  • Go
In this article
  1. Step 1: Plan namespaces, labels and the directory structure
  2. Step 2: One Deployment and Service per service
  3. Step 3: Each service's own ConfigMap and Secret
  4. Step 4: Stateful dependencies — databases and message queues
  5. Step 5: The external entry point and NetworkPolicy
  6. Step 6: Probes, resources, HPA and PDB
  7. Step 7: Combine base with dev/prod overlays using Kustomize
  8. Step 8: Migration Jobs, the release flow, rolling updates and rollback
  9. Step 9: Debugging when things go wrong
  10. Common pitfalls
  11. Key takeaways
  12. Next

In the previous article, “Deploying a Whole Microservice System with Docker Compose”, we used a single compose.yaml to run four services together on one host. This article moves the same system into a Kubernetes cluster and adds what Compose cannot give you: multiple hosts, zero-downtime updates, autoscaling and fine-grained access control.

“Deploying One Go Service with Docker and Kubernetes” already showed how to put one service on a cluster: one Deployment, one Service, three probes. Going from one to four, the hard part is not "copy the YAML four times" but the problems that only show up at the system level:

  • With four services and two or more environments, how do you organise the configuration so nothing is duplicated everywhere, yet you can still see exactly how each environment differs?
  • How do services find each other? How does external traffic get in, and how do you guarantee it can only come in through the entry point?
  • Should stateful components such as databases and message queues live in the cluster or be handed to a managed service?
  • When should database migrations run during a release? How do you go back when a new version is broken?
  • When something goes wrong, where do you start looking?

This article continues with the order system used throughout the series. Starting from the directory structure, it builds step by step a configuration you can actually apply to a cluster. All API versions in the examples are the current stable ones: apps/v1, autoscaling/v2, policy/v1, networking.k8s.io/v1, and Gateway API's gateway.networking.k8s.io/v1. Configuration is managed with Kustomize, which is built into kubectl.

Step 1: Plan namespaces, labels and the directory structure

First decide three things that affect every file that follows.

One: how to separate environments. There are two common approaches: one cluster per environment, or one namespace per environment within the same cluster. Putting production and development in different clusters gives the strongest isolation; a mistake or resource exhaustion in one environment cannot spill over into the other. Sharing one cluster between development and test, each in its own namespace, saves a lot of cost. In this article's examples the production namespace is shop and the development one is shop-dev. If your development environment is in a separate cluster, both can be called shop and nothing in the configuration below changes — because services address each other by short names, as explained in Step 2.

Two: label conventions. Kubernetes ties resources together with labels: a Service selects Pods by label, a NetworkPolicy uses labels to decide who may connect to whom, and a PDB uses labels to know which Pods to protect. Once labels get messy, these mechanisms silently stop working. The examples consistently use the labels Kubernetes recommends:

LabelPurposeExample values
app.kubernetes.io/nameWhich service; selectors use thisorder, api-gateway
app.kubernetes.io/part-ofWhich system it belongs toshop
app.kubernetes.io/componentThe role of the resourcemigrate, config

Three: the directory structure. Following the series' repository layout, the Kubernetes configuration lives in deploy/k8s/, split into base (shared by all environments) and overlays (the differences for each environment):

shop/                                  # module path example.com/shop
├── cmd/{api-gateway,order,payment,inventory}/main.go
├── internal/ …
├── api/proto/ …
└── deploy/
    ├── docker/Dockerfile              # shared by all four services (previous article)
    ├── compose/ …                     # the Compose configuration from the previous article
    └── k8s/
        ├── base/
        │   ├── kustomization.yaml     # lists the four services and shared resources
        │   ├── httproute.yaml         # external route to api-gateway
        │   ├── networkpolicy.yaml     # who may connect to whom
        │   ├── api-gateway/
        │   │   ├── kustomization.yaml
        │   │   ├── config.env
        │   │   ├── deployment.yaml
        │   │   ├── service.yaml
        │   │   ├── hpa.yaml
        │   │   └── pdb.yaml
        │   ├── order/                 # same as above, plus migrate-job.yaml
        │   ├── payment/               # same as order
        │   └── inventory/             # same as order
        └── overlays/
            ├── dev/
            │   ├── kustomization.yaml # namespace: shop-dev, secrets, fewer replicas
            │   ├── namespace.yaml
            │   ├── data/              # postgres, nats, redis for development
            │   └── secrets/           # not in version control
            └── prod/
                ├── kustomization.yaml # namespace: shop, image tags, more resources
                ├── namespace.yaml
                └── patches/

One subdirectory per service, each with its own kustomization.yaml, means "everything about order" lives in base/order/: adding a service is copying a directory, renaming it and adding one line to the list one level up.

Users Gateway (gateway-system namespace) HTTPRoute: shop.example.com Namespace shop Service api-gateway Pod × 2–N (HPA) Service order Service payment Service inventory Pod × 3–20 Pod × 2–10 Pod × 2–10 Services call each other via DNS: order.shop.svc.cluster.local:9090 (order:9090 within the namespace) Managed PostgreSQL One database per service NATS nats namespace (Helm) Managed Redis Cache, rate limiting, idempotency keys

Figure: External traffic reaches api-gateway through the Gateway and HTTPRoute, and backend services call each other via cluster DNS; in production the database and cache are outside the cluster, and NATS lives in its own namespace.

Step 2: One Deployment and Service per service

Write order in full first; the other three services follow the same shape. The Deployment describes "what to run and how to run it":

# deploy/k8s/base/order/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: order
  labels:
    app.kubernetes.io/name: order
spec:
  # No replicas: the replica count belongs to the HPA, otherwise every apply resets it
  revisionHistoryLimit: 5
  selector:
    matchLabels:
      app.kubernetes.io/name: order
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 0
  template:
    metadata:
      labels:
        app.kubernetes.io/name: order
    spec:
      terminationGracePeriodSeconds: 30
      securityContext:
        runAsNonRoot: true
        seccompProfile:
          type: RuntimeDefault
      topologySpreadConstraints:
        - maxSkew: 1
          topologyKey: kubernetes.io/hostname
          whenUnsatisfiable: ScheduleAnyway
          labelSelector:
            matchLabels:
              app.kubernetes.io/name: order
      containers:
        - name: app
          image: registry.example.com/shop/order:latest
          args: ['serve']
          ports:
            - name: http
              containerPort: 8080
            - name: grpc
              containerPort: 9090
          envFrom:
            - configMapRef:
                name: order-config
          env:
            - name: DB_PASSWORD_FILE
              value: /etc/shop/secrets/db-password
            - name: GOMEMLIMIT
              valueFrom:
                resourceFieldRef:
                  resource: limits.memory
          volumeMounts:
            - name: secrets
              mountPath: /etc/shop/secrets
              readOnly: true
          resources:
            requests:
              cpu: 100m
              memory: 128Mi
            limits:
              memory: 256Mi
          startupProbe:
            httpGet:
              path: /livez
              port: http
            periodSeconds: 2
            failureThreshold: 30
          readinessProbe:
            httpGet:
              path: /readyz
              port: http
            periodSeconds: 5
            failureThreshold: 3
          livenessProbe:
            httpGet:
              path: /livez
              port: http
            periodSeconds: 10
            failureThreshold: 3
          lifecycle:
            preStop:
              sleep:
                seconds: 5
          securityContext:
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: true
            capabilities:
              drop: ['ALL']
      volumes:
        - name: secrets
          secret:
            secretName: order-secrets

This file is longer than the one in the single-service article, and every extra part is there for a reason:

  • The tag on image doesn't matter. latest in base is only a placeholder; the real tag is rewritten to a git SHA by the overlay's images field (Steps 7 and 8).
  • args: ["serve"] matches the subcommand design from the previous article: the same image runs the service with serve and runs migrations with migrate up.
  • envFrom reads the ConfigMap, and the password is mounted as a file, paired with the previous article's envOrFile: the program reads the file that DB_PASSWORD_FILE points to. Compose's /run/secrets/… and the /etc/shop/secrets/… here differ only in path; the code doesn't change at all.
  • GOMEMLIMIT takes its value from the memory limit. By default the Go runtime doesn't know the container's memory limit; once it is set, the garbage collector collects more aggressively as it approaches the limit, reducing the chance of being OOMKilled. The memory curves measured in “Performance Tuning in Go: pprof and Benchmarks” are exactly what you use to decide this limit.
  • topologySpreadConstraints spreads a service's Pods across different nodes as far as possible, so one failed node doesn't take every replica with it.
  • The preStop sleep makes the Pod wait a few seconds before it receives the termination signal, giving Service endpoints and load balancers time to remove it from their lists, so no new requests arrive while it is shutting down. Distroless images have no sleep command, so this uses Kubernetes' built-in sleep action (supported only in newer Kubernetes versions; on older clusters the program can delay by itself after receiving the signal). After that, the program handles SIGTERM as in “Configuration and Graceful Shutdown in Go” and finishes up within terminationGracePeriodSeconds.
  • securityContext runs as non-root, makes the root filesystem read-only and drops all Linux capabilities. A static Go binary never needed these privileges, so tightening them costs nothing.

The Service gives this set of Pods a stable name and address:

# deploy/k8s/base/order/service.yaml
apiVersion: v1
kind: Service
metadata:
  name: order
  labels:
    app.kubernetes.io/name: order
spec:
  selector:
    app.kubernetes.io/name: order
  ports:
    - name: http
      port: 8080
      targetPort: http
    - name: grpc
      port: 9090
      targetPort: grpc

How services find each other. Cluster DNS creates a <service>.<namespace>.svc.cluster.local record for every Service: in the shop namespace, order's full name is order.shop.svc.cluster.local. Pods in the same namespace can write just order, and the DNS search domains fill in the rest. So api-gateway's configuration says ORDER_ADDR=order:9090 — exactly as in the previous article's Compose file, and unchanged whether it is deployed to shop or shop-dev. Only across namespaces do you need a longer name; for example, production NATS lives in the nats namespace, so you write nats.nats.svc.cluster.local.

Tip: gRPC uses long-lived HTTP/2 connections. An ordinary ClusterIP Service load-balances when a connection is established; once the connection is up, every request goes to the same Pod, and adding replicas doesn't spread the traffic by itself. Both solutions from “Service Discovery and Load Balancing for Go Microservices” apply here: create an extra headless Service with clusterIP: None so the gRPC client gets every Pod address via dns:///order-headless:9090 and spreads requests with round_robin, or let a service mesh load-balance per request.

The Deployments and Services for api-gateway, payment and inventory have the same structure, with only names and labels changed; api-gateway doesn't connect to a database, so it has no secret mount and only exposes port 8080.

Step 3: Each service's own ConfigMap and Secret

Configuration and secrets need to achieve two things: each service gets only its own, and Pods are replaced automatically when configuration changes. Kustomize's generators do both.

In each service's subdirectory, use configMapGenerator to generate a ConfigMap from an env file:

# deploy/k8s/base/order/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml
  - service.yaml
  - hpa.yaml
  - pdb.yaml
  - migrate-job.yaml
configMapGenerator:
  - name: order-config
    envs:
      - config.env
    options:
      labels:
        app.kubernetes.io/component: config
# deploy/k8s/base/order/config.env
LOG_LEVEL=info
LOG_FORMAT=json
DB_HOST=postgres
DB_NAME=order
DB_USER=order
PAYMENT_ADDR=payment:9090
INVENTORY_ADDR=inventory:9090
NATS_URL=nats://nats:4222
REDIS_ADDR=redis:6379
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector.monitoring:4317

There is one key detail in what the generator does: the generated ConfigMap's name carries a hash of its contents, for example order-config-t59gh52tg6, and the order-config written in the Deployment is automatically rewritten to that hashed name. This brings two benefits:

  1. Change the configuration and Pods roll automatically. When an ordinary ConfigMap is modified, running Pods don't re-read their environment variables; with the generator, a change in content changes the name, which changes the Deployment's Pod template, so Kubernetes performs a rolling update.
  2. Rollback brings the configuration back too. The old ReplicaSet references the ConfigMap by its old name, so with kubectl rollout undo code and configuration move together; you never get "old code, new configuration".

options.labels adds app.kubernetes.io/component: config to the ConfigMap. This is for the release flow in Step 8: apply only configuration and migration jobs first, then everything else.

Secrets come from different sources in different environments. In development, secretGenerator can generate them from local files (the files are not in version control):

# deploy/k8s/overlays/dev/kustomization.yaml (excerpt)
secretGenerator:
  - name: order-secrets
    files:
      - db-password=secrets/order-db-password.txt
    options:
      labels:
        app.kubernetes.io/component: config

In production, passwords should never appear in any form in the repository or on the CI filesystem. Common approaches are the External Secrets Operator syncing a Secret named order-secrets from a cloud key management service, Sealed Secrets putting encrypted ciphertext in the repository, or the Secrets Store CSI Driver mounting keys straight into the Pod. Whichever you use, to the Deployment it is just "there is a Secret called order-secrets"; base doesn't need to know where it comes from.

ItemWhere it livesExample
Configuration same in every envconfig.env in baseLog format, short service-to-service names
Configuration that differs per envConfigMap merged in the overlayDatabase host, cache address
Development secretsThe overlay's secretGeneratorTest passwords generated locally
Production secretsA key management service outsideSynced by tools such as External Secrets

Tip: Kubernetes Secrets are only base64-encoded by default, not encrypted. Make sure the cluster has encryption at rest enabled for etcd, and use RBAC to restrict who can read Secrets; anyone who can create Pods in a namespace can in practice read every Secret in that namespace.

Step 4: Stateful dependencies — databases and message queues

The microservices themselves are stateless; they can be killed, rescheduled and scaled at any time. Databases and message queues cannot. Where they live is one of the most important decisions when deploying a whole system:

OptionAdvantagesCosts
Managed cloud databaseSomeone else handles backups, patching, HA and capacityPrice, cloud lock-in
In-cluster OperatorDeclarative management, still with HA and automatic backupsYou must understand the Operator; storage must be reliable
Hand-written StatefulSetFull control, fastest to get workingBackups, upgrades and failover are all on you

A pragmatic combination: use a managed service for the production database, and a simple in-cluster StatefulSet for development. For the message queue it depends on the team's skills: components such as NATS and RabbitMQ have officially maintained Helm charts or Operators; install them in their own namespace (for example nats), and the four services connect via nats.nats.svc.cluster.local.

The development PostgreSQL can reuse the previous article's init script directly:

# deploy/k8s/overlays/dev/data/postgres.yaml
apiVersion: v1
kind: Service
metadata:
  name: postgres
spec:
  clusterIP: None
  selector:
    app.kubernetes.io/name: postgres
  ports:
    - name: postgres
      port: 5432
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
spec:
  serviceName: postgres
  replicas: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: postgres
  template:
    metadata:
      labels:
        app.kubernetes.io/name: postgres
    spec:
      containers:
        - name: postgres
          image: postgres:17-alpine
          env:
            - name: POSTGRES_USER
              value: postgres
            - name: POSTGRES_PASSWORD_FILE
              value: /run/secrets/postgres_password
            - name: PGDATA
              value: /var/lib/postgresql/data/pgdata
          ports:
            - name: postgres
              containerPort: 5432
          readinessProbe:
            exec:
              command: ['pg_isready', '-U', 'postgres', '-d', 'postgres']
            periodSeconds: 5
          resources:
            requests:
              cpu: 100m
              memory: 256Mi
            limits:
              memory: 512Mi
          volumeMounts:
            - name: data
              mountPath: /var/lib/postgresql/data
            - name: secrets
              mountPath: /run/secrets
              readOnly: true
            - name: initdb
              mountPath: /docker-entrypoint-initdb.d
              readOnly: true
      volumes:
        - name: secrets
          secret:
            secretName: postgres-secrets
        - name: initdb
          configMap:
            name: postgres-initdb
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ['ReadWriteOnce']
        resources:
          requests:
            storage: 5Gi

The difference between a StatefulSet and a Deployment is that it gives each Pod a stable name (postgres-0) and its own persistent volume: when the Pod is rescheduled, it mounts the same data again. With a headless Service (clusterIP: None), services still connect with postgres:5432. PGDATA points to a subdirectory because many storage classes put lost+found at the root of the mount point, and PostgreSQL refuses a non-empty directory during initialisation.

The secrets are mounted at /run/secrets with underscore-style file names such as order_db_password, so the 10-create-databases.sh used with Compose in the previous article doesn't need a single character changed; just put it in a ConfigMap via configMapGenerator and mount it in the init directory. In development, NATS and Redis are each just a Deployment plus a Service, structured much like the above, so they are not repeated here.

In production, the overlay only has to replace the addresses in the ConfigMap with the managed services' host names (Step 7); neither the four services' code nor base changes. This is the payoff for putting "addresses" in configuration rather than hard-coding them in the program.

Tip: When should you use Helm? When you are installing components someone else maintains (NATS, a PostgreSQL Operator, a monitoring stack, a Gateway controller), a Helm chart is the easiest way, because it packages years of accumulated configuration options. Your own four services are simple in structure and differ only in a few fields; with Kustomize you see the final YAML directly, which makes debugging more straightforward. You can also use both: install third-party components with Helm and manage your own services with Kustomize. Only when you need to package the whole system for other teams or customers to install, with lots of parameterisation, is it time to make your own services a Helm chart too.

Step 5: The external entry point and NetworkPolicy

In Compose, "only the gateway is exposed" relied on publishing a single port and putting the backends on an internal network. In Kubernetes this splits into two halves: letting external traffic reach api-gateway, and making every other path a dead end.

External routing. The traditional approach is Ingress (networking.k8s.io/v1); the new standard is Gateway API. Gateway API separates responsibilities: platform administrators own the Gateway (which ports to listen on, which certificate to use), and application teams only write HTTPRoutes (which host name and which paths go to which Service). The Kubernetes community announced the retirement of the ingress-nginx project at the end of 2025, so starting new systems directly on Gateway API is the safer choice:

# deploy/k8s/base/httproute.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: shop
spec:
  parentRefs:
    - name: public
      namespace: gateway-system
  hostnames:
    - shop.example.com
  rules:
    - matches:
        - path:
            type: PathPrefix
            value: /
      backendRefs:
        - name: api-gateway
          port: 8080

This assumes the platform has already created a Gateway named public in the gateway-system namespace and allows routes from other namespaces to attach to it. Note that the route points only to the api-gateway Service: routing, authentication and rate limiting all happen in the code from “Building an API Gateway in Go: Routing, Authentication and Rate Limiting”; the cluster's entry point only handles TLS and hands the traffic in.

If your cluster uses an Ingress controller, the equivalent is:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: shop
spec:
  ingressClassName: example-ingress-class # replace with the cluster's actual IngressClass
  tls:
    - hosts:
        - shop.example.com
      secretName: shop-tls
  rules:
    - host: shop.example.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: api-gateway
                port:
                  number: 8080

NetworkPolicy: make every other path a dead end. By default Kubernetes lets "every Pod connect to every Pod". Even though order's Service isn't exposed externally, any Pod in the cluster (including another team's, or a compromised one) can hit order:9090 directly and bypass api-gateway's authentication. NetworkPolicy fixes this by "deny everything first, then allow rule by rule":

# deploy/k8s/base/networkpolicy.yaml
# 1. Default deny: no Pod in the namespace accepts any incoming connection
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-ingress
spec:
  podSelector: {}
  policyTypes: ['Ingress']
---
# 2. api-gateway only accepts traffic from the Gateway controller's namespace
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: api-gateway-from-edge
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/name: api-gateway
  policyTypes: ['Ingress']
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: gateway-system
      ports:
        - port: 8080
---
# 3. Backend services only accept gRPC calls from api-gateway and order (the Saga coordinator)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: backend-from-callers
spec:
  podSelector:
    matchExpressions:
      - key: app.kubernetes.io/name
        operator: In
        values: ['order', 'payment', 'inventory']
  policyTypes: ['Ingress']
  ingress:
    - from:
        - podSelector:
            matchExpressions:
              - key: app.kubernetes.io/name
                operator: In
                values: ['api-gateway', 'order']
      ports:
        - port: 9090
---
# 4. The monitoring system may scrape /metrics from every service
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: metrics-from-monitoring
spec:
  podSelector:
    matchLabels:
      app.kubernetes.io/part-of: shop
  policyTypes: ['Ingress']
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: monitoring
      ports:
        - port: 8080

NetworkPolicies are additive: if any rule allows the traffic, it gets through; if no rule allows it, rule 1 blocks it. So when reading them, ask "which paths are open", not "which paths are closed". kubernetes.io/metadata.name is a label Kubernetes adds to every namespace automatically, and selecting namespaces with it is more reliable than adding your own label.

Rule 3 is fairly broad: both api-gateway and order can call all three backends. To be stricter, split it into one rule per service, for example inventory only accepts calls from order — the finer the rules, the more they turn "which service depends on which" into executable documentation, at the cost of remembering to update them whenever a new call relationship is added.

gateway-system monitoring Namespace shop (default-deny-ingress) allow :8080 api-gateway :8080/metrics of all Pods allow :9090 (can also call payment, inventory directly) order payment inventory order coordinates the Saga: may call payment, inventory Other Pods Direct calls to backends → denied (no rule allows them)

Figure: Only the Gateway can reach api-gateway, only api-gateway and order can reach the backends' gRPC port, and monitoring can only scrape metrics on 8080; every other connection is blocked by the default rule.

The development environment puts PostgreSQL, NATS and Redis in the same namespace, so they are also caught by the default deny; the dev overlay therefore additionally allows "Pods in the same namespace may connect to the data components". In production the database is outside the cluster; if you want to restrict outbound traffic, you need Egress rules as well, and must remember to allow DNS (port 53 in kube-system, UDP and TCP), or no service will be able to resolve any name.

Tip: NetworkPolicy only takes effect if the cluster's network plugin (CNI) supports it. Some network plugins accept NetworkPolicy objects but don't enforce them at all. Always test after deploying: from a temporary Pod, try to connect to order:9090 — it should fail.

Step 6: Probes, resources, HPA and PDB

For services to "hold up" in the cluster, you need these mechanisms, each doing its own job:

MechanismThe question it answersWhat happens if it's misconfigured
startupProbeHas the program finished starting?Too short: slow-starting services restart endlessly
readinessProbeCan it take traffic right now?Checks too many dependencies: one failure takes all offline
livenessProbeIs the program stuck and in need of a restart?Checks external dependencies: a DB blip restarts every Pod
resourcesHow much does it need, and how much can it use?No requests: scheduling goes wrong, HPA can't compute
HPAHow many replicas when traffic grows?Fights with the Deployment's replicas
PDBHow many can be missing at once during upkeep?Too strict: nodes can never be drained

The probes are already in Step 2's Deployment. What matters is their meaning: /livez only answers "the program itself is alive" and does not check the database; /readyz answers "I can handle requests" and may check whether its own connection pool is ready. When the database drops out briefly, a failing readiness probe takes the Pod out of load balancing temporarily, and it comes back once the connection recovers; if liveness also checked the database, Kubernetes would restart every Pod at once, turning a small problem into a full outage.

For resources, setting only CPU requests and no CPU limits is a common choice for Go services: a CPU limit causes throttling, which makes latency suddenly worse under high load. Memory, on the other hand, must have a limit, with the GOMEMLIMIT from earlier telling the Go runtime where that line is.

The HPA adjusts the replica count based on CPU utilisation:

# deploy/k8s/base/order/hpa.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: order
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: order
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
  behavior:
    scaleDown:
      stabilizationWindowSeconds: 300

averageUtilization is calculated relative to requests, so for Pods without CPU requests the HPA simply cannot work. scaleDown.stabilizationWindowSeconds makes it observe for a while before scaling down, so the replica count doesn't bounce up and down when traffic comes in waves. Each service's HPA is configured independently: api-gateway handles every request while order coordinates Sagas; their load shapes differ, so their replica ranges should be tuned separately. If CPU isn't a good signal (for example inventory mostly waits on the database), you can switch to custom metrics such as requests per second or queue length, which requires installing a metrics adapter.

A PDB protects against "voluntary disruptions" — evictions during node maintenance and cluster upgrades:

# deploy/k8s/base/order/pdb.yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: order
spec:
  maxUnavailable: 1
  selector:
    matchLabels:
      app.kubernetes.io/name: order

maxUnavailable: 1 is used instead of minAvailable because it makes sense at any replica count: with ten replicas only one is evicted at a time, and in development with a single replica eviction is still allowed, so node maintenance never gets stuck. Had it been minAvailable: 1, an environment with only one replica would leave kubectl drain waiting forever.

The pace of a rolling update is set by the Deployment's strategy. maxUnavailable: 0 with maxSurge: 25% means: start the extra new Pods first, and only shut down an old one once a new one passes readiness. Throughout the process the number of serving replicas never drops below the original count, at the cost of needing a little more capacity during the update.

Step 7: Combine base with dev/prod overlays using Kustomize

Everything written so far is in base; now use overlays to assemble two environments. The base entry point lists the four services and the shared resources:

# deploy/k8s/base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
labels:
  - pairs:
      app.kubernetes.io/part-of: shop
resources:
  - api-gateway
  - order
  - payment
  - inventory
  - httproute.yaml
  - networkpolicy.yaml

labels adds app.kubernetes.io/part-of: shop to every resource. It is used here instead of the old commonLabels because the latter also rewrites the Deployment's selector, and a selector cannot be changed once created — if you later want to adjust labels, the old approach makes the apply fail outright.

The development overlay:

# deploy/k8s/overlays/dev/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: shop-dev
resources:
  - namespace.yaml
  - ../../base
  - data/postgres.yaml
  - data/nats.yaml
  - data/redis.yaml
  - data/networkpolicy.yaml
configMapGenerator:
  - name: postgres-initdb
    files:
      - data/10-create-databases.sh
secretGenerator:
  - name: order-secrets
    files:
      - db-password=secrets/order-db-password.txt
    options:
      labels:
        app.kubernetes.io/component: config
  # payment-secrets and inventory-secrets: same as above
  - name: postgres-secrets
    files:
      - postgres_password=secrets/postgres-password.txt
      - order_db_password=secrets/order-db-password.txt
      - payment_db_password=secrets/payment-db-password.txt
      - inventory_db_password=secrets/inventory-db-password.txt
patches:
  - target:
      kind: HorizontalPodAutoscaler
    patch: |-
      - op: replace
        path: /spec/minReplicas
        value: 1
      - op: replace
        path: /spec/maxReplicas
        value: 2
  - target:
      kind: HTTPRoute
      name: shop
    patch: |-
      - op: replace
        path: /spec/hostnames/0
        value: dev.shop.example.com
images:
  - name: registry.example.com/shop/api-gateway
    newTag: dev
  - name: registry.example.com/shop/order
    newTag: dev
  - name: registry.example.com/shop/payment
    newTag: dev
  - name: registry.example.com/shop/inventory
    newTag: dev

The production overlay:

# deploy/k8s/overlays/prod/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: shop
resources:
  - namespace.yaml
  - ../../base
configMapGenerator:
  - name: order-config
    behavior: merge
    literals:
      - DB_HOST=order-db.prod.example.internal
      - NATS_URL=nats://nats.nats.svc.cluster.local:4222
      - REDIS_ADDR=shop-cache.prod.example.internal:6379
  # payment-config, inventory-config and api-gateway-config merge their own addresses the same way
patches:
  - path: patches/order-resources.yaml
  - target:
      kind: HorizontalPodAutoscaler
      name: order
    patch: |-
      - op: replace
        path: /spec/minReplicas
        value: 3
      - op: replace
        path: /spec/maxReplicas
        value: 20
images:
  - name: registry.example.com/shop/api-gateway
    newTag: 3f2c1a9e4b7d
  - name: registry.example.com/shop/order
    newTag: 3f2c1a9e4b7d
  - name: registry.example.com/shop/payment
    newTag: 3f2c1a9e4b7d
  - name: registry.example.com/shop/inventory
    newTag: 3f2c1a9e4b7d
# deploy/k8s/overlays/prod/patches/order-resources.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: order
spec:
  template:
    spec:
      containers:
        - name: app
          resources:
            requests:
              cpu: 500m
              memory: 512Mi
            limits:
              memory: 1Gi
# deploy/k8s/overlays/prod/namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
  name: shop
  labels:
    pod-security.kubernetes.io/enforce: restricted

The overlays use several of Kustomize's ways to modify resources, each suited to its own situation:

MethodSuited toExample
namespaceMoving a whole environment to a namespaceshop-dev, shop
imagesChanging only image tagsgit SHA
configMapGenerator + mergeOverriding some configuration valuesThe production database host
Strategic merge patch (file)Changing a block of structure, such as resourcesorder-resources.yaml
JSON 6902 patch (op: replace)Precisely changing one field, or all resources of a kindMinimum replicas for every HPA

The production namespace is labelled pod-security.kubernetes.io/enforce: restricted, so Pod Security Admission rejects any Pod that runs as root or asks for extra privileges. Step 2's securityContext was written precisely to pass this level; the migration Job needs the same settings, or it will be rejected in production.

Before applying, always look at the output first:

# Render the full YAML; no cluster connection needed
kubectl kustomize deploy/k8s/overlays/prod > /tmp/shop-prod.yaml

# Diff against the current state on the cluster
kubectl diff -k deploy/k8s/overlays/prod

# Trial run with server-side validation, without actually writing anything
kubectl apply -k deploy/k8s/overlays/prod --dry-run=server
base 4 × Deployment, Service HPA, PDB, migration Job ConfigMap, HTTPRoute NetworkPolicy overlays/dev namespace: shop-dev In-cluster postgres, nats, redis Secrets from secretGenerator HPA 1–2, tag dev overlays/prod namespace: shop Merged ConfigMap → managed databases Secrets synced externally, not in git Larger resources, tag = git SHA

Figure: base holds the resources shared by the four services; dev and prod only describe their own namespace, data components, secret sources, replica counts and image tags.

Step 8: Migration Jobs, the release flow, rolling updates and rollback

Migrations as a Job. The same idea as the previous article's one-off service: the same image with a different command:

# deploy/k8s/base/order/migrate-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
  name: order-migrate
  labels:
    app.kubernetes.io/name: order-migrate
    app.kubernetes.io/component: migrate
spec:
  backoffLimit: 2
  activeDeadlineSeconds: 600
  template:
    metadata:
      labels:
        app.kubernetes.io/name: order-migrate
        app.kubernetes.io/component: migrate
    spec:
      restartPolicy: Never
      securityContext:
        runAsNonRoot: true
        seccompProfile:
          type: RuntimeDefault
      containers:
        - name: migrate
          image: registry.example.com/shop/order:latest
          args: ['migrate', 'up']
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop: ['ALL']
          envFrom:
            - configMapRef:
                name: order-config
          env:
            - name: DB_PASSWORD_FILE
              value: /etc/shop/secrets/db-password
          volumeMounts:
            - name: secrets
              mountPath: /etc/shop/secrets
              readOnly: true
      volumes:
        - name: secrets
          secret:
            secretName: order-secrets

One property of Jobs dictates how the release flow is written: a Job's Pod template cannot be modified after creation. On the next release the image tag changes, and a plain apply fails. So the flow must delete the previous migration Job first and then create the new one. The other requirement is that the migration must finish before the new version's Pods go live, and schema changes must be backward compatible (add the column first, change the code, drop the old column later), because old and new versions run side by side during a rolling update.

The release flow. Chain build, push, retag, migrate, apply and wait into one script, run by CI after merging to main:

#!/usr/bin/env bash
# Outline of the release flow: run from the repository root
set -euo pipefail

SHA="$(git rev-parse --short=12 HEAD)"
REG="registry.example.com/shop"
NS="shop"
OVERLAY="deploy/k8s/overlays/prod"
SERVICES=(api-gateway order payment inventory)

# 1. Build and push the images, tagged with the git SHA (immutable and traceable)
for svc in "${SERVICES[@]}"; do
  docker buildx build \
    --build-arg SERVICE="$svc" \
    -f deploy/docker/Dockerfile \
    -t "$REG/$svc:$SHA" \
    --push .
done

# 2. Set the overlay's image tags to this SHA (requires the standalone kustomize)
(
  cd "$OVERLAY"
  for svc in "${SERVICES[@]}"; do
    kustomize edit set image "$REG/$svc=$REG/$svc:$SHA"
  done
)

# 3. Run database migrations first: delete the previous Job, apply only configuration and migration jobs
kubectl -n "$NS" delete job -l app.kubernetes.io/component=migrate --ignore-not-found --wait
kubectl apply -k "$OVERLAY" -l 'app.kubernetes.io/component in (config,migrate)'
kubectl -n "$NS" wait --for=condition=complete job -l app.kubernetes.io/component=migrate --timeout=10m

# 4. Apply everything and wait for each service's rolling update to finish
kubectl apply -k "$OVERLAY"
for svc in "${SERVICES[@]}"; do
  kubectl -n "$NS" rollout status "deployment/$svc" --timeout=5m
done

A few design points:

  • Tag with the git SHA, not latest. latest doesn't trigger a rolling update (the Pod template is unchanged), and there's no way to tell which version is running on the cluster. The SHA gives a one-to-one mapping between "which commit, which image, which rollout".
  • All four services use the same SHA. In a monorepo this gives the whole system's version a single name; even if a service's code hasn't changed, rebuilding its image takes almost no time thanks to the cache. If you have many services and want to release only those that changed, you can compare changed paths, but be careful with the shared internal/ packages.
  • apply -l applies only configuration and migrations first. The migration Job needs the new ConfigMap (with its hashed name) and Secret, so section 3 uses a label selector to pick the config and migrate resources and apply them first. If the migration fails, kubectl wait times out, the script aborts because of set -e, the Deployments are untouched, and the old version keeps serving.
  • kubectl wait only waits for "success". When a migration fails, it only aborts after the timeout; to fail faster, poll the Job's Failed condition separately in the script.
  • Commit the modified overlay back to the repository, or switch to GitOps: CI only builds the images and opens a commit that "changes the tags to the new SHA", and Argo CD or Flux detects the change in the cluster and applies it. That way the repository is always the single source of truth for the cluster's state, and the commit history tells you who released what and when.
1 git push Merge to main 2 Build images 4 services 3 Push tag = git SHA 4 Set tags kustomize edit 5 Migration Job wait for complete 6 apply -k overlays/prod 7 rollout status Rollout complete Failure: roll back Restore previous SHA The same SHA travels from build to cluster: images, overlay and rollout history all match If the migration fails, the flow stops at step 5 and the old version keeps serving

Figure: The same git SHA is carried from build to cluster; the new version is applied only after the migration succeeds, and if any step fails the flow stops and the old version keeps serving.

Rolling updates and rollback. rollout status waits until the new ReplicaSet is fully ready, or until progressDeadlineSeconds (600 seconds by default) is exceeded and the rollout is marked as failed. When a new version is broken, there are two ways back:

# Show a service's rollout history
kubectl -n shop rollout history deployment/order

# Emergency rollback to the previous version (affects only this Deployment)
kubectl -n shop rollout undo deployment/order

# Roll back to a specific revision
kubectl -n shop rollout undo deployment/order --to-revision=3

rollout undo is fast and suits stopping the bleeding in an emergency; because ConfigMap names carry hashes, the old version's configuration comes back too. But it leaves the cluster's state inconsistent with the repository: the next time someone runs the release flow, the broken version is applied again. So after stopping the bleeding, the proper fix is to revert that commit in the repository (or change the overlay's tags back to the previous SHA) and run the full release flow again. Database migrations are usually not rolled back: backward-compatible schema changes let the old code work on the new schema, which is much safer than writing "down migrations".

Step 9: Debugging when things go wrong

Once the whole system is deployed, the most common problems can be investigated from the outside in with a fixed set of commands:

# Overview: which Pods are unhealthy and how many times they've restarted
kubectl -n shop get pods -o wide
kubectl -n shop get deploy,svc,hpa,pdb

# Why a Pod won't start: look at the Events at the bottom
kubectl -n shop describe pod order-7c9d8f6b5-x2k4q

# Recent events in the namespace, sorted by time
kubectl -n shop get events --sort-by=.lastTimestamp

# Logs: all order Pods, and the logs from before the last crash
kubectl -n shop logs -l app.kubernetes.io/name=order --tail=100 --prefix
kubectl -n shop logs order-7c9d8f6b5-x2k4q --previous

# Did the Service select any Pods: empty endpoints mean a selector or readiness problem
kubectl -n shop get endpointslices -l kubernetes.io/service-name=order

# Bypass the entry point and connect to a service directly from your machine
kubectl -n shop port-forward svc/order 9090:9090

# Distroless has no shell: attach a debug container that shares the process namespace
kubectl -n shop debug -it order-7c9d8f6b5-x2k4q --image=busybox:1.37 --target=app

# When a rolling update is stuck
kubectl -n shop rollout status deployment/order
kubectl -n shop describe deployment order

Common symptoms and where to look:

SymptomCommon causesLook here first
ImagePullBackOffTag doesn't exist, registry authentication failedEvents in describe pod
CrashLoopBackOffMissing configuration, can't reach the database, crashes on startuplogs --previous
CreateContainerConfigErrorThe referenced ConfigMap or Secret doesn't existdescribe pod, get secret
PendingNot enough resources, PVC can't bind, affinity can't be satisfieddescribe pod, get events
Running but takes no trafficReadiness keeps failingProbe messages in describe pod, /readyz
Service unreachableSelector doesn't match Pod labels, wrong port name, NetworkPolicyget endpointslices, NetworkPolicy
OOMKilledMemory limit too low, no GOMEMLIMIT, a genuine leakLast State in describe pod, pprof
Rolling update stuckNew Pods never become ready, not enough resources for extra Podsrollout status, describe deploy

The rough order for debugging is: is the Pod up (get, describe) → what does the program say (logs) → is the network path open (endpointslices, port-forward, NetworkPolicy) → where does the request slow down or fail (tracing from “Observability in Go: Logs, Metrics and Tracing”). The first three steps need only kubectl; only the fourth needs observability tools. port-forward is especially useful: it lets you connect from your machine straight to a service, skipping the Gateway, api-gateway and NetworkPolicy, so in one go you can tell whether the problem is in the service itself or somewhere along the path.

Common pitfalls

  • The Deployment sets replicas and also has an HPA. Every apply resets the replica count to the value in the file, instantly cutting the replicas the HPA just added. Don't set replicas on a Deployment with an HPA.
  • Using the latest tag. With an unchanged Pod template there's no rolling update, and you can't tell which version is on the cluster; rollback becomes guesswork. Always use the git SHA.
  • The liveness probe checks the database. One database blip and every service is restarted together. Liveness checks only the program itself.
  • No CPU requests. The HPA computes utilisation relative to requests; without requests it can't scale, and the scheduler doesn't know where to place the Pod.
  • A PDB with minAvailable and a single replica. Nodes can never be drained and cluster upgrades get stuck. Use maxUnavailable: 1.
  • Assuming that creating a NetworkPolicy makes you safe. If the network plugin doesn't support it, the object exists but has no effect; if you restrict egress and forget to allow DNS, no service can resolve names. Test after deploying.
  • Running gRPC over ClusterIP and expecting even traffic. Long-lived connections make requests stick to a few Pods, and adding replicas doesn't help. Use a headless Service with client-side load balancing, or a service mesh.
  • Modifying the migration Job and applying it directly. A Job's Pod template is immutable; you must delete it and create it again, and it must finish before the new version goes live.
  • Adding shared labels with commonLabels. It rewrites the selector, so later label changes can't update the Deployment. Use labels and keep includeSelectors at its default of false.
  • Putting production passwords in secretGenerator files. Those files end up in CI's working directory or even get committed by mistake. Hand production secrets to an external key management service.
  • Calling it done after rollout undo. The cluster and the repository disagree, and the next release brings the broken version back. After stopping the bleeding, revert in the repository.
  • Forgetting the Pod Security level. The production namespace is restricted; a Job or sidecar container missing its securityContext will be refused, and the error appears in the ReplicaSet's or Job's events, not in the output of apply.

Key takeaways

  • One namespace (or one cluster) per environment: shop for production, shop-dev for development; services call each other by the short name order:9090, whose full name is order.shop.svc.cluster.local.
  • Each service has its own Deployment, Service, HPA, PDB and ConfigMap under deploy/k8s/base/<service>/, consistently using app.kubernetes.io/* labels.
  • A Deployment needs three probes, CPU requests, a memory limit, GOMEMLIMIT, a preStop delay and a non-root securityContext.
  • configMapGenerator's hashed names make configuration changes trigger rolling updates and bring configuration back on rollback; production secrets come from an external key management service.
  • Use managed services for production databases, install third-party components (NATS, Operators, monitoring) with Helm, and manage your own services with Kustomize.
  • There is only one way in: a Gateway API HTTPRoute (or Ingress) pointing to api-gateway; NetworkPolicy denies by default, then allows Gateway → api-gateway → backends rule by rule.
  • Kustomize: base holds shared resources, and overlays describe each environment's differences with namespace, images, ConfigMap merges and patches; check with kubectl kustomize and kubectl diff before applying.
  • Release flow: build images → tag with the git SHA and push → kustomize edit set image → delete the old Job, apply configuration and migrations, wait for completion → apply -k → rollout status.
  • Rollback: rollout undo to stop the bleeding, then revert in the repository; keep schema changes backward compatible rather than relying on down migrations.
  • Debug from the outside in: get/describe → logs (including --previous) → endpointslices/port-forward/NetworkPolicy → distributed tracing.

Next

At this point the whole order system runs in the cluster with multiple replicas, autoscales, and releases and rolls back safely. But not every system needs all this from day one: the next article, “From Monolith to Microservices: Evolving the Architecture at 100K, 1M and 10M Scale”, takes a step back and looks at how the same shop system should grow with scale, from a single monolith, stage by stage, into what it is now — and when to split, and when not to split yet.