Columns › 6. Deploying the system › Part 22
Deploying a Microservice System on Kubernetes
在 Kubernetes 部署整套微服務
Move the api-gateway, order, payment and inventory services onto Kubernetes with a Deployment and Service per service, Gateway API routing, NetworkPolicy, HPA and PDB, Kustomize overlays for dev and prod, and a release flow covering image tags, migrations, rollouts, rollback and debugging.
In this article
- Step 1: Plan namespaces, labels and the directory structure
- Step 2: One Deployment and Service per service
- Step 3: Each service's own ConfigMap and Secret
- Step 4: Stateful dependencies — databases and message queues
- Step 5: The external entry point and NetworkPolicy
- Step 6: Probes, resources, HPA and PDB
- Step 7: Combine base with dev/prod overlays using Kustomize
- Step 8: Migration Jobs, the release flow, rolling updates and rollback
- Step 9: Debugging when things go wrong
- Common pitfalls
- Key takeaways
- Next
In the previous article, “Deploying a Whole Microservice System with Docker Compose”, we used a single compose.yaml to run four services together on one host. This article moves the same system into a Kubernetes cluster and adds what Compose cannot give you: multiple hosts, zero-downtime updates, autoscaling and fine-grained access control.
“Deploying One Go Service with Docker and Kubernetes” already showed how to put one service on a cluster: one Deployment, one Service, three probes. Going from one to four, the hard part is not "copy the YAML four times" but the problems that only show up at the system level:
- With four services and two or more environments, how do you organise the configuration so nothing is duplicated everywhere, yet you can still see exactly how each environment differs?
- How do services find each other? How does external traffic get in, and how do you guarantee it can only come in through the entry point?
- Should stateful components such as databases and message queues live in the cluster or be handed to a managed service?
- When should database migrations run during a release? How do you go back when a new version is broken?
- When something goes wrong, where do you start looking?
This article continues with the order system used throughout the series. Starting from the directory structure, it builds step by step a configuration you can actually apply to a cluster. All API versions in the examples are the current stable ones: apps/v1, autoscaling/v2, policy/v1, networking.k8s.io/v1, and Gateway API's gateway.networking.k8s.io/v1. Configuration is managed with Kustomize, which is built into kubectl.
Step 1: Plan namespaces, labels and the directory structure
First decide three things that affect every file that follows.
One: how to separate environments. There are two common approaches: one cluster per environment, or one namespace per environment within the same cluster. Putting production and development in different clusters gives the strongest isolation; a mistake or resource exhaustion in one environment cannot spill over into the other. Sharing one cluster between development and test, each in its own namespace, saves a lot of cost. In this article's examples the production namespace is shop and the development one is shop-dev. If your development environment is in a separate cluster, both can be called shop and nothing in the configuration below changes — because services address each other by short names, as explained in Step 2.
Two: label conventions. Kubernetes ties resources together with labels: a Service selects Pods by label, a NetworkPolicy uses labels to decide who may connect to whom, and a PDB uses labels to know which Pods to protect. Once labels get messy, these mechanisms silently stop working. The examples consistently use the labels Kubernetes recommends:
| Label | Purpose | Example values |
|---|---|---|
app.kubernetes.io/name | Which service; selectors use this | order, api-gateway |
app.kubernetes.io/part-of | Which system it belongs to | shop |
app.kubernetes.io/component | The role of the resource | migrate, config |
Three: the directory structure. Following the series' repository layout, the Kubernetes configuration lives in deploy/k8s/, split into base (shared by all environments) and overlays (the differences for each environment):
shop/ # module path example.com/shop
├── cmd/{api-gateway,order,payment,inventory}/main.go
├── internal/ …
├── api/proto/ …
└── deploy/
├── docker/Dockerfile # shared by all four services (previous article)
├── compose/ … # the Compose configuration from the previous article
└── k8s/
├── base/
│ ├── kustomization.yaml # lists the four services and shared resources
│ ├── httproute.yaml # external route to api-gateway
│ ├── networkpolicy.yaml # who may connect to whom
│ ├── api-gateway/
│ │ ├── kustomization.yaml
│ │ ├── config.env
│ │ ├── deployment.yaml
│ │ ├── service.yaml
│ │ ├── hpa.yaml
│ │ └── pdb.yaml
│ ├── order/ # same as above, plus migrate-job.yaml
│ ├── payment/ # same as order
│ └── inventory/ # same as order
└── overlays/
├── dev/
│ ├── kustomization.yaml # namespace: shop-dev, secrets, fewer replicas
│ ├── namespace.yaml
│ ├── data/ # postgres, nats, redis for development
│ └── secrets/ # not in version control
└── prod/
├── kustomization.yaml # namespace: shop, image tags, more resources
├── namespace.yaml
└── patches/One subdirectory per service, each with its own kustomization.yaml, means "everything about order" lives in base/order/: adding a service is copying a directory, renaming it and adding one line to the list one level up.
Figure: External traffic reaches api-gateway through the Gateway and HTTPRoute, and backend services call each other via cluster DNS; in production the database and cache are outside the cluster, and NATS lives in its own namespace.
Step 2: One Deployment and Service per service
Write order in full first; the other three services follow the same shape. The Deployment describes "what to run and how to run it":
# deploy/k8s/base/order/deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: order
labels:
app.kubernetes.io/name: order
spec:
# No replicas: the replica count belongs to the HPA, otherwise every apply resets it
revisionHistoryLimit: 5
selector:
matchLabels:
app.kubernetes.io/name: order
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 25%
maxUnavailable: 0
template:
metadata:
labels:
app.kubernetes.io/name: order
spec:
terminationGracePeriodSeconds: 30
securityContext:
runAsNonRoot: true
seccompProfile:
type: RuntimeDefault
topologySpreadConstraints:
- maxSkew: 1
topologyKey: kubernetes.io/hostname
whenUnsatisfiable: ScheduleAnyway
labelSelector:
matchLabels:
app.kubernetes.io/name: order
containers:
- name: app
image: registry.example.com/shop/order:latest
args: ['serve']
ports:
- name: http
containerPort: 8080
- name: grpc
containerPort: 9090
envFrom:
- configMapRef:
name: order-config
env:
- name: DB_PASSWORD_FILE
value: /etc/shop/secrets/db-password
- name: GOMEMLIMIT
valueFrom:
resourceFieldRef:
resource: limits.memory
volumeMounts:
- name: secrets
mountPath: /etc/shop/secrets
readOnly: true
resources:
requests:
cpu: 100m
memory: 128Mi
limits:
memory: 256Mi
startupProbe:
httpGet:
path: /livez
port: http
periodSeconds: 2
failureThreshold: 30
readinessProbe:
httpGet:
path: /readyz
port: http
periodSeconds: 5
failureThreshold: 3
livenessProbe:
httpGet:
path: /livez
port: http
periodSeconds: 10
failureThreshold: 3
lifecycle:
preStop:
sleep:
seconds: 5
securityContext:
allowPrivilegeEscalation: false
readOnlyRootFilesystem: true
capabilities:
drop: ['ALL']
volumes:
- name: secrets
secret:
secretName: order-secretsThis file is longer than the one in the single-service article, and every extra part is there for a reason:
- The tag on
imagedoesn't matter.latestin base is only a placeholder; the real tag is rewritten to a git SHA by the overlay'simagesfield (Steps 7 and 8). args: ["serve"]matches the subcommand design from the previous article: the same image runs the service withserveand runs migrations withmigrate up.envFromreads the ConfigMap, and the password is mounted as a file, paired with the previous article'senvOrFile: the program reads the file thatDB_PASSWORD_FILEpoints to. Compose's/run/secrets/…and the/etc/shop/secrets/…here differ only in path; the code doesn't change at all.GOMEMLIMITtakes its value from the memory limit. By default the Go runtime doesn't know the container's memory limit; once it is set, the garbage collector collects more aggressively as it approaches the limit, reducing the chance of being OOMKilled. The memory curves measured in “Performance Tuning in Go: pprof and Benchmarks” are exactly what you use to decide this limit.topologySpreadConstraintsspreads a service's Pods across different nodes as far as possible, so one failed node doesn't take every replica with it.- The
preStopsleepmakes the Pod wait a few seconds before it receives the termination signal, giving Service endpoints and load balancers time to remove it from their lists, so no new requests arrive while it is shutting down. Distroless images have nosleepcommand, so this uses Kubernetes' built-in sleep action (supported only in newer Kubernetes versions; on older clusters the program can delay by itself after receiving the signal). After that, the program handles SIGTERM as in “Configuration and Graceful Shutdown in Go” and finishes up withinterminationGracePeriodSeconds. securityContextruns as non-root, makes the root filesystem read-only and drops all Linux capabilities. A static Go binary never needed these privileges, so tightening them costs nothing.
The Service gives this set of Pods a stable name and address:
# deploy/k8s/base/order/service.yaml
apiVersion: v1
kind: Service
metadata:
name: order
labels:
app.kubernetes.io/name: order
spec:
selector:
app.kubernetes.io/name: order
ports:
- name: http
port: 8080
targetPort: http
- name: grpc
port: 9090
targetPort: grpcHow services find each other. Cluster DNS creates a <service>.<namespace>.svc.cluster.local record for every Service: in the shop namespace, order's full name is order.shop.svc.cluster.local. Pods in the same namespace can write just order, and the DNS search domains fill in the rest. So api-gateway's configuration says ORDER_ADDR=order:9090 — exactly as in the previous article's Compose file, and unchanged whether it is deployed to shop or shop-dev. Only across namespaces do you need a longer name; for example, production NATS lives in the nats namespace, so you write nats.nats.svc.cluster.local.
Tip: gRPC uses long-lived HTTP/2 connections. An ordinary ClusterIP Service load-balances when a connection is established; once the connection is up, every request goes to the same Pod, and adding replicas doesn't spread the traffic by itself. Both solutions from “Service Discovery and Load Balancing for Go Microservices” apply here: create an extra headless Service with
clusterIP: Noneso the gRPC client gets every Pod address viadns:///order-headless:9090and spreads requests withround_robin, or let a service mesh load-balance per request.
The Deployments and Services for api-gateway, payment and inventory have the same structure, with only names and labels changed; api-gateway doesn't connect to a database, so it has no secret mount and only exposes port 8080.
Step 3: Each service's own ConfigMap and Secret
Configuration and secrets need to achieve two things: each service gets only its own, and Pods are replaced automatically when configuration changes. Kustomize's generators do both.
In each service's subdirectory, use configMapGenerator to generate a ConfigMap from an env file:
# deploy/k8s/base/order/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
- deployment.yaml
- service.yaml
- hpa.yaml
- pdb.yaml
- migrate-job.yaml
configMapGenerator:
- name: order-config
envs:
- config.env
options:
labels:
app.kubernetes.io/component: config# deploy/k8s/base/order/config.env
LOG_LEVEL=info
LOG_FORMAT=json
DB_HOST=postgres
DB_NAME=order
DB_USER=order
PAYMENT_ADDR=payment:9090
INVENTORY_ADDR=inventory:9090
NATS_URL=nats://nats:4222
REDIS_ADDR=redis:6379
OTEL_EXPORTER_OTLP_ENDPOINT=http://otel-collector.monitoring:4317There is one key detail in what the generator does: the generated ConfigMap's name carries a hash of its contents, for example order-config-t59gh52tg6, and the order-config written in the Deployment is automatically rewritten to that hashed name. This brings two benefits:
- Change the configuration and Pods roll automatically. When an ordinary ConfigMap is modified, running Pods don't re-read their environment variables; with the generator, a change in content changes the name, which changes the Deployment's Pod template, so Kubernetes performs a rolling update.
- Rollback brings the configuration back too. The old ReplicaSet references the ConfigMap by its old name, so with
kubectl rollout undocode and configuration move together; you never get "old code, new configuration".
options.labels adds app.kubernetes.io/component: config to the ConfigMap. This is for the release flow in Step 8: apply only configuration and migration jobs first, then everything else.
Secrets come from different sources in different environments. In development, secretGenerator can generate them from local files (the files are not in version control):
# deploy/k8s/overlays/dev/kustomization.yaml (excerpt)
secretGenerator:
- name: order-secrets
files:
- db-password=secrets/order-db-password.txt
options:
labels:
app.kubernetes.io/component: configIn production, passwords should never appear in any form in the repository or on the CI filesystem. Common approaches are the External Secrets Operator syncing a Secret named order-secrets from a cloud key management service, Sealed Secrets putting encrypted ciphertext in the repository, or the Secrets Store CSI Driver mounting keys straight into the Pod. Whichever you use, to the Deployment it is just "there is a Secret called order-secrets"; base doesn't need to know where it comes from.
| Item | Where it lives | Example |
|---|---|---|
| Configuration same in every env | config.env in base | Log format, short service-to-service names |
| Configuration that differs per env | ConfigMap merged in the overlay | Database host, cache address |
| Development secrets | The overlay's secretGenerator | Test passwords generated locally |
| Production secrets | A key management service outside | Synced by tools such as External Secrets |
Tip: Kubernetes Secrets are only base64-encoded by default, not encrypted. Make sure the cluster has encryption at rest enabled for etcd, and use RBAC to restrict who can read Secrets; anyone who can create Pods in a namespace can in practice read every Secret in that namespace.
Step 4: Stateful dependencies — databases and message queues
The microservices themselves are stateless; they can be killed, rescheduled and scaled at any time. Databases and message queues cannot. Where they live is one of the most important decisions when deploying a whole system:
| Option | Advantages | Costs |
|---|---|---|
| Managed cloud database | Someone else handles backups, patching, HA and capacity | Price, cloud lock-in |
| In-cluster Operator | Declarative management, still with HA and automatic backups | You must understand the Operator; storage must be reliable |
| Hand-written StatefulSet | Full control, fastest to get working | Backups, upgrades and failover are all on you |
A pragmatic combination: use a managed service for the production database, and a simple in-cluster StatefulSet for development. For the message queue it depends on the team's skills: components such as NATS and RabbitMQ have officially maintained Helm charts or Operators; install them in their own namespace (for example nats), and the four services connect via nats.nats.svc.cluster.local.
The development PostgreSQL can reuse the previous article's init script directly:
# deploy/k8s/overlays/dev/data/postgres.yaml
apiVersion: v1
kind: Service
metadata:
name: postgres
spec:
clusterIP: None
selector:
app.kubernetes.io/name: postgres
ports:
- name: postgres
port: 5432
---
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: postgres
spec:
serviceName: postgres
replicas: 1
selector:
matchLabels:
app.kubernetes.io/name: postgres
template:
metadata:
labels:
app.kubernetes.io/name: postgres
spec:
containers:
- name: postgres
image: postgres:17-alpine
env:
- name: POSTGRES_USER
value: postgres
- name: POSTGRES_PASSWORD_FILE
value: /run/secrets/postgres_password
- name: PGDATA
value: /var/lib/postgresql/data/pgdata
ports:
- name: postgres
containerPort: 5432
readinessProbe:
exec:
command: ['pg_isready', '-U', 'postgres', '-d', 'postgres']
periodSeconds: 5
resources:
requests:
cpu: 100m
memory: 256Mi
limits:
memory: 512Mi
volumeMounts:
- name: data
mountPath: /var/lib/postgresql/data
- name: secrets
mountPath: /run/secrets
readOnly: true
- name: initdb
mountPath: /docker-entrypoint-initdb.d
readOnly: true
volumes:
- name: secrets
secret:
secretName: postgres-secrets
- name: initdb
configMap:
name: postgres-initdb
volumeClaimTemplates:
- metadata:
name: data
spec:
accessModes: ['ReadWriteOnce']
resources:
requests:
storage: 5GiThe difference between a StatefulSet and a Deployment is that it gives each Pod a stable name (postgres-0) and its own persistent volume: when the Pod is rescheduled, it mounts the same data again. With a headless Service (clusterIP: None), services still connect with postgres:5432. PGDATA points to a subdirectory because many storage classes put lost+found at the root of the mount point, and PostgreSQL refuses a non-empty directory during initialisation.
The secrets are mounted at /run/secrets with underscore-style file names such as order_db_password, so the 10-create-databases.sh used with Compose in the previous article doesn't need a single character changed; just put it in a ConfigMap via configMapGenerator and mount it in the init directory. In development, NATS and Redis are each just a Deployment plus a Service, structured much like the above, so they are not repeated here.
In production, the overlay only has to replace the addresses in the ConfigMap with the managed services' host names (Step 7); neither the four services' code nor base changes. This is the payoff for putting "addresses" in configuration rather than hard-coding them in the program.
Tip: When should you use Helm? When you are installing components someone else maintains (NATS, a PostgreSQL Operator, a monitoring stack, a Gateway controller), a Helm chart is the easiest way, because it packages years of accumulated configuration options. Your own four services are simple in structure and differ only in a few fields; with Kustomize you see the final YAML directly, which makes debugging more straightforward. You can also use both: install third-party components with Helm and manage your own services with Kustomize. Only when you need to package the whole system for other teams or customers to install, with lots of parameterisation, is it time to make your own services a Helm chart too.
Step 5: The external entry point and NetworkPolicy
In Compose, "only the gateway is exposed" relied on publishing a single port and putting the backends on an internal network. In Kubernetes this splits into two halves: letting external traffic reach api-gateway, and making every other path a dead end.
External routing. The traditional approach is Ingress (networking.k8s.io/v1); the new standard is Gateway API. Gateway API separates responsibilities: platform administrators own the Gateway (which ports to listen on, which certificate to use), and application teams only write HTTPRoutes (which host name and which paths go to which Service). The Kubernetes community announced the retirement of the ingress-nginx project at the end of 2025, so starting new systems directly on Gateway API is the safer choice:
# deploy/k8s/base/httproute.yaml
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
name: shop
spec:
parentRefs:
- name: public
namespace: gateway-system
hostnames:
- shop.example.com
rules:
- matches:
- path:
type: PathPrefix
value: /
backendRefs:
- name: api-gateway
port: 8080This assumes the platform has already created a Gateway named public in the gateway-system namespace and allows routes from other namespaces to attach to it. Note that the route points only to the api-gateway Service: routing, authentication and rate limiting all happen in the code from “Building an API Gateway in Go: Routing, Authentication and Rate Limiting”; the cluster's entry point only handles TLS and hands the traffic in.
If your cluster uses an Ingress controller, the equivalent is:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: shop
spec:
ingressClassName: example-ingress-class # replace with the cluster's actual IngressClass
tls:
- hosts:
- shop.example.com
secretName: shop-tls
rules:
- host: shop.example.com
http:
paths:
- path: /
pathType: Prefix
backend:
service:
name: api-gateway
port:
number: 8080NetworkPolicy: make every other path a dead end. By default Kubernetes lets "every Pod connect to every Pod". Even though order's Service isn't exposed externally, any Pod in the cluster (including another team's, or a compromised one) can hit order:9090 directly and bypass api-gateway's authentication. NetworkPolicy fixes this by "deny everything first, then allow rule by rule":
# deploy/k8s/base/networkpolicy.yaml
# 1. Default deny: no Pod in the namespace accepts any incoming connection
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: default-deny-ingress
spec:
podSelector: {}
policyTypes: ['Ingress']
---
# 2. api-gateway only accepts traffic from the Gateway controller's namespace
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: api-gateway-from-edge
spec:
podSelector:
matchLabels:
app.kubernetes.io/name: api-gateway
policyTypes: ['Ingress']
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: gateway-system
ports:
- port: 8080
---
# 3. Backend services only accept gRPC calls from api-gateway and order (the Saga coordinator)
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: backend-from-callers
spec:
podSelector:
matchExpressions:
- key: app.kubernetes.io/name
operator: In
values: ['order', 'payment', 'inventory']
policyTypes: ['Ingress']
ingress:
- from:
- podSelector:
matchExpressions:
- key: app.kubernetes.io/name
operator: In
values: ['api-gateway', 'order']
ports:
- port: 9090
---
# 4. The monitoring system may scrape /metrics from every service
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
name: metrics-from-monitoring
spec:
podSelector:
matchLabels:
app.kubernetes.io/part-of: shop
policyTypes: ['Ingress']
ingress:
- from:
- namespaceSelector:
matchLabels:
kubernetes.io/metadata.name: monitoring
ports:
- port: 8080NetworkPolicies are additive: if any rule allows the traffic, it gets through; if no rule allows it, rule 1 blocks it. So when reading them, ask "which paths are open", not "which paths are closed". kubernetes.io/metadata.name is a label Kubernetes adds to every namespace automatically, and selecting namespaces with it is more reliable than adding your own label.
Rule 3 is fairly broad: both api-gateway and order can call all three backends. To be stricter, split it into one rule per service, for example inventory only accepts calls from order — the finer the rules, the more they turn "which service depends on which" into executable documentation, at the cost of remembering to update them whenever a new call relationship is added.
Figure: Only the Gateway can reach api-gateway, only api-gateway and order can reach the backends' gRPC port, and monitoring can only scrape metrics on 8080; every other connection is blocked by the default rule.
The development environment puts PostgreSQL, NATS and Redis in the same namespace, so they are also caught by the default deny; the dev overlay therefore additionally allows "Pods in the same namespace may connect to the data components". In production the database is outside the cluster; if you want to restrict outbound traffic, you need Egress rules as well, and must remember to allow DNS (port 53 in kube-system, UDP and TCP), or no service will be able to resolve any name.
Tip: NetworkPolicy only takes effect if the cluster's network plugin (CNI) supports it. Some network plugins accept NetworkPolicy objects but don't enforce them at all. Always test after deploying: from a temporary Pod, try to connect to
order:9090— it should fail.
Step 6: Probes, resources, HPA and PDB
For services to "hold up" in the cluster, you need these mechanisms, each doing its own job:
| Mechanism | The question it answers | What happens if it's misconfigured |
|---|---|---|
| startupProbe | Has the program finished starting? | Too short: slow-starting services restart endlessly |
| readinessProbe | Can it take traffic right now? | Checks too many dependencies: one failure takes all offline |
| livenessProbe | Is the program stuck and in need of a restart? | Checks external dependencies: a DB blip restarts every Pod |
| resources | How much does it need, and how much can it use? | No requests: scheduling goes wrong, HPA can't compute |
| HPA | How many replicas when traffic grows? | Fights with the Deployment's replicas |
| PDB | How many can be missing at once during upkeep? | Too strict: nodes can never be drained |
The probes are already in Step 2's Deployment. What matters is their meaning: /livez only answers "the program itself is alive" and does not check the database; /readyz answers "I can handle requests" and may check whether its own connection pool is ready. When the database drops out briefly, a failing readiness probe takes the Pod out of load balancing temporarily, and it comes back once the connection recovers; if liveness also checked the database, Kubernetes would restart every Pod at once, turning a small problem into a full outage.
For resources, setting only CPU requests and no CPU limits is a common choice for Go services: a CPU limit causes throttling, which makes latency suddenly worse under high load. Memory, on the other hand, must have a limit, with the GOMEMLIMIT from earlier telling the Go runtime where that line is.
The HPA adjusts the replica count based on CPU utilisation:
# deploy/k8s/base/order/hpa.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
name: order
spec:
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: order
minReplicas: 2
maxReplicas: 10
metrics:
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 70
behavior:
scaleDown:
stabilizationWindowSeconds: 300averageUtilization is calculated relative to requests, so for Pods without CPU requests the HPA simply cannot work. scaleDown.stabilizationWindowSeconds makes it observe for a while before scaling down, so the replica count doesn't bounce up and down when traffic comes in waves. Each service's HPA is configured independently: api-gateway handles every request while order coordinates Sagas; their load shapes differ, so their replica ranges should be tuned separately. If CPU isn't a good signal (for example inventory mostly waits on the database), you can switch to custom metrics such as requests per second or queue length, which requires installing a metrics adapter.
A PDB protects against "voluntary disruptions" — evictions during node maintenance and cluster upgrades:
# deploy/k8s/base/order/pdb.yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: order
spec:
maxUnavailable: 1
selector:
matchLabels:
app.kubernetes.io/name: ordermaxUnavailable: 1 is used instead of minAvailable because it makes sense at any replica count: with ten replicas only one is evicted at a time, and in development with a single replica eviction is still allowed, so node maintenance never gets stuck. Had it been minAvailable: 1, an environment with only one replica would leave kubectl drain waiting forever.
The pace of a rolling update is set by the Deployment's strategy. maxUnavailable: 0 with maxSurge: 25% means: start the extra new Pods first, and only shut down an old one once a new one passes readiness. Throughout the process the number of serving replicas never drops below the original count, at the cost of needing a little more capacity during the update.
Step 7: Combine base with dev/prod overlays using Kustomize
Everything written so far is in base; now use overlays to assemble two environments. The base entry point lists the four services and the shared resources:
# deploy/k8s/base/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
labels:
- pairs:
app.kubernetes.io/part-of: shop
resources:
- api-gateway
- order
- payment
- inventory
- httproute.yaml
- networkpolicy.yamllabels adds app.kubernetes.io/part-of: shop to every resource. It is used here instead of the old commonLabels because the latter also rewrites the Deployment's selector, and a selector cannot be changed once created — if you later want to adjust labels, the old approach makes the apply fail outright.
The development overlay:
# deploy/k8s/overlays/dev/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: shop-dev
resources:
- namespace.yaml
- ../../base
- data/postgres.yaml
- data/nats.yaml
- data/redis.yaml
- data/networkpolicy.yaml
configMapGenerator:
- name: postgres-initdb
files:
- data/10-create-databases.sh
secretGenerator:
- name: order-secrets
files:
- db-password=secrets/order-db-password.txt
options:
labels:
app.kubernetes.io/component: config
# payment-secrets and inventory-secrets: same as above
- name: postgres-secrets
files:
- postgres_password=secrets/postgres-password.txt
- order_db_password=secrets/order-db-password.txt
- payment_db_password=secrets/payment-db-password.txt
- inventory_db_password=secrets/inventory-db-password.txt
patches:
- target:
kind: HorizontalPodAutoscaler
patch: |-
- op: replace
path: /spec/minReplicas
value: 1
- op: replace
path: /spec/maxReplicas
value: 2
- target:
kind: HTTPRoute
name: shop
patch: |-
- op: replace
path: /spec/hostnames/0
value: dev.shop.example.com
images:
- name: registry.example.com/shop/api-gateway
newTag: dev
- name: registry.example.com/shop/order
newTag: dev
- name: registry.example.com/shop/payment
newTag: dev
- name: registry.example.com/shop/inventory
newTag: devThe production overlay:
# deploy/k8s/overlays/prod/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: shop
resources:
- namespace.yaml
- ../../base
configMapGenerator:
- name: order-config
behavior: merge
literals:
- DB_HOST=order-db.prod.example.internal
- NATS_URL=nats://nats.nats.svc.cluster.local:4222
- REDIS_ADDR=shop-cache.prod.example.internal:6379
# payment-config, inventory-config and api-gateway-config merge their own addresses the same way
patches:
- path: patches/order-resources.yaml
- target:
kind: HorizontalPodAutoscaler
name: order
patch: |-
- op: replace
path: /spec/minReplicas
value: 3
- op: replace
path: /spec/maxReplicas
value: 20
images:
- name: registry.example.com/shop/api-gateway
newTag: 3f2c1a9e4b7d
- name: registry.example.com/shop/order
newTag: 3f2c1a9e4b7d
- name: registry.example.com/shop/payment
newTag: 3f2c1a9e4b7d
- name: registry.example.com/shop/inventory
newTag: 3f2c1a9e4b7d# deploy/k8s/overlays/prod/patches/order-resources.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
name: order
spec:
template:
spec:
containers:
- name: app
resources:
requests:
cpu: 500m
memory: 512Mi
limits:
memory: 1Gi# deploy/k8s/overlays/prod/namespace.yaml
apiVersion: v1
kind: Namespace
metadata:
name: shop
labels:
pod-security.kubernetes.io/enforce: restrictedThe overlays use several of Kustomize's ways to modify resources, each suited to its own situation:
| Method | Suited to | Example |
|---|---|---|
namespace | Moving a whole environment to a namespace | shop-dev, shop |
images | Changing only image tags | git SHA |
configMapGenerator + merge | Overriding some configuration values | The production database host |
| Strategic merge patch (file) | Changing a block of structure, such as resources | order-resources.yaml |
JSON 6902 patch (op: replace) | Precisely changing one field, or all resources of a kind | Minimum replicas for every HPA |
The production namespace is labelled pod-security.kubernetes.io/enforce: restricted, so Pod Security Admission rejects any Pod that runs as root or asks for extra privileges. Step 2's securityContext was written precisely to pass this level; the migration Job needs the same settings, or it will be rejected in production.
Before applying, always look at the output first:
# Render the full YAML; no cluster connection needed
kubectl kustomize deploy/k8s/overlays/prod > /tmp/shop-prod.yaml
# Diff against the current state on the cluster
kubectl diff -k deploy/k8s/overlays/prod
# Trial run with server-side validation, without actually writing anything
kubectl apply -k deploy/k8s/overlays/prod --dry-run=serverFigure: base holds the resources shared by the four services; dev and prod only describe their own namespace, data components, secret sources, replica counts and image tags.
Step 8: Migration Jobs, the release flow, rolling updates and rollback
Migrations as a Job. The same idea as the previous article's one-off service: the same image with a different command:
# deploy/k8s/base/order/migrate-job.yaml
apiVersion: batch/v1
kind: Job
metadata:
name: order-migrate
labels:
app.kubernetes.io/name: order-migrate
app.kubernetes.io/component: migrate
spec:
backoffLimit: 2
activeDeadlineSeconds: 600
template:
metadata:
labels:
app.kubernetes.io/name: order-migrate
app.kubernetes.io/component: migrate
spec:
restartPolicy: Never
securityContext:
runAsNonRoot: true
seccompProfile:
type: RuntimeDefault
containers:
- name: migrate
image: registry.example.com/shop/order:latest
args: ['migrate', 'up']
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop: ['ALL']
envFrom:
- configMapRef:
name: order-config
env:
- name: DB_PASSWORD_FILE
value: /etc/shop/secrets/db-password
volumeMounts:
- name: secrets
mountPath: /etc/shop/secrets
readOnly: true
volumes:
- name: secrets
secret:
secretName: order-secretsOne property of Jobs dictates how the release flow is written: a Job's Pod template cannot be modified after creation. On the next release the image tag changes, and a plain apply fails. So the flow must delete the previous migration Job first and then create the new one. The other requirement is that the migration must finish before the new version's Pods go live, and schema changes must be backward compatible (add the column first, change the code, drop the old column later), because old and new versions run side by side during a rolling update.
The release flow. Chain build, push, retag, migrate, apply and wait into one script, run by CI after merging to main:
#!/usr/bin/env bash
# Outline of the release flow: run from the repository root
set -euo pipefail
SHA="$(git rev-parse --short=12 HEAD)"
REG="registry.example.com/shop"
NS="shop"
OVERLAY="deploy/k8s/overlays/prod"
SERVICES=(api-gateway order payment inventory)
# 1. Build and push the images, tagged with the git SHA (immutable and traceable)
for svc in "${SERVICES[@]}"; do
docker buildx build \
--build-arg SERVICE="$svc" \
-f deploy/docker/Dockerfile \
-t "$REG/$svc:$SHA" \
--push .
done
# 2. Set the overlay's image tags to this SHA (requires the standalone kustomize)
(
cd "$OVERLAY"
for svc in "${SERVICES[@]}"; do
kustomize edit set image "$REG/$svc=$REG/$svc:$SHA"
done
)
# 3. Run database migrations first: delete the previous Job, apply only configuration and migration jobs
kubectl -n "$NS" delete job -l app.kubernetes.io/component=migrate --ignore-not-found --wait
kubectl apply -k "$OVERLAY" -l 'app.kubernetes.io/component in (config,migrate)'
kubectl -n "$NS" wait --for=condition=complete job -l app.kubernetes.io/component=migrate --timeout=10m
# 4. Apply everything and wait for each service's rolling update to finish
kubectl apply -k "$OVERLAY"
for svc in "${SERVICES[@]}"; do
kubectl -n "$NS" rollout status "deployment/$svc" --timeout=5m
doneA few design points:
- Tag with the git SHA, not
latest.latestdoesn't trigger a rolling update (the Pod template is unchanged), and there's no way to tell which version is running on the cluster. The SHA gives a one-to-one mapping between "which commit, which image, which rollout". - All four services use the same SHA. In a monorepo this gives the whole system's version a single name; even if a service's code hasn't changed, rebuilding its image takes almost no time thanks to the cache. If you have many services and want to release only those that changed, you can compare changed paths, but be careful with the shared
internal/packages. apply -lapplies only configuration and migrations first. The migration Job needs the new ConfigMap (with its hashed name) and Secret, so section 3 uses a label selector to pick theconfigandmigrateresources and apply them first. If the migration fails,kubectl waittimes out, the script aborts because ofset -e, the Deployments are untouched, and the old version keeps serving.kubectl waitonly waits for "success". When a migration fails, it only aborts after the timeout; to fail faster, poll the Job'sFailedcondition separately in the script.- Commit the modified overlay back to the repository, or switch to GitOps: CI only builds the images and opens a commit that "changes the tags to the new SHA", and Argo CD or Flux detects the change in the cluster and applies it. That way the repository is always the single source of truth for the cluster's state, and the commit history tells you who released what and when.
Figure: The same git SHA is carried from build to cluster; the new version is applied only after the migration succeeds, and if any step fails the flow stops and the old version keeps serving.
Rolling updates and rollback. rollout status waits until the new ReplicaSet is fully ready, or until progressDeadlineSeconds (600 seconds by default) is exceeded and the rollout is marked as failed. When a new version is broken, there are two ways back:
# Show a service's rollout history
kubectl -n shop rollout history deployment/order
# Emergency rollback to the previous version (affects only this Deployment)
kubectl -n shop rollout undo deployment/order
# Roll back to a specific revision
kubectl -n shop rollout undo deployment/order --to-revision=3rollout undo is fast and suits stopping the bleeding in an emergency; because ConfigMap names carry hashes, the old version's configuration comes back too. But it leaves the cluster's state inconsistent with the repository: the next time someone runs the release flow, the broken version is applied again. So after stopping the bleeding, the proper fix is to revert that commit in the repository (or change the overlay's tags back to the previous SHA) and run the full release flow again. Database migrations are usually not rolled back: backward-compatible schema changes let the old code work on the new schema, which is much safer than writing "down migrations".
Step 9: Debugging when things go wrong
Once the whole system is deployed, the most common problems can be investigated from the outside in with a fixed set of commands:
# Overview: which Pods are unhealthy and how many times they've restarted
kubectl -n shop get pods -o wide
kubectl -n shop get deploy,svc,hpa,pdb
# Why a Pod won't start: look at the Events at the bottom
kubectl -n shop describe pod order-7c9d8f6b5-x2k4q
# Recent events in the namespace, sorted by time
kubectl -n shop get events --sort-by=.lastTimestamp
# Logs: all order Pods, and the logs from before the last crash
kubectl -n shop logs -l app.kubernetes.io/name=order --tail=100 --prefix
kubectl -n shop logs order-7c9d8f6b5-x2k4q --previous
# Did the Service select any Pods: empty endpoints mean a selector or readiness problem
kubectl -n shop get endpointslices -l kubernetes.io/service-name=order
# Bypass the entry point and connect to a service directly from your machine
kubectl -n shop port-forward svc/order 9090:9090
# Distroless has no shell: attach a debug container that shares the process namespace
kubectl -n shop debug -it order-7c9d8f6b5-x2k4q --image=busybox:1.37 --target=app
# When a rolling update is stuck
kubectl -n shop rollout status deployment/order
kubectl -n shop describe deployment orderCommon symptoms and where to look:
| Symptom | Common causes | Look here first |
|---|---|---|
ImagePullBackOff | Tag doesn't exist, registry authentication failed | Events in describe pod |
CrashLoopBackOff | Missing configuration, can't reach the database, crashes on startup | logs --previous |
CreateContainerConfigError | The referenced ConfigMap or Secret doesn't exist | describe pod, get secret |
Pending | Not enough resources, PVC can't bind, affinity can't be satisfied | describe pod, get events |
| Running but takes no traffic | Readiness keeps failing | Probe messages in describe pod, /readyz |
| Service unreachable | Selector doesn't match Pod labels, wrong port name, NetworkPolicy | get endpointslices, NetworkPolicy |
OOMKilled | Memory limit too low, no GOMEMLIMIT, a genuine leak | Last State in describe pod, pprof |
| Rolling update stuck | New Pods never become ready, not enough resources for extra Pods | rollout status, describe deploy |
The rough order for debugging is: is the Pod up (get, describe) → what does the program say (logs) → is the network path open (endpointslices, port-forward, NetworkPolicy) → where does the request slow down or fail (tracing from “Observability in Go: Logs, Metrics and Tracing”). The first three steps need only kubectl; only the fourth needs observability tools. port-forward is especially useful: it lets you connect from your machine straight to a service, skipping the Gateway, api-gateway and NetworkPolicy, so in one go you can tell whether the problem is in the service itself or somewhere along the path.
Common pitfalls
- The Deployment sets
replicasand also has an HPA. Everyapplyresets the replica count to the value in the file, instantly cutting the replicas the HPA just added. Don't setreplicason a Deployment with an HPA. - Using the
latesttag. With an unchanged Pod template there's no rolling update, and you can't tell which version is on the cluster; rollback becomes guesswork. Always use the git SHA. - The liveness probe checks the database. One database blip and every service is restarted together. Liveness checks only the program itself.
- No CPU requests. The HPA computes utilisation relative to requests; without requests it can't scale, and the scheduler doesn't know where to place the Pod.
- A PDB with
minAvailableand a single replica. Nodes can never be drained and cluster upgrades get stuck. UsemaxUnavailable: 1. - Assuming that creating a NetworkPolicy makes you safe. If the network plugin doesn't support it, the object exists but has no effect; if you restrict egress and forget to allow DNS, no service can resolve names. Test after deploying.
- Running gRPC over ClusterIP and expecting even traffic. Long-lived connections make requests stick to a few Pods, and adding replicas doesn't help. Use a headless Service with client-side load balancing, or a service mesh.
- Modifying the migration Job and applying it directly. A Job's Pod template is immutable; you must delete it and create it again, and it must finish before the new version goes live.
- Adding shared labels with
commonLabels. It rewrites the selector, so later label changes can't update the Deployment. Uselabelsand keepincludeSelectorsat its default of false. - Putting production passwords in
secretGeneratorfiles. Those files end up in CI's working directory or even get committed by mistake. Hand production secrets to an external key management service. - Calling it done after
rollout undo. The cluster and the repository disagree, and the next release brings the broken version back. After stopping the bleeding, revert in the repository. - Forgetting the Pod Security level. The production namespace is restricted; a Job or sidecar container missing its
securityContextwill be refused, and the error appears in the ReplicaSet's or Job's events, not in the output ofapply.
Key takeaways
- One namespace (or one cluster) per environment:
shopfor production,shop-devfor development; services call each other by the short nameorder:9090, whose full name isorder.shop.svc.cluster.local. - Each service has its own Deployment, Service, HPA, PDB and ConfigMap under
deploy/k8s/base/<service>/, consistently usingapp.kubernetes.io/*labels. - A Deployment needs three probes, CPU requests, a memory limit,
GOMEMLIMIT, apreStopdelay and a non-rootsecurityContext. configMapGenerator's hashed names make configuration changes trigger rolling updates and bring configuration back on rollback; production secrets come from an external key management service.- Use managed services for production databases, install third-party components (NATS, Operators, monitoring) with Helm, and manage your own services with Kustomize.
- There is only one way in: a Gateway API HTTPRoute (or Ingress) pointing to api-gateway; NetworkPolicy denies by default, then allows Gateway → api-gateway → backends rule by rule.
- Kustomize: base holds shared resources, and overlays describe each environment's differences with
namespace,images, ConfigMap merges and patches; check withkubectl kustomizeandkubectl diffbefore applying. - Release flow: build images → tag with the git SHA and push →
kustomize edit set image→ delete the old Job, apply configuration and migrations, wait for completion →apply -k→rollout status. - Rollback:
rollout undoto stop the bleeding, then revert in the repository; keep schema changes backward compatible rather than relying on down migrations. - Debug from the outside in: get/describe → logs (including
--previous) → endpointslices/port-forward/NetworkPolicy → distributed tracing.
Next
At this point the whole order system runs in the cluster with multiple replicas, autoscales, and releases and rolls back safely. But not every system needs all this from day one: the next article, “From Monolith to Microservices: Evolving the Architecture at 100K, 1M and 10M Scale”, takes a step back and looks at how the same shop system should grow with scale, from a single monolith, stage by stage, into what it is now — and when to split, and when not to split yet.