Columns › 8. Security and working with AI › Part 31
Microservice Security: Authentication, Authorisation, Secrets and Supply Chain
微服務的安全性:認證、授權、密鑰與供應鏈
Starting from a threat model of the shop system: validate JWTs at the gateway and propagate identity, enforce object-level authorisation in every service to stop BOLA, use mTLS between services, keep secrets out of code and logs, and guard the supply chain with govulncheck, SBOMs and image signing.
In this article
- What this article solves
- Step 1: Draw the threat model first, then decide what to defend against
- Step 2: Centralise authentication at the gateway: validate every field of the JWT
- Step 3: Pass the identity to the backend, in a way that can't be forged
- Step 4: Authorise in every service: stop BOLA
- Step 5: Use mTLS between services to confirm identity, and enforce least privilege
- Step 6: Keep secrets out of code, container images and logs, and make them rotatable
- Step 7: Input validation, parameterised queries, rate limiting and security headers
- Step 8: Guard the supply chain: dependencies, container images and the build process
- Step 9: Collect it into a checklist that cross-references the earlier articles
- Common pitfalls
- Key takeaways
- Next
In the previous article, “Handling Money in Go: Minor Units, Decimals, Currencies, Rounding and Allocation”, we made "getting the money right" solid; but however precise the arithmetic, if someone can see other people's orders, change other people's amounts, or walk off with the payment service's API key, all that effort is wasted. This article opens Part 8 by walking through the whole order system again from a security point of view.
What this article solves
Over the previous twenty-odd articles, we split the order system into four services, api-gateway, order, payment and inventory, connected them with gRPC and a message queue, and deployed them into the shop namespace in Kubernetes. Everything works, but every service we split out adds another set of network connections, another configuration, another database password and another container image, and therefore several more places to attack. In a monolith, "calling a function inside the same process" is inherently trusted; in microservices, every call crosses the network, and trust can no longer be taken for granted.
This article answers a series of very concrete questions:
- Who might attack this system, where would they get in, and what do they want? In other words, first draw a threat model.
- Where should authentication (who you are) happen? When the gateway validates a JWT, exactly which fields must it check?
- How is the verified identity passed safely to the backend services, without letting clients forge it?
- Why can't authorisation (what you are allowed to do) be done only at the gateway? What does BOLA, number one in the OWASP API Security Top 10 (2023), look like, and how do you stop it in Go?
- How do services confirm each other's identity? How is mTLS configured with Go's
crypto/tls? - Where should secrets such as database passwords and payment API keys live, and how are they rotated?
- What do input validation, SQL injection protection, request size limits, rate limiting, CORS and security headers each defend against?
- How do you write logs without writing tokens or personal data into them?
- How do you guard the supply chain of the code itself, its dependencies and its container images?
Security is not a feature; it is a set of habits spread across every layer. The approach here: each step maps to a program already written in an earlier article, points out what to add and where, and at the end everything is collected into a checklist that cross-references the earlier articles.
Step 1: Draw the threat model first, then decide what to defend against
Security work without a threat model often turns into "add every recommendation you've seen online", with the result that what really matters doesn't get done while minor things get done very thoroughly. A threat model doesn't have to be formal; answering four questions is enough: What are we protecting? Who will attack? Where do they get in? How much do we lose if we can't stop them?
For the order system, the assets to protect are roughly:
| Asset | Examples | Consequence if obtained or altered |
|---|---|---|
| Users' personal data | Name, email, address, phone | Leaks, regulatory liability |
| Order data | Who bought what, amounts, status | Seeing other people's orders, tampering with status (unpaid becomes paid) |
| Ability to pay | Payment API keys, refund permissions | Fraudulent charges, malicious refunds |
| Inventory | Reservations and releases | Stock maliciously tied up, overselling |
| The system itself | Database, cluster, CI | The whole system taken over |
There are roughly three kinds of attacker: anonymous external users (who can only reach the public API), logged-in users with bad intentions (who hold a valid token and want to see other people's data or gain permissions they shouldn't have), and someone who has already broken through a layer (for example, who has a shell in a container, or has hidden malicious code in a dependency). The second kind is the easiest to overlook, because "logged in" is often mistaken for "trusted".
Next, draw the trust boundaries: every time data crosses a boundary, ask again "who are you, and are you allowed to do this?"
Figure: Internet to gateway is the first boundary, gateway to in-cluster services is the second, and services to the database, message queue and external payment provider is the third; every line needs its own checks.
With this diagram, you can go through each arrow using a classification such as STRIDE (spoofing, tampering, repudiation, information disclosure, denial of service, elevation of privilege). Take the "gateway → order" line: could someone bypass the gateway and call order directly (spoofing)? Can a client forge the headers that carry identity (spoofing, elevation of privilege)? Is the connection encrypted (information disclosure)? Every step in the rest of this article answers questions like these.
Tip: Keep the threat model in the repo's
docs/and review it alongside the code. Coming back to update the diagram every time you add a service, a public API or an external dependency matters more than writing it very thoroughly once.
Step 2: Centralise authentication at the gateway: validate every field of the JWT
In “Building an API Gateway in Go: Routing, Authentication and Rate Limiting”, we made the gateway the single entry point and did authentication there. This step fills in the details of JWT validation. In practice, users usually log in to an identity provider (for example a self-hosted Keycloak or a cloud identity service) through OIDC (OpenID Connect), receive a signed JWT access token, and then put it in the Authorization: Bearer ... header on every API call.
Once the gateway receives a token, these are all the things it must check; miss one and you have a hole:
| Check | Why |
|---|---|
| Signature | Confirms the token was signed by the identity provider and hasn't been altered. The public key comes from JWKS, chosen by the header's kid |
| Algorithm | Only accept the expected algorithm (e.g. RS256 or ES256); never accept none, and never let the token decide to verify an asymmetric key with HMAC |
iss (issuer) | Must equal the URL of the identity provider you trust, so tokens signed by another provider can't be used |
aud (audience) | Must include this API's identifier; otherwise tokens obtained by other applications can also call your API |
exp, nbf | Reject if expired or not yet valid; allow a clock skew of a few tens of seconds at most |
| Required claims | For example, sub (the user ID) must not be empty |
Key rotation relies on JWKS (JSON Web Key Set): the identity provider publishes the list of currently valid public keys at a public URL (the jwks_uri in the OIDC discovery document), each key with its own kid. To rotate, the provider first adds the new key to the list, after a while starts signing with the new key, and only after another while removes the old key. What the gateway has to do is cache the JWKS, and fetch it again when it meets an unknown kid (with rate limiting, so an attacker can't use made-up kid values to make you hammer the identity provider).
Don't write JWT validation yourself. The example below uses github.com/coreos/go-oidc/v3, which handles discovery, JWKS caching and rotation, the signature, iss, aud and exp itself:
// Snippet: requires github.com/coreos/go-oidc/v3/oidc
package auth
import (
"context"
"net/http"
"strings"
"github.com/coreos/go-oidc/v3/oidc"
)
type Claims struct {
Subject string `json:"sub"`
Roles []string `json:"roles"`
}
func NewVerifier(ctx context.Context, issuer, audience string) (*oidc.IDTokenVerifier, error) {
provider, err := oidc.NewProvider(ctx, issuer) // reads the discovery document to get jwks_uri
if err != nil {
return nil, err
}
return provider.Verifier(&oidc.Config{
ClientID: audience, // checks aud
SupportedSigningAlgs: []string{oidc.RS256}, // only accept the expected algorithm
}), nil
}
// Authenticate is the gateway middleware: verify the token, delete incoming identity headers, then add the verified identity.
func Authenticate(v *oidc.IDTokenVerifier, next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
r.Header.Del("X-User-ID") // anything the client sends itself never counts
r.Header.Del("X-User-Roles")
raw, ok := strings.CutPrefix(r.Header.Get("Authorization"), "Bearer ")
if !ok {
http.Error(w, "unauthenticated", http.StatusUnauthorized)
return
}
tok, err := v.Verify(r.Context(), raw) // signature, iss, aud, exp
if err != nil {
http.Error(w, "unauthenticated", http.StatusUnauthorized)
return
}
var c Claims
if err := tok.Claims(&c); err != nil || c.Subject == "" {
http.Error(w, "unauthenticated", http.StatusUnauthorized)
return
}
r.Header.Set("X-User-ID", c.Subject)
r.Header.Set("X-User-Roles", strings.Join(c.Roles, ","))
r.Header.Del("Authorization") // the backend doesn't need the raw token, so don't pass it on
next.ServeHTTP(w, r)
})
}The IDTokenVerifier in this package's name was originally meant for OIDC ID tokens, but what it verifies is simply the signature and standard fields of an ordinary JWT; if your access token is a JWT with aud set to the API's identifier, it is used the same way. If the identity provider issues opaque tokens, query the token introspection endpoint instead. The principle is the same: verify once at the gateway, and the backend receives the verified result.
Three details that are easy to miss. First, the error response is always a terse 401; don't return details such as "invalid signature" or "expired", so attackers get fewer clues (write the details to the log). Second, the two r.Header.Del lines are critical, as the next step explains. Third, logout and revocation: a JWT stays valid until it expires, so access tokens should be short-lived (a few minutes to a quarter of an hour), paired with refresh tokens; only if you really need immediate revocation should you add a revocation list at the gateway.
Step 3: Pass the identity to the backend, in a way that can't be forged
After the gateway has verified the token, backend services need to know "who sent this request". There are two common approaches:
| Approach | Advantages | Watch out for |
|---|---|---|
The gateway adds headers such as X-User-ID | Simple; the backend doesn't need to understand JWTs | The backend must accept connections only from the gateway; the gateway must first delete same-named headers sent by the client |
| Forward the JWT unchanged; the backend verifies it again | The backend doesn't have to trust network location | Every service needs JWKS configured; the further a token travels, the larger the exposure if it leaks |
| The gateway exchanges it for a short-lived internal token | The backend can verify the source, and permissions can be narrowed | One more issuing component to maintain |
This series uses the first, and protects its preconditions with two things: the gateway always deletes incoming X-User-* headers first (the code in the previous step), and the backend only accepts connections from the gateway, enforced in Kubernetes by restricting sources with NetworkPolicy (see “Deploying a Microservice System on Kubernetes”), plus mTLS from Step 5 to confirm the peer's certificate really is the gateway's. Doing only one of the two isn't enough: without deleting the headers, a client that sends X-User-ID: admin itself becomes an administrator; without restricting the source, any compromised container in the cluster can impersonate any user.
Figure: Authentication happens once at the entry point; the gateway hands the verified identity to the backend, but "is this order yours?" has to be decided by the order service itself.
With gRPC, the identity goes in metadata: the gateway's client interceptor adds it, and the backend's server interceptor reads it and puts it into the context, exactly the same idea as HTTP headers (see “gRPC and Protocol Buffers in Go”). Events passed through the message queue are different: an event is "a fact that happened inside the system". For example, the UserID recorded in OrderCreated is part of the order data; consumers should not treat it as "the person making the current request", nor make permission decisions based on it.
Step 4: Authorise in every service: stop BOLA
Number one in the OWASP API Security Top 10 (2023 edition) is API1:2023 Broken Object Level Authorization (BOLA): the API looks up data by an ID in the path or parameters, but never confirms that the data belongs to the caller. The classic example is GET /orders/{id}: a logged-in alice changes o-1 in the URL to o-2 and sees bob's order. It ranks first not because it's technically hard, but because it is so easy to miss: the gateway only knows "this person is logged in"; it doesn't know whose o-2 is, and only the service that owns the data can make that judgement.
For reference, here are the ten items of the 2023 edition; the following steps map to several of them:
| ID | Name | Example in this system |
|---|---|---|
| API1 | Broken Object Level Authorization | Changing the ID to see someone else's order |
| API2 | Broken Authentication | The JWT's aud isn't checked; the none algorithm is accepted |
| API3 | Broken Object Property Level Authorization | A client sends "status":"paid" and it's written to the database; responses include internal fields |
| API4 | Unrestricted Resource Consumption | No request size limit, no rate limiting, ten thousand rows in one query |
| API5 | Broken Function Level Authorization | Ordinary users can call admin functions such as refunds |
| API6 | Unrestricted Access to Sensitive Business Flows | Bots place masses of orders to tie up stock |
| API7 | Server Side Request Forgery | A service sends requests to a URL the user supplies (e.g. a custom callback URL) |
| API8 | Security Misconfiguration | Wide-open CORS, stack traces in error messages, missing security headers |
| API9 | Improper Inventory Management | Old API versions and test endpoints still online and unmanaged |
| API10 | Unsafe Consumption of APIs | Trusting data returned by an external payment provider directly, without verifying signatures or validating it |
Common authorisation models are RBAC (role-based: the support role can see all orders) and ABAC (attribute-based: the order's UserID equals the caller, or the order's shop equals a shop the caller manages). In practice the two are often combined: RBAC decides "may you call this function" (stopping API5), and an ABAC-style owner check decides "may you touch this piece of data" (stopping API1).
The authorisation check belongs in the service layer, not in the handler or the repository, because: there may be several handlers (HTTP, gRPC, internal batch jobs), and they all reach the same service method; the repository only handles data access and shouldn't know "who the caller is". Below is a complete, directly runnable example that continues the structure from “Layered Architecture and Dependency Injection in Go” and uses httptest to run four cases:
// cmd/order: identity comes in from the gateway; object-level authorisation is checked in the service.
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"log/slog"
"net/http"
"net/http/httptest"
"strings"
)
type Status string
const (
StatusPending Status = "pending"
StatusPaid Status = "paid"
StatusCancelled Status = "cancelled"
)
type Order struct {
ID string `json:"id"`
UserID string `json:"user_id"`
Total int64 `json:"total"` // in cents
Status Status `json:"status"`
}
// Identity is what the gateway hands to the backend after verifying the token.
type Identity struct {
UserID string
Roles []string
}
func (id Identity) Has(role string) bool {
for _, r := range id.Roles {
if r == role {
return true
}
}
return false
}
type ctxKey struct{}
func identityFrom(ctx context.Context) (Identity, bool) {
id, ok := ctx.Value(ctxKey{}).(Identity)
return id, ok && id.UserID != ""
}
// withIdentity reads the headers added by the gateway. Precondition: the service only accepts connections from the gateway (mTLS),
// and the gateway first deletes same-named headers sent by the client.
func withIdentity(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
uid := r.Header.Get("X-User-ID")
if uid == "" {
http.Error(w, "unauthenticated", http.StatusUnauthorized)
return
}
var roles []string
if v := r.Header.Get("X-User-Roles"); v != "" {
roles = strings.Split(v, ",")
}
ctx := context.WithValue(r.Context(), ctxKey{}, Identity{UserID: uid, Roles: roles})
next.ServeHTTP(w, r.WithContext(ctx))
})
}
var ErrNotFound = errors.New("order not found")
type Repository interface {
Get(ctx context.Context, id string) (Order, error)
}
type memRepo map[string]Order
func (m memRepo) Get(_ context.Context, id string) (Order, error) {
o, ok := m[id]
if !ok {
return Order{}, ErrNotFound
}
return o, nil
}
type Service struct{ repo Repository }
// GetForUser is the line of defence against BOLA: after finding the order, confirm it belongs to the caller.
// If it doesn't, return ErrNotFound, so an attacker can't tell "doesn't exist" from "isn't yours".
func (s *Service) GetForUser(ctx context.Context, id string) (Order, error) {
who, ok := identityFrom(ctx)
if !ok {
return Order{}, ErrNotFound
}
o, err := s.repo.Get(ctx, id)
if err != nil {
return Order{}, err
}
if o.UserID != who.UserID && !who.Has("support") {
slog.WarnContext(ctx, "order access denied", "order_id", id, "user_id", who.UserID)
return Order{}, ErrNotFound
}
return o, nil
}
func (s *Service) handleGet(w http.ResponseWriter, r *http.Request) {
o, err := s.GetForUser(r.Context(), r.PathValue("id"))
if errors.Is(err, ErrNotFound) {
http.Error(w, "not found", http.StatusNotFound)
return
}
if err != nil {
http.Error(w, "internal error", http.StatusInternalServerError)
return
}
w.Header().Set("Content-Type", "application/json")
json.NewEncoder(w).Encode(o)
}
func main() {
svc := &Service{repo: memRepo{
"o-1": {ID: "o-1", UserID: "alice", Total: 129900, Status: StatusPaid},
"o-2": {ID: "o-2", UserID: "bob", Total: 4500, Status: StatusPending},
}}
mux := http.NewServeMux()
mux.HandleFunc("GET /orders/{id}", svc.handleGet)
h := withIdentity(mux)
for _, tc := range []struct{ user, roles, id string }{
{"alice", "", "o-1"}, // own order: 200
{"alice", "", "o-2"}, // someone else's order: 404
{"carol", "support", "o-2"}, // support role: 200
{"", "", "o-1"}, // no identity: 401
} {
req := httptest.NewRequest(http.MethodGet, "/orders/"+tc.id, nil)
if tc.user != "" {
req.Header.Set("X-User-ID", tc.user)
}
if tc.roles != "" {
req.Header.Set("X-User-Roles", tc.roles)
}
rec := httptest.NewRecorder()
h.ServeHTTP(rec, req)
fmt.Printf("user=%-6q order=%s -> %d\n", tc.user, tc.id, rec.Code)
}
}The results are 200, 404, 200 and 401. A few design decisions are worth explaining:
- Return 404, not 403, when it isn't yours. Returning 403 tells the attacker "this ID exists", which lets them enumerate valid order IDs. 404 makes "doesn't exist" and "isn't yours" look the same.
- Write a warning log entry on denial. Normal users almost never hit this path; a burst of them in a short time means someone is scanning IDs, which can be hooked up to alerts as in “Observability in Go: Logs, Metrics and Tracing”.
- Put the condition into SQL for list queries.
ListOrdersshouldn't fetch everything and then filter in Go; useWHERE user_id = $1. Otherwise pagination and counts leak how many records other people have. - Hard-to-guess IDs (such as UUIDs) help, but can't replace the check. IDs appear in URLs, logs and shared links, and will be seen sooner or later.
Writes are the same. For operations such as cancelling an order or changing an address, confirm the owner and status at the start of the service method, and put "check the owner" into table-driven tests (see “Testing in Go: Table-Driven Tests, httptest and Integration Tests”), so that every method that touches orders has a "someone else's order" test case. Such tests are cheap, yet they are the most effective way to keep BOLA from regressing.
Step 5: Use mTLS between services to confirm identity, and enforce least privilege
Internal calls such as gateway to order and order to inventory also have to answer "who are you?". Ordinary TLS only lets the client confirm the server's identity; mTLS (mutual TLS) has both sides present certificates, so the server can also confirm which service is calling. It solves two things at once: encryption of the connection (traffic inside the cluster may also be visible to other programs on the same node), and service identity (only the gateway can call order's public API, only order can call inventory's reservation API).
Figure: Chain verification proves "you are some legitimate service in the cluster"; the name check then decides "you may call me".
There are two ways to do it. The first is to hand it to a service mesh (such as Istio or Linkerd): a proxy next to each Pod automatically issues and rotates certificates and does mTLS, and the mesh's authorisation policies describe "who may call whom", with no changes to the Go program at all. The second is to configure crypto/tls yourself in Go, with certificates issued by a tool such as cert-manager and mounted into the Pod as a Secret. When you have only a few services, or want to understand how it works first, the second way is more transparent:
// mTLS: the order service only accepts callers whose certificates are signed by the in-cluster CA and whose identity is on the allowlist.
package main
import (
"crypto/tls"
"crypto/x509"
"errors"
"fmt"
"log"
"net/http"
"os"
"slices"
"time"
)
func loadPool(path string) (*x509.CertPool, error) {
pem, err := os.ReadFile(path)
if err != nil {
return nil, err
}
pool := x509.NewCertPool()
if !pool.AppendCertsFromPEM(pem) {
return nil, errors.New("no CA certificate found in " + path)
}
return pool, nil
}
// serverTLS: require and verify the client certificate, then check the service identity on it.
func serverTLS(caFile, certFile, keyFile string, allowed []string) (*tls.Config, error) {
ca, err := loadPool(caFile)
if err != nil {
return nil, err
}
return &tls.Config{
MinVersion: tls.VersionTLS13,
ClientAuth: tls.RequireAndVerifyClientCert,
ClientCAs: ca,
// Reread the files on every handshake, so rotating certificates needs no restart.
GetCertificate: func(*tls.ClientHelloInfo) (*tls.Certificate, error) {
c, err := tls.LoadX509KeyPair(certFile, keyFile)
return &c, err
},
// By the time we get here the chain is already verified; this only authorises "who may call me".
VerifyConnection: func(cs tls.ConnectionState) error {
if len(cs.PeerCertificates) == 0 {
return errors.New("no client certificate")
}
leaf := cs.PeerCertificates[0]
for _, name := range leaf.DNSNames {
if slices.Contains(allowed, name) {
return nil
}
}
return fmt.Errorf("caller %v not allowed", leaf.DNSNames)
},
}, nil
}
// clientTLS: present our own certificate when calling other services, and trust only the in-cluster CA.
func clientTLS(caFile, certFile, keyFile string) (*tls.Config, error) {
ca, err := loadPool(caFile)
if err != nil {
return nil, err
}
cert, err := tls.LoadX509KeyPair(certFile, keyFile)
if err != nil {
return nil, err
}
return &tls.Config{
MinVersion: tls.VersionTLS13,
RootCAs: ca,
Certificates: []tls.Certificate{cert},
}, nil
}
func main() {
dir := os.Getenv("TLS_DIR") // e.g. /etc/shop/tls, mounted from a Secret
cfg, err := serverTLS(dir+"/ca.crt", dir+"/tls.crt", dir+"/tls.key",
[]string{"api-gateway.shop.svc", "payment.shop.svc"})
if err != nil {
log.Fatal(err)
}
mux := http.NewServeMux()
mux.HandleFunc("GET /healthz", func(w http.ResponseWriter, _ *http.Request) { w.Write([]byte("ok")) })
srv := &http.Server{
Addr: ":8443",
Handler: mux,
TLSConfig: cfg,
ReadHeaderTimeout: 5 * time.Second,
}
// The certificate is already in TLSConfig, so both file name arguments are empty.
log.Fatal(srv.ListenAndServeTLS("", ""))
}
// newInventoryClient: the HTTP client order uses to call inventory.
func newInventoryClient(dir string) (*http.Client, error) {
cfg, err := clientTLS(dir+"/ca.crt", dir+"/tls.crt", dir+"/tls.key")
if err != nil {
return nil, err
}
return &http.Client{
Timeout: 3 * time.Second,
Transport: &http.Transport{TLSClientConfig: cfg},
}, nil
}tls.RequireAndVerifyClientCert makes the handshake fail outright when the client has no certificate, or its certificate wasn't signed by ClientCAs; VerifyConnection runs after chain verification succeeds and is used to decide "may this service call me". Here the certificate's DNS names serve as the identity; service meshes mostly use SPIFFE-format URIs (spiffe://...) in the certificate's URI SAN, whose corresponding field is leaf.URIs. gRPC uses the same tls.Config, passed in through credentials.NewTLS(cfg). Rereading the files in GetCertificate on every handshake is the simplest form of rotation; under heavy traffic you can switch to watching the files for changes, then reloading and caching.
mTLS solves identity; least privilege has to be applied at every layer:
- Database: one account per service, with access only to its own schema; order's account can't read payment's tables; application accounts have no DDL privileges, and migrations run at deploy time under a separate account (see the approach in “Deploying a Whole Microservice System with Docker Compose” of treating migrations as a one-off job).
- Message queue: each service can only publish and subscribe to the topics it needs; inventory can't publish
PaymentSucceeded. - Kubernetes: each service has its own ServiceAccount, and those that don't call the Kubernetes API have token automounting turned off; containers run as non-root with a read-only root filesystem; NetworkPolicy denies by default and only opens the connections that are needed.
- Outbound connections: only payment needs to reach the external payment provider; all outbound traffic from the other services can be blocked, which is also the most practical layer of defence against SSRF (API7).
Step 6: Keep secrets out of code, container images and logs, and make them rotatable
Secrets include database passwords, payment API keys, JWT signing private keys, TLS private keys and third-party service tokens. There are only three principles: not in version control, not in container images, not in logs; and on top of that, one more: they must be rotatable without changing the code.
The easiest way to break "not in version control" is to "put it in config.yaml just to test" and then forget to remove it; once it's pushed to the remote, it stays in the history even if the next commit deletes it, and must be treated as leaked and replaced immediately. Defences are .gitignore plus secret scanning before commit (a pre-commit hook or a scanning tool in CI), and paying special attention to configuration files in PR review. "Not in container images" means no COPY .env in the Dockerfile and no passwords passed through ARG/ENV: every layer of an image can be taken apart and inspected, and docker history also keeps build arguments.
There are roughly three ways to obtain secrets at run time:
| Method | How | Advantages | Disadvantages |
|---|---|---|---|
| Environment variables | A Kubernetes Secret injected through env | Simplest; the configuration loading from “Configuration and Graceful Shutdown in Go” can read it directly | Easily printed (environment dumps while debugging, error-reporting tools collecting the environment); updates need a Pod restart |
| Mounted files | A Secret mounted as a volume at /etc/shop/secrets/ | File permissions can be set; when the Secret is updated the files follow, and the program can reread them | The program has to handle reloading itself |
| Secret management service | Fetched at run time from Vault or a cloud Secret Manager, or synced into Secrets with External Secrets | Centralised auditing, fine-grained permissions, can issue dynamic short-lived credentials | One more dependency; startup has to handle it being unavailable |
This series recommends mounting secrets as files and using environment variables for (non-secret) configuration; medium and large teams can add a secret management service that syncs into Kubernetes Secrets, so the code doesn't need to know where the secrets come from. Remember that Kubernetes Secrets are only base64-encoded by default: the cluster should enable encryption at rest, and RBAC should restrict who can read Secrets.
Rotation should be designed in advance, not thought about only when something goes wrong: database passwords can rotate between two prepared accounts; JWT signing keys rely on JWKS publishing the old and new public keys at the same time; external payment keys follow the provider's process, which usually allows two keys to exist at once. What the program has to do is "read at more times than just startup": file-mounted secrets can be reread whenever a new connection is created, or the files can be watched for changes. The GetCertificate in Step 5 follows the same idea.
Step 7: Input validation, parameterised queries, rate limiting and security headers
This step groups API8 (misconfiguration), API4 (unrestricted resource consumption), API3 (object property level authorisation) and injection together, because they are all about "how incoming requests are handled".
Input validation belongs at the service boundary, with an allowlist mindset: define what is valid rather than enumerating what is dangerous. Quantities need upper and lower bounds, strings need maximum lengths, and enum values must be in the list. Use DisallowUnknownFields when decoding JSON, so a client that sends fields it has no business deciding, such as "status":"paid" or "total":1, is rejected outright. This is the simplest way to stop the "mass assignment" part of API3; responses likewise output only the fields listed in the DTO, never the whole database model serialised.
The fix for SQL injection is simple in Go: values always go to database/sql through placeholders (see “database/sql, Transactions and Connection Pools in Go”), never spliced into the SQL with fmt.Sprintf. Parts that can't be parameters (sort column, sort direction, table names) are mapped through an allowlist. Output encoding depends on where the output goes: JSON goes through encoding/json, HTML through html/template with automatic escaping; don't build strings yourself.
Request size is limited with http.MaxBytesReader, together with http.Server's ReadHeaderTimeout, ReadTimeout and MaxHeaderBytes; list APIs need a maximum page size. Rate limiting is done at the gateway per user and per IP (see the token bucket implementation in “Building an API Gateway in Go: Routing, Authentication and Rate Limiting”), with stricter quotas for sensitive flows such as login, placing orders and payment, which is also the first line of defence against API6.
Logs need the opposite way of thinking: whatever you write will be seen by many people and many systems. Never write tokens, passwords or card numbers; mask personal data such as email and phone before writing; user IDs can usually be written because debugging needs them. Rather than relying on everyone remembering, mask centrally with ReplaceAttr on the slog handler:
// Input validation, parameterised queries, and logs that don't leak tokens or personal data.
package main
import (
"context"
"database/sql"
"encoding/json"
"errors"
"fmt"
"log/slog"
"net/http"
"os"
"strings"
)
type CreateOrderRequest struct {
Items []struct {
SKU string `json:"sku"`
Quantity int `json:"quantity"`
} `json:"items"`
Note string `json:"note"`
}
func (r CreateOrderRequest) Validate() error {
if len(r.Items) == 0 || len(r.Items) > 50 {
return errors.New("items: need 1 to 50 items")
}
for i, it := range r.Items {
if it.SKU == "" || len(it.SKU) > 64 {
return fmt.Errorf("items[%d].sku: invalid", i)
}
if it.Quantity < 1 || it.Quantity > 99 {
return fmt.Errorf("items[%d].quantity: must be 1..99", i)
}
}
if len([]rune(r.Note)) > 200 {
return errors.New("note: too long")
}
return nil
}
func decodeCreate(w http.ResponseWriter, r *http.Request) (CreateOrderRequest, error) {
var req CreateOrderRequest
r.Body = http.MaxBytesReader(w, r.Body, 64<<10) // 64 KiB limit
dec := json.NewDecoder(r.Body)
dec.DisallowUnknownFields() // rejects fields the client shouldn't send, such as "status" and "total"
if err := dec.Decode(&req); err != nil {
return req, fmt.Errorf("decode: %w", err)
}
return req, req.Validate()
}
// Parameterised queries: values always go through placeholders, never spliced into the SQL string.
// Things that can't be parameters, such as the sort column, are mapped through an allowlist.
var sortColumns = map[string]string{"created": "created_at", "total": "total"}
func listOrders(ctx context.Context, db *sql.DB, userID, status, sort string) (*sql.Rows, error) {
col, ok := sortColumns[sort]
if !ok {
col = "created_at"
}
q := "SELECT id, total, status FROM orders WHERE user_id = $1 AND status = $2 ORDER BY " + col + " DESC LIMIT 50"
return db.QueryContext(ctx, q, userID, status)
}
// Logs: sensitive keys are always masked; emails keep only the domain.
var secretKeys = map[string]bool{"authorization": true, "token": true, "password": true, "card_number": true}
func redact(_ []string, a slog.Attr) slog.Attr {
k := strings.ToLower(a.Key)
switch {
case secretKeys[k]:
return slog.String(a.Key, "[REDACTED]")
case k == "email":
if _, domain, ok := strings.Cut(a.Value.String(), "@"); ok {
return slog.String(a.Key, "***@"+domain)
}
}
return a
}
func main() {
log := slog.New(slog.NewJSONHandler(os.Stdout, &slog.HandlerOptions{ReplaceAttr: redact}))
log.Info("login", "user_id", "u-42", "email", "alice@example.com", "token", "eyJhbGciOi...")
mux := http.NewServeMux()
mux.HandleFunc("POST /orders", func(w http.ResponseWriter, r *http.Request) {
req, err := decodeCreate(w, r)
if err != nil {
http.Error(w, "invalid request", http.StatusBadRequest)
return
}
fmt.Fprintf(w, "ok: %d items\n", len(req.Items))
})
_ = mux
_ = listOrders
}When it runs, email in the log becomes ***@example.com and token becomes [REDACTED]. Note that ReplaceAttr only sees key names; if someone prints an entire request struct under one key, it can't catch it. So the team's guidelines must still state clearly "don't print whole requests or headers into logs", and reviewers should watch for it. The same goes for tracing span attributes: don't record the Authorization header as an attribute.
Security headers and CORS are handled centrally at the gateway. A pure JSON API needs at least Content-Type: application/json and X-Content-Type-Options: nosniff, plus Strict-Transport-Security on HTTPS domains. CORS isn't a protection mechanism; it is a mechanism for "relaxing" the browser's same-origin policy: list only the front-end domains you really need, and never reflect the request's Origin back verbatim; when cookies are needed (Access-Control-Allow-Credentials: true), you certainly can't pair it with *. Error responses carry no stack traces or internal messages; the details go only into the log, with a request ID attached so support can match them up.
Step 8: Guard the supply chain: dependencies, container images and the build process
The last boundary lies outside your code: is what you're running really the code you think it is? Supply-chain attacks can happen in dependencies (malicious code planted, or known vulnerabilities), in base images, in the build environment, or by images being replaced after they're published to the registry.
Go has a good foundation here. go.sum records the hash of every module version, and together with the default checksum database (sum.golang.org), it ensures the same version downloads the same content anywhere (see “Go Project Layout and Go Modules”). Always commit go.sum; go mod verify confirms the local module cache hasn't been modified.
Check for known vulnerabilities with the official govulncheck. Its distinctive feature is call-graph analysis: it only reports when your program actually calls a vulnerable function, which is far less noisy than simply matching version numbers. Besides scanning source code, it can also scan compiled binaries:
# Dependency integrity
go mod verify
# Known vulnerabilities: analyses the call graph and only reports vulnerabilities actually reached
go install golang.org/x/vuln/cmd/govulncheck@latest
govulncheck ./...
# You can also check an already-compiled binary (e.g. one extracted from a container image)
go build -o bin/order ./cmd/order
govulncheck -mode=binary bin/order
# Generate an SBOM and provenance attestation while building the image (requires BuildKit and a registry that supports attestations)
docker buildx build --sbom=true --provenance=true \
-t registry.example.com/shop/order:1.4.2 --push -f deploy/Dockerfile .govulncheck also checks the Go standard library itself, so the fix it reports is sometimes "upgrade Go". In CI, run it on every PR, and additionally on a daily schedule against the main branch: new vulnerability advisories don't wait for you to change your code.
On the image side, “Deploying One Go Service with Docker and Kubernetes” already used a multi-stage build and a distroless base: the final image contains only a static binary and CA certificates, with no shell and no package manager, so even an attacker who gains execution has very few tools to work with. One step further:
- Pin the base image's digest (
@sha256:...) rather than just a tag, so the content behind a tag can't be swapped; then have an automated tool open PRs to update it regularly. - SBOM (software bill of materials) lists every component and version in the image. When a new vulnerability advisory comes out, you can directly ask "which images contain this component" instead of guessing one by one.
- Image signing: sign images at release with a tool such as Sigstore's cosign, and have the cluster use an admission controller to allow only images with valid signatures to be deployed. The idea is "only what CI built can go live": images someone
docker pushed by hand can't get into the cluster. - Build environment: CI tokens get only the permissions they need, third-party Actions are pinned to a commit SHA, and production deployment credentials are only available on protected branches. This is discussed further in “Bringing AI Workflows into the Team Process: PR Review, CI and Human Gates”.
Step 9: Collect it into a checklist that cross-references the earlier articles
Condense the previous eight steps into a table you can put into a PR template or a pre-release check. Every item maps to somewhere already implemented in this series, so you can find the code again:
| Item | What to confirm | Related articles |
|---|---|---|
| Threat model | A trust boundary diagram exists, and is updated for new services or new public APIs | Step 1 of this article |
| Authentication | The gateway checks the signature, algorithm, iss, aud and exp; JWKS caching and rotation | “Building an API Gateway in Go: Routing, Authentication and Rate Limiting” |
| Identity propagation | The gateway deletes incoming X-User-*; the backend only accepts connections from the gateway | “net/http and Middleware in Go”, “Go context: Cancellation, Timeouts and Request-Scoped Values” |
| Object-level authorisation | Every method that reads or writes orders checks the owner; tests include a "someone else's order" case | “Layered Architecture and Dependency Injection in Go”, “Testing in Go: Table-Driven Tests, httptest and Integration Tests” |
| Function-level authorisation | Refunds and back-office functions check roles | Step 4 of this article |
| Trust between services | mTLS or a service mesh; NetworkPolicy denies by default | “gRPC and Protocol Buffers in Go”, “Deploying a Microservice System on Kubernetes” |
| Least privilege | One database account per service, topic permissions, non-root, read-only filesystem | “Deploying a Whole Microservice System with Docker Compose”, “Deploying a Microservice System on Kubernetes” |
| Secrets | Not in git, images or logs; mounted as files; a rotation process exists | “Configuration and Graceful Shutdown in Go” |
| Input and injection | Allowlist validation, DisallowUnknownFields, parameterised queries, size limits | “database/sql, Transactions and Connection Pools in Go” |
| Resource consumption | Rate limiting, timeouts, list limits | “Building an API Gateway in Go: Routing, Authentication and Rate Limiting”, “Timeouts, Retries and Circuit Breakers” |
| External callbacks | Payment callbacks have signatures verified, idempotent handling, reconciliation | “Payment Pitfalls: Double Charges, Callbacks and Reconciliation”, “Idempotency and Distributed Locks” |
| Logs | ReplaceAttr masks tokens and personal data; access denials raise alerts | “Observability in Go: Logs, Metrics and Tracing” |
| Supply chain | go.sum committed, govulncheck in CI and on a schedule, distroless, SBOM, signing | “Deploying One Go Service with Docker and Kubernetes” |
This table isn't a list to finish in one go; it's a baseline you can raise step by step. The suggested order is to start with what "blocks the most common, highest-impact problems": object-level authorisation, complete JWT validation, keeping secrets out of git, and govulncheck in CI; then mTLS and least privilege; and finally items that need more infrastructure, such as signing and admission control.
Common pitfalls
- Authorising only at the gateway. The gateway can tell "logged in or not" and "has a certain role or not", but it can't tell "is this order yours". BOLA can only be stopped in the service that owns the data.
- The backend trusts an
X-User-IDfrom anyone. If the gateway doesn't delete incoming headers, or the backend can be called directly by other Pods in the cluster, identity headers are effectively self-service. - Checking only the JWT signature. Without checking
aud, other applications' tokens work too; without restricting the algorithm, you may face algorithm-confusion attacks; without checkingexp, a leaked token is valid forever. - Returning 403 leaks whether a resource exists. For resources owned by users, return 404 when it isn't theirs.
- Secrets in git history. Deleting the file doesn't undo the leak; any secret that was ever pushed must be replaced.
- Leaving debug logging on in production. The whole request printed while debugging often includes the
Authorizationheader and personal data. InsecureSkipVerify: trueleft in the code. Certificate verification turned off for convenience while testing goes live with everything else, and mTLS becomes meaningless. Test environments should use a test CA, not switch verification off.- CORS reflecting any
Originwhile allowing credentials. That lets any website call your API as the user. - Running
govulncheckonly once at the start of the project. New vulnerability advisories appear every day; run it continuously on a schedule. - Trusting responses and callbacks from the external payment provider. Callbacks need their signatures verified, and amounts and orders must be compared against your own data; see API10.
Key takeaways
- Draw the threat model and trust boundaries first; every time a boundary is crossed, authenticate and authorise again.
- Authenticate once at the gateway: verify the signature, restrict the algorithm, check
iss,audandexp/nbf, and handle key rotation with JWKS; leave JWT validation to a mature library. - The gateway deletes incoming identity headers before adding the verified identity; the backend only accepts connections from the gateway.
- BOLA is number one in the OWASP API Security Top 10 (2023): every service method that reads or writes a resource must check the owner, return 404 when it isn't theirs, and be guarded by table-driven tests.
- Use mTLS between services (
RequireAndVerifyClientCertplusVerifyConnectionchecking the caller's name) or a service mesh; configure the database, message queue and Kubernetes for least privilege. - Keep secrets out of git, images and logs; provide them through mounted files or a secret management service, and design rotation in advance.
- Allowlist input validation,
DisallowUnknownFields, parameterised queries, request size limits, rate limiting, security headers and strict CORS. - Use
slog'sReplaceAttrto mask tokens and personal data centrally. - Supply chain: commit
go.sum, putgovulncheckin CI and on a schedule, use distroless with pinned digests, SBOMs and image signing.
Next
Many parts of security ultimately come down to "every code change has been looked at carefully by someone, and the checks have run". The next article, “Building Go Microservices with Claude Code: Rules, Permissions and Security Review”, covers how to let an AI agent take part in development while keeping it within safe bounds through project rules, permission settings and hooks.