This resume is also available in English.

3. Splitting into microservices4 / 4 · All columns

Columns › 3. Splitting into microservices › Part 12

Service Discovery and Load Balancing for Go Microservices

服務發現與負載平衡

When services run as many instances whose IPs keep changing, how does a caller find the healthy ones? This article compares client-side and server-side load balancing, builds a heartbeat-based registry and round-robin, least-connections and consistent-hashing balancers in Go, and explains DNS-based discovery in Kubernetes.

  • Go
  • Microservices
  • Load Balancing
  • Kubernetes
In this article
  1. What this article solves
  2. Step 1: Client-side or server-side load balancing
  3. Step 2: The registry and heartbeats
  4. Step 3: Active and passive health checks
  5. Step 4: Three load-balancing algorithms
  6. Step 5: Why consistent hashing reduces movement
  7. Step 6: Service discovery in Kubernetes: Services and DNS
  8. Step 7: gRPC's long-lived connection problem and client-side load balancing
  9. Common pitfalls
  10. Key takeaways
  11. Next

In the previous article, “Building an API Gateway in Go: Routing, Authentication and Rate Limiting”, we funnelled all external traffic into api-gateway, but its routing table hard-codes the addresses of order, payment and inventory in the program. This article tackles that problem: when every service runs several instances, and instances scale out, scale in and get redeployed, how does a caller always know "which healthy instances exist right now", and how does it spread requests evenly across them?

What this article solves

In the monolith era, calling another module was just a function call; there was no such thing as "can't find the other side". Once you split into services, before order can call inventory it has to answer two questions:

  1. Service discovery: which instances of inventory exist right now? What are their addresses? Which of them are healthy?
  2. Load balancing: which one should this request go to?

The two are closely related but worth thinking about separately. Service discovery provides a "candidate list", and load balancing picks one from the list. The list may come from a registry, DNS or the platform's API; the picking may use round-robin, least connections or consistent hashing; and the picking may happen at the caller or in a load balancer in between.

This article will, in order:

  • Compare client-side and server-side load balancing.
  • Write a minimal registry in Go with heartbeats and a TTL, to understand what health checking really is.
  • Implement three common load-balancing algorithms and observe the effect of consistent hashing when scaling out.
  • See how Kubernetes builds all this in with Services and DNS, and what to watch out for with gRPC.

Step 1: Client-side or server-side load balancing

There are two basic architectures for where the load-balancing decision is made:

Client-side load balancing Server-side load balancing Order service has picking logic Registry Inventory A Inventory B Inventory C Connects directly, one hop fewer Order service Load balancer L4 / L7 Inventory A Inventory B Inventory C Simple caller, but one extra hop

Figure: on the left, the caller fetches the instance list from the registry, picks one itself and connects directly; on the right, the caller knows only one address, and a load balancer in the middle picks the backend.

AspectClient-side load balancingServer-side load balancing
Who picks the instanceA library in the callerA load balancer in the middle (L4 or L7)
Network pathConnects directly to the instance, one hop fewerPasses through an extra load balancer
Caller complexityHigh: must integrate service discovery, health state and the algorithmLow: needs only one fixed address
Multi-language supportEvery language needs a matching libraryLanguage-agnostic
Single point of failureNo central node (apart from the registry)The load balancer itself must be highly available
Effect on long-lived connectionsGood: can open a connection to each instanceL4 can only balance "connections", not the requests inside a connection
Typical examplesgRPC's built-in load balancing, service mesh sidecarsCloud load balancers, NGINX, Kubernetes Services

There is also a common middle ground: the service mesh. It places a sidecar proxy next to each service; to the application it looks like server-side load balancing (it only connects to the local proxy), but in reality it does client-side load balancing on every node.

The selection rule can be simple: with no special requirements, use the server-side load balancing the platform provides (such as a Kubernetes Service), which is the least effort; consider client-side load balancing only when you use a long-lived-connection protocol such as gRPC, or need to route by request content (for example with consistent hashing).

Step 2: The registry and heartbeats

Whichever architecture you choose, someone has to maintain the list of "which healthy instances exist right now". The most direct approach is a registry; common products include Consul, etcd, ZooKeeper and Nacos. Their core mechanism is actually quite simple:

  1. After an instance starts, it registers its service name and address with the registry.
  2. The instance sends regular heartbeats meaning "I'm still alive".
  3. If the registry receives no heartbeat from an instance within the TTL (time to live), it treats the instance as unhealthy and stops returning it in query results.
  4. When an instance shuts down gracefully, it deregisters itself rather than waiting for the TTL to expire.
  5. The caller queries the list of healthy instances for a service, usually caching it locally for a while or subscribing to change notifications.
Registry healthy if beat within TTL Inventory A Inventory B Inventory C Heartbeat lost → removed ① Register / heartbeat Order service ② Query the list ③ Call healthy instances directly

Figure: instances register with the registry and keep sending heartbeats; an instance whose heartbeats stop is taken out of service. After querying the healthy list, the caller calls the instances directly.

Below is a minimal registry written with the standard library. For simplicity, registration and heartbeat are the same action: each PUT /instances renews the lease once; GET /services/{service} returns only instances that have sent a heartbeat within the TTL; and DELETE lets an instance take itself out of service on graceful shutdown. Routing uses the methods and path parameters that http.ServeMux supports from Go 1.22:

// A minimal service registry: instances register and renew with regular heartbeats, and drop out automatically when heartbeats stop.
package main

import (
	"encoding/json"
	"log"
	"net/http"
	"sort"
	"sync"
	"time"
)

const ttl = 10 * time.Second // illustrative: no heartbeat for over 10 seconds means unhealthy

type instance struct {
	Service  string    `json:"service"`
	Addr     string    `json:"addr"`
	LastBeat time.Time `json:"last_beat"`
}

type registry struct {
	mu   sync.RWMutex
	byID map[string]*instance // key = service + "|" + addr
}

func (r *registry) beat(service, addr string, now time.Time) {
	r.mu.Lock()
	defer r.mu.Unlock()
	id := service + "|" + addr
	if in, ok := r.byID[id]; ok {
		in.LastBeat = now
		return
	}
	r.byID[id] = &instance{Service: service, Addr: addr, LastBeat: now}
}

func (r *registry) deregister(service, addr string) {
	r.mu.Lock()
	defer r.mu.Unlock()
	delete(r.byID, service+"|"+addr)
}

// healthy returns only instances with a heartbeat within the TTL.
func (r *registry) healthy(service string, now time.Time) []string {
	r.mu.RLock()
	defer r.mu.RUnlock()
	var addrs []string
	for _, in := range r.byID {
		if in.Service == service && now.Sub(in.LastBeat) < ttl {
			addrs = append(addrs, in.Addr)
		}
	}
	sort.Strings(addrs)
	return addrs
}

// reap periodically clears records that expired long ago, so the map doesn't only ever grow.
func (r *registry) reap(now time.Time) {
	r.mu.Lock()
	defer r.mu.Unlock()
	for id, in := range r.byID {
		if now.Sub(in.LastBeat) > 3*ttl {
			delete(r.byID, id)
		}
	}
}

type beatReq struct {
	Service string `json:"service"`
	Addr    string `json:"addr"`
}

func main() {
	reg := &registry{byID: map[string]*instance{}}
	go func() {
		for now := range time.Tick(ttl) {
			reg.reap(now)
		}
	}()

	mux := http.NewServeMux()
	// Registration and heartbeat are the same action: each PUT renews the lease once
	mux.HandleFunc("PUT /instances", func(w http.ResponseWriter, r *http.Request) {
		var req beatReq
		if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, 1<<10)).Decode(&req); err != nil ||
			req.Service == "" || req.Addr == "" {
			http.Error(w, "service and addr are required", http.StatusBadRequest)
			return
		}
		reg.beat(req.Service, req.Addr, time.Now())
		w.WriteHeader(http.StatusNoContent)
	})
	// Take yourself out of service on graceful shutdown instead of waiting for the TTL to expire
	mux.HandleFunc("DELETE /instances/{service}/{addr}", func(w http.ResponseWriter, r *http.Request) {
		reg.deregister(r.PathValue("service"), r.PathValue("addr"))
		w.WriteHeader(http.StatusNoContent)
	})
	mux.HandleFunc("GET /services/{service}", func(w http.ResponseWriter, r *http.Request) {
		w.Header().Set("Content-Type", "application/json")
		json.NewEncoder(w).Encode(reg.healthy(r.PathValue("service"), time.Now()))
	})

	log.Println("registry listening on :8500")
	log.Fatal(http.ListenAndServe(":8500", mux))
}

A few design points:

  • Health is "inferred": the registry doesn't know whether an instance can really handle requests, only whether it sends heartbeats on time. So heartbeats should be sent by the logic inside the instance that genuinely represents health, for example only when the database connection pool is confirmed usable, rather than by a separate goroutine that sends them unconditionally.
  • The TTL trade-off: too short a TTL and minor network jitter wrongly takes instances out of service; too long and callers learn that an instance has died only long after the fact. A common setting is a heartbeat interval of about a third of the TTL, which tolerates occasionally missing one or two.
  • Separate read and write locking: queries are far more frequent than heartbeats, so we use sync.RWMutex, a technique mentioned in “Goroutines, Channels and Concurrency Patterns in Go”.
  • Clearing expired records: reap periodically deletes records that haven't had a heartbeat for a long time, so the map doesn't only ever grow.

On the instance side, the job is to start sending regular PUTs once start-up completes, and in the shutdown flow from “Configuration and Graceful Shutdown in Go”, first DELETE itself, wait a little while for callers to update their lists, and only then stop accepting new requests. Do it in the reverse order and callers will send requests to an instance that has already shut down before their lists are updated.

Tip: this registry is itself a single point of failure, and its data lives only in memory. Production registries replicate data across several nodes with a consensus algorithm such as Raft. Once you understand the principle, use a mature product in production, or simply use the mechanism built into the platform (Step 6).

Step 3: Active and passive health checks

A heartbeat is a health signal that "the instance reports itself". The other approach is an external active check, and the two are often used together:

MethodWho runs itHowAdvantageDisadvantage
Heartbeat / TTLThe instance itselfPeriodically renews its lease with the registryLight load on the registry"Alive" doesn't mean "able to serve"
Active checkThe registry or the load balancerPeriodically calls /healthz or the gRPC Health serviceThe real state as seen from outsideCheck traffic grows with the number of instances
Passive checkThe callerWatches the error rate of real requests and temporarily ejects instances after consecutive failuresFastest reaction, no extra trafficNeeds failed requests to happen first

In Kubernetes, active checks are performed by the kubelet through probes, and there are two kinds of probe with different meanings:

  • Liveness probe: failure means the process is stuck and needs restarting. The check must be very lightweight and must not depend on external resources; otherwise a blip in the database restarts every instance.
  • Readiness probe: failure means "don't send me traffic for now", for example because the instance is still warming up or the database it depends on is temporarily unreachable. The instance is removed from the Service's endpoint list but is not restarted.

A passive check is the same concept as the circuit breaker in “Timeouts, Retries and Circuit Breakers”: instances that fail repeatedly are temporarily removed from the list, and after a while a small number of requests are let through to probe them.

Step 4: Three load-balancing algorithms

With a list of healthy instances, the next step is to pick one. The program below implements the three most commonly used algorithms, and you can run it directly to observe the results:

// Three load-balancing algorithms: round-robin, least connections and consistent hashing (with virtual nodes).
package main

import (
	"fmt"
	"hash/crc32"
	"slices"
	"strconv"
	"sync"
	"sync/atomic"
)

// ---------- Round-robin ----------

type RoundRobin struct {
	addrs []string
	next  atomic.Uint64
}

func (rr *RoundRobin) Pick() string {
	n := rr.next.Add(1) - 1
	return rr.addrs[n%uint64(len(rr.addrs))]
}

// ---------- Least connections ----------

type LeastConn struct {
	mu     sync.Mutex
	active map[string]int
}

// Acquire picks the instance with the fewest in-flight requests; always call the returned release when done.
func (lc *LeastConn) Acquire() (addr string, release func()) {
	lc.mu.Lock()
	defer lc.mu.Unlock()
	best := -1
	for a, n := range lc.active {
		// Break ties by name so the result is predictable
		if best == -1 || n < best || (n == best && a < addr) {
			addr, best = a, n
		}
	}
	lc.active[addr]++
	return addr, func() {
		lc.mu.Lock()
		lc.active[addr]--
		lc.mu.Unlock()
	}
}

// ---------- Consistent hashing ----------

type Ring struct {
	replicas int               // virtual nodes per instance
	hashes   []uint32          // sorted positions on the ring
	owner    map[uint32]string // position → instance
}

func NewRing(replicas int, addrs ...string) *Ring {
	r := &Ring{replicas: replicas, owner: map[uint32]string{}}
	for _, a := range addrs {
		r.Add(a)
	}
	return r
}

func (r *Ring) Add(addr string) {
	for i := range r.replicas {
		h := crc32.ChecksumIEEE([]byte(addr + "#" + strconv.Itoa(i)))
		r.hashes = append(r.hashes, h)
		r.owner[h] = addr
	}
	slices.Sort(r.hashes)
}

// Get walks clockwise from the key's position to the first virtual node.
func (r *Ring) Get(key string) string {
	h := crc32.ChecksumIEEE([]byte(key))
	i, _ := slices.BinarySearch(r.hashes, h)
	if i == len(r.hashes) {
		i = 0 // wrap around to the start of the ring
	}
	return r.owner[r.hashes[i]]
}

func main() {
	addrs := []string{"inventory-a:9003", "inventory-b:9003", "inventory-c:9003"}

	rr := &RoundRobin{addrs: addrs}
	for range 4 {
		fmt.Print(rr.Pick(), " ")
	}
	fmt.Println()

	lc := &LeastConn{active: map[string]int{}}
	for _, a := range addrs {
		lc.active[a] = 0
	}
	a1, done1 := lc.Acquire()
	a2, _ := lc.Acquire()
	done1() // the first request has finished
	a3, _ := lc.Acquire()
	fmt.Println("least-conn:", a1, a2, a3)

	// Consistent hashing: after adding a fourth instance, only some keys change owner
	ring := NewRing(100, addrs...)
	before := map[string]string{}
	for i := range 10000 {
		k := "SKU-" + strconv.Itoa(i)
		before[k] = ring.Get(k)
	}
	ring.Add("inventory-d:9003")
	moved := 0
	for k, old := range before {
		if ring.Get(k) != old {
			moved++
		}
	}
	fmt.Printf("consistent hash: %.1f%% of keys moved\n", float64(moved)/100)
}

Running it produces output like this (the consistent-hashing percentage varies slightly with the hash function and the number of virtual nodes; the figure is only illustrative):

inventory-a:9003 inventory-b:9003 inventory-c:9003 inventory-a:9003
least-conn: inventory-a:9003 inventory-b:9003 inventory-a:9003
consistent hash: 22.0% of keys moved

Round-robin: take turns in order, incrementing a counter with atomic.Uint64, no lock needed. When every request costs about the same and every instance has the same specification, round-robin is simple and effective. Its variant, "weighted round-robin", gives larger instances more requests.

Least connections: pick the instance with the fewest in-flight requests. When processing times vary widely (for example some queries take 5 milliseconds and others 2 seconds), round-robin can let one instance happen to pile up a heap of slow requests, while least connections automatically avoids busy instances. One implementation note: every pick must be paired with a release that is always called when the request finishes (usually with defer), otherwise the count only ever goes up. With many instances, scanning the whole list also gets more expensive; a common compromise is "pick two at random and choose the less busy one" (the power of two choices), which performs close to least connections at a much lower cost.

Consistent hashing: the same key always lands on the same instance, which suits situations that need "affinity", for example when each instance caches inventory data for certain SKUs locally and you want all requests for the same SKU to go to the same instance to improve the cache hit rate.

AlgorithmSuitable forWatch out for
Round-robinRequests of similar cost, instances of the same specificationSlow requests may concentrate on one instance
Weighted round-robinInstances of different specificationsWeights must be adjusted with the specification
Least connectionsWidely varying processing timesNeeds to track in-flight request counts
Power of two choicesMany instances, a cheap approximation of least connectionsNeeds load information
Consistent hashingLocal caches, stateful shardingHot keys can overload a single instance

Step 5: Why consistent hashing reduces movement

If you distribute keys with the most obvious hash(key) % N, then when the number of instances goes from 3 to 4, almost every key lands somewhere new, which is the same as invalidating every cache at once, and the backend database suddenly takes all the traffic. Consistent hashing exists to solve exactly this problem.

The approach is to imagine the range of hash values as a ring: each instance is placed at some position on the ring according to the hash of its name; each key also computes a position, and the first instance found walking clockwise from there is its owner.

Inventory A Inventory B Inventory C Inventory D (new) SKU-1 SKU-7 SKU-9 SKU-1 → Inventory B SKU-7 → Inventory C SKU-9 → Inventory D (belonged to A before D) Adding D moves only the keys between C and D Other keys stay put

Figure: a key walks clockwise from its position on the ring to the first node; when a new node D is added, only the keys between C and D are taken over by D, and the other keys stay put.

When a new instance is added, it only "takes over" the adjacent segment of the ring, and other keys are completely unaffected. Ideally, going from N instances to N+1, only about 1/(N+1) of the keys need to move; in the example, going from 3 to 4 instances, the theoretical figure is 25%, and the actual result is close to that.

replicas in the example is the number of virtual nodes. If each instance places only one point on the ring, the points are likely to be unevenly distributed and some instance ends up responsible for a particularly large segment. Placing tens to hundreds of virtual nodes per instance makes the distribution much more even; and when an instance goes away, its range is spread across several instances rather than all landing on the next neighbour.

Watch out for hot keys with consistent hashing: if one SKU is in a big promotion, all its requests land on the same instance. Common remedies are to spread hot keys out further, or to switch to "consistent hashing with bounded loads", where an instance whose load exceeds the average by a certain proportion passes requests on to the next node.

Step 6: Service discovery in Kubernetes: Services and DNS

If your services run on Kubernetes, the platform has already done most of the work above:

  • Pods get scheduled, restarted and change IPs, so you should never remember a Pod's IP directly.
  • A Service selects a group of Pods by label and provides a stable virtual IP (ClusterIP) and DNS name. Only Pods that pass their readiness probe appear in the Service's endpoints (EndpointSlice), which is a built-in registry plus health checking.
  • Cluster DNS (usually CoreDNS) lets you find a Service by name. Within the same namespace, plain inventory resolves; the full name is inventory.shop.svc.cluster.local.
  • kube-proxy (or an alternative network plugin) distributes connections sent to the ClusterIP among the Pods behind it on every node, which is a form of L4 server-side load balancing.
# A regular Service: a stable virtual IP, with kube-proxy distributing at the connection level
apiVersion: v1
kind: Service
metadata:
  name: inventory
  namespace: shop
spec:
  selector:
    app: inventory
  ports:
    - name: grpc
      port: 9003
      targetPort: 9003
---
# A headless Service: no virtual IP; DNS returns the IP of every ready Pod directly
apiVersion: v1
kind: Service
metadata:
  name: inventory-headless
  namespace: shop
spec:
  clusterIP: None
  selector:
    app: inventory
  ports:
    - name: grpc
      port: 9003
      targetPort: 9003

For ordinary HTTP/1.1 services, a regular Service is enough. This is also why, on Kubernetes, the gateway's routing table from the previous article can simply say http://order:8080: DNS resolves the name and the Service handles load balancing. The complete deployment configuration is demonstrated in “Deploying One Go Service with Docker and Kubernetes” and “Deploying a Microservice System on Kubernetes”.

Step 7: gRPC's long-lived connection problem and client-side load balancing

A regular Service has an often-overlooked trap: it balances "connections", not "requests". gRPC is built on HTTP/2, where one connection can carry a large number of concurrent requests and stays open for a long time. The result: each order instance establishes a connection to just one of the inventory Pods and then sends it every request; scale inventory out to 10 Pods and the 9 new ones may get almost no traffic.

The fix is to let the gRPC client do load balancing itself: get every Pod's IP through a headless Service, open a connection to each IP, then distribute requests round-robin. grpc-go has this built in; you only need to specify DNS resolution and the round_robin policy when creating the connection:

// Snippet: gRPC client-side load balancing (DNS resolves several IPs, round-robin distribution, with health checks)
package main

import (
	"log"

	"google.golang.org/grpc"
	"google.golang.org/grpc/credentials/insecure"
	_ "google.golang.org/grpc/health" // enable client-side health checking

	inventoryv1 "example.com/shop/api/proto/inventory/v1"
)

const serviceConfig = `{
  "loadBalancingConfig": [{"round_robin": {}}],
  "healthCheckConfig": {"serviceName": ""}
}`

func main() {
	// The headless Service makes DNS return each Pod's IP, and round_robin opens a connection to each IP
	conn, err := grpc.NewClient(
		"dns:///inventory-headless.shop.svc.cluster.local:9003",
		grpc.WithTransportCredentials(insecure.NewCredentials()),
		grpc.WithDefaultServiceConfig(serviceConfig),
	)
	if err != nil {
		log.Fatal(err)
	}
	defer conn.Close()
	inventory := inventoryv1.NewInventoryServiceClient(conn)
	_ = inventory // handed to order's service
}

Notes:

  • The dns:/// target: grpc-go resolves the name with DNS and re-resolves when connections run into problems. The headless Service makes DNS return the IP of every ready Pod.
  • round_robin: grpc-go's default policy is pick_first, which connects only to the first address; only by switching to round_robin does it create a subconnection to every address and use them in turn.
  • Health checks: once you set healthCheckConfig and import google.golang.org/grpc/health, the client checks every backend through the standard gRPC Health service and temporarily stops sending requests to unhealthy ones (the server must also register the Health service).
  • How quickly DNS updates: DNS doesn't push changes, so when new Pods join, the client only finds out when it re-resolves. On the server side you can set a maximum connection age (MaxConnectionAge in keepalive.ServerParameters) so that clients periodically reconnect and re-resolve, and new instances gradually receive traffic.

If you don't want to handle these details in every service, a service mesh (such as Istio or Linkerd) does L7 per-request load balancing in a sidecar or node proxy, and the application only needs to connect to a regular Service name. This trades infrastructure complexity for application simplicity, and suits teams that already have quite a few services.

Common pitfalls

  • Hard-coding instance addresses in configuration: it breaks as soon as instances scale out or are redeployed. Use service names and let service discovery resolve them.
  • Liveness probes that check external dependencies: when the database blips, every instance is restarted because its liveness probe fails, which makes things worse. Put external dependencies only in the readiness probe.
  • The wrong shutdown order: stop serving first and deregister afterwards, and callers will send requests to an instance that has already shut down before their lists update. Deregister (or fail the readiness probe) first, wait, then shut down.
  • Running gRPC through a regular Service and assuming it is load balanced: HTTP/2's long-lived connections concentrate requests on a few Pods. Switch to a headless Service with round_robin, or use a service mesh.
  • Distributing stateful requests with hash % N: as soon as the instance count changes, almost every key is redistributed and every cache is invalidated. Use consistent hashing.
  • Consistent hashing without virtual nodes: the distribution is uneven and a few instances carry most of the keys.
  • Forgetting to release the count in least connections: if the count isn't decremented when a request errors or panics, that instance looks busy forever. Release it with defer.
  • A registry without high availability: when it goes down, new callers can't find any service. Use a mature product in cluster mode, and have callers keep the last known list as a fallback.
  • A TTL that is too short or too long: too short and network jitter causes misjudgements; too long and failed instances keep receiving traffic. Set it from the heartbeat interval and the acceptable detection time.

Key takeaways

  • Service discovery answers "which healthy instances exist", and load balancing answers "which one gets this request".
  • Server-side load balancing keeps the caller simplest; client-side load balancing saves a hop and can handle long-lived connections and content-based routing. A service mesh sits in between.
  • The core of a registry is registration, heartbeats, TTL and deregistration; health is inferred, and heartbeats should reflect the real ability to serve.
  • Health checks come in three kinds: heartbeat, active and passive; Kubernetes liveness and readiness probes serve different purposes and must not be mixed up.
  • Round-robin suits requests of similar cost, least connections suits requests with widely varying processing times, and consistent hashing suits situations that need affinity, with virtual nodes needed for an even distribution.
  • On Kubernetes, Services and cluster DNS already provide service discovery and L4 load balancing; gRPC needs a headless Service with round_robin to balance across every Pod.

Next

Callers can now find healthy instances and spread requests across them, but any network call will inevitably run into slowness, failures and temporary unavailability. The next article, “Timeouts, Retries and Circuit Breakers”, explains how to set sensible timeouts, when to retry, and how to use circuit breaking to stop one failing service from dragging down the whole system.