Columns › 3. Splitting into microservices › Part 12
Service Discovery and Load Balancing for Go Microservices
服務發現與負載平衡
When services run as many instances whose IPs keep changing, how does a caller find the healthy ones? This article compares client-side and server-side load balancing, builds a heartbeat-based registry and round-robin, least-connections and consistent-hashing balancers in Go, and explains DNS-based discovery in Kubernetes.
In this article
- What this article solves
- Step 1: Client-side or server-side load balancing
- Step 2: The registry and heartbeats
- Step 3: Active and passive health checks
- Step 4: Three load-balancing algorithms
- Step 5: Why consistent hashing reduces movement
- Step 6: Service discovery in Kubernetes: Services and DNS
- Step 7: gRPC's long-lived connection problem and client-side load balancing
- Common pitfalls
- Key takeaways
- Next
In the previous article, “Building an API Gateway in Go: Routing, Authentication and Rate Limiting”, we funnelled all external traffic into api-gateway, but its routing table hard-codes the addresses of order, payment and inventory in the program. This article tackles that problem: when every service runs several instances, and instances scale out, scale in and get redeployed, how does a caller always know "which healthy instances exist right now", and how does it spread requests evenly across them?
What this article solves
In the monolith era, calling another module was just a function call; there was no such thing as "can't find the other side". Once you split into services, before order can call inventory it has to answer two questions:
- Service discovery: which instances of
inventoryexist right now? What are their addresses? Which of them are healthy? - Load balancing: which one should this request go to?
The two are closely related but worth thinking about separately. Service discovery provides a "candidate list", and load balancing picks one from the list. The list may come from a registry, DNS or the platform's API; the picking may use round-robin, least connections or consistent hashing; and the picking may happen at the caller or in a load balancer in between.
This article will, in order:
- Compare client-side and server-side load balancing.
- Write a minimal registry in Go with heartbeats and a TTL, to understand what health checking really is.
- Implement three common load-balancing algorithms and observe the effect of consistent hashing when scaling out.
- See how Kubernetes builds all this in with Services and DNS, and what to watch out for with gRPC.
Step 1: Client-side or server-side load balancing
There are two basic architectures for where the load-balancing decision is made:
Figure: on the left, the caller fetches the instance list from the registry, picks one itself and connects directly; on the right, the caller knows only one address, and a load balancer in the middle picks the backend.
| Aspect | Client-side load balancing | Server-side load balancing |
|---|---|---|
| Who picks the instance | A library in the caller | A load balancer in the middle (L4 or L7) |
| Network path | Connects directly to the instance, one hop fewer | Passes through an extra load balancer |
| Caller complexity | High: must integrate service discovery, health state and the algorithm | Low: needs only one fixed address |
| Multi-language support | Every language needs a matching library | Language-agnostic |
| Single point of failure | No central node (apart from the registry) | The load balancer itself must be highly available |
| Effect on long-lived connections | Good: can open a connection to each instance | L4 can only balance "connections", not the requests inside a connection |
| Typical examples | gRPC's built-in load balancing, service mesh sidecars | Cloud load balancers, NGINX, Kubernetes Services |
There is also a common middle ground: the service mesh. It places a sidecar proxy next to each service; to the application it looks like server-side load balancing (it only connects to the local proxy), but in reality it does client-side load balancing on every node.
The selection rule can be simple: with no special requirements, use the server-side load balancing the platform provides (such as a Kubernetes Service), which is the least effort; consider client-side load balancing only when you use a long-lived-connection protocol such as gRPC, or need to route by request content (for example with consistent hashing).
Step 2: The registry and heartbeats
Whichever architecture you choose, someone has to maintain the list of "which healthy instances exist right now". The most direct approach is a registry; common products include Consul, etcd, ZooKeeper and Nacos. Their core mechanism is actually quite simple:
- After an instance starts, it registers its service name and address with the registry.
- The instance sends regular heartbeats meaning "I'm still alive".
- If the registry receives no heartbeat from an instance within the TTL (time to live), it treats the instance as unhealthy and stops returning it in query results.
- When an instance shuts down gracefully, it deregisters itself rather than waiting for the TTL to expire.
- The caller queries the list of healthy instances for a service, usually caching it locally for a while or subscribing to change notifications.
Figure: instances register with the registry and keep sending heartbeats; an instance whose heartbeats stop is taken out of service. After querying the healthy list, the caller calls the instances directly.
Below is a minimal registry written with the standard library. For simplicity, registration and heartbeat are the same action: each PUT /instances renews the lease once; GET /services/{service} returns only instances that have sent a heartbeat within the TTL; and DELETE lets an instance take itself out of service on graceful shutdown. Routing uses the methods and path parameters that http.ServeMux supports from Go 1.22:
// A minimal service registry: instances register and renew with regular heartbeats, and drop out automatically when heartbeats stop.
package main
import (
"encoding/json"
"log"
"net/http"
"sort"
"sync"
"time"
)
const ttl = 10 * time.Second // illustrative: no heartbeat for over 10 seconds means unhealthy
type instance struct {
Service string `json:"service"`
Addr string `json:"addr"`
LastBeat time.Time `json:"last_beat"`
}
type registry struct {
mu sync.RWMutex
byID map[string]*instance // key = service + "|" + addr
}
func (r *registry) beat(service, addr string, now time.Time) {
r.mu.Lock()
defer r.mu.Unlock()
id := service + "|" + addr
if in, ok := r.byID[id]; ok {
in.LastBeat = now
return
}
r.byID[id] = &instance{Service: service, Addr: addr, LastBeat: now}
}
func (r *registry) deregister(service, addr string) {
r.mu.Lock()
defer r.mu.Unlock()
delete(r.byID, service+"|"+addr)
}
// healthy returns only instances with a heartbeat within the TTL.
func (r *registry) healthy(service string, now time.Time) []string {
r.mu.RLock()
defer r.mu.RUnlock()
var addrs []string
for _, in := range r.byID {
if in.Service == service && now.Sub(in.LastBeat) < ttl {
addrs = append(addrs, in.Addr)
}
}
sort.Strings(addrs)
return addrs
}
// reap periodically clears records that expired long ago, so the map doesn't only ever grow.
func (r *registry) reap(now time.Time) {
r.mu.Lock()
defer r.mu.Unlock()
for id, in := range r.byID {
if now.Sub(in.LastBeat) > 3*ttl {
delete(r.byID, id)
}
}
}
type beatReq struct {
Service string `json:"service"`
Addr string `json:"addr"`
}
func main() {
reg := ®istry{byID: map[string]*instance{}}
go func() {
for now := range time.Tick(ttl) {
reg.reap(now)
}
}()
mux := http.NewServeMux()
// Registration and heartbeat are the same action: each PUT renews the lease once
mux.HandleFunc("PUT /instances", func(w http.ResponseWriter, r *http.Request) {
var req beatReq
if err := json.NewDecoder(http.MaxBytesReader(w, r.Body, 1<<10)).Decode(&req); err != nil ||
req.Service == "" || req.Addr == "" {
http.Error(w, "service and addr are required", http.StatusBadRequest)
return
}
reg.beat(req.Service, req.Addr, time.Now())
w.WriteHeader(http.StatusNoContent)
})
// Take yourself out of service on graceful shutdown instead of waiting for the TTL to expire
mux.HandleFunc("DELETE /instances/{service}/{addr}", func(w http.ResponseWriter, r *http.Request) {
reg.deregister(r.PathValue("service"), r.PathValue("addr"))
w.WriteHeader(http.StatusNoContent)
})
mux.HandleFunc("GET /services/{service}", func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
json.NewEncoder(w).Encode(reg.healthy(r.PathValue("service"), time.Now()))
})
log.Println("registry listening on :8500")
log.Fatal(http.ListenAndServe(":8500", mux))
}A few design points:
- Health is "inferred": the registry doesn't know whether an instance can really handle requests, only whether it sends heartbeats on time. So heartbeats should be sent by the logic inside the instance that genuinely represents health, for example only when the database connection pool is confirmed usable, rather than by a separate goroutine that sends them unconditionally.
- The TTL trade-off: too short a TTL and minor network jitter wrongly takes instances out of service; too long and callers learn that an instance has died only long after the fact. A common setting is a heartbeat interval of about a third of the TTL, which tolerates occasionally missing one or two.
- Separate read and write locking: queries are far more frequent than heartbeats, so we use
sync.RWMutex, a technique mentioned in “Goroutines, Channels and Concurrency Patterns in Go”. - Clearing expired records:
reapperiodically deletes records that haven't had a heartbeat for a long time, so the map doesn't only ever grow.
On the instance side, the job is to start sending regular PUTs once start-up completes, and in the shutdown flow from “Configuration and Graceful Shutdown in Go”, first DELETE itself, wait a little while for callers to update their lists, and only then stop accepting new requests. Do it in the reverse order and callers will send requests to an instance that has already shut down before their lists are updated.
Tip: this registry is itself a single point of failure, and its data lives only in memory. Production registries replicate data across several nodes with a consensus algorithm such as Raft. Once you understand the principle, use a mature product in production, or simply use the mechanism built into the platform (Step 6).
Step 3: Active and passive health checks
A heartbeat is a health signal that "the instance reports itself". The other approach is an external active check, and the two are often used together:
| Method | Who runs it | How | Advantage | Disadvantage |
|---|---|---|---|---|
| Heartbeat / TTL | The instance itself | Periodically renews its lease with the registry | Light load on the registry | "Alive" doesn't mean "able to serve" |
| Active check | The registry or the load balancer | Periodically calls /healthz or the gRPC Health service | The real state as seen from outside | Check traffic grows with the number of instances |
| Passive check | The caller | Watches the error rate of real requests and temporarily ejects instances after consecutive failures | Fastest reaction, no extra traffic | Needs failed requests to happen first |
In Kubernetes, active checks are performed by the kubelet through probes, and there are two kinds of probe with different meanings:
- Liveness probe: failure means the process is stuck and needs restarting. The check must be very lightweight and must not depend on external resources; otherwise a blip in the database restarts every instance.
- Readiness probe: failure means "don't send me traffic for now", for example because the instance is still warming up or the database it depends on is temporarily unreachable. The instance is removed from the Service's endpoint list but is not restarted.
A passive check is the same concept as the circuit breaker in “Timeouts, Retries and Circuit Breakers”: instances that fail repeatedly are temporarily removed from the list, and after a while a small number of requests are let through to probe them.
Step 4: Three load-balancing algorithms
With a list of healthy instances, the next step is to pick one. The program below implements the three most commonly used algorithms, and you can run it directly to observe the results:
// Three load-balancing algorithms: round-robin, least connections and consistent hashing (with virtual nodes).
package main
import (
"fmt"
"hash/crc32"
"slices"
"strconv"
"sync"
"sync/atomic"
)
// ---------- Round-robin ----------
type RoundRobin struct {
addrs []string
next atomic.Uint64
}
func (rr *RoundRobin) Pick() string {
n := rr.next.Add(1) - 1
return rr.addrs[n%uint64(len(rr.addrs))]
}
// ---------- Least connections ----------
type LeastConn struct {
mu sync.Mutex
active map[string]int
}
// Acquire picks the instance with the fewest in-flight requests; always call the returned release when done.
func (lc *LeastConn) Acquire() (addr string, release func()) {
lc.mu.Lock()
defer lc.mu.Unlock()
best := -1
for a, n := range lc.active {
// Break ties by name so the result is predictable
if best == -1 || n < best || (n == best && a < addr) {
addr, best = a, n
}
}
lc.active[addr]++
return addr, func() {
lc.mu.Lock()
lc.active[addr]--
lc.mu.Unlock()
}
}
// ---------- Consistent hashing ----------
type Ring struct {
replicas int // virtual nodes per instance
hashes []uint32 // sorted positions on the ring
owner map[uint32]string // position → instance
}
func NewRing(replicas int, addrs ...string) *Ring {
r := &Ring{replicas: replicas, owner: map[uint32]string{}}
for _, a := range addrs {
r.Add(a)
}
return r
}
func (r *Ring) Add(addr string) {
for i := range r.replicas {
h := crc32.ChecksumIEEE([]byte(addr + "#" + strconv.Itoa(i)))
r.hashes = append(r.hashes, h)
r.owner[h] = addr
}
slices.Sort(r.hashes)
}
// Get walks clockwise from the key's position to the first virtual node.
func (r *Ring) Get(key string) string {
h := crc32.ChecksumIEEE([]byte(key))
i, _ := slices.BinarySearch(r.hashes, h)
if i == len(r.hashes) {
i = 0 // wrap around to the start of the ring
}
return r.owner[r.hashes[i]]
}
func main() {
addrs := []string{"inventory-a:9003", "inventory-b:9003", "inventory-c:9003"}
rr := &RoundRobin{addrs: addrs}
for range 4 {
fmt.Print(rr.Pick(), " ")
}
fmt.Println()
lc := &LeastConn{active: map[string]int{}}
for _, a := range addrs {
lc.active[a] = 0
}
a1, done1 := lc.Acquire()
a2, _ := lc.Acquire()
done1() // the first request has finished
a3, _ := lc.Acquire()
fmt.Println("least-conn:", a1, a2, a3)
// Consistent hashing: after adding a fourth instance, only some keys change owner
ring := NewRing(100, addrs...)
before := map[string]string{}
for i := range 10000 {
k := "SKU-" + strconv.Itoa(i)
before[k] = ring.Get(k)
}
ring.Add("inventory-d:9003")
moved := 0
for k, old := range before {
if ring.Get(k) != old {
moved++
}
}
fmt.Printf("consistent hash: %.1f%% of keys moved\n", float64(moved)/100)
}Running it produces output like this (the consistent-hashing percentage varies slightly with the hash function and the number of virtual nodes; the figure is only illustrative):
inventory-a:9003 inventory-b:9003 inventory-c:9003 inventory-a:9003
least-conn: inventory-a:9003 inventory-b:9003 inventory-a:9003
consistent hash: 22.0% of keys movedRound-robin: take turns in order, incrementing a counter with atomic.Uint64, no lock needed. When every request costs about the same and every instance has the same specification, round-robin is simple and effective. Its variant, "weighted round-robin", gives larger instances more requests.
Least connections: pick the instance with the fewest in-flight requests. When processing times vary widely (for example some queries take 5 milliseconds and others 2 seconds), round-robin can let one instance happen to pile up a heap of slow requests, while least connections automatically avoids busy instances. One implementation note: every pick must be paired with a release that is always called when the request finishes (usually with defer), otherwise the count only ever goes up. With many instances, scanning the whole list also gets more expensive; a common compromise is "pick two at random and choose the less busy one" (the power of two choices), which performs close to least connections at a much lower cost.
Consistent hashing: the same key always lands on the same instance, which suits situations that need "affinity", for example when each instance caches inventory data for certain SKUs locally and you want all requests for the same SKU to go to the same instance to improve the cache hit rate.
| Algorithm | Suitable for | Watch out for |
|---|---|---|
| Round-robin | Requests of similar cost, instances of the same specification | Slow requests may concentrate on one instance |
| Weighted round-robin | Instances of different specifications | Weights must be adjusted with the specification |
| Least connections | Widely varying processing times | Needs to track in-flight request counts |
| Power of two choices | Many instances, a cheap approximation of least connections | Needs load information |
| Consistent hashing | Local caches, stateful sharding | Hot keys can overload a single instance |
Step 5: Why consistent hashing reduces movement
If you distribute keys with the most obvious hash(key) % N, then when the number of instances goes from 3 to 4, almost every key lands somewhere new, which is the same as invalidating every cache at once, and the backend database suddenly takes all the traffic. Consistent hashing exists to solve exactly this problem.
The approach is to imagine the range of hash values as a ring: each instance is placed at some position on the ring according to the hash of its name; each key also computes a position, and the first instance found walking clockwise from there is its owner.
Figure: a key walks clockwise from its position on the ring to the first node; when a new node D is added, only the keys between C and D are taken over by D, and the other keys stay put.
When a new instance is added, it only "takes over" the adjacent segment of the ring, and other keys are completely unaffected. Ideally, going from N instances to N+1, only about 1/(N+1) of the keys need to move; in the example, going from 3 to 4 instances, the theoretical figure is 25%, and the actual result is close to that.
replicas in the example is the number of virtual nodes. If each instance places only one point on the ring, the points are likely to be unevenly distributed and some instance ends up responsible for a particularly large segment. Placing tens to hundreds of virtual nodes per instance makes the distribution much more even; and when an instance goes away, its range is spread across several instances rather than all landing on the next neighbour.
Watch out for hot keys with consistent hashing: if one SKU is in a big promotion, all its requests land on the same instance. Common remedies are to spread hot keys out further, or to switch to "consistent hashing with bounded loads", where an instance whose load exceeds the average by a certain proportion passes requests on to the next node.
Step 6: Service discovery in Kubernetes: Services and DNS
If your services run on Kubernetes, the platform has already done most of the work above:
- Pods get scheduled, restarted and change IPs, so you should never remember a Pod's IP directly.
- A Service selects a group of Pods by label and provides a stable virtual IP (ClusterIP) and DNS name. Only Pods that pass their readiness probe appear in the Service's endpoints (EndpointSlice), which is a built-in registry plus health checking.
- Cluster DNS (usually CoreDNS) lets you find a Service by name. Within the same namespace, plain
inventoryresolves; the full name isinventory.shop.svc.cluster.local. - kube-proxy (or an alternative network plugin) distributes connections sent to the ClusterIP among the Pods behind it on every node, which is a form of L4 server-side load balancing.
# A regular Service: a stable virtual IP, with kube-proxy distributing at the connection level
apiVersion: v1
kind: Service
metadata:
name: inventory
namespace: shop
spec:
selector:
app: inventory
ports:
- name: grpc
port: 9003
targetPort: 9003
---
# A headless Service: no virtual IP; DNS returns the IP of every ready Pod directly
apiVersion: v1
kind: Service
metadata:
name: inventory-headless
namespace: shop
spec:
clusterIP: None
selector:
app: inventory
ports:
- name: grpc
port: 9003
targetPort: 9003For ordinary HTTP/1.1 services, a regular Service is enough. This is also why, on Kubernetes, the gateway's routing table from the previous article can simply say http://order:8080: DNS resolves the name and the Service handles load balancing. The complete deployment configuration is demonstrated in “Deploying One Go Service with Docker and Kubernetes” and “Deploying a Microservice System on Kubernetes”.
Step 7: gRPC's long-lived connection problem and client-side load balancing
A regular Service has an often-overlooked trap: it balances "connections", not "requests". gRPC is built on HTTP/2, where one connection can carry a large number of concurrent requests and stays open for a long time. The result: each order instance establishes a connection to just one of the inventory Pods and then sends it every request; scale inventory out to 10 Pods and the 9 new ones may get almost no traffic.
The fix is to let the gRPC client do load balancing itself: get every Pod's IP through a headless Service, open a connection to each IP, then distribute requests round-robin. grpc-go has this built in; you only need to specify DNS resolution and the round_robin policy when creating the connection:
// Snippet: gRPC client-side load balancing (DNS resolves several IPs, round-robin distribution, with health checks)
package main
import (
"log"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials/insecure"
_ "google.golang.org/grpc/health" // enable client-side health checking
inventoryv1 "example.com/shop/api/proto/inventory/v1"
)
const serviceConfig = `{
"loadBalancingConfig": [{"round_robin": {}}],
"healthCheckConfig": {"serviceName": ""}
}`
func main() {
// The headless Service makes DNS return each Pod's IP, and round_robin opens a connection to each IP
conn, err := grpc.NewClient(
"dns:///inventory-headless.shop.svc.cluster.local:9003",
grpc.WithTransportCredentials(insecure.NewCredentials()),
grpc.WithDefaultServiceConfig(serviceConfig),
)
if err != nil {
log.Fatal(err)
}
defer conn.Close()
inventory := inventoryv1.NewInventoryServiceClient(conn)
_ = inventory // handed to order's service
}Notes:
- The
dns:///target: grpc-go resolves the name with DNS and re-resolves when connections run into problems. The headless Service makes DNS return the IP of every ready Pod. round_robin: grpc-go's default policy ispick_first, which connects only to the first address; only by switching toround_robindoes it create a subconnection to every address and use them in turn.- Health checks: once you set
healthCheckConfigand importgoogle.golang.org/grpc/health, the client checks every backend through the standard gRPC Health service and temporarily stops sending requests to unhealthy ones (the server must also register the Health service). - How quickly DNS updates: DNS doesn't push changes, so when new Pods join, the client only finds out when it re-resolves. On the server side you can set a maximum connection age (
MaxConnectionAgeinkeepalive.ServerParameters) so that clients periodically reconnect and re-resolve, and new instances gradually receive traffic.
If you don't want to handle these details in every service, a service mesh (such as Istio or Linkerd) does L7 per-request load balancing in a sidecar or node proxy, and the application only needs to connect to a regular Service name. This trades infrastructure complexity for application simplicity, and suits teams that already have quite a few services.
Common pitfalls
- Hard-coding instance addresses in configuration: it breaks as soon as instances scale out or are redeployed. Use service names and let service discovery resolve them.
- Liveness probes that check external dependencies: when the database blips, every instance is restarted because its liveness probe fails, which makes things worse. Put external dependencies only in the readiness probe.
- The wrong shutdown order: stop serving first and deregister afterwards, and callers will send requests to an instance that has already shut down before their lists update. Deregister (or fail the readiness probe) first, wait, then shut down.
- Running gRPC through a regular Service and assuming it is load balanced: HTTP/2's long-lived connections concentrate requests on a few Pods. Switch to a headless Service with
round_robin, or use a service mesh. - Distributing stateful requests with
hash % N: as soon as the instance count changes, almost every key is redistributed and every cache is invalidated. Use consistent hashing. - Consistent hashing without virtual nodes: the distribution is uneven and a few instances carry most of the keys.
- Forgetting to release the count in least connections: if the count isn't decremented when a request errors or panics, that instance looks busy forever. Release it with
defer. - A registry without high availability: when it goes down, new callers can't find any service. Use a mature product in cluster mode, and have callers keep the last known list as a fallback.
- A TTL that is too short or too long: too short and network jitter causes misjudgements; too long and failed instances keep receiving traffic. Set it from the heartbeat interval and the acceptable detection time.
Key takeaways
- Service discovery answers "which healthy instances exist", and load balancing answers "which one gets this request".
- Server-side load balancing keeps the caller simplest; client-side load balancing saves a hop and can handle long-lived connections and content-based routing. A service mesh sits in between.
- The core of a registry is registration, heartbeats, TTL and deregistration; health is inferred, and heartbeats should reflect the real ability to serve.
- Health checks come in three kinds: heartbeat, active and passive; Kubernetes liveness and readiness probes serve different purposes and must not be mixed up.
- Round-robin suits requests of similar cost, least connections suits requests with widely varying processing times, and consistent hashing suits situations that need affinity, with virtual nodes needed for an even distribution.
- On Kubernetes, Services and cluster DNS already provide service discovery and L4 load balancing; gRPC needs a headless Service with
round_robinto balance across every Pod.
Next
Callers can now find healthy instances and spread requests across them, but any network call will inevitably run into slowness, failures and temporary unavailability. The next article, “Timeouts, Retries and Circuit Breakers”, explains how to set sensible timeouts, when to retry, and how to use circuit breaking to stop one failing service from dragging down the whole system.