This resume is also available in English.

3. Splitting into microservices2 / 4 · All columns

Columns › 3. Splitting into microservices › Part 10

gRPC and Protocol Buffers in Go

gRPC 與 Protocol Buffers

Define service-to-service interfaces with a single .proto contract, generate type-safe Go code with protoc, implement unary and streaming calls, and learn deadlines, status codes, interceptors and the rules for evolving fields without breaking clients.

  • Go
  • gRPC
  • Protobuf
  • Microservices
In this article
  1. What this article solves
  2. Step 1: Design the proto contract
  3. Step 2: Generate the Go code
  4. Step 3: Implement the server
  5. Step 4: Implement the client: connections, deadlines and status codes
  6. Step 5: Interceptors: gRPC's middleware
  7. Step 6: Let the contract evolve: backward-compatibility rules
  8. Common pitfalls
  9. Key takeaways
  10. Next

In the previous article, “When to Split: Service Boundaries and Decomposition Strategies”, we reorganised the shop into three modules, order, payment and inventory, and made order talk to inventory only through the InventoryService interface. This article "moves that interface onto the network": once inventory becomes a separate service, how should order call it so that it gets type checking just like calling a local function, and can still handle timeouts and errors?

What this article solves

The most intuitive way for services to communicate is JSON over HTTP, which you can build with what you learned in “net/http and Middleware in Go”. But as the number of services grows, a few problems become more and more obvious:

  • The contract is scattered everywhere: what requests and responses look like is written only in documentation or in the other side's code. If the other side renames a field, you find out at runtime.
  • Every service rewrites serialisation and error handling: how status codes map, where error messages go, each service does it its own way.
  • No standard deadline propagation or streaming: you have to agree on headers yourselves and handle long-lived connections yourselves.

gRPC together with Protocol Buffers (Protobuf for short) standardises all of this in one go: a single .proto file describes the services and messages, and tools generate the code for both sides; transport uses HTTP/2, messages use a compact binary format; and deadlines, status codes, metadata and streaming all follow uniform rules.

This article works through the following in order:

  1. Design the .proto contract for the inventory service.
  2. Generate Go code with protoc plus protoc-gen-go and protoc-gen-go-grpc.
  3. Implement the server: a unary call and server streaming.
  4. Implement the client: grpc.NewClient, deadlines and status code handling.
  5. Use interceptors for logging, panic recovery and request ID propagation.
  6. Learn to change the contract without breaking old clients.

Tip: The Go code in this article depends on two third-party modules, google.golang.org/grpc and google.golang.org/protobuf, so it is presented as snippets. It uses the current APIs (for example grpc.NewClient, not the deprecated grpc.Dial).

Step 1: Design the proto contract

Following the series' project structure, contracts live under api/proto/, in directories by "service/version": api/proto/inventory/v1/inventory.proto. The version number goes in both the package name and the path, so that if an incompatible major change is ever truly needed, v1 and v2 can coexist for a while.

syntax = "proto3";

package shop.inventory.v1;

import "google/protobuf/timestamp.proto";

option go_package = "example.com/shop/api/proto/inventory/v1;inventoryv1";

// Inventory service: reserves stock for the order service, and lets clients subscribe to stock changes.
service InventoryService {
  // Unary call: one request, one response.
  rpc ReserveStock(ReserveStockRequest) returns (ReserveStockResponse);
  // Server streaming: after subscribing, the server keeps pushing stock changes.
  rpc WatchStock(WatchStockRequest) returns (stream StockEvent);
}

message LineItem {
  string sku = 1;
  int32 quantity = 2;
}

message ReserveStockRequest {
  string order_id = 1;
  repeated LineItem items = 2;
  // Field 3 used to be string warehouse and has been removed; both its number and name are reserved so they are never reused.
  reserved 3;
  reserved "warehouse";
}

message ReserveStockResponse {
  string reservation_id = 1;
  google.protobuf.Timestamp expires_at = 2;
}

message WatchStockRequest {
  repeated string skus = 1;
}

message StockEvent {
  string sku = 1;
  int32 available = 2;
  google.protobuf.Timestamp changed_at = 3;
}

The design considerations behind this file, one by one:

  • syntax = "proto3": currently the most widely used syntax. Newer Protobuf also supports the "editions" style, starting with edition = "2023", which turns the differences between proto2 and proto3 into individually configurable feature flags; for most new services proto3 is enough, and every toolchain supports it.
  • package shop.inventory.v1: Protobuf's own namespace, which appears in gRPC's full method names, e.g. /shop.inventory.v1.InventoryService/ReserveStock.
  • option go_package: the import path the generated Go code lives at, and its package name. The inventoryv1 after the semicolon is the package name, so that every version isn't called v1 and clashing with the others.
  • Every RPC has its own dedicated Request and Response messages: even if there is only one field now, don't use a bare string or a shared message. Dedicated messages let you add fields later without changing the method signature.
  • Field numbers are what the wire format actually recognises; names are only for humans. Numbers 1 to 15 take just one byte to encode, so keep them for the most frequently used fields.
  • reserved: for deleted fields, reserve both the number and the name, to stop anyone later reusing the same number and reading old data with a different meaning.
  • Use google.protobuf.Timestamp for time, not your own strings or integers, so there are no disputes about time zones and precision across languages.
  • Amounts: the shop always represents money as int64 "cents" (consistent with Order.Total); don't use double.

Step 2: Generate the Go code

You need three tools: the Protobuf compiler protoc, and two Go plugins. protoc-gen-go generates the message types, and protoc-gen-go-grpc generates the service's client and server interfaces.

# Install the two plugins (into $GOBIN, which must be on your PATH)
go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest

# Generate code from the project root, output next to the proto file
protoc -I api/proto \
  --go_out=api/proto --go_opt=paths=source_relative \
  --go-grpc_out=api/proto --go-grpc_opt=paths=source_relative \
  inventory/v1/inventory.proto

# This produces:
#   api/proto/inventory/v1/inventory.pb.go       message types
#   api/proto/inventory/v1/inventory_grpc.pb.go  Client/Server interfaces

# Add grpc and protobuf to go.mod
go get google.golang.org/grpc google.golang.org/protobuf

paths=source_relative makes the output location follow the .proto file's relative path, so the generated package's import path is example.com/shop/api/proto/inventory/v1, consistent with go_package.

inventory.proto The single contract protoc + go and go-grpc plugins inventory.pb.go Message types, serialisation inventory_grpc.pb.go Client and Server interfaces Both services generate code from the same proto, so type mismatches are caught at compile time

Figure: A single inventory.proto passes through protoc and two plugins to produce message types and service interfaces; the server and client both import the same generated code.

Should the generated code be committed to version control? Both approaches are common. Committing it means go build needs no extra tools, at the cost of remembering to regenerate; not committing it means adding a generation step in CI. Either way, it is a good idea to pin the commands above in a Makefile or go generate and lock the plugin versions, so different people don't generate different results. Some teams instead use tools like buf to manage generation settings and compatibility checks, which Step 6 mentions.

Step 3: Implement the server

protoc-gen-go-grpc generated an InventoryServiceServer interface; all we have to do is implement it and register it with a grpc.Server. Below is a trimmed-down version of cmd/inventory/main.go, containing the unary call ReserveStock, the server stream WatchStock, and two interceptors (explained in Step 5):

// Snippet: the inventory service's gRPC server
package main

import (
	"context"
	"log"
	"net"
	"sync"
	"time"

	"google.golang.org/grpc"
	"google.golang.org/grpc/codes"
	"google.golang.org/grpc/status"
	"google.golang.org/protobuf/types/known/timestamppb"

	inventoryv1 "example.com/shop/api/proto/inventory/v1"
)

type inventoryServer struct {
	// Embed Unimplemented: when the proto gains a method, the old implementation still compiles, and unimplemented methods return Unimplemented.
	inventoryv1.UnimplementedInventoryServiceServer

	mu    sync.Mutex
	stock map[string]int32
}

func (s *inventoryServer) ReserveStock(ctx context.Context, req *inventoryv1.ReserveStockRequest) (*inventoryv1.ReserveStockResponse, error) {
	if req.GetOrderId() == "" || len(req.GetItems()) == 0 {
		return nil, status.Error(codes.InvalidArgument, "order_id and items are required")
	}
	s.mu.Lock()
	defer s.mu.Unlock()
	for _, it := range req.GetItems() {
		if it.GetQuantity() <= 0 {
			return nil, status.Errorf(codes.InvalidArgument, "quantity of %s must be positive", it.GetSku())
		}
		if s.stock[it.GetSku()] < it.GetQuantity() {
			// A business "not enough" is FailedPrecondition; the caller should not retry blindly
			return nil, status.Errorf(codes.FailedPrecondition, "sku %s out of stock", it.GetSku())
		}
	}
	if err := ctx.Err(); err != nil { // if the caller has given up, don't deduct stock
		return nil, status.FromContextError(err).Err()
	}
	for _, it := range req.GetItems() {
		s.stock[it.GetSku()] -= it.GetQuantity()
	}
	return &inventoryv1.ReserveStockResponse{
		ReservationId: "rsv-" + req.GetOrderId(),
		ExpiresAt:     timestamppb.New(time.Now().Add(15 * time.Minute)),
	}, nil
}

func (s *inventoryServer) WatchStock(req *inventoryv1.WatchStockRequest, stream grpc.ServerStreamingServer[inventoryv1.StockEvent]) error {
	ticker := time.NewTicker(2 * time.Second)
	defer ticker.Stop()
	for {
		select {
		case <-stream.Context().Done(): // the client disconnected or timed out
			return status.FromContextError(stream.Context().Err()).Err()
		case <-ticker.C:
			s.mu.Lock()
			for _, sku := range req.GetSkus() {
				ev := &inventoryv1.StockEvent{Sku: sku, Available: s.stock[sku], ChangedAt: timestamppb.Now()}
				if err := stream.Send(ev); err != nil {
					s.mu.Unlock()
					return err
				}
			}
			s.mu.Unlock()
		}
	}
}

// A unary interceptor that logs each call's method, duration and status code.
func logUnary(ctx context.Context, req any, info *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (any, error) {
	start := time.Now()
	resp, err := handler(ctx, req)
	log.Printf("method=%s code=%s dur=%s", info.FullMethod, status.Code(err), time.Since(start))
	return resp, err
}

// Turns a panic into Internal, so the whole process doesn't crash.
func recoverUnary(ctx context.Context, req any, info *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (resp any, err error) {
	defer func() {
		if r := recover(); r != nil {
			log.Printf("panic in %s: %v", info.FullMethod, r)
			err = status.Error(codes.Internal, "internal error")
		}
	}()
	return handler(ctx, req)
}

func main() {
	lis, err := net.Listen("tcp", ":9003")
	if err != nil {
		log.Fatal(err)
	}
	srv := grpc.NewServer(
		grpc.ChainUnaryInterceptor(recoverUnary, logUnary), // run in order, from outermost to innermost
	)
	inventoryv1.RegisterInventoryServiceServer(srv, &inventoryServer{
		stock: map[string]int32{"SKU-1": 10, "SKU-2": 3},
	})
	log.Println("inventory gRPC listening on :9003")
	log.Fatal(srv.Serve(lis))
}

A few things worth noting:

  • Embedding UnimplementedInventoryServiceServer: this is the form grpc-go requires by default (embedded by value). When the proto later gains a new method, the old implementation still compiles, and calling the new method just gets an Unimplemented error rather than the whole service failing to compile.
  • Always return errors with status.Error / status.Errorf: return a plain error and the client only sees Unknown, with no way to tell whether to retry.
  • Read fields with getters: req.GetOrderId() is safe even when req is nil, which is the convention for Protobuf-generated code.
  • Check ctx: if the client has already given up (timed out or cancelled), don't do anything with side effects. status.FromContextError converts context.DeadlineExceeded into the corresponding gRPC status code. This is the principle from “Go context: Cancellation, Timeouts and Request-Scoped Values” applied to gRPC.
  • The streaming method's signature: the current generated code uses the generic type grpc.ServerStreamingServer[StockEvent]; Send sends one event, and stream.Context() is cancelled when the client disconnects.

gRPC defines four kinds of call, corresponding to where the stream keyword appears in the proto:

Client Server Unary Server streaming Client streaming Bidirectional streaming Plain arrows: requests; accent arrows: responses

Figure: A unary call is one question, one answer; server streaming is one request, many responses; client streaming is many requests, one response; and in bidirectional streaming both sides can keep sending.

Kind of callIn protoTypical use
Unaryrpc A(Req) returns (Resp)The vast majority of queries and commands, e.g. reserving stock
Server streamingrpc A(Req) returns (stream Resp)Subscribing to changes, returning large results in batches
Client streamingrpc A(stream Req) returns (Resp)Uploading lots of data, then returning a summary
Bidirectionalrpc A(stream Req) returns (stream Resp)Chat, real-time collaboration, long-running two-way sync

Prefer unary calls. Streaming is powerful, but load balancing, retries and debugging are all more complex: a long stream stays stuck to the same backend instance, and when the backend is redeployed you also have to handle reconnecting after the stream is interrupted.

Step 4: Implement the client: connections, deadlines and status codes

On the client side, the order service calls inventory. There are three key points: create the connection once, give every call a deadline, and decide what to do next based on the status code.

// Snippet: the order service calls the inventory service
package main

import (
	"context"
	"errors"
	"io"
	"log"
	"time"

	"google.golang.org/grpc"
	"google.golang.org/grpc/codes"
	"google.golang.org/grpc/credentials/insecure"
	"google.golang.org/grpc/status"

	inventoryv1 "example.com/shop/api/proto/inventory/v1"
)

func main() {
	// NewClient doesn't connect immediately; the connection is made on the first call. The whole process shares this one conn.
	conn, err := grpc.NewClient("dns:///inventory:9003",
		grpc.WithTransportCredentials(insecure.NewCredentials()), // use TLS in production
	)
	if err != nil {
		log.Fatal(err)
	}
	defer conn.Close()
	client := inventoryv1.NewInventoryServiceClient(conn)

	// Every call carries a deadline
	ctx, cancel := context.WithTimeout(context.Background(), 800*time.Millisecond)
	defer cancel()
	resp, err := client.ReserveStock(ctx, &inventoryv1.ReserveStockRequest{
		OrderId: "o-1001",
		Items:   []*inventoryv1.LineItem{{Sku: "SKU-1", Quantity: 2}},
	})
	switch status.Code(err) {
	case codes.OK:
		log.Printf("reserved %s until %s", resp.GetReservationId(), resp.GetExpiresAt().AsTime())
	case codes.FailedPrecondition:
		log.Printf("out of stock: %s", status.Convert(err).Message())
	case codes.DeadlineExceeded, codes.Unavailable:
		log.Printf("inventory temporarily unavailable: %v", err) // leave it to the retry policy
	default:
		log.Printf("reserve failed: %v", err)
	}

	// Server streaming: keep calling Recv until io.EOF or an error
	wctx, wcancel := context.WithTimeout(context.Background(), 10*time.Second)
	defer wcancel()
	stream, err := client.WatchStock(wctx, &inventoryv1.WatchStockRequest{Skus: []string{"SKU-1"}})
	if err != nil {
		log.Fatal(err)
	}
	for {
		ev, err := stream.Recv()
		if errors.Is(err, io.EOF) {
			break
		}
		if err != nil {
			log.Printf("watch ended: %v", status.Code(err))
			break
		}
		log.Printf("%s available=%d", ev.GetSku(), ev.GetAvailable())
	}
}

Connections: grpc.NewClient just creates a *grpc.ClientConn object without blocking until a connection succeeds; the real connection is made on the first call, and it reconnects automatically after a disconnect. A single ClientConn can hold several HTTP/2 connections, each running many requests at once, so the whole process should share one: create it in cmd/order/main.go and hand it to whatever needs it through dependency injection, rather than creating a new one per call. The target string dns:///inventory:9003 means resolving via DNS, which is closely tied to service discovery and is discussed in detail in “Service Discovery and Load Balancing for Go Microservices”.

Deadlines: gRPC has no timeout by default. A call without a deadline waits forever when the other side hangs, eventually dragging down the caller's goroutines and connections. The rule is simple: wrap every call with context.WithTimeout. Better still, gRPC puts the remaining time into the grpc-timeout header sent to the server, so the server-side ctx automatically carries the same deadline; if the server calls further downstream, it only has to pass the same ctx along, and the whole call chain shares one time budget.

API Gateway Budget: 1 s Order service 900ms left Inventory service 780ms left ctx ctx The grpc-timeout header carries the remaining time downstream If any hop times out, the whole chain gets DeadlineExceeded and stops wasting resources (numbers are illustrative only)

Figure: Each hop subtracts the time already spent and passes the remaining budget on; if any hop times out, the whole chain stops together.

Status codes: gRPC has a fixed set of status codes, which the client gets with status.Code(err). The most commonly used ones:

Status codeMeaningWhat the client should do
OKSuccess—
InvalidArgumentThe request itself is wrongDon't retry; fix the request
NotFoundThe resource doesn't existDon't retry
AlreadyExistsWhat you are creating already existsUsually treat as success (idempotent)
FailedPreconditionThe system state doesn't allow it, e.g. insufficient stockDon't retry blindly; report it to the user
PermissionDenied / UnauthenticatedNo permission / not logged inDon't retry
ResourceExhaustedQuota or rate limitTry again later
DeadlineExceededTimed outDepends on whether the operation is idempotent
UnavailableThe service is temporarily unavailableRetry after backing off
Internal / UnknownA server-side errorLog it and alert

Only when the server picks the right status code can the client make the right decision. The most common mistake is returning every failure as Internal, leaving the client unable to tell "business errors that shouldn't be retried" from "transient errors that can be". When and how to retry is discussed fully in “Timeouts, Retries and Circuit Breakers”.

Tip: DeadlineExceeded doesn't mean the server didn't execute. The request may already have successfully deducted stock, with the response just not getting back in time. So operations with side effects need an idempotency key, the subject of “Idempotency and Distributed Locks”.

Step 5: Interceptors: gRPC's middleware

In “net/http and Middleware in Go”, we added cross-cutting logic such as logging and recovery by wrapping http.Handler. gRPC's counterpart is the interceptor, which comes in four kinds: server-side and client-side, unary and streaming.

The server in Step 3 already uses two unary interceptors: recoverUnary turns a panic into an Internal error, and logUnary logs the method name, duration and status code. grpc.ChainUnaryInterceptor(recoverUnary, logUnary) runs them in order from outermost to innermost, like the layers of an onion from the outside in.

Clients can have interceptors too. The example below puts the request ID from ctx into gRPC metadata (the equivalent of HTTP headers), so downstream services' logs can be matched to the same request; on the server side it is read back with metadata.FromIncomingContext:

// Snippet: a client interceptor that puts the request ID into metadata for downstream services
package middleware

import (
	"context"

	"google.golang.org/grpc"
	"google.golang.org/grpc/metadata"
)

type requestIDKey struct{}

func RequestIDUnary(ctx context.Context, method string, req, reply any,
	cc *grpc.ClientConn, invoker grpc.UnaryInvoker, opts ...grpc.CallOption) error {
	if id, ok := ctx.Value(requestIDKey{}).(string); ok {
		ctx = metadata.AppendToOutgoingContext(ctx, "x-request-id", id)
	}
	return invoker(ctx, method, req, reply, cc, opts...)
}

// Read it back on the server side
func RequestIDFrom(ctx context.Context) string {
	if md, ok := metadata.FromIncomingContext(ctx); ok {
		if v := md.Get("x-request-id"); len(v) > 0 {
			return v[0]
		}
	}
	return ""
}

Attach it with grpc.WithChainUnaryInterceptor(RequestIDUnary) when creating the client connection. In practice, you usually don't need to write your own interceptors for logs, metrics and distributed tracing: OpenTelemetry provides a ready-made stats handler for grpc-go, introduced in “Observability in Go: Logs, Metrics and Tracing”. When writing your own interceptors, remember: metadata keys are always lowercase; don't put sensitive data in metadata; and errors inside an interceptor should also be returned as status errors.

Besides interceptors, a server usually registers two standard services before going live: health checking (grpc.health.v1.Health, used by load balancers and Kubernetes probes) and reflection (which lets tools like grpcurl list services, handy for debugging; whether to enable it in production depends on your security requirements).

Step 6: Let the contract evolve: backward-compatibility rules

Once services are deployed independently, old and new versions are bound to be live at the same time: inventory has been upgraded while an old order is still running, or the other way round. Protobuf's design gives you a chance to get this right, provided you follow the rules:

ChangeSafe?Notes
Add a field (with a new number)SafeOld code skips fields it doesn't recognise; new code reading old data gets zero values
Add an RPC methodSafeAn old server returns Unimplemented
Delete a field and reserved its number and nameSafeThat field in old data is skipped
Rename a fieldSafe on the wire, but breaks code and the JSON mappingAvoid where possible
Change a field numberUnsafeEquivalent to deleting the old field and adding a new one
Change a field typeMostly unsafeOnly a few compatible types are exceptions; don't rely on them
Reuse a deleted numberVery dangerousOld data is interpreted as something completely different
Rename the package or serviceUnsafeThe full method name changes, and old clients can't find it

There is also a semantic trap: proto3 scalar fields make no distinction between "not set" and "set to the zero value", so when quantity is 0 you can't tell whether the other side didn't send it or really sent 0. If the business needs to distinguish them, add optional before the field; the generated Go field becomes a pointer, and whether it is nil tells you if it was set.

Enums need care too: the first value must be 0, and it is recommended to name it XXX_UNSPECIFIED to mean "not set", e.g. ORDER_STATUS_UNSPECIFIED = 0, followed by ORDER_STATUS_PENDING, ORDER_STATUS_PAID and ORDER_STATUS_CANCELLED. That way, when old code receives a newly added enum value, at least it won't mistake it for some meaningful status.

Handing these rules to a tool to check is the most reliable approach. buf breaking compares the current proto with the version on the main branch and blocks incompatible changes directly in CI. When an incompatible major change is truly unavoidable, add a v2 package, let the two versions coexist, and remove v1 once every client has migrated.

Common pitfalls

  • Not setting a deadline: by default gRPC waits forever, so as soon as a downstream service hangs, upstream goroutines pile up. Every call needs a timeout.
  • Creating a new connection for every call: grpc.NewClient should be called once at startup and shared by the whole process; creating new ones wastes TCP and TLS handshakes and defeats load balancing.
  • Returning a plain error: without wrapping it in status.Error, the client only sees Unknown and cannot decide whether to retry.
  • Reusing field numbers: delete a field without reserved, and when someone later puts something else under the same number, old data is read as the wrong value.
  • Passing proto messages around as domain models: generated types carry internal state and must not be copied directly; and letting them seep into business logic means any contract change ripples through all your code. Convert them into your own domain types (such as Order) in the handler layer.
  • Overusing streaming: long streams stick to the same instance, causing uneven load and mass interruptions during deployments. If a unary call will do, use a unary call.
  • Using insecure.NewCredentials() in production: TLS or mTLS is recommended even inside the cluster, or leave it to a service mesh.
  • Messages that are too large: the default receive limit is about 4 MB. When you need to send a lot of data, switch to streaming in batches, or send the address of an object in object storage, rather than simply raising the limit.

Key takeaways

  • gRPC + Protobuf defines services with a single .proto contract, tools generate type-safe code for both sides, and transport uses HTTP/2.
  • Put contracts in api/proto/<service>/<version>/, give every RPC its own Request/Response, use Timestamp for time and int64 cents for money.
  • Generate code with protoc plus protoc-gen-go and protoc-gen-go-grpc, and pin the commands and plugin versions.
  • Servers embed Unimplemented...Server, return errors via status with the right status code, and check ctx before doing anything with side effects.
  • Clients create one connection with grpc.NewClient and share it across the process; every call carries a deadline, and the status code decides whether to retry.
  • Interceptors are gRPC's middleware, suited to logging, panic recovery, request IDs and tracing.
  • Evolve contracts by only adding, never renumbering, and reserved anything you delete; check with buf breaking in CI.

Next

Services can now talk to each other over gRPC, but browsers and external partners won't call internal gRPC services directly, and authentication, rate limiting and request IDs shouldn't be written separately in every service. The next article, “Building an API Gateway in Go: Routing, Authentication and Rate Limiting”, uses the standard library to write a small gateway that handles all of this centrally at the system's entry point.