Columns › 3. Splitting into microservices › Part 10
gRPC and Protocol Buffers in Go
gRPC 與 Protocol Buffers
Define service-to-service interfaces with a single .proto contract, generate type-safe Go code with protoc, implement unary and streaming calls, and learn deadlines, status codes, interceptors and the rules for evolving fields without breaking clients.
In this article
- What this article solves
- Step 1: Design the proto contract
- Step 2: Generate the Go code
- Step 3: Implement the server
- Step 4: Implement the client: connections, deadlines and status codes
- Step 5: Interceptors: gRPC's middleware
- Step 6: Let the contract evolve: backward-compatibility rules
- Common pitfalls
- Key takeaways
- Next
In the previous article, “When to Split: Service Boundaries and Decomposition Strategies”, we reorganised the shop into three modules, order, payment and inventory, and made order talk to inventory only through the InventoryService interface. This article "moves that interface onto the network": once inventory becomes a separate service, how should order call it so that it gets type checking just like calling a local function, and can still handle timeouts and errors?
What this article solves
The most intuitive way for services to communicate is JSON over HTTP, which you can build with what you learned in “net/http and Middleware in Go”. But as the number of services grows, a few problems become more and more obvious:
- The contract is scattered everywhere: what requests and responses look like is written only in documentation or in the other side's code. If the other side renames a field, you find out at runtime.
- Every service rewrites serialisation and error handling: how status codes map, where error messages go, each service does it its own way.
- No standard deadline propagation or streaming: you have to agree on headers yourselves and handle long-lived connections yourselves.
gRPC together with Protocol Buffers (Protobuf for short) standardises all of this in one go: a single .proto file describes the services and messages, and tools generate the code for both sides; transport uses HTTP/2, messages use a compact binary format; and deadlines, status codes, metadata and streaming all follow uniform rules.
This article works through the following in order:
- Design the
.protocontract for theinventoryservice. - Generate Go code with
protocplusprotoc-gen-goandprotoc-gen-go-grpc. - Implement the server: a unary call and server streaming.
- Implement the client:
grpc.NewClient, deadlines and status code handling. - Use interceptors for logging, panic recovery and request ID propagation.
- Learn to change the contract without breaking old clients.
Tip: The Go code in this article depends on two third-party modules,
google.golang.org/grpcandgoogle.golang.org/protobuf, so it is presented as snippets. It uses the current APIs (for examplegrpc.NewClient, not the deprecatedgrpc.Dial).
Step 1: Design the proto contract
Following the series' project structure, contracts live under api/proto/, in directories by "service/version": api/proto/inventory/v1/inventory.proto. The version number goes in both the package name and the path, so that if an incompatible major change is ever truly needed, v1 and v2 can coexist for a while.
syntax = "proto3";
package shop.inventory.v1;
import "google/protobuf/timestamp.proto";
option go_package = "example.com/shop/api/proto/inventory/v1;inventoryv1";
// Inventory service: reserves stock for the order service, and lets clients subscribe to stock changes.
service InventoryService {
// Unary call: one request, one response.
rpc ReserveStock(ReserveStockRequest) returns (ReserveStockResponse);
// Server streaming: after subscribing, the server keeps pushing stock changes.
rpc WatchStock(WatchStockRequest) returns (stream StockEvent);
}
message LineItem {
string sku = 1;
int32 quantity = 2;
}
message ReserveStockRequest {
string order_id = 1;
repeated LineItem items = 2;
// Field 3 used to be string warehouse and has been removed; both its number and name are reserved so they are never reused.
reserved 3;
reserved "warehouse";
}
message ReserveStockResponse {
string reservation_id = 1;
google.protobuf.Timestamp expires_at = 2;
}
message WatchStockRequest {
repeated string skus = 1;
}
message StockEvent {
string sku = 1;
int32 available = 2;
google.protobuf.Timestamp changed_at = 3;
}The design considerations behind this file, one by one:
syntax = "proto3": currently the most widely used syntax. Newer Protobuf also supports the "editions" style, starting withedition = "2023", which turns the differences between proto2 and proto3 into individually configurable feature flags; for most new services proto3 is enough, and every toolchain supports it.package shop.inventory.v1: Protobuf's own namespace, which appears in gRPC's full method names, e.g./shop.inventory.v1.InventoryService/ReserveStock.option go_package: the import path the generated Go code lives at, and its package name. Theinventoryv1after the semicolon is the package name, so that every version isn't calledv1and clashing with the others.- Every RPC has its own dedicated Request and Response messages: even if there is only one field now, don't use a bare
stringor a shared message. Dedicated messages let you add fields later without changing the method signature. - Field numbers are what the wire format actually recognises; names are only for humans. Numbers 1 to 15 take just one byte to encode, so keep them for the most frequently used fields.
reserved: for deleted fields, reserve both the number and the name, to stop anyone later reusing the same number and reading old data with a different meaning.- Use
google.protobuf.Timestampfor time, not your own strings or integers, so there are no disputes about time zones and precision across languages. - Amounts: the shop always represents money as
int64"cents" (consistent withOrder.Total); don't usedouble.
Step 2: Generate the Go code
You need three tools: the Protobuf compiler protoc, and two Go plugins. protoc-gen-go generates the message types, and protoc-gen-go-grpc generates the service's client and server interfaces.
# Install the two plugins (into $GOBIN, which must be on your PATH)
go install google.golang.org/protobuf/cmd/protoc-gen-go@latest
go install google.golang.org/grpc/cmd/protoc-gen-go-grpc@latest
# Generate code from the project root, output next to the proto file
protoc -I api/proto \
--go_out=api/proto --go_opt=paths=source_relative \
--go-grpc_out=api/proto --go-grpc_opt=paths=source_relative \
inventory/v1/inventory.proto
# This produces:
# api/proto/inventory/v1/inventory.pb.go message types
# api/proto/inventory/v1/inventory_grpc.pb.go Client/Server interfaces
# Add grpc and protobuf to go.mod
go get google.golang.org/grpc google.golang.org/protobufpaths=source_relative makes the output location follow the .proto file's relative path, so the generated package's import path is example.com/shop/api/proto/inventory/v1, consistent with go_package.
Figure: A single inventory.proto passes through protoc and two plugins to produce message types and service interfaces; the server and client both import the same generated code.
Should the generated code be committed to version control? Both approaches are common. Committing it means go build needs no extra tools, at the cost of remembering to regenerate; not committing it means adding a generation step in CI. Either way, it is a good idea to pin the commands above in a Makefile or go generate and lock the plugin versions, so different people don't generate different results. Some teams instead use tools like buf to manage generation settings and compatibility checks, which Step 6 mentions.
Step 3: Implement the server
protoc-gen-go-grpc generated an InventoryServiceServer interface; all we have to do is implement it and register it with a grpc.Server. Below is a trimmed-down version of cmd/inventory/main.go, containing the unary call ReserveStock, the server stream WatchStock, and two interceptors (explained in Step 5):
// Snippet: the inventory service's gRPC server
package main
import (
"context"
"log"
"net"
"sync"
"time"
"google.golang.org/grpc"
"google.golang.org/grpc/codes"
"google.golang.org/grpc/status"
"google.golang.org/protobuf/types/known/timestamppb"
inventoryv1 "example.com/shop/api/proto/inventory/v1"
)
type inventoryServer struct {
// Embed Unimplemented: when the proto gains a method, the old implementation still compiles, and unimplemented methods return Unimplemented.
inventoryv1.UnimplementedInventoryServiceServer
mu sync.Mutex
stock map[string]int32
}
func (s *inventoryServer) ReserveStock(ctx context.Context, req *inventoryv1.ReserveStockRequest) (*inventoryv1.ReserveStockResponse, error) {
if req.GetOrderId() == "" || len(req.GetItems()) == 0 {
return nil, status.Error(codes.InvalidArgument, "order_id and items are required")
}
s.mu.Lock()
defer s.mu.Unlock()
for _, it := range req.GetItems() {
if it.GetQuantity() <= 0 {
return nil, status.Errorf(codes.InvalidArgument, "quantity of %s must be positive", it.GetSku())
}
if s.stock[it.GetSku()] < it.GetQuantity() {
// A business "not enough" is FailedPrecondition; the caller should not retry blindly
return nil, status.Errorf(codes.FailedPrecondition, "sku %s out of stock", it.GetSku())
}
}
if err := ctx.Err(); err != nil { // if the caller has given up, don't deduct stock
return nil, status.FromContextError(err).Err()
}
for _, it := range req.GetItems() {
s.stock[it.GetSku()] -= it.GetQuantity()
}
return &inventoryv1.ReserveStockResponse{
ReservationId: "rsv-" + req.GetOrderId(),
ExpiresAt: timestamppb.New(time.Now().Add(15 * time.Minute)),
}, nil
}
func (s *inventoryServer) WatchStock(req *inventoryv1.WatchStockRequest, stream grpc.ServerStreamingServer[inventoryv1.StockEvent]) error {
ticker := time.NewTicker(2 * time.Second)
defer ticker.Stop()
for {
select {
case <-stream.Context().Done(): // the client disconnected or timed out
return status.FromContextError(stream.Context().Err()).Err()
case <-ticker.C:
s.mu.Lock()
for _, sku := range req.GetSkus() {
ev := &inventoryv1.StockEvent{Sku: sku, Available: s.stock[sku], ChangedAt: timestamppb.Now()}
if err := stream.Send(ev); err != nil {
s.mu.Unlock()
return err
}
}
s.mu.Unlock()
}
}
}
// A unary interceptor that logs each call's method, duration and status code.
func logUnary(ctx context.Context, req any, info *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (any, error) {
start := time.Now()
resp, err := handler(ctx, req)
log.Printf("method=%s code=%s dur=%s", info.FullMethod, status.Code(err), time.Since(start))
return resp, err
}
// Turns a panic into Internal, so the whole process doesn't crash.
func recoverUnary(ctx context.Context, req any, info *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (resp any, err error) {
defer func() {
if r := recover(); r != nil {
log.Printf("panic in %s: %v", info.FullMethod, r)
err = status.Error(codes.Internal, "internal error")
}
}()
return handler(ctx, req)
}
func main() {
lis, err := net.Listen("tcp", ":9003")
if err != nil {
log.Fatal(err)
}
srv := grpc.NewServer(
grpc.ChainUnaryInterceptor(recoverUnary, logUnary), // run in order, from outermost to innermost
)
inventoryv1.RegisterInventoryServiceServer(srv, &inventoryServer{
stock: map[string]int32{"SKU-1": 10, "SKU-2": 3},
})
log.Println("inventory gRPC listening on :9003")
log.Fatal(srv.Serve(lis))
}A few things worth noting:
- Embedding
UnimplementedInventoryServiceServer: this is the form grpc-go requires by default (embedded by value). When the proto later gains a new method, the old implementation still compiles, and calling the new method just gets anUnimplementederror rather than the whole service failing to compile. - Always return errors with
status.Error/status.Errorf: return a plainerrorand the client only seesUnknown, with no way to tell whether to retry. - Read fields with getters:
req.GetOrderId()is safe even whenreqis nil, which is the convention for Protobuf-generated code. - Check
ctx: if the client has already given up (timed out or cancelled), don't do anything with side effects.status.FromContextErrorconvertscontext.DeadlineExceededinto the corresponding gRPC status code. This is the principle from “Go context: Cancellation, Timeouts and Request-Scoped Values” applied to gRPC. - The streaming method's signature: the current generated code uses the generic type
grpc.ServerStreamingServer[StockEvent];Sendsends one event, andstream.Context()is cancelled when the client disconnects.
gRPC defines four kinds of call, corresponding to where the stream keyword appears in the proto:
Figure: A unary call is one question, one answer; server streaming is one request, many responses; client streaming is many requests, one response; and in bidirectional streaming both sides can keep sending.
| Kind of call | In proto | Typical use |
|---|---|---|
| Unary | rpc A(Req) returns (Resp) | The vast majority of queries and commands, e.g. reserving stock |
| Server streaming | rpc A(Req) returns (stream Resp) | Subscribing to changes, returning large results in batches |
| Client streaming | rpc A(stream Req) returns (Resp) | Uploading lots of data, then returning a summary |
| Bidirectional | rpc A(stream Req) returns (stream Resp) | Chat, real-time collaboration, long-running two-way sync |
Prefer unary calls. Streaming is powerful, but load balancing, retries and debugging are all more complex: a long stream stays stuck to the same backend instance, and when the backend is redeployed you also have to handle reconnecting after the stream is interrupted.
Step 4: Implement the client: connections, deadlines and status codes
On the client side, the order service calls inventory. There are three key points: create the connection once, give every call a deadline, and decide what to do next based on the status code.
// Snippet: the order service calls the inventory service
package main
import (
"context"
"errors"
"io"
"log"
"time"
"google.golang.org/grpc"
"google.golang.org/grpc/codes"
"google.golang.org/grpc/credentials/insecure"
"google.golang.org/grpc/status"
inventoryv1 "example.com/shop/api/proto/inventory/v1"
)
func main() {
// NewClient doesn't connect immediately; the connection is made on the first call. The whole process shares this one conn.
conn, err := grpc.NewClient("dns:///inventory:9003",
grpc.WithTransportCredentials(insecure.NewCredentials()), // use TLS in production
)
if err != nil {
log.Fatal(err)
}
defer conn.Close()
client := inventoryv1.NewInventoryServiceClient(conn)
// Every call carries a deadline
ctx, cancel := context.WithTimeout(context.Background(), 800*time.Millisecond)
defer cancel()
resp, err := client.ReserveStock(ctx, &inventoryv1.ReserveStockRequest{
OrderId: "o-1001",
Items: []*inventoryv1.LineItem{{Sku: "SKU-1", Quantity: 2}},
})
switch status.Code(err) {
case codes.OK:
log.Printf("reserved %s until %s", resp.GetReservationId(), resp.GetExpiresAt().AsTime())
case codes.FailedPrecondition:
log.Printf("out of stock: %s", status.Convert(err).Message())
case codes.DeadlineExceeded, codes.Unavailable:
log.Printf("inventory temporarily unavailable: %v", err) // leave it to the retry policy
default:
log.Printf("reserve failed: %v", err)
}
// Server streaming: keep calling Recv until io.EOF or an error
wctx, wcancel := context.WithTimeout(context.Background(), 10*time.Second)
defer wcancel()
stream, err := client.WatchStock(wctx, &inventoryv1.WatchStockRequest{Skus: []string{"SKU-1"}})
if err != nil {
log.Fatal(err)
}
for {
ev, err := stream.Recv()
if errors.Is(err, io.EOF) {
break
}
if err != nil {
log.Printf("watch ended: %v", status.Code(err))
break
}
log.Printf("%s available=%d", ev.GetSku(), ev.GetAvailable())
}
}Connections: grpc.NewClient just creates a *grpc.ClientConn object without blocking until a connection succeeds; the real connection is made on the first call, and it reconnects automatically after a disconnect. A single ClientConn can hold several HTTP/2 connections, each running many requests at once, so the whole process should share one: create it in cmd/order/main.go and hand it to whatever needs it through dependency injection, rather than creating a new one per call. The target string dns:///inventory:9003 means resolving via DNS, which is closely tied to service discovery and is discussed in detail in “Service Discovery and Load Balancing for Go Microservices”.
Deadlines: gRPC has no timeout by default. A call without a deadline waits forever when the other side hangs, eventually dragging down the caller's goroutines and connections. The rule is simple: wrap every call with context.WithTimeout. Better still, gRPC puts the remaining time into the grpc-timeout header sent to the server, so the server-side ctx automatically carries the same deadline; if the server calls further downstream, it only has to pass the same ctx along, and the whole call chain shares one time budget.
Figure: Each hop subtracts the time already spent and passes the remaining budget on; if any hop times out, the whole chain stops together.
Status codes: gRPC has a fixed set of status codes, which the client gets with status.Code(err). The most commonly used ones:
| Status code | Meaning | What the client should do |
|---|---|---|
OK | Success | — |
InvalidArgument | The request itself is wrong | Don't retry; fix the request |
NotFound | The resource doesn't exist | Don't retry |
AlreadyExists | What you are creating already exists | Usually treat as success (idempotent) |
FailedPrecondition | The system state doesn't allow it, e.g. insufficient stock | Don't retry blindly; report it to the user |
PermissionDenied / Unauthenticated | No permission / not logged in | Don't retry |
ResourceExhausted | Quota or rate limit | Try again later |
DeadlineExceeded | Timed out | Depends on whether the operation is idempotent |
Unavailable | The service is temporarily unavailable | Retry after backing off |
Internal / Unknown | A server-side error | Log it and alert |
Only when the server picks the right status code can the client make the right decision. The most common mistake is returning every failure as Internal, leaving the client unable to tell "business errors that shouldn't be retried" from "transient errors that can be". When and how to retry is discussed fully in “Timeouts, Retries and Circuit Breakers”.
Tip:
DeadlineExceededdoesn't mean the server didn't execute. The request may already have successfully deducted stock, with the response just not getting back in time. So operations with side effects need an idempotency key, the subject of “Idempotency and Distributed Locks”.
Step 5: Interceptors: gRPC's middleware
In “net/http and Middleware in Go”, we added cross-cutting logic such as logging and recovery by wrapping http.Handler. gRPC's counterpart is the interceptor, which comes in four kinds: server-side and client-side, unary and streaming.
The server in Step 3 already uses two unary interceptors: recoverUnary turns a panic into an Internal error, and logUnary logs the method name, duration and status code. grpc.ChainUnaryInterceptor(recoverUnary, logUnary) runs them in order from outermost to innermost, like the layers of an onion from the outside in.
Clients can have interceptors too. The example below puts the request ID from ctx into gRPC metadata (the equivalent of HTTP headers), so downstream services' logs can be matched to the same request; on the server side it is read back with metadata.FromIncomingContext:
// Snippet: a client interceptor that puts the request ID into metadata for downstream services
package middleware
import (
"context"
"google.golang.org/grpc"
"google.golang.org/grpc/metadata"
)
type requestIDKey struct{}
func RequestIDUnary(ctx context.Context, method string, req, reply any,
cc *grpc.ClientConn, invoker grpc.UnaryInvoker, opts ...grpc.CallOption) error {
if id, ok := ctx.Value(requestIDKey{}).(string); ok {
ctx = metadata.AppendToOutgoingContext(ctx, "x-request-id", id)
}
return invoker(ctx, method, req, reply, cc, opts...)
}
// Read it back on the server side
func RequestIDFrom(ctx context.Context) string {
if md, ok := metadata.FromIncomingContext(ctx); ok {
if v := md.Get("x-request-id"); len(v) > 0 {
return v[0]
}
}
return ""
}Attach it with grpc.WithChainUnaryInterceptor(RequestIDUnary) when creating the client connection. In practice, you usually don't need to write your own interceptors for logs, metrics and distributed tracing: OpenTelemetry provides a ready-made stats handler for grpc-go, introduced in “Observability in Go: Logs, Metrics and Tracing”. When writing your own interceptors, remember: metadata keys are always lowercase; don't put sensitive data in metadata; and errors inside an interceptor should also be returned as status errors.
Besides interceptors, a server usually registers two standard services before going live: health checking (grpc.health.v1.Health, used by load balancers and Kubernetes probes) and reflection (which lets tools like grpcurl list services, handy for debugging; whether to enable it in production depends on your security requirements).
Step 6: Let the contract evolve: backward-compatibility rules
Once services are deployed independently, old and new versions are bound to be live at the same time: inventory has been upgraded while an old order is still running, or the other way round. Protobuf's design gives you a chance to get this right, provided you follow the rules:
| Change | Safe? | Notes |
|---|---|---|
| Add a field (with a new number) | Safe | Old code skips fields it doesn't recognise; new code reading old data gets zero values |
| Add an RPC method | Safe | An old server returns Unimplemented |
Delete a field and reserved its number and name | Safe | That field in old data is skipped |
| Rename a field | Safe on the wire, but breaks code and the JSON mapping | Avoid where possible |
| Change a field number | Unsafe | Equivalent to deleting the old field and adding a new one |
| Change a field type | Mostly unsafe | Only a few compatible types are exceptions; don't rely on them |
| Reuse a deleted number | Very dangerous | Old data is interpreted as something completely different |
| Rename the package or service | Unsafe | The full method name changes, and old clients can't find it |
There is also a semantic trap: proto3 scalar fields make no distinction between "not set" and "set to the zero value", so when quantity is 0 you can't tell whether the other side didn't send it or really sent 0. If the business needs to distinguish them, add optional before the field; the generated Go field becomes a pointer, and whether it is nil tells you if it was set.
Enums need care too: the first value must be 0, and it is recommended to name it XXX_UNSPECIFIED to mean "not set", e.g. ORDER_STATUS_UNSPECIFIED = 0, followed by ORDER_STATUS_PENDING, ORDER_STATUS_PAID and ORDER_STATUS_CANCELLED. That way, when old code receives a newly added enum value, at least it won't mistake it for some meaningful status.
Handing these rules to a tool to check is the most reliable approach. buf breaking compares the current proto with the version on the main branch and blocks incompatible changes directly in CI. When an incompatible major change is truly unavoidable, add a v2 package, let the two versions coexist, and remove v1 once every client has migrated.
Common pitfalls
- Not setting a deadline: by default gRPC waits forever, so as soon as a downstream service hangs, upstream goroutines pile up. Every call needs a timeout.
- Creating a new connection for every call:
grpc.NewClientshould be called once at startup and shared by the whole process; creating new ones wastes TCP and TLS handshakes and defeats load balancing. - Returning a plain error: without wrapping it in
status.Error, the client only seesUnknownand cannot decide whether to retry. - Reusing field numbers: delete a field without
reserved, and when someone later puts something else under the same number, old data is read as the wrong value. - Passing proto messages around as domain models: generated types carry internal state and must not be copied directly; and letting them seep into business logic means any contract change ripples through all your code. Convert them into your own domain types (such as
Order) in the handler layer. - Overusing streaming: long streams stick to the same instance, causing uneven load and mass interruptions during deployments. If a unary call will do, use a unary call.
- Using
insecure.NewCredentials()in production: TLS or mTLS is recommended even inside the cluster, or leave it to a service mesh. - Messages that are too large: the default receive limit is about 4 MB. When you need to send a lot of data, switch to streaming in batches, or send the address of an object in object storage, rather than simply raising the limit.
Key takeaways
- gRPC + Protobuf defines services with a single
.protocontract, tools generate type-safe code for both sides, and transport uses HTTP/2. - Put contracts in
api/proto/<service>/<version>/, give every RPC its own Request/Response, useTimestampfor time andint64cents for money. - Generate code with
protocplusprotoc-gen-goandprotoc-gen-go-grpc, and pin the commands and plugin versions. - Servers embed
Unimplemented...Server, return errors viastatuswith the right status code, and checkctxbefore doing anything with side effects. - Clients create one connection with
grpc.NewClientand share it across the process; every call carries a deadline, and the status code decides whether to retry. - Interceptors are gRPC's middleware, suited to logging, panic recovery, request IDs and tracing.
- Evolve contracts by only adding, never renumbering, and
reservedanything you delete; check withbuf breakingin CI.
Next
Services can now talk to each other over gRPC, but browsers and external partners won't call internal gRPC services directly, and authentication, rate limiting and request IDs shouldn't be written separately in every service. The next article, “Building an API Gateway in Go: Routing, Authentication and Rate Limiting”, uses the standard library to write a small gateway that handles all of this centrally at the system's entry point.