Columns › 4. Reliability › Part 15
Message Queues and Event-Driven Design
訊息佇列與事件驅動
Replacing synchronous calls with events decouples services in time. This article compares work queues and pub/sub, covers Kafka, RabbitMQ and NATS concepts, then partitions and ordering, consumer groups, acks and redelivery, dead-letter queues, idempotent consumers and event schema versioning.
In this article
- What this article solves
- Step 1: Commands and events, queues and publish/subscribe
- Step 2: The core differences between Kafka, RabbitMQ and NATS
- Step 3: Partitions, ordering and consumer groups
- Step 4: Acknowledgement, redelivery and delivery semantics
- Step 5: Retries, dead-letter queues and idempotent consumers
- Step 6: Event schemas and versioning
- Step 7: Organising consumers in a Go service
- Common pitfalls
- Key takeaways
- Next
In the previous article, “Idempotency and Distributed Locks”, we made duplicate requests take effect only once, and noted that at-least-once delivery makes idempotency a necessity; this article turns parts of the order service's flow into asynchronous events and formally introduces message queues.
What this article solves
So far, the order flow in example.com/shop has been entirely synchronous calls: order calls inventory and waits for it to respond; then it calls payment and waits for it to respond. The previous few articles made this chain more robust with timeouts, retries and circuit breaking, but synchronous calls have a few fundamental limitations:
- Temporal coupling: at the moment of the call, the downstream must be online. During the few seconds the payment service is restarting for a deployment, orders fail.
- Latencies add up: the time the user waits is the sum of every hop's latency, and the more downstreams there are, the slower it gets.
- Fan-out is hard: once an order is paid, you need to notify the user, update reports, tell the warehouse to ship, award points… Every new requirement means the order service calls one more service, and gains one more point of failure.
- Peaks hit every layer directly: when traffic surges during a promotion, every layer has to withstand the peak at the same time.
A message queue separates "sending" from "processing" in time: the producer is done once it hands the message to the intermediary (the broker), and the consumer takes it out and processes it when convenient. If a downstream is temporarily down, the message waits in the queue; when a peak arrives, the queue acts as a buffer and consumers work through it at their own pace. The price is that the system becomes eventually consistent: at the moment the order succeeds, the notification may not have been sent yet and the reports may not have been updated, and you have to handle duplicates, out-of-order messages and failure retries, problems that are not obvious with synchronous calls. This article takes those problems apart one by one.
Step 1: Commands and events, queues and publish/subscribe
First, let's clarify two sets of concepts that are often confused. The first is the intent of a message:
- Command: asks a service to do something, such as
ReserveStockorSendEmail. It has a definite recipient, and the sender cares about the result. - Event: states a fact that has already happened, such as
OrderPlacedorOrderPaid. It is named in the past tense, and the sender neither knows nor cares who will handle it.
The power of event-driven design comes from the second kind: the order service only announces "order A-1001 has been paid", while notifications, reporting and shipping each subscribe on their own, and adding a new subscriber requires no changes at all to the order service's code. This is how the loose coupling discussed in “When to Split: Service Boundaries and Decomposition Strategies” is realised at the communication level.
The second set is the delivery model:
Figure: on the left, a work queue hands each message to only one of the workers, which suits sharing out work; on the right, with publish/subscribe every subscriber receives a copy, which suits event notifications.
- Work queue (point-to-point): several consumers compete for the same queue, and each message is processed by only one of them. Used to share load, for example sending emails or generating thumbnails.
- Publish/subscribe (pub/sub): every subscriber receives a copy of every message. Used to broadcast events.
In practice the two are often combined: "order events" use publish/subscribe so that inventory, notifications and reporting each get a copy, while the inventory service itself has three instances that share its copy as a work queue. Kafka's consumer groups are a concrete implementation of exactly this combination, which we'll look at in detail in Step 3.
Step 2: The core differences between Kafka, RabbitMQ and NATS
There are many message systems on the market; these three are the ones most often compared, and their design philosophies differ greatly:
| Aspect | Kafka | RabbitMQ | NATS (JetStream) |
|---|---|---|---|
| Core model | Partitioned append-only log | Exchanges route to queues | Subjects + optional persistent streams |
| After a message is consumed | Retained until the retention period ends, replayable | Deleted from the queue once acknowledged | Core doesn't store it; JetStream retains it per its retention policy |
| Consumption progress | Consumer groups track their own offsets | The broker tracks acknowledgement of each message | The consumer records acknowledgement progress |
| Ordering | Ordered within a partition | Ordered with a single queue and a single consumer | Ordered within a stream |
| Routing flexibility | Mainly by topic and partition | Very strong (direct, topic, fanout, headers) | Subject wildcards |
| Best for | High-throughput event streams, replay, many independent consumers | Work queues, complex routing, per-message acknowledgement | Lightweight, low latency, both request–reply and events |
Kafka's most important feature is that the log can be replayed: messages don't disappear once read, so a new service coming online can read the historical events from the beginning to build its own data; if you find a processing bug, you can fix the code, move the offset back and reprocess. RabbitMQ is a traditional message broker: once a message is acknowledged it is removed from the queue, and its routing rules are very flexible. The core of NATS (Core NATS) is pure "at-most-once" publish/subscribe with no persistence; JetStream adds persistent streams, acknowledgement and replay on top.
When choosing, rather than comparing performance figures, answer a few questions: do you need to replay history? Do you need complex routing? Does the team have the capacity to operate a Kafka cluster? Does your cloud provider offer a ready-made managed service? The concepts that follow are mostly explained in Kafka's terminology, because it lays out partitions, offsets and consumer groups most clearly, but the underlying principles apply to all these systems.
Step 3: Partitions, ordering and consumer groups
A single queue has a throughput limit, so Kafka splits a topic into several partitions. Each partition is an independent log that can only be appended to at the end, and each message within a partition has an increasing position number called the offset. Partitions can be spread across different brokers, with writes and reads in parallel.
Figure: topic orders has four partitions; the two consumers in group A (the inventory service) get two partitions each, while group B (the notification service) has only one consumer, which reads every partition; each group records its own offsets.
Partitions bring a key limitation: ordering only holds within a single partition. If order A-1001's OrderPlaced lands in partition 0 and its OrderCancelled in partition 2, a consumer may process the cancellation before the creation, and an order that has already been cancelled is treated as a new order. The solution is a partition key: the producer specifies a key for each message, and messages with the same key always land in the same partition via hashing. With the order number as the key, all events for the same order are guaranteed to be processed in order; there is no ordering between different orders, but that usually isn't needed.
The consumer group decides who reads which partition:
- Within a group, a partition is assigned to only one consumer at a time, which guarantees ordering within the partition and gives you a work queue.
- Different groups are independent of each other and record their own offsets, which gives you publish/subscribe.
- When there are more consumers than partitions, the extra consumers sit idle. The number of partitions is the upper limit on parallelism, so leave room for growth when planning, because adding partitions later changes which partition a key maps to and breaks existing ordering assumptions.
- When consumers join or leave, the group rebalances, and partitions may move to another consumer. This means information such as "which consumer processed this message before" can't live only in a consumer's memory.
Tip: hot keys make partitions uneven. If one large customer's events make up half the traffic, using the customer number as the key makes one partition particularly busy. When choosing a key, consider both "the scope that needs ordering" and "the traffic distribution"; the smaller the scope, the better.
Step 4: Acknowledgement, redelivery and delivery semantics
When a consumer processes a message, the most important decision is when to acknowledge (ack). In Kafka, acknowledgement takes the form of committing an offset, meaning "I have finished processing every message before this position"; in RabbitMQ, it is sending an ack for each message.
- Acknowledge, then process: if you crash halfway through processing, the message has already been acknowledged and is lost forever. This is at-most-once.
- Process, then acknowledge: if you crash after processing but before acknowledging, you receive the same message again after restarting. This is at-least-once.
The previous article, “Idempotency and Distributed Locks”, explained that most businesses choose at-least-once and then absorb duplicates with idempotent processing. Crashes aren't the only source of duplicates: if a consumer takes too long to process, exceeding the time limit the broker uses to judge it alive, it is kicked out of the group and a rebalance is triggered; another consumer takes over the same partition and starts reading from the last committed offset, so the half-processed message is processed twice. RabbitMQ is the same: unacknowledged messages are requeued when a connection drops. The producer side also creates duplicates: if the producer sends a message and doesn't receive the broker's acknowledgement, it resends, and the broker ends up with two copies (Kafka's idempotent producer setting eliminates duplicates at this stage, but not duplicates on the consumer side).
Taking Kafka with Go as an example, the key is to turn off auto-commit and commit only after processing succeeds (or the message has explicitly been sent to the dead-letter queue):
// Snippet: using github.com/segmentio/kafka-go
func consume(ctx context.Context, handle func(context.Context, kafka.Message) error) error {
r := kafka.NewReader(kafka.ReaderConfig{
Brokers: []string{"kafka:9092"},
GroupID: "inventory", // same group name = the same set of consumers sharing partitions
Topic: "orders.events",
})
defer r.Close()
dlq := &kafka.Writer{Addr: kafka.TCP("kafka:9092"), Topic: "orders.events.dlq"}
defer dlq.Close()
for {
m, err := r.FetchMessage(ctx) // FetchMessage doesn't commit automatically; ReadMessage does
if err != nil {
return err // also exits here when ctx is cancelled
}
if herr := handle(ctx, m); herr != nil {
// Retries were already done inside handle; if it still fails, send it with the reason to the dead-letter queue so the partition isn't blocked
werr := dlq.WriteMessages(ctx, kafka.Message{
Key: m.Key,
Value: m.Value,
Headers: append(m.Headers, kafka.Header{Key: "x-error", Value: []byte(herr.Error())}),
})
if werr != nil {
return werr // can't even write to the DLQ: don't commit, process it again after restarting
}
}
if err := r.CommitMessages(ctx, m); err != nil {
return err
}
}
}On the producer side, remember to specify the key and require acknowledgement from all in-sync replicas, for example &kafka.Writer{Addr: kafka.TCP("kafka:9092"), Topic: "orders.events", Balancer: &kafka.Hash{}, RequiredAcks: kafka.RequireAll}, and put the order number in kafka.Message.Key when writing.
Step 5: Retries, dead-letter queues and idempotent consumers
What should you do with messages whose processing fails? First apply the error classification from “Timeouts, Retries and Circuit Breakers”:
- Transient errors (a dropped database connection, a downstream 503): back off and retry a few times inside the consumer, and it usually succeeds.
- Permanent errors (a malformed message, a reference to an order that doesn't exist, a bug in the code): the result is the same however many times you retry; such messages are often called poison messages.
Poison messages are especially dangerous in an ordered partition: the consumer gets stuck on one, retrying endlessly, none of the messages after it can move, and the whole partition stalls. The solution is a dead-letter queue (DLQ): when retries exceed the limit or an error is judged permanent, send the message, together with the error reason, retry count, original partition and offset, to another topic, then commit the offset and carry on. The DLQ needs monitoring and alerting; a person or a tool examines the cause, and once it's fixed the messages are replayed back to the topic.
Figure: transient errors are retried after backing off; messages that exceed the retry count or can't be processed go to the dead-letter queue and are replayed to the topic once fixed.
Note that sending a message to the DLQ breaks the ordering for that key: if A-1001's OrderPaid goes into the DLQ, the subsequent OrderShipped will be processed first. If the business can't accept that, you can record the key when sending to the DLQ and hold back later messages with the same key as well, processing them in order once the problem is resolved. Whether that complexity is worth it depends on how strict the business is about ordering.
Another key point is the idempotent consumer. Every event needs a unique ID (generated by the producer and unchanged on resend), and the consumer records "this ID has been processed" in the same database transaction as the business processing, skipping it if received again. The dedupe state must live in the database rather than in the consumer's memory, because after a rebalance the partition may move to another instance.
The program below simulates this step together with the concepts from Step 3: four partitions, the order number as the key, two consumers forming one group, retries on transient errors, a poison message sent to the DLQ, and a duplicate event caused by a producer resend being skipped idempotently:
package main
import (
"errors"
"fmt"
"hash/fnv"
"sync"
)
type Event struct {
ID string // unique event ID, used by consumers for deduplication
Key string // partition key: events for the same order land in the same partition
Type string
Payload string
}
// Topic is a topic with n partitions; each partition is an append-only log
type Topic struct {
partitions [][]Event
}
func NewTopic(n int) *Topic { return &Topic{partitions: make([][]Event, n)} }
func (t *Topic) Publish(e Event) int {
h := fnv.New32a()
h.Write([]byte(e.Key))
p := int(h.Sum32() % uint32(len(t.partitions)))
t.partitions[p] = append(t.partitions[p], e)
return p
}
var ErrTransient = errors.New("transient failure")
var ErrPoison = errors.New("unprocessable message")
// Consumer is a member of a consumer group, responsible for the partitions assigned to it
type Consumer struct {
name string
maxRetries int
seen map[string]bool // IDs of events already processed (in practice kept in the database; see the inbox table)
committed map[int]int // committed offset for each partition
dlq *[]Event
mu *sync.Mutex
log []string
}
func (c *Consumer) Run(t *Topic, parts []int, handle func(Event, int) error) {
for _, p := range parts {
for off := c.committed[p]; off < len(t.partitions[p]); off++ {
e := t.partitions[p][off]
c.process(p, off, e, handle)
c.committed[p] = off + 1 // commit only once processed (succeeded or sent to the DLQ)
}
}
}
func (c *Consumer) process(p, off int, e Event, handle func(Event, int) error) {
if c.seen[e.ID] {
c.log = append(c.log, fmt.Sprintf("p%d@%d %s %s: duplicate delivery, skipped", p, off, e.Key, e.Type))
return
}
for attempt := 1; ; attempt++ {
err := handle(e, attempt)
if err == nil {
c.seen[e.ID] = true
c.log = append(c.log, fmt.Sprintf("p%d@%d %s %s: succeeded on attempt %d", p, off, e.Key, e.Type, attempt))
return
}
if errors.Is(err, ErrPoison) || attempt == c.maxRetries {
c.mu.Lock()
*c.dlq = append(*c.dlq, e)
c.mu.Unlock()
c.log = append(c.log, fmt.Sprintf("p%d@%d %s %s: %v, sent to the dead-letter queue", p, off, e.Key, e.Type, err))
return
}
}
}
func main() {
t := NewTopic(4)
events := []Event{
{"e1", "order-1", "OrderPlaced", "v1"},
{"e2", "order-2", "OrderPlaced", "v1"},
{"e3", "order-1", "OrderPaid", "v1"},
{"e4", "order-3", "OrderPlaced", "broken"},
{"e5", "order-2", "OrderPaid", "v1"},
{"e3", "order-1", "OrderPaid", "v1"}, // producer resend: the same event ID arrives again
}
for _, e := range events {
p := t.Publish(e)
fmt.Printf("publish %s %-11s → partition %d\n", e.Key, e.Type, p)
}
handle := func(e Event, attempt int) error {
if e.Payload == "broken" {
return ErrPoison
}
if e.ID == "e2" && attempt < 2 {
return ErrTransient // fails the first time, succeeds on retry
}
return nil
}
var dlq []Event
var mu sync.Mutex
// Two consumers form one group, and the four partitions are split evenly between them
assign := map[string][]int{"consumer-A": {0, 2}, "consumer-B": {1, 3}}
consumers := []*Consumer{}
var wg sync.WaitGroup
for _, name := range []string{"consumer-A", "consumer-B"} {
c := &Consumer{name: name, maxRetries: 3, seen: map[string]bool{}, committed: map[int]int{}, dlq: &dlq, mu: &mu}
consumers = append(consumers, c)
wg.Add(1)
go func() { defer wg.Done(); c.Run(t, assign[name], handle) }()
}
wg.Wait()
for _, c := range consumers {
fmt.Printf("[%s handles partitions %v]\n", c.name, assign[c.name])
for _, l := range c.log {
fmt.Println(" " + l)
}
}
fmt.Printf("dead-letter queue: %d message(s)\n", len(dlq))
}The result (the two consumers run in parallel, but the order within each one's partitions is fixed):
publish order-1 OrderPlaced → partition 1
publish order-2 OrderPlaced → partition 0
publish order-1 OrderPaid → partition 1
publish order-3 OrderPlaced → partition 3
publish order-2 OrderPaid → partition 0
publish order-1 OrderPaid → partition 1
[consumer-A handles partitions [0 2]]
p0@0 order-2 OrderPlaced: succeeded on attempt 2
p0@1 order-2 OrderPaid: succeeded on attempt 1
[consumer-B handles partitions [1 3]]
p1@0 order-1 OrderPlaced: succeeded on attempt 1
p1@1 order-1 OrderPaid: succeeded on attempt 1
p1@2 order-1 OrderPaid: duplicate delivery, skipped
p3@0 order-3 OrderPlaced: unprocessable message, sent to the dead-letter queue
dead-letter queue: 1 message(s)You can observe that order-1's three events all land in partition 1 and are processed in order; the e3 resent by the producer is skipped because its ID has already been processed; and order-3's bad message goes to the DLQ, but partition 3's offset still moves forward without getting stuck. Keeping the dedupe set in the consumer's memory in this example is only a simplification; in production, replace it with an inbox table in the database.
Step 6: Event schemas and versioning
Once an event is published, it becomes a public contract between services, and it is harder to change than an API: Kafka retains historical events and consumers may replay them months later; different consumers deploy on different schedules, so old and new versions of events and old and new versions of consumers exist at the same time.
When designing events, first prepare a fixed envelope that separates the metadata every event needs from the business content:
# Event envelope (shown as YAML; JSON or Protocol Buffers are common on the wire)
id: 01JCX8Z3T6Q8M4N2V7K5R9W1AB # unique event ID, used by consumers for deduplication; unchanged on resend
type: order.paid # event type, past tense
version: 2 # schema version of data
occurred_at: '2026-11-15T10:00:00Z'
source: order # the service that emitted the event
subject: A-1001 # partition key: events for the same order are processed in order
trace_id: 4bf92f3577b34da6a3ce929d0e0e4736 # ties into tracing; see the observability article
data:
order_id: A-1001
user_id: 42
total: 129900 # int64, in minor units (cents), consistent with Order.Total
currency: TWD # optional field added in v2; old consumers ignore itThe core principle of versioning is compatibility:
- Backward compatibility (new consumers can read old events): new fields need a default or must be nullable, so new code can still handle old events that lack the field.
- Forward compatibility (old consumers can read new events): old code should ignore fields it doesn't recognise rather than report an error. Go's
encoding/jsonignores unknown fields by default; Protocol Buffers also preserve unknown fields. - Never delete required fields, change a field's type or meaning, or reuse the name or number of a deleted field. The field-numbering rules discussed in “gRPC and Protocol Buffers in Go” apply to events too.
When you really need an incompatible change, don't change it in place; publish a new event type or a new topic instead (for example order.paid.v2), have the producer publish both during the transition period, and stop the old one once every consumer has migrated. At larger scale, you can introduce a schema registry, which checks whether a new schema is compatible with the old one when the producer publishes, stopping contract-breaking changes before they are deployed.
There is one more design choice: how much data should an event carry? Event notification carries only the ID, and consumers query back when they need details; events are small, but it creates coupling and load from the lookups. Event-carried state transfer puts all the data consumers need into the event, so consumers can be completely independent of the producer, but events get bigger and the data may be stale. Order events usually take the latter approach, carrying the amount and items so the notification and reporting services don't have to call back to the order service.
Step 7: Organising consumers in a Go service
Finally, let's discuss where consumers sit in the project structure. Following the layering in “Layered Architecture and Dependency Injection in Go”, message consumers are in the same layer as HTTP handlers: both are adapters that "turn external input into calls to the service". So the inventory service can add a consumer.go under internal/inventory/ that decodes events, checks versions and calls the business methods in service.go, while the business logic itself has no idea whether a request came from HTTP or a message.
A few practical recommendations:
- One goroutine per partition, processing in order; when you need more parallelism, add partitions rather than starting several goroutines within a partition and scrambling the order. If messages genuinely have no ordering requirement, you can hash by key onto several workers, so the same key still goes to the same worker.
- Processing time needs an upper bound. Wrap the processing of every message in
context.WithTimeout, so one message can't block the whole partition and you don't exceed the broker's liveness limit and trigger a rebalance. - On shutdown, first stop fetching, finish the messages in hand, commit offsets, then close the connection. This is covered in “Configuration and Graceful Shutdown in Go”, and it is especially important for consumers; otherwise every deployment causes a batch of duplicate processing.
- Monitor consumer lag: the gap between the latest offset and the committed offset. Steadily growing lag means consumption can't keep up with production, a signal to scale out consumers or investigate.
Common pitfalls
- Auto-committing offsets: messages are committed as soon as they are read, so if processing fails or the consumer crashes, the message is lost, which in effect becomes at-most-once.
- Not specifying a partition key: events for the same order scatter across partitions, and the cancellation is processed before the creation.
- Assuming the whole topic is ordered: ordering holds only within a partition; use a key to confine the scope that needs ordering to a single partition.
- Consumers that aren't idempotent: at-least-once delivery, rebalances and producer resends all cause the same event to be processed several times.
- Keeping dedupe state in consumer memory: after a rebalance the partition changes hands, and the new consumer doesn't know what has been processed.
- Retrying poison messages forever: one bad message blocks the whole partition; you need a retry limit and a dead-letter queue, and you need to monitor the DLQ.
- Processing in parallel with several goroutines within a partition: when committing an offset you can't be sure all the earlier messages are done, and ordering is scrambled.
- Changing event schemas in place: deleting fields or changing types breaks consumers that haven't been upgraded yet, and every replayed old event.
- Disguising commands as events: an "event" such as
SendEmailToUseris really bound to a single recipient, losing the decoupling that events are for. - Publishing messages directly inside a database transaction: the transaction rolls back but the message has already been sent, or the transaction commits but the message isn't sent; this is the problem the next article solves.
Key takeaways
- Message queues separate sending from processing in time, gaining decoupling, peak shaving and scalability, at the cost of eventual consistency and the complexity of handling duplicates, disorder and failures.
- Commands have a definite recipient, while events state facts that have already happened; work queues share out work, publish/subscribe broadcasts events, and consumer groups combine the two.
- Ordering holds only within a partition; use a partition key to confine events that need ordering to the same partition; the number of partitions is the upper limit on parallelism.
- Acknowledge after processing, accept at-least-once delivery, and implement idempotent consumers with event IDs plus database deduplication.
- Retry transient errors with backoff and send poison messages to a dead-letter queue; the DLQ needs monitoring and a replay process.
- Events are public contracts: a fixed envelope, only compatible changes, a new version for incompatible ones, and a schema registry to guard them where needed.
Next
Messages let services cooperate asynchronously, but there is still a gap: the order service needs to both "write to the database" and "publish an event", the two can't be in the same transaction, and a failure on either side causes inconsistency. The next article, “Distributed Transactions: Sagas and the Outbox Pattern”, uses sagas to coordinate cross-service flows and a transactional outbox to guarantee that data and events succeed together.