This resume is also available in English.

4. Reliability4 / 4 · All columns

Columns › 4. Reliability › Part 16

Distributed Transactions: Sagas and the Outbox Pattern

分散式交易:Saga 與 Outbox

Order, inventory and payment each own their database, so no single transaction spans them. This article explains why not two-phase commit, compares orchestrated and choreographed sagas with compensations, then uses a transactional outbox, a relay and an inbox to keep data and events consistent.

  • Go
  • Microservices
In this article
  1. What this article solves
  2. Step 1: Why not two-phase commit
  3. Step 2: Sagas and compensating actions
  4. Step 3: Orchestration and choreography
  5. Step 4: Implementing an orchestrated saga in Go
  6. Step 5: The dual write problem and the transactional outbox
  7. Step 6: Implementing the relay
  8. Step 7: The inbox and end-to-end consistency
  9. Common pitfalls
  10. Key takeaways
  11. Next

In the previous article, “Message Queues and Event-Driven Design”, we had services cooperate asynchronously through events and dealt with duplicates, reordering and dead letters. This article answers the last remaining question: when a single order spans three services and three databases, how do we make it "all succeed or all be undone"?

What this article solves

In the monolith days, placing an order was one database transaction: create the order, decrement stock, write the payment record, and if any step failed, ROLLBACK; the database guaranteed that nothing was left half done. The ACID properties introduced in “database/sql, Transactions and Connection Pools in Go” made this feel like a given.

Once we split into order, inventory and payment services, the principle that "each service owns its data" means each has its own database and none may read or write another's tables directly. Placing an order becomes three independent local transactions spread across three databases. This creates two kinds of problem:

  1. Consistency across services: stock has been reserved, but the charge fails. How do we give the reserved stock back? What state should the order end up in?
  2. Consistency between data and messages: after writing the order, the order service must publish an order.placed event. If it writes to the database first and then publishes, a crash in between means the event is never sent; if it publishes first and then writes, and the write fails, downstream services have already received an event about an order that does not exist. This is called the dual write problem.

This article solves the first problem with sagas, the second with a transactional outbox, and finally uses an inbox so that consumers can handle duplicates safely.

ProblemToolCore idea
A multi-step flow across servicesSagaA chain of local transactions, with compensations on failure
Writing to the database and publishingTransactional outboxWrite the event into the same database first, then let a relay send it
An event may be delivered twiceInboxThe consumer records the IDs of events it has processed

Step 1: Why not two-phase commit

The most intuitive idea is: since one database has transactions, find a transaction that can span several databases. That is two-phase commit (2PC): a coordinator first asks every participant "can you commit?" (the prepare phase); each participant locks the relevant data and answers; once everyone agrees, the coordinator tells them all to commit (the commit phase), and if anyone says no, everything is rolled back. The XA specification and databases' PREPARE TRANSACTION are implementations of this mechanism.

In theory 2PC provides genuine atomicity, but it is rarely used in a microservice architecture, for several reasons:

  • Blocking: after the prepare phase, participants must hold their locks until they hear the coordinator's decision. If the coordinator crashes at that point, participants can neither commit nor roll back, and the data stays locked until the coordinator recovers.
  • Availability multiplies: the whole transaction requires every participant to be online and healthy at the same time, so an outage in any one service makes ordering fail. This is exactly the temporal coupling microservices try to get rid of.
  • Latency and throughput: several network round trips, plus holding locks the whole time, severely limit throughput under high concurrency.
  • Uneven technology support: most messaging systems, NoSQL stores and external payment gateways do not support XA. You cannot ask a third-party payment service to take part in your two-phase commit.

So the mainstream choice in microservices is to give up atomicity and isolation across services and settle for eventual consistency: every step is a reliable local transaction, failures are handled with compensating actions, and the system reaches a consistent state after a while. That is a saga.

Step 2: Sagas and compensating actions

The term saga comes from a 1987 database paper, where it was originally used for long-running transactions. Its definition is simple: a saga is a sequence of local transactions T1, T2 … Tn, each Ti having a matching compensating action Ci; if Tk fails, run Ck-1 … C1 in reverse order.

Applied to placing an order:

StepServiceForward actionCompensation
T1orderCreate order, status pendingChange order to cancelled
T2inventoryReserve stockRelease the reservation
T3paymentChargeRefund
T4orderChange order to paid(Last step, none needed)
Forward steps (each a local transaction) T1 Create order status = pending T2 Reserve stock inventory service T3 Charge ✗ payment declined T4 Mark paid never runs Compensations (run in reverse, must be idempotent) C1 Cancel order status = cancelled C2 Release stock return reservation Walk back on failure Compensation is not a rollback: the order record stays, its status just becomes cancelled; others can see the intermediate states

Figure: when the charge fails, the saga walks back, first releasing the stock and then marking the order as cancelled; T4 never runs.

Compensation is fundamentally different from a database rollback, and this is the key to understanding sagas:

  • Compensation does not make data disappear; it adds a new business action that "cancels out" the earlier one. The order record is still there, only its status becomes cancelled; a refund is a new money movement, not the deletion of the charge record. This is actually good for auditing and reconciliation.
  • Compensations must be idempotent, because they may be retried; and compensations must eventually succeed, because a saga has no "compensation failed" exit. A compensation that hits a transient error keeps retrying, and when it truly cannot complete it must go into a manual handling process rather than be left alone.
  • Not every action can be compensated. A sent email cannot be unsent, and a shipped parcel cannot vanish into thin air. Such actions belong at the very end of the saga, executed only once you are sure you will never need to turn back.

This leads to the principle for ordering steps. A saga has a pivot step: every step before it can be compensated, and every step after it must be designed to "only ever succeed" (it can be retried until it does). In the order flow, the charge is usually the pivot: the stock reservation before the charge can be released, and confirming the order, notifying and shipping after the charge only need retries. Put the steps that are likely to fail (such as a declined payment) early and the steps that are hard to compensate late, and the saga becomes much simpler in most cases.

Step 3: Orchestration and choreography

There are two ways to organise a saga:

Orchestration Order orchestrator tracks saga state Order Inventory Payment commands + replies + Flow lives in one place, easy to read + Clear compensation order − Orchestrator must be highly available and persist its state Choreography Order service Inventory service Payment service OrderPlaced StockReserved PaymentCompleted + Loose coupling, no central point + Natural when there are few steps − Flow scattered across services, hard to trace, prone to cycles

Figure: in orchestration, one orchestrator issues commands to each service in turn and waits for replies; in choreography there is no centre, and each service decides the next step itself after hearing the previous step's event.

Orchestration: an orchestrator (a component inside the order service, or a separate workflow engine) explicitly remembers the saga's state, sends commands to each service in turn, and decides from the replies whether to move on or start compensating. The flow is defined in one place, so it is easy to read, test and trace; "which step is this order stuck on" is answered by looking at the orchestrator's state table. The cost is that the orchestrator itself must be reliable: its state must be persisted, and after a crash and restart it must be able to resume from where it stopped.

Choreography: there is no centre; each service subscribes to the previous step's event and emits its own. The order service emits OrderPlaced; the inventory service hears it, reserves stock and emits StockReserved; the payment service hears that, charges and emits PaymentCompleted; the order service hears that and changes the order to paid. On failure, the payment service emits PaymentFailed, and the inventory and order services each hear it and run their compensations.

AspectOrchestrationChoreography
Flow definitionCentralised in the orchestratorSpread across each service's event handlers
CouplingThe orchestrator knows every participantServices only know about events
TraceabilityHigh: look at one state tableLow: piece it together from distributed tracing
Adding a stepChange the orchestratorA new service subscribes to an event, but compensation logic may need changes in several places
RiskThe orchestrator becomes a bloated centreCyclic dependencies; nobody understands the full flow
Best forCore flows with many steps and complex compensationReactive flows with few, independent steps

A common practical combination: use orchestration for a core transactional flow such as placing an order, so the compensation order stays clear and controlled; use choreography for peripheral reactions such as "after an order is paid, send a notification, award points and update reports", where each service freely subscribes to the order.paid event.

Step 4: Implementing an orchestrated saga in Go

The orchestrator's core logic is actually short: run the steps in order and record the ones completed; if any step fails, run the compensations of the completed steps in reverse. The program below simulates this flow in memory, showing one successful order and one whose payment is declined:

package main

import (
	"context"
	"errors"
	"fmt"
)

// Step is one step of the saga: a forward action plus a compensation
type Step struct {
	Name       string
	Action     func(ctx context.Context, s *OrderSaga) error
	Compensate func(ctx context.Context, s *OrderSaga) error // nil means no compensation needed
}

// OrderSaga is the state of one order saga; in practice it is written to the database after every step
// Status reuses the Order statuses: pending / paid / cancelled
type OrderSaga struct {
	OrderID       string
	ReservationID string
	PaymentID     string
	Status        string
	Log           []string
}

func (s *OrderSaga) logf(format string, a ...any) {
	s.Log = append(s.Log, fmt.Sprintf(format, a...))
}

var ErrDeclined = errors.New("payment declined")

// Run executes the steps in order; if any step fails, the completed steps are compensated in reverse
func Run(ctx context.Context, s *OrderSaga, steps []Step) error {
	for i, st := range steps {
		if err := st.Action(ctx, s); err != nil {
			s.logf("✗ %s failed: %v, starting compensation", st.Name, err)
			for j := i - 1; j >= 0; j-- {
				if steps[j].Compensate == nil {
					continue
				}
				// Compensations must be idempotent and retried until they succeed (retries omitted here)
				if cerr := steps[j].Compensate(ctx, s); cerr != nil {
					s.Status = "needs_manual_fix"
					return fmt.Errorf("compensating %s failed: %w (cause: %v)", steps[j].Name, cerr, err)
				}
			}
			s.Status = "cancelled"
			return err
		}
		s.logf("✓ %s", st.Name)
	}
	s.Status = "paid"
	return nil
}

func orderSteps(cardOK bool) []Step {
	return []Step{
		{
			Name:       "create order (pending)",
			Action:     func(_ context.Context, s *OrderSaga) error { s.Status = "pending"; return nil },
			Compensate: func(_ context.Context, s *OrderSaga) error { s.logf("↺ order marked as cancelled"); return nil },
		},
		{
			Name: "reserve stock",
			Action: func(_ context.Context, s *OrderSaga) error {
				s.ReservationID = "rsv-" + s.OrderID
				return nil
			},
			Compensate: func(_ context.Context, s *OrderSaga) error {
				s.logf("↺ released stock reservation %s", s.ReservationID)
				return nil
			},
		},
		{
			Name: "charge",
			Action: func(_ context.Context, s *OrderSaga) error {
				if !cardOK {
					return ErrDeclined
				}
				s.PaymentID = "pay-" + s.OrderID
				return nil
			},
			Compensate: func(_ context.Context, s *OrderSaga) error {
				s.logf("↺ refunded %s", s.PaymentID)
				return nil
			},
		},
		{
			Name:   "confirm order (paid)",
			Action: func(_ context.Context, s *OrderSaga) error { return nil },
		},
	}
}

func main() {
	ctx := context.Background()
	for _, tc := range []struct {
		id     string
		cardOK bool
	}{{"A-1001", true}, {"A-1002", false}} {
		s := &OrderSaga{OrderID: tc.id}
		err := Run(ctx, s, orderSteps(tc.cardOK))
		fmt.Printf("order %s → %s (err=%v)\n", s.OrderID, s.Status, err)
		for _, l := range s.Log {
			fmt.Println("  " + l)
		}
	}
}

Output:

order A-1001 → paid (err=<nil>)
  ✓ create order (pending)
  ✓ reserve stock
  ✓ charge
  ✓ confirm order (paid)
order A-1002 → cancelled (err=payment declined)
  ✓ create order (pending)
  ✓ reserve stock
  ✗ charge failed: payment declined, starting compensation
  ↺ released stock reservation rsv-A-1002
  ↺ order marked as cancelled

This example is deliberately simple; a production system needs a few more things:

  • Persist the saga state: after every step, write the state (current step, each step's result, the reservation and payment IDs) to the database. When the orchestrator restarts, it scans for unfinished sagas and continues forwards or backwards from where they stopped.
  • Handle "unknown outcomes": a charge request times out and you do not know whether it went through. The example only compensates the steps before the failed one, but when the outcome is unknown, the failed step must also be treated as possibly successful: query again or retry the charge with an idempotency key and compensate only once you are sure nothing was charged, or simply run an idempotent refund (refunding a charge that does not exist is a no-op). This is why “Idempotency and Distributed Locks” requires every step to be safely retryable.
  • Retrying compensations: the example marks needs_manual_fix as soon as a compensation fails; in practice you should keep retrying with backoff first, and only hand over to a human and raise an alert after some time has passed.
  • Deadlines: the saga as a whole also needs a deadline, for example automatically releasing reserved stock that has gone unpaid for fifteen minutes, so that nothing stays stuck in pending forever.

Calls between steps can be synchronous gRPC, or commands and replies through a message queue. The benefit of messaging is that the orchestrator does not need downstream services to be online; the downside is that we are back to the dual write problem: the orchestrator must both "update the saga state" and "send the next command". That is exactly what the outbox in the next step solves.

Step 5: The dual write problem and the transactional outbox

Back to the order service creating an order and publishing an event. Whatever order you put them in, "writing to the database" and "sending a message" are two different systems, and there is no way to put them in the same transaction:

  • Commit to the database first, then publish: crash before publishing, and the order exists but the event is never sent, so stock is never reserved.
  • Publish first, then commit to the database: if the commit fails or rolls back, downstream services receive an event about an order that never existed.
  • Publish inside the database transaction: the message goes out before the transaction commits; if the transaction rolls back, you have the same problem as above, and consumers may query the data before the commit and read stale data.

The transactional outbox solves this: since only the database can guarantee atomicity, write the event into the same database too. The order service writes to the orders table and the outbox table in the same local transaction; if the transaction commits, the event is guaranteed to exist; if it rolls back, the event is guaranteed not to. A separate relay process then reads unpublished events from the outbox, sends them to the message broker and marks them as published once that succeeds.

Order service database One local transaction INSERT orders INSERT outbox Committed together, or neither exists Relay polling or CDC Message broker Inventory service inbox table dedupe by event ID mark as sent after publishing The relay may resend → at-least-once delivery; the inbox gives "effectively processed once" on the consumer side

Figure: the order and the outbox row are written in the same local transaction; the relay reads unpublished events and sends them to the message broker; the inventory service uses an inbox table to deduplicate by event ID.

-- The order service's outbox table
CREATE TABLE outbox (
    id           bigserial   PRIMARY KEY,           -- also determines publish order
    event_id     uuid        NOT NULL UNIQUE,       -- external event ID, used by consumers to deduplicate
    aggregate_id text        NOT NULL,              -- partition key: the order number
    type         text        NOT NULL,              -- e.g. order.placed
    payload      jsonb       NOT NULL,
    created_at   timestamptz NOT NULL DEFAULT now(),
    published_at timestamptz                         -- NULL means not sent yet
);
CREATE INDEX outbox_unpublished ON outbox (id) WHERE published_at IS NULL;

-- Create an order: business data and event in the same transaction
BEGIN;
INSERT INTO orders (id, user_id, total, status)
VALUES ('A-1001', 42, 129900, 'pending');
INSERT INTO outbox (event_id, aggregate_id, type, payload)
VALUES (gen_random_uuid(), 'A-1001', 'order.placed',
        '{"order_id":"A-1001","user_id":42,"total":129900}');
COMMIT;

-- The inventory service's inbox table: event ID dedup and the business write in the same transaction
CREATE TABLE inbox (
    event_id     uuid        PRIMARY KEY,
    processed_at timestamptz NOT NULL DEFAULT now()
);

BEGIN;
INSERT INTO inbox (event_id) VALUES ($1) ON CONFLICT DO NOTHING;
-- previous line affected 0 rows → already processed: ROLLBACK, then acknowledge the message anyway
UPDATE inventory
SET    reserved = reserved + $3
WHERE  sku = $2 AND stock - reserved >= $3;
COMMIT;

In Go, these two INSERTs are simply two ExecContext calls on the same *sql.Tx, written in internal/order/repository.go; the service layer calls a repository method that "creates the order and records the event" and does not need to know the messaging system exists. This means the order service does not depend on the message broker being online when it creates an order at all: if the broker is temporarily down, the event waits safely in the outbox.

Step 6: Implementing the relay

There are two common ways to build a relay:

  • Polling publisher: periodically query rows where published_at IS NULL, take a batch ordered by id, publish them one by one and update them. It is simple to implement and needs only SQL; the downsides are polling latency and the extra database load from frequent queries.
  • Change data capture (CDC, e.g. Debezium): read the database's transaction log (PostgreSQL's WAL, MySQL's binlog) and turn inserts into the outbox table into messages. Latency is low and there is no polling, but it adds another component to operate.

The core of a polling relay looks roughly like this:

// Snippet: PostgreSQL + database/sql; publish is implemented by the message queue client (e.g. kafka-go's Writer)
type outboxRow struct {
	ID      int64
	EventID string
	Key     string
	Type    string
	Payload []byte
}

func relayOnce(ctx context.Context, db *sql.DB, publish func(context.Context, outboxRow) error) (int, error) {
	tx, err := db.BeginTx(ctx, nil)
	if err != nil {
		return 0, err
	}
	defer tx.Rollback() // no-op once committed

	// FOR UPDATE SKIP LOCKED: multiple relay instances never get the same batch; run a single instance when strict ordering is required
	rows, err := tx.QueryContext(ctx, `
		SELECT id, event_id, aggregate_id, type, payload
		FROM outbox
		WHERE published_at IS NULL
		ORDER BY id
		LIMIT 100
		FOR UPDATE SKIP LOCKED`)
	if err != nil {
		return 0, err
	}
	var batch []outboxRow
	for rows.Next() {
		var r outboxRow
		if err := rows.Scan(&r.ID, &r.EventID, &r.Key, &r.Type, &r.Payload); err != nil {
			rows.Close()
			return 0, err
		}
		batch = append(batch, r)
	}
	rows.Close()
	if err := rows.Err(); err != nil {
		return 0, err
	}

	sent := 0
	for _, r := range batch {
		if err := publish(ctx, r); err != nil {
			break // stop here to preserve order; mark what was sent, retry the rest next round
		}
		if _, err := tx.ExecContext(ctx,
			`UPDATE outbox SET published_at = now() WHERE id = $1`, r.ID); err != nil {
			return 0, err
		}
		sent++
	}
	return sent, tx.Commit()
}

This code is called periodically by a goroutine in cmd/order/main.go (for example every 200ms, backing off when there is nothing to do), and is wired to the cancellation mechanism from “Go context: Cancellation, Timeouts and Request-Scoped Values” so that it stops on shutdown.

Look carefully at the relay's failure modes: if publish succeeds but the process crashes before tx.Commit(), this batch's published_at is never written and it will be published again in the next round. The outbox guarantees that "every event is published at least once", not exactly once. This is not a flaw but a deliberate trade-off: better a duplicate than a loss, with duplicates left to the consumer.

A few more practical details:

  • Ordering: sorting by id, running a single relay instance and stopping on failure preserves publish order; multiple instances with SKIP LOCKED raise throughput, but events for the same order may be sent by different instances in a different order. When you need both, shard by a hash of aggregate_id and run one relay per shard.
  • The event ID is decided when the row is written to the outbox, not generated at publish time, so a re-published event carries the same ID and consumers can deduplicate.
  • Clean-up: published rows should be deleted or moved to a history table periodically, so the outbox does not grow without limit and slow down queries.
  • Monitoring: the age of the oldest unpublished event is a good metric; if it keeps rising, the relay is stuck.

Step 7: The inbox and end-to-end consistency

The relay resends and the message queue redelivers, so consumers are bound to receive duplicate events. The inbox table in the SQL above is the consumer's countermeasure: in the same local transaction as the business work, insert the event ID first; only if the insert succeeds do you continue with the business logic, and if it fails the event has already been processed, so you simply acknowledge the message. Because the inbox record and the business write commit or roll back together, you never get "done but not recorded" or "recorded but not done".

Looking at the whole chain end to end:

  1. The order service writes the order and the outbox row in one local transaction; atomicity is guaranteed by the database.
  2. The relay publishes the outbox to the message broker at least once, possibly with duplicates.
  3. The message broker delivers the event to the inventory service at least once, possibly with duplicates.
  4. The inventory service writes the inbox row and the business data in one local transaction; idempotency is guaranteed by a unique constraint.

At-least-once delivery plus idempotent processing gives you effectively-once processing. No single component in the system achieves "exactly-once delivery", yet the end result is correct. Each step of a saga can be chained the same way: when the orchestrator updates the saga state, it writes the next command into the outbox; when a participant receives a command, it deduplicates with the inbox and, once done, writes its reply into its own outbox.

Finally, we have to be honest about the one letter sagas lack: isolation. A saga's intermediate states are visible to others; other requests may see an order whose "stock is reserved but has not been charged", or read data that is about to be undone before the compensation runs. Common countermeasures include:

  • Semantic lock: mark "in progress" with a status field, such as the order's pending. Other flows that see pending know it is not settled yet; for example, a return cannot be requested for a pending order.
  • Commutative updates: make updates applicable in any order, for example "increase or decrease the reserved quantity" rather than "set it to a value".
  • Reread and verify: before a critical step, read the data again and check it has not changed, much like the version number in optimistic locking.
  • Reorder steps: put checks that are likely to fail early, which lowers the chance of needing compensation and shortens the time intermediate states exist.

All of these countermeasures have costs, and the business has to decide which inconsistencies are tolerable and for how long. For a flow like payments, which tolerates no errors at all, you also need regular reconciliation to catch anything missed; “Payment Pitfalls: Double Charges, Callbacks and Reconciliation” goes into this in depth.

Common pitfalls

  • Forcing 2PC between microservices: as soon as the coordinator crashes, every participant's data stays locked, and external services do not support it anyway.
  • Treating compensation as rollback: a compensation is a new business action that leaves a record and can fail; it must be idempotent and keep retrying until it succeeds.
  • Putting non-compensable steps first: send the "order confirmed" notification before charging, and if the charge fails you cannot take it back.
  • Keeping saga state only in memory: after the orchestrator restarts, it does not know which sagas were halfway through, and stock stays reserved forever.
  • Treating an unknown outcome as a failure: compensating the earlier steps as soon as the charge times out, without handling the possibility that "the charge actually succeeded", so the goods never ship but the money is taken.
  • Publishing messages directly inside a database transaction: the message has already gone out when the transaction rolls back, or consumers query the data before it is committed.
  • Generating the outbox event ID at publish time: resent events get different IDs, and consumers cannot deduplicate.
  • Assuming the outbox gives exactly once: if the relay crashes after publishing but before marking, the event is resent; consumers must have an inbox or some other idempotency mechanism.
  • Running multiple relay instances while assuming order is preserved: SKIP LOCKED raises throughput, but events for the same key may arrive out of order.
  • Never cleaning up the outbox: the table grows and grows, polling queries get slower, and relay latency rises with them.
  • Ignoring isolation: other flows read a saga's intermediate state and make decisions on it; use semantic locks or status checks to state explicitly that something is "not settled yet".

Key takeaways

  • Once each service owns its database, no single transaction can span services; 2PC blocks, reduces availability and is not supported by external systems.
  • A saga is a chain of local transactions plus reverse compensations; compensations are business actions that must be idempotent and eventually succeed; non-compensable steps go after the pivot step.
  • Orchestration centralises the flow and is easy to trace, suiting core transactions; choreography is loosely coupled and suits peripheral reactions; the two are often combined.
  • The orchestrator must persist saga state, handle steps with unknown outcomes and set a deadline for the whole saga.
  • A transactional outbox writes the event and the business data in the same local transaction, and a relay publishes it by polling or CDC, guaranteeing at least once.
  • An inbox deduplicates by event ID on the consumer side; at-least-once delivery plus idempotent processing equals effectively-once processing.
  • Sagas have no isolation; reduce the risk from intermediate states with semantic locks, commutative updates, rereads and step ordering.

Next

At this point the order system stays correct through downstream failures, duplicate deliveries and partial failures, but when an order is stuck in pending it is still hard to answer "which service, which step, and why". The next article, “Observability in Go: Logs, Metrics and Tracing”, uses structured logs, metrics and distributed tracing to make the whole cross-service flow visible.