This resume is also available in English.

7. Pitfalls in practice1 / 5 · All columns

Columns › 7. Pitfalls in practice › Part 26

Concurrency Pitfalls: From Data Races to Overselling

併發常見問題:從資料競爭到超賣

Ten concurrency problems that show up when a Go microservice system meets real traffic, from data races, goroutine leaks and deadlocks to lost updates, overselling, hot rows, cache stampedes and duplicate messages, each with symptoms, causes, detection and fixes.

  • Go
  • Concurrency
  • Database
  • Microservices
In this article
  1. What this article solves
  2. Problem 1: Data races
  3. Problem 2: Concurrent map writes kill the program
  4. Problem 3: Loop variables captured by closures
  5. Problem 4: Goroutine leaks
  6. Problem 5: Deadlocks and lock ordering
  7. Problem 6: Unbounded goroutines and WaitGroup misuse
  8. Problem 7: Lost updates and overselling stock
  9. Problem 8: Hot rows
  10. Problem 9: Cache stampedes and the thundering herd
  11. Problem 10: Messages processed more than once
  12. Summary table
  13. Key takeaways
  14. Next

In the previous article, “From Monolith to Microservices: Evolving the Architecture at 100K, 1M and 10M Scale”, we saw how a system grows with scale, and the four services api-gateway, order, payment and inventory are now running in the cluster. Once the system works and scale grows, the real test begins: with more traffic, more replicas and more retries, plenty of problems that never appeared in local testing start to surface. This part, “Lessons from production”, looks back at the problems we hit most often along the way. This article covers concurrency; the next one covers payments.

What this article solves

Concurrency problems share one trait: they almost never happen in development, yet they are certain to happen in production. During development you are the only one clicking, and requests arrive one after another; once live, hundreds of requests read and write the same data in the same second, and any code that “reads, then decides, then writes back” can have someone cut in. Worse, these bugs usually don’t crash the program; they quietly compute the wrong answer: a counter is a little low, stock goes negative, the same message is processed twice. By the time a user or a reconciliation notices, several days have often passed.

This article looks at common concurrency problems in two layers. The first half covers problems inside Go programs: data races, concurrent map writes, loop variables captured by closures, goroutine leaks, deadlocks and spawning goroutines without a limit. The second half covers problems in the database and across services: lost updates and overselling, hot rows, cache stampedes and duplicate message processing. Both layers are really the same thing — “several actors doing a read-modify-write on the same state at the same time” — only the shared thing changes from a variable in memory to a row in the database, a key in the cache or a message in a message queue.

Each problem is covered in the same order: symptoms (what you will see), why (the underlying mechanism), how to reproduce or detect it (how to be sure that’s what it is) and finally how to fix it. Where a fix was taught in an earlier article, we only highlight the key points and link to it rather than repeat it.

Problem 1: Data races

Symptoms

Counters, statistics or cached values are occasionally wrong, and the result differs on every run; sometimes two fields in a struct contradict each other, for example an order’s Status is already paid but PaidAt is still the zero value. The most frustrating part is that adding a log line to observe it makes the problem disappear.

Why

The Go memory model says: when two goroutines access the same variable at the same time, at least one of them writes, and there is no synchronisation between them (no “happens-before” relationship established by a mutex, channel, sync/atomic, WaitGroup and so on), that is a data race and the program’s behaviour is not guaranteed. The most common example is n++: it looks like one line but is actually three steps — “read n, add one, write back”. Two goroutines read 5 at the same time, each adds one and each writes back 6, so two increments only produce 6 and one update is overwritten.

goroutine A shared variable n goroutine B reads n = 5 n = 5 reads n = 5 n = 5 writes back 5 + 1 n = 6 writes back 5 + 1 n = 6 Incremented twice, yet n only went from 5 to 6

Figure: Both goroutines read 5 first and then each writes back 6, so one of the increments simply vanishes.

Beyond “undercounting”, a data race can also expose reordering by the compiler and the CPU: another goroutine may see Status change before it sees PaidAt change, or may even read a half-written result of a multi-word value (a string, slice or interface). So a data race cannot be rationalised as “it’s only a statistic, being a little off doesn’t matter”.

How to reproduce or detect it

Go’s built-in race detector is the most effective tool: go test -race, go run -race or go build -race. It records every memory access at run time, and whenever two unsynchronised accesses actually execute, it prints the call stacks of both sides. The test file below contains the buggy version and two fixes:

package counter

import (
	"sync"
	"sync/atomic"
	"testing"
)

// Racy version: n++ is three steps, "read, add one, write back"
func countUnsafe(workers, times int) int {
	n := 0
	var wg sync.WaitGroup
	for range workers {
		wg.Add(1)
		go func() {
			defer wg.Done()
			for range times {
				n++
			}
		}()
	}
	wg.Wait()
	return n
}

// Fix 1: protect the whole read-modify-write with sync.Mutex
func countMutex(workers, times int) int {
	var (
		mu sync.Mutex
		n  int
		wg sync.WaitGroup
	)
	for range workers {
		wg.Add(1)
		go func() {
			defer wg.Done()
			for range times {
				mu.Lock()
				n++
				mu.Unlock()
			}
		}()
	}
	wg.Wait()
	return n
}

// Fix 2: for a single value, atomic is enough
func countAtomic(workers, times int) int64 {
	var n atomic.Int64
	var wg sync.WaitGroup
	for range workers {
		wg.Add(1)
		go func() {
			defer wg.Done()
			for range times {
				n.Add(1)
			}
		}()
	}
	wg.Wait()
	return n.Load()
}

func TestCountUnsafe(t *testing.T) {
	if got := countUnsafe(8, 1000); got != 8000 {
		t.Errorf("countUnsafe = %d, want 8000", got)
	}
}

func TestCountSafe(t *testing.T) {
	if got := countMutex(8, 1000); got != 8000 {
		t.Errorf("countMutex = %d, want 8000", got)
	}
	if got := countAtomic(8, 1000); got != 8000 {
		t.Errorf("countAtomic = %d, want 8000", got)
	}
}

Run with -race and you will see output like this (shortened):

$ go test -race ./...
==================
WARNING: DATA RACE
Read at 0x00c0000122b8 by goroutine 14:
  example.com/shop/internal/counter.countUnsafe.func1()
      .../counter_test.go:18 +0x94

Previous write at 0x00c0000122b8 by goroutine 9:
  example.com/shop/internal/counter.countUnsafe.func1()
      .../counter_test.go:18 +0xa4

Goroutine 14 (running) created at:
  example.com/shop/internal/counter.countUnsafe()
      .../counter_test.go:15 +0x74
==================
--- FAIL: TestCountUnsafe (0.00s)
    counter_test.go:67: countUnsafe = 7590, want 8000
    testing.go:1867: race detected during execution of test
FAIL

Read the report like this: “the read on line 18” and “the previous write on line 18” happened in two different goroutines, both of which were created on line 15. Following the stacks leads you to the shared variable.

Tip: The race detector only catches races that actually execute. It has no false positives but can miss races. So your tests must genuinely create concurrency (start several goroutines, use t.Parallel()), and go test -race ./... belongs in CI. It makes the program several times slower and noticeably increases memory use, so it is not normally left on in production, but you can run a -race build in a load-testing environment.

How to fix it

The principle fits in one sentence: shared mutable state must either be owned by a single goroutine, or every access to it must be synchronised. You can hand the state to a single goroutine and have everyone else send it messages over a channel (the approach covered in “Goroutines, Channels and Concurrency Patterns in Go”), or wrap the whole read-modify-write in a sync.Mutex.

As for choosing between atomic and mutex: when there is only one value and the operation itself is a single atomic action (increment, load, compare-and-swap), the sync/atomic types (atomic.Int64, atomic.Bool, atomic.Pointer[T]) are the simplest and fastest. As soon as several fields must stay consistent together, or you “check, then decide, then write”, use a mutex. A common mistake is to switch every field to atomic and assume that makes it safe: each field read or written on its own indeed won’t race, but a combination like if a.Load() > 0 { b.Store(...) } can still have someone cut in. That is a logical race condition; the race detector can’t catch it, and only design can prevent it.

Problem 2: Concurrent map writes kill the program

Symptoms

The service occasionally restarts entirely, and the last log line is fatal error: concurrent map writes or fatal error: concurrent map read and map write, followed by a long list of goroutine stacks. On Kubernetes you see the Pod’s restart count keep climbing.

Why

Go’s built-in map does not support concurrent reads and writes. The runtime keeps a “being written” flag in map operations, and when it detects another goroutine writing or reading at the same time, it ends the whole process with a fatal error. It is deliberately designed so that it cannot be caught with recover, because the map’s internal structure may already be corrupted and carrying on would only produce bugs that are even harder to track down. A common trigger is using a map as a global cache, for example caching product data in a map[string]Product in the order service, with request handlers reading while a background refresh writes.

How to reproduce or detect it

This detection is best-effort and doesn’t fire every time, but a test that starts a few goroutines reading and writing the same map will always be caught when run with -race. Another clue is the stack printed with the fatal error: it lists every goroutine at that moment, and searching for mapassign or mapaccess finds the two conflicting places.

How to fix it

The most direct approach is to wrap the map in a sync.RWMutex, using RLock for reads and Lock for writes. sync.Map only suits two cases: keys written once and then mostly read, or different goroutines working on disjoint sets of keys; otherwise it is not necessarily faster than RWMutex, and you lose type checking. When the write volume is very high, you can split the data into several shards by key hash, each with its own lock, so fewer goroutines contend for the same lock. One often-overlooked approach is wholesale replacement: a background job periodically builds a brand-new map and swaps it in all at once with an atomic.Pointer[map[string]Product], so readers need no lock at all — provided the map is never modified after it is built.

Problem 3: Loop variables captured by closures

Symptoms

You start a goroutine per order inside a loop to send notifications, and some orders get several emails while others get none; or you store the address of each element in a slice and find at the end that they all point to the same value.

Why

Before Go 1.22, i and v in for i, v := range items were a single variable for the whole loop; each iteration just wrote a new value into it. A closure captures the variable itself, not its value at that moment, so by the time the goroutine actually runs, it often reads the value from a later iteration, or even the last one; meanwhile the main loop is writing while the goroutine is reading, which is itself a data race.

Go 1.22 changed the semantics: each iteration creates a new variable, and the same goes for the three-clause for i := 0; i < n; i++. The change only applies when the module’s go.mod declares go 1.22 or later, so an old project that upgrades its toolchain without changing the go line keeps the old behaviour. Note that this change only fixes “the loop variable is shared”; it does not fix races caused by a closure writing to a shared outer variable — for example results = append(results, x) inside a goroutine still needs synchronisation.

How to reproduce or detect it

go vet’s built-in loopclosure check warns, under the old semantics, when a loop variable is captured by go or defer on the last line of the loop; -race also catches the race between the main loop writing and the goroutine reading. Checking the go version in go.mod is the first step, because it decides which semantics your code actually has.

How to fix it

For new projects, just set go.mod to 1.22 or later. If you must support older versions, there are two ways: add a line v := v inside the loop to create a new variable, or pass the value into the goroutine as a parameter: go func(v Item) { ... }(v). Even under the new semantics, explicitly passing the values you need as parameters is still a good habit: readers don’t have to work out which version’s rules the code depends on.

Problem 4: Goroutine leaks

Symptoms

Right after deployment everything is fine, but after a few days memory creeps up until Kubernetes kills and restarts the Pod as OOMKilled; on the dashboards the go_goroutines metric only ever rises. After the restart things are normal again, so the problem gets written off as “intermittent” and never fixed.

Why

Goroutines are not garbage-collected. As long as a goroutine is stuck on an action that will never complete — sending to a channel nobody receives from, reading from a channel that is never closed, waiting on a network call with no timeout — it keeps holding its stack and every object it references. The classic pattern is “start a goroutine to call the downstream service while the main flow uses select to wait for the result or a timeout”: after the timeout the main flow moves on, and when the background goroutine finishes and tries to send its result back on the channel, nobody is receiving any more.

package main

import (
	"context"
	"errors"
	"fmt"
	"runtime"
	"time"
)

// Simulates calling the inventory service, with variable latency
func callInventory() int {
	time.Sleep(200 * time.Millisecond)
	return 42
}

// Leaky version: after returning on timeout, nobody receives from ch any more,
// so the background goroutine is stuck on ch <- forever.
func stockLeaky(timeout time.Duration) (int, error) {
	ch := make(chan int)
	go func() { ch <- callInventory() }()
	select {
	case n := <-ch:
		return n, nil
	case <-time.After(timeout):
		return 0, errors.New("timeout")
	}
}

// Fixed version: a buffer of 1 means the send never blocks, and context carries cancellation.
func stockFixed(ctx context.Context) (int, error) {
	ch := make(chan int, 1)
	go func() { ch <- callInventory() }()
	select {
	case n := <-ch:
		return n, nil
	case <-ctx.Done():
		return 0, ctx.Err()
	}
}

func main() {
	before := runtime.NumGoroutine()
	for range 100 {
		_, _ = stockLeaky(10 * time.Millisecond)
	}
	time.Sleep(300 * time.Millisecond) // wait for the background work to finish
	fmt.Println("leaky: extra goroutines =", runtime.NumGoroutine()-before)

	before = runtime.NumGoroutine()
	for range 100 {
		ctx, cancel := context.WithTimeout(context.Background(), 10*time.Millisecond)
		_, _ = stockFixed(ctx)
		cancel()
	}
	time.Sleep(300 * time.Millisecond)
	fmt.Println("fixed: extra goroutines =", runtime.NumGoroutine()-before)
}

Run it with go run -race . and the output is leaky: extra goroutines = 100 and fixed: extra goroutines = 0: every timeout leaves a stuck goroutine behind, and under heavy traffic that becomes tens of thousands. Other common sources of leaks: forgetting to call the cancel returned by context.WithCancel, not calling Stop on a time.Ticker, not closing the Body after reading an HTTP response, and consumer loops that wait only on a channel without listening to ctx.Done().

How to reproduce or detect it

In production the most useful tool is the goroutine profile. With net/http/pprof enabled in the service (covered in “Performance Tuning in Go: pprof and Benchmarks” — remember to expose it only on an internal port), use go tool pprof http://localhost:6060/debug/pprof/goroutine to see which function has the most goroutines stuck on it, or open /debug/pprof/goroutine?debug=1 directly, which groups goroutines with identical stacks and counts them. A leak is very easy to spot: a particular chan send or select line has thousands of goroutines piled up in front of it, and the number grows over time. Take two snapshots some time apart and compare them; the group that grew is the culprit.

In tests you can borrow the idea behind goleak: when a test finishes, check whether any goroutines are still around that shouldn’t be. The open-source goleak library does exactly this; with the standard library alone, you can compare runtime.NumGoroutine() before and after, as the program above does, giving background work a moment to wind down before asserting.

How to fix it

Before starting any goroutine, answer one question: when, and for what reason, does it end? The practical rules: anything that can block must be cancellable via context (“Go context: Cancellation, Timeouts and Request-Scoped Values”); a channel that delivers a single result gets a buffer of 1, so the sender can never get stuck; cancel is always immediately followed by defer cancel(); long-running background loops must be told to stop, and waited for, as part of the flow in “Configuration and Graceful Shutdown in Go”.

Problem 5: Deadlocks and lock ordering

Symptoms

The service hasn’t crashed and CPU is low, but some requests never get a response and eventually all time out at the gateway. Only in tests or very small programs will you see fatal error: all goroutines are asleep - deadlock!. At the database layer, you occasionally get a deadlock detected error, and one transaction is forcibly aborted by the database.

Why

The condition for a deadlock: A holds lock 1 and waits for lock 2, B holds lock 2 and waits for lock 1, and both wait forever. The Go runtime only reports a deadlock when all goroutines are stuck; an HTTP server always has a goroutine waiting for new connections, so when only some goroutines deadlock it says nothing at all and simply looks “stuck”.

Go programs have a few variants that are especially easy to fall into: sync.Mutex is not reentrant, so if a goroutine, inside a method that already holds a lock, calls another method that takes the same lock, it deadlocks itself; with sync.RWMutex, taking the read lock again recursively while already holding it will also hang as soon as a writer is queued in between, because while the writer waits, new read locks queue behind it; and sending on a channel or calling an external service while holding a lock can extend the deadlock beyond the lock itself.

Deadlocks in the database work the same way: one order buys products A and B, another buys B and A, and the two transactions each lock the stock row of their first product before going for the second, so they block each other. PostgreSQL detects this and aborts one of the transactions (SQLSTATE 40P01), while MySQL InnoDB returns error 1213; if the aborted side doesn’t retry, the user sees the order fail.

How to reproduce or detect it

At the Go layer, the most effective approach is to look at every goroutine’s stack: sending SIGQUIT to the process (kill -QUIT <pid>) prints all stacks and then exits, or you can open /debug/pprof/goroutine?debug=2 to look without interrupting the service. Search the stacks for sync.(*Mutex).Lock and semacquire, and note the wait time shown after them, for example [semacquire, 12 minutes] — that is a lock that has been stuck for a long time. At the database layer, PostgreSQL’s log records the two transactions involved in a deadlock and the locks they were waiting for, and you can also query pg_locks together with pg_stat_activity to see who is waiting on whom right now; on MySQL, SHOW ENGINE INNODB STATUS shows the most recent deadlock.

How to fix it

The most fundamental fix is a fixed lock order: when several things must be locked at once, all code takes the locks in the same order. For stock, sort the product IDs before decrementing several products, then update or SELECT … FOR UPDATE them in that order, and deadlocks can’t happen. Supporting measures: keep the scope of a held lock as small as possible, and do no I/O, no channel sends and no external service calls inside a lock; when you need reentrancy, split the “already holds the lock” version into an internal method that doesn’t take the lock; keep database transactions short, and design bounded retries for errors such as 40P01 and 40001.

Problem 6: Unbounded goroutines and WaitGroup misuse

Symptoms

During batch jobs or traffic peaks, the service’s memory explodes, downstream services get knocked over and the database reports that its connections are full; or wg.Wait() returns early and half-finished results get used; occasionally you also see panic: sync: negative WaitGroup counter.

Why

Goroutines are cheap, so it’s easy to write code that “starts one goroutine per record”. Ten thousand orders means ten thousand simultaneous requests to the downstream service, and neither the downstream nor the connection pool mentioned in “database/sql, Transactions and Connection Pools in Go” can cope; large numbers of requests queue outside the pool, then time out and retry, creating even more requests. The number of goroutines is not the problem in itself; the pressure applied simultaneously to shared resources is.

There are three common misuses of WaitGroup: calling wg.Add(1) only inside the goroutine, so the main flow may reach Wait before the Add and return immediately; calling Done more times than Add, so the counter goes negative and panics; and passing a WaitGroup into a function by value, which copies it into a separate counter — go vet’s copylocks check catches this.

How to reproduce or detect it

Run the batch job once with input close to production volume while watching go_goroutines, the downstream request rate and the database connection count; whether the pressure grows linearly with the input is obvious at a glance. go vet and -race catch most WaitGroup problems.

How to fix it

Limit how many operations run at once with a semaphore (a channel of fixed capacity) or a fixed number of workers. The code below only starts a goroutine once it has acquired a slot, and also listens to the context, so no new work is started once the main flow is cancelled:

package main

import (
	"context"
	"fmt"
	"sync"
	"sync/atomic"
	"time"
)

// notify simulates sending order notifications, allowing only limit to run at once.
func notifyAll(ctx context.Context, orderIDs []int, limit int) (maxInFlight int64) {
	sem := make(chan struct{}, limit) // semaphore: capacity is the concurrency limit
	var (
		wg       sync.WaitGroup
		inFlight atomic.Int64
		peak     atomic.Int64
	)
	for _, id := range orderIDs { // since Go 1.22, id is a new variable each iteration
		select {
		case sem <- struct{}{}: // only start a goroutine once a slot is acquired
		case <-ctx.Done():
			wg.Wait()
			return peak.Load()
		}
		wg.Add(1) // Add before go, never inside the goroutine
		go func() {
			defer wg.Done()
			defer func() { <-sem }()
			cur := inFlight.Add(1)
			for { // record the maximum number running at once
				p := peak.Load()
				if cur <= p || peak.CompareAndSwap(p, cur) {
					break
				}
			}
			time.Sleep(5 * time.Millisecond) // pretend to call an external service
			_ = id
			inFlight.Add(-1)
		}()
	}
	wg.Wait()
	return peak.Load()
}

func main() {
	ids := make([]int, 1000)
	for i := range ids {
		ids[i] = i + 1
	}
	peak := notifyAll(context.Background(), ids, 20)
	fmt.Println("max running at once:", peak)
}

The output is max running at once: 20: however many records come in, no more than 20 requests hit the downstream service at once. The semaphore’s capacity should be chosen from what the downstream can handle and the size of the connection pool, not picked at random. The widely used golang.org/x/sync/errgroup also has SetLimit to do the same thing, and handles “cancel the others when one fails” as well. As for WaitGroup, the rules are: Add before the go statement, Done via defer, always pass it by pointer; from Go 1.25 you can also just use wg.Go(func() { ... }), which wraps both Add and Done for you.

Problem 7: Lost updates and overselling stock

Symptoms

A limited-edition product has only 1 left, yet 2 orders are placed at the same time; the stock column goes negative; or two admins edit product data at the same time and whoever saves last overwrites the other’s changes entirely.

Why

This is Problem 1 moved into the database. If the inventory service decrements stock by “first SELECT stock, check in Go whether there is enough, then UPDATE”, two transactions can both read 1, both decide “enough” and both decrement. If the UPDATE says SET stock = stock - 1, the result is -1, i.e. overselling; if it writes a value computed in the program, SET stock = 0, the stock figure looks right but two items were actually sold — that is a lost update.

Transaction T1 (order A) stock row Transaction T2 (order B) SELECT returns 1 SELECT returns 1 stock = 1 1 ≥ 1, can sell 1 ≥ 1, can sell stock - 1 stock = 0 stock - 1 stock = -1 Result: only 1 in stock, yet 2 sold (oversold) Fix: UPDATE … SET stock = stock - 1 WHERE id = $1 AND stock >= 1 The second affects 0 rows → report “sold out”; stock never goes negative

Figure: Both transactions read stock = 1 before the other writes, and both conclude “can sell”; putting the check into the UPDATE’s WHERE clause makes the second one fail.

Many people assume that using a transaction makes it safe, but what a transaction guarantees depends on the isolation level. The table below summarises the common anomalies and how the default levels of two databases behave:

Isolation levelDirty readNon-repeatable readPhantom readLost update (read, modify, write back)
Read Committed (PostgreSQL default)NoYesYesYes
Repeatable Read (MySQL InnoDB default)NoNoNot in PostgreSQL; not for plain reads in InnoDB, but locking reads and updates may see new rowsPostgreSQL raises an error asking you to retry; InnoDB: yes
SerializableNoNoNoNo (conflicting transactions fail and must be retried)

A few details that are easy to misunderstand: PostgreSQL’s Repeatable Read is snapshot isolation; when two transactions update the same row, the one that commits later gets could not serialize access due to concurrent update (SQLSTATE 40001), and the application must retry. But snapshot isolation still can’t prevent “write skew”, where two transactions each read data the other is about to change and then modify different rows. Under Repeatable Read in MySQL InnoDB, a plain SELECT reads the snapshot from the start of the transaction, but UPDATE and SELECT … FOR UPDATE read the latest committed version, so the “compute in the program, then write back” lost update still happens. Serializable is the safest, but it makes transactions fail on conflict, so it must be paired with retries, and it reduces throughput.

How to reproduce or detect it

Write an integration test (the approach from “Testing in Go: Table-Driven Tests, httptest and Integration Tests”) that sets stock to 1, starts 50 goroutines placing orders at the same time, and finally asserts that “the number of successful orders ≤ the original stock” and that stock is not below 0. This kind of test reproduces easily even on a local machine. In production you can run a periodic check query: for each product, whether the quantity in placed orders plus the remaining stock equals the total stock received; and SELECT * FROM inventory WHERE stock < 0.

How to fix it

There are three fixes, listed from most to least recommended, and all of them merge the “check” and the “write” into a single atomic action in the database:

-- Safety net: forbid negatives at the database level, so any approach that slips through fails instead of overselling
ALTER TABLE inventory ADD CONSTRAINT stock_non_negative CHECK (stock >= 0);

-- Fix 1: conditional UPDATE (most recommended; one statement does the check and the decrement)
UPDATE inventory
SET    stock = stock - $2, updated_at = now()
WHERE  product_id = $1 AND stock >= $2;
-- Rows affected = 1: decremented; = 0: not enough stock, report "sold out"

-- Fix 2: pessimistic locking, lock the row first and then check (use when the check is complex)
BEGIN;
SELECT stock FROM inventory WHERE product_id = $1 FOR UPDATE;
-- Check stock >= quantity in the application; ROLLBACK if not enough
UPDATE inventory SET stock = stock - $2 WHERE product_id = $1;
COMMIT;

-- Fix 3: optimistic locking, use a version column to detect "someone changed it after I read it"
UPDATE inventory
SET    stock = $2, version = version + 1
WHERE  product_id = $1 AND version = $3;
-- 0 rows affected: someone modified it first; re-read and try again (with a limit)

Fix 1 is already correct under PostgreSQL’s Read Committed: the second transaction waits for the first to commit, then re-evaluates the WHERE condition against the updated row; by then stock >= 1 no longer holds, so 0 rows are affected. On the Go side you just check whether result.RowsAffected() is 1, and if it is 0 return a sentinel error such as ErrOutOfStock (“Error Handling in Go: Wrapping, Sentinels and Custom Types”). Fix 2 suits cases where the check is too complex to fit in one WHERE clause; the cost is that every other transaction must queue while the lock is held, so the transaction must never call other services. Fix 3 suits low-contention scenarios such as editing product data in an admin panel; in high-contention flash sales most requests end up retrying and failing, which is actually slower.

Across services you need to think one step further: order creating the order and inventory decrementing the stock live in different databases and cannot share one transaction. The right approach is to treat the stock decrement as a step in a saga and compensate on failure (cancel the order or return the stock) — see “Distributed Transactions: Sagas and the Outbox Pattern”; the decrement request itself should also carry an idempotency key so that a retry doesn’t decrement twice.

Problem 8: Hot rows

Symptoms

Normally everything is fine, but the moment a particular product goes on sale, the inventory service’s latency jumps from a few milliseconds to several seconds, while the database CPU stays low; the connection pool fills up, and even unrelated products can’t be ordered. Looking at the database, you find large numbers of transactions all waiting on the lock for the same row.

Why

The fix for Problem 7 makes stock decrements correct, at the cost that updates to the same row must queue. Every UPDATE has to take the row’s write lock and only releases it when the transaction commits; if a transaction takes 5 ms from start to commit, that row can in theory handle at most about 200 updates per second, and any further requests can only queue. If the transaction does other work too (writing the order, writing logs, even calling external services), the lock is held longer and throughput drops further. Adding more service replicas doesn’t help at all, because the bottleneck is that one row in the database, and queued transactions keep holding connections from the connection pool.

How to reproduce or detect it

Use a load-testing tool to place orders for the same product ID concurrently, and watch how latency rises with concurrency. On PostgreSQL, query the number of connections in pg_stat_activity with wait_event_type = 'Lock', and the transactions in pg_locks waiting on the same relation and tuple; on MySQL, look at performance_schema.data_lock_waits.

How to fix it

Start with the cheapest option: keep lock hold times as short as possible. The stock-decrement transaction should only decrement stock; move other writes outside the transaction or process them asynchronously via the outbox; put the stock decrement as the transaction’s last step before committing, to shorten how long the row is locked.

If that isn’t enough, split one row into several: divide a product’s stock into N “stock buckets” (an inventory_buckets table, each row holding part of the stock), and on each order pick a random bucket with stock left to decrement; total stock is the sum of all buckets. The chance of conflict drops to roughly 1/N, at the cost that totals need summing, and when a bucket runs out you must switch buckets and retry, which gets fiddly for the last few items. Another common approach is to move flash-sale decrements to an in-memory store (for example Redis’s atomic decrement) to reject most requests beyond the available stock up front, write the successful ones back to the database asynchronously, and regularly reconcile the two counts. You can also route decrement requests for the same product through a message queue partitioned by product ID and have a single consumer process them in order, or even merge several decrements within a short window into one UPDATE. All of these trade complexity for throughput; first measure to confirm it really is a hot row, then pick the simplest option that is sufficient.

Problem 9: Cache stampedes and the thundering herd

Symptoms

As soon as a popular product’s cache entry expires, the database’s QPS instantly jumps dozens of times and latency spikes; in worse cases the slowed database makes queries time out, the cache never gets rebuilt, and the system enters a cycle of “the slower it gets, the more requests pile up; the more requests, the slower it gets”. The same thing happens when services restart or the cache cluster is restarted.

Why

On a cache miss, each request goes to the database by itself and then fills the cache. Normally that’s fine, but the instant a popular key expires, the hundreds of requests reading it all miss at once and all query the database at once. This is a cache stampede, also called the thundering herd. If many keys are written at the same time with the same TTL, they also expire at the same time. The same shape appears with retries: the moment a downstream service recovers, every client retrying on a fixed interval rushes in at once and knocks it over again.

No protection: cache stampede singleflight: merge identical requests Request 1 Request 1 Request 2 Request 2 Request N Request N Cache expired singleflight one per key Database queried N times Database queried once Other waiters share the same result; random expiry times stop keys expiring together

Figure: On the left, N requests miss at the same time and each queries the database; on the right, with singleflight only one request per key goes through to query, and the others wait and share the result.

How to reproduce or detect it

During a load test, delete a popular key by hand, or set a very short TTL, and see whether database queries spike. On the dashboards, plot the “cache hit rate” and the “database query count” on the same graph; during a stampede the two lines swing sharply in opposite directions.

How to fix it

The first step is to merge identical requests within the process. golang.org/x/sync/singleflight ensures only one call per key actually runs at a time; the other calls wait for it to finish and receive the same result:

// Snippet: internal/inventory/cache.go (requires golang.org/x/sync/singleflight)
type ProductCache struct {
	group singleflight.Group
	cache Cache      // local or Redis cache
	repo  Repository // reads from the database
}

func (c *ProductCache) Get(ctx context.Context, id string) (Product, error) {
	if p, ok := c.cache.Get(id); ok {
		return p, nil
	}
	v, err, _ := c.group.Do(id, func() (any, error) {
		// Don't let the first caller's cancellation affect other waiters, but still bound the query
		qctx, cancel := context.WithTimeout(context.WithoutCancel(ctx), 2*time.Second)
		defer cancel()
		p, err := c.repo.FindProduct(qctx, id)
		if err != nil {
			return Product{}, err
		}
		ttl := 5*time.Minute + time.Duration(rand.Int64N(int64(time.Minute))) // stagger expiry times randomly
		c.cache.Set(id, p, ttl)
		return p, nil
	})
	if err != nil {
		return Product{}, err
	}
	return v.(Product), nil
}

There are two details here: if the context used inside Do were simply the first caller’s, then as soon as that request was cancelled every waiter would fail with it, so context.WithoutCancel cuts off cancellation and we add our own timeout; and the TTL gets a random component so that keys written in the same batch don’t all expire at once.

singleflight only works within a single process, so if inventory has 10 replicas there can still be up to 10 queries. That is usually enough; if not, you can add: refreshing in the background before expiry (only one party refreshes while everyone else keeps reading the old value, i.e. stale-while-revalidate); using a distributed lock so that only one replica in the whole cluster rebuilds the cache (“Idempotency and Distributed Locks” discusses the limits of locks); and briefly caching “does not exist” for keys that aren’t found, to prevent cache penetration. The thundering herd caused by retries is handled with exponential backoff plus random jitter and circuit breakers — see “Timeouts, Retries and Circuit Breakers”.

Problem 10: Messages processed more than once

Symptoms

The same order sends two “payment successful” notifications; stock is decremented twice; loyalty points are added one extra time. The logs show the same message ID processed twice by the consumer, sometimes milliseconds apart, sometimes several minutes apart.

Why

As “Message Queues and Event-Driven Design” explained, most message queues offer an at-least-once delivery guarantee. If the consumer finishes its side effects but crashes, is redeployed or goes through a partition rebalance before it can acknowledge (ack or commit the offset), the message is delivered again; if the producer sends a message, gets no acknowledgement and resends, the queue also ends up with two identical messages. This is not a bug in the queue but the nature of distributed systems: “processing” and “acknowledging” are two actions, and something can always fail between them. Like Problem 1, it is a read-modify-write race, except the two actors may be two successive runs of the same consumer.

How to reproduce or detect it

In tests, feed the same message into the handler twice and assert that the final state is the same as sending it once; in a load-testing environment, randomly kill the consumer process after processing completes but before acknowledgement. In production, record message IDs in logs and traces, and periodically count how many times each ID is processed.

How to fix it

Make consumers idempotent rather than chasing “exactly-once” delivery. The most reliable approach is to write the business data and a “processed message” record in the same database transaction: create processed_messages(consumer, message_id) with a unique constraint and first INSERT … ON CONFLICT DO NOTHING; 0 rows affected means it was already processed, so acknowledge and skip; only when 1 row is affected do you perform the business update, then commit everything together. Another common approach is to make the business operation idempotent in itself, for example UPDATE orders SET status = 'paid' WHERE id = $1 AND status = 'pending', where running it a second time affects 0 rows. On the producer side, use the outbox to keep “writing data” and “publishing the event” consistent (“Distributed Transactions: Sagas and the Outbox Pattern”); the details of designing idempotency keys are in “Idempotency and Distributed Locks”.

Tip: Side effects such as sending emails, push notifications or calling external APIs can’t go inside a database transaction. Instead, record “to send” inside the transaction first, then have a separate step send according to that record and mark it done; a duplicate will then just see “already sent”.

Summary table

#ProblemTypical symptomsHow to detectHow to fix
1Data raceNumbers occasionally wrong, fields contradictgo test -racemutex, atomic, single owner
2Concurrent map writesfatal error: concurrent map writes-race, fatal stack traceRWMutex, sharding, wholesale replacement
3Captured loop variableDuplicate or wrong elements processedgo vet, check the go.mod versionGo 1.22+, pass as a parameter
4Goroutine leakMemory and goroutine count only risepprof goroutine profile, goleakcontext cancellation, buffered channel, defer cancel()
5DeadlockRequests hang with no response, deadlock detectedgoroutine dump, pg_locksFixed lock order, smaller lock scope, retries
6Unbounded goroutines, WaitGroup misuseDownstream knocked over, Wait returns earlyWatch resources grow with input, go vetSemaphore / worker pool, Add before go
7Lost updates and oversellingNegative stock, more sold than in stockConcurrent integration test, reconciliation queryConditional UPDATE, FOR UPDATE, version column
8Hot rowLatency explodes when one product goes on saleLock-wait statisticsShorter transactions, stock buckets, pre-decrement and queues
9Cache stampedeDatabase spike the moment the cache expiresHit rate versus query countsingleflight, random TTL, background refresh
10Duplicate message processingDuplicate notifications, double decrementsCount per message IDIdempotent consumers, dedup table, outbox

Key takeaways

  • Every concurrency problem comes from “several actors doing a read-modify-write on the same state at the same time”; the only difference is whether the shared thing is a variable, a database row, a cache key or a message.
  • go test -race belongs in CI, and tests must genuinely create concurrency; it has no false positives, but only catches races that actually execute.
  • Use atomic for a single value; use a mutex when several fields must stay consistent together or you “check, then write”. Switching every field to atomic does not mean there are no race conditions.
  • Concurrent map writes are a fatal error that cannot be recovered; a shared map must be locked or replaced wholesale.
  • Check the go version in go.mod: from Go 1.22 the loop variable is new each iteration; passing values into a goroutine as parameters is clearest.
  • Every goroutine needs a clear exit condition; find leaks with pprof’s goroutine profile and deadlocks with a goroutine dump.
  • When taking several locks or locking several rows, always use a fixed order; no I/O inside a lock.
  • Limit the pressure applied to downstream services with a semaphore or worker pool, and put wg.Add before go.
  • For stock decrements, put the check in the UPDATE’s WHERE clause, check the rows affected, and add CHECK (stock >= 0) as the last line of defence; a transaction is not the same as safety — know the isolation level you are using.
  • For hot rows, shorten transactions first, then consider buckets, pre-decrementing or queuing per product; for cache stampedes use singleflight plus random TTLs; messages will be duplicated, so consumers must be idempotent.

Next

The most expensive kind of concurrency problem is the kind that involves money. The next article, “Payment Pitfalls: Double Charges, Callbacks and Reconciliation”, takes apart the problems the payment service runs into one by one: double charges, callback ordering, amount calculations and daily reconciliation.