This resume is also available in English.

6. Deploying the system3 / 5 · All columns

Columns › 6. Deploying the system › Part 23

From Monolith to Microservices: Evolving the Architecture at 100K, 1M and 10M Scale

從單體到微服務:十萬、百萬、千萬級的架構怎麼演進

How the same online shop should evolve with scale: a modular monolith first, then horizontal scaling and caching, read replicas and the first extracted services, and finally sharding and multiple regions, with the signals that justify each step and a safe procedure for splitting a service out.

  • Architecture
  • Scalability
  • Microservices
In this article
  1. What this article solves
  2. Step 1: Define “scale” clearly
  3. Step 2: Getting started (10K) — a modular monolith is enough
  4. Step 3: 100K — horizontal scaling and caching, still a monolith
  5. Step 4: 1M — read/write splitting, message queues and the first extracted services
  6. Step 5: 10M — sharding, read models and multiple regions
  7. Step 6: Decision table — scale versus architectural choices
  8. Step 7: Safely splitting out a service
  9. Common pitfalls
  10. Key takeaways
  11. Next

In the previous article, “Deploying a Microservice System on Kubernetes”, we put the four services api-gateway, order, payment and inventory onto a cluster, and the whole system is now running. This article pauses and takes a step back to look at how a system like this should grow, stage by stage with scale, from a small monolith into what it is now — which is exactly the order this series follows: first write the monolith well (articles 5 to 8), then learn how to split it (from article 9 onwards), and only then deploy a whole microservice system.

What this article solves

“Our system has to handle a million users, so let's use microservices from the start.” It sounds far-sighted, but in practice it is often one of the most expensive mistakes a system can make. Architecture is not something decided once; it evolves together with traffic, data volume and team size. Each stage has its own most cost-effective approach, and doing the next stage's work early means paying real costs for capabilities you cannot use yet.

This article walks the whole evolution with the same online shop:

  • First define “scale” clearly: which numbers tell you what stage you are at.
  • Four stages — 10K, 100K, 1M and 10M — and for each one: what the system looks like, what the bottlenecks and signals are, what to do, what not to do too early, and how to decide whether to split.
  • A decision table that maps scale to architectural choices.
  • A procedure for “safely splitting out a service”, from finding the boundary all the way to deleting the old code.

The conclusion first: most systems never reach 10M scale in their entire lives, and many do not even need the microservices of the 1M stage. The point of this article is not to encourage you to keep climbing, but to let you know which stage you are at and what the signals for the next one look like — and, until those signals appear, to stay where you are with a clear conscience.

Step 1: Define “scale” clearly

Phrases like “100K scale” or “million scale” are common, but unless you say what you are counting, the discussion easily ends up at cross purposes. A system with a million registered members of whom only a few thousand log in each day needs a completely different architecture from one where a million people rush to buy at the same moment every day. So let's fix the axes first. This article describes scale with four numbers:

MetricMeaningWhy it matters
Registered usersHow many accounts are in the databaseDetermines data volume, index size and backup time
Daily active users (DAU)People who actually use it in a dayDetermines the normal traffic level
Peak requests/sec (RPS)How many requests arrive at the busiest momentDetermines how much compute you need and whether the database can cope
Daily orders (write volume)How many transactions per day must be writtenDetermines write pressure on the primary database and data growth rate

The four stages correspond roughly to the orders of magnitude below. Treat this table as an illustration, not as benchmark results: the same numbers may put no strain at all on a system with well-written queries, while a system that hits the database dozens of times per page will have fallen over long before. Hardware, data model, read/write ratio and traffic shape (steady or spiky) can all shift the thresholds by an order of magnitude.

StageRegistered users (approx.)DAU (approx.)Peak RPS (approx.)Daily orders (approx.)
10KTens of thousandsThousandsTensHundreds to thousands
100KHundreds of thousandsTens of thousandsHundredsThousands to tens of thousands
1MMillionsHundreds of thousandsThousandsTens to hundreds of thousands
10MTens of millions or moreMillions or moreTens of thousands or moreHundreds of thousands or more

What really decides whether to move to the next stage is not which line these numbers have crossed, but the signals: p99 latency starting to rise, database CPU staying high for long periods, the connection pool often making requests queue, a single deployment needing several teams to coordinate, one feature's failure taking down the whole site. These signals are listed concretely for every stage. The numbers tell you roughly where you are; the signals tell you whether it is time to move.

Tip: scale has another, often overlooked dimension — team size. A system's architecture tends to end up mirroring the organisation's communication structure (Conway's law, discussed in Step 4). When a five-person team runs twenty services, the pain usually comes not from traffic but from headcount.

Step 2: Getting started (10K) — a modular monolith is enough

What the system looks like

At this stage, the system is the program written in articles 5 to 8 of the series: a single Go binary that handles requests with net/http and middleware (“net/http and Middleware in Go”), internally split by business domain into three modules — internal/order, internal/payment and internal/inventory — that call each other only through interfaces and are wired together with dependency injection in cmd/ (“Layered Architecture and Dependency Injection in Go”). Behind it sits one PostgreSQL, accessed through the database/sql connection pool, and cross-module consistency is guaranteed by a single database transaction (“database/sql, Transactions and Connection Pools in Go”). Table-driven tests and integration tests give you the confidence to make changes (“Testing in Go: Table-Driven Tests, httptest and Integration Tests”).

Users shop modular monolith One binary, one image, one deployment Order module internal/order Payment module internal/payment Inventory module internal/inventory Modules call each other only through interfaces, wired up with dependency injection in cmd/ Cross-module consistency: a single database transaction guarantees it Connection pool One PostgreSQL Each module touches only its own tables

Figure: three modules in one binary, calling each other only through interfaces; one deployment, one database, and each module touches only its own tables.

Deployment can be very plain: one or two machines, one container image, one CI pipeline. To avoid downtime during deployments, run at least two replicas and handle SIGTERM as described in “Configuration and Graceful Shutdown in Go”. Use a managed database service, turn on automatic backups, and actually rehearse a restore once — a backup you have never rehearsed restoring is no backup at all.

Bottlenecks and signals at this stage

At this stage the real bottleneck is almost never technical; it is that the product is still finding its way: features change drastically, the data model changes often, and domain boundaries are not yet stable. Signals at the system level are usually mild:

  • CPU and memory of a single replica are still low; extra replicas exist purely for zero-downtime deployments.
  • Slow queries show up occasionally, usually because an index is missing.
  • A deployment takes just a few minutes, and the whole team can understand the entire program.

What to do

Spend your effort on making the monolith “structurally correct”; this is the foundation for every later step of evolution:

  1. Clear module boundaries: each module exposes only an interface, each table belongs to exactly one module, and modules do not directly JOIN each other's tables. This looks like unnecessary discipline now, but when you split services in Step 4 it decides whether the split takes weeks or months.
  2. Stateless: keep sessions in signed cookies or the database, and uploaded files in object storage, never on the local disk. A little more discipline now means that adding replicas in the next stage is just a matter of changing a number.
  3. Externalised configuration: database addresses and passwords are read from environment variables or files, so the same container image can run in any environment.
  4. Basic observability: structured logs, metrics for request latency and error rate, and a trace ID for every request. These become indispensable in Step 3, but they are cheapest to put in from day one.

What not to do too early

  • Don't start with microservices. The table in “When to Split: Service Boundaries and Decomposition Strategies” makes it clear: microservices trade network calls, eventual consistency and many deployments for independent releases, independent scaling and fault isolation. With a team of a few people and tens of RPS, you will barely use those three things, yet the costs are just as high. Worse, the boundaries are not yet stable at this point: a wrong boundary inside a monolith is fixed by changing an import; a wrong boundary between services means changing APIs, databases and deployments.
  • Don't start with Kubernetes. One binary plus a managed database runs perfectly well on the simplest platform (even a single virtual machine). A cluster is itself a system that has to be learnt and operated.
  • Don't start with a message queue, caching or sharding. Every component you add is one more thing that can fail and has to be monitored and backed up.

How to decide whether to split

At this stage the answer is almost always “don't split”. The only exception is a part with radically different non-functional requirements, such as image processing that needs special hardware, or data that regulations require to be isolated; even then, prefer “same codebase, another binary” over a fully independent service.

Step 3: 100K — horizontal scaling and caching, still a monolith

What the system looks like

When traffic grows to hundreds of requests per second, the first thing to do is usually not to split services but to run more copies of the same monolith and make the database do less work.

Users CDN static files Load balancer Monolith replica 1 Monolith replica 2 Monolith replica N Stateless: no sessions or uploads kept locally, so any replica can handle any request Redis cache Cache-aside, TTL PostgreSQL primary Indexes, slow queries, pool Background jobs Email and reports moved off the request path Observability: logs, metrics, traces p99 latency, error rate, database CPU, connection pool wait, cache hit rate

Figure: static files go to the CDN, dynamic requests are spread by a load balancer across several stateless monolith replicas; the cache absorbs repeated reads, background jobs move slow work off the request path, and observability is there from day one.

Bottlenecks and signals at this stage

  • Application CPU runs high at peak, and the p99 latency of a single replica rises. This is the easiest bottleneck to fix: add replicas.
  • Database connections approach the limit: with more replicas, the sum of every replica's connection pool may exceed the number of connections the database allows.
  • Slow queries multiply: as data grows, queries that used to be fast enough even with a full table scan start to slow down.
  • The same data is read over and over: product pages and home-page recommendations — most requests read the same small set of popular data.
  • Requests wait for things they shouldn't: sending a confirmation email synchronously after an order, generating a PDF receipt — when a third-party service slows down, the whole request slows down with it.

What to do

1. Stateless replicas behind a load balancer. Because Step 2 already made the app stateless, this is just a matter of changing the replica count from 2 to N. With health checks and graceful shutdown, neither deployments nor scaling drop requests. This is the cheapest and most direct way to scale.

2. Manage the connection pool. The number of replicas times the maximum connections per replica must not exceed the database's limit, and you need to leave room for migrations and admin tools. The SetMaxOpenConns, SetMaxIdleConns and SetConnMaxLifetime settings introduced in “database/sql, Transactions and Connection Pools in Go” must be recalculated here based on the replica count. With even more replicas, you can put a connection pool proxy (such as PgBouncer) in front of the database so that a large number of application connections share a small number of database connections. In your metrics, watch “time spent waiting for a connection”; it reflects problems earlier than “number of connections”.

3. Indexes and slow queries. Turn on the database's slow query log, use EXPLAIN ANALYZE to read the execution plan, and create indexes for the query conditions that are actually used — for example, “a user's orders sorted by time” needs a composite index on (user_id, created_at). While you are at it, fix N+1 queries (querying row by row inside a loop) and unnecessary SELECT *. This step costs the least and often brings the biggest improvement; before adding caches or hardware, make sure the queries themselves are right. The approach in “Performance Tuning in Go: pprof and Benchmarks” applies here too: measure first, then optimise.

4. Redis caching with cache-aside. On reads, check the cache first; only on a miss query the database and write the result back to the cache. On writes, write to the database first and delete the cache entry once that succeeds. Three details decide whether the cache is a helper or a new source of failures:

  • Always set a TTL, and add random jitter so that a large number of keys don't expire in the same second.
  • Delete rather than update the cache: deleting the key right after a write and letting the next read reload it has far fewer race conditions than “update the database and the cache at the same time”.
  • Prevent stampedes: the moment a hot key expires, hundreds or thousands of requests find the cache empty at once and all hit the database together. The fix is to let only one request per key query the database at a time while the others wait for its result.

The example below, which uses only the standard library, puts all three details together. In a real system the first layer would be Redis; here an in-memory map stands in for it, because the point is the flow:

// Cache-aside: check the cache first, query the database only on a miss;
// only one request per key goes to the database at a time, the others wait for its result (prevents stampedes).
package main

import (
	"context"
	"fmt"
	"math/rand/v2"
	"sync"
	"sync/atomic"
	"time"
)

type Item struct {
	SKU string
	Qty int
}

type Order struct {
	ID     int64
	UserID int64
	Items  []Item
	Total  int64 // in cents (minor units)
	Status string
}

type entry struct {
	order   Order
	expires time.Time
}

// call is a database query in progress; requests that arrive later wait for it to finish.
type call struct {
	done  chan struct{}
	order Order
	err   error
}

type OrderCache struct {
	mu       sync.Mutex
	items    map[int64]entry
	inflight map[int64]*call
	ttl      time.Duration
	load     func(ctx context.Context, id int64) (Order, error)
}

func NewOrderCache(ttl time.Duration, load func(context.Context, int64) (Order, error)) *OrderCache {
	return &OrderCache{
		items:    map[int64]entry{},
		inflight: map[int64]*call{},
		ttl:      ttl,
		load:     load,
	}
}

func (c *OrderCache) Get(ctx context.Context, id int64) (Order, error) {
	c.mu.Lock()
	if e, ok := c.items[id]; ok && time.Now().Before(e.expires) {
		c.mu.Unlock()
		return e.order, nil // hit
	}
	if cl, ok := c.inflight[id]; ok {
		c.mu.Unlock()
		select { // someone is already querying: wait for the result, but respect our own timeout
		case <-cl.done:
			return cl.order, cl.err
		case <-ctx.Done():
			return Order{}, ctx.Err()
		}
	}
	cl := &call{done: make(chan struct{})}
	c.inflight[id] = cl
	c.mu.Unlock()

	// Query with a separate context: if the first request is cancelled, it must not take down the others waiting
	loadCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), 2*time.Second)
	cl.order, cl.err = c.load(loadCtx, id)
	cancel()

	c.mu.Lock()
	delete(c.inflight, id)
	if cl.err == nil {
		// Add ±10% random jitter to the TTL so that lots of keys don't expire in the same second
		jitter := time.Duration(rand.Int64N(int64(c.ttl)/5)) - c.ttl/10
		c.items[id] = entry{order: cl.order, expires: time.Now().Add(c.ttl + jitter)}
	}
	c.mu.Unlock()
	close(cl.done)
	return cl.order, cl.err
}

// Invalidate is called *after* writing to the database: only delete the cache entry once the write succeeds; the next read reloads it.
func (c *OrderCache) Invalidate(id int64) {
	c.mu.Lock()
	delete(c.items, id)
	c.mu.Unlock()
}

func main() {
	var dbQueries atomic.Int64
	load := func(ctx context.Context, id int64) (Order, error) {
		dbQueries.Add(1)
		select {
		case <-time.After(50 * time.Millisecond): // simulate a slowish query
		case <-ctx.Done():
			return Order{}, ctx.Err()
		}
		return Order{ID: id, UserID: 42, Items: []Item{{SKU: "tea", Qty: 2}}, Total: 39800, Status: "paid"}, nil
	}
	cache := NewOrderCache(30*time.Second, load)

	var wg sync.WaitGroup
	for range 100 {
		wg.Add(1)
		go func() {
			defer wg.Done()
			ctx, cancel := context.WithTimeout(context.Background(), time.Second)
			defer cancel()
			if _, err := cache.Get(ctx, 1001); err != nil {
				fmt.Println("error:", err)
			}
		}()
	}
	wg.Wait()
	fmt.Println("100 concurrent requests, database queries:", dbQueries.Load())

	cache.Invalidate(1001)
	o, _ := cache.Get(context.Background(), 1001)
	fmt.Printf("Reloaded after invalidation: order %d %s %d cents, queries: %d\n", o.ID, o.Status, o.Total, dbQueries.Load())
}

When you run it you will see that 100 concurrent requests hit the database only once. context.WithoutCancel keeps the actual query unaffected if the first request is cancelled, while each waiter still respects its own timeout. Cache avalanches, hot keys and more variations of the cache and database falling out of sync are taken apart in detail in Problem 9 of “Concurrency Pitfalls: From Data Races to Overselling”.

Tip: the system must still be correct if the cache “disappears entirely” — just slower. Don't treat data that lives only in the cache (a shopping cart, for example) as a cache — that is a database, and its persistence and backups must be designed the way you would for a database.

5. Static files through a CDN. Hand images, JavaScript and CSS to a CDN, and put a content hash in the file names so you can set very long cache lifetimes. This step alone often removes more than half of the requests that reach the application.

6. Move background work off the request path. Sending emails, generating receipts and notifying third parties don't need to make the user wait. At this stage you don't need a separate message queue yet: create a jobs table in the same database, have the request insert a pending job within its transaction, and let a background worker pick jobs up with SELECT … FOR UPDATE SKIP LOCKED. Because the job is in the same transaction as the business data, “the order was written but the notification was never queued” simply cannot happen — this is in fact the precursor to the Outbox in Step 4.

7. Observability becomes routine. At 100K, the three things covered in “Observability in Go: Logs, Metrics and Tracing” must go from “being collected” to “being looked at”: p50/p99 latency and error rate for each endpoint, database CPU and connection pool wait, cache hit rate, the background job queue length, plus alerts on what users actually feel. What to do next depends entirely on what these metrics say.

8. Enforce module boundaries. With more code and more people, “modules talk only through interfaces” can no longer rely on a tacit understanding. You can use Go's internal directories to restrict imports, add a simple static check in CI that forbids internal/order from importing the implementation packages of internal/payment, or have code review reject any cross-module table access as a rule. If the boundaries hold, the services can actually be split in the next step.

What not to do too early

  • You don't need read/write splitting yet. As long as indexes and caching are done well, a sufficiently sized primary database can go a long way. The replication lag problems that read replicas bring (Step 4) are pure cost here.
  • You don't need microservices yet. The application bottleneck is solved by adding replicas; the monolith itself scales horizontally — a key fact many people skip over.
  • Don't cache everything. Every additional kind of cached data is one more chance of “inconsistent data”. Start by caching things that are read often, written rarely and can tolerate briefly stale data, such as product information; keep data that must be correct in real time, such as order status and stock levels, out of the cache for now.

How to decide whether to split

Still don't split. At this stage “splitting” happens at the module level: turn module boundaries from convention into enforcement, so that each module's tables, interfaces and tests can be looked at independently. If you find that two modules are always changed together and their tables JOIN each other, merging them or redrawing the boundary now costs far less than changing it after they have become services.

Step 4: 1M — read/write splitting, message queues and the first extracted services

What the system looks like

At thousands of requests per second and tens to hundreds of thousands of orders a day, a single database and a single deployment unit start to become real constraints. This is the first stage with a good reason to split out services — but only where it hurts.

Users Kubernetes cluster, namespace shop api-gateway Auth, rate limiting, routing order (old monolith) Everything else still here payment First service split out inventory Owns the inventory data Message queue Each service publishes events via an Outbox; consumers are idempotent order primary All writes go here payment database Only payment reads and writes inventory database Only inventory reads and writes Read replicas × 2 Async replication with lag: reading your own fresh writes must go back to the primary

Figure: api-gateway handles authentication and rate limiting in one place; payment and inventory are split out first and own their own databases, while all other features remain in the original monolith (order); services pass events through a message queue, and order's database gains read replicas to share the read load.

Bottlenecks and signals at this stage

  • Reads saturate the primary database: even with a cache, reads from reports, admin queries and list pages keep the primary's CPU high for long periods.
  • Writes and reads interfere with each other: one large report query slows down order writes.
  • Deployment becomes a coordination problem: several teams change the same codebase, a release has to wait until everyone's features are ready, and one team's bug forces the whole site to roll back.
  • Failures spread: when the third-party payment provider slows down, stuck requests fill up the connections and goroutines, and even browsing products is affected.
  • One part has a clearly different load shape: stock is decremented heavily during flash sales while order lookups stay steady; scaling the whole package together is wasteful.

What to do

1. Read replicas and read/write splitting. The primary handles only writes and the few reads that need strong consistency; other reads are spread across one or more read replicas. The biggest trap is replication lag: replicas replicate asynchronously, so if a user places an order and immediately jumps to “My orders”, and that read hits a replica that hasn't caught up yet, they see “the order I just placed has disappeared”. Common countermeasures:

  • Read-your-writes: a user who has just written data reads from the primary for a short while.
  • Strongly consistent reads always go to the primary: checking the order status before payment, checking the quantity before decrementing stock — these decisions must not be based on stale data.
  • Monitor replication lag: when the lag exceeds a threshold, temporarily route reads back to the primary, or take the replica out of the read pool.

The router below puts these rules together. It is generic; in production code T is simply *sql.DB:

// Read/write routing: writes always go to the primary; reads go to a read replica by default,
// but a user who has just written data reads from the primary for a short while, avoiding replication lag.
package main

import (
	"fmt"
	"sync"
	"sync/atomic"
	"time"
)

// Router is generic; in production code T is simply *sql.DB.
type Router[T any] struct {
	primary  T
	replicas []T
	next     atomic.Uint64
	stick    time.Duration // how long to stick to the primary after a write; must exceed normal replication lag

	mu        sync.Mutex
	lastWrite map[int64]time.Time // userID → time of last write
}

func NewRouter[T any](primary T, replicas []T, stick time.Duration) *Router[T] {
	return &Router[T]{primary: primary, replicas: replicas, stick: stick, lastWrite: map[int64]time.Time{}}
}

// Writer returns the primary and records that this user has just written.
func (r *Router[T]) Writer(userID int64) T {
	r.mu.Lock()
	r.lastWrite[userID] = time.Now()
	r.mu.Unlock()
	return r.primary
}

// Reader returns a database that can be read. strong means this read cannot accept stale data (e.g. checking status before payment).
func (r *Router[T]) Reader(userID int64, strong bool) T {
	if strong || len(r.replicas) == 0 {
		return r.primary
	}
	r.mu.Lock()
	t, ok := r.lastWrite[userID]
	if ok && time.Since(t) >= r.stick {
		delete(r.lastWrite, userID) // clear it once the sticky period is over so the map doesn't grow forever
		ok = false
	}
	r.mu.Unlock()
	if ok {
		return r.primary // read your own fresh writes
	}
	i := r.next.Add(1) % uint64(len(r.replicas))
	return r.replicas[i]
}

func main() {
	r := NewRouter("primary", []string{"replica-a", "replica-b"}, 2*time.Second)

	fmt.Println("User 1 browses orders:", r.Reader(1, false))
	fmt.Println("User 1 browses orders:", r.Reader(1, false))
	fmt.Println("User 1 places an order (write):", r.Writer(1))
	fmt.Println("User 1 views the order just placed:", r.Reader(1, false))
	fmt.Println("User 2 browses orders:", r.Reader(2, false))
	fmt.Println("User 2 checks before paying:", r.Reader(2, true))
}

This example keeps “who has just written” in the memory of a single process, so it is only illustrative: with multiple replicas, the next request may land on a different replica. In practice you can put the time of the last write (or the database's replication position, such as PostgreSQL's LSN) into the user's session or a cookie, so that any replica can make the decision.

2. A message queue for asynchronous work. The jobs table in the database starts to struggle at this point, and several services need to subscribe to the same event (once an order is placed, stock must be decremented, a notification sent and points added). This is when you switch to a real message queue and, following “Message Queues and Event-Driven Design”, let services communicate through events; consumers must be idempotent and there must be retries and a dead-letter queue.

3. Split out the first services along bounded contexts — only where it hurts. Following the method in “When to Split: Service Boundaries and Decomposition Strategies”, use the strangler fig pattern to move one piece at a time. In an e-commerce system, the usual first candidates are:

CandidateWhy it comes first
paymentDepends on third parties, latency is unpredictable, needs fault isolation; different compliance and audit requirements
inventoryWrite pressure concentrates during flash sales, needs independent scaling; clear boundary (only quantities and reservations)
orderUsually the core; stays in the original program while other features are moved out around it

The order is set by “pain”, not by “ease of splitting”: if your system hurts most in search or notifications, split that out first. The yardstick is the signals from Step 1 — once it is split out, can it be released independently, scaled independently, or have its failures isolated?

4. An API Gateway consolidates the entry point. With more services, authentication, rate limiting and routing shouldn't be written once in every service. Following “Building an API Gateway in Go: Routing, Authentication and Rate Limiting”, centralise them in api-gateway, which is also the routing layer of the strangler fig pattern: it decides which paths are still in the monolith and which have moved to new services. Calls between services must also get “Timeouts, Retries and Circuit Breakers”, so that when the payment provider slows down only payment is affected, rather than the whole site going down.

5. Each service owns its own database. This is the watershed for whether a split succeeds. If payment becomes an independent service but still reads and writes the payments table in the monolith's database directly, what you get is a “distributed monolith”: the deployments are separate, the coupling is not reduced at all, and you have added network calls on top. Each piece of data has exactly one owner, and other services get it through APIs or events; consistency across services is coordinated with the Sagas of “Distributed Transactions: Sagas and the Outbox Pattern”, events are published reliably within the same transaction using an Outbox, and every operation that may be retried is designed to be idempotent following “Idempotency and Distributed Locks”.

6. Kubernetes and automation. Three or four services, each with several replicas and its own release cadence, start to become too much to manage by hand. This is when you hand deployment to a cluster, which is exactly what “Deploying a Microservice System on Kubernetes” did: one Deployment per service, with an HPA scaling automatically based on load. For services where CPU is not a good metric (such as inventory, which mostly waits on the database), you can use a custom metric like requests per second instead, and make scaling up fast and scaling down slow:

# deploy/k8s/base/inventory/hpa.yaml
# Primarily requests per second per Pod, CPU as a secondary metric; requires a custom metrics adapter to be installed first
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: inventory
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: inventory
  minReplicas: 3
  maxReplicas: 30
  metrics:
    - type: Pods
      pods:
        metric:
          name: http_requests_per_second
        target:
          type: AverageValue
          averageValue: '200' # illustrative value: measure a single Pod's comfortable load with a load test, then fill it in
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - type: Percent
          value: 100
          periodSeconds: 30
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Pods
          value: 2
          periodSeconds: 60

With multiple metrics, the HPA computes the required replica count for each one separately and takes the largest. Don't fill in averageValue by gut feeling: use a load test to find how many requests a single Pod can handle while p99 latency is still acceptable, then apply a safety margin. And remember: when the application scales, the database does not scale with it — the replica ceiling times the connection pool size must still not exceed the database's connection limit.

7. Align team structure with service boundaries. Conway's law says that a system's structure mirrors the communication structure of the organisation that designs it. Use it in reverse: let one team fully own one or a few services (code, database, deployment, alerts, on-call), so that service boundaries are team boundaries. If one service needs three teams to change it together, or one team has to touch five services for every feature change, that is a sign the boundaries were drawn wrongly.

What not to do too early

  • Don't split into a dozen services in one go. Split out the one or two that hurt most, get the splitting process, monitoring and deployment running smoothly, then decide on the next one. Every service needs someone on call for it.
  • Don't shard yet. Read/write splitting, caching and partitioning large tables by time (PostgreSQL partitioned tables) can usually carry you a lot further. Sharding belongs to Step 5, and it is one of the least reversible decisions there is.
  • Don't build your own platform. A service mesh, a home-grown deployment system or hand-written service discovery are not worth it at the scale of a few services; use managed services and off-the-shelf tools.

How to decide whether to split

Before splitting out a service, you must at least be able to answer the following questions, and the answers must be “yes”:

  1. Does this part have a clear bounded context, and can its data have a single owner?
  2. Once split out, will it bring at least one benefit you actually need: independent releases, independent scaling or fault isolation?
  3. Does it not need strongly consistent transactions with other modules, or can it accept eventual consistency via a Saga?
  4. Is there a team willing and able to own it completely?
  5. Are automated deployment, centralised logging, metrics and distributed tracing all in place?

If the answer to any question is “no”, it is cheaper to tidy it up inside the modular monolith first than to split it out.

Step 5: 10M — sharding, read models and multiple regions

What the system looks like

At this magnitude, the write capacity of a single primary database, the availability of a single region and the operational capacity of a single team all become constraints. The system's focus shifts from “splitting services” to “splitting data” and “splitting geography”.

Global traffic management (DNS/Anycast) Region A (multiple availability zones) Edge: rate limiting, load shedding, api-gateway order payment inventory Plus search, recommendations, notifications… run by other teams Order database: sharded by user_id Shard 1 Shard 2 Shard 3 … Event stream (CDC, Outbox) Read model “My orders” list Search index Product and order search Region B Same services Active-passive: on standby Active-active: always live Data residency: some users' data must stay in this region Cross-region replication Asynchronous, with lag

Figure: global traffic management directs users to different regions; within a region, rate limiting and load shedding come first, order data is sharded by user_id, and an event stream feeds changes into read models and the search index; regions replicate to each other asynchronously.

Bottlenecks and signals at this stage

  • Primary writes hit the ceiling: the primary is already on the largest instance size and write throughput is still not enough, or a single table is so large that index maintenance, backup and restore, and schema changes all take hours.
  • Query requirements conflict: for the same order data, placing an order looks it up by order ID, “My orders” looks it up by user, customer support needs full-text search, and operations need aggregate statistics; one set of indexes cannot satisfy everyone.
  • A single-region outage is unacceptable: an incident in one cloud region means the whole business stops.
  • Traffic peaks far exceed the average: during a big promotion, instantaneous traffic may be several to dozens of times normal, and scaling cannot keep up.
  • Cost becomes a first-class problem: infrastructure spending is large enough to justify someone managing it full time.

What to do

1. Sharding: split one large table across several databases. The first step is usually partitioning within the same database: the data is still in one PostgreSQL, but physically split into several child tables. As long as a query includes the partition key, it scans only the relevant child tables, and maintenance (such as deleting old data) can be done one partition at a time:

-- order database: the orders table is hash-partitioned by user_id into 8 partitions (the count is illustrative)
CREATE TABLE orders (
    id         BIGINT      NOT NULL,
    user_id    BIGINT      NOT NULL,
    total      BIGINT      NOT NULL,             -- unit: cents
    status     TEXT        NOT NULL DEFAULT 'pending'
               CHECK (status IN ('pending', 'paid', 'cancelled')),
    created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
    PRIMARY KEY (user_id, id)                    -- primary keys and unique indexes must include the partition key
) PARTITION BY HASH (user_id);

CREATE TABLE orders_p0 PARTITION OF orders FOR VALUES WITH (MODULUS 8, REMAINDER 0);
CREATE TABLE orders_p1 PARTITION OF orders FOR VALUES WITH (MODULUS 8, REMAINDER 1);
CREATE TABLE orders_p2 PARTITION OF orders FOR VALUES WITH (MODULUS 8, REMAINDER 2);
CREATE TABLE orders_p3 PARTITION OF orders FOR VALUES WITH (MODULUS 8, REMAINDER 3);
CREATE TABLE orders_p4 PARTITION OF orders FOR VALUES WITH (MODULUS 8, REMAINDER 4);
CREATE TABLE orders_p5 PARTITION OF orders FOR VALUES WITH (MODULUS 8, REMAINDER 5);
CREATE TABLE orders_p6 PARTITION OF orders FOR VALUES WITH (MODULUS 8, REMAINDER 6);
CREATE TABLE orders_p7 PARTITION OF orders FOR VALUES WITH (MODULUS 8, REMAINDER 7);

-- An index created on the parent table is automatically created on every partition
CREATE INDEX orders_user_created_idx ON orders (user_id, created_at DESC);

-- A query that includes the partition key scans only one partition
EXPLAIN SELECT id, total, status
FROM orders
WHERE user_id = 42
ORDER BY created_at DESC
LIMIT 20;

Partitioning solves the problem of a single table being too large, but writes still happen on the same host. Only when a single host's write capacity is genuinely insufficient do you go further and shard: different users live on different database clusters. The choice of shard key decides everything:

  • Shard by user_id: all of a user's orders are on the same shard, so “My orders” queries only one shard and order transactions don't span shards. This is the most common choice in e-commerce.
  • Shard by order_id: writes are distributed most evenly, but “all orders of a given user” must query every shard.
  • Hot shards: when the shard key is unevenly distributed, one shard gets especially busy. A big promotion by one seller, or a flood of orders from one enterprise account, can turn a shard into a hot spot. Data keyed by seller is particularly prone to this.

A common design in practice is to first hash into a fixed number of logical shards (say 1024), then use a mapping table to map logical shards to physical databases. When you need to scale out, you just move some logical shards to a new database and update the mapping table, without recomputing where every row belongs. Encoding the logical shard into the order ID also lets a request that only has an order ID find the shard directly:

// Shard key: user_id is first hashed into a fixed number of logical shards, then a mapping table maps them to physical databases.
// Migrating only changes the mapping table; there is no need to recompute the hash of every row.
package main

import (
	"encoding/binary"
	"fmt"
	"hash/fnv"
)

const logicalShards = 1024 // fixed from the start, never changed later

// LogicalShard decides which logical shard a user's data belongs to.
func LogicalShard(userID int64) int {
	var b [8]byte
	binary.BigEndian.PutUint64(b[:], uint64(userID))
	h := fnv.New32a()
	h.Write(b[:])
	return int(h.Sum32() % logicalShards)
}

// ShardMap maps logical shards → physical databases; in practice it lives in configuration or a coordination service.
type ShardMap []shardRange

type shardRange struct {
	from, to int // from inclusive, to exclusive
	db       string
}

func (m ShardMap) DB(userID int64) string {
	s := LogicalShard(userID)
	for _, r := range m {
		if s >= r.from && s < r.to {
			return r.db
		}
	}
	panic(fmt.Sprintf("logical shard %d has no database mapped", s))
}

// OrderID encodes the logical shard into the low 10 bits of the order ID: the shard can be found from the order ID alone.
func OrderID(userID, seq int64) int64 {
	return seq<<10 | int64(LogicalShard(userID))
}

// ShardOfOrder recovers the logical shard from an order ID.
func ShardOfOrder(orderID int64) int {
	return int(orderID & (logicalShards - 1))
}

func main() {
	m := ShardMap{
		{0, 256, "orders-db-1"},
		{256, 512, "orders-db-2"},
		{512, 768, "orders-db-3"},
		{768, 1024, "orders-db-4"},
	}

	count := map[string]int{}
	for id := int64(1); id <= 100000; id++ {
		count[m.DB(id)]++
	}
	for _, r := range m {
		fmt.Printf("%s: %d users\n", r.db, count[r.db])
	}

	id := OrderID(42, 7)
	fmt.Printf("User 42's order %d is in logical shard %d (= %d)\n", id, ShardOfOrder(id), LogicalShard(42))
}

Tip: after sharding, cross-shard JOINs, cross-shard transactions and global uniqueness constraints (such as “a coupon code can be used only once”) all become difficult or expensive. That is why sharding comes last: it turns many things the database used to do for you into things the application has to handle itself. Before deciding to shard, make sure you have exhausted read/write splitting, partitioning, vertical splitting (putting different tables in different databases, i.e. the one-database-per-service of Step 4) and hardware upgrades.

2. CQRS and read models. Command Query Responsibility Segregation (CQRS) separates the “model used for writes” from the “model used for queries”: the write side keeps normalised data organised by shard key, and every change publishes an event through an Outbox or CDC (change data capture, which reads the database's transaction log directly); the read side subscribes to the events and builds data optimised for specific queries, such as a per-user pre-built order list, a full-text search index for customer support (a search engine such as Elasticsearch or OpenSearch), and an analytics database for operations. The price is that the read model always lags slightly behind writes, and the interface has to tolerate “an order just placed appears in search results only a few seconds later”.

3. Event streaming. At 1M, a message queue is for “handing off work”; at 10M, events often become the data backbone of the whole system: the same stream of order events feeds the read models, the search index, recommendations, risk control and the data warehouse at the same time. At this point you lean towards an event streaming platform that retains history and can be replayed (the Kafka type compared in “Message Queues and Event-Driven Design”). A new downstream system can replay events from the beginning to build its own data instead of asking for access to the upstream database. At this stage, the structure and versioning of events become a contract as important as an API.

4. Multiple availability zones first, then multiple regions. The first step to high availability is multi-AZ: several data centres within the same region, with very low latency, so the database can replicate synchronously and fail over automatically when one availability zone goes down. Most systems need nothing more. Multi-region, on the other hand, is a whole other order of complexity:

ModeHow it worksBenefitCost
Active-passive (primary/standby)One region serves traffic, another replicates the data and stands bySimple model, writes have only one primaryFailover may lose data not yet replicated; failover must be rehearsed
Active-activeSeveral regions serve reads and writes at the same timeServes users nearby, a single-region outage goes unnoticedWrite conflicts, cross-region consistency and data placement must all be redesigned

Active-active usually has to be designed together with sharding: each user has a “home region” where their writes happen, and other regions only read or forward, so the same record is never modified on both sides at once. There are also data residency regulations: some countries or industries require user data to stay in a particular region, which directly determines how shards map to regions. It is best to clarify this at the start of the design rather than patching it up later.

5. Rate limiting and load shedding. At this scale you will inevitably hit moments when “traffic exceeds capacity”, and the system must be able to reject in a planned way rather than collapse entirely. The rate limiting in “Building an API Gateway in Go: Routing, Authentication and Rate Limiting” is the first line of defence; the second is load shedding: when a service detects that it is overloaded (for example, queue wait time exceeds a threshold), it rejects some requests outright with a 503 so that the requests already in progress can complete. You can also prioritise features: placing orders and paying matter most, while recommendations, browsing history and reviews can be temporarily switched off at peak. At this scale, circuit breakers, timeouts, and backoff and limits on retries also go from “good habits” to “hard requirements” — uncontrolled retries turn a small failure into a total collapse.

6. Capacity planning and load testing. No more “add machines when something breaks”; instead, estimate ahead of time: how much traffic a big promotion is expected to bring, how much each service can handle while p99 latency is acceptable, how much write headroom the database has left — and run load tests with data and traffic shapes close to production. A load test should find not only “how much it can take” but also “how it breaks beyond that” — whether it gracefully rejects some requests or collapses in a cascade.

7. Platform team and cost. When there are many services and many teams, each team handling its own deployment, monitoring, databases and secrets management leads to a great deal of duplication. This is when it is worth setting up a platform team that turns the cluster, the deployment pipeline, observability and database templates into internal products, so business teams can serve themselves in a standard way. Cost must also become an engineering metric: infrastructure cost per order, idle resources, over-provisioned replica counts and cross-region transfer fees should all be measured and reviewed.

What not to do too early

Everything at this stage is expensive, so the list of things “not to do too early” is really the whole stage: don't shard without a write bottleneck, don't go multi-region without regulatory or availability requirements, don't adopt CQRS without conflicting query requirements, and don't build a platform team without many services. Many systems serve their users reliably for years at the 100K or 1M stage. That is not a failure; it is an architecture that fits its requirements.

How to decide whether to split

At 10M, what gets split changes from “services” to “data” and “regions”, and the criteria change with it:

  • Split data: primary writes have reached the limit, single-table maintenance time is unacceptable, and you have found a shard key that keeps the vast majority of transactions within a single shard.
  • Split read models: there are several conflicting query requirements, and the business can accept a brief lag in the read model.
  • Split regions: the loss from a single-region outage outweighs the cost of building and running multiple regions, or regulations require data residency.

Step 6: Decision table — scale versus architectural choices

The four stages condensed into one table. “—” means the stage usually doesn't need it yet; again, this is only a common starting point, and your own system's signals take precedence:

Architectural choice10K100K1M10M
Deployment unitModular monolithMonolith, multiple replicasMonolith + a few extracted servicesMany services, owned by many teams
Application scaling2 replicas for availabilityHorizontal scaling behind a load balancerCluster with HPA autoscalingMultiple availability zones, multiple regions
DatabaseOne PostgreSQLIndexes, slow queries, connection poolRead/write splitting, one database per servicePartitioning, sharding
Caching—Redis cache-asideMulti-level caching, hot-key protectionPer-region deployment, capacity planning
Asynchronous workSynchronous is fineDatabase jobs tableMessage queue, OutboxEvent streaming, CDC
Queries and searchDatabase queriesDatabase queries + indexesRead replicasCQRS read models, search engine
Entry pointThe monolith handles itLoad balancer, CDNAPI GatewayGlobal traffic management, rate limiting and load shedding
ObservabilityStructured logs, basic metricsLatency, error rate, alertsDistributed tracing, per-service dashboardsService level objectives, capacity and cost metrics
OrganisationOne teamOne team, an owner per moduleTeams split by serviceBusiness teams + a platform team

Read this table as “from left to right, moving one column only when a signal appears”, not as “the goal is 10M, so fill in the rightmost column straight away”. Each row can move independently: many systems still have their database in the 100K column while their entry point already needs an API Gateway; some systems have very little traffic but need multiple regions because of regulations.

Step 7: Safely splitting out a service

Whatever stage you decide to split at, the split should be a series of small, reversible steps, not a single cut-over. Below, extracting inventory from the monolith serves as the example:

1 Find the boundary Make it a standalone module inside the monolith, called only through its interface 2 Data ownership Only this module reads and writes its tables; other modules go through its interface 3 Sync data Give the new service its own database and keep copying changes with Outbox or CDC 4 Shadow traffic Send the same request to the new service too; compare the results only, never use them 5 Switch reads Shift reads to the new service by percentage; writes stay in the monolith 6 Switch writes The new service becomes the data owner and syncs back to the monolith for a while 7 Remove old code Delete the monolith's code, tables and sync mechanism Each step can be rolled back on its own; the old path stays until step 7

Figure: seven steps from finding the boundary to deleting the old code; until the last step the old path is still there, and if anything goes wrong at any step you can roll back to the previous one.

1. Find the boundary. Inside the monolith, first turn inventory into a standalone module: other modules may call it only through the inventory.Service interface and may not touch its tables directly. This step happens entirely inside the monolith, and the compiler and tests confirm that no call site has been missed. If some features cannot be cleanly separated however you cut them — for example, order and inventory data always need to be updated in the same transaction — that is a sign the boundary is wrong: redraw it first, or accept eventual consistency using reservations and a Saga.

2. Establish data ownership. Inventory all the code that reads or writes the inventory tables and change it all to go through the inventory module's interface. Pay special attention to places that are easy to miss, such as reports, admin pages and batch jobs — you can create a dedicated database account for the inventory tables and revoke access from the other modules' accounts, so any missed access fails outright. At this point the data is still in the monolith's database, but it now has a single owner.

3. Build the new service and sync the data. The new inventory service has its own database. The monolith is still the owner of the data: all writes still happen in the monolith, and the changes are then synced to the new database. There are two main ways to sync:

  • Outbox: the monolith writes the business data and an event in the same transaction, a relay publishes it to the message queue, and the new service consumes the events to update its own data (“Distributed Transactions: Sagas and the Outbox Pattern”).
  • CDC: read the database's transaction log directly and turn table changes into events. No changes to the monolith's code are needed, but the events say “a row changed” rather than “what happened in business terms”.

What you must avoid is dual writes at the application layer: having the monolith write its own database first and then call the new service to write again. Any failure between the two writes leaves the two sides permanently inconsistent, and it is hard to notice. Before syncing starts, do a full migration of the historical data, then use the sync mechanism to catch up on subsequent changes; finally, write a reconciliation program that regularly compares the data on both sides.

4. Shadow traffic. Have api-gateway (or an adapter layer inside the monolith) send a copy of inventory read requests to the new service as well, but only compare the results, never use them: users still get the monolith's response, and mismatches are logged. This step finds gaps in data syncing, behavioural differences in edge cases and performance problems in the new service at no risk at all. Shadow traffic only works for reads; write requests cannot simply be duplicated, or stock would be decremented twice.

5. Switch reads. Once the comparison results are stable, have the new service answer read requests by percentage: first 1%, then 10%, then all of them. Writes stay in the monolith, and the new service's data is still updated by the sync mechanism. Watch out for sync lag here: a read right after stock was decremented may get the pre-sync number, so reads that need strong consistency still go to the monolith. Whenever you find a problem, just set the percentage back to 0.

6. Switch writes. This is the only step that is genuinely risky: the owner of the data changes from the monolith to the new service. A common approach is to pick a low-traffic window, briefly freeze inventory writes (or switch over product by product in batches), confirm that syncing has fully caught up, then point the write routing at the new service. After the switch, run a reverse sync for a while: the new service's changes are synced back to the monolith's old tables, so that reports that haven't moved yet keep working and rolling back is still possible. All operations on the write path must be idempotent, because retries around the switch may reach both sides at the same time.

7. Remove the old code. After observing for long enough (at least one full business cycle, such as a month-end close), delete the inventory code, the old tables and the reverse sync from the monolith. This step is often postponed indefinitely, and the result is a system that forever has two copies of the code and two copies of the data, costing more to maintain than before the split. Put it in the plan; only then is the split complete.

Tip: every step should have a clear “done condition” and “rollback method” written into the plan, for example “move to step 5 only after the shadow comparison mismatch rate stays below the threshold for a full week” or “switch back if the error rate rises within an hour of switching writes”. The most common way service splits fail is not that the technology can't do it, but that the system stays in an uncertain state for too long — two systems, two copies of the data, and nobody dares to delete either.

Common pitfalls

  • Premature microservices. Splitting into a dozen services at 10K or 100K means the team spends most of its time on deployments, network errors, cross-service debugging and data consistency, and product development actually slows down. This is the most common and most expensive mistake; the monolith itself can scale horizontally.
  • Treating scale numbers as thresholds. “Split once we reach a million users” is not a meaningful rule. Look at signals, not numbers: latency, database load, connection pool wait, the cost of coordinating deployments, how far failures spread.
  • Adding replicas before going stateless. With sessions in local memory and uploads on the local disk, once you add a load balancer users get logged out at random and files vanish at random.
  • Skipping indexes and query optimisation and going straight to caching or more hardware. A query missing an index merely delays the explosion on a bigger machine, and a cache hides the problem until the moment the cache expires.
  • Caches without TTLs or stampede protection. Inconsistent cached data is never corrected; the moment a hot key expires, the database is knocked over.
  • Forgetting replication lag after read/write splitting. The order a user just placed “disappears”, and the order status read before payment is stale. Reads that need strong consistency must go to the primary.
  • Splitting the service but not the database. Several services share the same database and tables, and the result is a distributed monolith: the deployments are separate, the coupling is undiminished, and you get network calls and partial failures on top.
  • Dual writes at the application layer. Writing your own database and then calling another service means a failure between the two steps causes permanent inconsistency. Use an Outbox or CDC.
  • Sharding too early or picking the wrong shard key. Sharding is almost irreversible; a wrong shard key makes most queries span shards and concentrates hot spots on a few shards. Exhaust read/write splitting, partitioning and vertical splitting first.
  • Jumping straight to active-active multi-region. Write conflicts, cross-region consistency and data residency are each major projects. Get multiple availability zones right and rehearse active-passive failover first, then talk about active-active.
  • Knowing how to scale but not how to say no. A system without rate limiting and load shedding doesn't just slow down when traffic exceeds capacity; it collapses entirely, and uncontrolled retries push it down further.
  • Stopping halfway through a split. The old code and old tables are never deleted, and two systems coexist forever. A split plan must include the last step.
  • Ignoring team structure. When service boundaries don't match team boundaries, every feature change needs several teams to coordinate — you get none of the benefits of microservices and pay all of the costs.

Key takeaways

  • Describe scale with several numbers: registered users, DAU, peak RPS and daily orders; the orders of magnitude for each stage are only illustrative, and what really decides whether to move on are the signals.
  • 10K: a modular monolith + one PostgreSQL + one deployment. Get boundaries, statelessness, externalised configuration and basic observability right, and don't split services.
  • 100K: stateless replicas behind a load balancer, connection pools sized by replica count, indexes and slow queries, Redis cache-aside (TTL with jitter, delete after write, stampede protection), a CDN, background work moved off the request path, observability as routine, and enforced module boundaries. Still a monolith.
  • 1M: read/write splitting with replication lag handled, a message queue, splitting out only the most painful services along bounded contexts (commonly payment and inventory), an API Gateway, a database owned by each service, Saga/Outbox/idempotency, Kubernetes with HPA, and teams aligned with service boundaries.
  • 10M: partitioning and then sharding (a fixed number of logical shards + a mapping table, watching for hot spots), CQRS read models and a search engine, event streaming, multiple availability zones before multiple regions, rate limiting and load shedding, capacity planning and load testing, a platform team and cost management. Most systems never need to get here.
  • Safely splitting out a service: find the boundary → establish data ownership → give the new service a database and sync it with an Outbox or CDC → compare with shadow traffic → switch reads by percentage → switch writes with reverse sync → remove the old code. Every step must be reversible.
  • Premature microservices are a common and expensive mistake; staying at the stage that fits your needs is not falling behind, it is sound engineering judgement.

Next

Once the architecture grows, the database is usually the first thing to strain: queries get slower, and the first fix that comes to mind is an index. But adding an index to a table with tens of millions of rows that is taking orders can, with one wrong step, lock the whole table and take the service down. The next article, “Adding Indexes Safely: Indexing Large Tables Without an Outage”, carries on into the data layer: how to find the index you need from evidence, how to design it, and how to add it safely to a large table.