This resume is also available in English.

7. Pitfalls in practice3 / 5 · All columns

Columns › 7. Pitfalls in practice › Part 28

Preventing Duplicate Payments and Handling Payment Incidents: From the Button to the Provider, and How to Stop the Bleeding

支付防重複與事故應變:從按鈕到金流商,出事時怎麼止血

Duplicate payment orders can enter at any layer, from the button and client to the gateway, services, provider and webhooks; this article shows how each layer stops them. It then covers preventing, alerting on and responding to payment incidents, with Go code for query-before-retry and a stuck-payment sweeper.

  • 支付
  • 冪等
  • 事故
  • Go
In this article
  1. What this article solves
  2. Part 1: Where duplicates come from
  3. Layer 1: The pay button
  4. Layer 2: The client (web and app)
  5. Layer 3: Gateways, load balancers and retries between services
  6. Layer 4: The order service, one checkout makes one order
  7. Layer 5: The payment service, one attempt at a time per order
  8. Layer 6: The payment provider
  9. Layer 7: Webhooks and message redelivery
  10. Layer 8: What the user sees
  11. Wiring layers 4 and 5 together in Go
  12. Part 2: Payment incidents
  13. What payment incidents look like
  14. Prevention beforehand: keep incidents small
  15. Payment states: who can end unknown
  16. The incident runbook
  17. The stuck-payment sweeper
  18. Key takeaways
  19. Next

The previous article, “Payment Pitfalls: Double Charges, Callbacks and Reconciliation”, fixed ten payment problems one at a time: idempotency keys, state machines, timeouts that go to unknown, webhook signature checks, reconciliation and ledgers. Those are the parts; this article puts them together and answers two more practical questions. First, a payment passes through seven or eight layers between the user pressing the button and the provider actually charging the card, so at which layer do duplicate transaction orders appear, and what does each layer have to stop? Second, even when every layer is right, providers still go down, webhooks still stop arriving and a new release can still get amounts wrong, so when something really goes wrong, how do you notice, how do you stop the bleeding, and what do you do first?

What this article solves

Problem 1 of the previous article already said that "the idempotency key has to go end to end", but only in one paragraph. In practice the most common double charges don't come from a layer that does nothing; they come from every layer doing "roughly the right thing": the button is disabled but no idempotency key is generated, the client sends a key but generates a new one after a timeout, the payment service deduplicates but after a timeout charges again with a new merchant trade number. So Part 1 walks down the layers one by one, and for each layer answers three things: where duplicates come from, what this layer's guard is, and what this layer must not rely on.

Part 2 deals with incidents. Payment incidents are different from ordinary outages: when an ordinary service goes down, fixing it restores everything; when payments go wrong, every charge, cancellation and refund that happened during the incident is real money that moved, and after fixing the code you still have to recover the state of every single payment. This part covers the common kinds of incident, prevention and alerting beforehand, a runbook for responding during the incident, and finally a sweeper in Go that recovers the results of stuck payments.

We continue with the series' example: the online shop example.com/shop, where users go through api-gateway to the order service to create an order, the payment service charges the payment provider and on success emits an event through the Outbox that moves the order to paid, and the inventory service releases stock when an order is cancelled; data lives in PostgreSQL, and amounts are always int64 minor units. Payment states follow the previous article's state machine: created, pending, unknown, succeeded, failed, plus the refund states.

Part 1: Where duplicates come from

Start with the whole picture. A checkout passes through these layers on its way from the screen to the provider, and each layer has its own way of producing duplicates:

Layer Where duplicates come from This layer's guard Pay button Double clicks, resubmit after back Disable the button; checkout_token made on load Client Retry on timeout, app resends on resume Every retry reuses the same idempotency key Gateway / LB No answer, so send to another instance Retry only idempotent calls, at one layer Order service Two orders from one checkout Unique constraint (user_id, checkout_token) Payment service New number and charge again after a timeout One number per attempt; query before retry Provider A new number is a new transaction Rejects duplicate numbers or keys Webhooks, queue At-least-once delivery Dedupe by transaction ID or event ID What users see "Payment failed" invites paying again Show "being confirmed" when unknown Upper layers: fewer duplicates, better experience Lower layers: guarantee a single charge

Figure: From top to bottom are the eight layers a payment passes through; on the left is why this layer produces duplicates, on the right is this layer's guard. The lower the guard, the more reliable it is; the upper ones only reduce how often duplicates happen.

The diagram has an important direction: the higher the guard, the less you can trust it; the lower, the more reliable. A disabled button can be bypassed (reload, open two tabs, call the API directly), and the client's idempotency key depends on the client implementing it correctly, but a database unique constraint cannot be bypassed. So the upper guards exist to reduce the number of duplicates and improve the experience, and what really guarantees "charge only once" has to be database constraints and the state machine. Conversely, the lower guards alone are not enough either: they stop the duplicates, but users see a pile of error messages, and support calls will swamp you first.

Layer 1: The pay button

Where duplicates come from: on a slow network, users think the tap didn't register and tap again; touch screens often fire twice for one tap. After paying, pressing back or reloading can make the browser submit the form again.

The guard: disable the button and show "processing" as soon as it is pressed, until a result arrives; and generate a checkout_token (a random UUID) when the checkout page loads, carried by the form and every API request. Note that it is generated once on load, not each time the button is pressed: ten requests from ten clicks carry the same token. If the user really wants to buy again, they enter checkout again and get a new token.

What not to rely on: the disabled button. It exists only in the memory of this one tab; reloading, opening a second tab or reconnecting after a network drop all defeat it, and anyone calling the API doesn't have to go through the button at all. Disabling the button is for the experience, not a guard.

Layer 2: The client (web and app)

Where duplicates come from: the HTTP client retries automatically after a timeout or a dropped connection; an app sent to the background is suspended by the system and sends the request again when it returns to the foreground; an offline queue resends once the connection is back.

The guard: every retry reuses the same checkout_token as the idempotency key (for example in an Idempotency-Key header), however many retries and however long apart. The most common mistake is to generate the key inside the "send request" function, so that every retry generates a new key and idempotency is completely lost. The key belongs to "this payment intent" and lives in the checkout page's state (an app has to store it somewhere that survives the system killing the process), not to "this HTTP request".

The client also has to handle a "processing" response correctly: when the server returns 202 or 409 (a request with the same key is still in progress, see “Idempotency and Distributed Locks”), the client should switch to polling the payment status, not send the payment request again.

What not to rely on: that the client is always implemented correctly. Old app versions, third-party integrations and tampered requests can send no key or the wrong key, so the server-side guards cannot assume the idempotency key always exists and is always right; payment requests without a key should simply be rejected.

Layer 3: Gateways, load balancers and retries between services

Where duplicates come from: api-gateway, load balancers, service-mesh sidecars and gRPC clients can all be configured to "send again to another instance if upstream doesn't answer". The user pressed once, but the payment service receives two requests, possibly on two different instances, processed at the same time.

The guard: follow the principles of “Timeouts, Retries and Circuit Breakers”: retry only idempotent requests, and retry at only one layer. Applied to payments:

  • GET to query a payment's status can be retried.
  • POST to create a payment and charge may be retried only if the request carries an idempotency key and the downstream is known to deduplicate on it; otherwise the middle layers never retry and pass the error up.
  • The errors that are safe to retry are those where the request was definitely not processed: connection refused, or a 503 where the other side makes clear nothing was processed. A timeout, 502, 504 or a connection dropped after sending all mean "we don't know if it happened", and only idempotent requests may be retried.
  • 4xx is not retried (except 429, where you wait for Retry-After), and neither are business results (card declined).

Many gateways and load balancers retry some errors automatically by default, without looking at the HTTP method; before going live, check the defaults of the components you actually use, and explicitly turn them off or restrict them on payment paths.

What not to rely on: that the middle layers know which requests are idempotent. They only see the HTTP method and path, so the safest choice is for the middle layers on payment paths not to retry at all, with retries happening only where the meaning is understood: the client carrying the idempotency key, and the payment service itself, which queries before deciding.

Layer 4: The order service, one checkout makes one order

Where duplicates come from: duplicate requests that slip through the three layers above all end up as "create order" being called twice. If the order service INSERTs a new order every time, one checkout produces two orders, each paid separately, and the previous article's constraint of "at most one active payment per order" can't stop it, because they are two different orders.

The guard: use a database unique constraint to guarantee "one checkout intent maps to one order". Continuing with the orders table from “database/sql, Transactions and Connection Pools in Go”, add checkout_token:

ALTER TABLE orders ADD COLUMN checkout_token text;
CREATE UNIQUE INDEX orders_one_per_checkout ON orders (user_id, checkout_token);

-- Create the order: the same user and token arriving a second time inserts nothing
INSERT INTO orders (user_id, total, checkout_token)
VALUES ($1, $2, $3)
ON CONFLICT (user_id, checkout_token) DO NOTHING
RETURNING id;
-- No row returned: read the existing order and return it if the amount matches;
-- if not, the same token is being used for something else, so return 422

The unique key is (user_id, checkout_token) rather than just the token so that "someone else's token" can never hit your order, and a lookup checks the owner along the way. Once the order exists, its state can only go pending → paid or pending → cancelled following the previous article's rules, written with a conditional UPDATE that includes the current state.

What not to rely on: "SELECT to see if it exists, INSERT if not". Two concurrent requests both see "doesn't exist" and each insert an order, which is exactly the read-modify-write race from “Concurrency Pitfalls: From Data Races to Overselling”. Deduplication must rest on a unique constraint.

Layer 5: The payment service, one attempt at a time per order

This is the most critical layer in the chain, because it is the only place that sends charges to the provider directly. Any duplicate that slipped past the layers above can still be stopped here; if it slips through here, the money really is charged twice.

Where duplicates come from: three sources. First, concurrency: two payment requests for the same order arrive at the same time on two instances. Second, retries after a timeout: the first charge times out, the code treats it as a failure, and creates a new payment record with a new merchant trade number to charge again. Third, paying again: the first attempt really failed (card declined) and the user pays again with another card, which is a legitimate second attempt, but only once it is certain that the first one really failed.

The second source is the most common and the easiest to overlook, so it is worth drawing:

Wrong: new number after a timeout Payment service Provider 1. charge A01 charged, response lost ✕ 2. timeout, treated as failure 3. charge A02 (new number) 4. succeeded Result: charged twice Right: query the same number first Payment service Provider 1. charge A01 charged, response lost ✕ 2. timeout -> unknown 3. query A01 4. already succeeded Result: charged once, marked succeeded

Figure: On the left, after a timeout the charge is sent again with a new merchant trade number, which the provider sees as two different transactions, and both succeed; on the right, the service first queries with the same number, sees that the first attempt already succeeded, and does not charge again.

The guards are four, which together mean "each order has only one payment attempt at a time, and before moving to the next attempt, the previous one's result is always confirmed":

  1. Each attempt has a unique merchant trade number (many providers call it a merchant trade number or merchant order ID, and usually require it to be unique within a merchant). The number is made of "order ID + attempt number": resending the same attempt always reuses the same number, and only a new attempt after a definite failure gets a new one. Then, as long as the provider rejects duplicate numbers, there is one more guard.
  2. At most one in-progress attempt per order at a time: reuse the partial unique index payments_one_active_per_order on the previous article's payments table.
  3. Query before retry: when an attempt's result is unknown, never start the next charge directly; query the provider with the same merchant trade number. If it succeeded, mark it succeeded; only a definite failure allows a new attempt; if the transaction can't be found (the provider never got it), resend with the same number; if the query itself fails or the provider says it's still processing, do nothing and answer "being confirmed".
  4. Every state transition is conditional: changing pending to succeeded carries WHERE status = 'pending', and whoever loses the race reads the state again before deciding, so the two sides never both charge.

In the table, that means two more columns on the previous article's payments table, with the merchant trade number used directly as the idempotency key:

ALTER TABLE payments ADD COLUMN attempt_no int NOT NULL DEFAULT 1;
ALTER TABLE payments ADD COLUMN trade_no text UNIQUE;  -- the merchant trade number sent to the provider
ALTER TABLE payments ADD CONSTRAINT payments_one_row_per_attempt UNIQUE (order_id, attempt_no);

-- Create attempt n: inserted only for the first attempt, or if the previous one definitely failed
INSERT INTO payments (order_id, attempt_no, trade_no, idempotency_key, amount, currency, status)
SELECT $1, $2, $3, $3, $4, $5, 'pending'
WHERE $2 = 1
   OR EXISTS (SELECT 1 FROM payments WHERE order_id = $1 AND attempt_no = $2 - 1 AND status = 'failed')
ON CONFLICT DO NOTHING
RETURNING id;

ON CONFLICT DO NOTHING names no columns, so a conflict on any unique constraint (the same attempt, the same number, an order that already has an in-progress payment) makes this insert return no row, and the caller knows it is not the one that charges.

Layer 6: The payment provider

Where duplicates come from: even when every layer on our side is right, the provider still just does what it's asked: two charge requests with different numbers are two legitimate transactions.

The guard: use the provider's deduplication as the last line of insurance, not the only one. Two patterns are common: either the merchant trade number must be unique within a merchant, and a duplicate number is rejected or returns the original transaction; or the API supports an idempotency key header, and within a retention period a request with the same key returns the first result. Which one applies, what error a duplicate returns and how long keys are kept differ from provider to provider, so go by its official documentation, and write a test that confirms the "duplicate number" error is interpreted by us as "go and query the original", not as "failed".

What not to rely on: that the provider will always stop it. Some payment methods don't deduplicate, some idempotency keys are only kept for a while, some behave differently when the request body differs; and once we change the number ourselves, the provider has no reason at all to think it's a duplicate.

Layer 7: Webhooks and message redelivery

Where duplicates come from: the provider's webhooks are delivered at least once and resent until a 2xx comes back; our own "payment succeeded" event goes through a message queue, and consumers can receive it twice too (“Message Queues and Event-Driven Design”). Duplicates at this layer don't cause a second charge, but they cause a second shipment, a second notification email, a second ledger entry.

The guard: deduplicate webhooks by the provider's transaction ID or event ID and apply them idempotently through the state machine (problem 4 of the previous article); the payment service updates the state and writes the Outbox event in the same database transaction, and consumers deduplicate by writing the event ID into an Inbox (“Distributed Transactions: Sagas and the Outbox Pattern”). Every consumer that "does something when an event arrives" should ask itself: if the same event arrives twice, does it do the thing twice?

Layer 8: What the user sees

The last layer is not in the code but in the user's eyes. Many double payments are actually induced by the interface: show "payment failed, please try again" after a timeout, and of course the user pays again.

The guard:

  • When the result is unknown, show "payment is being confirmed; please don't pay again; we'll notify you once it's confirmed", with a link to check the order status; don't show "failed".
  • The order detail page shows the real payment status (being confirmed, succeeded, failed), not just what the front end remembers.
  • While an order has an in-progress payment, don't show a "pay again" button; show it only after the payment has definitely failed, and pressing it goes through Layer 5's "new attempt" flow.
  • Support scripts and the back office agree: when a customer calls to say "I think I paid twice", support should see each attempt's status and the provider's transaction ID, not a vague "paid".

Wiring layers 4 and 5 together in Go

Below, a standalone internal/payment package implements "one checkout, one order" and "query before retry". To keep the example runnable, data lives in memory, but every place that needs mutual exclusion is marked with the unique constraint or conditional UPDATE it corresponds to in the database.

// internal/payment/checkout.go
package payment

import (
	"context"
	"errors"
	"fmt"
	"sync"
)

type Status string

const (
	Pending   Status = "pending"   // charge sent or about to be sent, result not known yet
	Unknown   Status = "unknown"   // timeout, dropped connection, 5xx: we don't know if it happened
	Succeeded Status = "succeeded" // the provider explicitly said it succeeded
	Failed    Status = "failed"    // the provider explicitly said it failed (e.g. card declined)
)

var (
	ErrDeclined   = errors.New("provider: declined")                     // a definite failure
	ErrNotFound   = errors.New("provider: trade not found")              // no such merchant trade number
	ErrProcessing = errors.New("payment is being confirmed")             // shown to the user as "being confirmed", HTTP 202
	ErrKeyReused  = errors.New("checkout token reused, content differs") // HTTP 422
)

// Provider is the adapter for the payment provider. Of Charge's errors only ErrDeclined
// means "definitely not charged"; any other error (timeout, dropped connection, 5xx) means unknown.
type Provider interface {
	Charge(ctx context.Context, tradeNo string, amount int64, currency string) (txnID string, err error)
	Query(ctx context.Context, tradeNo string) (status Status, txnID string, err error)
}

type Order struct {
	ID       int64
	UserID   int64
	Token    string // checkout_token: generated once when the checkout page loads
	Amount   int64  // minor units
	Currency string
}

type Attempt struct {
	OrderID int64
	No      int    // which payment attempt this is for the order
	TradeNo string // the merchant trade number sent to the provider
	Status  Status
	TxnID   string // the provider's transaction ID
}

// TradeNo is built from the order ID and the attempt number: resending the same attempt reuses
// the same number, and only a new attempt after a definite failure gets a new one. Length and
// allowed characters depend on the provider's documentation.
func TradeNo(orderID int64, attempt int) string {
	return fmt.Sprintf("S%010dA%02d", orderID, attempt)
}

// Service simulates the database in memory: every place that needs the lock is a unique
// constraint or a conditional UPDATE in the database.
type Service struct {
	p        Provider
	mu       sync.Mutex
	nextID   int64
	orders   map[string]*Order // unique constraint (user_id, checkout_token)
	byID     map[int64]*Order
	attempts map[int64]*Attempt // each order's current attempt
}

func NewService(p Provider) *Service {
	return &Service{p: p, orders: map[string]*Order{}, byID: map[int64]*Order{}, attempts: map[int64]*Attempt{}}
}

// Checkout creates at most one order per user and checkout token.
func (s *Service) Checkout(userID int64, token string, amount int64, currency string) (Order, error) {
	if token == "" || amount <= 0 || len(currency) != 3 {
		return Order{}, errors.New("invalid checkout")
	}
	s.mu.Lock()
	defer s.mu.Unlock()
	key := fmt.Sprintf("%d/%s", userID, token)
	if o, ok := s.orders[key]; ok {
		if o.Amount != amount || o.Currency != currency {
			return Order{}, ErrKeyReused
		}
		return *o, nil // a resend: return the same order
	}
	s.nextID++
	o := &Order{ID: s.nextID, UserID: userID, Token: token, Amount: amount, Currency: currency}
	s.orders[key], s.byID[o.ID] = o, o
	return *o, nil
}

// Pay can be called any number of times; an order is charged successfully at most once.
func (s *Service) Pay(ctx context.Context, orderID int64) (Attempt, error) {
	a, o, err := s.claim(ctx, orderID)
	if err != nil || a.Status == Succeeded {
		return a, err
	}
	txnID, err := s.p.Charge(ctx, a.TradeNo, o.Amount, o.Currency)
	switch {
	case err == nil:
		got, _ := s.cas(a, Pending, Succeeded, txnID)
		return got, nil
	case errors.Is(err, ErrDeclined):
		got, _ := s.cas(a, Pending, Failed, "")
		return got, err
	default: // timeout or another error: don't guess, leave it to the next call or the sweeper
		got, _ := s.cas(a, Pending, Unknown, "")
		return got, ErrProcessing
	}
}

// claim decides whether this call charges and with which number. A Pending attempt in the
// result means "this call is the one that charges".
func (s *Service) claim(ctx context.Context, orderID int64) (Attempt, Order, error) {
	for {
		if err := ctx.Err(); err != nil {
			return Attempt{}, Order{}, err
		}
		o, a, err := s.load(orderID)
		if err != nil {
			return Attempt{}, Order{}, err
		}
		switch {
		case a == nil:
			if got, ok := s.create(orderID, 1); ok {
				return got, o, nil
			}
		case a.Status == Succeeded:
			return *a, o, nil // already paid: return it as is
		case a.Status == Pending:
			return *a, o, ErrProcessing // another request is charging right now: never charge again
		case a.Status == Failed:
			if got, ok := s.create(orderID, a.No+1); ok { // only a definite failure gets a new number
				return got, o, nil
			}
		case a.Status == Unknown:
			st, txnID, err := s.p.Query(ctx, a.TradeNo) // query before retry
			switch {
			case errors.Is(err, ErrNotFound): // the provider never got it: resend with the same number
				if got, ok := s.cas(*a, Unknown, Pending, ""); ok {
					return got, o, nil
				}
			case err != nil || (st != Succeeded && st != Failed):
				return *a, o, ErrProcessing // query failed or still processing: do nothing
			default:
				s.cas(*a, Unknown, st, txnID) // apply the result, the next loop acts on the new state
			}
		}
		// lost the race (another request changed the state first): read again
	}
}

func (s *Service) load(orderID int64) (Order, *Attempt, error) {
	s.mu.Lock()
	defer s.mu.Unlock()
	o, ok := s.byID[orderID]
	if !ok {
		return Order{}, nil, fmt.Errorf("order %d not found", orderID)
	}
	if a, ok := s.attempts[orderID]; ok {
		cp := *a
		return *o, &cp, nil
	}
	return *o, nil, nil
}

// create is INSERT ... ON CONFLICT (order_id, attempt_no) DO NOTHING.
func (s *Service) create(orderID int64, no int) (Attempt, bool) {
	s.mu.Lock()
	defer s.mu.Unlock()
	cur := s.attempts[orderID]
	if (cur == nil && no != 1) || (cur != nil && (cur.No != no-1 || cur.Status != Failed)) {
		return Attempt{}, false
	}
	a := &Attempt{OrderID: orderID, No: no, TradeNo: TradeNo(orderID, no), Status: Pending}
	s.attempts[orderID] = a
	return *a, true
}

// cas is UPDATE ... SET status = to WHERE order_id = ? AND attempt_no = ? AND status = from.
func (s *Service) cas(a Attempt, from, to Status, txnID string) (Attempt, bool) {
	s.mu.Lock()
	defer s.mu.Unlock()
	cur := s.attempts[a.OrderID]
	if cur == nil || cur.No != a.No || cur.Status != from {
		return a, false
	}
	cur.Status = to
	if txnID != "" {
		cur.TxnID = txnID
	}
	return *cur, true
}

The key points:

  • Pay can be called again any number of times: a double click, a client retry, a gateway resend, a user coming back a few minutes later and pressing again all go through the same entry point, and claim decides whether this call charges.
  • Only the call that wins Pending actually calls Charge; other concurrent calls see Pending and return ErrProcessing (shown to the user as "being confirmed"), without waiting and without charging.
  • Of the errors Charge returns, only an explicit decline moves the state to Failed; timeouts, dropped connections and every other error go to Unknown.
  • When the state is Unknown, the next call first calls Query with the same TradeNo. If it succeeded, there is no new charge; only "not found" resends with the same number; a failed query does nothing.
  • A new number is created only after the previous attempt is definitely Failed.

The heart of the tests is a fake provider where "the charge succeeds but the response times out". Every Charge call really creates a charge and it does not deduplicate for us, so "charge only once" depends entirely on our own code:

// internal/payment/checkout_test.go
package payment

import (
	"context"
	"errors"
	"fmt"
	"sync"
	"testing"
	"time"
)

// fakeProvider really creates a charge on every Charge call (it does not deduplicate for us),
// so "charge only once" depends entirely on our own code.
type fakeProvider struct {
	mu        sync.Mutex
	charges   []string          // the merchant trade number of every charge made
	status    map[string]Status // merchant trade number -> result
	timeout   bool              // the charge succeeds but the response times out
	declineN  int               // the first N charges are declined
	chargeDur time.Duration
}

func (f *fakeProvider) Charge(ctx context.Context, tradeNo string, amount int64, currency string) (string, error) {
	time.Sleep(f.chargeDur)
	f.mu.Lock()
	defer f.mu.Unlock()
	if f.status == nil {
		f.status = map[string]Status{}
	}
	if f.declineN > 0 {
		f.declineN--
		f.status[tradeNo] = Failed
		return "", ErrDeclined
	}
	f.charges = append(f.charges, tradeNo)
	f.status[tradeNo] = Succeeded
	if f.timeout {
		f.timeout = false // only the first one times out
		return "", context.DeadlineExceeded
	}
	return fmt.Sprintf("txn-%d", len(f.charges)), nil
}

func (f *fakeProvider) Query(ctx context.Context, tradeNo string) (Status, string, error) {
	f.mu.Lock()
	defer f.mu.Unlock()
	st, ok := f.status[tradeNo]
	if !ok {
		return "", "", ErrNotFound
	}
	return st, "txn-of-" + tradeNo, nil
}

func (f *fakeProvider) count() int {
	f.mu.Lock()
	defer f.mu.Unlock()
	return len(f.charges)
}

func TestTimeoutThenRetryChargesOnce(t *testing.T) {
	p := &fakeProvider{timeout: true}
	s := NewService(p)
	ctx := context.Background()
	o, _ := s.Checkout(7, "tok-1", 1999, "USD")

	a, err := s.Pay(ctx, o.ID)
	if !errors.Is(err, ErrProcessing) || a.Status != Unknown {
		t.Fatalf("first try: %v %v, want unknown + ErrProcessing", a.Status, err)
	}
	// the user or client retries: it queries first, sees success and does not charge
	a, err = s.Pay(ctx, o.ID)
	if err != nil || a.Status != Succeeded {
		t.Fatalf("retry: %v %v", a.Status, err)
	}
	if n := p.count(); n != 1 {
		t.Fatalf("charged %d times, want 1", n)
	}
}

// The wrong way: treat a timeout as failure and charge again with a new number.
func TestNaiveRetryChargesTwice(t *testing.T) {
	p := &fakeProvider{timeout: true}
	for i := 1; i <= 2; i++ {
		if _, err := p.Charge(context.Background(), fmt.Sprintf("naive-%d", i), 1999, "USD"); err == nil {
			break
		}
	}
	if n := p.count(); n != 2 {
		t.Fatalf("naive retry charged %d times; this test documents the bug", n)
	}
}

func TestDoubleClickConcurrent(t *testing.T) {
	p := &fakeProvider{chargeDur: 20 * time.Millisecond}
	s := NewService(p)
	var wg sync.WaitGroup
	for range 10 { // ten clicks on the same checkout page
		wg.Go(func() {
			o, err := s.Checkout(7, "tok-2", 1999, "USD")
			if err != nil {
				t.Error(err)
				return
			}
			if _, err := s.Pay(context.Background(), o.ID); err != nil && !errors.Is(err, ErrProcessing) {
				t.Error(err)
			}
		})
	}
	wg.Wait()
	if n := p.count(); n != 1 {
		t.Fatalf("charged %d times, want 1", n)
	}
	if s.nextID != 1 {
		t.Fatalf("%d orders, want 1", s.nextID)
	}
}

func TestDeclinedThenNewAttempt(t *testing.T) {
	p := &fakeProvider{declineN: 1}
	s := NewService(p)
	o, _ := s.Checkout(7, "tok-3", 1999, "USD")
	if _, err := s.Pay(context.Background(), o.ID); !errors.Is(err, ErrDeclined) {
		t.Fatalf("want declined, got %v", err)
	}
	a, err := s.Pay(context.Background(), o.ID) // pay again with another card
	if err != nil || a.No != 2 || a.TradeNo == TradeNo(o.ID, 1) {
		t.Fatalf("second attempt: %+v %v", a, err)
	}
}

func TestTokenReusedWithOtherAmount(t *testing.T) {
	s := NewService(&fakeProvider{})
	s.Checkout(7, "tok-4", 1999, "USD")
	if _, err := s.Checkout(7, "tok-4", 2999, "USD"); !errors.Is(err, ErrKeyReused) {
		t.Fatalf("want ErrKeyReused, got %v", err)
	}
}

TestNaiveRetryChargesTwice deliberately shows the wrong way: treat a timeout as failure, charge again with a new number, and the customer is charged twice. TestDoubleClickConcurrent uses ten goroutines to simulate ten clicks on the same checkout page and runs under -race to confirm that only one order is created, only one charge is made, and there is no data race (sync.WaitGroup.Go needs Go 1.25 or later):

$ go test -race ./...
ok  	example.com/shop/internal/payment	1.2s
ok  	example.com/shop/internal/sweeper	1.3s

Tip: in a real implementation, Pending can get stuck forever if the process crashes in the middle of calling Charge. Then claim just keeps returning ErrProcessing and never charges again; recovering its result is the job of the "stuck-payment sweeper" in Part 2. Better stuck waiting for someone than risking a second charge.

Part 2: Payment incidents

The guards of Part 1 make double charges rare, but payment systems depend on too many outside factors, and sooner or later there will be an incident. The point is not to guarantee nothing ever goes wrong but that incidents stay small, are noticed early, and handling them doesn't make things worse.

What payment incidents look like

The common kinds of payment incident are roughly these, and each shows a different first signal:

IncidentWhat happensFirst signal
Provider fully downEvery charge request fails or times outPayment success rate drops sharply, circuit breaker opens
Provider slowResponses go from a few hundred milliseconds to over ten seconds; timeouts and retries surge and drag us down tooCall latency rises, timeout rate rises, more unknown
Payments stuck in processingMany payments stay in pending or unknown, and users don't know whether they paidNumber of payments stuck for over 15 minutes
Double charges have happenedOne order has two successful charges at the providerDouble-charge detection query, complaints
Webhooks stopThe provider succeeded but we never get the notificationTime since the last webhook, unknown piling up
Reconciliation differences jumpThe number or amount of daily reconciliation differences suddenly growsReconciliation report
A new release gets amounts wrongAfter a deploy, payment amounts don't match order amounts, the currency is wrong, decimals are wrongPayments blocked by amount sanity checks, a sudden shift in the amount distribution
Certificate or key rotation failsSignature checks fail and every webhook is rejected, or authentication to the provider failsWebhook signature failure rate, the provider returning 401/403

What these have in common: most payment incidents don't bring the service down. HTTP responses are normal, CPU is normal, and the error rate may not even be high (a webhook failing its signature check returns 401, which looks like "someone sending junk"); only business metrics show it. That is why the alerts below are mainly business metrics.

Prevention beforehand: keep incidents small

Per-provider timeouts, circuit breakers and retry budgets

As in “Timeouts, Retries and Circuit Breakers”, each provider has its own timeouts, circuit breaker and connection pool, so one slow provider doesn't drag down every payment, or features that have nothing to do with payments. What is special about payments: when the circuit breaker is open, the user is told "payments are unavailable right now, please try again later", and these requests were definitely not sent, so the user can safely retry later; a timed-out request, on the other hand, is "unknown" and goes to unknown.

Retry budgets matter even more. When a provider slows down, retrying every timed-out request multiplies the traffic sent to it and makes it slower, a vicious circle. The payment service only resends charges under "query before retry" anyway, and the query requests need their own budget and rate limit too.

Switches: able to stop, able to switch

Prepare a few switches (feature flags) in advance that take effect within seconds without a deploy:

  • Stop new payments: the checkout page shows "payments are under maintenance", and payments already in progress are unaffected. It's the last resort, but it has to exist, and it has to be drilled regularly to confirm it really works.
  • Switch provider: with several providers, send new payments to the backup one. Switch only new payments; payments already in progress with the old provider still have to be queried and completed there.
  • Pause automation: background jobs such as automatic order cancellation on timeout, automatic refunds and automatic retries can make wrong decisions during an incident (for example cancelling an order that was actually paid but whose webhook hasn't arrived). Each must be pausable on its own.

Rate limits, canaries and amount sanity checks

  • Rate limits: limit payment requests per user, per card and per IP, to stop scripted attacks and runaway client retries; also respect the provider's rate limits on us, so our own bursts don't get us blocked.
  • Canary deploys: code that touches the payment path first serves a small share of traffic with the new version, and business metrics such as payment success rate and amount distribution decide whether to expand (“Automating Deployments and Troubleshooting with AI Workflows”). Bugs in payment code often don't bring the service down; they just make amounts or states wrong, so the canary's criteria must include business metrics.
  • Amount sanity checks: in the payment service, check one last time before sending the charge: the amount is greater than zero, doesn't exceed the limit for this merchant or product type, the currency is on the allow list, and the payment amount equals the order amount. If the check fails, reject and alert rather than sending it and letting the provider decide. This check turns "a new release got amounts wrong" from "thousands of wrong charges made" into "thousands blocked, with an alert".

Alert on business metrics

Technical metrics (error rate, latency, CPU) are still needed, but the most important payment alerts are business metrics. Below are some common alert rules in plain terms; the numbers are only a starting point, to be tuned to your own traffic and history:

MetricAlert condition (in plain terms)Possible incident
Payment success rateOver the last 10 minutes, the success rate is 10 or more percentage points below the same time last week, with at least 50 payments in the windowProvider outage, a bad release, a certificate problem
Stuck paymentsPayments in pending or unknown for over 15 minutes: more than 0 for 10 minutes in a rowSlow provider, webhooks stopped, a crashed process
Successful charges per orderAny order with two or more successful charges at the provider: notify immediatelyDouble charges
Payment attempts per orderMore than 3 attempts on one order, or the average attempts per order suddenly risingRunaway client retries, cards declined en masse
Webhook lagPayments are in progress, but the last webhook was 15 minutes agoWebhooks stopped, signature failures, URL blocked
Webhook signature failure rateMore than 1% of webhooks fail the signature checkKey rotation failed, someone forging webhooks
Payments blocked by amount checksMore than 0 in the last 10 minutesA release getting amounts wrong
Reconciliation differencesDaily differences more than 3 times the 30-day average, or the difference amount above a thresholdAny kind of silent error

Alerts like "double charges" and "blocked by amount checks" should have a threshold of 0: they don't happen in normal operation, so a single occurrence deserves a look. The other alerts must not be too sensitive, or after too many night-time pages, people start ignoring them.

Payment states: who can end unknown

During an incident, most of the work is recovering the results of payments stuck in pending and unknown. First, make the rule clear: unknown can only end with a result "confirmed with the provider", and several sources can do that confirming; they all use the same conditional transitions, so whoever arrives first, none overwrites another.

Processing pending Succeeded succeeded Result unknown unknown Failed failed Query the provider user retry, webhook, sweeper, recon, manual definite success timeout, dropped connection, 5xx failed found succeeded not found: resend with same number Definite decline (e.g. card declined) To pay again after failed, create a new attempt with a new merchant trade number

Figure: Timeouts, dropped connections and 5xx move a payment to unknown; user retries, webhooks, the sweeper, reconciliation and manual handling can only move it to succeeded or failed after confirming with the provider; if the transaction isn't found, it is resent with the same number.

The incident runbook

The easiest mistake during an incident is rushing to "fix it" and skipping confirmation: seeing a pile of unknown and marking them all failed, seeing double charges and refunding them in bulk. The order below is deliberate; each step gets the right information for the next one.

1. Detect alerts, complaints, provider notices 2. Declare name a commander, start the timeline 3. Stop the bleeding switches, rollback, pause automation 4. Communicate "being confirmed, don't pay again" 5. Scope the timeline and SQL queries 6. Confirm query API, statements into a provider table 7. Fix sweeper, orders, idempotent refunds 8. Reconcile, review rerun reconciliation, blameless review Before step 6: no state changes, no refunds, no cancellations

Figure: From detecting, declaring, stopping the bleeding and communicating, to scoping, confirming state with the provider and fixing, then reconciling and a blameless review; no refund or cancellation happens before the state is confirmed.

1. Detect

Sources are the alerts above, complaints reported by support, and the provider's status page or notifications. Whoever gets it first does the smallest check: is it one user's problem, or do many payments show the same thing?

2. Declare the incident

As soon as money is involved and more than one user is affected, declare an incident; don't wait to be sure of the cause. Assign an incident commander who makes the decisions and coordinates, open a dedicated channel, and start a timeline from this moment: when it was found, what was done when, what was seen. The review afterwards and the scoping both rely on the timestamps in this timeline.

3. Stop the bleeding

The goal is to stop the damage from spreading, not to find the root cause:

  • Provider down or slow: the circuit breaker should already be open; if needed, use the switches to move to the backup provider, or stop new payments.
  • A bug from a new release: roll back immediately; don't fix forward in the middle of an incident.
  • Pause automation that may make wrong decisions: automatic order cancellation on timeout, automatic refunds, automatic retries.
  • Webhooks stopped or failing signature checks: check the webhook URL, certificates and keys; webhooks are usually resent, so the backlog arrives once it's fixed.

Stopping the bleeding does not include changing data. At this stage, don't change the state of any payment or order.

4. Communicate

Give support and users the right message: "some payment results are being confirmed; please don't pay again; we'll let you know once they're confirmed". The checkout page and order pages show the same message. Update status internally at regular intervals, even if there's no progress: "no progress; next update in 30 minutes".

5. Scope what's affected

Using the timestamps in the timeline and database queries, list the payments and orders that may be affected. Some common queries:

-- Stuck payments: counts and the oldest time, by status
SELECT status,
       count(*)                                                        AS payments,
       count(*) FILTER (WHERE updated_at < now() - interval '15 minutes') AS older_than_15m,
       min(updated_at)                                                 AS oldest
FROM payments
WHERE status IN ('pending', 'unknown')
GROUP BY status;

-- Payment succeeded but the order is cancelled (victims of automatic cancellation during the incident)
SELECT o.id AS order_id, p.id AS payment_id, p.amount, p.currency, p.provider_txn_id
FROM orders o
JOIN payments p ON p.order_id = o.id
WHERE o.status = 'cancelled'
  AND p.status = 'succeeded'
  AND p.updated_at >= $1;  -- when the incident started

-- Payment amount different from the order amount during a deploy
SELECT p.id AS payment_id, p.order_id, o.total, p.amount
FROM payments p
JOIN orders o ON o.id = p.order_id
WHERE p.created_at BETWEEN $1 AND $2  -- from the deploy to the rollback
  AND p.amount <> o.total;

6. Confirm the real state with the provider

This is the most important step of the whole process: during an incident, the state in our own database can't be trusted. unknown may actually be a success, failed may have been written wrongly by buggy code, and successes during a webhook outage weren't recorded at all. So before any fix, use the provider's query API (or statement files) to get the real state of every relevant transaction, and store it in a table:

-- Transactions synced from provider queries, webhooks and statements; the provider's transaction ID is authoritative
CREATE TABLE provider_transactions (
  provider_txn_id text        PRIMARY KEY,
  trade_no        text        NOT NULL,     -- the merchant trade number we sent
  status          text        NOT NULL,     -- the status the provider reports
  amount          bigint      NOT NULL,     -- minor units
  currency        char(3)     NOT NULL,
  created_at      timestamptz NOT NULL,     -- the transaction time on the provider's side
  synced_at       timestamptz NOT NULL DEFAULT now()
);
CREATE INDEX provider_transactions_trade_no ON provider_transactions (trade_no);

With this table, double charges can be found with one query: an order with two or more successful transactions at the provider.

-- Double-charge detection: orders with two or more successful charges at the provider
SELECT p.order_id,
       count(*)                                                  AS succeeded_charges,
       sum(t.amount)                                             AS charged_total,
       array_agg(t.provider_txn_id ORDER BY t.created_at)        AS provider_txn_ids
FROM provider_transactions t
JOIN payments p ON p.trade_no = t.trade_no
WHERE t.status = 'succeeded'
  AND t.created_at >= now() - interval '7 days'
GROUP BY p.order_id
HAVING count(*) > 1;

-- Successful transactions the provider has and we have no record of at all: trace by hand
SELECT t.*
FROM provider_transactions t
LEFT JOIN payments p ON p.trade_no = t.trade_no
WHERE t.status = 'succeeded' AND p.id IS NULL;

The detection query doubles as everyday monitoring: run it every 15 minutes, and if the result is not empty, trigger the "successful charges per order" alert above.

Payments stuck in pending and unknown are handed to the sweeper in the next section, which queries the provider one by one.

7. Fix: confirm first, then fix one by one, idempotently

The order and principles of fixing:

  • Stuck payments: the sweeper moves them to succeeded or failed according to the query result, through the same conditional transitions.
  • Payment succeeded but order cancelled: restore the order according to the business rules (if stock is still available) or refund in full, and notify the user.
  • Double charges: keep the charge recorded on the order and refund the rest. Refunds must be idempotent and recorded: each charge to refund gets a fixed idempotency key and is written into the refunds table, and a refund worker then sends it to the provider following problem 7 of the previous article. Rerunning the script, or two people running it at once, never refunds twice.
CREATE TABLE refunds (
  id              bigint      GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
  provider_txn_id text        NOT NULL,                  -- the charge to refund
  idempotency_key text        NOT NULL UNIQUE,
  amount          bigint      NOT NULL CHECK (amount > 0),
  currency        char(3)     NOT NULL,
  reason          text        NOT NULL,
  status          text        NOT NULL DEFAULT 'pending',
  created_by      text        NOT NULL,                  -- who decided it, for the audit trail
  created_at      timestamptz NOT NULL DEFAULT now()
);

-- Refund only successful charges that are not the one recorded on the order; one charge never gets more than one refund record
INSERT INTO refunds (provider_txn_id, idempotency_key, amount, currency, reason, created_by)
SELECT t.provider_txn_id, 'dup-refund:' || t.provider_txn_id, t.amount, t.currency,
       'duplicate_charge', $1                            -- e.g. the incident ID plus who ran it
FROM provider_transactions t
JOIN payments p    ON p.trade_no = t.trade_no
JOIN payments kept ON kept.order_id = p.order_id
                  AND kept.status IN ('succeeded', 'partially_refunded', 'refunded')
WHERE t.status = 'succeeded'
  AND t.provider_txn_id <> kept.provider_txn_id
ON CONFLICT (idempotency_key) DO NOTHING
RETURNING provider_txn_id, amount;

Why you can't refund in bulk first: refunding according to our own database's state before confirming it causes two kinds of error. One is refunding what shouldn't be: treating an unknown that was actually the only successful charge as a duplicate and refunding it, so the customer gets both the refund and the goods. The other is missing refunds: a charge that succeeded while webhooks were down has no record on our side, and looking only at our own database you will never find it. A refund is real money moving too; a wrong refund means collecting the money from the customer again, which is even harder to handle than charging twice. So the order is always: confirm → record → execute idempotently → confirm again.

8. Reconcile and review

Once the incident is handled, rerun a full reconciliation for the incident window and confirm every difference has been handled or has an owner. Then hold a blameless postmortem: rebuild what happened from the timeline, why the existing guards didn't stop it, why the alert was late (or why there was none), which steps took too long; the action items get owners and deadlines, for example "add an alert on webhook lag", "query the payment status before automatically cancelling an order", "add a pre-validation step to key rotation". The review looks at the system and the process, not at the person who pressed the deploy button: if one person's single mistake can cause an incident, what needs fixing is the system that let the mistake happen.

The stuck-payment sweeper

Payments stuck in pending and unknown exist in small numbers all the time (a lost webhook, a process restarted in the middle of a charge), and during an incident thousands can pile up at once. The sweeper is a background job that runs periodically, queries the provider for each one, and changes the state only on a definite result.

Start a run before = now - MinAge Take a batch: id > after ORDER BY id LIMIT n No rows: done for each row Wait for the rate limit; stop if ctx is cancelled Query the provider with the same trade number Succeeded or failed conditional update; no change = handled Still processing no change, next run Query failed no change, count error after = last id of the batch

Figure: Each run first fixes a time cutoff, then takes batch after batch in id order; before each query it waits for the rate limit, and only a succeeded or failed result leads to a conditional update, where no change means a webhook already handled it; it can be cancelled at any point.

Some design points:

  • Only touch old enough payments: payments newer than MinAge (for example 15 minutes) are still in the normal flow or waiting for a webhook, so don't compete with them. The cutoff is fixed at the start of each run so that a run doesn't keep chasing payments that newly became stuck.
  • Keyset pagination: take rows with id > last row of the previous batch, not OFFSET. While the sweep runs, earlier payments leave the "stuck" set as they're handled; with OFFSET the whole set shifts forward and some payments are skipped, while keyset pagination isn't affected and scans every row exactly once.
  • Rate limit: leave at least a minimum gap between provider queries. During an incident the provider may already be struggling, and firing thousands of queries at once gets you rate limited, or makes it even slower.
  • Conditional transitions: a payment is changed only if its state is still the one seen during the scan; no change means a webhook or a user retry handled it first, which is normal, not an error.
  • No answer, no change: a failed query or a provider still processing leaves the payment as it is, to be queried again next run. The sweeper never "guesses" a result.
  • Can be stopped and rerun: every step watches ctx, so it stops at once on a deploy or shutdown (“Configuration and Graceful Shutdown in Go”); because every write is conditional, rerunning it or running two instances at once does no harm beyond wasting some queries. If you really want to avoid duplicate queries, use a lease from “Idempotency and Distributed Locks” so only one instance runs at a time.

In PostgreSQL, the queries and the update look like this:

-- Index only stuck payments, so a scan doesn't read the whole table
CREATE INDEX payments_stuck ON payments (id) WHERE status IN ('pending', 'unknown');

-- Stuck: keyset pagination
SELECT id, trade_no, status
FROM payments
WHERE status IN ('pending', 'unknown')
  AND id > $1               -- the id of the last row in the previous batch
  AND updated_at < $2       -- the cutoff fixed at the start of this run
ORDER BY id
LIMIT $3;

-- Transition: change only if the state is still the one seen during the scan
UPDATE payments
SET status = $3, provider_txn_id = coalesce(NULLIF($4, ''), provider_txn_id), updated_at = now()
WHERE id = $1 AND status = $2;
-- 1 row affected: changed; 0 rows affected: someone else handled it first

The Go side depends on only two interfaces, so both the database and the provider can be swapped for test fakes:

// internal/sweeper/sweeper.go
package sweeper

import (
	"context"
	"time"
)

type Payment struct {
	ID      int64
	TradeNo string
	Status  string // pending or unknown
}

type Store interface {
	// Stuck returns up to limit pending/unknown payments with id > afterID, last updated before
	// before, in ascending id order.
	Stuck(ctx context.Context, afterID int64, before time.Time, limit int) ([]Payment, error)
	// Transition changes the status only if it is still from; it reports whether it did.
	Transition(ctx context.Context, id int64, from, to, txnID string) (bool, error)
}

type Provider interface {
	Query(ctx context.Context, tradeNo string) (status, txnID string, err error)
}

type Sweeper struct {
	Store    Store
	Provider Provider
	MinAge   time.Duration // anything newer is left to the normal flow and webhooks
	Batch    int
	Interval time.Duration // minimum gap between queries, to protect the provider and avoid its rate limit
	Now      func() time.Time
}

type Result struct {
	Scanned, Resolved, AlreadyDone, StillPending, QueryErrors int
}

// Run does one pass: it queries the provider once per stuck payment and changes the status only
// on a definite result. It can be stopped at any time and rerun; conditional updates stop a
// payment from being changed twice.
func (s *Sweeper) Run(ctx context.Context) (Result, error) {
	var r Result
	tick := time.NewTicker(s.Interval)
	defer tick.Stop()
	before := s.Now().Add(-s.MinAge) // fixed at the start, so a run doesn't chase new rows
	var after int64
	for {
		batch, err := s.Store.Stuck(ctx, after, before, s.Batch)
		if err != nil || len(batch) == 0 {
			return r, err
		}
		for _, p := range batch {
			select {
			case <-ctx.Done():
				return r, ctx.Err()
			case <-tick.C: // rate limit
			}
			r.Scanned++
			st, txnID, err := s.Provider.Query(ctx, p.TradeNo)
			switch {
			case err != nil:
				r.QueryErrors++ // no answer, no change; try again next run
			case st == "succeeded" || st == "failed":
				ok, err := s.Store.Transition(ctx, p.ID, p.Status, st, txnID)
				if err != nil {
					return r, err
				}
				if ok {
					r.Resolved++
				} else {
					r.AlreadyDone++ // a webhook or a user retry got there first
				}
			default:
				r.StillPending++
			}
		}
		after = batch[len(batch)-1].ID // keyset: the next batch starts after the last row
	}
}

Once Transition succeeds, the event that moves the order to paid, the notification email and the ledger entries all go out through the Outbox exactly as in the normal flow; the sweeper only "recovers the result" and doesn't have its own follow-up processing, so the two paths never behave differently.

The tests confirm three things: every old enough stuck payment is queried exactly once (keyset pagination neither skips nor repeats), the one that a webhook handles first during the scan isn't overwritten, and the ones whose query fails are left as they are; plus that cancelling ctx stops the run at once:

// internal/sweeper/sweeper_test.go
package sweeper

import (
	"context"
	"errors"
	"fmt"
	"sort"
	"sync"
	"testing"
	"time"
)

var t0 = time.Date(2026, 11, 15, 12, 0, 0, 0, time.UTC)

type row struct {
	Payment
	updated time.Time
}

type fakeStore struct {
	mu   sync.Mutex
	rows map[int64]*row
}

func (f *fakeStore) Stuck(ctx context.Context, afterID int64, before time.Time, limit int) ([]Payment, error) {
	f.mu.Lock()
	defer f.mu.Unlock()
	var out []Payment
	for _, r := range f.rows {
		if r.ID > afterID && r.updated.Before(before) && (r.Status == "pending" || r.Status == "unknown") {
			out = append(out, r.Payment)
		}
	}
	sort.Slice(out, func(i, j int) bool { return out[i].ID < out[j].ID })
	return out[:min(limit, len(out))], nil
}

func (f *fakeStore) Transition(ctx context.Context, id int64, from, to, txnID string) (bool, error) {
	f.mu.Lock()
	defer f.mu.Unlock()
	r := f.rows[id]
	if r.Status != from {
		return false, nil
	}
	r.Status = to
	return true, nil
}

type fakeProvider struct {
	mu      sync.Mutex
	queried map[string]int
	before  func(tradeNo string) // runs before a query, to simulate a webhook getting there first
}

func (f *fakeProvider) Query(ctx context.Context, tradeNo string) (string, string, error) {
	if f.before != nil {
		f.before(tradeNo)
	}
	f.mu.Lock()
	f.queried[tradeNo]++
	f.mu.Unlock()
	var n int
	fmt.Sscanf(tradeNo, "T%d", &n)
	switch {
	case n%7 == 0:
		return "", "", errors.New("provider 503")
	case n%5 == 0:
		return "pending", "", nil
	case n%2 == 0:
		return "failed", "", nil
	default:
		return "succeeded", fmt.Sprintf("txn-%d", n), nil
	}
}

func TestSweepEveryStuckPaymentOnce(t *testing.T) {
	st := &fakeStore{rows: map[int64]*row{}}
	for id := int64(1); id <= 25; id++ {
		updated := t0.Add(-time.Hour)
		if id > 22 {
			updated = t0.Add(-time.Minute) // too new, this run leaves it alone
		}
		st.rows[id] = &row{Payment{id, fmt.Sprintf("T%d", id), "unknown"}, updated}
	}
	p := &fakeProvider{queried: map[string]int{}}
	p.before = func(tradeNo string) {
		if tradeNo == "T3" { // while the sweeper queries, a webhook marks payment 3 succeeded first
			st.Transition(context.Background(), 3, "unknown", "succeeded", "txn-3")
		}
	}
	s := &Sweeper{Store: st, Provider: p, MinAge: 15 * time.Minute, Batch: 10,
		Interval: time.Millisecond, Now: func() time.Time { return t0 }}

	r, err := s.Run(context.Background())
	if err != nil {
		t.Fatal(err)
	}
	if r.Scanned != 22 || len(p.queried) != 22 {
		t.Fatalf("scanned %d, queried %d distinct, want 22", r.Scanned, len(p.queried))
	}
	for no, n := range p.queried {
		if n != 1 {
			t.Errorf("%s queried %d times", no, n)
		}
	}
	if r.AlreadyDone != 1 || r.Resolved+r.AlreadyDone+r.StillPending+r.QueryErrors != 22 {
		t.Fatalf("result %+v", r)
	}
	if got := st.rows[14].Status; got != "unknown" { // a failed query changes nothing
		t.Fatalf("payment 14 is %s", got)
	}
	again, _ := s.Run(context.Background()) // a rerun only touches the ones still without a result
	if again.Resolved != 0 || again.Scanned != r.StillPending+r.QueryErrors {
		t.Fatalf("second run %+v", again)
	}
}

func TestSweepStopsOnCancel(t *testing.T) {
	st := &fakeStore{rows: map[int64]*row{1: {Payment{1, "T1", "pending"}, t0.Add(-time.Hour)}}}
	s := &Sweeper{Store: st, Provider: &fakeProvider{queried: map[string]int{}}, MinAge: time.Minute,
		Batch: 10, Interval: time.Hour, Now: func() time.Time { return t0 }}
	ctx, cancel := context.WithTimeout(context.Background(), 10*time.Millisecond)
	defer cancel()
	r, err := s.Run(ctx)
	if !errors.Is(err, context.DeadlineExceeded) || r.Scanned != 0 {
		t.Fatalf("got %+v %v", r, err)
	}
}

The fake provider decides results by the trade number (multiples of 7 fail to query, multiples of 5 are still processing, even numbers failed, the rest succeeded), and all 22 old enough stuck payments are scanned once; the second run only touches the ones still without a result. In production, also give the sweeper metrics: how many it scanned per run, how many it resolved, how many queries failed, how old the oldest stuck payment is; these numbers feed the "stuck payments" alert above.

Key takeaways

  • Duplicate transaction orders can come from eight layers: the button, the client, gateway and service retries, the order service, the payment service, the provider, webhooks and message redelivery, and an interface that nudges users into paying again. The upper guards reduce how often it happens and improve the experience; what really guarantees one charge is database constraints and the state machine.
  • The idempotency key belongs to "this payment intent": generated once when the checkout page loads, with every retry carrying the same key; orders are unique on (user_id, checkout_token).
  • Middle layers retry only idempotent requests, and only one layer retries; on payment paths, the middle layers are best not retrying at all.
  • In the payment service, each attempt has a unique merchant trade number and resends of the same attempt reuse it; when the result is unknown, query before retry, and only a definite failure gets a new number.
  • When the result is unknown, show the user "being confirmed", not "failed".
  • Most payment incidents don't bring the service down, so alerts rely on business metrics: success rate, stuck payments, double charges, webhook lag, signature failures, amount checks and reconciliation differences.
  • Prepare switches beforehand (stop payments, switch provider, pause automation), per-provider timeouts and circuit breakers, retry budgets, rate limits, canaries and amount sanity checks.
  • The order of incident handling: detect, declare, stop the bleeding, communicate, scope, confirm the state with the provider, fix idempotently, reconcile and hold a blameless review. No refunds and no state changes before the state is confirmed.
  • The stuck-payment sweeper: a fixed cutoff, keyset pagination, a rate limit, conditional transitions, no answer means no change, stoppable and rerunnable at any time.
CheckDone?
checkout_token generated once when the checkout page loads☐
Every retry carries the same idempotency key; payment requests without one are rejected☐
Gateways and load balancers on payment paths don't retry automatically☐
Orders have a (user_id, checkout_token) unique constraint☐
Each attempt has a unique merchant trade number, reused on resend☐
Unknown results are queried first; only a definite failure creates a new attempt☐
Webhook and event consumers deduplicate by ID☐
The screen says "being confirmed" when the result is unknown☐
Switches to stop payments, switch provider and pause automation work and have been drilled☐
Amount sanity checks before a charge is sent☐
Business-metric alerts and the double-charge detection query☐
A stuck-payment sweeper and its metrics☐
Runbook: no refunds before the state is confirmed☐

Next

All of this article's incident handling comes down to one thing: during an incident our own data can't be trusted, and only by taking the provider's records as authoritative and matching them one by one do we learn which payments are stuck, which are duplicated, and which we never recorded at all. The sweeper handles the few we "know are stuck", while reconciliation is about finding the ones we "don't know are wrong". The next article, “Reconciliation: Matching Payment Provider Statements Reliably”, does reconciliation properly from start to finish: how to get the statements, how to match them by transaction ID, how to handle day boundaries and fees, and how to track each difference until it's closed.