Columns › 7. Pitfalls in practice › Part 27
Payment Pitfalls: Double Charges, Callbacks and Reconciliation
支付常見問題:重複扣款、回呼與對帳
The hardest part of payments is not failure but not knowing whether a charge succeeded. Ten common pitfalls, from double charges, state machines and timeouts to webhook verification and ordering, cancellations, amounts, refunds, reconciliation, ledgers and sensitive data, with a checklist.
In this article
- What this article solves
- Problem 1: Double charges
- Problem 2: States jump around and a success is overwritten by a failure
- Problem 3: Treating a timeout as a failure
- Problem 4: Webhooks that are forged, resent, out of order, or arrive before the synchronous response
- Problem 5: A webhook and a user cancellation happen at the same time
- Problem 6: Calculating amounts with float64
- Problem 7: Getting refunds and partial refunds wrong
- Problem 8: No reconciliation, so discrepancies pile up until they can't be traced
- Problem 9: Editing balances directly, with no ledger or audit log
- Problem 10: Card numbers and sensitive data in the logs
- Summary table
- Key takeaways
- Next
In the previous article, “Concurrency Pitfalls: From Data Races to Overselling”, we looked at the many kinds of concurrent read-modify-write problems in Go programs and databases. This article turns to the most expensive kind of concurrency problem: the kind that happens with money. When the online shop's payment service goes wrong, the result is not a broken layout; it is a customer charged twice, a merchant paid too little, or books that don't balance and have to be traced by hand, one record at a time.
What this article solves
Payment flows have three awkward properties that ordinary service calls don't. First, they involve an external payment provider whose systems we don't control: networks drop, responses are slow and notifications get resent. Second, they are asynchronous: once a charge request is sent, the final result often only arrives later through a callback (webhook); the synchronous response only means "received". Third, their mistakes are hard to undo: money charged twice has to go through a refund, and a missing ledger record is only discovered through reconciliation.
So the goal of a payment system is not "never make a mistake" but: doing the same thing several times counts only once (idempotency), every payment's state is known at all times (a state machine), an unclear result is never guessed (queries and reconciliation), and every change leaves a record (ledgers and audit logs). Each of the ten problems below happens when one of these rules is broken.
We continue with the series' example: orders in the order service have three states, pending, paid and cancelled, and the amount Total is stored as an int64 in minor units; the payment service charges the payment provider, a payment has its own finer-grained states, and on success it emits an event that moves the order to paid. Each problem is covered in the order "symptom, why, how to fix".
Problem 1: Double charges
Symptom
A customer reports that the same order was charged twice, with two identical amounts on their credit card statement; in the back office you see two successful payment records for the same order_id, only a few hundred milliseconds or a few seconds apart.
Why
Duplicate requests are almost impossible to avoid, and they come from at least four layers: the user double-clicks the pay button on a slow connection, or presses Back after paying and submits again; the HTTP client in the front end or app retries automatically after a timeout; the retry logic in api-gateway or between services (“Timeouts, Retries and Circuit Breakers”) resends when no response arrives; and the message queue's at-least-once delivery causes the "start charge" event to be consumed twice. If any one of these layers treats a resend as a new request, the payment service sends a second charge to the payment provider, and the provider sees two perfectly legitimate requests.
How to fix
The answer is the idempotency key from “Idempotency and Distributed Locks”, but the point is that it must be "end to end": the idempotency key is generated at the very top of the chain and passed all the way down. Concretely, when the checkout page loads, it generates an idempotency key representing "this one payment intent"; the button submit, front-end retries and gateway retries all carry the same value; the payment service uses it as a unique key when creating the payment record; and when calling the payment provider, it passes its own payment ID or idempotency key as the merchant order number or in the provider's idempotency key field, so the provider can block duplicates too.
CREATE TABLE payments (
id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
order_id bigint NOT NULL,
idempotency_key text NOT NULL UNIQUE,
amount bigint NOT NULL CHECK (amount > 0), -- minor units
currency char(3) NOT NULL, -- ISO 4217, e.g. 'USD'
status text NOT NULL DEFAULT 'created',
provider_txn_id text UNIQUE, -- the payment provider's transaction ID
refunded_amount bigint NOT NULL DEFAULT 0 CHECK (refunded_amount >= 0),
created_at timestamptz NOT NULL DEFAULT now(),
updated_at timestamptz NOT NULL DEFAULT now(),
CHECK (refunded_amount <= amount)
);
-- At most one payment per order that is "still in progress or succeeded"
CREATE UNIQUE INDEX payments_one_active_per_order ON payments (order_id)
WHERE status IN ('created', 'pending', 'unknown', 'succeeded', 'partially_refunded');
-- Create a payment: a second request with the same idempotency key inserts nothing
INSERT INTO payments (order_id, idempotency_key, amount, currency)
VALUES ($1, $2, $3, $4)
ON CONFLICT (idempotency_key) DO NOTHING
RETURNING id, status;
-- No row returned: look up the existing one; if order_id and amount match, return its current status;
-- if not, the same key is being reused for something else: return 422 and do not chargeThere are two lines of defence here. The idempotency key stops "resends of the same intent"; the partial unique index stops "different idempotency keys for the same order", for example when the user opens two tabs and checks out in each. You need both, and both must be database constraints rather than check-then-write in code, or you are back to the read-modify-write race from the previous article. The front end should also disable the button after submitting, but that only improves the experience; it can't be a line of defence.
Tip: when a duplicate request arrives while the first is still being processed, return "processing" and let the client query later, rather than waiting or doing it again; if there is already a result, return the previous result unchanged.
Problem 2: States jump around and a success is overwritten by a failure
Symptom
The payment provider's dashboard shows the charge succeeded, but our payment record says failed, the order is cancelled automatically, and the customer has paid but never receives the goods; or a payment that was fully refunded is changed back to succeeded.
Why
A payment's state comes from more than one source: the synchronous response, webhooks, active queries, reconciliation and manual operations. If every source runs UPDATE payments SET status = $2 WHERE id = $1 as soon as it gets a result, the final state depends on who wrote last rather than on the facts. The classic case: the failure notification for the first attempt arrives late and overwrites the successful retry; or the reconciliation job overwrites a newer state with stale data.
How to fix
Define an explicit state machine: which states may move to which, with everything else rejected.
Figure: a payment goes from created to pending, and the result may be succeeded, failed or the temporary unknown; only a successful payment can be refunded, and failed and refunded are final states.
Write the rules as code with tests, so every "not allowed" has a matching case:
package payment
import (
"fmt"
"testing"
)
type Status string
const (
Created Status = "created"
Pending Status = "pending"
Unknown Status = "unknown" // timeout or unclear result; wait for a query or webhook
Succeeded Status = "succeeded"
Failed Status = "failed"
PartiallyRefunded Status = "partially_refunded"
Refunded Status = "refunded"
)
// Only the allowed transitions are listed; anything not listed is rejected.
var allowed = map[Status][]Status{
Created: {Pending, Failed},
Pending: {Succeeded, Failed, Unknown},
Unknown: {Succeeded, Failed},
Succeeded: {PartiallyRefunded, Refunded},
PartiallyRefunded: {PartiallyRefunded, Refunded},
}
func CanTransition(from, to Status) bool {
for _, s := range allowed[from] {
if s == to {
return true
}
}
return false
}
func Transition(from, to Status) error {
if from == to && from != PartiallyRefunded {
return nil // duplicate notification: already in this state, treat as success and do nothing
}
if !CanTransition(from, to) {
return fmt.Errorf("illegal transition %s -> %s", from, to)
}
return nil
}
func TestTransition(t *testing.T) {
cases := []struct {
from, to Status
ok bool
}{
{Pending, Succeeded, true},
{Unknown, Succeeded, true},
{Succeeded, Succeeded, true}, // webhook sent twice
{Failed, Succeeded, false}, // a final state can't be flipped; needs manual handling
{Succeeded, Failed, false}, // a late failure notification can't overwrite success
{Created, Refunded, false}, // can't refund what was never paid
{Succeeded, PartiallyRefunded, true},
}
for _, c := range cases {
err := Transition(c.from, c.to)
if (err == nil) != c.ok {
t.Errorf("%s -> %s: err=%v, want ok=%v", c.from, c.to, err, c.ok)
}
}
}The check in code is only the first layer; when writing to the database, put the "current state" in the WHERE clause, for example UPDATE payments SET status = 'succeeded' WHERE id = $1 AND status IN ('pending', 'unknown'). If it affects 0 rows, re-read the row and decide whether it is "already in this state" (a duplicate notification, treat as success) or "an illegal transition" (log an alert and hand it to a person). This is the same technique as the conditional UPDATE that prevented overselling in the previous article: it stops two concurrent updates from overwriting each other.
There are two principles when designing the states. First, final states are never flipped automatically: if the payment provider says a payment succeeded after it was failed, that is an anomaly that needs manual handling, not an ordinary transition. Second, the states must include "don't know": unknown is not failure, and the next problem explains why it has to exist. The order's states stay simple: when a payment enters succeeded, an event is emitted through the outbox so that order moves the order from pending to paid, while the payment details stay in the payment service.
Problem 3: Treating a timeout as a failure
Symptom
A call to the payment provider times out, the program reports "payment failed", the user pays again, and both payments succeed; or the system automatically cancels the order and returns the stock, and only the next day do you find that the money had in fact been taken.
Why
A timeout only means "no response arrived within the time limit"; it doesn't mean the other side did nothing. The request may never have arrived, may have arrived and failed, or may have been charged successfully with the response lost on the way back. To the caller, these three cases look exactly the same. Treating a timeout as a failure means making the wrong decision in one of three possibilities, and the consequence of that mistake is charging the customer again or cancelling an order that has been paid.
How to fix
Timeouts, dropped connections and 5xx responses all go to unknown; they are never judged as failures directly. Once in unknown:
- Query the payment provider for the transaction's status using the same merchant order number or idempotency key, and only move to
succeededorfailedonce you have a definite result. - If the query also fails or the provider shows it as still processing, query again from a background job with backoff, while also waiting for the webhook.
- If it is still unclear after a while, hand it over to the next day's reconciliation and to manual handling; meanwhile the order stays
pending, the stock is not released, and the user is not allowed to pay again with a new idempotency key.
If you really do need to retry the charge itself, always reuse the same idempotency key so the payment provider treats it as the same charge; if the provider doesn't support idempotency keys, don't retry the charge automatically, only query. This is also why “Timeouts, Retries and Circuit Breakers” stresses that only idempotent operations may be retried automatically: for payments, a charge request without an idempotency guarantee is never retried.
What the user sees must be honest too: "Confirming your payment. Please don't pay again; we'll notify you once it's confirmed" is far better than a wrong "Payment failed, please try again".
Problem 4: Webhooks that are forged, resent, out of order, or arrive before the synchronous response
Symptom
Someone sends a "payment succeeded" request straight to the webhook URL and the order is marked as paid; the same payment sends two success emails; the webhook keeps being resent because we return 500 every time; or the synchronous response says "processing" and moves a payment that the webhook had already marked successful back to pending.
Why
The webhook URL is public and anyone can send requests to it, so an unverified webhook means anyone can mark an order as paid. Even when the source is trusted, webhook delivery, like a message queue, is "at least once": if the payment provider doesn't get a 2xx it resends, with intervals ranging from seconds to hours; networks and retries mean the arrival order of several notifications is not guaranteed to match the order in which the events happened; and the webhook and the synchronous response travel by different paths, so the webhook can perfectly well arrive before the synchronous response.
Figure: the webhook first moves the payment to succeeded, and the late synchronous response can't move it back to processing; the redelivered webhook has the same state, so it gets 200 straight away.
How to fix
Verify first, then apply idempotently. For verification: check the signature the way the payment provider specifies, commonly an HMAC with a shared secret over "timestamp + raw request body", or verifying a signature with the provider's public key; you must verify the raw bytes, not parse into a struct and serialise it back; use a constant-time comparison to avoid timing attacks; and check the timestamp, rejecting requests that are too old to prevent replays.
package webhook
import (
"crypto/hmac"
"crypto/sha256"
"encoding/hex"
"io"
"net/http"
"net/http/httptest"
"strconv"
"strings"
"testing"
"time"
)
const maxSkew = 5 * time.Minute
func sign(secret []byte, ts string, body []byte) string {
m := hmac.New(sha256.New, secret)
m.Write([]byte(ts + "."))
m.Write(body)
return hex.EncodeToString(m.Sum(nil))
}
// Handler only verifies; once verified it hands over to apply (check the state, update idempotently).
func Handler(secret []byte, now func() time.Time, apply func(body []byte) error) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
body, err := io.ReadAll(io.LimitReader(r.Body, 1<<20)) // take the raw bytes first; don't parse and re-serialise
if err != nil {
http.Error(w, "bad body", http.StatusBadRequest)
return
}
ts := r.Header.Get("X-Timestamp")
sec, err := strconv.ParseInt(ts, 10, 64)
if err != nil || now().Sub(time.Unix(sec, 0)).Abs() > maxSkew {
http.Error(w, "stale", http.StatusUnauthorized) // block replays
return
}
got, err := hex.DecodeString(r.Header.Get("X-Signature"))
want, _ := hex.DecodeString(sign(secret, ts, body))
if err != nil || !hmac.Equal(got, want) { // constant-time comparison
http.Error(w, "bad signature", http.StatusUnauthorized)
return
}
if err := apply(body); err != nil {
http.Error(w, "retry later", http.StatusInternalServerError) // let the provider resend
return
}
w.WriteHeader(http.StatusOK)
})
}
func TestHandler(t *testing.T) {
secret := []byte("test-secret")
now := time.Unix(1_800_000_000, 0)
h := Handler(secret, func() time.Time { return now }, func([]byte) error { return nil })
body := `{"payment_id":"pay_1","status":"succeeded","amount":1000}`
ts := strconv.FormatInt(now.Unix(), 10)
send := func(sig, ts string) int {
req := httptest.NewRequest(http.MethodPost, "/webhook", strings.NewReader(body))
req.Header.Set("X-Timestamp", ts)
req.Header.Set("X-Signature", sig)
rec := httptest.NewRecorder()
h.ServeHTTP(rec, req)
return rec.Code
}
if c := send(sign(secret, ts, []byte(body)), ts); c != 200 {
t.Errorf("valid: %d", c)
}
if c := send(sign([]byte("wrong"), ts, []byte(body)), ts); c != 401 {
t.Errorf("forged: %d", c)
}
old := strconv.FormatInt(now.Add(-time.Hour).Unix(), 10)
if c := send(sign(secret, old, []byte(body)), old); c != 401 {
t.Errorf("replayed: %d", c)
}
}Once verified, apply should:
- Not fully trust the webhook's content: check that the amount, currency and merchant order number in the webhook match the record in
payments; for important states, you can also confirm once more through the payment provider's query API. - Let the state machine decide whether to update: if it is already in the same state, do nothing and return 200; if the state is "older" (for example "processing" arrives after success), ignore it; log illegal transitions and raise an alert. Handling of the synchronous response goes through the same rules, so when it arrives late it doesn't move the state backwards.
- Store the "notifications received": write every webhook verbatim into a notification log table (including the provider's event ID), deduplicated with a unique constraint on the event ID; later investigations and reconciliation both rely on it.
- Respond quickly: return 2xx once it is verified and written to the database, and do follow-up work such as sending emails and updating the order asynchronously through the outbox (“Distributed Transactions: Sagas and the Outbox Pattern”). When processing fails, return 5xx so the provider resends; that is far better than returning 200 without having handled it.
Problem 5: A webhook and a user cancellation happen at the same time
Symptom
The user stays on the payment page too long, the order is cancelled on timeout and its stock released; a few seconds later the payment-succeeded webhook arrives, and you end up with "order cancelled, but money taken". Or the other way round: the user has just pressed cancel as the payment-succeeded webhook arrives, each side updates on its own, and you end up with the order cancelled and the payment succeeded.
Why
Cancellation and payment success are two independent events, each arriving by a different path, and neither knows about the other. If both sides do "read the order, check, write back", it is a classic read-modify-write race; if both use conditional updates, the database guarantees only one side wins, but the losing side still has to deal with what has already happened in the real world: the money has been taken and can't simply be ignored.
How to fix
Make the rules explicit and have both sides use conditional updates:
- Cancel the order:
UPDATE orders SET status = 'cancelled' WHERE id = $1 AND status = 'pending'. - Mark as paid after payment succeeds:
UPDATE orders SET status = 'paid' WHERE id = $1 AND status = 'pending'.
Only one of the two statements affects 1 row. If the payment-success side affects 0 rows and the order is already cancelled, that means "payment succeeded but order cancelled", and you must automatically issue a full refund (with an idempotency key), or, depending on business rules, try to restore the order (provided the stock is still there), and notify the user. The point is that this branch must be written out explicitly and tested, rather than being allowed to fall into "illegal transition, ignore".
Ways to make this less likely: query the payment status once before cancelling an order on timeout; keep the payment page's validity shorter than the order's hold time; and for payments already in pending or unknown, delay cancelling the order until the result is clear.
Problem 6: Calculating amounts with float64
Symptom
An order total shows as 299.99000000000001; discounting three items separately and adding them up comes to one cent less than discounting the whole order; reconciliation is off by a few cents and nobody can find out why.
Why
float64 is a binary floating-point number and can't represent decimal fractions like 0.1 exactly; 0.1 + 0.2 gives 0.30000000000000004. Every operation can add a little error, and it adds up into mistakes in comparisons, sums and reconciliation. Another common problem is not recording the currency: the same 1000 means very different amounts in different currencies, and currencies also have different numbers of minor-unit digits.
How to fix
Always store amounts as integer minor units (for example, US dollars counted in cents) in an int64, just like Order.Total, and use bigint in the database; always pass and store the amount together with its currency; keep the places that need rounding (discounts, tax, currency conversion, allocation) in explicit functions that state which rounding rule is used and who gets the remainder, guaranteeing that the parts add up to the original amount. When exchanging data with the payment provider, check whether its fields are in minor units or in whole units, and never send amounts as floating-point numbers in JSON.
There is a lot of detail in this topic: when you need a decimal type, how many decimal places each currency has, the difference between banker's rounding and ordinary rounding, and how to allocate without losing a cent. The next article, “Handling Money in Go: Minor Units, Decimals, Currencies, Rounding and Allocation”, covers it in full.
Problem 7: Getting refunds and partial refunds wrong
Symptom
The same payment is refunded twice; partial refunds add up to more than the original payment; customer support clicks refund in the back office, the screen says it succeeded, but it actually failed at the payment provider and the customer never gets the money.
Why
A refund is itself another payment operation, and problems 1 to 4 all happen to it again: repeated clicks and retries, unclear timeouts, asynchronous result notifications. On top of that, partial refunds have to be accumulated, and "read the refunded amount, add this one, check it isn't over, write back" is yet another read-modify-write.
How to fix
Treat refunds as a resource of their own: create a refunds table in which each refund has its own idempotency key, amount, status (pending, succeeded, failed, unknown) and the payment provider's refund ID. When issuing a refund, use one conditional update that does the check and reserves the amount at the same time:
UPDATE payments SET refunded_amount = refunded_amount + $2 WHERE id = $1 AND status IN ('succeeded', 'partially_refunded') AND refunded_amount + $2 <= amount
If it affects 0 rows, either the state doesn't allow it or the amount is too large; only after it succeeds do you call the payment provider. When a refund definitely fails, give the amount back; when it definitely succeeds, move the payment to partially_refunded or refunded based on the accumulated amount. The support interface must show the refund's real status, not "submitted". Also, many payment providers don't complete refunds immediately, and some have refund deadlines or don't support partial refunds; check these rules before issuing the refund, rather than letting the request fail after it is sent.
Problem 8: No reconciliation, so discrepancies pile up until they can't be traced
Symptom
At month-end settlement, the total collected according to our system doesn't match the amount paid out by the payment provider; a few of our payments are unknown and nobody has dealt with them; a few show as successful at the provider while we have no record of them at all.
Why
Even with idempotency, state machines and webhooks all done correctly, two independent systems will still drift apart: every webhook redelivery fails, a particular release has a bug, someone operates directly in the provider's dashboard, time zones and day boundaries differ, fees are deducted from payouts. Without regular comparison, these discrepancies only blow up all at once at month end, and the longer it takes, the harder it is to find the cause and the people involved.
How to fix
Set up daily reconciliation: every day, get the payment provider's reconciliation file (or pull the previous day's transaction details via the API) and match it against the internal payment and refund records one by one, by transaction ID.
Figure: after matching both sides by transaction ID, the results fall into four categories; matching ones are marked as reconciled, and the other three go into a discrepancy queue to be handled one by one.
The matching results roughly fall into four categories, each with its own handling:
Type Internal record Payment provider Common cause Handling
MATCHED succeeded 1000 succeeded 1000 — Mark as reconciled
MISSING_REMOTE succeeded (none) Different day boundary or time zone; wrongly judged succeeded Compare again next day; if still missing, query it and correct
MISSING_LOCAL (none) or unknown succeeded Every webhook failed; no follow-up after a timeout Confirm by query and record the state; refund if order cancelled
MISMATCH succeeded 1000 succeeded 900 or refunded Amount changed, partial refund not synced; fees Investigate one by one, correct the record and write an audit logA few implementation points: cut both sides' data with the same time zone and day boundary, and also bring in transactions from a window before and after, so transactions that cross midnight aren't wrongly flagged as missing; reconcile fees, payouts and refunds separately rather than only comparing totals; the reconciliation job itself must be idempotent, giving the same result when the same day is rerun; discrepancy handling needs an owner and a deadline, and every correction writes an audit log entry instead of editing the database directly. In the long run, the number and types of discrepancies are also a good monitoring metric: a sudden increase usually means something has broken.
Problem 9: Editing balances directly, with no ledger or audit log
Symptom
A user's wallet balance doesn't match the sum of their transaction history; a balance goes negative; support or an engineer edits an account's balance directly to fix the data, and afterwards nobody can say where that money came from or where it went.
Why
If the system has only a balance column and every top-up, payment and refund adds to or subtracts from it directly, that number is the only truth, and it has no history: any duplicate execution, read-modify-write race or manual edit can quietly make it wrong, and afterwards it can't be recomputed or proven correct.
How to fix
Use double-entry bookkeeping: each transaction is written as a set of entry lines, each line recording a debit or credit to one account, and for every transaction total debits must equal total credits. Money never appears or disappears out of nowhere; it only moves from one account to another.
Figure: the top-up and the payment each write two entry lines with equal debits and credits; the user wallet's balance is derived from the entries and can be recomputed and verified at any time.
CREATE TABLE accounts (
id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
owner text NOT NULL, -- e.g. user:7, merchant:9, platform:bank
currency char(3) NOT NULL,
balance bigint NOT NULL DEFAULT 0 -- a cached value derived from entries; can be recomputed at any time
);
CREATE TABLE journal_entries (
id bigint GENERATED ALWAYS AS IDENTITY PRIMARY KEY,
idempotency_key text NOT NULL UNIQUE, -- each business event is recorded only once
kind text NOT NULL, -- topup / payment / refund
created_at timestamptz NOT NULL DEFAULT now()
);
CREATE TABLE ledger_lines (
entry_id bigint NOT NULL REFERENCES journal_entries (id),
account_id bigint NOT NULL REFERENCES accounts (id),
debit bigint NOT NULL DEFAULT 0 CHECK (debit >= 0),
credit bigint NOT NULL DEFAULT 0 CHECK (credit >= 0),
CHECK ((debit = 0) <> (credit = 0)) -- each line is either a debit or a credit, not both
);
-- Payment of 3000 minor units: the wallet (account 7) pays the merchant payable (account 9), all in one transaction
BEGIN;
UPDATE accounts SET balance = balance - 3000 WHERE id = 7 AND balance >= 3000;
-- 0 rows affected = insufficient balance, ROLLBACK
INSERT INTO journal_entries (idempotency_key, kind) VALUES ('pay-order-1001', 'payment') RETURNING id;
-- assume the returned id is 42
INSERT INTO ledger_lines (entry_id, account_id, debit, credit) VALUES (42, 7, 3000, 0), (42, 9, 0, 3000);
UPDATE accounts SET balance = balance + 3000 WHERE id = 9;
COMMIT;
-- Daily check: find unbalanced entries (should always be empty)
SELECT entry_id FROM ledger_lines GROUP BY entry_id HAVING sum(debit) <> sum(credit);The rules boil down to a few: no entry, no balance change, with both done in the same database transaction; the balance is only a cache of the entries, regularly recomputed from them and compared; deductions use a conditional update to stop the balance going negative, exactly like preventing overselling in the previous article; entries are append-only, so a mistake is reversed by writing an opposite entry rather than by deleting or updating; and every entry carries an idempotency key so duplicate events aren't recorded twice.
The audit log is another layer: every manual operation (adjusting a balance, changing a state by hand, forcing a refund) must record who did it, when, why, what was changed, and the values before and after, and like the entries it is append-only. Manual adjustments themselves should also be done by writing entries, not by editing balance directly.
Problem 10: Card numbers and sensitive data in the logs
Symptom
Searching the logs turns up full credit card numbers, expiry dates or security codes; the span attributes in the tracing system contain the whole payment request; the error reporting service receives request bodies containing card numbers.
Why
For debugging, many services write the full request and response to the logs in middleware (the request logging in “net/http and Middleware in Go”), or print the whole struct on error. Logs, traces and error reports are usually copied into several systems, kept for a long time and open to many people; a card number appearing anywhere brings all of these systems into the audit scope of the Payment Card Industry Data Security Standard (PCI DSS), and a leak is a serious security incident.
How to fix
The most fundamental fix is to keep card numbers off our servers entirely: use the payment provider's hosted payment page or embedded hosted fields, so the card number goes straight from the user's browser to the payment provider and we only receive a token representing the card or the payment. This greatly reduces the scope that has to comply with PCI DSS. Supporting measures include:
- Use an allow list for logs, recording only explicitly listed fields (payment ID, order ID, amount, status, error code) rather than the whole request body; the structured logging in “Observability in Go: Logs, Metrics and Tracing” suits this well.
- Implement
slog.LogValuerorString()on the types of sensitive values so they are masked automatically when printed. - If the raw content of webhooks has to be kept, store it in a table with controlled access, not in ordinary logs.
- Keep the webhook signing secret and the payment provider's API keys in secret management (“Configuration and Graceful Shutdown in Go”), never in code or container images, and rotate them regularly.
- The security code (CVV) must never be stored under any circumstances.
Summary table
| # | Problem | Why it happens | How to avoid it |
|---|---|---|---|
| 1 | Double charges | Double clicks, client and gateway retries, message redelivery | End-to-end idempotency keys, unique constraints, one active payment per order |
| 2 | States jumping around | Several sources overwrite the state directly | An explicit state machine, conditional UPDATEs, final states never flipped |
| 3 | Timeout treated as failure | The result is actually unknown on a timeout | Move to unknown, query, retry with the same idempotency key, reconcile |
| 4 | Forged, resent, out-of-order webhooks | Public URL, at-least-once delivery, different paths | Signature and timestamp checks, compare content, apply idempotently via the state machine |
| 5 | Webhook racing with cancellation | Two independent events each doing read-modify-write | Conditional updates on both sides, explicit handling for the loser (automatic refund) |
| 6 | float64 amounts | Binary floating point can't represent decimals exactly | int64 minor units, with currency, centralised and explicit rounding |
| 7 | Refunds calculated wrongly | Refunds also get duplicated, time out and are asynchronous | Refunds as their own resource, idempotency keys, conditional accumulation |
| 8 | No reconciliation | Two systems will inevitably drift apart | Daily matching by transaction ID, a discrepancy queue, audit logs |
| 9 | Balances without a ledger | A single number has no history and can't be recomputed | Double-entry bookkeeping, debits equal credits, append-only |
| 10 | Sensitive data leaks | Logs and traces record whole requests | Hosted payment fields, log allow lists, masking and secret management |
Key takeaways
Before going live, you can check the following list item by item:
- The idempotency key is generated on the checkout page and passed all the way to
paymentand the payment provider; the database has unique constraints, with at most one active payment per order. - Payments and refunds both have explicit state machines; every write uses a conditional UPDATE that includes the current state, with tests covering every forbidden transition.
- Timeouts, dropped connections and 5xx responses always go to
unknown, with queries and background follow-up; a charge request without an idempotency guarantee is never retried automatically. - Webhooks have their signatures verified (raw bytes, constant-time comparison, timestamp), their amount and currency compared and are deduplicated by event ID, and late, resent, out-of-order webhooks and ones that arrive before the synchronous response are all handled.
- "Payment succeeded but order cancelled" has an explicit handling path (automatic refund or restoring the order) and tests.
- Amounts are
int64minor units with a currency, and rounding and allocation live in explicit functions. - Refunds have their own idempotency keys and states, and accumulated refunds use a conditional update to prevent over-refunding.
- Daily reconciliation matches by transaction ID, and discrepancies have a queue, an owner and a deadline.
- Wallets and balances use double-entry bookkeeping: no entry, no balance change; entries are append-only; debits and credits are checked for balance and balances recomputed regularly.
- Every manual operation has an audit log entry; card numbers never pass through our own servers, logs only record fields on the allow list, and secrets live in secret management.
Next
This article fixed ten common problems one by one, but real duplicate transactions rarely come from just one place: a user double-clicks, the app retries after a timeout, the gateway retries, the queue redelivers, and if any layer fails to stop it, an extra transaction order appears; and when something really goes wrong, what you do first and what you do next decides how much it costs. The next article, “Preventing Duplicate Payments and Handling Payment Incidents: From the Button to the Provider, and How to Stop the Bleeding”, follows a payment from the button all the way to the provider, stops duplicates at every layer, and then covers prevention, alerting and the response steps for payment incidents.