Hard · reference design

Payment webhook processor

Turn payment provider webhooks into correct order and ledger updates: never lose an event, never apply one twice, and survive events that arrive late, out of order or not at all.

1 Requirements

Functional

Receive webhooks from a payment provider (payment succeeded, refunded, disputed).
Verify that every webhook really comes from the provider.
Update orders and the ledger exactly once in effect, whatever the delivery order.
Find and repair events that were never delivered.

Non-functional

Never lose an event, and keep the raw payload for audit and replay.
Answer within the provider timeout, a few seconds at most.
Correctness over speed: a payment applied twice or not at all is an incident.

Out of scope: Checkout and card collection. Accounting reports.

2 Core entities

Webhook event: The raw payload, its provider event id, type and signature.
Inbox row: One per event id: payload, received at, status, attempts, last error.
Payment: Internal state: pending, paid, refunded or disputed.
Ledger entry: An immutable record of money moving, in integer minor units.

3 API

POST /webhooks/{provider}

Verifies the signature, stores the event and returns 2xx immediately.

POST /admin/events/{id}/replay

Re-runs a stored event after a fix. Safe because handlers are idempotent.

4 Data flow

The provider sends a signed event.
The endpoint verifies it, stores it in the inbox and answers 2xx.
A processor picks it up and applies it to payments and the ledger in one transaction.
A daily reconciliation compares the ledger with the provider and repairs gaps.

5 High-level design

Press play to follow one request through the system. Every step explains what happens and the trade-off behind the choice.

Payment providerSigned events

Webhook endpointVerify, store, 2xx

Inbox tableUnique event id

Work queueRetries, backoff

Dead lettersAlert + replay

ProcessorIdempotent handler

ReconciliationDaily vs provider

Payments + ledgerOne transaction

Follow one "payment succeeded" event from the provider to the ledger. Press play, or step through the stages to see what happens and what each choice costs.

6 Deep dives

How do we handle events that arrive out of order?

Option	Gives	Costs
Apply events in arrival order	Simple.	A late event can overwrite a newer state.
Compare event timestamps	No extra calls.	Several events can share a timestamp, and clocks are not a reliable order.
Allowed transitions plus fetching the current object	Order no longer matters; the provider is the source of truth.	One API call per important event.

Choice. A state machine that refuses backward moves, and a fetch of the current object for money-changing events.

How do we keep side effects from running twice?

Option	Gives	Costs
Send emails inside the handler	Direct.	A retry after a crash sends them again, or sends them for a rolled-back change.
Transactional outbox	Side effects are recorded in the same transaction and sent once after commit.	One more table and worker.

Choice. A transactional outbox. It is the standard way to tie a database change to a message without distributed transactions.

How long do we keep raw events?

Option	Gives	Costs
Delete after processing	Least storage.	No replay and no audit trail.
Keep for a fixed window, then purge	Replay and audit for as long as they are useful.	Storage, and payloads may contain personal data to protect.

Choice. Keep them for a fixed window, longer than the provider retry window, with access restricted and a scheduled purge.

In one sentence

Verify, store and answer fast; deduplicate on the event id; apply with a state machine in one transaction; send side effects through an outbox; and reconcile daily because webhooks cannot prove nothing was missed.