Design a Payment Service
Online shops send us their customers' payments. Each payment must be charged exactly once even when the shop retries after a timeout, recorded in a ledger whose books always balance, and settled through an outside payment processor that is slow and rate limited. A fraud score is wanted on every payment, receipts go out afterwards, and every day the ledger is checked against the processor's own report.
Constraints
- A payment request must finish within 800 ms
- About 1,500 payments per second at peak, and twice that during a big sale
- A payment is charged exactly once, even when the caller retries it
- Every payment is written to a ledger, and all of its entries succeed together or not at all
- Cards are charged through an external payment processor with a rate limit you buy by the plan
- A fraud score is wanted, but a slow fraud check must not stop payments
- Keep infrastructure under $16,500 a month
- The ledger holds about 2 billion entries across 40 million accounts, and every payment carries a unique key chosen by the caller
Scenarios to survive
Typical peak
Steady payments at the usual peak.
Interviewer asks: A payment touches a fraud check, the ledger and the processor. Which of those must happen before you answer the shop, and in what order?
Big sale (2x)
A shopping event doubles the number of payments.
Interviewer asks: Which component hits its limit first when payments double, and which limit can you not buy your way past?
Fraud scoring hangs
Whatever runs fraud scoring becomes 40x slower.
Interviewer asks: The fraud check hangs. Do you wait, skip it, or decide with simple rules, and who decides?
A ledger database node fails
The ledger database loses a node.
Interviewer asks: The ledger store loses a node in the middle of a payment. How do you know whether the customer was charged?
A whole region goes down
The region that holds your main servers and ledger is unreachable. Payments keep arriving from the US and Europe.
Interviewer asks: US East is gone. What keeps taking payments, where is the ledger now, and what happened to payments that were in flight?
What a good design must do
- Payments finish within the latency target at peak
- The big sale passes without failed payments
- Total monthly cost stays within $16,500
- A hung fraud check does not stop payments
- Something writes the ledger, and it is all-or-nothing
- Something takes the money from the card using the processor
- Something sends receipts
- Something reconciles the ledger with the processor
- The ledger survives losing a database node
- The design explains how a retried request is recognised
- Every query is answered through a key or an index, never by reading a whole table
- Payments keep working when a whole region is lost