Payment Systems: Beyond ACID, Into Reliability

The most surprising thing about designing a payment system is that the core problem isn’t handling money, it’s handling requests reliably.

Let’s watch a payment happen. Imagine a user, Alice, wants to send $10 to Bob.

sequenceDiagram
    participant Alice
    participant PaymentService
    participant BankA
    participant BankB

    Alice->>PaymentService: Initiate Payment $10 to Bob
    PaymentService->>BankA: Debit Alice $10
    BankA-->>PaymentService: Debit Success
    PaymentService->>BankB: Credit Bob $10
    BankB-->>PaymentService: Credit Success
    PaymentService-->>Alice: Payment Complete

This looks straightforward, but what if the network glitches between PaymentService and BankA after Alice’s debit succeeds but before the credit to Bob is confirmed? Alice’s money is gone, but Bob hasn’t received it. This is where idempotency becomes critical.

Idempotency means an operation can be performed multiple times without changing the result beyond the initial application. For payments, this is crucial. If Alice’s request to pay Bob gets sent twice, we must ensure Bob only gets credited $10 once. We achieve this by giving each payment request a unique request_id.

When the PaymentService receives a payment request, it first checks if a payment with that request_id has already been processed.

• If yes: It returns the previous result (e.g., "Payment already processed successfully").
• If no: It proceeds with the payment, records the request_id and its outcome, and then returns the result.

This prevents duplicate transactions.

Now, how do we know the money moved correctly? This is where Ledgers come in. A ledger is an immutable, append-only record of all financial transactions. Think of it as the authoritative source of truth for balances.

Instead of just updating a balance in a database, every financial event (debit, credit) is recorded as a distinct entry on the ledger.

Ledger Entry 1:
  Timestamp: 2023-10-27T10:00:00Z
  Type: DEBIT
  Account: Alice
  Amount: $10.00
  Request ID: req-abc-123

Ledger Entry 2:
  Timestamp: 2023-10-27T10:00:05Z
  Type: CREDIT
  Account: Bob
  Amount: $10.00
  Request ID: req-abc-123

Balances are then calculated by summing up all entries for an account. This makes auditing and reconciliation much simpler, as the ledger is the single source of truth.

The challenge then becomes ensuring Consistency. We want the ledger to accurately reflect the real-world state of money. This is where distributed transaction concepts, like two-phase commit (2PC) or simpler event-driven approaches, become relevant.

Consider a payment that involves multiple steps, like debiting Alice’s account at Bank A and crediting Bob’s account at Bank B. We need to ensure either both happen, or neither happens.

A simplified, event-driven approach might look like this:

1. Payment Service:
   • Generates a unique payment_id.
   • Records a "PaymentInitiated" event with payment_id and request_id.
   • Sends a "DebitCommand" to Bank A.
2. Bank A:
   • Processes debit.
   • Publishes a "DebitedEvent" (success or failure).
3. Payment Service (listens for DebitedEvent):
   • If "DebitedEvent" is success:
     • Records a "PaymentDebited" event.
     • Sends a "CreditCommand" to Bank B.
   • If "DebitedEvent" is failure:
     • Records a "PaymentFailed" event.
     • (Optional: Revert any local state changes).
4. Bank B:
   • Processes credit.
   • Publishes a "CreditedEvent" (success or failure).
5. Payment Service (listens for CreditedEvent):
   • If "CreditedEvent" is success:
     • Records a "PaymentCompleted" event.
   • If "CreditedEvent" is failure:
     • Records a "PaymentFailed" event.
     • Initiates a compensation transaction: Sends a "RefundCommand" to Bank A to credit Alice back.

This event-driven flow, combined with idempotency on commands and unique payment_ids for internal state, helps maintain consistency. The ledger itself is the final, immutable record of these events.

The real trick with ledgers is that you don’t calculate balances by querying a current_balance column that gets updated. Instead, you read all the transactions for an account and sum them up on demand. This means your accounts table might only store metadata, while the transactions table is the actual source of truth, and it only ever grows.

When designing, think about the state machine of a payment: initiated, debited, credited, failed, refunded. Each transition should be an immutable event on the ledger, triggered by external services or internal logic, and protected by idempotency keys.

The next concept you’ll grapple with is how to handle reconciliation between your internal ledger and the statements from actual banks, especially when dealing with fees and chargebacks.