How to Implement a Ledger in Distributed Systems

What is a ledger?

Think of a Ledger as a "book of truth" — for example, a company's accounting spreadsheet where every transaction that has ever happened is recorded. Just like in an accounting spreadsheet, we do not edit or remove records; we only add new entries. That gives us immutability, which is an excellent principle in financial systems. The most common way to handle this is through debit and credit events, together with the corresponding amount for each transaction.

---

How do you do it?

Keep a distinct Ledger for each currency type, such as BRL or USD, to simplify business logic and calculations. To prevent duplicate processing in a distributed system, every operation must carry a unique idempotency key.

Concurrency management is another pillar for avoiding balance inconsistencies. One approach is optimistic locking, which uses a versioning field such as version or last_sequence. The alternative is pessimistic locking, which guarantees exclusive access to the Ledger through an external service like Redis.

---

Persistence and Concurrency

Ledger persistence can be implemented in a database such as MongoDB. For optimistic locks, use a field like last_sequence on the Ledger document itself. For pessimistic locking, use a distributed cache with low latency, such as Redis.

The idempotency key depends on the system structure, but it is generally a combination of data such as timestamp + from + to + amount. That key can be stored in a distributed cache with a TTL (Time To Live).

---

Handling Money in Distributed Systems

Why use int and not float?

Never use floating-point types (float) to store money. Work with integers, storing the value in the smallest unit of the currency (e.g., R$ 123.45 is stored as the integer 12345). All mathematical operations use these integers, eliminating precision errors.

Structure for Multiple Currencies

Monetary values should be represented by a structure that combines the amount and the currency, following the ISO 4217 standard. That structure should contain the amount (integer) and the currency (e.g., "BRL"), preventing errors such as adding dollars directly to reais.

---

Applying Optimistic Locking and Transactions

To guarantee consistency, recording a transaction must be atomic. Here's a step-by-step of the full flow:

1. Idempotency Check (Fail Fast): Before starting the database transaction, check the idempotency key in a fast cache such as Redis. If the key already exists, the operation has already been processed — return success immediately and avoid unnecessary work.

2. Start the Retry Loop: Because optimistic locking can fail, run all business logic inside a loop that attempts the operation a few times (e.g., 3 attempts) before giving up.

3. Read the Current State: Inside the loop and within a new MongoDB transaction, read the Ledger document to get its current balance and last_sequence.

4. In-Memory Calculation: Calculate the new balance in your application: new_balance = current_balance + transaction_amount.

5. Atomic Write Execution: Execute both write operations within the same transaction:

insertOne into the transactions collection with the data for the new movement.
updateOne on the ledgers collection. This update is the key to optimistic locking: it must match by _id and by the last_sequence you read in step 3. The update then modifies the balance and increments last_sequence.

1. Result:

Success: If updateOne finds the document and the transaction is committed, there was no conflict. Exit the loop and return success.

Failure: If the transaction fails because last_sequence did not match, another process changed the Ledger. The retry loop continues to the next iteration, restarting from step 3. Add a short wait (backoff) between attempts.

If the loop is exhausted without success, the operation failed and an error should be returned to the client.

---

What does my data look like?

ledgers

{
 "_id": ObjectId(),
 "balance": {
    "amount": 12345,
    "currency": "BRL"
 }, 
 "last_sequence": 1,
 "last_transactions":[
   {
     "_id": ObjectId(),
     "ledger_id": ObjectId(),
     "timestamp": UnixTime,
     "sequence": 1,
     "change": {
        "amount": 5000,
        "currency": "BRL"
     },
     "idempotency_key": "timestamp-from-to-amount"
   }
 ]
}

transactions

{
     "_id": ObjectId(),
     "ledger_id": ObjectId(),
     "timestamp": UnixTime,
     "sequence": 1,
     "change": {
        "amount": 5000,
        "currency": "BRL"
     },
     "idempotency_key": "timestamp-from-to-amount"
}

---

Final Considerations

Building a reliable Ledger rests on four pillars:
Immutability — an append-only transaction history;
Precision — integers for monetary calculations, avoiding floating-point errors;
Consistency — atomic transactions with concurrency control such as optimistic locking;
Idempotency — protection against duplicate operations.

Once those foundations are solid, the next step is scalability. For Ledgers with millions of transactions, techniques such as Snapshots (periodic balance pictures) to optimize reads and archiving of old transactions become crucial to keep the system performant over time.