Every core banking system has maintenance windows. Legacy systems — the kind running inside the majority of U.S. banks — have several per week. Planned downtime for log rotation, batch reconciliation, software updates. This is normal and expected.
FedNow is not normal. FedNow requires 24/7/365 availability. No windows. No exceptions. If your bank is a FedNow participant, you cannot go offline to do maintenance.
These two facts are in direct conflict. Resolving that conflict is the job of the Shadow Ledger.
The Problem in Detail
When a FedNow credit transfer arrives at your bank:
- The FedNow gateway receives the ISO 20022
pacs.008message - The message must be accepted or rejected within 20 seconds
- If accepted, funds must be credited to the recipient's account
- The core banking system must record the transaction
Steps 1–3 are time-critical. Step 4 requires the core banking system to be online.
During a maintenance window, the core is unavailable. Under a naive integration, incoming FedNow payments would fail — the bank would be forced to return the payment, degrade its FedNow participation status, and potentially violate its service agreement with the Federal Reserve.
The same problem existed in Brazil. When we integrated Santander Brazil into PIX in 2020, the bank's core had multiple weekly maintenance windows. PIX, like FedNow, required 24/7 availability. We couldn't eliminate the maintenance windows. We couldn't violate PIX's availability requirements. We had to architect around the conflict.
The solution was the Shadow Ledger.
What the Shadow Ledger Is
The Shadow Ledger is a real-time, in-memory (with durable write-ahead log) transaction store that sits between the payment network and the core banking system. It is not a replacement for the core ledger. It is a buffer — a temporary authoritative record of in-flight and recently settled transactions that allows the system to function when the core is unavailable.
FedNow Gateway
│
▼
Anti-Corruption Layer
│
▼
Shadow Ledger ◄──── always available
│
▼
Core Banking System ◄──── sometimes unavailable
When the core is online, the Shadow Ledger acts as a pass-through: transactions are recorded in both the Shadow Ledger and the core in near-real-time.
When the core is offline (maintenance window), the Shadow Ledger becomes the authoritative record. Incoming transactions are accepted, credited in the Shadow Ledger, and queued for reconciliation when the core comes back up.
The Data Model
Each Shadow Ledger entry captures the minimum state needed to accept a payment and later reconcile it with the core:
public record ShadowEntry(
String transactionId, // FedNow end-to-end ID
String accountId, // internal account identifier
BigDecimal amount,
Currency currency,
Instant receivedAt,
EntryState state, // PENDING | CREDITED | RECONCILED | FAILED
String rawIso20022, // original pacs.008 payload
Instant reconciledAt // null until core confirms
) {}
EntryState drives the reconciliation state machine:
- PENDING — received from FedNow, not yet processed
- CREDITED — credited in Shadow Ledger, core unavailable
- RECONCILED — confirmed by core, shadow entry can be archived
- FAILED — core rejected on reconciliation (requires manual review)
Accepting Payments During a Maintenance Window
The critical design decision: when the core is down, do you accept or reject incoming payments?
Rejecting is safer from a consistency standpoint but violates FedNow availability requirements and creates a terrible customer experience. A payment that arrives at 2am during a maintenance window simply fails.
Accepting requires you to make a commitment you can't immediately fulfill — you're promising the sender's bank that funds have been credited before your core says so. This requires confidence in your reconciliation path.
We chose to accept. The reasoning:
- The Shadow Ledger's write-ahead log gives us durable, crash-safe storage. The entry survives a power failure.
- Reconciliation is deterministic — the core will come back up, and we have the full ISO 20022 payload to replay.
- The only failure mode is a core rejection on reconciliation, which requires manual review regardless. That's a better outcome than a failed payment.
The acceptance logic in the FedNow gateway handler:
public PaymentResult accept(Pacs008Message message) {
ShadowEntry entry = ShadowEntry.fromPacs008(message);
// Always write to shadow ledger first
shadowLedger.persist(entry.withState(PENDING));
if (coreAvailabilityProbe.isAvailable()) {
CoreResult result = coreAdapter.credit(entry);
shadowLedger.updateState(entry.transactionId(),
result.success() ? CREDITED : FAILED);
return result.success()
? PaymentResult.accepted()
: PaymentResult.rejected(result.reason());
} else {
// Core is down — accept optimistically, queue for reconciliation
shadowLedger.updateState(entry.transactionId(), CREDITED);
reconciliationQueue.enqueue(entry.transactionId());
return PaymentResult.accepted(); // ← the key decision
}
}
Reconciliation
When the core comes back up, the reconciliation job replays every CREDITED entry in arrival order:
@Scheduled(fixedDelay = 30_000)
public void reconcile() {
if (!coreAvailabilityProbe.isAvailable()) return;
List<ShadowEntry> pending = shadowLedger.findByState(CREDITED);
for (ShadowEntry entry : pending) {
try {
CoreResult result = coreAdapter.credit(entry);
if (result.success()) {
shadowLedger.updateState(entry.transactionId(), RECONCILED);
} else {
shadowLedger.updateState(entry.transactionId(), FAILED);
alertOps(entry, result.reason());
}
} catch (Exception e) {
log.error("Reconciliation failed for {}", entry.transactionId(), e);
// Leave as CREDITED — will retry next cycle
}
}
}
Idempotency is critical here. The core adapter must handle duplicate credit attempts gracefully — the reconciliation job may replay an entry that was partially processed before a crash. In OpenFedNow, each adapter implements an idempotency key derived from the FedNow end-to-end transaction ID.
The Write-Ahead Log
The Shadow Ledger's durability guarantee comes from a write-ahead log (WAL). Before any state transition is confirmed to the caller, it is written to the WAL. On startup, the WAL is replayed to reconstruct the in-memory state.
This gives us crash safety: if the JVM dies between accepting a payment and writing to the database, the WAL replay catches it on restart.
public void persist(ShadowEntry entry) {
wal.append(WalEntry.of(entry)); // durable first
inMemoryStore.put(entry.transactionId(), entry); // then in-memory
database.insert(entry); // then database (async ok)
}
The database write is intentionally async. Consistency flows from WAL → memory → database, not the other way around.
What This Buys You
A bank running this pattern can:
- Schedule core maintenance windows normally
- Accept FedNow payments 24/7 without degradation
- Reconcile automatically when the core returns
- Handle core crashes (not just planned maintenance) with the same recovery path
- Meet FedNow's availability SLA without architectural surgery to the core
At Santander Brazil, this pattern ran in production for over 18 months. The reconciliation queue was non-empty on every maintenance window. It processed without a single data loss event.
OpenFedNow
This pattern is implemented in OpenFedNow, a free open-source dual-rail middleware framework for FedNow and RTP integration. The Shadow Ledger is Layer 3 of five. Pre-built adapters for Fiserv, FIS, and Jack Henry. Apache 2.0.
Technical whitepaper (with full architecture documentation): doi.org/10.5281/zenodo.21114113
Questions, issues, or PRs welcome.
Daniel Stelzer Mori is a payments architect and the creator of OpenFedNow. He served as Application Architect for Santander Brazil's PIX integration from 2019–2021.
Top comments (0)