A blockchain node can tell you what happened on-chain, but it does not automatically give your application the exact data model it needs.
Suppose a DeFi protocol has processed millions of deposits, withdrawals, borrows, repayments, and liquidations. A frontend may need to answer a simple question:
What is this user's current position?
Reading the blockchain directly for every request would mean repeatedly scanning blocks, transactions, contract storage, and event logs. That approach quickly becomes expensive and difficult to scale.
Blockchain indexers solve this problem by creating a data-processing layer between the blockchain and the application.
They consume blocks through RPC endpoints, identify relevant contract events, decode ABI-encoded data, process events in canonical order, apply state transitions, and persist the resulting state in a queryable database.
At a high level:
Blockchain
↓
RPC
↓
Blocks & Logs
↓
Event Decoder
↓
State Processor
↓
Database
↓
API
↓
Application
The interesting engineering problem is not simply collecting events. It is reconstructing a consistent application state from an ordered stream of blockchain activity.
What Smart Contract State Means to an Indexer
A smart contract's state is the collection of values stored by the contract at a particular point in the blockchain.
For example, a lending protocol may maintain values such as:
user → collateral
user → debt
user → interest index
market → liquidity
market → borrow rate
The blockchain contains the transactions and contract execution that modify these values.
An indexer can build an application-oriented representation such as:
UserPosition
user_address
market_address
collateral_amount
borrow_amount
last_updated_block
The important distinction is that events are historical facts, while indexed state is a derived representation.
For example:
Deposit(user, 5 ETH)
Borrow(user, 2000 USDC)
Repay(user, 500 USDC)
can result in:
Collateral = 5 ETH
Debt = 1500 USDC
The indexer derives that representation by processing events in sequence.
Getting Blockchain Data Through RPC
The indexing pipeline usually begins with an RPC connection to a blockchain node or provider.
Depending on the chain and node implementation, an indexer may use JSON-RPC methods such as:
eth_blockNumber
eth_getBlockByNumber
eth_getLogs
eth_getTransactionReceipt
For example, an indexer can request logs for a contract over a block range:
The node returns matching logs from that range.
A production indexer normally does not request an unlimited block range. Instead, it processes blocks in manageable chunks.
For example:
100000 → 100500
100501 → 101000
101001 → 101500
Identifying the Events That Matter
An indexer rarely needs every event emitted by every contract.
Instead, it defines the contracts and event signatures it cares about.
For example, a token indexer may monitor:
Transfer(address,address,uint256)
Approval(address,address,uint256)
The event signature is represented by a topic hash.
Conceptually:
Event Definition
↓
Keccak/Event Signature
↓
Topic0
↓
Matching Blockchain Logs
When an eth_getLogs response contains a matching topic, the indexer knows that the log corresponds to an event it understands.
This filtering is important for performance. Processing irrelevant logs at scale can create unnecessary RPC traffic, CPU work, and database writes.
The exact batch size depends on the chain, RPC provider, response size, rate limits, and event density.
This is the first major engineering consideration: the indexer must reliably move through the chain without losing its position or overwhelming the RPC layer.
ABI Decoding: Turning Logs into Structured Events
Raw logs are not usually returned as convenient application objects.
A typical log contains fields such as:
address
topics[]
data
blockNumber
transactionHash
transactionIndex
logIndex
The indexer uses the contract ABI to understand how those values should be interpreted.
Consider:
event Transfer(
address indexed from,
address indexed to,
uint256 value
);
The two indexed addresses are represented in topics, while value is encoded in the event data.
A decoder transforms the raw representation into something like:
{
"event": "Transfer",
"from": "0xAlice",
"to": "0xBob",
"value": "50000000000000000000"
}
The indexer can then convert the raw integer into the appropriate application representation, such as:
50 tokens
depending on the token's decimals.
This ABI-decoding stage is critical because incorrectly interpreting event data can result in incorrect application state even when the blockchain data itself is perfectly valid.
Event Ordering Is Part of State Reconstruction
An indexer cannot simply process events in the order they arrive from different workers.
State transitions must follow blockchain ordering.
A useful ordering key is:
(block_number, transaction_index, log_index)
For example:
Block 500
├── Transaction 0
│ ├── Log 0
│ └── Log 1
│
├── Transaction 1
│ └── Log 0
│
└── Transaction 2
└── Log 0
The indexer should process these in their canonical sequence.
This becomes particularly important when several events affect the same entity.
Consider:
Block 500:
Deposit +100
Transfer -40
If the indexer applies the transfer before the deposit, it could temporarily derive an impossible balance.
Therefore, parallel ingestion and sequential state application often need to be separated architecturally.
Reconstructing State from Events
Once events have been decoded and ordered, the indexer applies them as state transitions.
A simplified model is:
new_state = apply(previous_state, event)
For example:
Initial balance = 0
Mint 100
→ balance = 100
Transfer 30
→ balance = 70
Transfer 20
→ balance = 50
A simple pseudocode implementation could look like:
def process_event(event, state):
if event.type == "Mint":
state.balance += event.amount
elif event.type == "Transfer":
state.balance -= event.amount
return state
A production indexer obviously needs considerably more logic, including address validation, token decimals, multiple accounts, transaction boundaries, database transactions, rollback handling, and failure recovery.
But the underlying principle remains:
Previous State + Event → New State
Database Schema for Indexed State
After processing events, the indexer needs to persist the resulting data.
A simplified schema for a token indexer might look like:
Keeping both raw normalized events and derived state can be useful.
The event table provides an audit trail and makes reprocessing possible, while the derived tables provide fast application queries.
For example:
SELECT balance
FROM token_balances
WHERE wallet_address = '0x...';
is considerably more practical for an application than scanning historical blockchain logs for every request.
Checkpoints: Knowing Where the Indexer Stopped
A production indexer also needs a checkpoint system.
Imagine the indexer has successfully processed block 2,500,000.
It can store:
last_processed_block = 2500000
last_processed_hash = 0xabc...
If the process crashes, it can resume from a known position instead of starting from the beginning.
A checkpoint table might look like:
CREATE TABLE indexer_checkpoint (
id INT PRIMARY KEY,
block_number BIGINT,
block_hash VARCHAR(66),
updated_at TIMESTAMP
);
The checkpoint should generally be updated only after the corresponding data has been successfully persisted.
Otherwise, the indexer could record a block as processed even though some of its state changes were never committed.
This creates an important consistency rule:
Persist the state and advance the checkpoint as one logical unit of progress.
Handling Reorganizations
One of the most important challenges is blockchain reorganization.
Suppose the indexer processes:
Block 100
↓
Block 101A
↓
Block 102A
It derives application state from those blocks.
Later, the canonical chain becomes:
Block 100
↓
Block 101B
↓
Block 102B
The events from 101A and 102A may no longer belong to the canonical chain.
A robust indexer needs to detect this situation.
That is why storing the block hash alongside the checkpoint is useful.
The indexer can compare:
Stored block hash
vs
Current canonical block hash
If they differ, the indexer knows that its local view needs reconciliation.
A simplified rollback process could be:
Detect mismatch
↓
Find common ancestor
↓
Rollback affected indexed state
↓
Remove/reverse affected events
↓
Process canonical blocks again
↓
Update checkpoint
Some architectures avoid immediately exposing very recent blocks as final application state and instead wait for additional confirmations. Others implement explicit rollback support.
The correct strategy depends on the blockchain and the application's consistency requirements.
Handling Missed Events and RPC Failures
RPC infrastructure introduces another failure surface.
For example:
Indexer → RPC Provider
↓
Timeout
If the indexer simply continues without verifying the response, it could silently skip blockchain data.
Production systems commonly use:
- Retry policies
- Request timeouts
- Block-range backfilling
- Multiple RPC providers
- Rate-limit handling
- Failed-job queues
- Data consistency checks
Suppose the indexer successfully processed:
100000 → 101000
but the RPC request for:
101001 → 101500
failed.
The indexer should not advance its checkpoint to 101500.
Instead:
Checkpoint: 101000
↓
Retry 101001 → 101500
↓
Process successfully
↓
Advance checkpoint
This simple rule prevents gaps in the indexed chain history.
A Simplified Indexer Loop
The entire process can be represented with pseudocode:
while True:
latest_block = get_latest_block()
while checkpoint < latest_block:
from_block = checkpoint + 1
to_block = min(from_block + BATCH_SIZE - 1, latest_block)
logs = get_logs(from_block, to_block)
events = decode_logs(logs)
events.sort(
key=lambda e: (
e.block_number,
e.transaction_index,
e.log_index
)
)
begin_transaction()
for event in events:
`apply_state_change(event)
store_event(event)
save_checkpoint(to_block)
commit_transaction()
checkpoint = to_block`
A real implementation would need additional handling for RPC failures, duplicate events, reorgs, finality, concurrent workers, database failures, and chain-specific behavior.
Still, this loop captures the core indexing model:
Fetch → Decode → Order → Apply → Persist → Checkpoint
Why This Architecture Matters for Web3 Applications
Once the indexing layer is reliable, applications can query structured data rather than repeatedly interpreting raw blockchain activity.
A DeFi frontend can request:
GET /users/0x123/positions
and receive:
{
"collateral": "5 ETH",
"debt": "1500 USDC"
}
An NFT application can request the current owner of a token.
A trading application can query historical trades.
A blockchain explorer can provide transaction and contract activity through indexed tables.
In each case, the application is consuming a derived data layer, while the blockchain remains the underlying source of truth.
This separation also makes it easier to scale application APIs independently from blockchain RPC infrastructure.
The Complete Reconstruction Pipeline
Putting everything together:
Blockchain
↓
RPC Node
↓
Block / Log Fetching
↓
Event Filtering
↓
ABI Decoding
↓
Canonical Ordering
↓
State Transitions
↓
Database Persistence
↓
Checkpoint Update
↓
API
↓
Web3 Application
The indexer's job is therefore more sophisticated than simply copying blockchain data into a database.
It is maintaining a continuously updated, application-oriented representation of blockchain activity while accounting for ordering, failures, missed blocks, and chain reorganizations.
Conclusion
Blockchain indexers provide the infrastructure that turns low-level blockchain activity into data applications can actually use.
The process starts with RPC calls that retrieve blocks and logs. The indexer filters relevant events, decodes ABI-encoded values, orders them according to their position in the chain, and applies each event as a state transition.
The resulting state is persisted in database tables optimized for application queries. Checkpoints allow the indexer to resume safely after failures, while block hashes and rollback mechanisms help it recover from chain reorganizations.
The fundamental model is:
Raw Logs
↓
RPC Ingestion
↓
ABI Decoding
↓
Canonical Ordering
↓
State Reconstruction
↓
Database
↓
API
↓
Application
For developers and CTOs building blockchain applications, the key architectural question is not simply “How do we read blockchain data?”
It is:
“How do we maintain a reliable, queryable representation of blockchain state as the chain continuously changes?”
That is the core problem blockchain indexing is designed to solve.
Top comments (1)
Rebuilding state from logs works until a reorg or a contract that changes storage without emitting an event, then the index quietly drifts from the chain. The pattern I like is a periodic checkpoint: read a few balances directly via eth_call at block N and diff them against the indexed value, so drift shows up as a number instead of a user complaint. How do you handle reorg depth, do you wait for finality or roll back and replay?
iin1005h1528