Solana Indexer Decision Checklist
Building a Solana indexer involves several critical choices. Use this checklist to evaluate your options:
- Data Freshness Requirement: Do you need real-time (sub-second) or near-real-time (seconds to minutes)?
- Throughput: How many transactions per second does your application need to consume? Are bursts expected?
- Operational Tolerance: Can your team manage a validator node, Kafka, and ClickHouse, or do you prefer a managed pipeline?
- Cost Model: Consider infrastructure costs (validators, storage, compute) vs. API subscription fees.
- Data Volume & Retention: How much historical data must be stored, and for how long?
- Query Patterns: Will you run ad-hoc SQL queries, GraphQL, or only predefined aggregations?
- Team Skills: Does your team have experience with Rust, Node.js, or streaming infrastructure like Kafka?
Approaches to Indexing Solana Data
Indexing Solana is fundamentally different from EVM chains because of the account model and high throughput (thousands of TPS). There are three primary approaches:
- RPC Polling – Simplest, but slow and expensive at scale.
- Geyser Plugin + gRPC Stream – Real-time, efficient, but requires validator access.
- Managed Webhooks / Streams – Trade control for convenience.
RPC Polling
Use getSignaturesForAddress and getTransaction to fetch transactions for a program or account. Works for low-volume dApps but hits rate limits quickly on public RPCs.
curl https://api.mainnet-beta.solana.com -X POST -H "Content-Type: application/json" -d '
{
"jsonrpc": "2.0",
"id": 1,
"method": "getSignaturesForAddress",
"params": ["Vote111111111111111111111111111111111111111", {"limit": 5}]
}'
Downsides:
- Rate limited by most RPC providers.
- Not real-time; you must poll on an interval.
- High request cost for large programs.
Geyser Plugin + Yellowstone gRPC
This is the gold standard for real-time indexing. Run a Solana validator with a Geyser plugin that streams account updates, transactions, and logs via gRPC. The Yellowstone gRPC plugin is the most popular.
Requirements:
- A Solana validator node (or access to one).
- Geyser plugin compiled and configured.
- gRPC client (e.g., in Node.js, Rust, or Go).
Example Node.js gRPC client:
const grpc = require('@grpc/grpc-js');
const protoLoader = require('@grpc/proto-loader');
const package = protoLoader.loadSync('geyser.proto');
const geyser = grpc.loadPackageDefinition(package).geyser;
const client = new geyser.Geyser('localhost:10000', grpc.credentials.createInsecure());
const call = client.subscribe();
call.write({
slots: { filterByCommitment: { commitmentLevel: 1 } },
transactions: { filter: { accounts: { any: ['Vote111111111111111111111111111111111111111'] } } }
});
call.on('data', (data) => console.log(data));
Pros: Real-time, low latency, efficient streaming.
Cons: Requires running a validator (high ops cost) or paying for a managed gRPC stream service.
Managed Webhooks / Streams
Services like Helius, QuickNode Streams, and Shyft offer managed pipelines. They handle the validator/Geyser complexity and deliver data via webhooks or gRPC streams.
Webhook example (Helius-style):
POST /webhook
{
"webhookURL": "https://your-backend.com/solana-webhook",
"transactionTypes": ["ANY"],
"accountAddresses": ["TokenkegQfeZyiNwAJbNbGKPFXCWuBvf9Ss623VQ5DA"],
"webhookType": "enhanced"
}
Pros: No infrastructure management, built-in retries, enrichment.
Cons: Vendor lock-in, cost scales with data volume, limited control.
Evaluation Criteria for Solana Indexer Infrastructure
| Criterion | What to Check | Why It Matters |
|---|---|---|
| Data Freshness | Is delivery sub-second or batched? | Real-time apps need low latency; analytics can tolerate seconds. |
| Throughput Capacity | Max events/second; burst handling | Solana can produce >10K TPS; pipeline must keep up. |
| Operational Overhead | Validator management, queue setup, DB ops | Directly impacts team velocity and infrastructure cost. |
| Scalability | Horizontal scaling of consumers, storage | Data volume can grow exponentially with programs and users. |
| Query Flexibility | SQL, GraphQL, or pre-aggregated views | Analysts need ad-hoc querying; dashboards need fast aggregations. |
| Cost Predictability | Fixed subscription vs. variable compute/storage | Managed services charge per event; DIY has hardware + DevOps cost. |
| Reliability & Failover | Retry logic, replay capability, SLAs | Indexer downtime means missed data; backfill must be supported. |
| Data Retention | Hot vs. cold storage; archival strategy | Historical data enables trending analysis but increases storage cost. |
Common Pitfalls and Troubleshooting
- Rate Limiting: Public RPCs throttle polling; always use a dedicated RPC with higher limits for indexing workloads. OnFinality provides scalable RPC endpoints suitable for moderate polling needs.
- Missing Transactions: Geyser plugins can drop events under heavy load. Ensure your pipeline has a backfill mechanism (e.g., periodic batch fetch from RPC).
- Schema Drift: Program updates may change instruction layout. Use IDL parsing or handle dynamic decoding.
- Storage Scaling: Solana produces ~1 TB of transaction data per month. Plan for cold storage or data pruning.
- gRPC Connection Drops: Use reconnection logic with exponential backoff; many managed streams include automatic reconnection.
Key Takeaways
- For real-time apps, avoid RPC polling; use Geyser gRPC or managed streams.
- Evaluate total cost: DIY validator + streaming infrastructure vs. managed API subscriptions.
- Always plan for backfill: even the best streams can miss events during outages.
- Choose a storage backend that matches your query patterns: ClickHouse for analytics, Postgres for transactional queries.
- Managed services reduce ops but introduce variable costs; DIY gives control but requires dedicated DevOps.
Frequently Asked Questions
What is a Solana indexer?
A Solana indexer ingests on-chain data (transactions, accounts, logs) and stores it in a queryable format (database, data warehouse) for analytics, dashboards, or application features.
Can I use a standard RPC endpoint for indexing?
Yes, for low-volume cases. But production indexing needs higher throughput and real-time streams, making Geyser plugins or managed webhooks preferable.
Do I need to run my own validator to index Solana?
Not necessarily. You can use managed gRPC streams or RPC polling with a dedicated node. However, for maximum flexibility and control, running a validator with a Geyser plugin is common.
How does OnFinality support Solana indexing?
OnFinality offers Solana RPC endpoints suitable for polling and transaction fetching, as well as dedicated node infrastructure for teams that need custom validator setups. See our RPC pricing for rate limits and plans.
What's the difference between Geyser and Yellowstone?
Geyser is Solana's plugin interface that emits account and transaction updates. Yellowstone is a specific open-source Geyser plugin that exposes a gRPC stream. They are often used together.
Should I use webhooks or gRPC streams?
Webhooks are easier to set up but have higher latency and no ordering guarantees. gRPC streams provide ordered, low-latency data and are recommended for real-time use cases.
Related resources
Originally published at OnFinality.
Top comments (0)