DEV Community: Suliman Mokhtar

Zero-Latency DeFi: Parsing Raw Solana AMM Accounts in Rust

Suliman Mokhtar — Tue, 01 Sep 2026 06:55:03 +0000

Originally published on xroot.dev.

In high-frequency Web3 infrastructure, relying on a TypeScript SDK or a third-party pricing API means you are already too late.

If you are building an arbitrage bot, a sniper, or a real-time indexer on Solana, reading data through abstracted REST endpoints introduces hundreds of milliseconds of latency. To build institutional-grade infrastructure, you have to bypass the middleman. You need to pull the raw binary state of the Automated Market Maker (AMM) directly from the RPC node and deserialize it natively in memory.

Here is how to reverse-engineer Solana DeFi pools and parse raw account data in Rust at microsecond speed.

The Anatomy of a Solana Account & The Anchor Discriminator

Beneath the abstractions of the Solana ecosystem, an account's data is fundamentally just a continuous array of bytes (&[u8]). When a smart contract writes to an account, it serializes its state into this raw byte buffer.

If the AMM was built using the Anchor framework — which the vast majority of modern Solana DeFi protocols are — the account data doesn't just start with the struct variables. Anchor prepends an 8-byte discriminator to the beginning of the data payload.

This discriminator is calculated using the first 8 bytes of the SHA256 hash of the string "account:StructName". It acts as a safety check: if you try to deserialize an AMM pool account but the first 8 bytes don't match the expected hash, the program knows you passed the wrong account type and immediately aborts.

Bytes 0–7 (Anchor Discriminator) → Bytes 8–N (Raw struct data (Borsh-serialized))

To parse the account data yourself, your first step is always identifying and slicing off those first 8 bytes.

Reverse-Engineering the AMM Struct

You cannot parse binary data without knowing its exact memory layout. We need to map the byte layout of the DeFi pool — such as a Raydium CPMM or a pump.fun bonding curve — into a tightly packed Rust struct.

Instead of paying the "Borsh tax" (the CPU overhead of standard deserialization), we will engineer this for zero-copy deserialization. By using the bytemuck crate and #[repr(C)], we force the Rust compiler to lay out our struct exactly as it appears in the raw memory buffer.

use bytemuck::{Pod, Zeroable};
use solana_program::pubkey::Pubkey;

// #[repr(C)] guarantees predictable memory alignment
// matching the on-chain byte array.
#[repr(C)]
#[derive(Clone, Copy, Debug, Pod, Zeroable)]
pub struct BondingCurveState {
    pub virtual_token_reserves: u64,
    pub virtual_sol_reserves:   u64,
    pub real_token_reserves:    u64,
    pub real_sol_reserves:      u64,
    pub token_total_supply:     u64,
    pub complete:               u8, // u8 instead of bool for predictable byte sizing
}

By deriving Pod (Plain Old Data) and Zeroable, we are making a strict contractual promise to the Rust compiler: this struct contains no pointers, no dynamic strings, and is simply a safe bag of bytes. The compiler enforces this at compile time — if you accidentally include a Vec or a String, it will refuse to compile.

Rule: Every field in a Pod struct must have a known, fixed size. Use u8 instead of bool, and [u8; 32] instead of Pubkey if Pubkey doesn't implement Pod. Predictable sizing is the contract.

Deserialization at Microsecond Speed

When you request an account from a Solana RPC node, the node returns the data as a base64-encoded string. A junior developer would decode the string, parse it into JSON, and map it to a TypeScript object. We are going to decode the base64 string, strip the 8-byte Anchor discriminator, and instantly cast the remaining bytes directly to our Rust struct memory using a zero-copy pointer.

use base64::{Engine as _, engine::general_purpose::STANDARD};

pub fn parse_bonding_curve(base64_data: &str) -> BondingCurveState {
    // 1. Decode the base64 RPC response into a raw byte vector
    let raw_bytes = STANDARD
        .decode(base64_data)
        .expect("Failed to decode base64");

    // 2. Strip the 8-byte Anchor Discriminator
    //    (sha256("account:BondingCurveState")[..8])
    let account_data = &raw_bytes[8..];

    // 3. Zero-Copy Cast: reinterpret the bytes directly as the struct.
    //    This bypasses heap allocation entirely — nanosecond execution.
    let curve_state: &BondingCurveState = bytemuck::from_bytes(account_data);

    *curve_state
}

Because we aren't allocating new memory or reading field-by-field, this execution takes nanoseconds. The bytemuck::from_bytes call does not copy a single byte — it reinterprets the existing memory address as a pointer to our struct. We didn't move the library; we just pointed our code to the shelf.

TypeScript SDK (REST + JSON parse) — 200–800 ms

Rust + Borsh deserialization — ~10–50 µs

Rust + bytemuck zero-copy cast — ~100–500 ns ✓

Calculating Real-Time Price (The Business Value)

Data engineering is useless without applying business logic. Now that the raw AMM state is held securely in our Rust backend's memory, we can calculate the real-time token price instantly. Standard AMMs use the constant product formula (x × y = k). By extracting the virtual reserves from our parsed struct, calculating the exact price natively in Rust becomes trivial.

impl BondingCurveState {
    pub fn get_token_price_in_sol(&self) -> f64 {
        // Prevent division-by-zero panics
        if self.virtual_token_reserves == 0 {
            return 0.0;
        }

        // Apply the constant product ratio: price = y / x
        let sol_reserves   = self.virtual_sol_reserves as f64;
        let token_reserves = self.virtual_token_reserves as f64;

        sol_reserves / token_reserves
    }
}

If we hook this logic up to a NATS/WebSocket pipeline — as covered in the Solana Firehose ingestion series — we can stream raw base64 account data, zero-copy deserialize it, calculate the price, and fire it to a trading algorithm before a standard SDK even finishes opening its HTTP connection.

Note: For real-world use, you should validate the discriminator manually before calling bytemuck::from_bytes. A mismatched account type will cause a panic in debug mode and undefined behavior in release. Always assert: &raw_bytes[..8] == EXPECTED_DISCRIMINATOR.

Conclusion & System Impact

By reverse-engineering the binary state of Solana accounts and leveraging Rust's bytemuck, we achieve a fundamental infrastructure advantage:

Zero RPC Pricing Overhead: We don't pay for third-party price feed APIs. We read directly from chain truth.
Microsecond Deserialization: Dropping standard JSON/Borsh parsing reduces CPU overhead to near zero — from milliseconds to nanoseconds per event.
Direct Chain Truth: We read the exact mathematical reserves of the pool, making our metrics immune to frontend lag or aggregator delays.
Composable with NATS Pipelines: This parsing layer slots directly into the high-throughput pipeline described in the 3-part Solana data pipeline series.

Permissioned Tokens on Solana: How Token ACL Works — and How to Tell It From a Honeypot

Suliman Mokhtar — Sun, 30 Aug 2026 15:25:36 +0000

Originally published on xroot.dev.

On-chain, a regulated fund token and a honeypot scam are the same shape.

Both are Token-2022 mints. Both keep an active freeze authority. Both set DefaultAccountState to Frozen, so every new holder's account starts locked. One of them is a European money-market fund following its regulator's rules; the other is a trap built to let you buy and never sell. Every token scanner I tested reads them identically: red flags, high risk, score zero.

The reason this now matters is sRFC-37, the Token ACL standard — the Solana Foundation's official mechanism for permissioned tokens. It has been live on mainnet since March 2026, real institutional money already uses it, and the entire real-world-asset wave forming on Solana is going to ship in this exact shape. This post covers how it works end to end — and the structural check that separates a compliant token from a trap, verified against the chain rather than anyone's metadata.

Why Permissioned Tokens Exist at All

A tokenized treasury fund, a regulated stablecoin, a security token — their issuers are not allowed to let anyone hold them. KYC requirements, sanctions screening, court orders, investor-accreditation rules: the issuer must be able to control who holds the asset and stop specific wallets, or the asset cannot legally exist on a public chain.

Solana had two ways to build that before, and both hurt:

Transfer hooks run issuer code on every transfer — but every DEX, wallet and protocol touching the token must implement the hook interface, so composability dies at exactly the venues that create liquidity.
Manual freeze-and-thaw keeps standard transfers — but every new holder starts frozen and waits for the issuer to co-sign a thaw. Onboarding becomes a support ticket, and the issuer signs forever.

Token ACL is the third path: keep the freeze mechanism — the one lever the token program already enforces everywhere — but make thawing self-service against a published rulebook.

The Mechanics: Delegated Freeze, Permissionless Thaw

Three pieces make a Token ACL mint, and each is verifiable on-chain:

Default-frozen accounts. The mint carries Token-2022's DefaultAccountState extension set to Frozen — every token account anyone creates starts locked.
Freeze authority delegated to a program. The issuer hands the mint's freeze authority to a MintConfig PDA owned by the Token ACL program (TACLkU6…52TP). From then on, freeze and thaw run through code, not through the issuer's wallet.
A published gate. The MintConfig points at a Gate Program that answers one question: can_thaw_permissionless(wallet). The mint's metadata declares it under a token_acl key so wallets and SDKs can discover it.

The holder experience collapses to one instruction: create your (frozen) token account, call the permissionless thaw, and the Token ACL program asks the gate whether you qualify. If yes, your account unlocks in the same transaction — no issuer signature, no waiting. If the gate says no, you stay frozen. That is the whole trick: the issuer's compliance rules became a public, self-service API.

The Foundation ships a reference gate — the Allow/Block List program (GATEzz…iULz) — with four modes:

Mode	Who can thaw
Allow	only listed wallets (KYC allowlist)
Block	everyone except listed wallets (sanctions)
AllowAllEoas	any normal wallet; program-owned accounts stay frozen
Composite	allow list AND block list — block always wins

And because sRFC-37 only specifies the interface between Token ACL and gates, an issuer can replace the reference gate with anything — an on-chain KYC registry, an oracle-driven sanctions feed, an identity protocol — without touching the token itself.

What's Actually Live: the Chain, Not the Press Release

The standard was proposed in October 2025, the official guide with mainnet program addresses published January 12, 2026 — and the program's own transaction history says the rest. Its first mainnet transaction landed on March 6, 2026; its most recent, three days before this post. Total transactions ever: 514. This is genuinely early.

Early — but not theoretical. Pull a recent transaction and decode the mint it touches, and you find real institutional money:

Fact	Value
Token	Spiko Digital Assets Cash and Carry Fund (eurSPKCC)
Default state	frozen
Freeze authority	a PDA owned by the Token ACL program
metadata.token_acl	the official Allow/Block-List gate

A regulated French tokenized fund, running the textbook sRFC-37 setup — plus the rest of the institutional Token-2022 stack: pausable transfers, a permanent delegate, permissioned burn, a mint close authority. Keep that list in mind; it is the second half of this story.

The Honeypot Problem — and the Check That Actually Works

Here is the collision. DefaultAccountState: Frozen plus an active freeze authority is the classic honeypot signature — the pattern scam tokens use so buyers can receive but never sell. Every scanner rightly screams at it. But it is also, byte for byte, the shape of every compliant sRFC-37 token. As RWA issuance grows, scanners face a choice: learn the standard, or misread an entire asset class as scams — and train users to ignore the warning that actually matters.

The tempting shortcut is to trust the metadata: if the mint declares token_acl, call it permissioned. That shortcut is a vulnerability. Metadata is free-form — any scammer can paste a token_acl field into a honeypot and inherit the standard's legitimacy. The claim costs nothing.

What cannot be faked is account ownership. The verification that holds:

1 — The mint is default-frozen (the extension is on and set to frozen).
2 — The freeze authority is an account OWNED by the Token ACL program. Not "matches a claimed address" — you fetch the freeze authority's account and check its owner. Only the Token ACL program can own its MintConfig PDAs.
3 — The metadata declares the gate, so you can tell the holder which rulebook decides their thaw — the official Allow/Block-List gate, or a custom program worth reading before you buy.

The two failure modes write themselves: structure without the claim is still verifiably permissioned; a claim without the structure is a scam wearing the standard's clothes — and deserves a harder flag than an ordinary honeypot, not a pass.

I shipped exactly this three-condition check into xroot's free token audit this week. On eurSPKCC it now reads "permissioned token — freeze runs through the Token ACL standard, thaw is self-service through the official gate" where every other tool still prints the honeypot warning. On a mint that merely claims the standard, it prints the harder flag.

Honest Reading: the Standard Doesn't Vouch for Everything

One thing a good audit must not do is let Token ACL launder the rest of the mint. Remember eurSPKCC's other extensions — pausable transfers, permanent delegate, permissioned burn. On an institutional token those exist for regulators: seize on a court order, pause in an incident, burn-and-reissue on a recovery. They are disclosed design. They are also, mechanically, total issuer power over your balance — and Token ACL verifies the freeze mechanism only. A scammer can run a real, verified Token ACL setup and keep a permanent delegate that drains wallets.

So the honest verdict for a permissioned token is two sentences, not one score: this token follows the official standard for who may hold it — and its issuer can freeze, pause, seize, or burn your position, by design. Whether that is fine depends entirely on who the issuer is. For a regulated fund with a prospectus, it is the product working. For an anonymous team, it is the same red flag it always was.

What This Means From Here

For issuers: permissioned tokens on Solana stopped being a custom-engineering project. Mint with default-frozen state, delegate freeze to Token ACL, pick a gate mode, declare it in metadata — and holders onboard themselves against your rulebook.
For holders: a frozen-by-default token is no longer automatically a scam — and no longer automatically safe. The question moved from "is there a freeze authority?" to "who holds it, and through what?"
For tooling: 514 transactions in, this is the cheapest moment there will ever be to learn the standard. The RWA wave will not wait for the scanners.

— Check Any Token, Free —

Permissioned standard, or honeypot wearing its clothes?

Paste any Solana mint into xroot's free audit. It verifies Token ACL structurally — the freeze authority's owning program, never the metadata claim — names the gate that decides who can hold the token, and still tells you plainly about every seize, pause and burn power the issuer kept.

Audit a Token ↗All Solana Tools

Sources: the Token ACL documentation, the official sRFC-37 guide, the token-acl and ABL gate repositories, the sRFC-37 forum thread, and the Token ACL program's own mainnet history and the eurSPKCC mint, read directly from the chain on August 30, 2026.

Solana's 4,096-Byte Transactions: What v1 Breaks and How to Fix It

Suliman Mokhtar — Fri, 28 Aug 2026 05:59:26 +0000

Originally published on xroot.dev.

Solana's transaction size limit — 1,232 bytes, unchanged since genesis — is being raised to 4,096 bytes. The catch: the extra room only exists inside a brand-new wire format, and the day its feature gate activates, code that merely reads transactions can start failing.

The upgrade ships as two proposals: SIMD-0296 raises the size ceiling, and SIMD-0385 defines the v1 transaction format that carries it. Legacy and v0 transactions keep working exactly as they do today — if you never touch v1, nothing about sending changes for you. But reading is a different story: one v1 transaction inside a block is enough to make an un-upgraded getBlock call fail outright, and indexers that scan ComputeBudget instructions will silently record zeros for every v1 transaction they see.

Here is what actually changes, why it changes, the three ways it breaks existing apps — loudly, silently, and sneakily — and the exact fix for each. Everything below is verified against the proposal texts, then exercised end-to-end against a local Agave 4.2 validator rather than taken from the headlines.

Why 1,232 Bytes Existed, and Why It Can Finally Move

The old limit was never a design goal — it was plumbing. Transactions originally travelled as single UDP datagrams, so they had to fit the minimum IPv6 MTU of 1,280 bytes minus 48 bytes of headers: 1,232 bytes for everything — signatures, accounts, instructions, data. Solana's networking has since moved to QUIC, whose specification (RFC 9000) imposes no explicit stream size limit. The physical reason for the cap is gone; SIMD-0296 removes it.


Old limit (legacy / v0)	1,232 bytes
New limit (v1 only)	4,096 bytes (~3.3×)

What never fit in 1,232 bytes: zero-knowledge proofs (Confidential Transfers), Winternitz one-time signatures, nested institutional multisigs, BLS and other non-precompiled signature schemes. Teams worked around it with Jito bundles — which are not atomic at the protocol level. The proposal picked 4,096 by measuring real bundle traffic (roughly half of all bundles fit in 2,048 bytes, 65% in 6,144) and then aligning to the 4 KiB memory page validators already manage, so a transaction never spans multiple pages.

Worth knowing: SIMD-0296 introduces no new per-byte fee. The expectation instead is that schedulers will price large transactions via priority fees — a bigger envelope will cost more to land, just not on a published curve.

v1 Is a New Wire Format, Not a Bigger v0

SIMD-0385 does not stretch the old envelope — it redesigns it. A v1 transaction announces itself with version byte 129 (0x81) at offset zero, and then diverges from v0 in four structural ways:

Resource requests become message fields. Compute unit limit, loaded-accounts data size, heap size, and priority fee move out of ComputeBudget instructions into a fixed-position transactionConfig bitmask in the message itself. ComputeBudget instructions inside a v1 transaction still execute — as no-ops that burn compute units and configure nothing.
Address Lookup Tables are gone. All accounts are inline — up to 64 full 32-byte addresses, duplicates rejected at sanitization. Validators no longer load and deserialize ALT accounts before they can even parse a transaction.
Fixed-width instruction headers. Each instruction's header (program index, account count, data length) is separated from its variable-length payload, so parsers compute instruction boundaries directly instead of walking the buffer sequentially.
Signatures move to the tail. The message comes first, signatures last with no length prefix — the count is derived from the header.

The hard limits:

Constraint	v1 limit
Transaction size	4,096 bytes
Accounts	64, inline only — no lookup tables
Instructions	64
Signatures	12
Accounts per instruction	255

And the config fields that replace ComputeBudget instructions:

Config field	Encoding	Default when unset
Priority fee	u64 LE, total lamports	0
Compute unit limit	u32 LE	0 — not 200k/instruction
Loaded accounts data size	u32 LE	0 — not 64 MiB (cost model floors at 32 KiB)
Heap size	u32 LE, 1 KiB multiples in [32 KiB, 256 KiB]	32 KiB

The unit change nobody will notice until their fee stats corrupt: v0 priority fees are micro-lamports per compute unit; the v1 priority fee is an absolute total in lamports. Any pipeline that averages, compares, or estimates fees across versions must normalize first — the raw numbers are not even the same dimension.

The ALT removal is a real trade-off, and it lands hardest on DeFi routing. About 62% of current v0 transactions reference at least one lookup table. Converted to inline addresses, half of them grow by under 420 bytes and 90% by under 1,400 — comfortably inside the new envelope. But dense multi-table routes expand by 1,500+ bytes, and the 64-account cap does not move, so broad multi-pool aggregator strategies stay account-bound, not byte-bound. (A draft proposal, SIMD-0596, would raise the cap to 96.)

The Part That Breaks Your App: Reading

The loud failure. The moment the gate activates, getTransaction, getBlock, and blockSubscribe return JSON-RPC error -32015 for anything v1 unless you pass maxSupportedTransactionVersion: 1. It is the exact rerun of the v0 migration of 2022 — with the same nasty amplification: one v1 transaction anywhere in a block fails the entire getBlock response, not just that transaction.

// Un-upgraded reader — fails with -32015 once any v1 tx lands in the block
const block = await connection.getBlock(slot, {
  maxSupportedTransactionVersion: 0,
});

// Fixed — pass the integer 1 (the string "1" is rejected)
const block = await connection.getBlock(slot, {
  maxSupportedTransactionVersion: 1,
});

This parameter is backwards compatible — you can and should ship it today, before activation. Responses for v1 transactions carry the new transactionConfig object inside the message; v0 and legacy responses are unchanged.

The silent failure. Every indexer I have seen extracts priority fees and compute limits by scanning for ComputeBudget instructions. Against a v1 transaction that scan finds nothing and returns zero — without an error. Your pipeline keeps running and quietly writes wrong fee data, wrong CU data, and fee estimates skewed by transactions that appear free. The fix: read transactionConfig when it is present, fall back to instruction scanning when it is not, and normalize the units before anything compares them.

The sneaky failure. Geyser and gRPC streams have no version gate at all — there is no parameter to reject v1, so the new transactions simply arrive. Yellowstone builds before 15.1.1 silently downgrade v1 to v0 on the wire, and protobuf stubs generated before yellowstone-grpc-proto 12.6.0 have no Message.config field to decode. Regenerate your stubs, and detect the version structurally: check for the presence of config on the message first, then the versioned flag — in that order, because a v1 message is also "versioned".

Sending v1: Zero Defaults and New Units

Nothing forces you to send v1 — legacy and v0 stay valid indefinitely. Adopt it when you need the bytes. When you do, the sharpest edge is that v1 resource limits default to zero. Legacy and v0 gave you 200k compute units per instruction and 64 MiB of loaded accounts for free; a v1 transaction that doesn't explicitly request a compute unit limit and a loaded-accounts data size fails at execution, after you have already paid to land it.

// @solana/kit ≥ 8.0.0 — every limit is explicit in v1
const message = pipe(
  createTransactionMessage({ version: 1 }),
  (m) => setTransactionMessageComputeUnitLimit(120_000, m),
  (m) => setTransactionMessageLoadedAccountsDataSizeLimit(262_144, m),  // 32 KiB pages
  (m) => setTransactionMessagePriorityFeeLamports(10_000n, m),          // TOTAL lamports, not per-CU
);

The working recipe: simulate with both limits maxed out, read the consumed values from the simulation, then set the real limits from the measurement — rounding the loaded-accounts size up to the next 32 KiB page, because the cost model floors it at 32 KiB anyway. Remember there are no lookup tables to lean on: every account is inline, 64 at most. And send with encoding: 'base64' — base58 encoding is hard-capped at 1,232 bytes, so a large v1 transaction cannot even be submitted through it.

Two audiences with homework even if they never build a v1 transaction:

On-chain programs that introspect ComputeBudget. No sysvar exposes the v1 message config to on-chain code, and ComputeBudget instructions in v1 are no-ops — a program gating behavior on introspected compute budget simply cannot see it for v1 callers. Stop gating on it.
Fee sponsors and co-signers. A fee cap enforced by scanning ComputeBudget instructions does not bind a v1 transaction at all. Decode the raw bytes, check byte zero for 0x81, and read the limits from transactionConfig before you sign.

The Upgrade Matrix, and Where the Rollout Stands

Minimum versions that understand v1 — upgrading past these is the single highest-leverage preparation step:

Library	Minimum version	Support
`@solana/kit` (TypeScript)	8.0.0	read + send
`@solana/web3.js` 3.x	rc-0.3 (upcoming)	read + send
`@solana/web3.js` 1.x	1.99.0 (upcoming)	read only
`solana-*` crates (Rust)	4.2.x	read + send
`solders` (Python)	0.29.0	read + send
`solana-go`	1.23.0 (or 2.0.0 on `/v2`)	read + send
`yellowstone-grpc-proto`	12.6.0	first stubs with `Message.config`
Yellowstone geyser plugin	15.1.1	stops downgrading v1 → v0
`@triton-one/yellowstone-grpc`	6.0.0	decodes the config field

As of late August 2026 the feature gate is not yet active on testnet, devnet, or mainnet. The format ships with Agave 4.2 — whose mainnet feature activations began rolling out the week of August 17 — and the Solana Foundation's stated expectation is mainnet activation within weeks. You can exercise v1 locally today with Solana CLI ≥ 4.2 or Surfpool ≥ 1.5, and check the gate yourself at any time:

solana -u m feature status txv1aq4pp281K9um3tnPgkfX8UqtFT6wcVW3hNezGLL

I ran both checks while writing this. The gate reports inactive on mainnet, testnet, and devnet as of August 28. And on a local CLI 4.2.1 test validator — where the feature is already live — a @solana/kit 8 transaction built exactly as in the sending section lands on-chain with byte zero 0x81, reads back as version 1 with its transactionConfig populated, and fails with -32015 the moment it is fetched with maxSupportedTransactionVersion: 0. The behavior in this post is reproduced, not paraphrased.

Runnable end-to-end examples in TypeScript, Rust, Go, and Python live in the Solana Foundation's transaction-v1-examples repository.

If You're Here Because Something Already Broke

Symptom to fix, in the order you are likely to hit them after activation:

Symptom	Fix
Error `-32015` from `getTransaction` / `getBlock` / `blockSubscribe`	Add `maxSupportedTransactionVersion: 1` — as an integer, not a string — to every call
Whole blocks failing to fetch	Same fix — one v1 transaction fails the entire `getBlock` response until the parameter is raised
Priority fees or CU limits suddenly reading zero	Your ComputeBudget instruction scan cannot see v1 — read `transactionConfig` when present, then fall back to scanning
Fee stats or estimates off by orders of magnitude	Unit mismatch: v0 is micro-lamports per CU, v1 is total lamports — normalize before aggregating
"Transaction too large" on send despite v1	You are on base58 (capped at 1,232 bytes) — send with `encoding: 'base64'`
v1 transactions failing with compute errors	Limits default to zero — explicitly set the compute unit limit and loaded-accounts data size
Geyser/gRPC stream shows v1 traffic as v0, or missing config	Upgrade Yellowstone plugin ≥ 15.1.1, regenerate protobuf stubs ≥ 12.6.0, detect on `Message.config` presence

Bigger Envelopes, Sharper Edges

The upgrade itself is easy to like: ZK proofs, big multisigs, and heavyweight signature schemes finally land as single atomic transactions instead of bundle acrobatics, and validators get a format they can parse without touching state. But it is the first new transaction format since v0 in 2022, and the damage pattern will be the same — not the teams sending new transactions, but the readers who never opted in.

The preparation is cheap and safe to do today:

Upgrade your SDKs past the matrix above.
Ship maxSupportedTransactionVersion: 1 now — it is backwards compatible.
Audit anything that scans ComputeBudget instructions for fees or limits.
Only then think about whether your own transactions want the extra 2,864 bytes.

Sources: the official Larger Transaction Sizes upgrade page, the SIMD-0296 and SIMD-0385 proposals, and Solana's ALT trade-off analysis.

I build and operate real-time Solana data pipelines — Geyser/gRPC ingestion, decoding, and analytics at firehose scale. If your indexer, wallet backend, or trading system needs to survive this migration, get in touch or read the pipeline series.

Robinhood Chain: Three Things the Docs Don't Say

Suliman Mokhtar — Tue, 25 Aug 2026 13:18:05 +0000

Originally published on xroot.dev.

I spent a day reading Robinhood Chain with eth_call instead of reading about it. Three things turned up that no documentation page mentions — and one of them is a mistake I made first.

Robinhood Chain went live on 1 July 2026: an Arbitrum Orbit L2 settling to Ethereum, ETH for gas, ~0.02 gwei, and — unusually for a chain run by a regulated US brokerage — genuinely permissionless deployment. That much the docs say, and all of it checks out.

What follows is the part that only shows up if you query the chain yourself. Each finding is also a lesson about a different way of trusting the wrong artefact: a marketing page, an ABI, and a getter. The through-line is the same one that runs through parsing raw AMM accounts instead of using an SDK: prefer evidence the subject cannot author.

Establishing Ground Truth in Three Calls

Before trusting anything about a chain, ask the chain. Chain IDs get transcribed wrong, RPC URLs get stale, and "mainnet is live" is a claim with at least four distinct meanings.

RPC=https://rpc.mainnet.chain.robinhood.com

# Chain ID: 0x1237 = 4663
curl -s -X POST $RPC -H 'content-type: application/json' \
  -d '{"jsonrpc":"2.0","id":1,"method":"eth_chainId","params":[]}'

# Is this really a Nitro chain? Arbitrum precompiles answer 0xfe.
curl -s -X POST $RPC -H 'content-type: application/json' \
  -d '{"jsonrpc":"2.0","id":1,"method":"eth_getCode",
       "params":["0x000000000000000000000000000000000000006b","latest"]}'

An Arbitrum Nitro chain exposes its precompiles as accounts whose code is the single byte 0xfe. That one read is worth more than any "built on Arbitrum" badge: it is a property the chain cannot fake without actually being one.

The same trick answers the question that decides whether a third party can build anything at all. Rather than trusting "permissionless", estimate gas for a contract creation from an address the chain has never seen. A deployer allowlist rejects it. This one quotes a price.

Chain ID4663 (0x1237)

StackArbitrum Orbit / Nitro → Ethereum

Gas tokenETH · ~0.023 gwei

DeploymentPermissionless — no allowlist

CREATE2's deterministic-deployment proxy, Multicall3, Permit2 and Safe v1.4.1 are all present at their canonical addresses. As EVM chains go, this one arrived furnished.

The Chain Owner's Precompiles Are Public — and One Has Been Busy

Arbitrum chains expose their own governance through ArbOwnerPublic at 0x…006b. It is readable by anyone. Two calls tell you who controls the chain:

# getAllChainOwners()          -> selector 0x516b4e0f
# getAllTransactionFilterers() -> selector 0x595fbb5a
# (derive both with keccak256(signature)[0..4] — do not trust a table)

getAllChainOwners()          -> [ 0x2a153c6a…005C09 ]   # an UpgradeExecutor proxy
getAllTransactionFilterers() -> [ 0xebDc18A1…24b7   ]   # ← one authorised filterer

That second address is worth understanding. ArbOS added protocol-level transaction screening for Orbit chains: an authorised filterer registers a transaction hash, and from then on the state transition function forcibly fails it — including a transaction force-included through L1, which is normally the escape hatch that makes a rollup censorship-resistant.

Most write-ups stop at "the capability exists". But whether a capability has been used is a question an EOA answers for free, because its nonce is public and its history is indexed:

eth_getTransactionCount(0xebDc18A1…24b7) -> 0x17cc   # = 6,092 transactions

# What are they? Page the explorer's txlist and look at the destination + selector.
to:       0x0000000000000000000000000000000000000074   # the filter precompile
selector: 0xcb470491                                    # addFilteredTransaction(bytes32)
status:   1                                             # every one succeeded

6,092 transactions, and the ones I sampled are all filter calls. The first landed 30 June 2026 at 14:53 UTC — the day before the public mainnet launch. The most recent was 9 August. That is roughly 150 filtered transactions a day across the chain's first six weeks, with no published criteria, no volume disclosure, and no appeals process I could find.

I want to be precise about what this does and does not establish. It shows the mechanism is operational rather than dormant. It does not tell you what was filtered or why — the hashes reveal nothing about intent — and the filterer EOA carries no name or public tag, so attributing it to Robinhood is inference, not proof. The question I actually care about, and could not answer from the blocklist alone, is whether a contract deployment has ever been filtered.

If you are considering deploying something with a revenue stream on this chain, that is the risk to price: not a lawsuit, but a switch.

An ABI Is Not a Contract — I Got This Wrong First

Robinhood's tokenized stocks are ordinary ERC-20s. Whether they restrict transfers has been publicly disputed, so I went to settle it. I pulled the implementation behind the proxy, read its ABI, and searched the function list for the usual suspects — canTransfer, isWhitelisted, an identity registry, anything ERC-3643 shaped. Nothing. I concluded transfers were unrestricted apart from a global pause.

That conclusion was wrong, and the reason is a detail worth carrying around:

A Solidity modifier is inlined into the functions it guards. It never appears as its own ABI entry. An ABI enumerates what you can call. It says nothing about what happens on the way in. You cannot prove the absence of a restriction from an ABI — only from source or bytecode.

The verified source — 191,692 characters of it — settles the question immediately:

modifier onlyNotBlocked(address account) {
    if (IAccessControlsRegistry(ACCESS_CONTROLLED_REGISTRY).isBlocked(account)) {
        revert Blocked(account);
    }
    _;
}

function transfer(address to, uint256 value) public override
    onlyNotPaused
    onlyNotBlocked(to)
    onlyNotBlocked(_msgSender())
    returns (bool)

function transferFrom(address from, address to, uint256 value) public override
    onlyNotPaused
    onlyNotBlocked(from)
    onlyNotBlocked(to)
    onlyNotBlocked(_msgSender())
    returns (bool)

onlyNotBlocked appears fifteen times. canTransfer, whitelist and allowlist appear zero times. So both halves of my original answer needed splitting apart:

No allowlist. You do not need permission to receive one. That part was right, and it is why these tokens compose with ordinary DeFi at all.
But a per-address blocklist, checked on both sides of every transfer and on the caller. This is the USDC/USDT model — default-open, revocable — not ERC-3643's default-closed model.

That distinction is the whole commercial question. Default-open means a wallet, a portfolio tracker or an AMM works normally. Default-closed would mean none of them do. It is a good outcome — it is simply not the one the documentation states, because the documentation does not mention the blocklist at all.

Identity: Ask for Evidence the Subject Cannot Author

Search the chain's explorer for TSLA and you get fifty results. One is the real tokenized Tesla. Among the rest is a token that copies the genuine name character for character, and a cluster whose addresses all end in the same few characters — the signature of a script, not an issuer.

So how do you identify the real one? There is a getter that looks perfect for it: ACCESS_CONTROLLED_REGISTRY(), which on a genuine stock token returns the registry address. Call it on the impostor and it reverts. Job done?

No — because a getter is code, and the thing you are interrogating wrote it. Any contract can implement that function to return whatever value makes it look legitimate. It happens to work here only because these particular fakes did not bother.

The stronger check reads storage the contract cannot lie about. These tokens are beacon proxies, so the beacon address lives at the fixed EIP-1967 slot — and eth_getStorageAt bypasses contract code entirely:

SLOT=0xa3f0ad74e5423aebfd80d3ef4346578335a9a72aeaee59ff6cb3582b35133d50  # EIP-1967 beacon

eth_getStorageAt(TSLA, SLOT) -> 0x…e10b6f6b275de231345c20d14ab812db62151b00
eth_getStorageAt(AAPL, SLOT) -> 0x…e10b6f6b275de231345c20d14ab812db62151b00
eth_getStorageAt(NVDA, SLOT) -> 0x…e10b6f6b275de231345c20d14ab812db62151b00
eth_getStorageAt(SPY,  SLOT) -> 0x…e10b6f6b275de231345c20d14ab812db62151b00

eth_getStorageAt(the impostor, SLOT) -> 0x0000…0000

Four genuine tokens, one beacon, one factory. The impostor returns nothing, because it is not a beacon proxy and has no such slot to populate. A storage read is not an opinion and cannot be forged by the contract being read.

Never match on name or symbol. Memecoins on this chain have started appending the exact • Robinhood Token suffix to their own names — there is one called "Hoodrat • Robinhood Token". Any pipeline that discovers assets by string match has an impersonation vector, not a check.

One Address Holds Identity, Transferability and the Kill Switch

Follow the three findings and they converge on the same contract. The beacon that proves a token is genuine is the same registry the blocklist is read from — and the same registry that can halt every stock token at once:

function paused() public view returns (bool) {
    StockStorage storage $ = _getStockStorage();
    return $.paused || IAccessControlsRegistry(ACCESS_CONTROLLED_REGISTRY).paused();
}

Read that disjunction carefully. A stock token is paused if its own flag is set or if the registry's global flag is set. One transaction against 0xe10b…1b00 freezes the entire tokenized equity market on this chain. The same registry answers isBlocked, so an address blocked once is blocked on every stock token simultaneously.

Both flags read false today, and the multiplier that adjusts share counts for splits and dividends sits at exactly 1e18. Nothing is being exercised. That is rather the point: this is what the healthy state looks like, and the capabilities are invisible from the outside unless you go and read them.

None of this is scandalous. A regulated tokenized security needs a pause for corporate actions and a blocklist for sanctions compliance; an instrument that could not do those things could not legally represent a stock. The observation is narrower and, I think, more useful: on an ordinary token these powers are a red flag, and on a regulated one they are a requirement — so any generic token scanner pointed at a tokenized stock will produce a confidently wrong verdict.

What I Would Take Away

Probe with a control. Every "is X deployed" sweep should include an address you know is empty. If your method cannot produce a negative, it is not measuring anything.
Derive selectors, do not look them up. Four bytes of keccak is cheaper than trusting a table, and it catches the case where the function you think you are calling does not exist.
Never infer absence from an ABI. Modifiers are inlined. So are hooks. Presence is provable from an ABI; absence is not.
Prefer storage over getters, and nonces over announcements. Rank your evidence by how hard it would be for the subject to fabricate.
Capability is not usage — but usage is usually measurable. The gap between "a filter exists" and "it has run 6,092 times" was one nonce read away.

Robinhood Chain is open, well-provisioned and easy to build on, and I would take everything above as an argument for reading carefully rather than an argument against the chain. It is the same habit that made the economics of Solana's rent cut read so differently from the headlines about it. The uncomfortable finding is not that a brokerage put controls on its own tokenized securities — it is how much of that only exists in storage slots and nonces, and how little of it exists in prose.

— Check a Token Before You Trust It —

Fifty results for one ticker. One of them is real.

I build free, read-only token reports that do these checks for you — identity from storage rather than from a name, and an honest "unknown" where the chain cannot answer. No wallet connection, no signature.

Run a Token Report ↗Lock Down a Token

Everything here is reproducible against https://rpc.mainnet.chain.robinhood.com and the chain's Blockscout instance. Reference: Robinhood Chain docs, Arbitrum on transaction filtering, and L2BEAT's risk breakdown. Figures read on 25 August 2026; re-run them before relying on any of it.

xroot.dev is not affiliated with, endorsed by, or sponsored by Robinhood Markets, Inc. "Robinhood Chain" is used here only to name the public blockchain this article examines. Nothing here is financial advice.

Solana's 90% Rent Cut: The Economics of SIMD-0437

Suliman Mokhtar — Sat, 22 Aug 2026 07:19:09 +0000

Originally published on xroot.dev.

Every account on Solana carries a refundable SOL deposit — and the network just cut the price of that deposit by 90%.

SIMD-0437, shipping with Agave 4.2, is being covered as "Solana gets cheaper". That is true, and it matters for anyone creating accounts at scale. But the more interesting consequence runs the other way: because the change is strictly a relaxation, every account funded at the old rate keeps every lamport it was funded with. The SOL already locked across millions of forgotten token accounts just became a fixed, non-replenishing inventory — worth ten times more per account than anything created after activation.

Here is what rent actually is, exactly what SIMD-0437 changes, and what the cut means for reclaims, airdrops, ZK compression, and mint costs — verified against the proposal text rather than the headlines.

Rent Is a Refundable Deposit, Not a Fee

Solana charges for on-chain storage by requiring every account to hold a minimum SOL balance — the rent-exempt minimum — proportional to the account's size. Under the long-standing rate of 6,960 lamports per byte, a standard token account (ATA) requires a deposit of roughly 0.002 SOL.

The deposit is not spent. Close the account, and the lamports return to whatever address you designate. The catch is that nothing closes accounts automatically: sell a token to zero and its empty ATA stays open — holding its deposit — indefinitely. One memecoin season commonly leaves a wallet with 50–200 dead token accounts. Tens of millions of empty accounts have already been closed by people who noticed; far more are still sitting open.

SIMD-0437: 90% Off, in Five Feature-Gated Steps

SIMD-0437 lowers lamports_per_byte from 6,960 to 696 — not in one jump, but in five independently feature-gated reductions. It ships with Agave 4.2, whose mainnet feature activations began rolling out in late August 2026.

Current rate6,960 lamports / byte

Steps 1–46,333 → 5,080 → 2,575 → 1,322

Step 5 — final rate696 lamports / byte (−90%)

At the final rate, opening a token account costs about 0.0002 SOL instead of about 0.002 SOL. For anything that creates accounts in bulk, the math shifts by an order of magnitude: an airdrop to 10,000 fresh wallets used to immobilize roughly 20 SOL in recipient account rent; the same drop will soon immobilize about 2 SOL.

The Asymmetry Nobody Prices In: Old Accounts Keep Their Old Deposit

The detail buried in the proposal text: SIMD-0437 is strictly a relaxation of existing constraints. Accounts are allowed to hold less, never forced down to the new minimum. And CloseAccount returns the account's actual lamport balance — whatever was deposited at creation time — not the current rent minimum.

Consequence: every token account created before activation still returns ~0.002 SOL on close — forever. Every account created after full activation returns ~0.0002 SOL. The reclaimable stock sitting on-chain today is a finite inventory that stops replenishing. It does not expire — but it never grows again, and the accounts only get easier to forget.

Where that inventory hides, in rough order of how often people are surprised:

Empty token accounts. Every token sold (or rugged) to zero left one behind at ~0.002 SOL. Active wallets routinely hold 0.1–0.4 SOL in dead ATAs.
Failed program deploy buffers. An interrupted solana program deploy strands a buffer account holding rent for the entire binary — often 1–5 SOL per failed attempt. Developers lose more here than traders do in token accounts.
Wrapped SOL accounts. wSOL left over from DEX interactions is both a balance and a rent deposit, and unwraps back to plain SOL.

None of this requires trusting a third party with keys. Closing an account is a standard instruction your own wallet signs, with the refund directed to an address you choose.

Second-Order Effects for Builders

Two knock-on effects are worth re-running your numbers for:

ZK compression loses most of its rent argument. Compressed accounts' rent advantage falls from roughly 400× to roughly 40×. Still decisive for six-figure recipient lists; no longer decisive for a 500-wallet drop. If compression was on your roadmap purely for rent savings, the math has changed underneath it.

Fixed protocol fees become the dominant cost. With storage nearly free, per-mint costs are increasingly whatever the protocol on top charges — Metaplex's 0.0015 SOL fee becomes the largest line item of a Core NFT mint, not the rent beneath it.

A Cheaper Chain, and a Closing Window

The rent cut is unambiguously good for Solana: cheaper onboarding, cheaper airdrops, cheaper mints. But it also quietly finalizes a ledger: the SOL locked under the old rate is the most valuable per-account reclaim the network will ever offer, and every account created from here on is worth a tenth as much to close.

— Find Out What Your Wallets Are Holding —

See the exact total before you sign anything.

I built a scanner for exactly this: it lists every empty token account, wSOL balance, and stranded deploy buffer in a wallet, and shows the exact SOL total you would get back — before you approve a single transaction.

Scan Your Wallet ↗Recover Failed Deploys

Sources: the SIMD-0437 proposal and Helius's Agave 4.2 overview.

Zero-Data-Loss Analytics: Scaling NATS and ClickHouse for Web3

Suliman Mokhtar — Fri, 14 Aug 2026 09:41:20 +0000

In Part 1: Economics & System Design, we established our cloud economics and architecture blueprint. In Part 2: Ingesting the Solana Firehose with Rust, we built a resilient, zero-copy Rust ingestion service that decodes raw transactions and pumps structured swap events into NATS.

Now, we have to solve the final piece of the puzzle: Storage and Real-Time Delivery.

Piping millions of events into a traditional relational database like PostgreSQL will instantly lock tables, bloat disk space, and crash under heavy read/write concurrency. To analyze billions of historical swaps while simultaneously pushing live market feeds to end users with sub-second latency, we need a two-tier consumption layer: ClickHouse for analytical storage, and Axum WebSockets for live event fanout.

01. Consuming NATS JetStream & ClickHouse Schema Design

When consuming high-velocity data streams, database maintenance, schema migrations, or unexpected container restarts are inevitable. If your consumer reads directly from an in-memory queue, restarting your database means dropping millions of incoming swap events.

NATS JetStream: Persistence as a Safety Net

To guarantee zero data loss, we configure NATS with JetStream persistence. JetStream writes incoming messages to an encrypted, distributed disk stream. If our database goes offline for 15 minutes, NATS retains the stream and seamlessly replays every missing message the moment the database re-establishes its connection.

💾 JetStream vs. core NATS: Core NATS is fire-and-forget. JetStream adds durable, persistent, replayable streams with consumer acknowledgements. For a financial data pipeline, JetStream is non-negotiable. A consumer crash with core NATS means a data gap; with JetStream it means a brief lag before catch-up.

The ClickHouse Schema

For storage, ClickHouse is the clear winner. As a column-oriented DBMS, ClickHouse compresses data by up to 80% and can execute aggregate queries across billions of rows in milliseconds — the exact performance profile a swap analytics dashboard demands.

-- Production ClickHouse Schema for Solana Swaps
CREATE TABLE IF NOT EXISTS solana_swaps
(
    signature      String,
    slot           UInt64,
    block_time     DateTime CODEC(DoubleDelta, ZSTD),
    program_id     LowCardinality(String),
    signer         String,
    token_in       String,
    token_out      String,
    amount_in      UInt64,
    amount_out     UInt64,
    created_at     DateTime DEFAULT now()
)
ENGINE = MergeTree()
PARTITION BY toYYYYMM(block_time)
ORDER BY (program_id, token_in, token_out, block_time)
SETTINGS index_granularity = 8192;

By using the MergeTree engine, partitioning by month, and selecting an ordering key optimised for token pair filtering — (program_id, token_in, token_out, block_time) — ClickHouse skips 99% of the disk data during typical analytical queries.

⚡ Key insight: Using specialised codecs like DoubleDelta and ZSTD on timestamp columns dramatically reduces disk footprint, allowing billions of swap records to run on low-cost NVMe storage. LowCardinality(String) on program_id further reduces the column's on-disk size by up to 90%.

High-Throughput Batch Ingestion The single most common mistake engineers make with ClickHouse is executing single-row INSERT statements for every incoming event. ClickHouse is designed for bulk writes; single-row inserts create millions of tiny data parts on disk, triggering severe write amplification and bringing the cluster to its knees.

Dual-Threshold Flushing with tokio::select!
Our Rust consumer service pools incoming NATS messages into memory buffers and executes bulk inserts using two thresholds:
Threshold 1 (Batch size): ≥ 10,000 rows → flush immediately
Threshold 2 (Time interval): every 1.0 s → flush if buffer non-empty
tokio::select! races both futures concurrently. Whichever fires first triggers the flush. This means during peak volume the pipeline flushes at 10k-row boundaries (maximising throughput), and during quiet periods it flushes every second (minimising latency for dashboards).

use clickhouse::Client;

async fn run_clickhouse_consumer(
    mut nats_sub: JetStreamSubscription,
    client:   Client,
) {
    let mut buffer   = Vec::with_capacity(10_000);
    let mut interval = tokio::time::interval(Duration::from_secs(1));

    loop {
        tokio::select! {
            // Priority 1: Fill buffer from the NATS stream
            Some(msg) = nats_sub.next() => {
                if let Ok(swap) = bincode::deserialize::<DecodedSwap>(&msg.payload) {
                    buffer.push(swap);
                    msg.ack().await.ok();
                }

                if buffer.len() >= 10_000 {
                    flush_to_clickhouse(&client, mut buffer).await;
                }
            }
            // Priority 2: Flush periodically during low volume
            _ = interval.tick() => {
                if !buffer.is_empty() {
                    flush_to_clickhouse(&client, mut buffer).await;
                }
            }
        }
    }
}

By decoupling row-by-row arrivals into structured bulk inserts, we maintain near-zero CPU usage on the database cluster while keeping end-to-end swap visibility latency under one second at all volume levels.

03. The Real-Time Interface: Decoupled WebSocket Fanout

Frontend applications — live trading dashboards, arbitrage bots, portfolio trackers — need to see swaps the exact millisecond they happen. Polling ClickHouse repeatedly to check for new rows creates unnecessary read pressure on your analytical storage and introduces artificial latency.

Instead, we build a dedicated, lightweight WebSocket service using Axum that listens directly to the NATS subject — completely independent of the ClickHouse consumer.

Helius RPC ──> Rust Ingester ──> NATS JetStream ──┬──> ClickHouse (Batch / Analytical)
                                                  └──> Axum WS (Live Fanout / Clients)

Because NATS supports pub/sub topics out of the box, our Axum WebSocket service acts as a pure fanout router:

When a client connects to wss://api.xroot.dev/ws/swaps?pair=SOL-USDC, Axum subscribes them to the corresponding NATS subject (e.g. solana.swaps.raydium).
When the Rust Ingester publishes a decoded swap to NATS, NATS broadcasts it simultaneously to all active WebSocket clients and the ClickHouse batch consumer — no polling, no secondary query.
Thousands of concurrent frontend WebSocket connections consume live data feeds with zero impact on database performance, since ClickHouse is never touched by the fanout path.

🔀 Why not query ClickHouse for live data? ClickHouse excels at analytical aggregation — it is not designed for sub-millisecond single-row lookups. Routing live streaming clients through NATS keeps the database free for the heavy aggregation workloads it was built for: hourly volume, OHLCV candles, top-pair rankings.

Conclusion & System Summary

We have successfully engineered an end-to-end, high-throughput Web3 data pipeline from scratch. Here is the complete picture:

Filtered RPC Ingestion: Optimised Helius WebSocket subscriptions cap costs at under 8M credits/day on a $999/month budget — no wasted credit burn.
Zero-Copy Rust Engine: carbon + bounded tokio channels on a cost-effective 4-CPU pod handle thousands of events per second with no GC pauses and no memory leaks.
Guaranteed Delivery: NATS JetStream persists the stream to disk, ensuring zero data loss during downstream outages and enabling seamless replay on reconnect.
Analytical Storage & Fanout: A partitioned ClickHouse MergeTree table delivers sub-second analytical queries over billions of rows. A decoupled Axum WebSocket service fans out live swap events to frontend clients with zero database pressure.

Ingesting the Solana Firehose: High-Throughput Decoding with Rust

Suliman Mokhtar — Wed, 12 Aug 2026 06:15:57 +0000

In Part 1: Economics & System Design, we established the blueprint. We proved that by optimizing Helius RPC subscriptions and decoupling our ingestion layer from our database using NATS, we can track all Solana swap volume without burning our runway on infrastructure costs.

Now, we have to actually build the ingestion engine.

When you are dealing with thousands of transactions per second across Raydium, Orca, and Jupiter, traditional REST polling is obsolete. We need a persistent, real-time WebSocket connection. But keeping a socket alive at this scale, decoding the payload without causing CPU spikes, and forwarding it to our broker requires strict memory management.

Here is how we architect the Rust ingestion service to survive the firehose on a highly cost-effective 4-CPU pod.

01. Connection Resilience & The Reconnection Loop

Public and private WebSocket connections drop. Network blips happen. Helius might briefly cycle a node. If your Rust service uses a naive TCP stream that panics on a closed socket, your pipeline is dead in the water and your database starts missing swaps.

🔁 Principle: Resilience is not a feature. It is the foundation. A production ingestion service must detect a dead socket and re-establish its connection automatically — with no human intervention and no data gap.

We wrap our WebSocket connection in a tokio asynchronous reconnection loop equipped with exponential backoff and heartbeat monitoring. Instead of crashing, the service detects a missed ping/pong heartbeat or a dropped stream, pauses, and safely reconnects.

async fn stream_with_retry(ws_url: &str, tx: mpsc::Sender<RawEvent>) {
    let mut backoff = Duration::from_secs(1);

    loop {
        match connect_async(ws_url).await {
            Ok((mut socket, _)) => {
                tracing::info!("Connected to Helius Enhanced WebSocket");
                backoff = Duration::from_secs(1); // Reset on success

                while let Some(msg) = socket.next().await {
                    process_message(msg, &tx).await;
                }
            }
            Err(e) => {
                tracing::error!("Disconnected: {}. Retrying in {:?}...", e, backoff);
                tokio::time::sleep(backoff).await;
                // Exponential backoff, capped at 30 s
                backoff = std::cmp::min(backoff * 2, Duration::from_secs(30));
            }
        }
    }
}

The backoff starts at 1 second and doubles on each consecutive failure, capped at 30 seconds. On a successful reconnect, it resets to 1 second. This pattern ensures the service does not hammer Helius during an outage, while recovering instantly under normal network conditions.

By handling socket death gracefully, this loop ensures that our container never has to be manually restarted by Kubernetes or Docker during a network hiccup. The entire reconnection lifecycle is invisible to the downstream NATS publisher.

Zero-Copy Decoding with the Carbon Crate The fastest way to max out a 4-CPU server is attempting to fully deserialize every single transaction payload on Solana into a massive JSON object. The vast majority of the firehose is noise — NFT mints, token transfers, and governance votes that our swap indexer doesn't care about. If we allocate memory for those, we lose.

Filtering by Program Discriminators
To solve this, we leverage the Carbon framework. Carbon is an indexing framework built for Solana that allows us to pipeline our data: Datasource → Decoder → Processor.

Instead of deserializing the entire block, we configure our Helius datasource to apply a RpcBlockSubscribeFilter. This drops non-swap transactions at the RPC layer in nanoseconds — before they ever enter our process memory. For the transactions that do come through, Carbon's generated decoders turn the raw byte arrays directly into typed Rust structures without unnecessary allocations.

use carbon_core::{
    pipeline::Pipeline,
    processor::Processor,
};
use carbon_rpc_block_subscribe_datasource::{
    Filters, RpcBlockSubscribe,
};

// 1. Filter noise at the RPC layer — nanosecond cost
let datasource = RpcBlockSubscribe::new(
    helius_ws_url.to_string(),
    Filters::new(
        RpcBlockSubscribeFilter::MentionsAccountOrProgram(
            RAYDIUM_PROGRAM_ID.to_string(),
        ),
        None,
    ),
);

// 2. Carbon pipeline: Datasource → Decoder → Processor
Pipeline::builder()
    .datasource(datasource)
    .instruction(RaydiumDecoder, SwapProcessor::new(nats_tx))
    .build()?
    .run()
    .await;

⚡ Key insight: By filtering by Program ID at the RPC layer, a 4-CPU pod can comfortably track all Raydium and Orca swap volume. Non-swap transactions are dropped in nanoseconds. Memory is only allocated for actual swap events that pass the discriminator check.

By strictly filtering by Program Discriminators (like Raydium's program ID) and using Carbon to map byte arrays directly into structs, our CPU utilization stays incredibly low, easily fitting within our compute constraints.

Managing Backpressure & NATS Publishing We are now successfully isolating and decoding swap events. But what happens when Solana processes a massive burst of volume in a single slot, and our internal network to NATS briefly lags for 10 milliseconds?

If our pipeline is perfectly synchronous, the WebSocket reader will block while waiting for the NATS publish to succeed. The incoming TCP buffer will instantly fill up, and Helius will aggressively disconnect our socket for being too slow.

Decoupling I/O with Bounded Channels
To prevent this, we completely decouple our I/O tasks using tokio::sync::mpsc (Multi-Producer, Single-Consumer) bounded channels.

// Bounded channel — acts as an in-memory ring buffer
let (tx, mut rx) = mpsc::channel::<DecodedSwap>(10_000);

// Task 1: Producer — WebSocket reader / Carbon pipeline
tokio::spawn(async move {
    run_carbon_pipeline(tx).await;
});

// Task 2: Consumer — NATS publisher, drains independently
tokio::spawn(async move {
    let nc = async_nats::connect("nats://localhost:4222").await.unwrap();

    while let Some(swap) = rx.recv().await {
        let payload = bincode::serialize(&swap).unwrap();

        if let Err(e) = nc.publish(
            "solana.swaps.raydium",
            payload.into(),
        ).await {
            tracing::error!("NATS publish failed: {}", e);
        }
    }
});

The bounded channel with a capacity of 10,000 acts as a shock absorber. The WebSocket reader (Producer) can instantly dump decoded swaps into the channel and immediately go back to reading the socket.

🛡 The NATS publisher (Consumer) drains the channel at its own pace. If a downstream service lags, the bounded buffer absorbs the spike without ever blocking the incoming WebSocket stream. If the buffer fills completely, the producer applies backpressure naturally — preventing unbounded memory growth.

This two-task architecture is the key to operating safely on a cost-effective pod. The WebSocket reader and the NATS publisher run as independent tokio tasks on the same thread pool, coordinating through the channel without any locks, mutexes, or shared mutable state.

What's Next?
Our Rust service is now resilient, highly efficient, and pumping structured data into our message broker without dropping packets. To summarize what we have built:
Exponential backoff reconnection loop — the pipeline self-heals from network blips with no human intervention and no data gap.
Carbon framework with program discriminator filtering — non-swap transactions are dropped at the RPC layer in nanoseconds, keeping CPU and memory usage minimal.
Bounded mpsc channels — the WebSocket reader and NATS publisher run as fully decoupled async tasks, absorbing traffic spikes without blocking the incoming socket.
Data in motion is useless without a place to query it. In Part 3: Zero-Data-Loss Analytics with ClickHouse, we will consume these NATS streams, design an optimal ClickHouse schema for billions of swap rows, and build the real-time WebSocket interface for the frontend.

Architecting a Solana Data Pipeline: Economics & System Design

Suliman Mokhtar — Mon, 10 Aug 2026 11:12:23 +0000

Architecting a Solana Data Pipeline: Economics & System Design

If you have ever tried to track every single swap across Raydium, Orca, and Jupiter in real-time, you already know the painful truth: Solana is simply too fast for traditional Web2 architecture.

Standard REST API polling will get you rate-limited instantly. If you try to blindly open a WebSocket and dump the raw payload into a standard database, your server will run out of memory, drop packets, and corrupt your dataset. Drinking from the Solana firehose requires a paradigm shift in how we handle data ingestion.

This is Part 1 of a 3-part series where we will design a resilient, zero-data-loss swap indexer. We will architect the system, write a high-throughput ingestion service in Rust, and scale the analytics using ClickHouse.

Before we write a single line of code, we have to solve the hardest problem in Web3 infrastructure: Cloud Economics.

01. Cloud Economics & The RPC Math

The fastest way to kill a Web3 startup is uncontrolled RPC usage.

When you subscribe to real-time blockchain events, every message, parsed transaction, and account fetch burns RPC credits. If you scale your ingestion blindly, your infrastructure bill will scale exponentially right alongside it.

For high-throughput Solana pipelines, Helius is the industry standard, but you have to engineer your system to fit their pricing model.

Currently, the Helius Professional Plan offers 200 million monthly credits for $999/month. If you want to track all swap volume on Solana today, optimizing your WebSocket subscriptions and transaction fetching is mandatory. By filtering intelligently and only decoding the exact transaction signatures you need, tracking all swaps consumes roughly 5 to 8 million credits a day.

Metric	Value / Estimate
Helius Professional Plan	200M credits / month · $999
Average Daily Swap Volume	~3M swaps / day
Credits per Enhanced Transaction	~1–2 credits
Estimated Daily Credit Burn	5–8M credits / day
Peak Monthly Total	≈ 240M credits — within plan ✓

This keeps your monthly consumption comfortably under the 200M credit ceiling, proving that high-throughput Web3 infrastructure doesn't have to bankrupt the startup — if engineered correctly.

02. The Core Architecture Flow

To handle this volume without dropping data, we must decouple the ingestion speed from the database write speed.

Helius RPC ──> Rust Ingester (4-CPU) ──> NATS Broker ──┬──> ClickHouse (Analytics)
└──> Axum WS (Live Feeds)

The Ingestion Layer (Rust)

We are processing thousands of messages a second. Traditional languages will choke on garbage collection pauses at this scale. By using Rust, we can safely manage memory and handle the intense workload of streaming and decoding raw transactions on a highly cost-effective 4-CPU server pod.

The Message Broker (NATS)

This is the MVP of the entire architecture. If ClickHouse goes down for a split-second or experiences a CPU spike, the Rust ingester doesn't care. It simply publishes the decoded payload to NATS. NATS handles the backpressure and guarantees message replayability, ensuring zero data loss.

Storage & Interface (ClickHouse & WebSockets)

ClickHouse independently consumes the NATS stream to build tables for heavy historical analytics. Simultaneously, a separate lightweight WebSocket service can listen to the exact same NATS subjects to push live swap events directly to frontend clients.

03. Network Topology: The Zero-Latency Requirement

Having the right hardware and software stack is useless if your network topology is flawed.

When processing real-time financial data, external internet routing is your biggest enemy. To prevent latency lag and bandwidth bottlenecks, your Rust Ingester, NATS broker, and ClickHouse database must be deployed within the same local network (Virtual Private Cloud).

⚡ Rule: Keep all internal service-to-service communication inside the VPC. Internal latency is measured in microseconds. External internet routing adds 5–50ms per hop — which at thousands of messages per second means a queue that grows faster than it is consumed.

By keeping all internal communication localized, you eliminate external SSL handshake overhead and internet routing hops. The only time data touches the public internet is when it arrives from Helius, and when it is served to your end user.

Conclusion & Next Steps

We now have the complete economic and architectural foundation. To summarize:

Helius Professional Plan with filtered WebSocket subscriptions keeps costs predictable and within budget even at peak Solana swap volume.
Rust is the only language that sustains this memory footprint safely at the required throughput with no GC pauses.
NATS decouples ingestion speed from write speed, absorbing backpressure and guaranteeing zero data loss during downstream hiccups.
VPC co-location reduces inter-service latency from milliseconds to microseconds and eliminates the public attack surface.

Now that our blueprint is locked in, we need to handle the incoming firehose. In Part 2: Ingesting the Solana Firehose with Rust, we will write the high-throughput service that decodes raw transactions using the carbon crate, handles reconnection logic, and publishes structured events to NATS with zero data loss.