DEV Community: LordGhostX

SIMD-0204: Building the Foundation for Slashing on Solana

LordGhostX — Sat, 10 May 2025 17:20:40 +0000

In proof-of-stake blockchains, validators are responsible for keeping the network secure and functioning by verifying transactions and voting on consensus. To make sure they act honestly, many networks implement slashing. Slashing is a mechanism that punishes validators with bad intentions by taking away some of their staked tokens [1].

Blockchains like Ethereum, Cosmos, and Polkadot already rely on slashing to discourage bad behavior, like double signing or going offline for too long. The idea is to make attacks expensive and encourage delegators to support trustworthy validators [2].

Solana, known for its high speed and performance, hasn’t had slashing built into the protocol [3]. Validators could act against the network’s best interests without facing consequences. That’s starting to change with SIMD-0204: Slashable Event Verification [4], which introduces an on-chain program for verifying and recording validator violations. It sets up the infrastructure for slashing to become part of Solana’s future.

What Is Slashing?

Slashing is a protocol-level mechanism used in many proof-of-stake networks to penalize validators that break the rules. When a validator misbehaves, a portion of their staked tokens can be removed or “slashed” to discourage malicious behavior and reinforce honest participation [1][2].

Figure: Slashing Activity on Ethereum

This dashboard from Rated Explorer [5] shows real-time data on validator slashing in Ethereum, including the total number of penalized validators, how often slashing has occurred, and penalty sizes. As of May 10, 2025, 474 validators have been slashed across 129 separate incidents.

The types of behavior that lead to slashing vary by network, but common examples include:

Double Signing: Signing two different blocks at the same block height or block slot.
Equivocation: Sending conflicting messages or blocks to different parts of the network during consensus.
Downtime: Failing to stay online and participate in consensus harms the network’s reliability and can lead to slashing.
Surround Voting: Submitting votes that contradict earlier ones to try to split the network or manipulate the network.

Some proof-of-stake networks also offer rewards to people who detect and report violations. If you submit valid proof of misbehavior, you might get a share of the slashed tokens [13]. This helps keep everyone watching and incentivizes the community to hold validators accountable.

Why Slashing Is Challenging on Solana

Slashing has been part of many proof-of-stake networks for years, but Solana didn’t build it into the protocol early on. That’s mainly because of how the network is designed.

Solana is built for speed and performance. Validators are expected to cast and share votes quickly, sometimes before memory is fully written to disk. This helps the network stay fast, but it also increases the risk of local-state corruption [3]. In this kind of environment, automatic slashing becomes risky since it could end up punishing validators for errors that aren’t malicious.

Another challenge is that many types of misbehavior aren’t easy to detect synchronously. Things like duplicate blocks or invalid votes often need to be verified after they happen by collecting and comparing data from across the network [4]. That makes instant punishment harder to get right.

To handle this, Solana has mostly relied on softer tools like fork choice rules and gossip between validators [6]. These can help correct the network when things go wrong, but they don’t come with penalties or keep a record of what happened. For a network that values speed and resilience, slashing has always been a tricky thing to get right.

SIMD-0204: Solana’s First Step Toward Slashing

Slashing on Solana needed a starting point. The network needed a way to prove, not just suspect, that a validator misbehaved. SIMD-0204 provides that foundation by introducing a new on-chain program that verifies and records certain types of validator violations.

The program focuses on a specific violation to start: duplicate block production. If a valid proof is submitted, the violation is recorded on-chain. These reports don’t result in slashing yet, but they create a permanent, verifiable record of the violation.

Duplicate block production has caused real-world issues. In September 2022, a validator malfunction led to duplicate blocks in the same slot, triggering a consensus failure and halting Solana's mainnet for over 8 hours [14].

This step is intentionally limited in scope. SIMD-0204 doesn’t modify stakes or rewards. Its goal is to establish observability, giving the protocol, developers, and stakers a shared view into validator behavior that’s backed by cryptographic proof, not rumor or assumption.

How SIMD-0204 Works

When SIMD-0204 is activated, it deploys the Slashing Program [7]. This program is enshrined, meaning it's deployed directly by the protocol at a fixed address. To report a violation, a user submits proof using the DuplicateBlockProof instruction. This proof contains two conflicting shreds for the same slot, signed by the same validator. The program performs a series of checks:

It verifies that the shreds are formatted correctly.
It confirms that the shreds represent a real conflict using known duplication criteria.
It uses the Ed25519 Program [8] to ensure the same validator signed both shreds.

There’s also a second instruction, CloseViolationReport, which lets the reporter reclaim the lamports they paid to store the proof. That’s only allowed after a waiting period of three (3) epochs to give indexers and dashboards time to process the data.

Thanks to SIMD-0180: Vote Account Address Keyed Leader Schedule [9], block production is now linked to vote accounts rather than validator identities. This makes it possible to associate reported violations with the exact stake that earned a leader slot, which helps slashing in the future.

Future proposals like SIMD-0212: Slashing [10] will build on this reporting system to enforce actual penalties. But for now, the focus is on proving the network can detect and record violations reliably without introducing risk for honest validators.

What’s Stored On-Chain

After a proof passes verification, the program stores the violation in a new ProofReport account at a Program Derived Address (PDA). This account contains metadata about the violation:

struct ProofReport {
    version: u8,            // Current version of the report format
    reporter: Pubkey,       // Public key of the fee payer who submitted the report
    destination: Pubkey,    // Account to receive lamports when the report is closed
    epoch: Epoch,           // Epoch when the report was created
    violator: Pubkey,       // Public key of the violating validator
    slot: Slot,             // Slot at which the violation occurred
    violation_type: u8,     // Type of violation (e.g., duplicate block)
}

In addition to the metadata, the account also stores the submitted proof for future reference:

struct DuplicateBlockProofData {
    shred1_length: u32,    // Length in bytes of the first shred
    shred1: &[u8],         // Byte array containing the first shred’s data
    shred2_length: u32,    // Length in bytes of the second shred
    shred2: &[u8],         // Byte array containing the second shred’s data
}

This combination ensures that each report includes not only who did what and when but also the raw data needed to verify the violation later.

Security and Design Considerations

The Slashing Program is deployed using the BPF Upgradeable Loader, but its upgrade authority is set to None. This means the program cannot be updated after it’s deployed, ensuring consistent behavior and preventing tampering, even by core developers.

During the design review [11], there was discussion about whether to use the newer Loader-v4. The team chose to stick with the BPF Upgradeable Loader for compatibility while leaving the door open to switch if Loader-v4 becomes standard in the future.

To prevent abuse or noise in the program:

The program checks whether a ProofReport for the same slot and violator already exists. If it does, the new submission is rejected.
Reports must be properly formed and cryptographically verifiable because malformed or replayed submissions are automatically discarded.
A future slashing mechanism will rely on this data, so accuracy and trust in the program matter. SIMD-0204 starts with just one type of violation: duplicate block production. This focused approach keeps things manageable while developers and the community build confidence in the program.

Technical Details to Highlight:

Epoch-bound Activation: The program is activated at the boundary of a new epoch when the feature flag is turned on. This keeps integration predictable and reduces risk during rollout.
Submission Deadline: To prevent stale or delayed reports, proofs must be submitted within approximately one epoch’s worth of slots (432,000 [6]) from when the violation occurred.
No Rewards or Penalties (Yet): The program does not currently enforce any penalties or modify validator rewards. Its role is limited to verifying and recording. Future proposals, such as SIMD-0212, are expected to introduce the mechanisms for enforcing penalties based on these reports.
Byte-Level Validation: The program performs low-level operations, including fixed-size buffers, little-endian encoding, and offset-based deserialization to interpret and validate proofs. Signature verification is handled through instruction introspection and signature offsets.
Minimum Report Retention: Reports must remain on-chain for at least three (3) epochs before they can be closed. This window gives dashboards and indexers time to process the data. Only one (1) epoch is truly needed, so this may be reduced in the future.

Building Validator Transparency

SIMD-0204 lays the groundwork not only for future slashing but also for observing validator behavior in a consistent and verifiable way. Anyone can submit valid proof of violation: validators, bots, monitoring tools, or independent observers running their infrastructure [12]. This open access helps ensure the network is not solely reliant on core validators to flag bad actors.

As more reports are recorded on-chain, third-party tools are expected to build on top of this data. Dashboards, indexers, and analytics platforms can track validator behavior over time, creating the foundation for metrics like a slashing index or reputation score. These tools help delegators make better-informed staking decisions and encourage validators to maintain good practices [12].

While no rewards or penalties are enforced yet, this proposal provides a starting point for community-driven transparency. Future incentive structures will likely rely on it.

What’s Next for Slashing on Solana?

SIMD-0204 introduces the core infrastructure for recording validator violations, but actual slashing requires policy, economics, and enforcement. That’s where future proposals like SIMD-0212 come in. This upcoming proposal focuses on how penalties are calculated and applied using the reports generated by SIMD-0204.

Other improvements and alternatives being discussed include [10][12]:

New Violation Types
- Double voting
- Unstaking before a report is recorded (to avoid penalties)
Potential Protocol Changes
- A cooldown extension: allowing reports to still be submitted after stake deactivation.
- Shorter epochs: reducing the time window during which bad actors can operate before being penalized.
Design Alternatives
- Slashing only the validator’s self-stake.
- Deactivating stake without slashing.
- Rewarding reporters for submitting valid proofs.
- Redirecting slashed funds to an insurance pool or protocol treasury.

These decisions will shape how aggressive or conservative Solana’s slashing policy becomes. Community input and governance will play a key role as slashing evolves.

Implications for the Solana Ecosystem

As with any major protocol change, the introduction of slashing introduces both opportunity and complexity across the Solana ecosystem [4][10][12].

Potential Benefits

Increased validator accountability and overall network safety.
Enables new incentive structures and more robust slashing economics.
Transparent violation history (via ProofReports) supports third-party tools that help delegators evaluate validator behavior.
Slashing risk may encourage delegators to spread stakes across more validators, supporting decentralization.

Challenges and Trade-Offs

False Positives: If verification logic fails, honest validators could be penalized, so rigorous testing and auditing are needed.
Deterrence Risk: Without clear education and framing, slashing could discourage new or smaller participants from staking.
Operational Overhead: Adds complexity for indexers, RPC providers, and dashboard maintainers.
Governance Sensitivity: Activating slashing and tuning its parameters require broad community alignment.
Incentive Complexity: Reward systems (e.g., for reporters) must be carefully designed to avoid issues like frontrunning, manipulation, or abuse.

Conclusion

SIMD-0204: Slashable Event Verification [4] marks a key milestone in Solana’s efforts to promote validator accountability. It introduces an on-chain program for verifiable reporting of validator violations, building the foundation for future slashing logic, economic policy, and governance participation.

By making validator behavior observable and provable, SIMD-0204 empowers dashboards, analytics platforms, and delegators to make more informed decisions. As proposals like SIMD-0212: Slashing [10] are implemented, this positions the network for a more secure and decentralized validator landscape.

Looking ahead, the success of slashing in Solana will depend on real-world use. Signals such as valid reports submitted, adoption by validators and tooling, and implementation of slashing mechanisms will matter. A decline in duplicate blocks and growth in tools built around ProofReports would point to increased transparency and accountability across the network.

References

[1] Ledger Academy. “What is Slashing in Crypto?”. https://www.ledger.com/academy/topics/blockchain/what-is-slashing

[2] Binance Academy. “Slashing”. https://academy.binance.com/en/glossary/slashing

[3] Agave Validator Documentation. “Slashing Roadmap”. https://docs.anza.xyz/proposals/optimistic-confirmation-and-slashing

[4] Ashwin Sekar. “SIMD-0204: Slashable Event Verification”. https://github.com/solana-foundation/solana-improvement-documents/blob/main/proposals/0204-slashable-event-verification.md

[5] Rated Explorer. “Slashing Activity Dashboard”. https://explorer.rated.network/slashings

[6] Ryan Chern, Helius. “Consensus on Solana”. https://www.helius.dev/blog/consensus-on-solana

[7] Solana Program Library. “Slashing Program”. https://github.com/solana-program/slashing

[8] Solana Documentation. “Programs”. https://solana.com/docs/core/programs

[9] Justin Starry. “SIMD-0180: Vote Account Address Keyed Leader Schedule”. https://github.com/solana-foundation/solana-improvement-documents/blob/main/proposals/0180-vote-account-leader-schedule.md

[10] Ashwin Sekar. “SIMD-0212: Slashing”. https://github.com/solana-foundation/solana-improvement-documents/pull/212

[11] Ashwin Sekar and Jon Cinque. “SIMD-0204 Loader Decision”. https://github.com/solana-foundation/solana-improvement-documents/pull/204#discussion_r1866733325

[12] Ashwin Sekar. “Solana Foundation Validator Discussion - Dec 12 2024”. https://youtu.be/7Hmkaj-5QUU?t=709

[13] Everstake. “What Is Slashing in Crypto and How Does It Affect You?” https://everstake.one/blog/what-is-slashing-in-crypto-and-how-does-it-affect-you

[14] Solana Foundation. “09-30-22 Solana Mainnet Beta Outage Report” https://solana.com/news/09-30-22-solana-mainnet-beta-outage-report

Solana at Web2 Speed: Real-Time Performance Without Fragmentation

LordGhostX — Fri, 09 May 2025 06:54:36 +0000

Blockchain performance remains a major hurdle for building real-time applications. Metrics like latency, throughput, and responsiveness determine whether a decentralized network can support use cases such as multiplayer games, financial services, or automated systems.

Most blockchains are too slow for this. Bitcoin has a block time of 10 minutes and handles around 3 to 7 transactions per second [1][2]. Ethereum produces blocks roughly every 12 seconds and supports about 15 to 30 transactions per second [1][2]. Other networks fall in between: Cardano targets a 20-second block time, and Polkadot ranges from 6 to 12 seconds depending on network configuration [1].

Solana stands out as the fastest general-purpose blockchain today. It has an average block time of 400 milliseconds and can handle up to 65,000 transactions per second in optimal conditions [2]. This level of speed enables high-frequency, low-cost applications that are not feasible on most other networks. However, when applications demand sub-second responsiveness at a massive scale, even Solana encounters performance bottlenecks.

This leads to a key question: How can blockchains go faster without compromising the things that make them worth building on? Most existing solutions either fragment the ecosystem, compromise trust assumptions or introduce unnecessary complexities for developers and users.

Where Speed Comes at a Cost

Over the years, several approaches have emerged to improve blockchain performance. While each offers technical benefits, they often introduce trade-offs that reduce composability, increase complexity, or compromise trust.

Layer 2 Rollups: Higher Throughput, Delayed Finality

Layer 2 rollups process transactions off the main chain to reduce congestion and stabilize fees, then publish proofs back to Layer 1 [3]. This setup can significantly increase throughput, taking Ethereum from 15 to 30 transactions per second on-chain to thousands off-chain [1][3]. However, rollups introduce latency due to delayed settlement and rely on fraud or validity proofs for security. They also break atomic composability, making it difficult for contracts on Layer 2 to interact with those on Layer 1 [3].

Figure: ZK Rollup Transaction Process

Transactions are batched off-chain and submitted to Layer 1 with a SNARK proof for verification. This increases throughput and reduces congestion but adds settlement delays and limits composability.

Source: Messari, Polygon: A Multi-Sided Approach to ZK Scaling

AppChains and Layer 3s: Custom Performance, Isolated State

AppChains and Layer 3 solutions give developers full control over their execution environment, allowing them to tailor performance, security, and governance to the specific needs of a single application [4][5]. This specialization helps reduce congestion, lower fees, and improve efficiency for use cases like gaming or DeFi. However, these systems often operate independently from the main chain, which fragments the state and isolates users.

Sharding: Parallelism with Coordination Overhead

Sharding scales the blockchain by splitting the network into smaller parts called shards. Each shard handles its transactions and data, which helps increase throughput and reduce congestion [6]. However, cross-shard communication is complex, often asynchronous, and can limit how easily apps interact with each other in real-time.

Figure: Sharding Architecture with Beacon Chain Coordination

This diagram shows a network divided into multiple shard chains, each handling its transactions and data. A central Beacon Chain coordinates between the shards to maintain network consistency.

Source: Ledger Academy, What is Sharding?

Sidechains: Flexibility at the Cost of Trust

Sidechains are blockchains that run in parallel to a main chain but operate independently, often with their own consensus mechanisms and execution environments [7]. Assets can be moved between the main chain and the sidechain, typically through a bridge, allowing the sidechain to function with more flexibility. However, this independence comes with weaker security guarantees. Users must trust separate validator sets or bridge infrastructure, which can be vulnerable to failures or attacks.

Figure: Main Chain and Sidechain Interaction

Sidechains run alongside the main chain and use smart contracts to manage asset transfers and validation. They offer flexibility but rely on separate validators and bridges, which can introduce trust risks.

Source: UC Berkeley, Blockchain technology beyond cryptocurrencies: the future of last-mile delivery autonomous solutions

State Channels: Fast Finality but Limited Scope

This allows high throughput and low fees but requires opening and closing the channel with on-chain transactions. Channels are efficient for repeated interactions between fixed parties which limits flexibility and composability across broader applications [8].

Figure: How State Channels Work

Participants interact off-chain within a state channel, exchanging signed messages that represent state changes. Only the final state is submitted to the blockchain for settlement, reducing on-chain congestion and enabling faster and cheaper transactions.

Source: CSIRO Blockchain Patterns, Payment Channel (aka., State Channel)

The Trade-Off

The pattern is clear across these models. Performance improvements often come at the cost of interoperability, additional trust assumptions, or increased development overhead. These trade-offs make it harder to build real-time, composable applications that take full advantage of the blockchain’s potential.

Figure: Trade-offs Across Ethereum's Scaling Solutions

A side-by-side comparison of Ethereum’s major scaling strategies shows how throughput gains often come at the expense of composability, trust assumptions, or implementation complexity.

Source: Daniel Asaboro, Breaking the Blockchain Barrier: The Economic Transformation Enabled by Ephemeral Rollups on Solana

The Idea of “Ephemeral” in Blockchain

The word ephemeral means something short-lived or temporary [9]. In the context of blockchain, it refers to execution environments that exist only briefly and do not maintain their permanent state. Rather than creating a new chain or separate network, ephemeral systems handle specific tasks or short bursts of activity and then shut down once they are complete. This idea is now being applied to blockchain performance through Ephemeral Rollups, which offer fast, lightweight execution when needed while avoiding fragmentation and extra infrastructure.

Ephemeral Rollups by MagicBlock

MagicBlock is building the future of real-time, high-performance, fully on-chain applications that run with the speed and responsiveness of Web2 [9]. It introduced Ephemeral Rollups, a new runtime model that enables low-latency execution for demanding use cases such as games, DeFi protocols, and automation tools without sacrificing speed, security, or composability [10][11].

At the core, MagicBlock operates as a specialized Solana Virtual Machine (SVM) runtime. Instead of deploying a new chain, developers can delegate specific accounts to Ephemeral Rollups for fast execution [10][11]. The underlying programs, smart contracts, and assets remain on Solana’s mainnet, preserving full composability with the rest of the ecosystem.

These Ephemeral Rollups support block times as low as 10 to 50 milliseconds, enable gasless transactions and offer features like tick-based execution and custom event loops [10][11]. Developers can spin up high-speed environments as needed without duplicating infrastructure or breaking the unified state of the Solana network.

Figure: Real-Time Execution with MagicBlock Ephemeral Rollups

This diagram shows how Ephemeral Rollups operate with 10 millisecond block times and periodically sync back to Solana’s 400 millisecond main runtime. Developers can run high-speed, real-time workloads without leaving Solana’s composable ecosystem.

Source: MagicBlock, Why MagicBlock?

Performance and Economic Benefits

Developers can go beyond what is possible on Solana using Ephemeral Rollups while preserving full composability with existing programs and protocols. Developers and users gain the speed and flexibility of a custom execution environment without compromising on security, composability, or ecosystem compatibility [11].

No Fragmentation

Programs, assets, and application logic remain on the Solana mainnet. Developers do not need to deploy to a new chain or maintain a separate environment. All improvements to Solana instantly benefit applications using Ephemeral Rollups [11].

Atomic Composability

Delegated accounts can still be read by Solana, and undelegated accounts remain fully active [11]. This allows seamless interaction across protocols and avoids interoperability issues seen in many L2, appchains, and sidechain architectures.

Familiar Tools and Ecosystem Compatibility

Developers can continue using existing programming languages, libraries, and dev tools without modification. There is no need to rewrite applications or reconfigure infrastructure, making integration straightforward and low-risk [11].

High Responsiveness and Custom Execution

Ephemeral Rollups support block times as low as 10 milliseconds. Developers can define custom execution logic, including tick-based updates, event-driven tasks, and session runtimes. This enables real-time UX in areas like gaming, AI, DePIN, or financial automation where responsiveness is critical [11].

Low-Cost, Scalable Infrastructure

Gasless or near-zero-cost execution makes high-frequency interactions practical. Ephemeral Rollups are provisioned dynamically and scale with demand, eliminating the need for always-on infrastructure and reducing costs for developers and infra providers.

Open Infrastructure and DePIN Potential

Anyone can run Ephemeral Rollup environments and earn fees for providing compute. This creates opportunities for decentralized infrastructure networks to emerge, supporting real-time applications and being rewarded for their contributions [11].

Overall, Ephemeral Rollups give developers the best of both worlds: real-time performance with no compromises on the security, state integrity, or composability that make Solana powerful for on-chain applications.

Use Cases Supercharged by Ephemeral Rollups

Ephemeral Rollups make it possible to build decentralized applications that require high-speed, low-latency execution without relying on centralized infrastructure. By enabling sub-50 millisecond transaction processing with minimal or no fees, MagicBlock opens the door to real-time use cases across several industries.

Gaming

Games can now run with 10 millisecond block times, allowing players to move, interact, and compete in real-time. Game state updates instantly within the rollup, while assets and logic remain secured on Solana. A working example is Solana-Generals [12], a multiplayer strategy game powered by Ephemeral Rollups.

Financial Services

MagicBlock enables real-time settlement, live price discovery, and automated trade execution without off-chain oracles. This allows developers to build DeFi protocols with transparent logic and faster execution than what is possible on most L1 or L2 networks. The Real-Time Pricing Oracle [13] offers an example of how to build latency-sensitive financial tools on Solana using Ephemeral Rollups.

AI and Automation

On-chain AI agents and automation tools can benefit from fast, event-driven execution. Ephemeral Rollups provide the responsiveness needed for real-time inference and decision-making, allowing AI to interact directly with the live blockchain state. Developers can start with templates like Super Smart Contracts [14] to create smart contracts enhanced by AI.

Payments and Microtransactions

Ephemeral Rollups support instant, programmable transfers with very low overhead. This makes them ideal for subscriptions, pay-per-use models, or global remittances. MagicBlock's On-Chain Payments Demo [15] shows a token transfer that can delegate and execute both on-chain and in the Ephemeral Rollup.

DePIN and IoT

Real-world infrastructure networks need fast coordination. Ephemeral Rollups offer sub-50 millisecond state updates and horizontal scaling for sensor networks, smart devices, and DePIN protocols [11].

Figure: Ephemeral Rollup Activity on Solana

This dashboard from Dune Analytics tracks real-time usage of MagicBlock’s Ephemeral Rollups, including daily delegations and unique users. The upward trend highlights increasing developer adoption and real-world engagement.

Source: MagicBlock, Dune Analytics

Why Performance Without Fragmentation Matters

Many scaling solutions improve performance by creating separate environments. Layer 2 rollups, sidechains, and app-specific chains often rely on duplicating infrastructure, deploying new chains, or fragmenting the state of the network [3][4][5][6][7][8]. While these strategies can increase throughput, they make composability harder and add complexity for developers and users.

MagicBlock takes a different approach. Instead of building a separate chain or relying on trust bridges, it improves Solana directly. Ephemeral Rollups run as part of the Solana runtime with programs, assets, and accounts remaining on the mainnet, and composability is preserved across the entire network [10][11].

By preserving a single, global state, MagicBlock lets developers scale their applications without losing the benefits of shared infrastructure. Real-time performance and low-cost execution are achieved without isolating users or creating new trust assumptions [10][11].

The Future: Composable, Real-Time On-Chain Worlds

As blockchain performance improves, entirely new categories of applications become possible. Autonomous worlds, AI-native agents, real-time multiplayer games, and reactive financial systems can all exist fully on-chain without sacrificing user experience.

Ephemeral Rollups remove the traditional bottlenecks. Applications can respond in milliseconds, scale dynamically, and remain composable with the broader ecosystem. There is no need to compromise between speed and connectivity.

MagicBlock makes this a reality by providing real-time execution within Solana’s existing environment. It brings the performance of Web2 to decentralized apps, allowing them to feel fast, interactive, and seamless. For developers, this unlocks new creative and economic possibilities.

Conclusion

Ephemeral Rollups mark a new chapter in blockchain performance. Rather than fragmenting the network or adding layers of complexity, MagicBlock enhances Solana with real-time, composable execution environments. Developers can now build applications that are fast, scalable, and deeply integrated with the rest of the ecosystem.

By supporting low-latency execution, horizontal scaling, and gasless interactions, Ephemeral Rollups remove the bottlenecks that have long held back decentralized applications. Use cases like real-time games, AI agents, financial automation, and IoT coordination can finally live fully on-chain without compromising on speed or user experience.

MagicBlock proves that performance and composability no longer need to be trade-offs. Real-time dApps are not just possible, they are the future of blockchain-native applications.

References

[1] Nervos Knowledge Base. “What is Block Time in Blockchain? A Complete Guide”. https://www.nervos.org/knowledge-base/block_time_in_blockchain_(explainCKBot)

[2] Prachi Pandey. “Solana (SOL) Transaction Fees, Speeds, and Limits: Everything You Need to Know”. https://fuze.finance/blog/solana-transaction-fees-speeds-and-limits/

[3] Gemini Cryptopedia. “Blockchain Scalability and Privacy: The Rollup Ecosystem”. https://www.gemini.com/cryptopedia/layer-2-scaling-zk-rollup-optimistic-rollup-ethereum

[4] Coinbase Learn. “What is an application-specific blockchain (AppChain)?”. https://www.coinbase.com/learn/crypto-glossary/what-is-an-application-specific-blockchain-appchain

[5] Coinbase Learn. “What are Layer 3 blockchains and what is the difference with Layer 2 blockchains?”. https://www.coinbase.com/learn/crypto-glossary/what-are-layer-3-blockchains-and-what-is-the-difference-with-layer-2-blockchains

[6] Bitpanda Academy. “What is Sharding?”. https://www.bitpanda.com/academy/en/lessons/what-is-sharding/

[7] Wikipedia. ”Blockchain”. https://en.wikipedia.org/wiki/Blockchain

[8] Ledger Academy. “State Channels Meaning”. https://www.ledger.com/academy/glossary/state-channels

[9] Merriam-Webster. “Ephemeral”. https://www.merriam-webster.com/dictionary/ephemeral

[10] Gabriele Picco and Andrea Fortugno. ”Ephemeral Rollups are All You Need”. arXiv:2311.02650 [cs.DC]. https://arxiv.org/abs/2311.02650

[11] MagicBlock. “MagicBlock Documentation”. https://docs.magicblock.gg/

[12] MagicBlock. “Solana-Generals”. https://github.com/magicblock-labs/solana-generals

[13] MagicBlock. “Real-Time Pricing Oracle”. https://github.com/magicblock-labs/real-time-pricing-oracle

[14] Gabriele Picco. “Super Smart Contracts”. https://github.com/GabrielePicco/super-smart-contracts

[15] MagicBlock. “Dummy Token Transfer”. https://github.com/magicblock-labs/magicblock-engine-examples/tree/main/dummy-token-transfer

Raising Block Limits on Solana: SIMD-0250 and Its Economic Effects

LordGhostX — Fri, 09 May 2025 06:51:57 +0000

SIMD-0250, a Solana Improvement Document (SIMD), proposes raising the network’s maximum block Compute Units (CUs) to 60 million. It builds on SIMD-0207, which proposes to raise the limit from 48 million to 50 million. The only change made in both proposals is to the Max Block Units. Other limits, like Max Writable Account Units, Max Vote Units, and Max Block Accounts Data Size Delta, stay the same. This means more room for non-vote transactions, which helps boost total transaction throughput.

With more space in each block, the network can process more transactions per second. This helps reduce congestion and lowers the need for users to outbid each other on fees, especially during periods of high demand. For apps handling a high volume of transactions or those with small dollar values (like games or microtransactions), this kind of fee relief is a big deal. Messari’s State of Solana Q3 2023 report pointed to low fees and fast confirmations as major drivers of growth in consumer-facing use cases like gaming, DePIN, and social platforms.

For validators, this change comes with trade-offs. While it can increase fee earnings, it also raises hardware demands. Over time, this could raise the bar for running a validator and limit participation to better-resourced operators. This concern is not new. The Solana Foundation’s March 2023 Validator Health Report already emphasized that validator performance and resilience will need to scale alongside network growth.

Developers, on the other hand, stand to gain from the extra compute available. Apps that rely on batching transactions or updating on-chain state in real-time, like DeFi order books, multiplayer games, or NFT platforms, can run more smoothly. The larger block size also gives the network better breathing room during traffic spikes, helping avoid lags or dropped transactions when demand rises.

Still, there are risks. Larger blocks may take longer to execute and propagate across the network, placing extra load on RPC nodes, explorers, and indexers. If these systems are not updated to handle the new block size, they might lag or fail to keep up. The change is backward-compatible, but outdated nodes could reject the new, larger blocks entirely.

In summary, SIMD-0250 makes a relatively simple change with wide-ranging effects. It expands Solana’s ability to handle more activity, improve app performance, and stabilize fees, but it adds more pressure on infrastructure and validators. Like many scalability upgrades, its success depends on how well the ecosystem keeps up and whether that added capacity translates into meaningful new use.

References

SIMD-0207: Raise the Block Limit to 50M CUs. Retrieved from https://github.com/solana-foundation/solana-improvement-documents/blob/main/proposals/0207-raise-block-limits-to-50M.md
SIMD-0250: Increase Block Limit to 60M CUs. Retrieved from https://github.com/solana-foundation/solana-improvement-documents/blob/main/proposals/0250-raise-block-limits-to-60M.md
Messari’s State of Solana Q3 2023. Retrieved from https://messari.io/report/state-of-solana-q3-2023
Solana Foundation Validator Health Report: March 2023. Retrieved from https://solana.com/news/validator-health-report-march-2023

Decentralizing Starknet: Architecture, Plans, and MEV Impact

LordGhostX — Wed, 11 Sep 2024 05:32:46 +0000

Starknet is an Ethereum Layer 2 (L2) that leverages Zero-Knowledge (ZK) technology to deliver high transaction throughput and low costs while maintaining Ethereum's core principles. However, its reliance on a centralized sequencer for transaction processing introduces vulnerabilities such as single points of failure and scalability bottlenecks.

This article explores Starknet's current centralized architecture, the network's plan toward decentralization, the motivations driving this transition, and how these changes could reshape the future of the blockchain ecosystem, including the dynamics of MEV (Maximal Extractable Value).

Overview

Starknet's early success can be attributed to its centralized architecture, which facilitated rapid development. However, as the network scales, this centralization poses significant risks, including single points of failure and constraints on scalability.

Transitioning to a decentralized model is necessary to overcome these limitations and improve security through distributed control, enhance censorship resistance, and ensure long-term sustainability. This shift represents more than just a technical upgrade; it is pivotal for Starknet's continued growth and resilience.

Starknet's move toward decentralization will set a new benchmark in the blockchain space, inspiring other networks to adopt similar models. This evolution can potentially strengthen the resilience and openness of the entire ecosystem.

Starknet's strategic plan, as detailed in its introduction and presentations, outlines a gradual transition to decentralization, involving the distribution of control to independent nodes and the implementation of community-driven governance.

Starknet Sequencer

The Starknet sequencer is a specialized entity responsible for ordering and processing transactions before they are verified and submitted to Ethereum. Here's how it functions:

Transaction Ordering: The sequencer gathers and organizes incoming transactions in a specific order.
Execution: Once ordered, the sequencer executes these transactions.
Batching: After execution, the sequencer groups multiple transactions into a batch, known as a rollup for efficiency.
Block Production: The sequencer then produces blocks that contain these processed transaction batches.

Beyond ordering transactions, the sequencer is also responsible for maintaining the network's liveness. It ensures that the network continues to process transactions and produce new blocks, keeping Starknet responsive and operational.

However, the centralization of sequencers introduces potential vulnerabilities, such as censorship, where a sequencer might choose to exclude certain transactions, and security risks, where a single point of failure could jeopardize the entire network.

Decentralization Plans

The following initiatives highlight the steps Starknet is taking to distribute control, minimize vulnerabilities, and strengthen its infrastructure:

Community-run sequencers: To mitigate the risks of centralized sequencers, Starknet plans to decentralize this role by enabling community-run sequencers. This will distribute power and responsibility across multiple entities, reducing the influence of any single sequencer and enhancing the system's overall resilience.
Decentralized transaction ordering: By decentralizing the transaction ordering process, Starknet aims to prevent any single entity from controlling the sequence of transactions. This ensures a more transparent and equitable process, strengthens network security, and boosts participant trust.
Improved security and censorship resistance: Deploying multiple sequencers will significantly enhance network security by reducing the risk of censorship and malicious activities. A decentralized sequencer network makes it far more difficult for any single actor to undermine Starknet's integrity, reinforcing its commitment to security and censorship resistance.

Due to its proven reliability and robustness, Tendermint has been proposed as a strong candidate for Starknet's initial consensus mechanism in its decentralization efforts.

Tendermint

Tendermint is a Byzantine Fault Tolerant (BFT) consensus protocol used for securing decentralized networks. It allows a network of validators to agree on the next block in a blockchain, even if some validators are malicious, ensuring high reliability and security.

The protocol assumes there are N = 3f + 1 validators, where f represents the number of potentially malicious (Byzantine) validators. Tendermint operates in two key dimensions: height (the current block) and round (the iterative process to reach consensus). Multiple rounds may be required for each block height, with a single leader or proposer selected per round. This structured approach ensures validators can securely and efficiently agree on the next block while maintaining decentralization and resilience.

Consensus Process

Propose: In the initial phase, a validator is chosen to propose a new block, typically through a round-robin selection, to ensure fair and systematic rotation among validators. This step starts the consensus process by submitting a candidate block for consideration.
Prevote: After the block is proposed, validators cast their initial votes on whether to accept it. The prevote phase helps gauge initial support but does not finalize the block. It serves as an early signal of consensus among validators.
Precommit: If a supermajority (usually two-thirds) of validators prevote in favor of the block, the process moves to the precommit stage. This stronger agreement shows a more definitive step towards finalizing the block, though it is not yet final.
Commit: In the final phase, if a supermajority of precommits is reached, the block is committed to the blockchain. This step makes the block final and irreversible, securing it as a permanent part of the blockchain's history. The commit phase concludes the Tendermint consensus process, ensuring complete agreement among validators on the block.

Starknet ProofChain

Starknet proposes using the Tendermint protocol with an innovative ProofChain approach. This divides blocks into K proofchains based on height, where ProofChain(H) = H mod K. Each block contains proofs for the latest non-empty block and any empty blocks in its chain, improving security, scalability, and efficiency.

Before diving into ProofChains, it's important to understand the two types of blocks:

Blocks containing transactions (depicted in green) must include a proof to be valid.

Empty blocks (depicted in red) contain no transactions and no proof.

Here are the key points:

Blocks divided into K ProofChains: This structure allows parallel processing and verification, improving scalability by allowing multiple proofchains to be processed simultaneously.

Each block contains proofs for previous blocks: Every block includes cryptographic proofs for the preceding blocks, creating a robust verification chain. This ensures that the validity of each block is anchored in earlier blocks, securing the blockchain’s integrity.

Enhanced security: The braided ProofChain structure makes it significantly harder for an attacker to alter the blockchain's history. Recursive proofs, which include transaction execution and prior block verification, further reinforce network security.
Improved block verification efficiency: By organizing proofs into different chains, the process of verifying new blocks is streamlined, reducing computational demands. This speeds up the consensus process, especially in larger networks.

The zero-knowledge proof (ZKP) required for the L1 state update can be efficiently constructed by folding K ZKPs (one from each ProofChain).

For example, Block 12 must include a recursive proof that verifies the execution of transactions proof in Block 3 and also validates the empty blocks 6 and 9.

Decentralization Challenges

By distributing the proposing, voting, and committing roles across many validators, Starknet reduces centralization risks while enhancing security and reliability. With its checks and balances, Tendermint's structured consensus process ensures that Starknet remains decentralized and trustless, which is essential for its long-term success and scalability.

However, there are several challenges to consider:

Network Latency: Delays in the message (vote) delivery can cause timeouts and slow down the consensus process, impacting network performance.
Message Overhead: Tendermint O(n²) message overhead increases with more validators, potentially reducing consensus efficiency as the network scales.
Speed Constraints: The slowest validator limits the process, creating a bottleneck and vulnerability to delays as the validator count grows.

Starknet Staking

Starknet is introducing STRK staking in Q4 2024 to boost decentralization and secure the network, becoming the first Layer 2 to enable such a mechanism. In this initial phase, neither StarkWare nor the Starknet Foundation will participate in staking, making them ineligible for staking rewards. Additionally, all locked tokens will not be eligible for staking during this phase.

See the SNIP 18: Staking's First Stage on Starknet proposal for more details.

Network Rewards

Incentives will be essential for maintaining Starknet's security as it decentralizes. It will motivate participants to stake tokens and act in the network's best interest. This ensures long-term sustainability and attracts a diverse set of validators. The potential reward types include:

Transaction Fees: Validators may earn a share of fees for processing transactions, aligning their incentives with network growth and usage.
Block Rewards: Validators could receive newly minted tokens for proposing and validating blocks, providing a consistent base reward for their participation.
Proof Rewards: Unique to Starknet, validators might be rewarded for generating and verifying cryptographic proofs, which is essential for maintaining the network's zero-knowledge proof system.
MEV (Maximal Extractable Value): Validators can earn additional revenue by optimizing the transaction order within blocks. While profitable, it requires careful management to ensure fairness.
Staking Rewards: Participants who stake tokens to secure the network may earn rewards, encouraging long-term commitment and a healthy distribution of staked tokens.

Decentralization Impact on MEV

Decentralization tends to reduce the potential for extractive MEV (Maximal Extractable Value) by distributing power across a wider array of participants, reducing the chance of any single entity exploiting transaction ordering. However, it can also increase competition among validators, driving the emergence of more advanced and decentralized MEV strategies. Here are some predictions for the MEV landscape when Starknet becomes decentralized:

More equitable distribution of MEV opportunities: As sequencing becomes decentralized, MEV capture will be more evenly distributed among participants, reducing concentration in the hands of a single entity and promoting fairer access.
Innovation in transparent MEV strategies: Decentralization is likely to drive the development of new MEV strategies that are more transparent and equitable, enabling a level playing field for all participants.
Potential reduction in overall MEV: Increased competition due to broader accessibility may decrease the total extractable MEV as more participants compete for the same opportunities, driving down the overall value that can be captured.
Emergence of specialized MEV extraction protocols: The decentralized environment could lead to the rise of advanced tools and protocols designed for MEV extraction and a more distributed and competitive network.
Integration of MEV-resistant mechanisms: Starknet and similar platforms may adopt protocol-level features to mitigate MEV exploitation, promoting greater fairness and stability within the network.
Shift towards community-oriented MEV distribution: We may see a move towards models where MEV is distributed more equitably among network participants, potentially through community-driven MEV auctions or collective action mechanisms, reducing monopolization by a select few.

Conclusion 👋

Starknet's transition to decentralization will significantly improve the network's security, scalability, and fairness. By decentralizing the sequencer role and implementing the Tendermint consensus protocol and ProofChain, Starknet aims to overcome centralization risks while preserving a trustless and resilient ecosystem.

If you're interested in diving deeper into MEV on Starknet, check out my previous posts:

For additional resources on Starknet MEV, explore the comprehensive materials available here: https://github.com/Icedcool/StarknetMEV/.

If you want to discuss MEV on Starknet or have valuable insights to share, join our MEV discussion group.

Understanding Starknet Data with Dune Dashboards

LordGhostX — Tue, 27 Aug 2024 04:09:11 +0000

In my previous posts, Data to Queries: Exploring Starknet Data with Dune and Queries to Insights: Visualizing Starknet Data with Dune, I walked you through accessing Starknet's raw data on Dune, writing SQL queries to extract that data, and visualizing it through dashboards.

Now, let's dive deeper into understanding and interpreting the data in existing Starknet dashboards. This will help you gain meaningful insights from the visualizations, make informed decisions, and fully leverage the value of the data.

Exploring Starknet Dashboards

Here are a few dashboards on the Caravanserai page that you might find interesting:

Starknet Overview: Provides a comprehensive view of key metrics like transaction volumes, active addresses, and contract deployments.
Starknet DEXs: Provides detailed insights into liquidity flows, daily swap volumes, and market sentiment across various decentralized exchanges.
Starknet Arbitrage Tracker: Tracks real-time arbitrage opportunities by monitoring price discrepancies and potential profits within specific blocks.
Starknet DEX Analytics: Analyzes liquidity distribution, swap activities, and identifies market trends and whale movements.
Starknet MEV: Focuses on monitoring and analyzing Maximal Extractable Value (MEV) within Starknet.

Starknet Overview Dashboard

Let's start with the Starknet Overview dashboard and understand its key metrics.

The Strk Price section shows the price trends of Starknet's STRK token over time, comparing it with Ethereum (ETH). This comparison helps you assess STRK's performance in relation to ETH.

The Blocks section provides insight into network activity, including:

Average Daily Created Blocks: The average number of blocks created daily indicates the network's activity.
Average Daily Transactions per Block: Tracks the average number of transactions per block, showing how busy the network is and how much each block is used.
Average Block Creation Time: Shows how long it usually takes to make a block, helping you see if the network is running faster or slower.
Blocks, Average Block Creation Time, and Transactions per Block Over Time: A historical view of how many blocks were made, how long they took, and how many transactions were in each block so you can spot trends and changes over time.

The Transactions section provides insights into network transaction activity:

Transaction Count and TPS Over Time: Tracks the total number of transactions and the transactions per second (TPS) over time, giving insight into the network's activity and performance.
Average Daily Transaction Count: The average number of daily transactions, reflecting the network's regular usage.
Average TPS: Displays the average transactions per second, highlighting the network's speed and efficiency.
Daily Share L1 vs L2 Accepted Transactions: Compares the daily distribution of transactions accepted on Layer 1 (L1) vs Layer 2 (L2), showing where most activity occurs.
Daily Share of Blob vs Call Data Transactions: This chart breaks down the daily transactions into blob data and call data types, providing insight into the nature of the transactions being processed.
Transaction Reversion Rate and Number of Successful and Reverted Transactions Over Time: Monitors the rate at which transactions are reverted and compares successful vs. reverted transactions over time.
Average Transactions per User: The average number of transactions each user makes indicates user engagement and activity levels.
Daily Number of Transactions by User Type: Categorizes and counts transactions by different users, providing insight into how various user groups interact with the network.
Daily Share of Reverted Transactions by User Type: Analyzes which types of users are most likely to have transactions reverted, offering a deeper understanding of transaction reliability by user category.

The Users section focuses on user engagement:

Number of Active and New Addresses Over Time: Tracks the number of active users and new addresses created over time, showing growth and engagement trends.
Total Users: Displays the total number of unique users on the network, giving a sense of the overall user base size.
Average Daily Active Users: The average number of users actively engaging with the network daily, reflecting consistent user activity and network usage.

The Fees section analyzes transaction costs on the network:

Total Paid Fees and Average Fee per Transaction Over Time: Tracks the total fees paid and the average fee per transaction over time, giving insights into cost trends and network demand.
Total Paid Fees: The cumulative fees paid on the network reflect how cheap or expensive it is to use Starknet.
Average Fee per Transaction: Displays the average fee users pay per transaction, helping you understand the cost efficiency of transactions over time.

Starknet DEXs Dashboard

Let's explore data on Starknet's decentralized exchanges using the Starknet DEXs dashboard.

The Overview section provides a snapshot of DEX activity:

Average Daily Volume: This shows the average daily trading volume on Starknet DEXs, indicating overall market activity.
Average Daily Swaps: Tracks the average number of swaps executed daily, reflecting the frequency of trades on the platform.
Average Daily Swappers: Displays the average number of unique users making swaps daily, highlighting user engagement.
Daily Volume of Swaps, Number of Swaps, and Number of Swappers: Offers a detailed view of daily trading volume, swap counts, and the number of active users, helping you track trends over time.

The DEXs section details activity across different DEXs:

Daily Swapped Volume by DEX: Displays the total trading volume on each DEX per day, showing where most trading activity is occurring.
Daily Share of Volume by DEX: Breaks down the percentage of the total daily volume attributed to each DEX, highlighting the market share of each platform.
Daily Number of Swaps by DEX: Tracks the daily number of swaps executed on each DEX, indicating how frequently trades happen across different exchanges.
Daily Share of Swaps by DEX: Shows the proportion of total daily swaps that occur on each DEX, providing insight into which platforms are most active.

The Tokens section offers insights into token trading:

Daily Volume of Swaps by Token: Shows the total trading volume for each token per day, indicating which tokens are being traded the most.
Daily Share of Volume by Token: This breaks down the percentage of the total daily trading volume attributed to each token, highlighting different tokens popularity and trading activity.

Starknet MEV Dashboard

Next, let's explore MEV activity on Starknet using the Starknet MEV dashboard.

The Liquidation Overview section offers a comprehensive view of liquidation activities:

Total Repaid Debts: Shows the cumulative amount of debts repaid through liquidations, reflecting the scale of debt resolution on the network.
Total Liquidated Collaterals: Displays the total value of collaterals liquidated, giving insight into the assets seized during liquidation events.
Total Paid Fees: Indicates the total fees paid in the process of liquidations, highlighting the cost associated with these transactions.
Daily Number of Liquidations, Liquidators, and Users: Tracks the daily count of liquidation events, the participants involved in liquidating assets, and the users affected.
Daily Volume of Liquidated Debts, Collaterals, and TX Fees: A daily breakdown of the volume of debts liquidated, collaterals seized, and transaction fees incurred, offering a detailed view of the liquidation process over time.

The Liquidators section provides detailed information about those executing liquidations:

The Platforms & Tokens section breaks down liquidation data by platform and token:

Daily Net Profit and Loss of Liquidators by Platform: Tracks the daily net profit and loss of liquidators, categorized by platform, showing which platforms are most profitable or costly for liquidators.
Share of Debts by Token: Displays the proportion of total debts associated with each token, highlighting which tokens are most commonly involved in liquidation events.
Share of Collaterals by Token: This chart shows the distribution of collateral types by token, providing insight into the assets most frequently used as collateral in liquidations.

The Liquidation Transactions section provides detailed information on liquidation transactions on Starknet:

Advanced Analysis with Materialized Tables

Materialized tables on Dune store query results for faster analysis or reusing query results in other queries. You can explore these through Caravanserai's public queries.

For example, you can use the dune.caravanserai.result_starknet_dex_pairs table to identify the most actively traded pairs:

SELECT
  pair_name,
  SUM(volume) AS total_volume,
  AVG(price) AS average_price,
  MAX(price) AS max_price,
  MIN(price) AS min_price
FROM
  dune.caravanserai.result_starknet_dex_pairs
WHERE
  timestamp >= NOW() - INTERVAL '1 hour'
GROUP BY
  pair_name
ORDER BY
  total_volume DESC;

Or retrieve recent liquidation events using the dune.caravanserai.result_starknet_liquidations table:

SELECT
  liquidation_id,
  liquidated_token,
  liquidation_value,
  timestamp
FROM
  dune.caravanserai.result_starknet_liquidations
WHERE
  timestamp >= NOW() - INTERVAL '24 hours'
ORDER BY
  timestamp DESC;

If you're new to writing and running queries on Dune, I recommend starting with my article Data to Queries: Exploring Starknet Data with Dune.

Conclusion

In this article, we explored Starknet dashboards and materialized tables on Dune and learned how to interpret them for valuable insights. These dashboards can help you better understand Starknet's operations, optimize strategies, and stay ahead of market trends.

If you're interested in analyzing Starknet data, finding opportunities in the ecosystem, or sharing valuable insights, join our MEV discussion group to connect with like-minded individuals.

Data to Queries: Exploring Starknet Data with Dune

LordGhostX — Mon, 19 Aug 2024 00:36:26 +0000

Dune has recently integrated Starknet, one of Ethereum's most promising Layer 2. This integration allows you to easily interact with, visualize, query, and analyze data from Starknet directly on Dune.

If you're looking to create dashboards, you may want to check out my detailed guide: Queries to Insights: Visualizing Starknet Data with Dune.

This article will walk you through querying Starknet data on Dune, which you can use to create insightful dashboards.

Accessing Starknet's Data

Dune transforms raw blockchain data into SQL databases, enabling you to write DuneSQL queries to retrieve and analyze the data, visualize it with dashboards, develop data-driven applications, and many more.

1. Open the "Create" dropdown menu in Dune's navigation bar

2. Choose "New Query" from the options

3. You'll be directed to the "New Query" page

4. Click "Raw Blockchain Data" on the Data Explorer

5. Search and click on "starknet"

You can now explore and filter all Starknet-related tables, accessing data on blocks, calls, on-chain events, and transactions.

6. Explore the data tables

You can also filter columns within any table. In the image below, you can see the data types of the fields and the SQL editor on the right side for writing queries.

Take some time to explore the available data on Dune. When ready, you can start writing queries to extract the needed data.

Writing DuneSQL Queries for Starknet

DuneSQL is an easy-to-read query language similar to other SQL variations. To learn more about DuneSQL and its features, check out the Dune documentation.

1. Query Transaction Details by Hash

Let's write a query to retrieve the details of a transaction by its hash:

SELECT
  *
FROM
  starknet.transactions
WHERE
  transaction_hash = 0x0716bd0f37d806ceb45dff6d4d0620034c1d127df8a619e6febe9dfc034ef33e;

This query fetches all the columns for the transactions table with a specified transaction_hash. Click the "Run" button to execute:

You can create parameters for your queries that allow you to adjust variables on the fly and explore data dynamically:

SELECT
  *
FROM
  starknet.transactions
WHERE
  transaction_hash = {{tx_hash::text}};

2. Querying Latest Block Details

Let's write a query to retrieve the details of the latest block on Starknet:

SELECT
  time,
  number,
  tx_count,
  hash
FROM
  starknet.blocks
ORDER BY
  date DESC
LIMIT
  1;

This query fetches the creation time, block number, transaction count, and hash of the latest created block on Starknet. Click the "Run" button to execute:

3. Sorting Smart Contracts by Transaction Count

Let's write a more practical query that fetches the top contracts on Starknet by the number of transactions over the last 30 days:

SELECT
  contract_address,
  COUNT(*) AS transaction_count
FROM
  starknet.transactions
WHERE
  block_time >= CURRENT_TIMESTAMP - INTERVAL '30' DAY
  AND contract_address IS NOT NULL
GROUP BY
  contract_address
HAVING
  COUNT(*) > 0
ORDER BY
  transaction_count DESC
LIMIT
  10;

Explanation of the query:

SELECT: The query retrieves two columns: contract_address and a count of all transactions associated with each contract_address, labeled transaction_count.
FROM: The data is pulled from the transactions table within the starknet schema.
WHERE:
- Filters the transactions to include only those that occurred within the last 30 days (block_time >= CURRENT_TIMESTAMP - INTERVAL '30' DAY).
- Excludes rows where contract_address is NULL to ensure you only count transactions tied to an actual contract address.
GROUP BY: Groups the results by contract_address so that the COUNT function aggregates the number of transactions per contract address.
HAVING: This condition ensures that only contract addresses with at least one transaction (HAVING COUNT(*) > 0) are included in the results. Although the WHERE clause already filters out NULL addresses, this is an additional safeguard.
ORDER BY: Orders the grouped results by transaction_count in descending order, so the most active contract addresses appear first.
LIMIT: Limits the output to the top 10 contract addresses with the highest transaction counts.

4. Retrieving $STRK Token Price

You can also query the price of tokens on Dune! Here's a modified query, inspired by the Starknet Overview Dashboard, that retrieves the price of Starknet's $STRK token:

SELECT
  DATE(minute) AS "Date",
  AVG(price) AS "STRK Price"
FROM
  prices.usd
WHERE
  blockchain = 'ethereum'
  AND contract_address = 0xca14007eff0db1f8135f4c25b34de49ab0d42766
GROUP BY
  1
ORDER BY
  "Date"

Visualizing Results From Your Queries

You can take your queries a step further by visualizing their generated data and even adding it to a dashboard. We'll be visualizing the query that fetches $STRK prices.

1. Click the "New" button beside "Query Results"

2. Select the chart type you want to create

3. Click the "Add Visualization" Button

4. Explore Dune's various visualizations

You can take your queries and visualizations further by adding them to a dashboard, allowing the data to flow and tell compelling stories.

Starknet Resources on Dune

Dune provides a growing selection of Starknet-specific queries and dashboards, offering valuable insights into the ecosystem. One standout resource is the Caravanserai page, featuring in-depth dashboards and queries of various Starknet metrics: https://dune.com/caravanserai

For more Starknet-related content, use Dune's search function to discover queries, dashboards, and contributors: https://dune.com/discover/content/relevant?q=starknet. As Starknet evolves, we can expect these analytics resources on Dune to expand, offering even deeper insights into its on-chain data.

Conclusion

In this article, we've explored how to query and visualize Starknet data on Dune and how to create dynamic queries by creating parameters for them.

If you're interested in analyzing Starknet data, finding opportunities in the ecosystem, or sharing valuable insights, join our MEV discussion group to connect with like-minded individuals.

Queries to Insights: Visualizing Starknet Data with Dune

LordGhostX — Fri, 09 Aug 2024 20:06:15 +0000

Starknet's on-chain data offers a wealth of insights into its blockchain ecosystem. It provides a comprehensive view of user activity, transaction patterns, and network interactions, allowing analysts to discover valuable trends and behaviors. The ability to analyze on-chain data is important for making informed decisions, optimizing protocols, and identifying emerging opportunities within the network.

This article will guide you through transforming raw Starknet data into compelling visual narratives using Dune. We'll explore leveraging Dune's powerful querying capabilities and visualization tools to create dynamic, interactive dashboards that bring Starknet's on-chain data to life.

What is Starknet?

Starknet is an Ethereum Layer 2 (L2) designed to deliver high transaction throughput and low costs while maintaining Ethereum's core principles. Starknet uses Zero-Knowledge (ZK) technology, specifically ZK-STARK proofs, to verify transaction validity off-chain before committing them to the Ethereum mainnet.

Starknet's Architecture

Starknet's architecture is built to optimize the Ethereum ecosystem:

Scalability: By processing transactions off-chain and then batching them for submission to Ethereum, Starknet significantly increases the network's capacity.
Cost Efficiency: The reduced computational load on Ethereum translates to lower gas fees for users.
Security: ZK-STARK proofs ensure that all transactions are valid without revealing sensitive information, maintaining high security.
Compatibility: Starknet is designed to work seamlessly with Ethereum, allowing developers to build scalable applications that benefit from Ethereum's established ecosystem.

The Value of Starknet Data

Starknet's on-chain data is a goldmine of information for various stakeholders in the blockchain space:

Developers: Can analyze transaction patterns to optimize their dApps and smart contracts.
Investors: Gain insights into user adoption, token flows, and emerging trends.
Researchers: Study the effectiveness of L2 solutions and their impact on the broader Ethereum ecosystem.
Protocol Designers: Use data to identify bottlenecks and improve network efficiency.
Market Analysts: Track the growth of the Starknet ecosystem and its integration with DeFi and other blockchain applications.

By analyzing Starknet's on-chain data, we can gain valuable insights into the network's health, user behavior, and the adoption of Layer 2 solutions in the Ethereum ecosystem.

What is Dune?

Dune is a powerful blockchain data analytics platform that enables users to query, visualize, and share insights from various blockchain networks, including Starknet. By leveraging Dune's capabilities for Starknet analysis, you can:

Query Complex Data: Use Dune SQL to query and extract Starknet's on-chain data, allowing for deep dives into specific metrics and trends.
Create Dynamic Visualizations: Transform raw data into intuitive charts, graphs, and tables that make complex blockchain information understandable.
Develop Compelling Dashboards: Combine multiple queries and visualizations to create holistic views of Starknet's ecosystem, monitoring key real-time performance indicators.
Track Ecosystem Metrics: Monitor essential data points such as daily active addresses, transaction volumes, gas usage, and smart contract interactions.
Analyze Token Flows: Gain insights into token transfers, liquidity pools, and trading patterns within the Starknet network.
Collaborate and Share: Leverage Dune's community features to build upon existing queries, share your findings, and contribute to the collective understanding of Starknet's performance.

Key Starknet Data Points

Let's explore some of the key areas where we can extract valuable insights on Starknet:

Transaction Data: Information about each transaction, including unique identifiers, involved addresses, transferred values, gas fees, and timestamps. This helps understand network usage patterns and economic activity.
Block Data: Provides a structural overview of the blockchain, including block numbers, hashes, timestamps, and transaction counts. This helps analyze network growth and validate the chain's integrity.
Smart Contract Data: Information about deployed contracts and their current state helps developers and analysts study decentralized application (dApp) usage and smart contract interactions.
Account Data: This includes account balances and transaction counters (nonce), giving a view of user activity and asset distribution across the network.
Rollup-Specific Data: Unique to Layer 2 solutions like Starknet, this includes ZK-STARK proofs and state updates, which are important for understanding the roll-up mechanism's efficiency and security.
Token Data: Tracks the creation, transfer, and usage of various tokens on the network, including both fungible and non-fungible tokens (NFTs).
Protocol Metrics: Provides insights into the overall health and performance of the network, including total value locked (TVL), daily active users, and transaction throughput.
Interoperability Data: Captures cross-chain interactions and bridge usage information, highlighting Starknet's connection to the broader Ethereum ecosystem.

Starknet Resources on Dune

Visualizing Data with Dune Dashboards

Dune dashboards are a powerful tool for visualizing and presenting blockchain data in an intuitive, interactive format. These dashboards allow you to create comprehensive analytical tools by combining various data visualization widgets and explanatory text. Key features include:

Versatile Widgets: You can use various visualization types such as charts, graphs, tables, and metrics to represent data effectively.
Text Integration: Explanatory text widgets can be added to your data visualizations to provide context, analysis, or instructions.
Multimedia Support: Images and GIFs can be embedded within text widgets, enhancing your dashboard's visual appeal and clarity.
Customization: Dashboards are highly customizable, allowing you to arrange widgets and create a cohesive design that best communicates your insights.
Sharing and Collaboration: Easily share your dashboards with others or collaborate on their development.

Creating Dune Dashboards

To create a new dashboard on Dune:

1. Open the "Create" dropdown menu in Dune's navigation bar

2. Choose "New Dashboard" from the options

3. Enter a descriptive name that reflects your dashboard's focus

4. Click "Save and Open" to create your dashboard

5. Click "Edit" to modify the dashboard and add visualizations

6. Click "Add Visualization" and search for queries to visualize

7. Click "Add" next to each query you want to include

8. Click "Done" to finalize your dashboard setup

After adding queries and arranging the visualizations, your dashboard might resemble the following example:

Important: Dune dashboards are not real-time. To view the latest Starknet data and keep your analysis current:

Refresh the dashboard by clicking the Run button in the top right corner.

Update specific widgets using their refresh icons for dashboards with multiple queries.

Leverage Dune's scheduling feature to automate regular updates for key queries.

Consider performance impact when refreshing frequently, especially for complex queries.

Best Practices

The following best practices will help you design insightful, accessible, and engaging dashboards that maximize the value of Dune:

Group related metrics: For intuitive navigation, organize by categories such as network activity, economics, and performance.
Prioritize key metrics: Focus on the most important data points to maintain clarity and prevent information overload.
Maintain visual consistency: Use similar chart types for related data and a cohesive color scheme.
Leverage interactivity: Implement filters, parameters, and drill-down options for in-depth exploration.
Optimize layout: Arrange widgets logically, placing the most important data in prominent positions.
Provide context: Add concise text explanations to help users interpret data and understand its significance.
Ensure performance: Optimize query efficiency and widget loading for a smooth user experience.
Share and collaborate: Share your dashboard and engage with the community by forking existing dashboards.
Enhance discoverability: Use clear titles and descriptions to make your dashboard discoverable in Dune
Clarify data freshness: Communicate the update frequency for time-sensitive analysis and consider including a Last Updated timestamp.

Conclusion

In this article, we've covered the basics of Starknet and why its data is so valuable, explored Dune, and demonstrated how to find and visualize key resources. We've also learned how to create and manage dashboards for Starknet's unique data landscape.

If you're interested in analyzing Starknet data, finding opportunities in the ecosystem, or sharing valuable insights, join our MEV discussion group to connect with like-minded individuals.

Starknet Protocols: Key Features and MEV Opportunities

LordGhostX — Wed, 31 Jul 2024 22:30:07 +0000

You can build a wide range of dApps on Starknet just as you would on Ethereum or other Layer 1 blockchain. Starknet hosts numerous dApps and protocols that offer impressive features and present various MEV opportunities.

This article explores some protocols and dApps built on Starknet and the MEV opportunities they present.

Top Protocols on Starknet

By building on Starknet, you can access a scalable, cost-effective blockchain that leverages Ethereum's benefits while allowing you to innovate beyond limits. You can find the top projects on Starknet by total value locked (TVL) on DeFi Llama:

Nostra

Nostra is a comprehensive DeFi protocol on Starknet that integrates multiple financial services into a single platform. Nostra is on a mission to power flexible crypto actions and provide an excellent user experience. Key features of Nostra include:

Staking: Liquid-stake your $STRK to earn yield and receive Nostra Staked STRK (nstSTRK) tokens. Use nstSTRK in DeFi apps across Starknet and other dApps on Ethereum.
Lending and Borrowing: With Nostra, you can lend your crypto assets to earn interest or use your deposits to borrow other assets. It supports stablecoins like USDT, USDC, and DAI, as well as volatile assets like ETH and WBTC.
Liquidity Pools: You can deposit your crypto into Nostra's liquidity pools to earn swap fees and yield.
Swapping: You can swap and trade various tokens on Starknet. Nostra uses AVNU's DEX aggregator to tap into Starknet's liquidity network, ensuring you get the best available prices.
Bridging: Nostra supports seamless bridging of assets across more than 20 blockchains, allowing you to transfer your crypto quickly and securely with minimal fees.
UNO Stablecoin: Nostra also offers the $UNO stablecoin, a fully collateralized, Starknet-native USD stablecoin. UNO provides a stable asset for transactions, earning yield, and hedging against volatility.

Ekubo Protocol

Ekubo is a next-generation Automated Market Maker (AMM) on Starknet, designed for super-concentrated liquidity, extensibility, and providing users with low fees and high returns. Key features of Ekubo include:

Swapping: You can efficiently swap your tokens with the lowest gas fees. Ekubo also offers Dollar-Cost Average (DCA) orders, allowing you to place orders that execute over time for more efficient and programmatic entry and exit from positions.
Liquidity Pools: You can provide liquidity to Ekubo's pools to earn returns, benefiting from optimized liquidity usage and reduced gas costs.
Extensions: You can leverage Ekubo's efficient and secure core AMM protocol to create new order types, trading strategies, oracles, and many more using extensions.

zkLend

zkLend is a native lending protocol on Starknet that provides secure and user-friendly lending and borrowing services that are efficient and decentralized. Key features of zkLend include:

Deposits: You can deposit assets into zkLend's lending pools to earn competitive interest based on real-time supply and demand.
Borrowing: You can borrow against your deposits, using them as collateral for secure and efficient access to funds.
Flash Loans: zkLend supports flash loans for complex and time-sensitive transactions, allowing you to execute loans without upfront collateral requirements.

Nimbora

Nimbora is a Starknet-based infrastructure enabling users to earn and borrow on Layer 1 protocols with significantly reduced costs. Key features of Nimbora include:

Earning: You can deposit into Nimbora's pools to earn interest on your assets. Nimbora leverages multiple strategies to maximize yield opportunities and returns.
Borrowing: You can use your deposits as collateral to borrow other assets from Nimbora's pools. Nimbora helps you discover the best borrowing rates on Ethereum, enhancing your investment strategies for maximum efficiency.

Vesu

Vesu is a fully open, transparent, immutable, and user-friendly lending protocol built on Starknet. Vesu allows you to supply and borrow crypto assets without intermediaries, offering a decentralized public infrastructure that is free for everyone. Key features of Vesu include:

Earning: You can deposit assets into Vesu's lending pools to earn returns and yield. Vesu ensures maximum capital efficiency by aggregating assets to serve as shared liquidity for all borrowers in the same pool.
Borrowing: You can use your deposited assets as collateral to securely and efficiently borrow other assets, gaining access to funds without relying on intermediaries.

mySwap

mySwap is a concentrated liquidity AMM on Starknet, designed to provide an exceptional user experience and APR for users and liquidity providers based on the Uniswap v2 model. Key features of mySwap include:

Swapping: You can swap your tokens with low gas fees and minimize slippage. By concentrating liquidity within specific price ranges, mySwap facilitates efficient trades and maximizes your returns.
Liquidity Pools: You can earn higher returns on mySwap by concentrating your liquidity within specific price ranges, allowing for more efficient trades and optimized yield.

MEV Opportunities on Starknet Protocols

Starknet processes transactions using a First-Come-First-Serve (FCFS) model, preventing front-running. However, there are other MEV (Maximum Extractable Value) opportunities you can explore:

Arbitrage

Arbitrage involves taking advantage of price differences for an asset across different exchanges or markets. You can profit by simultaneously buying low in one market and selling high in another, provided the transaction fees don’t consume all the profit. Arbitrage benefits traders, helps balance prices across protocols, and improves market liquidity.

For example, suppose Ethereum (ETH) is priced at $1,900 on Nostra and $1,920 on Ekubo. You can purchase ETH on Nostra and immediately sell it on Ekubo, pocketing a $20 profit per ETH (minus transaction fees). However, performing precise calculations to ensure your arbitrage remains profitable is important, as transaction fees and market fluctuations can impact your potential gains.

There are various forms of arbitrage that extend beyond the basic concept:

Cross-Layer Arbitrage: Conducting arbitrage across multiple blockchain networks or layers. For example, you buy ETH on Ekubo (Starknet), bridge to Ethereum, and sell it on Uniswap (Ethereum).
Pool to Pool Arbitrage: Taking advantage of price discrepancies between different liquidity pools within the same or different protocols. For example, you buy a token from Nostra's liquidity pool and sell it in Ekubo's pool, where the price is higher.
Atomic Arbitrage: Executing a series of transactions within a single block to ensure all steps are completed simultaneously, minimizing risk. For example, you can use flash loans to buy and sell assets within the same transaction to capture price differences.
Triangular Arbitrage: Involves trading between three different assets to take advantage of discrepancies in their exchange rates. For example, you sell STRK for ETH, ETH for NSTR, and then NSTR back to STRK.
Statistical Arbitrage: Using statistical models to identify and exploit pricing inefficiencies. For example, you use historical data to predict and trade assets based on expected price movements.

To ensure profitability, each type of arbitrage requires a deep understanding of the markets and precise calculations, including transaction fees and market volatility.

Liquidation

Liquidation occurs when borrowers fail to maintain their collateral ratios and their assets are sold off. By monitoring lending protocols like zkLend and Vesu for undercollateralized loans, you can execute liquidations and earn a portion of the collateral as a reward. This process helps keep the lending platform solvent and protects lenders' funds.

Conclusion

Starknet's growing DeFi ecosystem presents exciting opportunities for users and developers. Its FCFS model reduces traditional MEV strategies like front-running, but searchers can still profit through arbitrage and liquidations.

As Starknet evolves, we can expect more innovative protocols and MEV opportunities. Whether you're seeking better yields, developing new dApps, or exploring MEV strategies, Starknet offers many possibilities.

If you want to discuss MEV on Starknet or have valuable insights to share, join our MEV discussion group.

Introduction to MEV on Starknet

LordGhostX — Thu, 25 Jul 2024 15:04:59 +0000

Maximal extractable value (MEV) refers to the maximum value that can be extracted from block production more than the standard block reward and gas fees by including, excluding, and changing the order of transactions in a block.

MEV has become more popular since Ethereum transitioned to Proof-of-Stake (PoS). The rise of Layer 2's like Starknet introduced new dynamics to the MEV landscape. Understanding MEV within these ecosystems is important in comprehending the implications of on-chain competition and its effects on network stability and user experience.

MEV on Ethereum

MEV in the Ethereum ecosystem presents a complex duality with beneficial and potentially detrimental impacts on the network.

Inefficiency and unfairness:

MEV can lead to network congestion and higher gas fees.
It may create an uneven playing field, favoring well-resourced and technically sophisticated actors.

Innovation and security incentives:

MEV drives advancements in transaction execution mechanisms.
It offers additional incentives to enhance network security and reliability.

Market health:

MEV helps with pricing consistency across protocols. Active MEV ensures that prices remain the same across different platforms, reducing the risk of bad debt on lending positions.
While some MEV practices can be harmful, useful MEV, such as arbitrage, helps maintain healthy markets by improving liquidity and price discovery.

MEV extraction strategies

There are multiple strategies to extract value from block production:

Front-running: Placing a transaction ahead of a known future transaction. For example, if a large trade is detected in the mempool, a front-runner might place a buy order just before it, then sell immediately after, profiting from the price impact.
Back-running: Positioning a transaction immediately after a target transaction. This can be used to capitalize on state changes caused by the target transaction, such as taking advantage of a newly created arbitrage opportunity.
Sandwich attacks: Combining front-running and back-running to profit from price movements. A sandwicher places a buy order before a large swap, then a sell order immediately after, profiting from the temporary price impact.
Liquidations: Liquidating under-collateralized positions to earn fees or acquire discounted collateral. Bots monitor lending protocols for positions close to liquidation thresholds and quickly execute liquidation transactions when conditions are met.

MEV infrastructure and tooling

The MEV ecosystem has developed various tools and platforms to manage and optimize MEV extraction. These solutions seek to balance the benefits of MEV while minimizing its negative impacts:

Flashbots: A research and development organization dedicated to addressing the challenges and opportunities of MEV in the Ethereum ecosystem.
Eden Network: A transaction ordering protocol designed to democratize MEV extraction and protect users from malicious MEV practices.
Manifold Finance: A platform focusing on gas cost optimization and MEV protection for DeFi users, enhancing the overall efficiency and security of transactions.
MEV-Boost: An implementation of Proposer-Builder Separation (PBS) for Ethereum validators. It allows them to access a competitive block-building market and earn a fair share of the MEV extracted in the blocks they propose.

Searchers, solvers, and builders

The MEV ecosystem on Ethereum involves several key participants:

Searchers: Participants who identify and exploit MEV opportunities using various strategies.
Solvers: Participants that optimize and execute MEV strategies. They work closely with searchers to refine extraction processes, ensuring maximum efficiency and profitability.
Builders: Participants responsible for constructing bundles of transactions submitted to the network. They collaborate with searchers and solvers to assemble transactions in a way that maximizes extracted value.

Trading bots (searchers)

Most MEV strategies are executed by automated trading bots that identify and exploit opportunities in real-time:

Arbitrage bots: Monitor token prices across exchanges and protocols, executing trades to profit from differences and help ensure uniform prices.
Front-running bots: Place buy orders before executing trades to capitalize on price movements.
Back-running bots: Identify opportunities after specific transactions and quickly execute follow-ups to extract value.
Sandwich bots: Implement sandwich attacks by placing orders before and after large trades to profit from price impacts.
Liquidation bots: Monitor lending protocols for positions near liquidation thresholds, quickly executing liquidation transactions when conditions are met.

Many of these trading bots are integrated within various MEV infrastructure platforms, aiming to reduce the risks of manipulative tactics.

MEV on Starknet

Starknet is an Ethereum Layer 2 (L2) designed to deliver high transaction throughput and low costs while maintaining Ethereum's core principles.

Starknet operates as a Zero-Knowledge Rollup (ZK-Rollup), using a STARK proof to bundle transactions off-chain before submitting a single proof to the Ethereum mainnet. MEV on L2's like Starknet differs from L1 due to increased throughput, reduced transaction costs, First-Come-First-Serve (FCFS) model, account abstraction, and censorship resistance. Here's how:

Higher TPS: L2's process more transactions simultaneously, reducing MEV opportunities and lowering the profitability of front-running.
Lower transaction fees: Reduced fees decrease the incentives for MEV extraction.
First-Come-First-Serve (FCFS) model: The Starknet sequencer processes transactions as they are received, eliminating the need for transaction reordering and mempool monitoring.
Account abstraction: Starknet's native account abstraction allows anyone to handle transactions, minimizing exposure to MEV.
Censorship resistance: Starknet's design prevents entities from monopolizing MEV extraction, ensuring a fairer ecosystem.

However, MEV opportunities still exist on Starknet. The interaction between L1 and L2 creates complex transaction flows that are "MEV-exploitable", allowing searchers to capitalize on information and timing differences between these layers for profit.

Profitable MEV strategies on L2s

Due to the L2 design, MEV strategies that rely on gas fee manipulation are less effective. However, searchers are still exploring profitable strategies. Here are some notable ones:

Atomic arbitrage

Atomic arbitrage involves profiting from price differences across exchanges and protocols (such as Ekubo and Nostra). The lower transaction fees and high TPS on L2s like Starknet make it easier and faster for bots to execute these arbitrage strategies efficiently. For a practical guide on creating an atomic arbitrage bot, check out Maksim's blog and Matteo's blog.

Liquidations

MEV bots can profit from liquidations on lending protocols when collateral values drop below certain thresholds. When this happens, bots can quickly act to liquidate these positions for profit. For a practical guide on building a liquidation bot, check out Kristan's blog on ZKLend.

Statistical arbitrage

This strategy uses statistical models to predict price movements and identify profitable trading opportunities. While it can be riskier, the high throughput and low transaction costs on L2s make it easier to execute effectively.

Cross-layer sandwich attacks

Cross-layer sandwich attacks involve monitoring interactions between L1 and L2 to place transactions that manipulate prices. Searchers can observe pending transactions on L1 and strategically place transactions on L2 to influence prices in their favor.

All these strategies are possible on Ethereum and its L2s. However, many L2s are exploring MEV auctions and decentralized sequencers to mitigate MEV risks for end users.

Starknet ecosystem strategies and MEV mitigation

Starknet actively employs strategies to create a fair ecosystem and mitigate MEV on the mainnet. The community is engaged in healthy discussions and multiple initiatives focused on addressing MEV on Starknet.

Starknet is improving TPS and performance by implementing parallel transactions, resulting in lower fees, reduced congestion, and fairer transactions.

The community is also developing tools and engaging in conversations to address MEV's impact. Here are some notable ones:

Starknet Analyzer: This repository is valuable for extracting and analyzing MEV-related data on Starknet. Check it out: https://github.com/d-s-i/starknet-analyzer.
Entro: A Python package providing functionalities for Starknet on-chain exploration, analyzing Starknet contract data, and data backfilling. Check it out: https://github.com/nethermindeth/entro.

Looking ahead, the future of L2s is promising, with the potential implementation of decentralized sequencers to decentralize the network and MEV auctions to reduce MEV risks.

Conclusion

I hope this article has introduced you to the world of MEV on Ethereum and Starknet. You should now be able to compare the MEV dynamics between these layers and better understand how MEV works on Starknet.

If you want to discuss MEV on Starknet or have valuable insights to share, join our MEV discussion group.

This piece is part of a series exploring MEV on Starknet, so stay tuned for more insights and discussions in the upcoming articles.

Build and deploy a web app with Python, Flask, and Doppler

LordGhostX — Mon, 04 Oct 2021 22:57:07 +0000

Picture this: You’re building an app that uses global configuration values (public and private keys, environment execution configurations, database credentials, domain names, and URLs), and you need to update them frequently. The first solution you’ll come up with is to manage your global values using ENV files. Still, soon enough, you’ll realize that this approach can be a burden and is unscalable, especially in a distributed architectural system.

Suppose you found a solution that serves as a universal secrets manager and allows you to synchronize environment variables at scale efficiently. Would this help you solve your problem better than ENV files? The answer is YES, and the best solution for your issue is Doppler.

This tutorial will teach you how to integrate Doppler into your apps and store your secrets with it. In this article, you will build and deploy a demo app with Python (Flask) and Heroku, then manage its secrets and environment variables using Doppler.

What is Doppler?

Doppler is a universal secrets manager that allows you to efficiently synchronize environment variables across devices, environments, and team members. It will enable you to store, share, and access secrets from a centralized source with support for collaboration, access controls, versioning, secrets documentation, and many more.

Benefits of using Doppler

Doppler provides many unique features and functionalities that make it more than just a storage for secrets. Here are some of them:

Doppler simplifies the synchronization of secrets and app configuration across devices, environments, and team members.
Doppler allows your teams to collaborate and organize secrets across multiple projects and environments.
Doppler helps developers automate the process of updating secrets in their apps across several locations.
Doppler lets you deploy everywhere, be it Docker, serverless environments, or anywhere you work.
Doppler boosts your team's productivity by removing the burden of keeping ENV files in sync.
Doppler supports secrets versioning, meaning you can easily track changes in real-time and roll back broken changes.
Doppler also lets you create documentation and notes for secrets values, so you know their usage.

Getting started with Doppler

To continue with this tutorial, you need an account with Doppler. Create one on the Doppler website using your email address, Google account, or Secure SSO if you don't have one.

Step 1: Create a Doppler Project

Projects are used in Doppler to universally organize and manage your secrets from local development to production.

Click on the Create Project button in the Doppler dashboard, as shown in the image below:

Step 2: Store your secrets in Doppler

To store secrets in Doppler, select the Project you want to configure from your dashboard. Click on the Project you created earlier:

Then, choose the environment execution mode (development, staging, production) to configure your secrets. Select the dev config:

Next, configure the Project config and store secrets in it. Click on the Add First Secret button to add secrets manually or the Import Secrets button to fetch them from an ENV, JSON or YAML file.

Don't forget to press the Save button after adding/updating your secrets in Doppler.

Step 3: Install the Doppler CLI

The Doppler CLI is a lightweight binary that allows you to access your secrets in any environment (local, development, CI/CD, staging, production). Follow this guide to install it on your machine.

Step 4: Authenticate the Doppler CLI

To connect the Doppler CLI to your account and access the secrets of your Projects, you need to be authenticated on it using an access token. Doppler CLI provides a doppler login command to authenticate yourself via your browser (for local development).

In the terminal, type the following:

doppler login

You only need to do this once per workplace. If you have multiple workplaces, you can scope each login to a separate directory.

Step 5: Set up a Doppler Project

After installing and authenticating the Doppler CLI, you need to configure it with a Project in your development environment to fetch its secrets. Doppler CLI provides a doppler setup command to configure your app with a Project.

Change the current directory to your app directory, then type the following in the terminal:

cd ./your/project/directory  
doppler setup

Now, select the Project and config you want to configure your app to use:

You can also use a doppler.yaml file to pre-configure the Doppler Project and config for your app. Follow this guide to learn how.

Doppler allows you to set up secrets for multiple projects on a single machine simultaneously by scoping them to specific directories.

Step 6: Inject Doppler into running processes

After setting up the Project and config of your secrets for your app, you can inject your secrets as environment variables into running processes using doppler run. Let’s see an example where we access our secrets from the Python shell.

In your terminal, type the following:

doppler run -- python3

import os  
print(os.getenv("DB_URL"))  
print(os.getenv("PAYMENT_KEY"))

Here, you launched a process (python3) and injected your secrets (that we created earlier) into it using doppler run. Then, you fetched the secrets from environment variables using os.getenv().

You can learn more ways to inject Doppler into running processes from this guide.

Setting up the Python app

Now that you have successfully injected your secrets into a running process using Doppler, let's start building a Python (Flask) web app that uses the secrets you stored in Doppler.

Step 1: Install the Flask web framework

You need to install Flask on your machine. In the terminal, type the following:

pip install Flask

Step 2: Create the Flask app

Create a file named app.py in the same directory you set up Doppler and save the following code in it:

from flask import *

app = Flask(__name__)


@app.route("/")
def home():
    return "Hello World!"


if __name__ == "__main__":
    app.run(debug=True)

Step 3: Start the app with Doppler

Start the app using doppler run to inject the secrets into the Flask server. In the terminal, type the following:

doppler run -- python3 app.py

When you open the http://127.0.0.1:5000 URL in your browser, you will get a response like this:

Integrating Doppler with Python and Flask

Let's fetch the secrets from Doppler and use them in the Python (Flask) web app. Since Doppler injects secrets as environment variables, you can easily retrieve them using the built-in os.getenv() function provided by Python like you would when using ENV files.

Step 1: Update the Flask app

Update the app.py you created earlier with the code below:

import os
from flask import *

app = Flask(__name__)


@app.route("/")
def home():
    response = f"""
    <h1>Doppler Secrets</h1>

    <h3>DB_URL: {os.getenv('DB_URL')}</h3>
    <h3>API_HOST: {os.getenv('API_HOST')}</h3>
    <h3>PAYMENT_KEY: {os.getenv('PAYMENT_KEY')}</h3>
    <h3>ADMIN_USERNAME: {os.getenv('ADMIN_USERNAME')}</h3>
    """

    return response


if __name__ == "__main__":
    app.run(debug=True)

Step 2: Start the app with Doppler

Start the app using doppler run and open the http://127.0.0.1:5000 URL in your browser to confirm Doppler injected your secrets correctly.

doppler run -- python3 app.py

Step 3: Update the secrets in Doppler

Change the secrets from your Doppler dashboard to see how easy, efficient, and scalable the synchronization of your environment variables has become.

Deploying the Flask app

Doppler provides integrations for many cloud providers, including Heroku, AWS, Digital Ocean, Azure, Docker, GCP, and many more to keep your secrets synchronized wherever you develop and deploy.

Doppler also simplifies the whole cloud provider integration process, only requiring you to set things up with a couple of clicks. Let's see how to integrate Doppler with Heroku.

Step 1: Create an app in Heroku

Step 2: Integrate Doppler with Heroku

Head over to the INTEGRATIONS tab in the stg (staging) config setup of your Doppler Project.

Don't forget to configure your secrets in the stg config of your Project.

Next, click on the Add Integration button and select Heroku from the list of cloud providers.

Then, authorize Doppler to have access to your Heroku account.

Finally, complete the integration process by setting the following:

Heroku Project Type: Either a Heroku App or Pipeline.
App: The name of the Heroku app that you created earlier (doppler-demo).
Config to sync: The config that Doppler should inject into your app (stg).
Import Options: Whether to import previously configured environment variables in Heroku to Doppler.

Click on the Set Up Integration button after providing all the necessary information.

You have successfully integrated Doppler with Heroku. Whenever you make changes (adding/updating) to your Project secrets, Doppler will automatically synchronize them with your Heroku app's Config Vars.

Step 3: Deploy your app to Heroku

Deploy your app to Heroku as you would typically do.

Conclusion

By integrating Doppler with Python (Flask) and Heroku, you built and deployed a web app with efficient and scalable secrets management using minimal effort. You saw how to store, share, and access secrets using Doppler and integrate your Project with a cloud provider.

You can also learn more about Doppler from the official documentation. If you have any questions, contact me on Twitter: @LordGhostX

Streaming Fauna Documents with Python

LordGhostX — Fri, 30 Jul 2021 18:44:53 +0000

With Fauna’s document streaming feature, you can get real-time updates about changes made to your database.

In this tutorial, you will learn how to use Fauna’s document streaming feature in a Python application.

Here for the code alone? Head over to the implementation section of this article.

Introduction

What is Streaming?

Streaming refers to when a program continuously reads and fetches data from a given source in real-time. For example, while watching a YouTube video or downloading a file from the internet, you stream the data from the origin and process it on your client’s device instantaneously.

Streaming, in some ways, can arguably be a better alternative to polling. Polling refers to when a client periodically makes queries to a data source to get updates. It is more expensive than streaming as the client sends many requests only to get a single update or none at all. Polling is only aware of changes when the query results return.

What is Fauna?

Fauna is a flexible database delivered as a secure data API providing two interfaces: GraphQL and the Fauna Query Language (FQL). It offers functionality to store collections, indexes, and other databases. To learn more about Fauna, visit the official documentation.

NOTE: The rest of this article assumes you have a fair understanding of Fauna and Python. Check out my introductory tutorial first if you don't.

Setting up our Fauna database

Creating the Fauna database

We need to first create the database to use in Fauna’s dashboard. If you have not created an account on Fauna before now, do so here.

In the dashboard, click on the NEW DATABASE button, provide a name for the database, then press SAVE. We will also be ticking the Pre-populate option to get demo data that will be used in this article:

Generate the database security key

Then, we need to create a security key to connect the database to the application. Go to the Security tab on the Fauna sidebar (located on the left side of the screen), click the NEW KEY button, provide the necessary information, and press SAVE:

Fauna will then present you with your Secret Key. Copy the key immediately and store it somewhere easily retrievable because it will only be displayed once.

Implementing document streaming with Python

Fauna features built-in support for document streaming. Whenever changes in a particular document are streamed, all the clients subscribed to it will be notified in real-time.

Initializing our Fauna client

Let’s write some code to implement this functionality!.

First, we will install the Fauna Python driver, importing the required libraries, and creating a database client.

In the terminal, type:

pip install faunadb

After that, create a Python script and save the code below in it:

from faunadb import query as q  
from faunadb.objects import Ref  
from faunadb.client import FaunaClient  

client = FaunaClient(secret="FAUNA_SECRET_KEY")

Selecting a document to stream

Next, we will fetch the document from the Fauna database collections that we want to stream:

doc = client.query(
    q.get(
        q.ref(q.collection("customers"), "101")
    )
)

Creating an OnStart function

We will start by defining an on_start function, which will be triggered when a document stream starts:

def on_start(event):
    print(f"started stream at {event.txn}")
    data = {
        "firstName": "John",
        "lastName": "Smith",
        "telephone": "719-872-4470"
    }
    client.query(q.update(doc["ref"], {"data": data}))

In our on_start function, we printed out the current timestamp of the transaction emitting the event (in microseconds) provided by event.txn. Then, the streamed document, so Fauna notifies our application in real-time.

Creating an OnVersion function

Next, we will define an on_version function. It will be triggered when there is a change in the streamed document and Fauna has sent a notification to the subscribing application:

def on_version(event):
    print(f"on_version event at {event.txn}")
    print(f"Event action: {event.event['action']}")
    print(f"Changes made: {event.event['diff'].get('data')}")

In our on_version function, we printed out the current timestamp again (in microseconds), then printed out the event action provided by event.event['action']. We then printed out the changes made to the document.

Creating an OnError function

After that, we will define an on_error function. It will be triggered when an error occurs while processing the data in the on_version:

def on_error(event):
    print(f"Received error event {event}")

In our on_error function, we printed out the event that triggered the issue along with its error message.

Starting the document streaming

We will now begin document streaming:

options = {"fields": ["document", "diff", "action"]}
stream = client.stream(doc["ref"], options, on_start, on_error, on_version)
stream.start()

In the code above, we defined the fields we want in every event update. Then, we passed the on_start, on_error, and on_version functions to the stream object and triggered it with the start method.

You should get an image like the one below when the entire script executes:

NOTE: You can stop active streams using the stream.close() method.

A few things to note

At the time of writing, Fauna’s document streaming is still an Early Access feature. Hence there are some limitations to using it, such as:

You cannot have more than 100 active stream connections to Fauna from a single client device.
Document streaming is currently not supported in Node.js clients due to an HTTP/2 implementation issue.
You can only stream Fauna documents. It is not possible to stream collections or any schema document.
Streams can only report events to the fields and values within a document’s data field.
There is currently no support for GraphQL subscriptions.

Conclusion

In this article, we learned about Fauna’s document streaming capabilities and how easy it is to integrate Fauna’s document streaming into a Python application.

You can learn more about Fauna document streaming from the official documentation. If you have any questions, don't hesitate to contact me on Twitter: @LordGhostX.

Building a Contact Manager with Python, Flask, and HarperDB

LordGhostX — Wed, 02 Jun 2021 19:45:13 +0000

What is HarperDB?

HarperDB is an easy database solution that helps collect, distribute, and analyze data throughout your organization. It provides unmatched flexibility, speed, security, and value for every data use case your organization and project require.

Benefits of Using HarperDB in Your Next Project

Seamless integration into any project and tool using a straightforward REST API. It also comes with support for multiple languages via various drivers and plugins.
HarperDB provides the ability to run your database anywhere, ranging from edge to cloud and anywhere in between.
Support for executing complex, performant, ACID-compliant SQL queries on JSON with no data duplication.
Functionality to search databases with JSON filters and SQL queries without the need of an ORM.
You enjoy the flexibility of a NoSQL database with the security, consistency, and durability of an RDBMS by leveraging a dynamic schema with attribute level and role-based security in a single storage engine.
HarperDB Studio offers a sleek and intuitive management UI that allows you to browse, query, and manage data all in one place, with no coding required.

Getting Started with HarperDB

Creating the HarperDB Database Instance

We need to create a database for the contact manager. If you have not created an account on HarperDB before now, create one on HarperDB’s website.

In the dashboard, click on Create New HarperDB Cloud Instance as shown in the image below.

Then, select a database instance type. A HarperDB instance is where you host your database. It can either be local (user-installed instance) or configured and managed on HarperDB servers (cloud instance). In this tutorial, we will be using the cloud instance.

You will need to provide information about your cloud instance, like name and authentication credentials. Click on the Instance Details button after supplying this information.

Note: Store the credentials somewhere secure and easily retrievable as we will need them in later parts of this tutorial.

You are also required to provide the specification of the cloud instance. HarperDB allows you to choose the RAM, storage size, and region your instance should have.

Once you have provided all the configuration details for the cloud instance, click on the Confirm Instance Details button, then finalize everything to create it.

NOTE: Cloud instances take several minutes to create. You’ll need to wait throughout this process. HarperDB also sends notifications via email on the instance creation status.

Creating the Database Schemas

Let’s set up the necessary database schema for our application. A schema is a collection of tables in HarperDB, analogous to a database in SQL. Without schemas, you cannot store data in the database.

To create a schema, first select the cloud instance in your dashboard, navigate to the schemas section by the left, and then provide a name for it. In this tutorial, we will be using the name contacts_repo as displayed in the image below:

Creating the Schema Tables

Let’s set up tables in our database schema. A table is analogous to tables in SQL databases like MySQL and collections in NoSQL databases like MongoDB.

To create a table. First, navigate to the tables section by the left side of your dashboard, then provide a name and hash_attribute for it. A hash_attribute identifies every record in the database uniquely, and it is synonymous with Primary Keys in a relational database.

Here, we will set the name to contacts and hash_attribute to contact_id as shown in the image below:

Setting Up the Demo Application

Cloning the Contact Manager Repository

For the sake of convenience, I have written the user interface of the contact manager (with Flask and Bootstrap) that we will use in this article. To get started, you will need to clone my repository and initialize the app like so:

git clone https://github.com/LordGhostX/harperdb-contact-manager
cd harperdb-contact-manager

The user-interface directory contains only the user interface of the contact manager. In contrast, the full-application directory contains the entire application (HarperDB + Python + Flask) that we will build in this tutorial.

Installing the Project’s Requirements

You need to install the contact manager’s requirements before you can run it. In your terminal, type:

cd user-interface  
pip install -r requirements.txt

Run the application to make sure it’s working. In your terminal, type:

To view the user interface of our application, head to your browser and open the URL http://127.0.0.1:5000, you should get a response similar to the image below:

Integrating HarperDB with Python and Flask

Now that we have set up the base application and database schemas, let’s move to communicating with the database using Python, as shown in the SDK documentation.

Installing HarperDB Python SDK

We need to get the Python SDK for HarperDB. It provides functionalities that allow an application to make queries to the HarperDB database instance. It’s available on pip, making it installable with a single command in the terminal.

In the terminal, type:

pip install harperdb

Configuring HarperDB Python SDK

We need to import the HarperDB SDK into our Flask application to use it. To do this, add the following code to the app.py file:

import harperdb

We also have to initialize the Python SDK client with the database instance authentication credentials we saved earlier. Add the following code to the app.py file right after initializing the Flask app.

app.config["SECRET_KEY"] = "SECRET_KEY"
db = harperdb.HarperDB(
    url="HARPERDB_URL",
    username="HARPERDB_USERNAME",
    password="HARPERDB_PASSWORD"
)

Note: Don’t forget to replace the credentials placeholders with the appropriate values for your application.

The HARPERDB_URL is present in the HarperDB dashboard, where all the database instances are displayed.

Testing the HarperDB Database Connection

To test our HarperDB cloud instance and credentials, we will be querying our database for its description. To do this, add the block of code below right before running the application in the app.py file.

print(db.describe_all())

Run the app.py file. If you get a response like an image below, the configuration is correct. If you don’t, check if your credentials are valid and correct.

Making Queries to HarperDB

Creating New Documents in HarperDB

To create new documents in our HarperDB schema, we need to run an INSERT query in the database via the Python SDK. HarperDB provides two methods to do this:

The SQL way: This involves running a SQL query to insert data into the database.
The NoSQL way: This involves using the insert method of the Python SDK to insert data into the database.

Here is sample Python code to demonstrate both methods:

# SQL way
db.sql("INSERT INTO contacts_repo.contacts (name, email) VALUES ('LordGhostX', 'lordghostx@email.com')")

# NoSQL way
result = db.insert("contacts_repo", "contacts", [{"name": "LordGhostX", "email": "lordghostx@email.com"}])

NOTE: If you omit the hash_attribute metadata when executing an INSERT query, HarperDB will automatically generate one for you and store it.

The code above executes an INSERT query to your HarperDB cloud database. It creates a document in the contacts table, in the contacts_repo schema with the name and email metadata.

We can integrate this into our contact manager by updating the contacts route with the code below:

@app.route("/contacts/", methods=["GET", "POST"])
def contacts():
    if request.method == "POST":
        action = request.form.get("action")
        contact_data = {
            "name": request.form.get("name", "").strip(),
            "job": request.form.get("job", "").strip(),
            "email": request.form.get("email", "").strip(),
            "phone": request.form.get("phone", "").strip()
        }

        if action == "addContact":
            db.insert("contacts_repo", "contacts", [contact_data])
            flash("Successfully added new contact", "success")

        return redirect(url_for("contacts"))

    return render_template("contacts.html")

Reading Documents from HarperDB

To read documents from our HarperDB tables, we need to run a SELECT or SEARCH query in the database via the Python SDK. HarperDB provides two methods to do this:

The SQL way: This involves running a SQL query to select the data from the database.
The NoSQL way: This involves using either the search_by_hash or search_by_value methods of the Python SDK to select data from the database.

Here is sample Python code to demonstrate both methods:

# SQL way
db.sql("SELECT * FROM contacts_repo.contacts")

# NoSQL way
db.search_by_hash("contacts_repo", "contacts", ["hash_attribute"]) # returns all documents with the specified hash attributes

db.search_by_value("contacts_repo", "contacts", "name", "LordGhostX") # returns all documents with the specified values

The code above executes a SELECT and SEARCH query to your HarperDB cloud database. It fetches all the documents in the contacts table, in the contacts_repo schema with the specified hash attributes or values.

We can integrate this into our contact manager by updating the rendering of the contacts route in the app.py file with the code below:

contacts_data = db.sql(
    "SELECT * FROM contacts_repo.contacts ORDER BY name")
return render_template("contacts.html", contacts_data=contacts_data)

We also need to update the contacts page to render the data fetched from the HarperDB request. Update the contacts table body <tbody> in the contacts.html file with the code below:

<tbody>
  {% for contact in contacts_data %}
  <tr>
    <td id="contactName-{{ contact.contact_id }}">{{ contact.name }}</td>
    <td id="contactEmail-{{ contact.contact_id }}">{{ contact.email }}</td>
    <td id="contactPhone-{{ contact.contact_id }}">{{ contact.phone }}</td>
    <td id="contactJob-{{ contact.contact_id }}">{{ contact.job }}</td>
    <td>
      <button class="btn btn-primary" data-toggle="modal" data-target="#contactModal" onclick="displayContactInfo('{{ contact.contact_id }}')">View Details</button>
    </td>
  </tr>
  {% endfor %}
</tbody>

Updating Documents in HarperDB

To update documents in our HarperDB tables, we need to run an UPDATE query in the database via the Python SDK. HarperDB provides two methods to do this:

The SQL way: This involves running a SQL query to update the data in the database.
The NoSQL way: This involves using the update method of the Python SDK to update data in the database.

Here is sample Python code to demonstrate both methods:

# SQL way
db.sql("UPDATE contacts_repo.contacts SET email='lordghostx@newemail.com' WHERE contact_id='sample_hash_attribute'")

# NoSQL way
db.update("contacts_repo", "contacts", [{"contact_id": "sample_hash_attribute", "email": "lordghostx@newemail.com"}])

The code above executes an UPDATE query to your HarperDB cloud database. It updates the document in the contacts table, in the contacts_repo schema with the specified hash attributes with the new values supplied.

We can integrate this into our contact manager by updating the action handling in the contacts route with the code below:

if action == "updateContact":
    contact_data["contact_id"] = request.form.get("contactID")
    db.update("contacts_repo", "contacts", [contact_data])
    flash("Successfully updated contact information", "success")

Deleting Documents from HarperDB

To delete documents from our HarperDB tables, we need to run a DELETE query in the database via the Python SDK. HarperDB provides two methods to do this:

The SQL way: This involves running a SQL query to delete the data from the database.
The NoSQL way: This involves using the delete method of the Python SDK to delete data from the database.

Here is sample Python code to demonstrate both methods:

# SQL way
db.sql("DELETE FROM contacts_repo.contacts WHERE name='LordGhostX'")

# NoSQL way
db.delete("contacts_repo", "contacts", ["sample_hash_attribute"])

The code above executes a DELETE query to your HarperDB cloud database. It deletes the document in the contacts table, in the contacts_repo schema with the specified hash_attribute, or has the value LordGhostX in the name column.

We can integrate this into our contact manager by updating the logout route with the code below:

@app.route("/contacts/delete/<string:contact_id>")
def delete_contact(contact_id):
    db.delete("contacts_repo", "contacts", [contact_id])
    flash("Successfully deleted contact information", "success")
    return redirect(url_for("contacts"))

Conclusion

By integrating HarperDB with Python and Flask, we built a contact manager with CRUD functionalities with minimal effort. We saw how to create cloud databases, manage them, and execute queries using HarperDB.

The source code of the contact manager is available on GitHub. It was exciting integrating HarperDB into a Flask application, and I can’t wait to see the amazing things you create with it!

If you have any questions, don't hesitate to contact me on Twitter: @LordGhostX