DEV Community

Cover image for Exploring the methods of looking into Ethereum’s transaction pool
Eugene Aseev
Eugene Aseev

Posted on • Edited on • Originally published at chainstack.com

Exploring the methods of looking into Ethereum’s transaction pool

#blockchain #ethereum #txpool #mempool

Introduction

The mempool of the Ethereum mainnet—called transaction pool or txpool—is the dynamic in-memory area where pending transactions reside before they are included in a block and thus become static.

The notion of a global txpool is a bit abstract as there is no single defined pool for all pending transactions. Instead, each node on the Ethereum mainnet has its own pool of transactions and, combined, they all form the global pool.

The thousands of pending transactions that enter the global pool by being broadcast on the network and before being included in a block are an always changing data set that's holding millions of dollars at any given second.

There's a lot that one can do here—and many do and make this txpool business a highly competitive market.

A few examples, listed in the order of inconspicuous to contentious:

  • Yield farming — you can watch the transactions movement between DeFi applications to be one of the first to detect the yield farming profitability changes.
  • Liquidity providing — you detect the potential liquidity movements in and out of DeFi applications and act on the changes.
  • Arbitrage — you can detect the movement of transactions that will affect token prices at different DEXes and make your arbitrage trades based on that.
  • Front running — you can automatically grab the existing pending transactions, simulate them to identify the potential profit if the transaction is executed, copy the transaction and swap the existing address with your own, and submit it with a higher miner fee so that your transaction gets executed on-chain before the one you are copying.
  • Doing a MEV — MEV stands for Miner Extractable Value, and it's based on that miners are theoretically free to include any transactions in the blocks and/or reorder them. This is a variation of the front running where instead of submitting your transaction with a higher fee to the same pool you picked it from, you get it directly into a block through a miner and bypass the pool.

To run any of the described scenarios, you need access to the Ethereum txpool, and you need the methods to retrieve the transactions from the txpool. While Chainstack has you covered with fast dedicated nodes for the former, this article focuses on all the ways you can look into the txpool.

Retrieving pending transactions with Geth

Since pending transactions are your targets in the txpool space, we are now going to make this a structured effort and focus on answering the following questions while accompanying the answers with practical examples:

  • How do I retrieve pending transactions?
  • Why do I view global or local pending transactions?
  • Can I view global pending transactions without txpool namespace?

There are a few ways to retrieve pending transactions.

  • Filters
  • Subscriptions
  • Txpool API
  • GraphQL API

Before we start, lets clarify a few things:

  • Global pending transactions refer to pending transactions that are happening globally, which includes your newly created local pending transactions.
  • Local pending transactions refer strictly to the transactions that you created on your local node. Note that you need 'personal' namespace enabled for Geth to send local transactions.
  • Pending transactions refer to transactions that are pending due to various reasons, like extremely low gas, out of order nonce, etc.
  • Chainstack is using Geth (Go Ethereum) client.

Filters

When we create a filter on Geth, Geth will return a unique filter_id. Do note that this filter_id will only live for 5 minutes from the last query on that specific filter. If you query this filter after 5 minutes, the filter will not exist anymore.

Creating a pending transaction filter and retrieving data from it

cURL

Create filter

Request body:

'{"jsonrpc":"2.0","method":"eth_newPendingTransactionFilter","params":[],"id":73}'
Enter fullscreen mode Exit fullscreen mode

Response body:

{ "id":73, "jsonrpc": "2.0", "result": "0xb337f6e2f833ecffc6e9315ba06cd03d" }
Enter fullscreen mode Exit fullscreen mode
Web3.js

Filter for pending transactions is not supported anymore. Please use Subscriptions.

Web3.py

Create filter

pending_transaction_filter= web3.eth.filter('pending')
Enter fullscreen mode Exit fullscreen mode

Access filter

print(pending_transaction_filter.get_new_entries())
Enter fullscreen mode Exit fullscreen mode

Common sources of confusion on filters

Web3.py and the pending argument

Why does web3.py have their input arguments as pending instead of a dictionary which contains the usual filter parameters like fromBlock, toBlock, address, topics.

This is because Web3 does the mapping internally. If we look at the web3.py source code, when web3.py receives a string pending, it will be mapped to eth_newPendingTransactionFilter, and when web3.py receives a dictionary, it is mapped to eth_newFilter.

To add to this, web3.py has get_new_entries as well as get_all_entries for filters but get_all_entries does not work in our case. This is because eth_newPendingTransactionFilter does not have get_all_entries available for it.

From latest block to pending block filter

Why doesn't the following filter give me real-time pending transactions?

web3.eth.filter({'fromBlock': 'latest', 'toBlock': 'pending'})
Enter fullscreen mode Exit fullscreen mode

A filter only returns new_entries() when the state has changed. This specific filter state changes only when there is a new latest block or pending block. Thus, you will only receive changes when there is a new latest block or pending block, i.e. (eth.getBlock('latest') / pending).

The getPendingTransactions filter

Why is the following giving me a different or empty result?

web3.eth.getPendingTransactions().then(console.log)
Enter fullscreen mode Exit fullscreen mode

This function is mapped to eth.pendingTransactions which is a function to check local pending transactions and does not give you global transactions.

Based on the Geth source code, only pendingTransactions that has its from field matching with your personal account will be shown.

Subscriptions

Subscriptions is real-time streaming of data from server to client through WebSocket. You will need a constantly active connection to stream such events. You cannot use curl for this and have to use a WebSocket client like websocat if you want to access it via command line. Once executed, a stream of pending transaction IDs will start flowing in.

For other supported subscriptions, check the Geth documentation: Supported Subscriptions.

Creating a subscription

Websocat

Connect to node

websocat wss://username:password@ws-nd-123-456-789.p2pify.com
Enter fullscreen mode Exit fullscreen mode

Create subscription

Request body:

{"id": 1, "method": "eth_subscribe", "params": ["newPendingTransactions"]}
Enter fullscreen mode Exit fullscreen mode

Response body:

{"jsonrpc":"2.0","id":1,"result":"0x2d4f3eb938cdb0b6fa9052de7c4da5de"}
Enter fullscreen mode Exit fullscreen mode

Incoming stream

{"jsonrpc":"2.0","method":"eth_subscription","params":{"subscription":"0x2d4f3eb938cdb0b6fa9052de7c4da5de","result":"0xee426dbaef2a432d0433d5de600347e97b6a8a855daf8765c18cf1b7efc53461"}}
...
Enter fullscreen mode Exit fullscreen mode

Txpool API

Txpool is a Geth-specific API namespace that keeps pending and queued transactions in the local memory pool. Geth's default is 4096 pending and 1024 queued transactions. However, Etherescan reports a much bigger number of pending transactions. If we view Geth's txpool, we will not be able to get all of the transactions. Once the pool of 4096 is full, Geth replaces older pending values with newer pending transactions.

If you need a bigger pool to be stored on your node, the flag --txpool.globalslots can be adjusted to a higher value on Geth CLI options. Do note that the larger the number, the bigger the payload size.

If you see your txpool_status to be empty, it can mean your node has not fully synced.

The txpool namespace is only supported on Chainstack dedicated nodes.

Creating a txpool filter

cURL

Create filter

Request body:

{"jsonrpc":"2.0","method":"eth_newPendingTransactionFilter","params":[],"id":73}
Enter fullscreen mode Exit fullscreen mode

Response body:

{ "jsonrpc": "2.0", "id": 73, "result": { "pending": {...}, "queued": {...} } }
Enter fullscreen mode Exit fullscreen mode
Web3.js

TxPool Module.

Web3.py

Geth API.

GraphQL API

The biggest advantage of using GraphQL is that you can filter out the specific fields of the transaction that you want. The query in GraphQL goes through the elements within the txpool. Thus, its limitations are the same as the ones of txpool as stated above.

The following is an example which shows the information of a pending transaction.

Query example:

query {
  pending {
    transactions {
      hash
      gas
      value
      gasPrice
      nonce
      r
      s
      v
      inputData
      status
      gasUsed
      cumulativeGasUsed
      from {
        address
      }
      to {
        address
      }
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

GraphQL API is currently supported on Chainstack dedicated Ethereum nodes.

I hope this blog serves you well in understanding the various ways of retrieving pending transactions.

Additional resources

Join our community of innovators

Have you already explored what you can achieve with Chainstack? Get started for free today.

Top comments (0)

Read next

imsushant12 profile image

Interview Questions on AWS EC2 and Compute Services

Sushant Gaurav -

readymade-ui profile image

Building Responsive Layouts with Tailwind CSS

ReadymadeUI -

bharu_0fec38bd0fb51b76742 profile image

Getting Started with Java: A Beginner's Guide

Bharu -

gamedevdigest profile image

Game Dev Digest — Issue #262 - VFX, XR, Replay Systems, and More

Game Dev Digest - The Newsletter On Unity Game Dev -