How I built a Bitcoin indexer using ZeroMQ

#blockchain #backend

Indexers are node operators that fetch data from a node then transform it and store on a database to provide easy access to the blockchain data. You can use it to build a history of user's transactions, or get a transaction/block to build a decentralized application.

ZeroMQ is an open source messaging library that you can connect and use some socket patterns. We will use a pub-sub pattern to subscribe on the node and listen new blocks and transactions on real time to store on our database and have a fast access than accessing directly the blockchain, and have a easy way to scale it.

Bitcoind is a node interacting who will allow us to interact with the blockchain and get the desired data.

Subscribing to Bitcoind using ZeroMQ

To Listen to new blocks and transactions you need to run a node locally on your own infrastructure, you can download bitcoind here

Using ZeroMQ you can set a address to publish your events (blocks and transactions) then on each new event your node emit and the application can deal with it.

To run a node with ZeroMQ using bitcoind:

bitcoind -signet -zmqpubrawblock=tcp://127.0.0.1:3000 -zmqpubrawtx=tcp://127.0.0.1:3000

After run the node, you can install zeromq package and listen rawblock and rawtx event, then insert on your database.

import zmq from "zeromq";

const SOCKET_ADDRESS = "tcp://127.0.0.1:3000"

const connectZeroMQ = () => {
    const socket = zmq.socket('sub');
    socket.connect(SOCKET_ADDRESS);
    socket.subscribe('');
    return socket;
}

const zmq = connectZeroMQ();

zmq.on('rawblock', (error, block) => {
    if(err !== null) return;
    // parse and store on your database
})

zmq.on('rawtxn', (error, txn) => {
    if(err !== null) return;
    // parse and store on your database
})

Conclusion

You can achieve this result on different ways, but ZeroMQ gives to you more security and reliability to receive the blocks and transactions on real-time, this code can be used in production, but I highly recommend that you create a script to check every block and transaction from time to time to make sure none are lost.

In my repository below you can find all the code above connected to the database, and getting the lost blocks, or old blocks, feel free to contribute =)
https://github.com/EmanuelCampos/bitcoin-indexer

If you have some suggestions/feedbacks you can call me on:
Twitter