DEV Community: Massimo Luraschi

Speed up NFT indexing with Subsquid

Massimo Luraschi — Fri, 31 Mar 2023 08:07:26 +0000

Summary

This series of articles is aimed at providing examples and explanation for some of the aspects of features of Subsquid.

This instalment wants to showcase two things, only one of which is actually inherently part of Subsquid SDK. The second one is still something that the SDK makes possible and easier to achieve. Furthermore, it's a possibility that many developers might not know, so it's worth mentioning.

Although the title of this article specifically talks about NFTs, the subjects here treated can actually be translated to any indexing project where it's necessary to repeatedly invoke a smart contract function to access its state.

The topics covered by this article are Server Extensions (through custom resolvers) and Multicall contract.

The project discussed in this article is hosted on a GitHub repository:

https://github.com/RaekwonIII/bored-ape-yacht-club-indexing

The repository is also configured, so you can run the project on Gitpod:

Introduction

As the series of articles on use cases of Subsquid framework grows, I want to add more and more examples and mention more and more features, including some that the average Subsquid developer might not be aware of.

This time it's the turn of Server Extensions. Subsquid documentation has an extensive documentation page detailing this functionality, so this page is mostly aimed at providing a practical example.

On top of this, because it's better to have more examples than less, I am taking this project as a chance to show how to use external API calls to grab the token metadata from the tokenURI, similarly to how it was done on the ENS article.

The Multicall contract is also a very interesting talking point which should become a staples for every developer that needs to repeatedly perform calls to smart contract functions to access the contract's status. Aggregating these transactions into a single call will contribute to efficiency of your indexer and maintain a good overall performance.

Project setup

If you haven’t already, you need to install Subsquid CLI first:

npm i -g @subsquid/cli@latest

As with most squid ETLs, it all starts with creating a project from a template. In a terminal, launch the command:

sqd init bored-ape-yacht-club -t evm

Here, bored-ape-yacht-club is the name we are going to give to our project, you can change it to anything you like. And -t evm specifies what template should be used and makes sure the setup will create a project from the EVM-indexing template.

Let’s also install the dependencies, by launching this command:

npm i

Contract ABI

The next step was importing the ABI into the project and generate bindings for the ABI. I have found the BAYC token contract on Ethereum block explorer and noted the address.

I then generated TypeScript boilerplate code to interact with it, thanks to Subsquid's CLI command:

sqd typegen 0xbc4ca0eda7647a8ab7c2061c2e118a18a936f13d

If you take a look at the commands.json file, specifically at the typegen shortcut defined in it, you'll be able to see it's adding the --multicall option.

This means the command will also generate a file named multicall.ts containing specific bindings for the Multicall contract I have discussed in the introduction. This will be useful in the indexing logic.

Schema

For this project, I didn't steer much away from other similar NFT indexing projects I have shown in previous articles. As usual, I am interest in Transfers, in theTokens, as well as the Owner of each token, and I'm also capturing Contract information.

I did, however, add the imageUrl field to the Token model, because this is part of what I wanted to achieve by using the Multicall contract. Here's the schema:

type Token @entity {
  id: ID!
  owner: Owner
  uri: String
  imageUrl: String
  transfers: [Transfer!]! @derivedFrom(field: "token")
  contract: Contract
}

type Owner @entity {
  id: ID!
  ownedTokens: [Token!] @derivedFrom(field: "owner")
}

type Contract @entity {
  id: ID!
  name: String! @index
  symbol: String!
  totalSupply: BigInt!
  tokens: [Token!]! @derivedFrom(field: "contract")
}

type Transfer @entity {
  id: ID!
  token: Token!
  from: Owner
  to: Owner
  timestamp: DateTime! @index
  block: Int! @index
  transactionHash: String!
}

Let's save this content in the schema.graphql file and update the TypeScript models, by running the command:

sqd codegen

Data indexing

For the data processing itself, I didn't want to create anything too complex, because it's not the focus of the project. Also, I am not going to provide the full code listing here, I'd recommend instead to take a look at the repository on GitHub.

Instead, I'm going to provide a list of functionalities that the logic accomplishes, and some code snippets to showcase how they are implemented:

configure the indexer to extract data generated by the BAYC smart contract and filter for the event I am interested in: Transfer

const contractAddress = "0xBC4CA0EdA7647A8aB7C2061c2E118A18a936f13D".toLowerCase();

const multicallAddress = "0x5ba1e12693dc8f9c48aad8770482f4739beed696".toLowerCase();

const processor = new EvmBatchProcessor()
.setDataSource({
  chain: process.env.RPC_ENDPOINT,
  archive: "https://eth.archive.subsquid.io",
})
.addLog(contractAddress, {
  filter: [
    [
      events.Transfer.topic,
    ],
  ],
  data: {
    evmLog: {
      topics: true,
      data: true,
    },
    transaction: {
      hash: true,
    },
  },
});

Process the batch of events that Subsquid Archive provides to my processor at recurring intervals and verify the event kind, and use a separate function to extract, save data in a temporary data interface array, then use another function to process this data array and save information on the database

processor.run(new TypeormDatabase(), async (ctx) => {
  const baycDataArr: BAYCData[] = [];

  for (let c of ctx.blocks) {
    for (let i of c.items) {
      if (i.address === contractAddress && i.kind === "evmLog") {
        if (i.evmLog.topics[0] === events.Transfer.topic) {
          const baycData = handleTransfer({
            ...ctx,
            block: c.header,
            ...i,
          });
          baycDataArr.push(baycData);
        }
      }
    }
  }

  await saveBAYCData(
    {
      ...ctx,
      block: ctx.blocks[ctx.blocks.length - 1].header,
    },
    baycDataArr
  );
});

type BAYCData = {
  id: string;
  from: string;
  to: string;
  tokenId: bigint;
  timestamp: Date;
  block: number;
  transactionHash: string;
};

Here is the handleTransfer function, responsible for decoding the Transfer event and save data in a BAYCData interface.

function handleTransfer(
  ctx: LogHandlerContext<
    Store,
    { evmLog: { topics: true; data: true }; transaction: { hash: true } }
  >
): BAYCData {
  const { evmLog, block, transaction } = ctx;

  const { from, to, tokenId } = events.Transfer.decode(evmLog);

  const baycData: BAYCData = {
    id: `${transaction.hash}-${evmLog.address}-${tokenId.toBigInt()}-${
      evmLog.index
    }`,
    from,
    to,
    tokenId: tokenId.toBigInt(),
    timestamp: new Date(block.timestamp),
    block: block.height,
    transactionHash: transaction.hash,
  };
  return baycData;
}

Token URI via Multicall contract

At this point, the transferred NFTs have been identified and the transfer information is stored in the baycDataArr array, but what is making sure this information is permanent, is the saveBAYCData function.

Its logic is acutally very similar to the one in the ENS tokens article, so I am not going to repeat it. Furthermore, it's possible to use the code stored in the GitHub repository as a reference (which I advise looking at).

I am going to instead focus on the parts that make this project worth talking about. Right at the very end of the function, after the loop over the baycDataArr array has ended, and before saving on the database, this code section below is where the Multicall contract is used to fetch the tokenURI, by aggregating multiple calls into a single one.

const maxHeight = maxBy(baycDataArr, data => data.block)!.block;
const multicall = new Multicall(ctx, {height: maxHeight}, multicallAddress);

ctx.log.info(`Calling multicall for ${baycDataArr.length} tokens...`);

const results = await multicall.tryAggregate(functions.tokenURI, baycDataArr.map(data => [contractAddress, [BigNumber.from(data.tokenId)]] as [string, BigNumber[]]), 100);
results.forEach((res, i) => {
  let t = tokens.get(baycDataArr[i].tokenId.toString());
  if (t) {
    let uri = '';
    if (res.success) {
      uri = <string>res.value;
    } else if (res.returnData) {
      uri = <string>functions.tokenURI.tryDecodeResult(res.returnData) || '';
    }
    t.uri = uri;
    if (!tokenIdToImageUrl.has(t.id)) tokensWithNoImage.push(t.id);
  }
})

ctx.log.info(`Done`);

It's worth noting that tokenIdToImageUrl is a Map defined at the very top of the file, which act as a "cache" of tokens we have already encountered and whose image we have already fetched.

Then, right after processing all results from the multicall contract, we can fetch the image url we need with this piece of code (note: api is an instance of the Axios API library, which is also instantiated at the top of the file):

await Promise.all(tokensWithNoImage.map( async (id) => {
  const t = tokens.get(id);
  if (t && t.uri) {
    try {
      const res = await api.get(t.uri);
      tokenIdToImageUrl.set(id, res.data.image);
      t.imageUrl = res.data.image;
    }
    catch (error) {
      console.log(error);
    }
  }
}))

Custom Resolver

The data processing implemented so far made sure I saved all the Transfers, but what if I wanted to do some aggregations? Let's imagine I wanted to know the daily total transfer, how would I access this kind of data?

There is a common pattern offered by GraphQL servers, which might be less known by Subsquid developers. This is, to write your own custom resolver as a server extension, and specify custom queries.

I set on to create an index.ts file under the src/server-extension/resolvers and I built my own daily aggregations using simple SQL queries. Here is how to get the daily Transfers of BAYC NFTs:

// src/server-extension/resolvers/index.ts
import 'reflect-metadata'
import type { EntityManager } from 'typeorm'
import { Field, ObjectType, Query, Resolver } from 'type-graphql'
import { Transfer } from '../../model'

@ObjectType()
export class TransfersDayData {
    @Field(() => Date, { nullable: false })
    day!: Date

    @Field(() => Number, {nullable: false})
    count!: number

    constructor(props: Partial<TransfersDayData>) {
        Object.assign(this, props)
    }
}

@Resolver()
export class TransfersDayDataResolver {
    constructor(private tx: () => Promise<EntityManager>) {}

    @Query(()=>[TransfersDayData])
    async getTransfersDayData(): Promise<TransfersDayData[]> {
        const manager = await this.tx()
        const repository = manager.getRepository(Transfer)

        const data: {
            day: string
            count: number
        }[] = await repository.query(`
            SELECT DATE(timestamp) AS day, COUNT(*) as count
            FROM transfer
            GROUP BY day
            ORDER BY day DESC
        `)
        return data.map(
            (i) => new TransfersDayData({
                day: new Date(i.day),
                count: i.count
            })
        )
    }
}

Start indexing

If you have followed along and want to try this at home, at this point you should launch the database Docker container and generate a new migration file (this will also take care of removing any existing migration files, relating to the template project):

sqd up
sqd migration:generate

Launch the processor:

sqd process

And finally, in a separate terminal window, launch the GraphQL service:

sqd serve

Let’s open the browser at the address: localhost:4350/graphql and test our work so far.

Let’s find all the tokens that have a value in the uri field, and get some information about them, including the uri itself:

query MyQuery {
    tokens(limit: 10, where: {uri_isNull: false}) {
        id
        uri
        owner {
            id
        {
    }
}

But ultimately, thanks to the code section visiting this very same URL, we were able to save the image url, which we can query:

query MyQuery {
    tokens(limit: 10, where {imageUrl_isNull: false}) {
        id
        imageUrl
        uri
    }
}

Let’s copy the image from the field and paste the URL in another tab, and that’s our bored ape.

We can also use the server extension we built, and select the day and the count, and when we execute it, we will get a list of days and the related count of transfers for each day. And the information here was retrieved via a custom SQL query.

Gitpod

Another option, as mentioned at the start of the article, is to run the project on Gitpod by clicking on the related link on the project’s README file:

Which will launch the entire project in a cloud-hosted execution environment for you.

Conclusion

In conclusion, with this article I wanted to prove to be able to

fully index the Bored Ape Yacht Club NFT collection
fetch their metadata, including their token URL in a performant way, thanks to Multicall contract
use API calls to these endpoints to locally save the URLs of the images
create our own personalized query by extending the GraphQL server and implementing a custom resolver

So, in this video, we learned that Subsquid SDK gives you the freedom to use external libraries and perform API calls to augment your on-chain data, that Multicall contract helps you keep the performance of your indexer in check, and that custom resolvers can allow developers to introduce custom manipulations and aggregations on top of indexed data.

If you found this article interesting, and you want to read more, please follow me and most importantly, Subsquid.

Subsquid socials:

Boost your dApp development productivity with local indexing

Massimo Luraschi — Mon, 27 Mar 2023 09:19:23 +0000

Summary

Blockchain data indexing is a performance enhancer for dApps, but it’s a pain to develop your application locally, because, somehow, no one thought about indexing a local EVM node… 😡😤

…Until Subsquid decided to put an end to this.

By working with an indexing middleware from the very early stages of a project, developers can build their dApp locally, deploying it on a local node, and index that local node with a squid.

This generates a much smoother developer experience, safer applications that can be thoroughly tested prior to the smart contract being deployed, and a much tighter development cycle, which leads to faster and more efficient deliveries.

The project discussed in this article is available at this repository:

github.com/RaekwonIII/local-evm-indexing/

For the development of the squid indexer, Ganache is going to be taken as a reference, because of these factors:

the provided MetaContract is simple, yet "interesting" enough from an indexing point of view, to create an example
the truffle console allows to easily create transactions from the command line and emulate smart contract actions

But the same actions can be done with Hardhat and Anvil (and very likely, there will be future articles about them).

Introduction

I have been dipping my toes in smart contract development for a little while now, and recently I have published an article detailing a full walkthrough on how to improve the performance of a dApp (an NFT marketplace, in this case), using blockchain indexing.

While developing this project, I really put myself in the shoes of developers that are using Subsquid’s SDK on a daily basis, and I started to realize one of the most annoying facts that might hinder indexer’s adoption, dApp development, and ultimately, the user experience.

The blockchain industry has recognized pretty quickly, that smart contract development needs local development environment. And this has brought to the creation of tools like Truffle, then Hardhat, and (as I recently discovered) Anvil.

These tools allow developers to launch a minimal EVM node, to which developers can deploy, run, test the smart contract(s) they are developing. But a decentralized application is not just the smart contract itself: what about the frontend?

Developing dApp frontend locally

I wanted to show that connecting the frontend of a dApp to an indexing middleware leads to better performance, compared to direct calls to the blockchain. But while developing this project, I needed to index my local Ethereum node, running on Hardhat.

This was a sudden realization for me, and a breakthrough-moment.

dApp frontend local development is being treated as an after-thought

Without the ability to index their local node, a typical developer journey looks pretty much like this:

build the application with connections to smart contract to work locally
deploy on a testnet
change project to use indexing middleware
finish testing in this configuration
deploy to mainnet/production

But this is not the best developer experience, nor the safest and most efficient way to work.

Luckily, Subsquid put an end to this, by publishing Docker images for their Ethereum Archive (which you can find on DockerHub). By launching this container, an Archive service will be running on your machine, ingesting the local node’s blocks.

This allows developers to build a squid indexer, based on the local node data, exactly the same way they would do if the contract was deployed on testnet.

Let’s build it!

As mentioned at the start of the article, I chose to use Truffle suite and Ganache for an and-to-end project.

As previously explained, this choice was made out of convenience, but the same result could be achieved on Hardhat, using ethers library, as it's shown in their documentation (as a matter of fact, I might be writing a future article about this, stay tuned for that 😉).

The following section will cover the setup, a sample smart contract from Ganache, its deployment to the local node, and the generation of some transactions.

For this tutorial, I also chose to start with a squid project, and we are going to use it to manage Ganache configuration. This is not strictly necessary, it is only useful to work on one single folder and keep this tutorial as simple as possible.

Squid project setup

If you haven’t already, you need to install Subsquid CLI first:

npm i -g @subsquid/cli@latest

In order to create a new squid project, simply open a terminal and launch the command:

sqd init local-evm-indexing -t evm

Where local-evm-indexing is the name we decided to give to our project, you can change it to anything you like. And -t evm specifies what template should be used and makes sure the setup will create a project from the EVM-indexing template.

The next two sections are instructions dedicated to the two main options for local Ethereum development: Hardhat and Ganache. For that, and other similar tools I might not be aware of, the configuration should still be very similar.

Ganache setup

In order to start working with Ganache node, it’s necessary to Install Truffle and Ganache packages.

1. Truffle project, sample contract

Let’s create a new truffle project, for example by downloading a sample contract. In a console window, from the project’s main folder, launch this command:

truffle unbox metacoin

To compile the contracts, launch this command in the same console window:

truffle compile

You should find the contract’s ABI at the location: build/contracts/MetaCoin.json, it will be useful for indexing.

2. Create a workspace

First of all, make sure the development configuration, in the networks section of truffle-config.js is uncommented, and has these values.

development: {
  host: "127.0.0.1",     // Localhost (default: none)
  port: 8545,            // Standard Ethereum port (default: none)
  network_id: "1337",       // Any network (default: none)
},

Note: It is not mandatory to change port or network_id, but I initially set them to the same value that Hardhat uses, out of convenience. It's important that they are the same as the Ganache server, though.

If you decide not to change them, please be aware of the values, and correctly apply these in the Archive image parameters.

Launch the Ganache tool and select the New Workspace (Ethereum option).

In the following window, provide a name for the workspace, and link the Truffle project we just created, by clicking on Add project button and selecting the truffle-config.js file in the project's root folder. Finally, select the Server tab at the top.

In this window, change the server configuration to the exact values reported in this image.

If you encounter issues with these settings, try and change the HOSTNAME value to 0.0.0.0 — all interfaces.

Note: It is not mandatory to change PORT NUMBER or NETWORK ID, but they are set to the same value as Hardhat, so the rest of the Tutorial will look the same.

The AUTOMINE option is disabled, same as in the Hardhat section of this tutorial, for debugging purposes. For more information, consult the official documentation.

3. Deploy the contract

With the Truffle suite, you can simply deploy any contract in the contracts directory by running

truffle migrate

You should, a this point, see a log similar to this:

Compiling your contracts...
===========================
> Everything is up to date, there is nothing to compile.

Starting migrations...
======================
> Network name:    'development'
> Network id:      1337
> Block gas limit: 6721975 (0x6691b7)
1_deploy_contracts.js
=====================
   Replacing 'ConvertLib'
   ----------------------
   > transaction hash:    0x268debaaefe5ed181d0f7eb41f8c0a12b378a0c076910507b654fa6818889c49
   > Blocks: 0            Seconds: 0
   > contract address:    0x65cb02943D325A5aaC103eAEea6dB52B561Ee347
   > block number:        13
   > block timestamp:     1675251575
   > account:             0x62B9eB74708c9B728F41a2D6d946E4C736f1e038
   > balance:             99.99684864
   > gas used:            157568 (0x26780)
   > gas price:           20 gwei
   > value sent:          0 ETH
   > total cost:          0.00315136 ETH
   Linking
   -------
   * Contract: MetaCoin <--> Library: ConvertLib (at address: 0x65cb02943D325A5aaC103eAEea6dB52B561Ee347)
   Replacing 'MetaCoin'
   --------------------
   > transaction hash:    0x35a35781d5b4947285caf8d9f901d9be7a675821b9c4ec934bc2be2d7e9e85cb
   > Blocks: 0            Seconds: 4
   > contract address:    0xc0Bb32fc688C4cac4417B8E732F26836C594C258
   > block number:        14
   > block timestamp:     1675251580
   > account:             0x62B9eB74708c9B728F41a2D6d946E4C736f1e038
   > balance:             99.98855876
   > gas used:            414494 (0x6531e)
   > gas price:           20 gwei
   > value sent:          0 ETH
   > total cost:          0.00828988 ETH
   > Saving artifacts
   -------------------------------------
   > Total cost:          0.01144124 ETH
Summary
=======
> Total deployments:   2
> Final cost:          0.01144124 ETH

Make sure to take note of the contract address, we'll be using this later, while indexing. In the case reported here, it's 0xc0Bb32fc688C4cac4417B8E732F26836C594C258.

4. Generate test transactions

Then, I ended up following the steps described in this Truffle documentation page to generate a few transactions on the MetaCoin contract I had previously deployed.

I launched the console first:

truffle console

Your prompt should have changed to this:

truffle(development)>

Then obtained an instance of the contract, and used it to trigger the sendCoin function, and send coin to 4 other accounts.

let instance = await MetaCoin.deployed()
let accounts = await web3.eth.getAccounts()
instance.sendCoin(accounts[1], 10, {from: accounts[0]})
instance.sendCoin(accounts[2], 10, {from: accounts[0]})
instance.sendCoin(accounts[3], 10, {from: accounts[0]})
instance.sendCoin(accounts[4], 10, {from: accounts[0]})

I can also verify that the first account now has 40 “coins” less than the initial 10000 allotted by Ganache:

truffle(development)> let balance = await instance.getBalance(accounts[0])
truffle(development)> balance.toNumber()
9960

Archive Docker image

In order to index the local Ethereum node, we need to use a pre-built Docker image of Subsquid’s Ethereum Archive. You can do it, by creating a docker-compose.archive.yml file (I have added one in the archive folder) and paste this:

version: "3"

services:
  worker:
    image: subsquid/eth-archive-worker:latest
    environment:
      RUST_LOG: "info"
    ports:
      - 8080:8080
    command: [
            "/eth/eth-archive-worker",
            "--server-addr", "0.0.0.0:8080",
            "--db-path", "/data/db",
            "--data-path", "/data/parquet/files",
            "--request-timeout-secs", "300",
            "--connect-timeout-ms", "1000",
            "--block-batch-size", "10",
            "--http-req-concurrency", "10",
            "--best-block-offset", "10",
            "--rpc-urls", "http://host.docker.internal:8545/",
            "--max-resp-body-size", "30",
            "--resp-time-limit", "5000",
            "--query-concurrency", "16",
    ]
    # Uncomment this section on Linux machines.
    # The connection to local RPC node will not work otherwise.
    # extra_hosts:
    #   - "host.docker.internal:host-gateway"
    volumes:
      - database:/data/db

volumes:
  database:

Now, let's open the commands.json file, and add archive-up and archive-down to the list of commands. It's not mandatory, but these shortcuts are quicker to type, and easier to remember! 😅

{
    "$schema": "https://cdn.subsquid.io/schemas/commands.json",
    "commands": {
      // ...
      "archive-up": {
        "description": "Start a local EVM Archive",
        "cmd": ["docker-compose", "-f", "archive/docker-compose.archive.yml", "up", "-d"]
      },
      "archive-down": {
        "description": "Stop a local EVM Archive",
        "cmd": ["docker-compose", "-f", "archive/docker-compose.archive.yml", "down"]
      },
      // ...
    }
  }

Then start the service by opening the terminal and launching the command:

sqd archive-up

Squid development

It's finally time to index the smart contract events we generated earlier, with Subsquid’s SDK. Using the contract’s ABI and contract address from previous steps.

If you want to know more about how to develop your squid ETL, please head over to the dedicated tutorial on Subsquid’s official documentation.

Schema

Since we are using MetaCoin contract as a reference, so we'll build a schema that makes sense with respect to the data it produces.

type Transfer @entity {
  id: ID!
  block: Int!
  from: String! @index
  to: String! @index
  value: BigInt!
  txHash: String!
  timestamp: BigInt!
}

And now let's generate the TypeScript model classes for this schema. In a console window, from the project’s root folder, type this command:

sqd codegen

ABI facade

We also need to generate TypeScript code to interact with the contract, and for that, we need the contract's ABI.

Luckily, Truffle generates that when it compiles the Solidity code, and puts the ABI under build/contracts(and Hardhat does the same under artifacts/contracts/). So let's use Subsquid CLI tools and in a console window, from the project’s root folder, type this command:

sqd typegen build/contracts/MetaCoin.json

Processor configuration

Let's open the processor.ts file, and make sure we import the new Transfer model, as well as the ABI facade. From this, we are only going to import the events object.

import { Transfer } from './model';
import { events } from './abi/MetaCoin';

Because we want to index our local Archive, the data source of the processor class needs to be set to the local environment.

I have also createad a constant for the address of the contract we deployed, and always remember to toLowerCase() your contract addresses, to avoid issues.

And finally, we want to configure the processor to request data about the Transfer event:

// ...
const contractAddress = "0xc0bb32fc688c4cac4417b8e732f26836c594c258".toLowerCase()
const processor = new EvmBatchProcessor()
  .setDataSource({
    chain: "http://localhost:8545",
    archive: "http://localhost:8080",
  })
  .addLog(contractAddress, {
    filter: [[events.Transfer.topic]],
    data: {
      evmLog: {
        topics: true,
        data: true,
      },
      transaction: {
        hash: true,
      },
    },
  });
// ...

You can also use environment variables, just like shown in this complete end-to-end project example. This will make sure the project will stay the same, and only configuration files containing environment variables will be changed, depending on the deployment.

Process events

We now need to change the logic of the processor.run() function, to apply our custom logic.

In order to process Transfer events, we need to "unbundle" all the events our processor receives from the Archive in a single batch, and decode each one of them.

The best practice in this scenario, is to use an array as a temporary data storage, and save all the processed transfers at the end, in another batch operation. This will yield better performance, as data storages such as the Postgres database we are using at the moment, are optimized for this. The code should look like this:

processor.run(new TypeormDatabase(), async (ctx) => {
  const transfers: Transfer[] = []
  for (let c of ctx.blocks) {
    for (let i of c.items) {

      if (i.address === contractAddress && i.kind === "evmLog"){
        if (i.evmLog.topics[0] === events.Transfer.topic) {

          const { _from, _to, _value } = events.Transfer.decode(i.evmLog)

          transfers.push(new Transfer({
            id: `${String(c.header.height).padStart(10, '0')}-${i.transaction.hash.slice(3,8)}`,
            block: c.header.height,
            from: _from,
            to: _to,
            value: _value.toBigInt(),
            timestamp: BigInt(c.header.timestamp),
            txHash: i.transaction.hash
          }))
        }
      }
    }
   }
   await ctx.store.save(transfers)
});

Run and test

It's finally time to perform a full test. Let's build the project first:

sqd build

Then, we need to launch the database Docker container, by running the command:

sqd up

Next, we need to remove the database migration file that comes with the template, and generate a new one, for our new database schema:

sqd migration:clean
sqd migration:generate

And we can finally launch the processor:

sqd process

In order to test our data, we can launch the GraphQL server as well. In a different console window, run:

sqd serve

And perform a test query:

query MyQuery {
  transfers {
    amount
    block
    from
    id
    timestamp
    to
    transactionHash
  }
}

Conclusions

What I showed you here is that thanks to Subsquid, it is possible to start working with an indexing middleware from the very early stages of a project.

Developers can build their dApp by building their smart contract locally, deploying it on a local node, and indexing that local node with a squid. This leads to a development and testing cycle with much tighter iterations.

Furthermore, since the indexing middleware is present from the start, fewer changes have to be made (just changing the chain, address, and the contract address itself).

If you found this article interesting, and you want to read more, please follow me and most importantly, Subsquid.

Subsquid socials:

Advancing dApp development with Hardhat Indexing: A Game-Changer for Ethereum Devs

Massimo Luraschi — Fri, 24 Mar 2023 08:52:24 +0000

Summary

Blockchain data indexing helps dApp developers to build better, more performant decentralized applications, ultimately improving the user experience.

One of the most important ways they do this is by decoupling of smart contract development from frontend and, when needed, backend development.

But traditionally, there has been a lack of solutions to help local development in this sense, no one thought about indexing a local EVM node…

But Subsquid did think of it

And this article explains how.

Having an indexing middleware from the start of a project, when no contracts have been deployed on-chain yet, means developers can build the smart contract locally (just as they do now), and index the local node as well. This enables early development of frontend as well, with reduced coupling to the smart contract, or its blockchain.

A smoother, better development experience leads to better dApps, which can be tested earlier and better, before deploying the smart contract on-chain (not even on testnet!).

Introduction

A little over a month ago, I have written an article on how to index a development Ethereum node, using Subsquid and the Ganache/Truffle suite.

I have been positively surprised by the appreciation of the developer community over this very useful feature offered by Subsquid's framework, and over the article itself.

The inspiration for the article came from a project I had built a little over a month earlier, for which I wrote a different article. In this initial project, I was actually using a Hardhat local node for my smart contract development. I decided to switch to Truffle for the final article, because it had a more intuitive demo contract, and because the command line tool made it easier to create manual transactions.

In a quest to provide developers with as many options possible, as much material possible and the maximum number of examples, I decided to revisit this material, and write a guide on how to index a Hardhat node.

The project discussed in this article is available in this repository:

%[https://github.com/RaekwonIII/local-evm-indexing/]

The project

If you haven’t already, you need to install Subsquid CLI first:

npm i -g @subsquid/cli@latest

In order to create a new squid project, simply open a terminal and launch the command:

sqd init local-evm-indexing -t evm

1. Create a Hardhat project

In order to start working with Hardhat, it’s necessary to install the package.

In a console window, from the squid project’s root folder, launch:

npx hardhat

Note: If you get an error referencing your README.md or other files, delete them and run npx hardhat again. Alternatively, choose a different folder, but doing so will slightly diverge from this tutorial.

And follow the guided initialization process. For the purpose of this tutorial, I have chosen a TypeScript project and left other choices as default.

Configure Hardhat automining (optional)

Then, open the hardhat.config.ts (the file extension will be .js if you selected a JavaScript file in the previous step), and add this in the HardhatUserConfig object (or the module.exports for a JavaScript file):

const config: HardhatUserConfig = {
  defaultNetwork: "hardhat",
  networks: {
    hardhat: {
      chainId: 1337,
      mining: {
        auto: false,
        interval: [4800, 5200]
      }
    },
  },
  solidity: "0.8.17",
};

The mining configuration will continuously mine blocks, even if no events or transactions are executed, it can be useful for debugging. You can read more about it on hardhat's official documentation.

Finally, launch the node with this command:

npx hardhat node

2. Deploy the contract (and generate transactions)

Hardhat ships with its own demo contract, called Lock, but I found myself more at ease with the MetaCoin contract from Truffle, which part of the reason why the first article uses Ganache and Truffle as an example.

In order to better compare the setup between the two articles and the two options (Ganache and Hardhat), I am going to continue using the MetaCoin contract in this article.

My suggestion is to just copy the Solidity source code from the repository and paste the two files in the contracts folder of the Hardhat project.

The next thing to do is change the deploy.ts (or .js, depending on how you initialized the project) script, located in the scripts folder. It needs to be edited in order to deploy our the ConvertLib library, as well as our ne. MetaCoin contract.

The hardhat-ethers plugin adds the ethers.js library to the Hardhat node, making it possible to use its functionalities. It's possible to use it to deploy the contract, but since our goal is to create transactions, we can also add some code to do that as well.

It should look like this:

import { ethers } from "hardhat";

async function main() {

  const ConvertLib = await ethers.getContractFactory("ConvertLib");
  const convertLib = await ConvertLib.deploy();

  await convertLib.deployed();

  const MetaCoin = await ethers.getContractFactory("MetaCoin", {
    libraries: {
      ConvertLib: convertLib.address,
    },
  });
  const metaCoin = await MetaCoin.deploy();

  let instance = await metaCoin.deployed();

  console.log(`MetaCoin  deployed to ${metaCoin.address}`);
  let accounts = await ethers.getSigners()
  console.log(`There are a total of ${accounts.length} accounts`)
  await instance.sendCoin(accounts[1].address, 10, {from: accounts[0].address})
  console.log(`Sent 10 coin from ${accounts[0].address} to ${accounts[1].address}`)
  await instance.sendCoin(accounts[2].address, 10, {from: accounts[0].address})
  console.log(`Sent 10 coin from ${accounts[1].address} to ${accounts[2].address}`)
  await instance.sendCoin(accounts[3].address, 10, {from: accounts[0].address})
  console.log(`Sent 10 coin from ${accounts[1].address} to ${accounts[3].address}`)
  await instance.sendCoin(accounts[4].address, 10, {from: accounts[0].address})
  console.log(`Sent 10 coin from ${accounts[1].address} to ${accounts[3].address}`)
}

// We recommend this pattern to be able to use async/await everywhere
// and properly handle errors.
main().catch((error) => {
  console.error(error);
  process.exitCode = 1;
});

Then open a different terminal (the first one is occupied by the running node), and run this command to deploy the contract:

npx hardhat run scripts/deploy.ts --network localhost

Note: please pay attention to the address of the deployedMetaCoin smart contract, as it will be used in the indexer logic section below, and it may vary from the one shown there.

The next sections about the Subsquid Archive and the development of the Squid indexer, are actually unchanged from the original article, and this is not only on purpose, but it's actually the point:

We have changed the tool running the local Ethereum node, but the indexing part stays the same

Hardhat offers the ability to interact with the running blockchain node through a console. If you don’t want to generate transactions on deployment, you can open the console and run the necessary code later on.

Using it is as easy as running the command:
npx hardhat console

Archive Docker image

version: "3"

services:
  worker:
    image: subsquid/eth-archive-worker:latest
    environment:
      RUST_LOG: "info"
    ports:
      - 8080:8080
    command: [
            "/eth/eth-archive-worker",
            "--server-addr", "0.0.0.0:8080",
            "--db-path", "/data/db",
            "--data-path", "/data/parquet/files",
            "--request-timeout-secs", "300",
            "--connect-timeout-ms", "1000",
            "--block-batch-size", "10",
            "--http-req-concurrency", "10",
            "--best-block-offset", "10",
            "--rpc-urls", "http://host.docker.internal:8545",
            "--max-resp-body-size", "30",
            "--resp-time-limit", "5000",
    ]
    # Uncomment this section on Linux machines.
    # The connection to local RPC node will not work otherwise.
    # extra_hosts:
    #   - "host.docker.internal:host-gateway"
    volumes:
      - database:/data/db
volumes:
  database:

Now, let’s open the commands.json file, and add archive-up and archive-down to the list of commands. It's not mandatory, but these shortcuts are quicker to type, and easier to remember! 😅

{
    "$schema": "https://cdn.subsquid.io/schemas/commands.json",
    "commands": {
      // ...
      "archive-up": {
        "description": "Start a local EVM Archive",
        "cmd": ["docker-compose", "-f", "archive/docker-compose.archive.yml", "up", "-d"]
      },
      "archive-down": {
        "description": "Stop a local EVM Archive",
        "cmd": ["docker-compose", "-f", "archive/docker-compose.archive.yml", "down"]
      },
      // ...
    }
  }

Then start the service by opening the terminal and launching the command:

sqd archive-up

Squid development

It’s finally time to index the smart contract events we generated earlier, with Subsquid’s SDK. Using the contract’s ABI and contract address from previous steps.

If you want to know more about how to develop your squid ETL, please head over to the dedicated tutorial on Subsquid’s official documentation.

Schema

Since we are using MetaCoin contract as a reference, so we'll build a schema that makes sense with respect to the data it produces.

type Transfer @entity {
  id: ID!
  block: Int!
  from: String! @index
  to: String! @index
  value: BigInt!
  txHash: String!
  timestamp: BigInt!
}

And now let’s generate the TypeScript model classes for this schema. In a console window, from the project’s root folder, type this command:

sqd codegen

ABI facade

We also need to generate TypeScript code to interact with the contract, and for that, we need the contract’s ABI.

Similarly to Truffle, Hardhat generates that when it compiles the Solidity code, and puts the ABI under artifacts/contracts(while Truffle does the same under build/contracts/). So let's use Subsquid CLI tools and in a console window, from the project’s root folder, type this command:

sqd typegen artifacts/contracts/MetaCoin.sol/MetaCoin.json

Processor configuration

Let’s open the processor.ts file, and make sure we import the new Transfer model, as well as the ABI facade. From this, we are only going to import the events object.

import { Transfer } from './model';
import { events } from './abi/MetaCoin';

Because we want to index our local Archive, the data source of the processor class needs to be set to the local environment.

I have also created a constant for the address of the contract we deployed, and always remember to toLowerCase() your contract addresses, to avoid issues.

Note: This is where you need to use the contract address that has been logged by the deploy.ts script.

And finally, we want to configure the processor to request data about the Transfer event:

// ...
const contractAddress = "0x9fE46736679d2D9a65F0992F2272dE9f3c7fa6e0".toLowerCase()
const processor = new EvmBatchProcessor()
  .setDataSource({
    chain: "http://localhost:8545",
    archive: "http://localhost:8080",
  })
  .addLog(contractAddress, {
    filter: [[events.Transfer.topic]],
    data: {
      evmLog: {
        topics: true,
        data: true,
      },
      transaction: {
        hash: true,
      },
    },
  });
// ...

Process events

We now need to change the logic of the processor.run() function, to apply our custom logic.

In order to process Transfer events, we need to "unbundle" all the events our processor receives from the Archive in a single batch, and decode each one of them.

The best practice in this scenario is to use an array as a temporary data storage and save all the processed transfers at the end, in another batch operation. This will yield better performance, as data storages such as the Postgres database we are using at the moment, are optimized for this. The code should look like this:

processor.run(new TypeormDatabase(), async (ctx) => {
  const transfers: Transfer[] = []
  for (let c of ctx.blocks) {
    for (let i of c.items) {
      if (i.address === contractAddress && i.kind === "evmLog"){
        if (i.evmLog.topics[0] === events.Transfer.topic) {
          const { _from, _to, _value } = events.Transfer.decode(i.evmLog)
          transfers.push(new Transfer({
            id: `${String(c.header.height).padStart(10, '0')}-${i.transaction.hash.slice(3,8)}`,
            block: c.header.height,
            from: _from,
            to: _to,
            value: _value.toBigInt(),
            timestamp: BigInt(c.header.timestamp),
            txHash: i.transaction.hash
          }))
        }
      }
    }
   }
   await ctx.store.save(transfers)
});

Run and test

It’s finally time to perform a full test. Let’s build the project first:

sqd build

Then, we need to launch the database Docker container, by running the command:

sqd up

Next, we need to remove the database migration file that comes with the template, and generate a new one, for our new database schema:

sqd migration:clean
sqd migration:generate

And we can finally launch the processor:

sqd process

In order to test our data, we can launch the GraphQL server as well. In a different console window, run:

sqd serve

And perform a test query:

query MyQuery {
  transfers {
    block
    from
    id
    timestamp
    to
    txHash
    value
  }
}

Conclusions

What I wanted to show with this article is that thanks to Subsquid, it is possible to have an indexing middleware at the same time as developing smart contract(s) for a project.

Furthermore, since the indexing middleware is present from the start, fewer changes have to be made (just changing the chain, address and the contract address itself).

If you found this article interesting, and you want to read more, please follow me and most importantly, Subsquid.

Subsquid socials:

Don't trust, verify: Indexing ENS Domains with Subsquid

Massimo Luraschi — Tue, 21 Mar 2023 10:27:00 +0000

Summary

I leveraged Subsquid's indexing framework to build my very own API that maps an ENS name to its tokenId and vice versa. Why this is needed or important? Read below 😉

The project discussed in this article is available at this repository, on the ens-workshop branch:

github.com/RaekwonIII/ethereum-name-service-indexing/

The repository is also configured, so you can run the project on Gitpod:

Introduction

I have recently published a series of videos and articles to educate about Blockchain Data Indexing.

It specifically discusses Ethereum, although it's applicable to all the EVM networks that Subsquid is available for, plus Substrate networks, with a couple of adaptations I'll probably discuss in a future article. It's a full course consisting of multiple "lessons", introducing Subsquid, the basic concepts, and a few practical tutorials to practice using the SDK.

While creating these tutorials, I choose Ethereum Name Service as an example, because it's a famous project, and quite frankly, also because I take these changes to study some subjects I am interested in (sue me! 😛).

What I didn't realize was that, in doing so, I ended up discovering not one, but two peculiar aspects of the NFTs created by the project, and consequently, two strong cases in favor of blockchain data indexing, and more specifically, two great examples of Subsquid's superiority in this aspect.

Ethereum Domains are NFTs…but they are odd

First of all, this needs to be clarified: whenever someone is registering a domain name on the Ethereum Name Service app, they are minting a Non-Fungible Token from the ENS collection.

This token looks and behaves like a common ERC-721 token: as a matter of fact, the project migrated the registrar in 2019, and since then, it's a contract that is compliant to this standard.

It does, however, have its peculiarities.

Token ID calculation

Unlike usual NFT collections, the tokenId of ENS name is the uint256representation of the hash of the label (vitalik for vitalik.eth). If someone wanted to derive the tokenId for a given ENS name (in this case: vitalik, which computes to 79233663829379634837589865448569342784712482819484549289560981379859480642508), they would have to do it like this:

const ethers = require('ethers')
const BigNumber = ethers.BigNumber
const utils = ethers.utils
const name = 'vitalik'
const labelHash = utils.keccak256(utils.toUtf8Bytes(name))
const tokenId = BigNumber.from(labelHash).toString()

On the flip-side, deriving the ENS name from the tokenId is not as easy. The reason for this was that the fixed-length hash allows to register infinite length of names, but the trade-off is that, because the name is hashed, this is a one-way operation: you cannot obtain ENS names from a tokenId.

All of this is explained in greater detail on ENS documentation page.

Token URL and metadata

Simply put, the ENS contract does not have .tokenURI. The project has, instead, created a separate metadata service which NFT marketplaces like OpenSea can fetch metadata for ENS such as registration data, expiration date, name length, etc.

The problem with this service? It’s using the tokenId to obtain the metadata.

Downsides of these oddities

Let's say you want to visualize the Nyan cat image that Vitalik has associated with his vitalik.eth name: how would you do that?

So this is exactly the adventure I embarked on.

Let's get building!

The project in Lesson 2 of the Blockchain Data Indexing course (you can find the project on this GitHub page) was already indexing ENS tokens, but in order to keep it simple and don't introduce too many variables to the learning process, it stopped at tracking transfers of tokens.

const processor = new EvmBatchProcessor()
.setDataSource({
  chain: process.env.RPC_ENDPOINT,
  archive: "https://eth.archive.subsquid.io",
})
.addLog(contractAddress, {
  filter: [
    [
      events.Transfer.topic,
    ],
  ],
  data: {
    evmLog: {
      topics: true,
      data: true,
    },
    transaction: {
      hash: true,
    },
  },
});

Modifying the schema

Because the objective is to improve the current project and save additional data, we need to change the schema first. I have added new fields, such as name, imageURI, uri and the registrar expiration date as expires :

type Token @entity {
  id: ID!
  owner: Owner
  name: String
  imageURI: String
  uri: String
  expires: BigInt,
  transfers: [Transfer!]! @derivedFrom(field: "token")
  contract: Contract
}

type Owner @entity {
  id: ID!
  ownedTokens: [Token!] @derivedFrom(field: "owner")
}

type Contract @entity {
  id: ID!
  name: String! @index
  symbol: String!
  totalSupply: BigInt!
  tokens: [Token!]! @derivedFrom(field: "contract")
}

type Transfer @entity {
  id: ID!
  token: Token!
  from: Owner
  to: Owner
  timestamp: BigInt!
  block: Int! @index
  transactionHash: String!
}

After the publishing of Lesson 2, Subsquid published a new version of our CLI, with improved commands, make sure to install it with

npm i -g @subsquid/cli@latest

And if you are working on the same project from the lesson, make sure to add this commands.json file to your root folder.

Following these changes, we need to update the TypeScript models, so let's run the command:

sqd codegen

Processor configuration

What is truly interesting for the ENS project, is the ability to index the NameRegisteredand NameRenewed events as well. This way, we are able to capture every time a name is registered, renewed and when it is transferred to a different owner.

For this, we need to change the part where EvmBatchProcessor class is instantiated and configured:

const processor = new EvmBatchProcessor()
  .setDataSource({
    chain: process.env.RPC_ENDPOINT,
    archive: "https://eth.archive.subsquid.io",
  })
  .addLog(contractAddress, {
    filter: [
      [
        events.NameRegistered.topic,
        events.NameRenewed.topic,
        events.Transfer.topic,
      ],
    ],
    data: {
      evmLog: {
        topics: true,
        data: true,
      },
      transaction: {
        hash: true,
      },
    },
  });

Of course, we also need to adjust how we process each item, because we will have logs generated by three different events, with three different topics, and they have to be treated separately:

processor.run(new TypeormDatabase(), async (ctx) => {
  const ensDataArr: ENSData[] = [];
  for (let c of ctx.blocks) {
    for (let i of c.items) {
      if (i.address === contractAddress && i.kind === "evmLog") {
        if (i.evmLog.topics[0] === events.NameRegistered.topic) {
          const ensData = handleNameRegistered({
            ...ctx,
            block: c.header,
            ...i,
          });
          ensDataArr.push(ensData);
        }
        if (i.evmLog.topics[0] === events.NameRenewed.topic) {
          const ensData = handleNameRenewed({
            ...ctx,
            block: c.header,
            ...i,
          });
          ensDataArr.push(ensData);
        }
        if (i.evmLog.topics[0] === events.Transfer.topic) {
          const ensData = handleTransfer({
            ...ctx,
            block: c.header,
            ...i,
          });
          ensDataArr.push(ensData);
        }
      }
    }
  }

  await saveENSData(
    {
      ...ctx,
      block: ctx.blocks[ctx.blocks.length - 1].header,
    },
    ensDataArr
  );
});

Notice the three different handle* functions. They are not too different from each other, but I'd argue they are still different enough to deserve to be separate:

function handleNameRegistered(
  ctx: LogHandlerContext<
    Store,
    { evmLog: { topics: true; data: true }; transaction: { hash: true } }
  >
): ENSData {
  const { evmLog, block, transaction } = ctx;

  const { id, owner, expires } = events.NameRegistered.decode(evmLog);

  const ensData: ENSData = {
    id: `${transaction.hash}-${evmLog.address}-${id.toBigInt()}-${
      evmLog.index
    }`,
    to: owner,
    tokenId: id.toBigInt(),
    timestamp: BigInt(block.timestamp),
    expires: expires.toNumber(),
    block: block.height,
    transactionHash: transaction.hash,
  };

  return ensData;
}

function handleNameRenewed(
  ctx: LogHandlerContext<
    Store,
    { evmLog: { topics: true; data: true }; transaction: { hash: true } }
  >
): ENSData {
  const { evmLog, block, transaction } = ctx;

  const { id, expires } = events.NameRenewed.decode(evmLog);

  const ensData: ENSData = {
    id: `${transaction.hash}-${evmLog.address}-${id.toBigInt()}-${
      evmLog.index
    }`,
    tokenId: id.toBigInt(),
    timestamp: BigInt(block.timestamp),
    expires: expires.toNumber(),
    block: block.height,
    transactionHash: transaction.hash,
  };

  return ensData;
}

function handleTransfer(
  ctx: LogHandlerContext<
    Store,
    { evmLog: { topics: true; data: true }; transaction: { hash: true } }
  >
): ENSData {
  const { evmLog, block, transaction } = ctx;

  const { from, to, tokenId } = events.Transfer.decode(evmLog);

  const ensData: ENSData = {
    id: `${transaction.hash}-${evmLog.address}-${tokenId.toBigInt()}-${
      evmLog.index
    }`,
    from,
    to,
    tokenId: tokenId.toBigInt(),
    timestamp: BigInt(block.timestamp),
    block: block.height,
    transactionHash: transaction.hash,
  };
  return ensData;
}

Augment on-chain data with external APIs

We are getting to the juicy bit here.

Finally, I have explored the tokenId and tokenURI issues mentioned above, and changed the saveENSData function to:

Fetch the token metadata from ENS metadata service
Process this information and extract the name of a given token (establishing the tokenId ↔️ name relation) and the URL of the image
Save this data in the Token entity, on the database

async function saveENSData(
  ctx: BlockHandlerContext<Store>,
  ensDataArr: ENSData[]
) {
  const tokensIds: Set<string> = new Set();
  const ownersIds: Set<string> = new Set();

  for (const ensData of ensDataArr) {
    tokensIds.add(ensData.tokenId.toString());
    if (ensData.from) ownersIds.add(ensData.from.toLowerCase());
    if (ensData.to) ownersIds.add(ensData.to.toLowerCase());
  }

  const transfers: Set<Transfer> = new Set();

  const tokens: Map<string, Token> = new Map(
    (await ctx.store.findBy(Token, { id: In([...tokensIds]) })).map((token) => [
      token.id,
      token,
    ])
  );

  const owners: Map<string, Owner> = new Map(
    (await ctx.store.findBy(Owner, { id: In([...ownersIds]) })).map((owner) => [
      owner.id,
      owner,
    ])
  );

  for (const ensData of ensDataArr) {
    const {
      id,
      tokenId,
      from,
      to,
      block,
      expires,
      transactionHash,
      timestamp,
    } = ensData;

    // the "" case handles absence of sender, which means it's not an actual transaction
    // it's the name being registered
    let fromOwner = owners.get(from || "");
    if (from && fromOwner == null) {
      fromOwner = new Owner({ id: from.toLowerCase() });
      owners.set(fromOwner.id, fromOwner);
    }

    // the "" case handles absence of receiver, which means it's not an actual transaction
    // it's the name being renewed
    let toOwner = owners.get(to || "");
    if (to && toOwner == null) {
      toOwner = new Owner({ id: to.toLowerCase() });
      owners.set(toOwner.id, toOwner);
    }

    const tokenIdString = tokenId.toString();
    let token = tokens.get(tokenIdString);
    if (token == null) {
      token = new Token({
        id: tokenIdString,
        contract: await getOrCreateContractEntity(ctx.store),
      });
      tokens.set(token.id, token);
    }
    token.owner = toOwner;

    if (expires) {
      token.expires = BigInt(expires);
      if (expires * 1000 < Date.now()) {
        // ctx.log.info(`Token ${tokenId} has expired`);
        continue;
      }
      let name = "",
        uri = "",
        imageURI = "";
      try {
        // https://metadata.ens.domains/docs
        let tokenURI = `https://metadata.ens.domains/mainnet/${contractAddress}/${tokenId}`;
        const meta = await axios.get(tokenURI);

        imageURI = meta.data.image;
        name = meta.data.name;
        uri = meta.data.url;
      } catch (e) {
        ctx.log.warn(`[API] Error during fetch token ${tokenId} metadata`);
        if (e instanceof Error) ctx.log.warn(`${e.message}`);
      } finally {
        token.name = name;
        token.uri = uri;
        token.imageURI = imageURI;
      }
    }

    if (toOwner && fromOwner) {
      const transfer = new Transfer({
        id,
        block,
        timestamp,
        transactionHash,
        from: fromOwner,
        to: toOwner,
        token,
      });

      transfers.add(transfer);
    }
  }

  await ctx.store.save([...owners.values()]);
  await ctx.store.save([...tokens.values()]);
  await ctx.store.save([...transfers]);
}

The important part is where the metadata service API is called, using axios library:

let tokenURI = `https://metadata.ens.domains/mainnet/${contractAddress}/${tokenId}`;
const meta = await axios.get(tokenURI);

I want to take a moment to emphasize the ability to call arbitrary external services to obtain additional information, and add to the data we extract from the blockchain.

This way, we are building a service, providing data that would not be possible to produce by purely relying on the blockchain.

Don't trust, verify

The ENS documentation acknowledges the shortcomings of their architectural choice, and provides an alternative, which is using a third party service, or downloading this service's dataset.

One might brush this off as non-consequential, but if you truly subscribe to the motto “Don’t Trust, Verify”, being able to obtain this information independently, in a decentralized way, and without relying on third party services is paramount.

Local indexing

If you have followed along and want to try this at home, you should delete the previous migration file:

rm -rf db/migrations/*js

Launch the database Docker container and generate a new migration file:

sqd up
sqd migration

Launch the processor:

sqd process

And finally, in a separate terminal window, launch the GraphQL service:

sqd serve

We can, for example, test out our initial question: "what is the image linked to vitalik.eht ENS name?" with this query:

query MyQuery {
  tokens(where: {name_contains: "vitalik"}) {
    expires
    id
    imageURI
    name
    uri
    owner {
      id
    }
  }
}

And if we copy paste the imageURI in our browser, we'll eventually land on vitalik's Nyan cat animated image:

Gitpod

Another option, as mentioned at the start of the article, is to run the project on Gitpod by clicking on the related link on the project's README file:

Which will launch the entire project in a cloud-hosted execution environment for us.

Conclusions

What I showed you here is that, thanks to Subsquid, you can build your own indexer, that is able to map every tokenId to its corresponding ENS name, allowing for querying in the opposite direction (get ENS name from tokenId ).

If you found this article interesting, and you want to read more, please follow me and most importantly, Subsquid.

Subsquid socials:

How to Build a performant and scalable Full Stack NFT Marketplace

Massimo Luraschi — Tue, 21 Mar 2023 08:03:40 +0000

by adding an indexing middleware

This article branches off from Nader Dabit's fantastic walkthrough on how to build a Full Stack NFT Marketplace.

I give Nader credit for the idea, and I consider this project a homage to his work.

Introduction

I was inspired into making this video after watching Nader's video on his YouTube channel. It's a proof of concept, functional NFT marketplace, and a great example of a full stack DApp, including a smart contract, as well as frontend.

When I saw it, because of what I do on a daily basis, I instantly noticed something: it is using direct calls to the contract to gather data for the frontend. As a matter of fact, it actually has some custom functions in the smart contract that have been developed specifically for this purpose.

Talking to DApp devs in the past year, I know this is actually quite common, especially in the early stages of development. You basically need to build the frontend and you need a data source: what better data source than the contract itself? It's relatively simple, as the contract is already there, and does not require additional components.

But this actually does not hold up in the long run. The performance of direct RPC calls is bad, it does not scale and so once the contract is deployed to mainnet (and in some cases even earlier, on testnet) you start to notice that it is necessary to add an indexing middleware. This requires rebuilding a decent chunk of the frontend, because instead of relying on the smart contract calls, the dApp now performs requests to a web service. This means duplicating the efforts of frontend developers, it's wasted time, and as we all know, time is money.

But developers are "forced" to do this because there are no easy alternatives. Or there were, and it's what I am going to show in this article.

Project and setup

I have forked Nader's repository in my own GitHub account, you can check it out here:

RaekwonIII/subsquid-ethereum-nextjs-marketplace

In the main branch you can find the end result, but if you want to follow along, with the article, I advise you to clone the repository and checkout the workshop-start branch, which has a couple of necessary changes. For example, at the top of the create-nft.js file, instead of the basic client:

// pages/create-nft.js

const client = ipfsHttpClient('https://ipfs.infura.io:5001/api/v0')

I had to add code to provide authentication when uploading images to IPFS, because Infura node requires it.

// pages/create-nft.js

const auth =
'Basic ' + Buffer.from(process.env.NEXT_PUBLIC_IPFS_PROJECT_ID + ':' + process.env.NEXT_PUBLIC_IPFS_PROJECT_SECRET).toString('base64');

const client = ipfsHttpClient({
    host: 'infura-ipfs.io',
    port: 5001,
    protocol: 'https',
    headers: {
        authorization: auth,
    },
});

I have also added a docker-compose.yml file and a squid folder at the root of the project, but we are going to talk about this in the next chapter.

Small changes aside, this represents a snapshot of Nader's project at the end of his walkthrough. We can launch the project with a series of commands. Run hardhat node:

npx hardhat node

Then deploy the contract (in a different terminal):

npx hardhat run scripts/deploy.js --network localhost

And finally run the app:

npm run dev

When we visit the home page, it is loading the NFTs up for sale, using a special smart contract function called fetchMarketItems():


// pages/index.js

  async function loadNFTs() {
    /* create a generic provider and query for unsold market items */
    const provider = new ethers.providers.JsonRpcProvider()
    const contract = new ethers.Contract(marketplaceAddress, NFTMarketplace.abi, provider)
    const data = await contract.fetchMarketItems()
    // ...
  }

Which we can find in the solidity code. The original video even shows how to code this, but in short, it's doing what the comment says: it's returning unsold items:


//contracts/NFTMarketplace.sol

    /* Returns all unsold market items */
    function fetchMarketItems() public view returns (MarketItem[] memory) {
      uint itemCount = _tokenIds.current();
      uint unsoldItemCount = _tokenIds.current() - _itemsSold.current();
      uint currentIndex = 0;

      MarketItem[] memory items = new MarketItem[](unsoldItemCount);
      for (uint i = 0; i < itemCount; i++) {
        if (idToMarketItem[i + 1].owner == address(this)) {
          uint currentId = i + 1;
          MarketItem storage currentItem = idToMarketItem[currentId];
          items[currentIndex] = currentItem;
          currentIndex += 1;
        }
      }
      return items;
    }

The same goes for the dashboard and my-nfts pages, where the same initialization is done. It's using a different function, with a different logic, but the concept remains the same.

Local development, local indexer

One of the main changes I have done in this branch was adding a squid subfolder, which contains a template of a squid data indexer, and by the end of this article, this is going to be my data source for the Next.js frontend.

This squid indexer relies on another component called Archive, and this is where the docker-compose.yml file in the root folder comes into play. You can read more about this in our docs, but Archives ingest raw blockchain data, which you can then further transform and customize in your own squid, which is essentially an ETL pipeline.

This file defines two services, called ingester and worker, that ingest data from the node, normalize it and present it via a REST API. I have also created my own images, and hosted them in my own registry, because the Ethereum Archive is being actively developed at the moment, and as soon as it stabilizes, Subsquid will likely provide public images.

A quick note about indexing Hardhat, is that the local node's default configuration only "mines" new blocks, when there is an interaction with the chain, like a smart contract function call. To make it easier to follow the Archive ingesting, as well as the squid indexing, I have changed, located in hardhat.config.js. I took the chance to also add configuration to deploy on Goerli network:

// hardhat.config.js
require("@nomiclabs/hardhat-waffle");
const fs = require('fs');
const infuraId = fs.readFileSync(".infuraid").toString().trim() || ""; // process.env.INFURA_NODE_ID;//

const account = fs.readFileSync(".account").toString().trim() || "";

module.exports = {
  defaultNetwork: "hardhat",
  networks: {
    hardhat: {
      chainId: 1337,
      mining: {
        auto: false,
        interval: [4800, 5200]
      }
    },
    goerli: {
      // Infura
      url: `https://goerli.infura.io/v3/${infuraId}`,
      accounts: [account]
    },
    /*
    matic: {
      // Infura
      // url: `https://polygon-mainnet.infura.io/v3/${infuraId}`,
      url: "https://rpc-mainnet.maticvigil.com",
      accounts: [process.env.privateKey]
    }
    */
  },
  solidity: {
    version: "0.8.4",
    settings: {
      optimizer: {
        enabled: true,
        runs: 200
      }
    }
  }
};

I did not find a more elegant way to read INFURA_NODE_ID, or the private key of the throw-away account I'll be using for testnet from the environment variables, so I resorted to writing them down into .infuraid and .account files, which are not part of the repository

With that being said, let's get back to our terminal and launch the Archive services, running the command:

docker compose up -d

Now we can start the squid development.

Squid development

In order for us to use the squid indexer as our data source, we need to customize the template to our needs and launch it.

Schema

The first thing to do is to define the schema, which determines both the database and API structure.

In order for our frontend to access the data it needs, we need to keep track of all the Tokens created and its metadata.

I also need to track the Owner of an NFT.

The Contract entity is something I usually have when I index NFTs from multiple collections. For the sake of this article, I am going to keep it, but I am actually going to hard-code most of the information for it, for simplicity, as it's not very relevant to the purpose of this project.

Lastly, every time an NFTs is sold, we are going to create a new Transfer object, which is keeping track, for example, of the historical price of every transfer.

Here's how the squid/schema.graphql file should look like:

type Token @entity {
  id: ID!
  owner: Owner
  uri: String
  transfers: [Transfer!]! @derivedFrom(field: "token")
  contract: Contract
  name: String,
  description: String
  imageURI: String
  price: BigInt!
  forSale: Boolean
}

type Owner @entity {
  id: ID!
  ownedTokens: [Token!] @derivedFrom(field: "owner")
}

type Contract @entity {
  id: ID!
  name: String! @index
  symbol: String! @index
  # contract URI updated once e.g. a day
  contractURI: String
  address: String
  # timestamp when the contract URI was updated last
  contractURIUpdated: BigInt @index
  totalSupply: BigInt!
  mintedTokens: [Token!]! @derivedFrom(field: "contract")
}

type Transfer @entity {
  id: ID!
  token: Token!
  from: Owner
  to: Owner
  price: BigInt!
  timestamp: BigInt! @index
  block: Int! @index
  transactionHash: String! @index
}

Now, before doing anything else, we need to install the indexer's dependencies. Let's make sure to navigate to the squid subfolder, and then run the command:

npm i

In the same terminal window, launch the command:

make codegen

This is a command from Subsquid's SDK, which generates TypeScript models from this schema we just defined. You can find them under source/model/generated. It has created one file for each one of the entities in the schema and each file defines a class with the same properties from the schema. This is an interface with the database and we are going to use this to save our data on the DB.

Smart contract and ABI

Now we need to make sure we are able to interact with the smart contract and to do this, we need the smart contract ABI. Luckily for us, it has been generated for us by Hardhat, and we can find it under the artifacts folder. Let's head back to the terminal and run:

npx squid-evm-typegen src/abi ../artifacts/contracts/NFTMarketplace.sol/NFTMarketplace.json

This is another command from Subsquid's SDK, and it generates TypeScript facades for EVM transactions, logs and calls. It needs these two arguments: first argument is the destination folder, the second one is the ABI location.

It has created 3 files, the interesting one is src/abi/NFTMarketplace.ts. It defines two objects:

one has LogEvent classes to decode smart contract Events and access their topics
another one with Func classes that allow us to decode smart contract functions and access their input

And a Contract class to directly call smart contract functions that are not payable from our indexer. We are going to use the tokenURI() function later on. Another use case for this class would be to fetch the contract's metadata (instead of hard-coding it, like I am going to do later on), by calling name(), symbol() and other similar functions.


// src/abi/NFTMarketplace.ts
import * as ethers from 'ethers'
import {LogEvent, Func, ContractBase} from './abi.support'
import {ABI_JSON} from './NFTMarketplace.abi'

export const abi = new ethers.utils.Interface(ABI_JSON);

export const events = {
    Approval: new LogEvent<([owner: string, approved: string, tokenId: ethers.BigNumber] & {owner: string, approved: string, tokenId: ethers.BigNumber})>(
        abi, '0x8c5be1e5ebec7d5bd14f71427d1e84f3dd0314c0f7b2291e5b200ac8c7c3b925'
    ),
    ApprovalForAll: new LogEvent<([owner: string, operator: string, approved: boolean] & {owner: string, operator: string, approved: boolean})>(
        abi, '0x17307eab39ab6107e8899845ad3d59bd9653f200f220920489ca2b5937696c31'
    ),
    MarketItemCreated: new LogEvent<([tokenId: ethers.BigNumber, seller: string, owner: string, price: ethers.BigNumber, sold: boolean] & {tokenId: ethers.BigNumber, seller: string, owner: string, price: ethers.BigNumber, sold: boolean})>(
        abi, '0xb640004f1d14576d0c209e240cad0410e0d8c0c33a09375861fbadae2588a98d'
    ),
    Transfer: new LogEvent<([from: string, to: string, tokenId: ethers.BigNumber] & {from: string, to: string, tokenId: ethers.BigNumber})>(
        abi, '0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef'
    ),
}

export const functions = {
    approve: new Func<[to: string, tokenId: ethers.BigNumber], {to: string, tokenId: ethers.BigNumber}, []>(
        abi, '0x095ea7b3'
    ),
    balanceOf: new Func<[owner: string], {owner: string}, ethers.BigNumber>(
        abi, '0x70a08231'
    ),
    createMarketSale: new Func<[tokenId: ethers.BigNumber], {tokenId: ethers.BigNumber}, []>(
        abi, '0xbe9af536'
    ),
    createToken: new Func<[tokenURI: string, price: ethers.BigNumber], {tokenURI: string, price: ethers.BigNumber}, ethers.BigNumber>(
        abi, '0x72b3b620'
    ),
    fetchItemsListed: new Func<[], {}, Array<([tokenId: ethers.BigNumber, seller: string, owner: string, price: ethers.BigNumber, sold: boolean] & {tokenId: ethers.BigNumber, seller: string, owner: string, price: ethers.BigNumber, sold: boolean})>>(
        abi, '0x45f8fa80'
    ),
    fetchMarketItems: new Func<[], {}, Array<([tokenId: ethers.BigNumber, seller: string, owner: string, price: ethers.BigNumber, sold: boolean] & {tokenId: ethers.BigNumber, seller: string, owner: string, price: ethers.BigNumber, sold: boolean})>>(
        abi, '0x0f08efe0'
    ),
    fetchMyNFTs: new Func<[], {}, Array<([tokenId: ethers.BigNumber, seller: string, owner: string, price: ethers.BigNumber, sold: boolean] & {tokenId: ethers.BigNumber, seller: string, owner: string, price: ethers.BigNumber, sold: boolean})>>(
        abi, '0x202e3740'
    ),
    getApproved: new Func<[tokenId: ethers.BigNumber], {tokenId: ethers.BigNumber}, string>(
        abi, '0x081812fc'
    ),
    getListingPrice: new Func<[], {}, ethers.BigNumber>(
        abi, '0x12e85585'
    ),
    isApprovedForAll: new Func<[owner: string, operator: string], {owner: string, operator: string}, boolean>(
        abi, '0xe985e9c5'
    ),
    name: new Func<[], {}, string>(
        abi, '0x06fdde03'
    ),
    ownerOf: new Func<[tokenId: ethers.BigNumber], {tokenId: ethers.BigNumber}, string>(
        abi, '0x6352211e'
    ),
    resellToken: new Func<[tokenId: ethers.BigNumber, price: ethers.BigNumber], {tokenId: ethers.BigNumber, price: ethers.BigNumber}, []>(
        abi, '0xe219fc75'
    ),
    'safeTransferFrom(address,address,uint256)': new Func<[from: string, to: string, tokenId: ethers.BigNumber], {from: string, to: string, tokenId: ethers.BigNumber}, []>(
        abi, '0x42842e0e'
    ),
    'safeTransferFrom(address,address,uint256,bytes)': new Func<[from: string, to: string, tokenId: ethers.BigNumber, data: string], {from: string, to: string, tokenId: ethers.BigNumber, data: string}, []>(
        abi, '0xb88d4fde'
    ),
    setApprovalForAll: new Func<[operator: string, approved: boolean], {operator: string, approved: boolean}, []>(
        abi, '0xa22cb465'
    ),
    supportsInterface: new Func<[interfaceId: string], {interfaceId: string}, boolean>(
        abi, '0x01ffc9a7'
    ),
    symbol: new Func<[], {}, string>(
        abi, '0x95d89b41'
    ),
    tokenURI: new Func<[tokenId: ethers.BigNumber], {tokenId: ethers.BigNumber}, string>(
        abi, '0xc87b56dd'
    ),
    transferFrom: new Func<[from: string, to: string, tokenId: ethers.BigNumber], {from: string, to: string, tokenId: ethers.BigNumber}, []>(
        abi, '0x23b872dd'
    ),
    updateListingPrice: new Func<[_listingPrice: ethers.BigNumber], {_listingPrice: ethers.BigNumber}, []>(
        abi, '0xae677aa3'
    ),
}

export class Contract extends ContractBase {

    balanceOf(owner: string): Promise<ethers.BigNumber> {
        return this.eth_call(functions.balanceOf, [owner])
    }

    fetchItemsListed(): Promise<Array<([tokenId: ethers.BigNumber, seller: string, owner: string, price: ethers.BigNumber, sold: boolean] & {tokenId: ethers.BigNumber, seller: string, owner: string, price: ethers.BigNumber, sold: boolean})>> {
        return this.eth_call(functions.fetchItemsListed, [])
    }

    fetchMarketItems(): Promise<Array<([tokenId: ethers.BigNumber, seller: string, owner: string, price: ethers.BigNumber, sold: boolean] & {tokenId: ethers.BigNumber, seller: string, owner: string, price: ethers.BigNumber, sold: boolean})>> {
        return this.eth_call(functions.fetchMarketItems, [])
    }

    fetchMyNFTs(): Promise<Array<([tokenId: ethers.BigNumber, seller: string, owner: string, price: ethers.BigNumber, sold: boolean] & {tokenId: ethers.BigNumber, seller: string, owner: string, price: ethers.BigNumber, sold: boolean})>> {
        return this.eth_call(functions.fetchMyNFTs, [])
    }

    getApproved(tokenId: ethers.BigNumber): Promise<string> {
        return this.eth_call(functions.getApproved, [tokenId])
    }

    getListingPrice(): Promise<ethers.BigNumber> {
        return this.eth_call(functions.getListingPrice, [])
    }

    isApprovedForAll(owner: string, operator: string): Promise<boolean> {
        return this.eth_call(functions.isApprovedForAll, [owner, operator])
    }

    name(): Promise<string> {
        return this.eth_call(functions.name, [])
    }

    ownerOf(tokenId: ethers.BigNumber): Promise<string> {
        return this.eth_call(functions.ownerOf, [tokenId])
    }

    supportsInterface(interfaceId: string): Promise<boolean> {
        return this.eth_call(functions.supportsInterface, [interfaceId])
    }

    symbol(): Promise<string> {
        return this.eth_call(functions.symbol, [])
    }

    tokenURI(tokenId: ethers.BigNumber): Promise<string> {
        return this.eth_call(functions.tokenURI, [tokenId])
    }
}

Implement mapping logic

Now that all of this is done, let's focus on the main logic, which is found in the processor.ts file. As the name suggests, this is where we define how to process blockchain data. The template has a minimalistic setup, it's indexing transactions towards the Null address and it's printing a log every time it encounters a new transaction.

Before starting to make changes here, I want to mention we are going to use some environment variables. The list includes the address of the marketplace contract, which you can find in the logs of hardhat node, for example, or as a console message of the command that you use for deploying it. The rest of the variables are credentials for an Infura node and Infura IPFS service, to allow us to pin images of the NFTs, as well as their metadata. The Squid itself needs a couple more variables, and they have to do with the Archive's URL, which is the containers we launched earlier, and chain's RPC node, in our case this is the hardhat node.

Create a .env file in the project's root and add these variables, then get an Infura (or equivalent) node and IPFS service, and put the correct values:

# .env

CLIENT_URL="http://localhost:8545"
NEXT_PUBLIC_SQUID_URL="http://localhost:4350/graphql"
NEXT_PUBLIC_WORKSPACE_URL=$CLIENT_URL
NEXT_PUBLIC_INFURA_NODE_ID=""
NEXT_PUBLIC_MARKETPLACE_ADDRESS=""
NEXT_PUBLIC_PRIVATE_KEY=""
NEXT_PUBLIC_IPFS_PROJECT_ID=""
NEXT_PUBLIC_IPFS_PROJECT_SECRET=""

Also, create a squid/.env file and paste this content:

# squid/.env

DB_NAME=squid
DB_PORT=23798
GQL_PORT=4350
ETHEREUM_WSS="http://localhost:8545"
ARCHIVE_URL="http://localhost:8080"

Here is a summary of the changes we will need to make to processor.ts:

change the data source to the ARCHIVE_URL environment variable, and add the chain parameter, assigning the ETHEREUM_WSS variable (the name is actually misleading, http(s) is also accepted)
add a addLog function call to receive data for the MarketItemCreated event
change the current addTransaction function call to request createMarketSale function data, and add a second addTransaction for the resellToken function, as well
add a MarketItemData to save event/function data while processing the batch, before saving to the database
implement a getOrCreateContractEntity function to deal with the Contract model instance, which we'll treat as a singleton. This is where I am going to hard-code contract information, but you can go ahead and use the Contract class from our ABI facades file to fetch data directly from the on-chain contract
implement three functions named handleMarketItemCreatedEvent, handleCreateMarketSaleFunction, handleResellTokenFunction, to process Event/Function data, and return a MarketItemData data interface
implement a saveItems function that accepts an array of MarketItemData (generated by the previous three functions processing the batch), that will save data models on the database in bulk, with some performance optimizations.
change the code inside the for loops in the call back function of of processor.run() to verify what kind of item the current iteration has encountered, and call the right handleX function, then add the MarketItemData to an array created at the top
in the same call back function of processor.run(), at the very end, outside the for loops, call the saveItems function

And here's the final result:

// squid/src/processor.ts

import { Store, TypeormDatabase } from "@subsquid/typeorm-store";
import {
  EvmBatchProcessor,
  BlockHandlerContext,
  LogHandlerContext,
  TransactionHandlerContext,
} from "@subsquid/evm-processor";
import {
  events,
  functions,
  Contract as ContractAPI,
} from "./abi/NFTMarketplace";
import { Contract, Owner, Token, Transfer } from "./model";
import { In } from "typeorm";
import { BigNumber } from "ethers";
import axios from "axios";

const contractAddress =
  process.env.NEXT_PUBLIC_MARKETPLACE_ADDRESS?.toLowerCase() ||
  "0x0000000000000000000000000000000000000000";

const processor = new EvmBatchProcessor()
  .setDataSource({
    chain: process.env.ETHEREUM_WSS,
    archive: process.env.ARCHIVE_URL || "https://goerli.archive.subsquid.io/",
  })
  .addLog(contractAddress, {
    filter: [[events.MarketItemCreated.topic]],
    data: {
      evmLog: {
        topics: true,
        data: true,
      },
      transaction: {
        hash: true,
      },
    },
  })
  .addTransaction(contractAddress, {
    sighash: functions.createMarketSale.sighash,
    data: {
      transaction: {
        from: true,
        input: true,
        to: true,
      },
    },
  })
  .addTransaction(contractAddress, {
    sighash: functions.resellToken.sighash,
    data: {
      transaction: {
        from: true,
        input: true,
        to: true,
      },
    },
  });

processor.run(new TypeormDatabase(), async (ctx) => {
  const marketItemData: MarketItemData[] = [];

  for (const block of ctx.blocks) {
    for (const item of block.items) {
      if (item.kind === "evmLog" && item.address === contractAddress) {
        if (item.evmLog.topics[0] === events.MarketItemCreated.topic) {
          ctx.log.info("found MarketItemCreated Event!");
          const marketItemDatum = handleMarketItemCreatedEvent({
            ...ctx,
            block: block.header,
            ...item,
          });
          marketItemData.push(marketItemDatum);
        }
      }
      if (item.kind === "transaction" && item.address === contractAddress) {
        if (
          item.transaction.input.slice(0, 10) ===
          functions.createMarketSale.sighash
        ) {
          ctx.log.info("found createMarketSale transaction!");

          const marketItemDatum = handleCreateMarketSaleFunction({
            ...ctx,
            block: block.header,
            ...item,
          });
          marketItemData.push(marketItemDatum);
        }
        if (
          item.transaction.input.slice(0, 10) === functions.resellToken.sighash
        ) {
          ctx.log.info("found resellToken transaction!");
          const marketItemDatum = handleResellTokenFunction({
            ...ctx,
            block: block.header,
            ...item,
          });
          marketItemData.push(marketItemDatum);
        }
      }
    }
  }

  await saveItems(
    {
      ...ctx,
      block: ctx.blocks[ctx.blocks.length - 1].header,
    },
    marketItemData
  );
});

type MarketItemData = {
  id: string;
  from?: string;
  to?: string;
  tokenId: bigint;
  price?: bigint;
  forSale: boolean;
  timestamp: bigint;
  block: number;
  transactionHash: string;
};

let contractEntity: Contract | undefined;

export async function getOrCreateContractEntity(
  store: Store
): Promise<Contract> {
  if (contractEntity == null) {
    contractEntity = await store.get(Contract, contractAddress);
    if (contractEntity == null) {
      contractEntity = new Contract({
        id: process.env.NEXT_PUBLIC_MARKETPLACE_ADDRESS,
        name: "MassimoTest",
        symbol: "XYZ",
        totalSupply: 0n,
      });
      await store.insert(contractEntity);
    }
  }
  return contractEntity;
}

function handleCreateMarketSaleFunction(
  ctx: TransactionHandlerContext<
    Store,
    {
      transaction: {
        from: true;
        input: true;
        to: true;
      };
    }
  >
): MarketItemData {
  const { transaction, block } = ctx;

  const { tokenId } = functions.createMarketSale.decode(transaction.input);
  const transactionHash = transaction.input;
  const addr = transaction.to;

  const marketItem: MarketItemData = {
    id: `${transactionHash}-${addr}-${tokenId.toBigInt()}-${transaction.index}`,
    tokenId: tokenId.toBigInt(),
    to: transaction.from || "",
    forSale: false,
    timestamp: BigInt(block.timestamp),
    block: block.height,
    transactionHash: transaction.input,
  };

  return marketItem;
}

function handleResellTokenFunction(
  ctx: TransactionHandlerContext<
    Store,
    {
      transaction: {
        from: true;
        input: true;
        to: true;
      };
    }
  >
): MarketItemData {
  const { transaction, block } = ctx;

  const { tokenId, price } = functions.resellToken.decode(transaction.input);

  const transactionHash = transaction.input;
  const addr = transaction.to?.toLowerCase();

  const marketItem: MarketItemData = {
    id: `${transactionHash}-${addr}-${tokenId.toBigInt()}-${transaction.index}`,
    tokenId: tokenId.toBigInt(),
    price: price.toBigInt(),
    forSale: true,
    timestamp: BigInt(block.timestamp),
    block: block.height,
    transactionHash: transaction.input,
  };

  return marketItem;
}

function handleMarketItemCreatedEvent(
  ctx: LogHandlerContext<
    Store,
    { evmLog: { topics: true; data: true }; transaction: { hash: true } }
  >
): MarketItemData {
  const { evmLog, transaction, block } = ctx;

  // the owner and seller fields are mislabelled
  // when a MarketItem is created, the "seller" is the owner
  // the NFT is just temporarily transferred to the contract itself
  // I decide to use the `sold` field to distinguish between
  // "my nfts" and "my nfts up for sale", reversing the logic
  const { tokenId, seller, owner, price, sold } =
    events.MarketItemCreated.decode(evmLog);

  const marketItem: MarketItemData = {
    id: "", // `${transaction.hash}-${addr}-${tokenId.toBigInt()}-${evmLog.index}`,
    tokenId: tokenId.toBigInt(),
    to: seller,
    price: price.toBigInt(),
    forSale: !sold, // when the item is created, `sold` is set to false, so it's up for sale
    timestamp: BigInt(block.timestamp),
    block: block.height,
    transactionHash: transaction.hash,
  };

  return marketItem;
}

async function saveItems(
  ctx: BlockHandlerContext<Store>,
  transfersData: MarketItemData[]
) {
  const tokensIds: Set<string> = new Set();
  const ownersIds: Set<string> = new Set();

  for (const transferData of transfersData) {
    tokensIds.add(transferData.tokenId.toString());
    if (transferData.from) ownersIds.add(transferData.from.toLowerCase());
    if (transferData.to) ownersIds.add(transferData.to.toLowerCase());
  }

  const transfers: Set<Transfer> = new Set();

  const tokens: Map<string, Token> = new Map(
    (await ctx.store.findBy(Token, { id: In([...tokensIds]) })).map((token) => [
      token.id,
      token,
    ])
  );

  const owners: Map<string, Owner> = new Map(
    (await ctx.store.findBy(Owner, { id: In([...ownersIds]) })).map((owner) => [
      owner.id,
      owner,
    ])
  );

  for (const transferData of transfersData) {
    const {
      id,
      tokenId,
      from,
      to,
      block,
      transactionHash,
      price,
      forSale,
      timestamp,
    } = transferData;
    const contract = new ContractAPI(ctx, { height: block }, contractAddress);

    // the "" case handles absence of sender, which means it's not an actual transaction
    // likely, it's the token being minted and put up for sale
    let fromOwner = owners.get(from || "");
    if (from && fromOwner == null) {
      fromOwner = new Owner({ id: from.toLowerCase() });
      owners.set(fromOwner.id, fromOwner);
    }

    // the "" case handles absence of receiver, which means it's not an actual transaction
    // likely it's the token being put up for re-sale
    let toOwner = owners.get(to || "");
    if (to && toOwner == null) {
      toOwner = new Owner({ id: to.toLowerCase() });
      owners.set(toOwner.id, toOwner);
    }

    const tokenIdString = tokenId.toString();

    let token = tokens.get(tokenIdString);

    let tokenURI,
      name,
      description,
      imageURI = "";
    try {
      tokenURI = await contract.tokenURI(BigNumber.from(tokenId));
    } catch (error) {
      ctx.log.warn(`[API] Error during fetch tokenURI of ${tokenIdString}`);
      if (error instanceof Error) ctx.log.warn(`${error.message}`);
    }

    if (tokenURI !== "") {
      // fetch metadata and assign name, description, imageURI
      const meta = await axios.get(tokenURI || "1");

      imageURI = meta.data.image;
      name = meta.data.name;
      description = meta.data.description;
    }

    if (token == null) {
      token = new Token({
        id: tokenIdString,
        uri: tokenURI,
        name,
        description,
        imageURI,
        price,
        contract: await getOrCreateContractEntity(ctx.store),
      });
      tokens.set(token.id, token);
    }
    token.owner = toOwner;
    token.forSale = forSale;
    token.price = price || token.price; // change the price ONLY if changed by function/event

    if (toOwner && fromOwner) {
      const transfer = new Transfer({
        id,
        block,
        timestamp,
        transactionHash,
        from: fromOwner,
        to: toOwner,
        price: token.price,
        token,
      });

      transfers.add(transfer);
    }
  }

  await ctx.store.save([...owners.values()]);
  await ctx.store.save([...tokens.values()]);
  await ctx.store.save([...transfers]);
}

Once we have implemented our code to decode all the data, processe it like we wanted, and save, we are ready test it out. To do so, let's start the database container. In a terminal, from the squid folder, launch the command:

make up

Then we need to build our TypeScript code:

make build

Then, one thing I didn't mention before, under the db/migrations folder, there is a javascript file, which is used to initialize the database schema for us. We need to delete the old one, and generate a new one, by running the command:

make migration

Let's launch the processor, by running the command:

make process

We should be seeing logs, telling us we came across some events or functions. There's one more thing we can do, we can launch the GraphQL server. In a different terminal window, since the make process command is blocking, run:

make serve

The log tells us which port the server is listening to: 4350. This means we could open our browser to the url: http://localhost:4350/graphql. Here, we can use the GraphQL playground to test some queries, and find out, for example, that this one:

      query MyQuery {
        tokens(orderBy: id_ASC, where: {AND: [
          {owner: { id_eq: "INSERT_ADDRESS_HERE" }}, 
          {forSale_eq: true}
        ]}) {
          description
          forSale
          id
          imageURI
          name
          price
          uri
          owner {
            id
          }
        }
      }

Can be used to fetch NFTs being listed by a certain user, and it's going to be useful in the next section.

Changing the App to use our API

Once the API is up and running, we can go back to our IDE, and start changing the App pages, to use it, instead of querying the contract. The create-nft.js file does not need our intervention, because the contract interaction is necessary for the minting.

Dashboard page

But we can start with the dashboard.js. Here we have this section, in the loadNFTs function, which we are going to replace.

// pages/dashboard.js

// ...

  async function loadNFTs() {

// ...

    const contract = new ethers.Contract(marketplaceAddress, NFTMarketplace.abi, signer)
    const data = await contract.fetchItemsListed()

    const items = await Promise.all(data.map(async i => {
      const tokenUri = await contract.tokenURI(i.tokenId)
      const meta = await axios.get(tokenUri)
      let price = ethers.utils.formatUnits(i.price.toString(), 'ether')
      let item = {
        price,
        tokenId: i.tokenId.toNumber(),
        seller: i.seller,
        owner: i.owner,
        image: meta.data.image,
      }
      return item
    }))

    setNfts(items)
    setLoadingState('loaded') 
  }
// ...

In order to use the squid API, we need to:

get the address of the person logged in with Metamask wallet
set headers of an API request
create a requestBody object, containing the GraphQL query we had developed at the end of last chapter, with a caveat: it will accept a variable, containing the address of the connected Metamask wallet, as the owner of the NFTs
create an options objects for API request options
use the axios library to perform an API request, using the options we created
process the response into an items array, similarly to the code snippet above
call the setNfts function, providing the items array we just created

Here's the end result:

// pages/dashboard.js

import { ethers } from 'ethers'
import { useEffect, useState } from 'react'
import axios from 'axios'
import Web3Modal from 'web3modal'

export default function CreatorDashboard() {
  const [nfts, setNfts] = useState([])
  const [loadingState, setLoadingState] = useState('not-loaded')
  useEffect(() => {
    loadNFTs()
  }, [])
  async function loadNFTs() {
    const web3Modal = new Web3Modal({
      network: 'mainnet',
      cacheProvider: true,
    })
    const connection = await web3Modal.connect()
    const provider = new ethers.providers.Web3Provider(connection)
    const signer = provider.getSigner()

    const owner = (await signer.getAddress()).toLowerCase();

    const headers = {
      'content-type': 'application/json',
    };
    const requestBody = {
      query: `query MyQuery ($owner: String!){
        tokens(orderBy: id_ASC, where: {AND: [
          {owner: { id_eq: $owner }}, 
          {forSale_eq: true}
        ]}) {
          description
          forSale
          id
          imageURI
          name
          price
          uri
          owner {
            id
          }
        }
      }`,
      variables: { owner }
    };
    const options = {
      method: 'POST',
      url: process.env.NEXT_PUBLIC_SQUID_URL,
      headers,
      data: requestBody
    };
    try {
      const response = await axios(options);

      const squiditems = await Promise.all(response.data.data.tokens.map(async i => {
        let item = {
          price: ethers.utils.formatUnits(i.price, 'ether'),
          tokenId: Number(i.id),
          owner: i.owner.id,
          image: i.imageURI,
          tokenURI: i.uri,
          name: i.name,
          description: i.description,
        }
        return item
      }))

      setNfts(squiditems)
      setLoadingState('loaded')
    } catch (err) {
      console.log('ERROR DURING AXIOS REQUEST', err);
    }
  }
  if (loadingState === 'loaded' && !nfts.length) return (<h1 className="py-10 px-20 text-3xl">No NFTs listed</h1>)
  return (
    <div>
      <div className="p-4">
        <h2 className="text-2xl py-2">Items Listed</h2>
          <div className="grid grid-cols-1 sm:grid-cols-2 lg:grid-cols-4 gap-4 pt-4">
          {
            nfts.map((nft, i) => (
              <div key={i} className="border shadow rounded-xl overflow-hidden">
                <img src={nft.image} className="rounded" />
                <div className="p-4 bg-black">
                  <p className="text-2xl font-bold text-white">Price - {nft.price} Eth</p>
                </div>
              </div>
            ))
          }
        </div>
      </div>
    </div>
  )
}

Home page

We can now apply the same changes to the other pages, starting with the index.js file, which manages the home page.

In the home page, we see all NFTs up for sale, regardless of the owner. So this means the query we used before can be recycled, we just need to get rid of the second condition, and every mention of the variable, because it's no longer used.

// pages/index.js

import { ethers } from 'ethers'
import { useEffect, useState } from 'react'
import axios from 'axios'
import Web3Modal from 'web3modal'

import {
  marketplaceAddress
} from '../config'

import NFTMarketplace from '../artifacts/contracts/NFTMarketplace.sol/NFTMarketplace.json'

export default function Home() {
  const [nfts, setNfts] = useState([])
  const [loadingState, setLoadingState] = useState('not-loaded')
  useEffect(() => {
    loadNFTs()
  }, [])
  async function loadNFTs() {
    const headers = {
      'content-type': 'application/json',
    }
    const requestBody = {
      query: `query AllNFTsForSale {
        tokens(where: {forSale_eq: true}, orderBy: id_ASC) {
          description
          forSale
          id
          imageURI
          name
          price
          uri
          owner {
            id
          }
        }
      }
      `,
    }
    const options = {
      method: 'POST',
      url: process.env.NEXT_PUBLIC_SQUID_URL,
      headers,
      data: requestBody
    }
    try {
      const response = await axios(options);

      const squiditems = await Promise.all(response.data.data.tokens.map(async i => {
        let item = {
          price: ethers.utils.formatUnits(i.price, 'ether'),
          tokenId: Number(i.id),
          owner: i.owner.id,
          image: i.imageURI,
          tokenURI: i.uri,
          name: i.name,
          description: i.description,
        }
        return item
      }))

      setNfts(squiditems)
      setLoadingState('loaded')
    } catch (err) {
      console.log('ERROR DURING AXIOS REQUEST', err);
    }
  }
  async function buyNft(nft) {
    /* needs the user to sign the transaction, so will use Web3Provider and sign it */
    const web3Modal = new Web3Modal()
    const connection = await web3Modal.connect()
    const provider = new ethers.providers.Web3Provider(connection)
    const signer = provider.getSigner()
    const contract = new ethers.Contract(marketplaceAddress, NFTMarketplace.abi, signer)

    /* user will be prompted to pay the asking proces to complete the transaction */
    const price = ethers.utils.parseUnits(nft.price.toString(), 'ether')   
    const transaction = await contract.createMarketSale(nft.tokenId, {
      value: price
    })
    await transaction.wait()
    loadNFTs()
  }
  if (loadingState === 'loaded' && !nfts.length) return (<h1 className="px-20 py-10 text-3xl">No items in marketplace</h1>)
  return (
    <div className="flex justify-center">
      <div className="px-4" style={{ maxWidth: '1600px' }}>
        <div className="grid grid-cols-1 sm:grid-cols-2 lg:grid-cols-4 gap-4 pt-4">
          {
            nfts.map((nft, i) => (
              <div key={i} className="border shadow rounded-xl overflow-hidden">
                <img src={nft.image} />
                <div className="p-4">
                  <p style={{ height: '64px' }} className="text-2xl font-semibold">{nft.name}</p>
                  <div style={{ height: '70px', overflow: 'hidden' }}>
                    <p className="text-gray-400">{nft.description}</p>
                  </div>
                </div>
                <div className="p-4 bg-black">
                  <p className="text-2xl font-bold text-white">{nft.price} ETH</p>
                  <button className="mt-4 w-full bg-pink-500 text-white font-bold py-2 px-12 rounded" onClick={() => buyNft(nft)}>Buy</button>
                </div>
              </div>
            ))
          }
        </div>
      </div>
    </div>
  )
}

My NFTs page

Here we load all the NFTs in possession of the Metamask wallet address, which are not for sale. So once again, the initial query can be recycled, this time we need to switch the forSale logic to false.

// pages/my-nfts.js

import { ethers } from 'ethers'
import { useEffect, useState } from 'react'
import axios from 'axios'
import Web3Modal from 'web3modal'
import { useRouter } from 'next/router'

const auth =
'Basic ' + Buffer.from(process.env. NEXT_PUBLIC_IPFS_PROJECT_ID + ':' + process.env. NEXT_PUBLIC_IPFS_PROJECT_SECRET).toString('base64');

export default function MyAssets() {
  const [nfts, setNfts] = useState([])
  const [loadingState, setLoadingState] = useState('not-loaded')
  const router = useRouter()
  useEffect(() => {
    loadNFTs()
  }, [])

  async function loadNFTs() {
    const web3Modal = new Web3Modal({
      network: "mainnet",
      cacheProvider: true,
    })
    const connection = await web3Modal.connect()
    const provider = new ethers.providers.Web3Provider(connection);
    const signer = provider.getSigner();

    const owner = await (await signer.getAddress()).toLowerCase();

    const headers = {
      'content-type': 'application/json',
    };
    const requestBody = {
      query: `query MyQuery ($owner: String!){
        tokens(orderBy: id_ASC, where: {AND: [
          {owner: { id_eq: $owner }}, 
          {forSale_eq: false}
        ]}) {
          description
          forSale
          id
          imageURI
          name
          price
          uri
          owner {
            id
          }
        }
      }`,
      variables: { owner }
    };
    const options = {
      method: 'POST',
      url: process.env.NEXT_PUBLIC_SQUID_URL,
      headers,
      data: requestBody
    };

    try {
      const response = await axios(options);

      const squiditems = await Promise.all(response.data.data.tokens.map(async i => {
        let item = {
          price: ethers.utils.formatUnits(i.price, 'ether'),
          tokenId: Number(i.id),
          owner: i.owner.id,
          image: i.imageURI,
          tokenURI: i.uri,
          name: i.name,
          description: i.description,
        }
        return item
      }))
      setNfts(squiditems)
      setLoadingState('loaded') 
    }
    catch (err) {
      console.log('ERROR DURING AXIOS REQUEST', err);
    }
  }
  function listNFT(nft) {
    console.log('nft:', nft)
    router.push(`/resell-nft?id=${nft.tokenId}&tokenURI=${nft.tokenURI}`)
  }
  if (loadingState === 'loaded' && !nfts.length) return (<h1 className="py-10 px-20 text-3xl">No NFTs owned</h1>)
  return (
    <div className="flex justify-center">
      <div className="p-4">
        <div className="grid grid-cols-1 sm:grid-cols-2 lg:grid-cols-4 gap-4 pt-4">
          {
            nfts.map((nft, i) => (
              <div key={i} className="border shadow rounded-xl overflow-hidden">
                <img src={nft.image} className="rounded" />
                <div className="p-4 bg-black">
                  <p className="text-2xl font-bold text-white">Price - {nft.price} Eth</p>
                  <button className="mt-4 w-full bg-pink-500 text-white font-bold py-2 px-12 rounded" onClick={() => listNFT(nft)}>List</button>
                </div>
              </div>
            ))
          }
        </div>
      </div>
    </div>
  )
}

Finally, the resell-nft.js page, similarly to create-nft.js does not need any changes.

Tests and Goerli deployment

Next.js has auto reload, so every change in our code is automatically considered. So if we reload the page, and open the inspector, in the Networks tab, we can see that there's a graphql request, and that's where the data to show it in the page comes from.

We managed to develop an NFT API, by indexing blockchain data, and we integrated it in our project. All of this was done on our local development environment, which means when this application will go live, the changes are going to be minimal.

As a matter of fact, it is possible to deploy this project on Goerli, the Ethereum testnet, and index it right away.

First of all, if you don't have them already, set up you Metamask with a real ETH address (I advise you to use a throw-away account for these tests), then head over to a faucet and get some test ETH, if you don't have enough already.

And finally, we need to deploy our contract to the testnet. From the project's root folder, in a terminal, run the command:

npx hardhat run scripts/deploy.js --network goerli

The good news is, to re-configure our App and index our contract, we just need to change a couple of environment variables:

contract address on Goerli (note it down, after launching the deploy script)
Ethereum rpc endpoint
the archive URL, because we are going to use one of the many official Archives from Subsquid

# .env
# ...
NEXT_PUBLIC_MARKETPLACE_ADDRESS="NEW_ADDRESS"
# ...

# squid/.env
# ...
ETHEREUM_WSS="https://goerli.infura.io/v3/YOUR_INFURA_ID"
ARCHIVE_URL="https://goerli.archive.subsquid.io/"

We can stop our squid, and the Next.js app, then launch them again with the new configuration. Bear in mind, to properly test out the project, you'll need to switch accounts in Metamask to a real ETH address.

Conclusions

With just a couple of changes to environment variables, we are testing out our application when the contract is deployed on testnet, without having to rewrite our DApp. I chose to deploy the project on Goerli, but Subsquid just announced Polygon compatibility, so you can also deploy it on Mumbai network.

This is, in my opinion, a very good way to future-proof your application, making sure that the user experience is not degraded by the RPC bottleneck.

At Subsquid is continuously developing, and will continue adding compatibilities to new networks, and deploy archives for them. So head over to the website, and to the docs to know more, and follow Subsquid on social media to stay up to date.

If you like to see more of these demo projects, follow me on Twitter, and feel free to reach out to me, and request/suggest in the comments.