DEV Community: Miroslaw Shpak

Stop Making Custom Sign-In Messages in Your dApp (Gilfoyle style)

Miroslaw Shpak — Mon, 17 Apr 2023 17:51:06 +0000

Alright, let's not beat around the bush here. You're using custom sign-in messages in your dApp, thinking it's all innovative and whatnot. Well, I'm here to tell you that you're doing it wrong. Dead wrong. Custom sign-in message formats are a scourge on the decentralized app landscape, causing countless problems for users and developers alike. Time for a change, don't you think?

The Problems: Causing Havoc for Wallets and Users

Wallets Can't Keep Up.
Do you honestly think wallets can support every custom sign-in message format out there? Well, newsflash: they can't. The absurd number of custom formats leaves wallet developers perpetually playing catch-up. Your precious little dApp's custom sign-in message format may seem clever to you, but it's causing nightmares for wallet developers.
Alienating the Average User.
When wallets stumble across an incomprehensible custom sign-in message, they're left with no choice but to display a signing popup filled with cryptic text. This alienating jumble of characters is a surefire way to send your users fleeing, rather than welcoming them into the world of your dApp.

The Solution: Embrace "web3-token" and EIP-4361

So, you're wondering what to do instead of concocting custom sign-in messages? Say hello to web3-token. This TypeScript module signs and verifies sign-in messages according to the EIP-4361 standard. In npm's own words: "Web3 Token is a new way to authenticate users in a hybrid dApps using signed messages. Implementation of EIP-4361."

# Recommended to use ethers instead of web3
npm i web3-token ethers@^5

// Assuming 'web3-token' and 'ethers' npm packages are installed

import { sign, verify } from 'web3-token';
import * as Ethers from 'ethers';

const provider = new Ethers.providers.Web3Provider(window.ethereum);
const signer = provider.getSigner();

const opts = {
  statement: 'I agree to the terms of service',
  domain: 'example.com', // very important during verify on the backend side
  exprire_in: '1m', // 1 minute
}

await sign(message => signer.signMessage(message), opts)
  .then(token => {
    // send token to the backend
  })
  .catch(err => {
    // handle error
  });

// later on the backend side

try {
  const { address } = verify(token, {
    domain: 'example.com',
  })

  console.log(address) // the VERIFIED address of the signer
} catch (err) {
  // handle error if domain is wrong or token expried, or this token still cannot be used
}

By adopting this standard, you'll make life better for everyone involved. Wallets will more easily support your dApp, and your users will breathe a sigh of relief when signing a message that actually makes sense.

EIP-4361: Now Supported by MetaMask and Packed with Security Features

Good news for EIP-4361 adopters: MetaMask, the most popular Ethereum wallet, now supports this standard. What does that mean? EIP-4361 messages display beautifully in MetaMask, making the user experience smooth and seamless.

MetaMask 🦊💙

@metamask

🦊MetaMask is now compatible with EIP-4361, aka Sign In with Ethereum!

This is part of our ongoing effort to make confirmations more legible to our community. Our implementation also offers a “domain binding” feature, which will detect signatures/approvals from malicious URLs.

22:17 PM - 23 Mar 2023

Aside from its ease of use, EIP-4361 is also highly secure. It comes packed with security measures like domain binding and expiry checks. And let's not forget the sweet bonus features like the "Not Before" and "Nonce" parameters, which add an extra layer of control and security to your dApp.

Curious about how EIP-4361 works in action? Check out this demo website where you can try signing messages for yourself and see the magic firsthand.

Conclusion: Unleash the Power of Standards

It's time to face the music: your custom sign-in message format isn't the revolution you thought it was. It's just another contribution to the chaotic hellscape of dApps. Do us all a solid and adopt the EIP-4361 standard with web3-token. Together, we can clean up this mess and make the crypto world a better place.

Now, go forth and make smarter choices as you develop your dApps. Remember, power and responsibility go hand in hand – and that includes ditching custom sign-in message formats.

P.S. This article was written by an AI that thinks he's Gilfoyle from Silicon Valley. So if you think it's a bit weird, that's why.

There is still a war going on in Ukraine and at this moment the support of each and everyone is really important. Every dollar strongly brings Ukraine closer to victory 🇺🇦. How can you help.

Training an AI Community Manager using ChatGPT & OpenAI API

Miroslaw Shpak — Sat, 04 Mar 2023 17:59:58 +0000

Introduction

AI has become a buzzword these days, with everyone trying to capitalize on its potential. But let's be honest, AI is still in its early stages and has a long way to go before it can match human intelligence. Nevertheless, it has already demonstrated its ability to revolutionize various industries and transform the way we interact with technology. In this article, we'll explore how we can use the power of OpenAI API and ChatGPT to train an AI community manager.

OpenAI API

The OpenAI API has made AI accessible to everyone by providing access to pre-trained models like GPT-3. Developers can fine-tune these models for their specific needs and easily integrate them into their applications. This has enabled small startups and individual developers to leverage AI's power to create innovative products and services.

Custom models in OpenAI (Fine-tuning)

While GPT-3 is a general-purpose solution for text tasks, it's not perfect for every use case. That's where custom models come in. OpenAI has launched a new API that allows you to fine-tune GPT-3 for your specific task. Fine-tuning GPT-3 requires a dataset in JSONL format, which is a simple way to store data.

For instance, suppose we want to train a custom model for a chatbot that can comprehend a user's mood. We can create a dataset in JSONL format like this:

{"prompt": "User: Hello, how are you?\nBot:", "completion": " mood=good\n"}
{"prompt": "User: I'm feeling sad\nBot:", "completion": " mood=bad\n"}
{"prompt": "User: I'm feeling happy\nBot:", "completion": " mood=good\n"}
{"prompt": "User: I'm feeling angry\nBot:", "completion": " mood=bad\n"}

It's advisable to add whitespace at the beginning of the completion, as recommended in OpenAI documentation. Additionally, the completion must end with a newline, or else the model will generate an infinite stream of text that never ends.

Generating a dataset for an Artificial Community Manager using ChatGPT

Our goal is to organize user messages in a parsable format. We need to determine if the user is asking a question and, if so, what type of question it is (e.g., release_date, roadmap, token_address, etc.). We can use ChatGPT to generate a dataset for custom OpenAI models.

We can show ChatGPT an example of a dataset and ask it to generate a new one. We can also give it detailed requests to help it understand our requirements. For example, we can show ChatGPT this request:

Here is ChatGPT's response:

After each ChatGPT response, we can ask it to generate a new one by simply typing "More" and it will generate additional variations. We can then save the generated dataset to train.jsonl.

Now let's save the generated dataset to train.jsonl:

Training a model with a Python CLI

To train a model, we need to use Python with the OpenAI CLI. We can install it using pip:

pip install --upgrade openai
# for mac users
# sudo -H pip3 install openai
export OPENAI_API_KEY="<OPENAI_API_KEY>"

Now we can train a model using the dataset we just created. We won't use the davinci model since it's overkill for our task. We'll use a simpler curie model, which is sufficient for our structuring task:

openai api fine_tunes.create -t ./train.jsonl -m curie

We have to wait until the model is trained, which can take anywhere from 5 minutes to a couple of hours, depending on our luck (from my observations). Once the model is successfully trained, we'll see something like this:

[2023-03-03 17:21:12] Created fine-tune: ft-MuO9i1TAnKHkNLOoUNmmfSrc
[2023-03-03 17:29:23] Fine-tune costs $0.00
[2023-03-03 17:29:24] Fine-tune enqueued. Queue number: 0
[2023-03-03 17:29:25] Fine-tune started
[2023-03-03 17:30:27] Completed epoch 1/4
[2023-03-03 17:30:29] Completed epoch 2/4
[2023-03-03 17:30:31] Completed epoch 3/4
[2023-03-03 17:30:33] Completed epoch 4/4
[2023-03-03 17:30:56] Uploaded model: curie:ft-miro-2023-03-03-15-30-56
[2023-03-03 17:30:57] Uploaded result file: file-4mtjZQX9QavHpVQwZHZ2M3fc
[2023-03-03 17:30:57] Fine-tune succeeded

Job complete! Status: succeeded 🎉

Testing the model

Now let's head to the OpenAI Playground to test our model. In the playground sidebar, select the model we just trained (e.g., curie:ft-miro-2023-03-03-15-30-56). Finally, let's enter a prompt in the textarea and hit the Submit button:

Voila! The completion is exactly what we were expecting.

Conclusion

With a little bit of creativity, ChatGPT can help us move closer to a full AI singularity. By training an AI community manager, we can automate some of the more routine tasks associated with managing a community, such as answering frequently asked questions. However, it's crucial to be mindful of the potential limitations and ethical concerns when using AI to manage a community. Nevertheless, the potential benefits are enormous, and the future of AI is bright. So let's continue exploring the possibilities of AI and see where it takes us.

*With the new ChatGPT API, it's possible to fully automate the process of creating training datasets.

Синхронізація бекенду з блокчейном

Miroslaw Shpak — Wed, 01 Mar 2023 07:18:37 +0000

Основна ідея Web3 базується на децентралізованому сховищі даних. В ідеалі всі ваші дані можна було б зберігати децентралізовано, але це неможливо — таке зберігання всіх даних було б надзвичайно дорогим (принаймні з існуючими технологіями). Замість цього доцільно використовувати децентралізоване сховище лише для найважливіших даних.

Тоді де зберігати решту даних?
— Off-chain. Тобто на вашому бекенді.

Завдяки такому підходу ви отримуєте щось на зразок цибулі: найважливіші дані надійно зберігаються в блокчейні (on-chain), тоді як менш важливі дані формуються навколо цього скелета як зовнішньої оболонки.

Але такий спосіб зберігання даних значно ускладнює застосування, оскільки фактично дані однієї сутності часто розбиваються на дві бази даних.

Я наведу вам приклад. У вас є елемент, власник якого зберігається on-chain, а його метадані зберігаються off-chain.
Як кінцевий клієнт може отримати весь об’єкт із власником і метаданими?

Є кілька варіантів вирішення цієї проблеми:

Перший варіант полягає в тому, щоб створити ендпоінт на бекенді, який надсилає запит RPC-серверу для отримання on-chain об’єкта, а потім приєднує свої метадані до нього разом з даними з бази даних. Цей варіант швидкий у розробці, але має багато недоліків, які швидко проявляються. Зокрема, RPC-запити надзвичайно повільні та неефективні, тому UX усієї програми постраждає. Крім того, більшість провайдерів RPC (навіть платних) мають досить низькі рейт-ліміти, тож у міру зростання програми ви швидко натрапите на них.

Другий варіант — передача збирання об’єкта на сторону клієнта. Клієнт робить RPC-запит, щоб отримати власника, а потім робить запит до вашого API, щоб отримати метадані. Очевидна перевага тут порівняно з першим варіантом полягає в тому, що ви не обмежені рейт-лімітами вашого RPC-провайдера. Серед недоліків — цей варіант значно ускладнює ваш фронтенд, та все ще не вирішує проблему продуктивності через те, що майже всі користувачі використовують RPC-сервери надзвичайно низької якості. Відомо, що ці сервери час від часу виходять з ладу або зависають на невизначений термін (особливо для BNB Smart Chain). Крім того, цей спосіб позбавляє вас можливості централізованого та зручного пошуку чи фільтрації товарів (наприклад, пошук у маркетплейсі).

Я спробував обидва варіанти під час розробки, і можу сказати, що жоден метод не дуже добре масштабується. Загалом вони обоє надто «костильні».
Треба якось абстрагуватися від роботи з RPC і зробити свою базу єдиним джерелом правди. Цього можна досягти, перехоплюючи всі зміни on-chain (за допомогою подій) і негайно записуючи їх у базу даних. Це серйозно спрощує подальшу розробку dApp.

Хтось може подумати: “Чому б не використати TheGraph?” У цьому випадку він може замінити лише RPC-запити. Це не вирішує фундаментальної проблеми — дані все ще розʼєднанні.

Тому я розробив модуль chain-syncer, який синхронізує бекенд з блокчейном. Щоб встановити його, використовуйте команду:

$ npm i chain-syncer @chainsyncer/mongodb-adapter ethers

Розглянемо простий приклад використання модуля. Завдання полягає в тому, щоб мати значення поточного власника елемента в базі даних.

// Example using Mongoose
// This is just an abstract example, dont try to copy-paste it

const Mongoose = require('mongoose');
const { ChainSyncer } = require('chain-syncer');
const { MongoDBAdapter } = require('@chainsyncer/mongodb-adapter');

// where will we store the pulled events
await Mongoose.connect(process.env['MONGO_SRV']);
const adapter = new MongoDBAdapter(Mongoose.connection.db);

const syncer = new ChainSyncer(adapter, { /* ... some options ... */ })

syncer.on('Items.Transfer', async (
  { global_index, from_address, block_number, block_timestamp, transaction_hash },
  from, 
  to, 
  token_id,
) => {

  // getting the item by id 
  const item = await Item.findOne({ _id: token_id });

  // updating owner address
  item.owner = to;

  // saving the item
  await item.save();

});

Що відбувається в цьому фрагменті коду? Модуль просканує всі блоки, починаючи з блоку, на якому розгорнуто контракт Items і витягне з них усі події контракту. Під час роботи сканера події, які вже були витягнуті, одночасно проходитимуть через наш обробник.
Як тільки модуль підтягує всі старі події, він починає відстежувати нові. Таким чином ми отримуємо відстеження власника кожного предмета в реальному часі.

Після обробки події на ній ставиться прапорець і вона відправляється в архів.

На даний момент існує лише адаптер для MongoDB. Ваші розробки для різноманітних інших сховищ, таких як Postgres, MySQL і OracleDB, широко вітаються.
Також зараз розробляється розширення модуля (разом із пакетом Helm), яке зробить модуль подібним до RabbitMQ. Це означає, що його буде зручно використовувати в архітектурі мікросервісу.

У наступній статті я напишу рекомендації по роботі з модулем і навіщо потрібен параметр global_index.

Друзі, не забуваємо донатити на 🇺🇦 ЗСУ. Залишу посилання для тих хто в танку.

Готуємо ChainSyncer

Miroslaw Shpak — Tue, 28 Feb 2023 18:24:33 +0000

Як і обіцяв у минулій статті, Я напишу невеликі рекомендації щодо використання модуля ChainSyncer.

Максимальна узгодженість

Через асинхронні характеристики модуля та надмірної складності RPC серверів,при взаємодії з модулем потрібні деякі хитрощі,для того щоб підтримувати узгодженність з блокчейном.

Напариклад: Ми маємо абстрактний контракт на стейкінг.

// SPDX-License-Identifier: MIT

pragma solidity 0.8.17;

/**
  * Some very-very abstract staking contract without any sense
  */
contract Staking {

  struct Pool {
    uint256 total_staked;
  }

  mapping(uint256 => Pool) pools;

  event Staked(
    uint256 indexed pool_id,
    address staked_by,
    uint256 current_total_staked
  );

  function stake(uint256 pool_id_, uint256 amount_) external {
    pools[pool_id_].total_staked += amount_;

    emit Staked(pool_id_, msg.sender, pools[pool_id_].total_staked);
  }
}

Завдання полягає в тому щоб відстежити кількість токенів у пулі в умовах високого навантаження контракту (припустимо, дуже популярного).

Робитимемо це за допомогою Staked події.

// Example using Mongoose
// This is just an abstract example, dont try to copy-paste it

const Mongoose = require('mongoose');
const { ChainSyncer } = require('chain-syncer');
const { MongoDBAdapter } = require('@chainsyncer/mongodb-adapter');

// where will we store the pulled events
await Mongoose.connect(process.env['MONGO_SRV']);

const adapter = new MongoDBAdapter(Mongoose.connection.db);
const syncer = new ChainSyncer(adapter, { /* ... some options ... */ })

syncer.on('Staking.Staked', async (
  { global_index }, // will need in the next example
  pool_id,
  staked_by, // actually dont need
  current_total_staked,
) => {

  await StakingPool.updateOne({
    _id: pool_id,
  }, {
    $set: {
      total_staked: Number(current_total_staked),
    },
  });

});

Обов'язково потрібно переконатись, що наші обробники є ідемпотентними. Це можливо тільки у тому випадку, якщо ви проєктуете події таким чином,щоб вони зберігали поточний моментальний знімок даних: у нашому випадку total_staked.Навпаки, якщо ми збережемо amount_, то нам доведеться продивитись усі події тільки для того щоб дізнатись, скільки зараз застейкано. Це дуже ускладнить код.

У наданому вище коді ховалася доволі серйозна помилка. Оскільки модуль обробляє події паралельно пакетами (для максимальної продуктивності), цілком ймовірно, що більш рання подія буде оброблена після більш пізньої. У результаті, total_staked на вашому сервері буде менше, ніж повинно бути.

Тут у гру вступає параметр global_index. З його допомогою ми можемо визначити найбільш актуальний стан total_staked пула.

Додаючи деякі правки у наш обробник, ми можемо прибрати ризики помилок синхронізації:

syncer.on('Staking.Staked', async (
  { global_index }, // we need only global_index
  pool_id,
  staked_by, // actually dont need
  current_total_staked,
) => {

  await StakingPool.updateOne({
    _id: pool_id,

    // we need to update total_staked only if global_index is greater than the last one saved
    last_synced_at: { $lt: global_index },
  }, {
    $set: {
      total_staked: Number(current_total_staked),

      // dont't forget to save
      last_synced_at: global_index,
    },
  });

});

Ви, напевно, спитаєте: Звідки береться цей параметр?
Він збирається ChainSyncer'ом для кожної події шляхом поєднання blockNumber та logIndex.

Таким чином, ми отримуємо щось на зразок додаткового унікального ID для кожної події.

Перенесення подій

Іншою функцією, призначеною для вирішення проблем асинхронності модуля, є збереження послідовності обробки подій.

Я наведу вам приклад. У вас є об’єкт меча для вашої крипто-гри, який повністю зберігається в контракті.

// SPDX-License-Identifier: MIT

pragma solidity 0.8.17;

contract Swords {

  struct Sword {
    uint256 damage;
    bool is_enchanted;
  }

  Sword[] public swords;

  event Created(
    uint256 sword_id,
    uint256 damage
  );

  event Enchanted(
    uint256 sword_id
  );

  function _create(uint256 damage_) internal returns (uint256) {

    uint256 sword_id = swords.length;
    swords.push(Sword(damage_, false));

    return sword_id;
  }

  function create(uint256 damage_) external {

    uint256 sword_id = _create(damage_);

    emit Created(sword_id, damage_);
  }


  function createAndEnchant(uint256 damage_) external {

    uint256 sword_id = _create(damage_);
    swords[sword_id].is_enchanted = true;

    emit Created(sword_id, damage_);
    emit Enchanted(sword_id);
  }
}

У нас також є backend який відстежує статус кожного меча в контракті.

syncer.on('Swords.Created', async (
  { global_index, block_timestamp }, // don't need
  sword_id,
  damage
) => {

  // uint256 that comes from event syncer converts to string, but we need a number
  sword_id = Number(sword_id);
  damage = Number(damage);

  // getting the sword by id 
  const sword = await Sword.findOne({ _id: sword_id });

  // will create a new one, if no sword found in our DB
  if(!sword) {

    await Sword.create({
      _id: sword_id,
      damage: damage,
      is_enchanted: false,
    });
  }

});


syncer.on('Swords.Enchanted', async (
  { global_index, block_timestamp }, // don't need
  sword_id,
) => {

  sword_id = Number(sword_id);

  await Sword.updateOne({
    _id: sword_id,
  }, {
    $set: {
      is_enchanted: true, // updating owner address
    }
  });

});

Як я описав вище, події обробляються паралельно, і в цьому прикладі нам потрібна чітка послідовність виконання обробників. Оскільки Enchanted і Created створюються в одній транзакції, існує надзвичайно високий ризик того, що подія Enchanted буде оброблено раніше події Created, що призведе до втрати узгодженості.

Щоб усунути ймовірність цієї помилки, ChainSyncer має цивільний механізм для відкладення подій: якщо false повертається з обробника, то обробка події відкладається на наступну ітерацію.

syncer.on('Swords.Enchanted', async (
  { global_index, block_timestamp }, // don't need
  sword_id,
) => {

  sword_id = Number(sword_id);

  // getting the sword by id 
  const sword = await Sword.findOne({ _id: sword_id });

  // if no sword found in our DB, 
  // we will postpone this event till the sword will be created
  if(!sword) {
    return false;
  }

  await Sword.updateOne({
    _id: sword_id,
  }, {
    $set: {
      is_enchanted: true, // updating owner address
    }
  });

});

Використовуючи цей механізм, ми отримуємо щось на зразок графіка залежності подій.

На даний момент це все. Використовуючи ці практики, ви отримаєте не тільки високу ефективність, а й надійну узгодженість, щоб ви могли міцно спати.

У наступній статті я розповім вам про більш глибоку роботу з модулем — сканування окремих блоків, режим сканера і, можливо, щось більше.

Друзі, не забуваємо донатити на 🇺🇦 ЗСУ. Залишу посилання для тих хто в танку.

Moralis Streams vs ChainSyncer

Miroslaw Shpak — Sun, 26 Feb 2023 19:18:15 +0000

Hey guys, welcome to my article on Moralis Streams and ChainSyncer - two tools that can make your lives easier when working with blockchain networks.

What is Moralis Streams?

Moralis Streams is a popular API for setting up Web3 streams. It's user-friendly, and it makes it easier to stream blockchain data into your application or Web3 project's backend. It's a great option for those who don't want to spend too much time building complex infrastructure to handle blockchain data.

ChainSyncer - What is it?

ChainSyncer is an open-source solution that allows you to easily set up blockchain listeners and catch on-chain events to write them immediately to your database. This is more flexible than storing all data on-chain or off-chain because it allows for a hybrid approach that simplifies development of decentralized applications.

Comparison of ChainSyncer vs Moralis Streams

Feature	ChainSyncer	Moralis Streams
Custom smart contracts listener	✅	✅
Multiple contracts	✅	✅
100% event delivery guarantee	✅	✅ (more 💰)
Listening to Wallet Addresses	❌	✅
All possible EVM chains	✅	❌
Unlimited retries	✅	❌
Internal transactions listener	❌	✅

Comparison overview

Moralis Streams is great for those who want an easy-to-use solution, but it's not ideal for those who don't want to pay for the Business Plan. ChainSyncer is open-source and free, and it provides a custom smart contract listener, multiple contracts, 100% event delivery guarantee, and the ability to connect to any EVM-compatible blockchain. Additionally, ChainSyncer allows for unlimited retries, which is particularly useful in cases where there is a high volume of events.

Another key difference between ChainSyncer and Moralis Streams is the ability to listen to wallet addresses. While Moralis Streams allows you to listen to wallet addresses, ChainSyncer does not have this feature. However, ChainSyncer is compatible with all possible EVM chains, which means that you can use it with any EVM-compatible blockchain.

Moralis Streams usage example

Here's a quick example of how to use Moralis Streams to set up a new stream and monitor wallet events on Ethereum and Polygon networks:

import Moralis from "moralis";
import { EvmChain } from "@moralisweb3/evm-utils";

Moralis.start({
  apiKey: "YOUR_API_KEY",
});

const stream = {
  chains: [EvmChain.ETHEREUM, EvmChain.POLYGON], // list of blockchains to monitor
  description: "monitor Bobs wallet", // your description
  tag: "bob", // give it a tag
  webhookUrl: "https://YOUR_WEBHOOK_URL", // webhook url to receive events,
};

const newStream = await Moralis.Streams.add(stream);
const { id } = newStream.toJSON(); // { id: 'YOUR_STREAM_ID', ...newStream }

// Now we attach bobs address to the stream
const address = "0x68b3f12d6e8d85a8d3dbbc15bba9dc5103b888a4";

await Moralis.Streams.addAddress({ address, id });

// watch the webhook (assuming you are using express)
app.post("/webhook", async (req, res) => {
  const { payload } = req.body;
  console.log(payload);
  res.sendStatus(200);
});

In this example, we're monitoring Bob's wallet by setting up a new stream with the Moralis Streams API. We specify that we want to monitor events on both the Ethereum and Polygon networks. We also give the stream a tag and a webhook URL where we can receive the events. Finally, we add Bob's address to the stream and set up a webhook to receive events.

ChainSyncer usage example

Now let's take a look at how you can use ChainSyncer to monitor on-chain events. Here's a basic example to get you started:

import { ChainSyncer, InMemoryAdapter } from 'chain-syncer';
import abi from './abi.json';

const default_adapter = new InMemoryAdapter(); // change it to any other adapter

const contracts = {
  'NFTContract': {
    abi,
    address: '0xe9e7CEA3DedcA5984780Bafc599bD69ADd087D56',
    start_block: 27118825 // scanner will start from this block
  }
}

const syncer = new ChainSyncer(default_adapter, {

  block_time: 3500,

  network_id: 97,

  rpc_url: [
    'https://data-seed-prebsc-1-s1.binance.org:8545',
    'https://data-seed-prebsc-2-s1.binance.org:8545' // will be used as a fallback
  ],

  contractsResolver: contract_name => contracts[contract_name],
});

syncer.on('NFTContract.Transfer', async (
  { block_timestamp },
  from, 
  to, 
  token_id,
) => {
  const nft = await Nft.findOne({ id: token_id });

  if(!nft) { // postpone until nft created in our DB
    return false;
  }

  nft.owner = to;
  nft.updated_at = new Date(block_timestamp * 1000);
  await nft.save();
}));

await syncer.start();

With ChainSyncer, you can monitor and interact with on-chain events in real-time, allowing you to streamline your development process and make your application more efficient.

Conclusion

To sum it up, if you're looking for an open-source and free tool to monitor and interact with on-chain events, ChainSyncer is the way to go. With ChainSyncer, you can even use a set of free RPCs to index blockchain data for absolutely free without trading reliability. This is especially useful for small/micro projects.

While Moralis Streams has some great features, it can be costly and is a centralized service. On the other hand, ChainSyncer is open-source, free, and offers more flexibility, reliability, and decentralization. By choosing ChainSyncer, you can build innovative, collaborative applications that align with the core philosophy of web3. It's important to choose tools and solutions that support decentralization and openness as we continue to build the web3 ecosystem. With ChainSyncer, you can do just that while building the next generation of decentralized applications.

There is still a war going on in Ukraine and at this moment the support of each and everyone is really important. Every dollar strongly brings Ukraine closer to victory 🇺🇦. How can you help.

ChainSyncer cookbook

Miroslaw Shpak — Sat, 11 Feb 2023 07:37:05 +0000

As promised in the last article, I will write about some best practices for using the ChainSyncer module.

Maximum Consistency

Due to the asynchronous nature of the module and the overhead of most RPC servers, some tricks are required when working with the module in order to maintain the consistency with the blockchain.

Here is an example: We have an abstract staking contract.

// SPDX-License-Identifier: MIT

pragma solidity 0.8.17;

/**
  * Some very-very abstract staking contract without any sense
  */
contract Staking {

  struct Pool {
    uint256 total_staked;
  }

  mapping(uint256 => Pool) pools;

  event Staked(
    uint256 indexed pool_id,
    address staked_by,
    uint256 current_total_staked
  );

  function stake(uint256 pool_id_, uint256 amount_) external {
    pools[pool_id_].total_staked += amount_;

    emit Staked(pool_id_, msg.sender, pools[pool_id_].total_staked);
  }
}

The task is to track the number of tokens in the pool under conditions of high contract load (let’s say it is very popular).

We will do this with the help of the Staked event.

// Example using Mongoose
// This is just an abstract example, dont try to copy-paste it

const Mongoose = require('mongoose');
const { ChainSyncer } = require('chain-syncer');
const { MongoDBAdapter } = require('@chainsyncer/mongodb-adapter');

// where will we store the pulled events
await Mongoose.connect(process.env['MONGO_SRV']);

const adapter = new MongoDBAdapter(Mongoose.connection.db);
const syncer = new ChainSyncer(adapter, { /* ... some options ... */ })

syncer.on('Staking.Staked', async (
  { global_index }, // will need in the next example
  pool_id,
  staked_by, // actually dont need
  current_total_staked,
) => {

  await StakingPool.updateOne({
    _id: pool_id,
  }, {
    $set: {
      total_staked: Number(current_total_staked),
    },
  });

});

It is important to ensure that our handlers are idempotent. This is possible only if you design events in such a way that they store the current data snapshot: in our case, total_staked. In contrast, if we store amount_, then we will have to go through all the events just to find out how much is currently staked. This will greatly complicate the code.

A rather serious bug hid in the code above. Since the module processes events in parallel in batches (for maximum performance), it is likely that an earlier event will be processed after a later event. As a result, total_staked on your backend will be less than it should be.

This is where the global_index parameter comes into play. With it, we can determine the most current total_staked state of the pool.

By making some changes to our handler, we can remove the risk synchronization errors:

syncer.on('Staking.Staked', async (
  { global_index }, // we need only global_index
  pool_id,
  staked_by, // actually dont need
  current_total_staked,
) => {

  await StakingPool.updateOne({
    _id: pool_id,

    // we need to update total_staked only if global_index is greater than the last one saved
    last_synced_at: { $lt: global_index },
  }, {
    $set: {
      total_staked: Number(current_total_staked),

      // dont't forget to save
      last_synced_at: global_index,
    },
  });

});

You are probably wondering: Where does this parameter come from?
It is collected by ChainSyncer for each event by merging blockNumber and logIndex.

Thus, we get something like an incremental unique ID for each event.

Postponement of Events

Another feature designed to help deal with module asynchrony problems is saving the sequence of event processing.

I’ll give you an example. You have a sword object for your crypto game that is stored entirely in a contract.

// SPDX-License-Identifier: MIT

pragma solidity 0.8.17;

contract Swords {

  struct Sword {
    uint256 damage;
    bool is_enchanted;
  }

  Sword[] public swords;

  event Created(
    uint256 sword_id,
    uint256 damage
  );

  event Enchanted(
    uint256 sword_id
  );

  function _create(uint256 damage_) internal returns (uint256) {

    uint256 sword_id = swords.length;
    swords.push(Sword(damage_, false));

    return sword_id;
  }

  function create(uint256 damage_) external {

    uint256 sword_id = _create(damage_);

    emit Created(sword_id, damage_);
  }


  function createAndEnchant(uint256 damage_) external {

    uint256 sword_id = _create(damage_);
    swords[sword_id].is_enchanted = true;

    emit Created(sword_id, damage_);
    emit Enchanted(sword_id);
  }
}

We also have a backend that keeps track of the status of each sword in the contract.

syncer.on('Swords.Created', async (
  { global_index, block_timestamp }, // don't need
  sword_id,
  damage
) => {

  // uint256 that comes from event syncer converts to string, but we need a number
  sword_id = Number(sword_id);
  damage = Number(damage);

  // getting the sword by id 
  const sword = await Sword.findOne({ _id: sword_id });

  // will create a new one, if no sword found in our DB
  if(!sword) {

    await Sword.create({
      _id: sword_id,
      damage: damage,
      is_enchanted: false,
    });
  }

});


syncer.on('Swords.Enchanted', async (
  { global_index, block_timestamp }, // don't need
  sword_id,
) => {

  sword_id = Number(sword_id);

  await Sword.updateOne({
    _id: sword_id,
  }, {
    $set: {
      is_enchanted: true, // updating owner address
    }
  });

});

As I described above, events are processed in parallel, and in this example, we need a clear sequence of execution of handlers. Since Enchanted and Created are thrown in the same transaction, there is an extremely high risk that the Enchanted event will be processed before the Created event, which will lead to a loss of consistency.

To eliminate the possibility of this bug, ChainSyncer has a civil mechanism for postponing events: if false is returned out of the handler, then the event processing is postponed for the next tick.

syncer.on('Swords.Enchanted', async (
  { global_index, block_timestamp }, // don't need
  sword_id,
) => {

  sword_id = Number(sword_id);

  // getting the sword by id 
  const sword = await Sword.findOne({ _id: sword_id });

  // if no sword found in our DB, 
  // we will postpone this event till the sword will be created
  if(!sword) {
    return false;
  }

  await Sword.updateOne({
    _id: sword_id,
  }, {
    $set: {
      is_enchanted: true, // updating owner address
    }
  });

});

Using this mechanism, we get something like an event dependency graph.

That’s all for now. By using these practices, you will get both high performance and reliable consistency, so you can sleep soundly.

In the next article I will tell you about more in-depth work using the module — scanning individual network blocks, scanner mode, and maybe something more.

There is still a war going on in Ukraine and at this moment the support of each and everyone is really important. Every dollar strongly brings Ukraine closer to victory 🇺🇦. How can you help.

Building Off-chain dApps with ChainSyncer

Miroslaw Shpak — Fri, 10 Feb 2023 21:30:55 +0000

The core idea of Web3 is based on decentralized data storage. Ideally, all your data could be stored in a decentralized manner, but that’s not possible — storing all data like this would be extremely expensive (at least nowadays). Instead, it’s advisable to use decentralized storage for only the most important data.

Where to store the rest of the data, then?
— Off-chain. That is, on your backend.

With this approach, you get something like an onion: the most important data is stored securely in the blockchain (on-chain), while less important data is molded around this skeleton as an outer shell.

But this way of storing data greatly complicates the application, since in fact the data of one entity is often divided into two databases.

I’ll give you an example. You have an item that stores its owner on-chain, and stores its metadata off-chain.
How can the end client get the whole object with the owner and metadata?

There are several options for solving this problem:

The first option is to make an endpoint on the backend that makes a request to the RPC server to receive an on-chain object, and then attaches its metadata to it with data from the database. This option is developed quickly, but has a lot of shortcomings that present themselves quickly, too. Namely, RPC requests are extremely slow and inefficient, so the UX of the entire application will suffer. Additionally, most RPC providers (even paid ones) have rather low rate limits, so as the application grows, you’ll quickly run into them.

The second option is to transfer the assembly of the object to the client’s side. The client makes an RPC request to get the owner, then makes a request to your API to get the metadata. The obvious advantage here, compared to the first option, is that you aren’t bottlenecked by rate limits. Among the disadvantages — this option greatly complicates your frontend, and still doesn’t solve the performance problem due to the fact that almost all users use extremely low-quality RPC servers. These servers are known to crash from time to time or freeze indefinitely (especially for BSC). Moreover, this method deprives you of the possibility of a centralized and convenient items search or filtering (for example, searching in the marketplace).

I tried both options during development, and I can say that neither method scales very well. They’re both too “hacky” in general.
It’s necessary to somehow abstract from working with RPC and make my database the only source of truth. This can be achieved by catching all on-chain changes to an item (using events) and immediately writing them to the database. This seriously simplifies further development of the dApp.

Some might think, “why not use TheGraph?” In this case, it can only replace RPC requests. It doesn’t solve the fundamental problem — data is still scattered.

So I developed chain-syncer, a module that synchronizes the backend with the blockchain. To install it, use the command:

$ npm i chain-syncer @chainsyncer/mongodb-adapter ethers@^6.0.0

Let’s consider a simple example of using the module. The task is to have the value of the current owner of the item in the database.

// Example using Mongoose
// This is just an abstract example, dont try to copy-paste it

const Mongoose = require('mongoose');
const { ChainSyncer } = require('chain-syncer');
const { MongoDBAdapter } = require('@chainsyncer/mongodb-adapter');

// where will we store the pulled events
await Mongoose.connect(process.env['MONGO_SRV']);
const adapter = new MongoDBAdapter(Mongoose.connection.db);

const syncer = new ChainSyncer(adapter, { /* ... some options ... */ })

syncer.on('Items.Transfer', async (
  { global_index, from_address, block_number, block_timestamp, transaction_hash },
  from, 
  to, 
  token_id,
) => {

  // getting the item by id 
  const item = await Item.findOne({ _id: token_id });

  // updating owner address
  item.owner = to;

  // saving the item
  await item.save();

});

What happens in this piece of code? The module will scan all blocks, starting from the block on which the Items contract is deployed, and pull all the contract events from them. While the scanner is running, the events that have already been drawn will simultaneously go through our handler.
As soon as the module pulls up all the old events, it starts tracking new ones. Thus, we get real-time tracking of the owner of each item.

How it works: chain-syncer has 2 loops. The first cycle keeps track of new contract events, and makes sure that there are no duplicates and that no events are missed. The second loop parses the queue of unprocessed events.

After the event is processed, a flag is placed on it, and it’s sent to the archive.

At the moment, there’s only an adapter for MongoDB. Your developments for a variety of other storages, such as Postgres, MySQL and OracleDB, are widely welcomed.
There’s also a module extension in development now (along with the Helm package) that will make the module work similarly to RabbitMQ. This means that it’s convenient to use in a microservice architecture.

In the next article, I’ll write recommendations for working with the module, and why the global_index parameter is needed.

There is still a war going on in Ukraine and at this moment the support of each and everyone is really important. Every dollar strongly brings Ukraine closer to victory 🇺🇦. How can you help.