DEV Community: EQ LAB

Maximizing Performance: Optimization Techniques for Substrate Blockchains

EQ LAB — Wed, 02 Aug 2023 08:39:14 +0000

In the decentralized world, optimization is crucial to enhance the efficiency and scalability of various systems. From distributed computing to network protocols, we continually strive to maximize performance and improve user experiences. The blockchain, with its promise of transparent, secure, and decentralized transactions, is no exception. In this article, we'll analyze optimization techniques tailored for Substrate blockchains, shedding light on practical tips to elevate the performance of these innovative platforms.

Before we explore the intricacies of optimization in the blockchain space, let us first establish its significance in the broader context of decentralization.

Why We Optimize Systems

Optimization empowers us to streamline processes, minimize resource consumption, and increase system efficiency. It enables us to overcome challenges in scalability, latency, and transaction throughput. Through efficient algorithms, leveraging parallel processing, and optimizing data structures, we can realize the true potential of decentralized systems.

Transitioning our focus to Substrate blockchains, we unravel unique characteristics and challenges they present to us. As a modular framework for building blockchain solutions, Substrate provides a rich set of tools and functionalities. However, achieving optimal performance on Substrate blockchains requires a deep understanding of the underlying technology and thoughtful consideration of design choices. At the end of the day, a faster blockchain means a better product and more possible design choices.

Armed with this understanding, we embark on a practical journey, presenting a range of optimization techniques tailored for Substrate blockchains. By optimizing storage and implementing caching strategies, we provide actionable insights to improve the performance and responsiveness of Substrate-based networks.

Approaches to Optimization in a Centralized World

There are several approaches to optimization in the centralized world. Let's explore these strategies in more detail:

Scaling: Scaling involves expanding the system's capacity to handle a larger transaction volume. There are two main scaling techniques:

Horizontal Scaling: This approach involves adding extra servers to the network, effectively distributing the transaction load among multiple nodes. We can accommodate a growing number of users and transactions with horizontal scaling.
Vertical Scaling: Vertical scaling focuses on enhancing the capabilities of existing servers by upgrading hardware components such as CPU, memory, or storage, allowing us to improve performance by increasing available computational resources.
Choosing the Best Instruments: Selecting a programming language, databases, and message queues like Rabbit MQ affects the efficiency of applications. Opting for technologies well-suited to the project enables us to improve performance and streamline development.
Parallelization and Asynchrony: Leveraging parallel processing and asynchronous programming techniques allows tasks to be executed concurrently in a multithreaded environment, enhancing overall efficiency.
Database Optimization: Optimizing the database architecture and configuration is essential for efficient data storage and retrieval. We can reduce latency and enhance query processing speed through techniques such as indexing, sharding, and data compression.
Code Optimization: Choosing the most effective algorithms and data structures for implementing various functionalities within the application is crucial for streamlined functionality. Optimized code enhances the efficiency and responsiveness of the system, resulting in improved overall performance.
Caching Frequently Used Data and Query Results: We avoid redundant computations and minimize the load by storing frequently used data in a cache.

Incorporating these optimization techniques into the development process enhances performance, scalability, and user experience. Leveraging the power of optimization, we meet market demands by delivering reliable systems capable of scaling up with a growing market.

Having understood the streamlining options in centralized systems, let's explore how we can improve blockchain performance for Substrate-based platforms.

Optimizing Substrate Development for Blockchain Performance

When optimizing Substrate development, it's crucial to understand the differentiating factors between Substrate and traditional backend systems. A significant distinction is the access to storage speeds.

In traditional development, memory calls are often made with little consideration for optimization. However, in Substrate, every memory call is expensive. Additionally, Substrate imposes certain limitations on the choice of tools, parallelism, and caching options.

Below is a table explaining the theoretical approaches to optimize blockchain performance within the context of Substrate:

Although certain limitations exist, we can use caching management and reducing storage calls to achieve notable performance improvement within the Substrate framework.

So, from these theoretical examples, what are the practical steps we can take to optimize Substrate blockchains?

Practical Approaches for Optimizing Substrate Blockchains

Before proceeding, let's first comprehend how caching works in the Substrate framework:

Quick Access: Accessing cached data is much faster than accessing data from storage.
Cache Population: Data is written to the cache when it's first called from storage.
Least Used Object Removal: The least used object will be removed if the cache becomes full to accommodate new entries.

Let's now examine practical ways to optimize Substrate blockchains.

Utilize Constants: Storing parameters in constants is preferable to pallet storage for customized pallets. Constants aren't kept in storage and therefore don't require resources to read from storage. Although modifying a constant requires updating the chain, the performance improvement is significant enough to outweigh this drawback.
Do not iterate on HashMap: EVM developers who transitioned to Substrate frequently use HashMap to store data. Iterations can be costly, and they don't take advantage of the cache. To address this, we at Equilibrium utilize VecMap - a vector data storage that combines the benefits of a Map interface with high speed.
Storing Pallet Parameters: We prioritize a minimal number of data structures and encourage the creation of large ones. It has a negligible impact on performance while reducing the amount of memory calls notably improves efficiency. In our experience, reading/writing a vector with 1k elements is comparable to a single element. Minimizing the number of read/write operations is essential for optimal performance.
Storing Dynamic Data: Let's look at assets as an example. If our chain doesn't allow users to create new assets, and we often need to access parameters such as price, we can store these parameters in a VecMap instead of a HashMap. This way, the asset will be stored in the cache most of the time, resulting in faster transfers.
Using System Account Data: Before executing an extrinsic, Substrate performs two essential tasks: it checks the nonce and deducts the transaction fee. This action consolidates the System storage into the Account and caches the balances storage. In the cases of Polkadot and Kusama, these two storages are merged, enabling a single read-and-write operation to handle both functions. Expanding the account data provides a suitable place to store frequently used account parameters such as scores, ratings, and nicknames. By leveraging this approach, we can avoid excessive interactions with storage and optimize performance.
Utilize Off-Chain Workers: Off-chain workers, a powerful Substrate feature, contribute to optimization efforts. They allow delegating heavy tasks to be performed off-chain and only verify the result on-chain. At Equilibrium, we apply this approach to manage margin calls. We utilize off-chain workers to monitor accounts and verify margin calls off-chain. We eliminate the need to check all accounts on-chain, enhancing overall performance.
Conduct Extensive Stress Testing: To ensure the optimal performance of our blockchain, we understand the importance of thorough stress testing. While benchmarks provide a preliminary understanding of how complex an extrinsic is, they don't consider critical factors like caching and storage size, which are essential for performance optimization. That's why we subject our system to rigorous stress testing.

We develop a deeper understanding of the actual duration of transactions and identify potential areas for improvement through the stress tests. They allow us to fine-tune our system, optimizing it for enhanced performance.

Additional Blockchain Optimization Ideas

In addition to the previous optimization techniques, let's mention a couple ideas to further enhance the performance of Substrate blockchains.

-Custom Caching: One way to optimize performance is by implementing custom caching, which involves preloading data when adding an extrinsic (a transaction) to the pool. We can often determine the necessary data and efficiently preload it into the cache by analyzing the extrinsic’s arguments, ensuring faster access.
-Deparallelizing data access: Currently, the parallelization implementation in Substrate does not allow storage calls and is thus largely ineffective. For Substrate technology to be competitive with, for example, payments giants, deparallelizing data access is mandatory.

These practical optimization techniques for Substrate blockchains allow us to improve system performance. We continue to explore and experiment with optimization techniques to realize the full potential of Substrate-based blockchain systems.

How IBC and XCM Drive Interoperability In Polkadot and Beyond

EQ LAB — Wed, 02 Aug 2023 08:25:13 +0000

As blockchain technology continues to evolve, achieving seamless communication and interoperability between different chains has become a crucial objective. In this article, we delve into two prominent protocols - Inter-Blockchain Communication (IBC) and Cross-Consensus Messaging (XCM) - that enable secure, permissionless, and trustless transfer of data between blockchains. We will explore both IBC and XCM, their functionality, features, and potential use cases.

At its core, IBC is an interoperability protocol that facilitates the transfer of data, assets, and logic between distinct blockchains. The protocol operates on the principles of simplicity, flexibility, and decentralization, offering a powerful solution for connecting blockchain networks.

Three abstraction layers: channels, connections, clients. Channels to identify application, connections for double identity verification, and clients to track consensus state of recipient chains.

When we send a packet on one chain we will submit a commitment proof, it will be recorded into the state. Then a relayer will construct a packet and add a proof that this was submitted into state and then submit this on the receiving chain which can then verify it using the Light Client of chain A that they have stored on chain B.

There are different Light Clients per consensus type. IBC's scope has expanded beyond the Tendermint consensus chains, thanks to the efforts of projects like Composable, which now support Substrate-based chains.

On top of clients there are connections. They operate once clients have been set up. It is a double identity verification. Both channels and connections are set up by handshakes.

The last layer of abstraction is channels. They are also set up with a handshake. In combination with the port they identify the application. Channel for a fungible token transfer over IBC could look like transfer/channel-0/base-denom where ‘transfer’ is the port name.

Relaying is permissionless and can be incentivized. IBC v4 and onwards contains fee middleware. Two implementations of relayer software available: one is Hermes developed in Rust and another is Go Relayer.

Let’s look at the IBC packet lifecycle using ICS-20 fungible token transfer standard.

Tokens on the left side are locked up by a user on Chain A, and after the tokens are locked, the packet is sent. The Light Client verification algorithm on Chain B checks whether the tokens on Chain A have been locked and mints IBC vouchers representing tokens of Chain A on Chain B.

Here’s a more generalized packet flow visualization:

This packet flow representation can be applied not only to fungible tokens but also to any data packets. Flow can be done in two different ways: by using interchain accounts or via custom applications.

Let’s look at custom IBC applications first.

Custom IBC applications allow for the development of completely customizable data transfers between chains. The exact requirements for module application to be IBC compatible can be found in the documentation. Specific application logic can then be added to such IBC compatible modules. Any data can be sent to the corresponding counterparty module. ICS-x means that the token will have a number instead of the ‘x’ which refers to a particular application.

Let’s now examine interchain accounts.

The interchain accounts option allows the use of functionality on Chain B from Chain A without leaving Chain A's environment.

Chains can implement both or either controller or host functionality. Host chain hosts ICA, executes sent txs. Controller chain triggers actions to be executed on the Host chain.

Let’s now explore a scenario where the relayer didn’t pick up the packet.

To avoid funds getting locked up forever a timeout functionality is implemented. The cool thing about IBC is it allows you to query not only receipts but also non-receipts. Such a query allows the user to submit a timeout command and revert the original send packet logic.

ICA have ordered channels. If such a channel has a timeout it is closed automatically.

Functionality that doesn't go in the IBC core can be used as middleware and reused for different applications thus building a middleware stack.

This diagram shows that we can keep stacking middleware and create functionality that way. Here packet and write_ack go from base application to Core IBC. Handshake, acks and timeouts go from Core IBC through the middleware to the base application. Order matters and must be replicated on the counterparty chain.

XCM is a cross-chain messaging protocol that enables the transfer of data and assets between chains in a more sophisticated and programmable way than ever before. XCM V3 includes improvements in three key areas: expectations and branching, introspection and safe dispatches, and asset exchange and NFTs.

Let’s now examine expectations and branching in more detail.

There are three new instructions: ExpectAsset, ExpectError and ExpectOrigin. They are related to three existing registers: Holding, Error, Origin. This allows for branching: changing the flow depending on errors thrown.

Safe dispatches allow for version control of destination chain and specific pallet. There is a new register called ‘Transact Status Register’ and four new instructions: QueryPallet, ExpectPallet, ReportTransactStatus, ClearTransactStatus. The new register holds the result of the most recent transact operation. QueryPallet gives back the instances of a specific pallet or module. ExpectPallet throws an error if the expected version does not match the one we get back. ReportTransactStatus reports what the Transact Status Register holds and ClearTransactStatus clears the data.

The goal of these statuses is to ensure no unexpected behavior is happening.

The second big category introduced in XCM v3 is Functional Multichain Decomposition. It is designed to allow for building utilizing logic from multiple chains. The changes here can be grouped into three categories:

Remote Locking
Context/ID for tracking messages & queries
Asset Namespacing

Remote Locking enables chains to use assets in other chains. Four new instructions here:

LockAsset
UnlockAsset
NoteUnlockable
RequestUnlock

One of the goals for Functional Multichain Decomposition is to enable tracking XCM messages. A new register ‘Topic’ is introduced. It can be set to any value and used as an ID by itself or in combination with ‘Origin’ to create a unique identifier.

There is a way to let ExmExecutor know about the context of the instruction using the XcmContext struct. It has three fields: Origin, XcmHash, Topic. This allows to track XCM message’s context and thus have an application spread between different shards.

The third big area of note is bridging. The four key improvements here are:

Universal Location
Message Exporting
Two stage Send and Export
Logical Origins

Universal Location is a new and unique location, a parent of all locations (different consensus systems). It encompasses relay chains and parachains. It has no parent and allows to use ‘Context’ for locating within the Universal Location.

There are three new SendXcm impls to manage routing over a bridge. They cover three scenarios:

The bridge is local
The bridge is in Local consensus but not on local chain (no fees expected)
The bridge is not local (fees expected)

Two stage Send and Export means that SendXcm is now a two-stage process: validate and deliver. Validate does price discovery and returns a ticket. Deliver executes the transaction.

The rise of IBC and XCM protocols represents significant milestones in achieving seamless communication and interoperability across blockchain networks. Through their secure, permissionless, and trustless nature, IBC and XCM empower developers to create decentralized applications that transcend the boundaries of individual chains. By enabling data transfer, asset exchange, and sophisticated programmability, these protocols pave the way for the next generation of blockchain innovation, facilitating collaboration, scalability, and enhanced user experiences in the decentralized ecosystem.

How Binary Heaps Are Utilized In A Leveraged Trading Protocol On EVM

EQ LAB — Wed, 02 Aug 2023 08:17:19 +0000

EQ Lab develops a variety of different products on multiple blockchain platforms. Among others, there is am EVM-based decentralized lending protocol that allows users to take up to 20x leveraged long and short positions on crypto assets across different DEX-es and AMMs. In this article we will explore how this product utilizes binary heaps to handle margin calls.

Pools

The protocol is comprised of liquidity pools. Each pool consists of a quote asset (stablecoin) and a base asset (risk asset). For each pool there is a corresponding AMM pool where quote and base assets can be exchanged. Lending pools allow to deposit base or quote asset, go long or short (borrow the other asset with up to 20x leverage), withdraw base or quote asset, and close long/short positions.

Four indexes are calculated for each pool. They are baseCollateralCoeff, baseDebtCoeff, quoteCollateralCoeff, quoteDebtCoeff. These are used to determine interest rates.

Positions

There are three types of positions: Lend, Long, and Short When a user creates any of these positions a discounted balance is stored instead of the “real” balance. This is how it is calculated:

discountedBalance = realBalance / coeff

There are separate coefficients for Lend, Long and Short positions. For example if a Lend position is created by depositing quote asset the following calculation applies:

discountedQuoteAmount = realAmount / quoteCollateralCoeff

The calculation for Short and Long positions is as follows:

discountedBaseAmount = realAmount / quoteDebtCoeff

Short and Long positions are then stored in two separate collections. They are sorted in descending order of debtAmount/collateralAmount. For short positions this is calculated as baseAmount/quoteAmount, and for long positions the calculation is quoteAmount/baseAmount.

These sorted positions are stored in a Max binary heap.

Any user action that changes a position’s quoteAmount/baseAmount ratio triggers a recalculation of the binary heap such that the first element is always the riskiest position.

Interest rates

Interest rates are scaled proportionally to asset volatility and leverage. Total long leverage and total short leverage is calculated for each pool. Users who take up long or short positions pay interest fees to collateral providers. Fee accrual is done by changing baseCollateralCoeff, quoteCollateralCoeff coefficients. Fee write-off occurs by modifying the baseDebtCoeff and quoteDebtCoeff.

Margin calls

User action triggers a heap check of long and short positions leverages. Leverage is calculated for the riskiest position (heap root). It is then compared to the max leverage pool parameter. If the position’s leverage exceeds max leverage, the position is liquidated. In this case collateral is swapped and used to cover liabilities. The rest is then distributed between lenders. If there is not enough collateral to settle, the lenders cover the difference. These actions are performed by modifying baseCollateralCoeff or quoteCollateralCoeff. This way individual balances don’t need to be changed.

This design allows to run an automated liquidator as well as potentially implement no-liquidation systems in future versions of the protocol. It is more capital efficient compared to designs that implement collateral, debt and balances positions for each user because liquidity can be reused: the same capital can be used to go both long and short simultaneously. This could ultimately result in a superior product and user experience.

Proposed Solution For Ledger Integration Compatible With All Polkadot Parachains

EQ LAB — Sun, 19 Mar 2023 20:51:45 +0000

In this article, we’ll discuss the issues of Ledger integration by Polkadot parachains and present a possible solution. Ledger integration with Polkadot parachains presents numerous opportunities for users yet requires each parachain to implement the support. To enhance the user experience, we've proposed the development of one app implemented for use across multiple parachains using a specific signature standard - similar to Keplr in Cosmos or Metamask with EVM-based blockchains.

Downsides of custom Ledger app integration by parachains

The integration of custom Ledger apps by parachains presents various challenges.

Ledger developers are busy, which leads to extended response times.
Each parachain features a unique account resulting in a special derivation path for each app. The implementation makes the application experience non-intuitive and confusing.
Additionally, using Ledger to participate in crowdloans requires exporting mnemonics to less secure environments, which is time-consuming and unsupported by all projects.
Derivation path selection functionality requires developers to add a new option for every new parachain, which complicates the user experience.
Finally, parachains do not have a custom Ledger app, forcing users to rely on less secure browser extensions and mobile apps. The app acts as a decoder with hard-coded pallet numbers and methods, which, combined with long response times, can cause users to lose access to funds for months.

To provide a clearer picture of the challenges described above, the table below summarizes the issues and their impact on the integration of custom Ledger apps by parachains.

Proposed solution

A viable solution is eliminating calldata decoding on the device to address the issues mentioned above. Decoding calldata for each transaction signature is the primary reason each chain requires a unique app, as exemplified by the Ethereum app decoding transfer transactions, but not most others. However, with Ledger, users can sign transactions in BLIND SIGNING mode without decoding them. Users can verify transaction data in Metamask and then sign the transaction in Ledger, providing an improved compromise between security and user experience for the entire ecosystem.

In contrast, Polkadot apps currently decode calldata in browser extensions and mobile apps. As a result, it is possible to show the transaction hash on Ledger and the extension/app to ensure that the correct transaction is signed on Ledger. To enhance security, we recommend displaying the parachain's Genesis hash. If the parachain is on the list of known chains, the name of the parachain can be displayed instead of the Genesis hash.

This solution is a balanced approach that improves security and user experience for the entire ecosystem.

Demonstration

Let’s now examine our app demo to highlight how the proposed solution could work.

First, we need to open Substrate app on Ledger.

We’ll then run the demo script.

Initial logs:

Getting information from Ledger
App version: {
"return_code": 36864,
"error_message": "No errors",
"test_mode": false,
"major": 0,
"minor": 1,
"patch": 0,
"deviceLocked": false,
"target_id": "31100004"
}
Ledger account: {
"pubKey": "d8a1f451162e6163488001efd01ecc0f9db7494834664b34d0fd225d67f0b72d",
"address": "15u3TJt3hc2oevVZi6YToyjdaymttFscQEoQNLn5CgLRvNtL",
"return_code": 36864,
"error_message": "No errors"
}

Signing transaction on Equilibrium parachain demonstration

Signer ss58 address: cg817PN4M6ZkNJ7Epx8QyqxR7Cb3JxYaZrEHyqSZQMcFXhzJy

Tx hash: 0x57e11ae24382d340e5195def4ff8c57da3c9f1645e61225870e9c713c6a7220d

Let’s compare the details with Ledger in order to verify the transaction.

Here’s the transaction hash display on Ledger:

The app detected the addition of Equilibrium Genesis hash to the list of known chains and printed out the chain name:

We’ll then approve transaction in Ledger and return to the console.

Ledger response: {
"return_code": 36864,
"error_message": "No errors",
"signature": "00cc82b71a16f1d41dc5c0d390e606762d2a9a6f0bdcaf66e59127a1c1917958dc403882a88a5d4ff5429b08b700a8da947f9d2e25270f23f960b222a4998d8707"
}
Signature is valid: true

Signed tx as bytes: 0x85028400d8a1f451162e6163488001efd01ecc0f9db7494834664b34d0fd225d67f0b72d00cc82b71a16f1d41dc5c0d390e606762d2a9a6f0bdcaf66e59127a1c1917958dc403882a88a5d4ff5429b08b700a8da947f9d2e25270f23f960b222a4998d8707520014000f0071650000000000004c388c1b04512ee6dd3afd3355fe0498f55c57773d1f4862bdf6aa27d12e387f00ca9a3b000000000000000000000000
Signed tx as json: {
"isSigned": true,
"method": {
"args": {
"asset": "25,969",
"to": "cg4q1NRDwhj1ocFmfnJubbSxekchEdDXBc95KCQxHdzfVcBtL",
"value": "1,000,000,000"
},
"method": "transfer",
"section": "eqBalances"
},
"era": {
"MortalEra": {
"period": "8",
"phase": "5"
}
},
"nonce": "5",
"signature": "0xcc82b71a16f1d41dc5c0d390e606762d2a9a6f0bdcaf66e59127a1c1917958dc403882a88a5d4ff5429b08b700a8da947f9d2e25270f23f960b222a4998d8707",
"signer": {
"Id": "cg817PN4M6ZkNJ7Epx8QyqxR7Cb3JxYaZrEHyqSZQMcFXhzJy"
},
"tip": "0"
}

We can review the signed transaction on chain after sending it.

Submitted tx hash: 0x7ca2d7464003d7f561b29411c286c831607ad9d98f190329b408a629f91be99e
Subscan link: https://equilibrium.subscan.io/extrinsic/0x7ca2d7464003d7f561b29411c286c831607ad9d98f190329b408a629f91be99e

Conclusion

Ledger integration can be a drawn-out and tedious process. Our design for a common Ledger app is a compromise between security and user experience that could decrease the barrier to Ledger integration for both new and existing Polkadot parachains.

For more detailed information please refer to the app, demo, and example which includes Equilibrium, Bifrost, and Khala transactions.

Solidity Source Mapping

EQ LAB — Sun, 29 Jan 2023 16:42:22 +0000

1. Intro

Contrary to its commonly-praised simplicity, Solidity is no joke. Don’t be deceived by its outward resemblance with general-purpose programming languages: Solidity and Ethereum run-time environment itself are fraught with many ‘traps and pitfalls’, which, being stumbled upon, can greatly increase the time for debugging.

Here are just some of the tricky peculiarities:

There are several different types of memory available to the contract;
Turing incompleteness (although there are almost no restrictions compared to Bitcoin)
The basic data type is a 256-bit integer (32 bytes);
The byte order is big-endian, unlike in x86, where it is little-endian;
The solidity compiler may generate an error without specifying a line or a source file which it could not compile;
Some seemingly innocuous and concise language constructs can unfold into a large number of inefficient bytecode.

In this post we delve deeper into the latter intricacy and will pin down what exactly the compiler turns solidity code into. For such purposes, disassemblers and decompilers are commonly used.

Using a disassembler on its own isn’t trivial. You need to be well versed in the instructions of the Ethereum virtual machine, bear in mind an insane amount of facts and precedents, mentally perform stack operations, and understand the nuances of Ethereum inner workings. Searching for compilation results for code sections of interest in the assembler commands listing is a long and exhausting task. Here the decompiler can certainly come in handy.

Decompilers accept byte-code as an input and restore the high-level code in accordance with a specific set of heuristics. With such code at hand it’s easier to compare blocks of source code with assembler instructions. The decompiler requires only bytecode to work, if any. It can also retrieve the bytecode by itself, if the working network and the contract address are specified.

However, the output code of the decompiler is very different from the original one. There are several reasons for that. First of all, there aren’t yet any well-developed decompilers for Solidity, which implies that you need to be ready for all sorts of bugs and glitches. Moreover, as source-code variable names are often lost during compiling, the decompiler is forced to substitute them with its own, devoid of any meaning (a, b, c, d, ... or var1, var2, var3, ...). Such dissimilarities lead to the need for another time-consuming source code mapping and manual collation.

Thus, it takes a lot of time and effort to figure out what several problem lines of code on solidity turn into. However, there is a better way.

The most popular framework for working with Solidity code is Truffle. When compiling, in addition to the bytecode of the contract, it generates plenty of other useful data. And it is this information which will help us in our endeavours!

Using this data we can form a comparison of the source code and the instructions in two convenient views at once: either matching a source-code to the instructions or the other way around. Now, let’s look at a simplified representation of “instruction — code fragment”.

2. Mapping instructions with code fragments

After a successful contract compilation, Truffle writes a set of json-files to the build/contracts subdirectory of the project. Each of these files corresponds to a source-code file in Solidity and contains the following information:

Contract bytecode;
Abi (the information required to call the methods of the contract and refer to its fields);
Contract source code;
Source map;
Abstract syntax tree;
Contract address after deploying to the network.

For our purposes, we will consider Contract source code, Contract bytecode and Source map. While the first one is more or less obvious, the other two need to be discussed in more detail. Let's start by looking at the Source map.

2.1. Source Map

The purpose of the Source map is to match each contract instruction to the section of the source code from which it was generated. To make the matching possible, the address of the source code fragment is stored in the source map for every instruction. Simple compression is used to save storage space: certain address fields are omitted if they are the same as in the previous instruction.

Now let's look at the Source map format. The delimiters are semicolon and colon: the former is used between the source map elements, and the latter — between the fields in the element. The field consists of three main elements determining the address of the corresponding fragment of the source code (in order of succession: offset from the beginning of the file, fragment length, file identifier). The fourth element is the type of jump instruction, but it isn’t important for source mapping.

The figure above shows the beginning of the source map, which contains information for 5 instructions. While the instr1 element contains all fields, the instr2 and instr3 are empty, which means that the address of the source code fragment for the second and the third instructions in the contract bytecode is the same as for instr1. In the instr4 only the first two fields are set, indicating that only the offset and length have changed, with the file ID and the type of transition instruction unchanged. In the instr5 there are three fields, so only the type of transition instruction remains the same as in instr1. A value of -1 in the file ID field indicates that the instruction does not match any of the user source files.

To sort through the code parsing, first let’s break down the Source map into elements:

const items = sourceMap.split(';');

Now go through them, storing current field values in the accumulator in case the following elements will lack the data in one or more fields. Even though, at first glance, it seems like the Array.prototype.reduce() function can be of use, there are better options. It is inconvenient to use, for you need to take care of saving the state of the accumulator and aggregating the results every step of the way. The scan() function is arguably better suited for our purposes:

function scan(arr, reducer, initialValue) {
let accumulator = initialValue;

const result = [];

for (const currValue of arr) {
const curr = reducer(accumulator, currValue);
accumulator = curr;
result.push(curr); }

return result;
}

It stores the accumulator values at each step by design, which is exactly what we need in this case. With the help of **scan() **it is now easy to process all the Source map elements:

scan(items, sourceMapReducer, [0, 0, 0, '-', -1]);

The sourceMapReducer() function splits each Source map element into separate fields. At the same time, we save the record number in the Source map, as it will be needed later. If a value is missing, the respective one from the accumulator is taken.

function sourceMapReducer(acc, curr) {

const parts = curr.split(':');

const fullParts = [];

for (let i = 0; i < 4; i++) {

const fullPart = parts[i] ? parts[i] : acc[i];

fullParts.push(fullPart);

}

const newAcc = [

parseInt(fullParts[0]),

parseInt(fullParts[1]),

parseInt(fullParts[2]),

fullParts[3],

acc[4] + 1

];

return newAcc;

}

2.2. Contract bytecode

In json, bytecode is represented as a huge number in hexadecimal notation.

The EVM (Ethereum Virtual Machine) bytecode format is quite simple. All commands fit into one byte, with only exception being the instructions which load constants onto the stack. The size of the constant can vary from 1 byte to 32, and it is encoded in the first byte of the instruction. The remaining bytes contain a constant.

Bytecode parsing can be narrowed down to a sequential byte reading of a hexadecimal number. If the byte read is a constant loading instruction, we determine the number of bytes and immediately read the whole constant.

function parseBytesToInstructions(bytes) {
const instructions = [];

let i = 0;
while (i < bytes.length) {
const byte = bytes[i];

let instruction = [];
if (firstPushInstructionCode <= byte && byte <= lastPushInstructionCode) {
const pushDataLength = byte - firstPushInstructionCode + 1;
const pushDataStart = i + 1;
const pushData = bytes.slice(pushDataStart, pushDataStart + pushDataLength);

instruction = [byte, ...pushData];
i += pushDataLength + 1;
} else {
instruction = [byte];
i++;
}

instructions.push(instruction);
}

return instructions;
}

2.3. Aggregating the data collected

Now we have all the information required to form the first representation: an array with disassembled contract instructions, and another array with data sufficient to determine the corresponding block of source code for each instruction. Therefore, to create the “instruction — code fragment” representation, we can simply go through these two arrays simultaneously “gluing together” information from them elementwise.

3. Mapping code snippets with instructions

It may seem that by constructing the “instruction — source code” representation in the previous section, we killed two birds with one stone and there’s nothing left to do to form the “source code — instruction” representation. This would be true, indeed, if a single instruction in the source map would match exactly one line of source code. Unfortunately, that’s not the case.

One instruction can be assigned to several lines in the source code file at once. Moreover, instructions can refer to the whole method body or even to the whole contract! Quite often a hundred consecutive instructions are tied to the entire code of the one contract. The following bunch of instructions, quite as big as the previous one, is tied to some method of the contract, and the next bunch — to the whole contract again. How useful such a flow with loads of duplicate lines of source code can be is very much in question.

The better solution is to build a tree structure from the blocks of code, integrating the smaller ones (contract methods) into large blocks (the contract itself). Then to each node of this tree we can tie all the corresponding instructions. The data structure like that is way easier to analyze.

3.1. Building a tree

We will build a structure step by step, iterating through all contract instructions and adding them one by one to the tree. Two blocks of source code can be in one of the following relationships:

blocks are equal
the first block contains the second
the first block is included in the second
blocks intersect
blocks do not intersect

Now, when we khow to define the relationship between the two blocks, we can build our tree with the addNode() function. It finds the right place for a new block in the tree, recursively scanning it starting from the root. Let's look at a simplified version this function:

function addNode(currentNode, block) {

const relation = classifyBlock(currentNode.block, block);

outer:

switch (relation) {

case equalBlock:

// do nothing

break;

case childBlock:

const [children, nonChildren] = splitChildren(currentNode.children, block);

if (children) {

currentNode.children = [

...nonChildren,

createNode(block, children),

];

} else {

for (const childNode of currentNode.children) {

const childRelation = classifyBlock(childNode.block, block);

switch (childRelation) {

case childBlock:

addNode(childNode, block);

break outer;

case neighborhoodBlock:

continue;

default:

throw new Error('Unknown relation');

}

currentNode.children.add(createNode(block));

}

break;

default:

throw new Error('Unknown relation');

}

First, we define the relation of the added block to the block of the current node in the tree (line 2). If the blocks are equal, it means that the block already exists in the tree and no further action is required (lines 6-8).

Where it gets interesting is when the block under consideration is a child of the block of the current node (lines 7-31). Here we need to consider three possible scenarios: The new block is the parent for some child blocks of the current node (lines 12-15).

A new block is a child of one of the child blocks of the current node (lines 17-28)

The new block is a direct child of the current node (line 29)

In the first and third case we rebuild the immediate children of the current node accordingly. However, in the second scenario we need to recursively go one level down.

Now let’s consider an example. Suppose the body of our contract consists of 6 instructions n1-n6, which correspond to the lines of the source code as follows:

n1: (1;15)
n2: (1;15)
n3: (1;7)
n4: (8;15)
n5: (3;3)
n6: (4;4)

The first instruction is n1, and the corresponding block (1; 15) becomes the root of our tree structure:

The n2 instruction has the same block as the previous one, so the tree doesn’t change:

The block (1; 7) of the instruction n3 is nested in (1; 15), so we rebuild the structure:

The block (8; 15) is also embedded in the root element of the tree (1; 15). However, this block is neither a parent for (1; 7), nor a child, so the tree gets this form:

The block (3; 3) is nested in (1; 15) and (1; 7), thus becoming a child of (1; 7):

Similarly, (4; 4) is embedded in (1; 15) and (1; 7) but it doesn’t intersect with (3; 3); therefore, it becomes the second child node of (1; 7)

Thus, all the blocks are in place and the we have our tree structure built. Now we can enhance it with useful information such as the text of the source code for the blocks, as well as some assembler instructions.

4. Conclusion

When compiling the contracts, Truffle generates very useful data which can help you to grasp how your code actually behaves and what exactly it does in the blockchain. This is also a convenient way to analyze bottlenecks in your code. However, this data is not exactly intelligible from the first sight, and today you’ve learned how to understand the format of it and figure out a way to present it in a simple and convenient form for further analysis.

Thank you for bearing with us through this long and technical post. We hope you enjoyed it and found it useful. Stay tuned for more insights from us, we can’t wait to share with you more of our hands-on Blockchain experience and fun stories!

Code hard. Build big. Decentralize.

Yours,

EQ LAB Team

Utilizing Multichain Bridge and Moonbeam to bring Liquidity to Polkadot Parachains

EQ LAB — Fri, 27 Jan 2023 17:15:27 +0000

Unlocking Ethereum liquidity is widely considered of paramount importance to any DeFi protocol’s success no matter the chain it chooses to launch on.

This DefiLlama chart illustrates almost 60% of all Defi TVL is locked on Ethereum. This makes bridging liquidity from Ethereum a high priority for Equilibrium.

This article will examine Equilibrium’s bridging solution that allows for cross-chain transfers between EVM networks and Polkadot parachains. It’s a unique contribution to the parachain ecosystem since it pioneers the concept of one parachain using another as a bridge. A brief overview of the underlying tech will also be provided. Let’s start with the basics.

Multichain is an open-source project that builds interoperable infrastructure for cross-chain interactions. It supports over eighty blockchain networks with approximately$1.6B TVL. Multichain has a history of reliability and trustworthiness stretching back to July 2020. Let’s examine Multichain’s Cross-Chain Bridge and Router.

Multichain Cross-Chain Bridge

The bridge fundamentally allows an asset to be sent from one chain to another. Let’s briefly describe how it works.

First an asset is locked on a token wrapper controlled by the Router contract on the source chain side. An MPC (Multi-party computation) Network with 28 validators verifies the transactions on the source chain and then signs token minting or withdrawal transactions on the target chain. A wrapped asset is then minted or native asset is withdrawn on the target chain. Wrapped assets can be burned to facilitate transfers from the target chain to the original one.

More info on the Cross-Chain Bridge.
Active MPC validator set can be found here.

Multichain Cross-Chain Router

The Router enables asset transfers for native tokens and those created with Multichain Bridge (bridged) between two or more chains. Liquidity pools support native assets since Multichain can not mint or burn those tokens. This requires tokens to be supplied to liquidity pools externally. Bridged assets do not require liquidity pools because Multichain controls the supply of those assets by minting and burning bridged tokens.

It’s also possible to combine native and bridged assets when a project adds support for extra chains via the router. Tokens on the chains supported by Multichain are considered bridged, while pre-existing tokens are considered native.

More info on the Cross-Chain Router.

SMPC network

Multichain uses a Secure Multi Party Computation (SMPC) network of nodes. These nodes generate parts of the private key for signing transactions. An algorithm selects a set of nodes from the network to do this. Selected nodes then sign transactions collectively. This mechanism is used for every supported network.

More info on SMPC.

Moonbeam EVM parachain

Moonbeam is a parachain on Polkadot designed as an onramp for developers. It’s an Ethereum-compatible L1 smart contract platform. Moonbeam is one of the ecosystem's biggest and best-known projects, the first winner of parachain auctions on Polkadot. Moonbeam enables developers to go cross-chain with their existing Ethereum dapps as well as create new cross-chain projects.

XCM

XCM stands for Cross-Consensus Message. It is how Polkadot brings interoperability to its projects. XCM format defines how messages are sent between blockchains, effectively connecting parachains to the relay chain and each other. This delivers on the promise of cross-chain application interoperability as more and more projects launch on Polkadot parachains allowing users to interact with all of them from a single dapp.

The main use case for XCM currently is accessing tokens in cross-chain dapps. Smart contracts on Moonbeam can communicate directly to perform transactions and other activities.

More info about XCM on Polkadot Wiki and Moonbeam.

XC-20 tokens

XC-20 is a token standard for ERC-20 tokens on Moonbeam. These tokens are cross-chain-ready and transferable across the entire Polkadot ecosystem. This is useful for applications that want to integrate native tokens as ERC-20s.

See Moonbeam docs for a detailed breakdown of how XC-20 works.

How it all comes together on Equilibrium

In Equilibrium’s case, both EQ and EQD tokens are controlled by Multichain, meaning that Multichain routers have minting and burning rights for these tokens. These minting and burning actions allow for control over the supply of these assets on the chain where the smart contract resides. This means all that is required for the bridge to work is a supply of assets on the chain where the token was originally minted. This is what bridging looks like for EQD:

The Equilibrium team developed a proxy contract that implements AnycallProxyBase interface and allows token transfer from Moonbeam to another parachain in the Polkadot ecosystem. This solution may be used by any team which wants to use a Multichain bridge in their parachain.

For a more in-depth look into how it works see github.

Moonbeam contains system contracts that connect EVM and Substrate parts of the network.
Using this contract, users may transfer XC-20 tokens from their Moonbeam EVM address to another parachain with substrate-type addresses.

The xTokens -transfer method allows sending XC-20 and pays fees in this token.

Using this method will move XC-20 tokens from a user to a sovereign account of the destination parachain in Moonbeam. The same value will be deposited to a recipient in the destination parachain.

Equilibrium put together two great things - a custom proxy call and "xTokens" contract and made automated deposits to Equilibrium or any other parachain from Multichain-supported networks possible.

Here is a generalized overview of what happens to the tokens under the hood using WBTC, ETH and USDC as an example:

Equilibrium implemented XcmTransferProxy for transferring tokens from Moonbeam to another parachain without additional user transactions. The code for this is on Github.

Here’s how it streamlines the token transfer process:

Withdrawal flow optimization

One of future Moonbeam runtime releases will enable the EVM call by XCM feature. It will be a powerful tool enabling interaction with EVM contracts on Moonbeam from any parachain.
Withdrawals from Equilibrium or any parachain will be enabled without additional user transactions on Moonbeam after the feature is released.

This is what withdrawal flow looks like now compared to after optimization:

Bridging liquidity across different chains is a complex and challenging process to streamline. Equilibrium has made a momentous step forward by developing a custom bridging solution. It benefits the project and ecosystem as a whole in two significant ways. Firstly it unlocks Defi TVL on Etherium and brings it to Polkadot parachains, and secondly, it facilitates the realization of Equilibrium’s mission to make EQD a cross-chain stablecoin.

This kind of cooperation between Equilibrium and Moonbeam in developing a solution that can be used by all parachains and benefits the entire ecosystem is the manifestation of a long-term vision of interoperability and value accrual that is now becoming a reality.

EQ Lab & The Future of Decentralized Finance

EQ LAB — Fri, 27 Jan 2023 16:52:48 +0000

EQ Lab is a software engineering company specializing in building robust and innovative Web3 products on top of multiple decentralized platforms, including Ethereum, Polkadot, TON, and others. Our core expertise lies in the field of finance and decentralized economy but is not limited to ranging from high-load infrastructure to applications incorporating NFT-based structures.

The flagship project in our portfolio is Equilibrium, a cutting-edge all-in-one DeFi platform that leverages the power of the Substrate development framework and Polkadot's parachain architecture. It provides users with a seamless and secure experience offering a wide range of features such as a decentralized stablecoin, lending, borrowing, and trading. Equilibrium’s mission is solving capital inefficiency in DeFi, making Equilibrium a versatile and powerful solution for high-leverage yet safe operations with digital assets.

What sets EQ Lab apart is our valuable blend of financial expertise and technical know-how.

Some of the team’s notable past achievements include:

GMRA-compliant p2p lending protocol on Ethereum called Nitrogen. It supported various ERC-20 tokens and sophisticated risk management models from TradFi repo markets.
EOSIO-based collateral-backed stablecoin protocol (EOSDT). The protocol leveraged EOSIO blockchain speed and deferred transactions to optimize liquidations thus lowering collateral requirements.
Bridge between TON blockchain and heterogenous networks. It adhered to a federated relay structure and was built using TON’s Fift and C++ languages.
Sophisticated risk-based approach to collateral backed loans on Polkadot which allowed to build a unified money-market, a stablecoin and a spot margin trading engine with leverages up to 20x.
The Curated - a digital catalog that allows for purchasing of digital art collections picked by a panel of experts.
Expertise and extensive research in the field of ZKPs allowed to build a proof of concept for rust-based blockchains (Substrate and Polkadot in particular).

As you can see from the list above, it’s not just Polkadot at the center of EQ Lab focus. The Substrate experience means the team is well-versed in building complex custom modular blockchain solutions. XCM technology is one example that has the potential to be utilized to transfer messages between different ecosystems and implemented outside of Polkadot.

EQ Lab team’s first-hand experience building on Polkadot has led to a number of unique technical solutions on offer to help less experienced teams and projects looking to get into the space.

Here is a non-exhaustive list of custom features and areas of expertise EQ Lab team perfected while building Equilibrium:

Unique bailsmen liquidation mechanism
Robust oracle solution
XCM implementation experience
Testnets and full emulation integration testing
Custom assets and balances, balances double accounting
Bridging solutions for EVM chains
Blockchain backend development
Custom balance checker and algorithmic asset management
Blockchain storage and trx throughput optimization

The aim is to create innovative and secure DeFi solutions that meet the needs of our clients using our extensive knowledge and experience. Partnering with EQ Lab will give you access to our team's expertise and resources and open the doors to DeFi.