DEV Community: Ivan Romanovich 🧐

Unleash the Power of the TON Blockchain: A Step-by-Step Guide to Data Collection with dton.io

Ivan Romanovich 🧐 — Sun, 15 Dec 2024 12:48:42 +0000

For any technical task on TON, you need to use indexers. Indexers are services that aggregate blockchain transactions within themselves, enrich the data and allow you to get this data in the required form.

Without using such services, for each request for information, you would have to parse a bunch of blockchain blocks to return the data. In this article, I will show you how to make GraphQL queries in dton.io on the TON blockchain. Let's take a simple task and go through the entire path of forming a request and at the same time consider the main features of the indexer.

What is dton.io?

Dton.io is a blockchain indexer, which means it collects information from each new block into its database. You can make GraphQL queries to this database and thus collect historical information without parsing the entire chain of blocks.

The two most important sources of dton.io data are the transaction table and the view of the latest state of wallets/accounts in the network. By making queries to this view and table, you can collect almost any information.

A big plus of dton.io is data enrichment, in addition to native transaction and block data, dton.io collects information from the smart contract data registry, and also enriches popular token standards on TON with data, such as NFT - the standard of non-fungible tokens and Jetton - the standard of fungible tokens.

Data enrichment allows you to reduce the number of necessary queries.

How does the request formation process work

The top-level algorithm is always the following:
understand how the task looks at the smart contract level
iterative request collection + combating emerging problems (like timeout on very large requests), validating the correctness of the data

Due to the specifics of the smart contract architecture on TON, the first part is often the most difficult - one logical operation, for example, supplying liquidity to the pool, can involve several contracts and, of course, searching for where among the 5 contracts the information you need is transmitted can only be obtained by delving into smart contracts.

By the way, you can run the requests that we will consider below here - https://dton.io/graphql/

Formulating the task

Within the framework of this tutorial, we will take a task that is understandable to a wide range: find the largest NFT buyers for the selected collection.

The premise of such a request may be the desire to find someone to sell your NFT to. But in this tutorial, this is just a convenient task through which you can show most of the functionality of dton.io..

What does this mean in terms of smart contracts

Before jumping headlong into the fields available in the indexer and thinking about how we aggregate data on the amount and number of transactions, let's understand how NFT sales in TON occur.

Almost any transaction implies a certain sales contract to which the NFT is “moved”, this contract implements the logic of the sale, it can be either a simple sale to anyone willing to pay the amount, or complex logic in the form of an auction.

Accordingly, from the data point of view, a sale can be called a transaction in which the owner of the NFT changes. You can see examples of different sales contracts here - https://github.com/ton-blockchain/token-contract/blob/main/nft/nft-sale.fc. This is the simplest sales contract written in the FUNC language.

Now let's try to make the first request.

Let's get the latest transactions with nft sales

So, we are interested in transactions with the transfer of ownership of NFT, so we will use the transaction representation:

      {
        raw_transactions(
        ) {
        }
      }

Now let's try to fill in the request, you can see all available fields in the documentation https://docs.dton.io/transactions-and-account-states .
Let's take the Telegram Username NFT collection as an example, as well as already completed transactions:

      {
        raw_transactions(
          parsed_seller_is_closed: 1
          parsed_nft_collection_address_address: "80D78A35F955A14B679FAA887FF4CD5BFC0F43B4A4EEA2A7E6927F3701B273C2"
        ) {
          nft_address: address
          col_address: parsed_nft_collection_address_address
        }
      }

Fields starting with parsed are enriched, not native fields. Now, it is important for us to remove transactions that do not occur with NFT, for this we will add the field account_state_state_init_code_has_get_nft_data: 1 .

Here the question may arise what is get_nft_data, the easiest way to distinguish different types of smart contracts is their signature, smart contracts of certain standards must contain certain functions and calls. The NFT standard in TON assumes the presence of the get_nft_data method.

      {
        raw_transactions(
          parsed_seller_is_closed: 1
          account_state_state_init_code_has_get_nft_data: 1
          workchain: 0
          page: 0
          page_size: 10
          parsed_nft_collection_address_address: "80D78A35F955A14B679FAA887FF4CD5BFC0F43B4A4EEA2A7E6927F3701B273C2"
        ) {
          prev_owner: parsed_seller_nft_prev_owner_address_address
          new_owner: parsed_seller_nft_new_owner_address_address
          nft_address: address
          col_address: parsed_nft_collection_address_address
          price: parsed_seller_nft_price
        }
      }

We will also immediately add parsed values of the price of both the previous and new owner. I suggest executing the request.

Timeout problem

After executing the request we will see:

{
  "data": {},
  "errors": [
    "Timeout exceeded, simplify your query (https://docs.dton.io/query-speed-optimization-timeout)"
  ]
}

The problem is that our query is processed across the entire indexer database. Possible solutions to the problem are described here https://docs.dton.io/query-speed-optimization-timeout. The simplest option is to add a time filter.

For this, we need filters.

Filtering the query by time

Filters in dton.io can be added to fields using underscores. The generation time filter is one of the most common - it allows you to “not run” the query through the entire database.

Let's narrow our task and search only for the last month:

      {
        raw_transactions(
          parsed_seller_is_closed: 1
          account_state_state_init_code_has_get_nft_data: 1
          workchain: 0
          page: 0
          page_size: 10
          gen_utime__gt: "2024-12-04 00:00:00"
          parsed_nft_collection_address_address: "80D78A35F955A14B679FAA887FF4CD5BFC0F43B4A4EEA2A7E6927F3701B273C2"
        ) {
          prev_owner: parsed_seller_nft_prev_owner_address_address
          new_owner: parsed_seller_nft_new_owner_address_address
          nft_address: address
          col_address: parsed_nft_collection_address_address
          price: parsed_seller_nft_price
        }
      }

The list of available filters is here - https://docs.dton.io/filters

In the request you can see page and page_size, this is a normal pagination. Transaction and account status displays support a maximum of 150 records per page.

Let's try to run the request, we'll get something like this:

{
  "data": {
    "raw_transactions": [
      {
        "prev_owner": "CF01D60CC8924D3C34C54A5787B9175BCD8D45D9ADA91371F93318DF2B76EBDB",
        "new_owner": "69B8D2CA45C14F056FED73236A6A0CB7AB5BA2B1A0F2E1AD784D84F7B4454D81",
        "nft_address": "8C08EBCDDEDB79E6FF3AA1B4F0A65388D16FE2F1D86BFBA46E25833A24A659F4",
        "col_address": "80D78A35F955A14B679FAA887FF4CD5BFC0F43B4A4EEA2A7E6927F3701B273C2",
        "price": 4746195291
      },
      {
        "prev_owner": "50FC81C8CFEA8780CEF8B636D3643ED9F7E5ADD3D4E8045510FD19AF2EAC337D",
        "new_owner": "0E4A5829EEBC85FB049DD63B5CCF801D3DC411780ABEC22F8F8918B66B852337",
        "nft_address": "E20696CD521BF88D729E06827587D2908E0ED94ED5910A79D26069C4F9F261AB",
        "col_address": "80D78A35F955A14B679FAA887FF4CD5BFC0F43B4A4EEA2A7E6927F3701B273C2",
        "price": 8479134769
      }
…
  ]
  },
  "errors": []
}

Removing wash trade

As I mentioned at the beginning of the article, query assembly is an iterative process, which means it's time to get back to the business task. For the statistics we are interested in, it is important for us to remove wash trade, for example, when resale occurs between one's own wallet.

Filters consisting of logical operators will help us with this, in this case not. We will remove from the query transactions in which the old and new owners are the same.

      {
        raw_transactions(
          parsed_seller_is_closed: 1
          account_state_state_init_code_has_get_nft_data: 1
          workchain: 0
          page: 0
          page_size: 10
          gen_utime__gt: "2024-12-04 00:00:00"
          not__: {parsed_seller_nft_prev_owner_address_address: new_owner}
          parsed_nft_collection_address_address: "80D78A35F955A14B679FAA887FF4CD5BFC0F43B4A4EEA2A7E6927F3701B273C2"
        ) {
          prev_owner: parsed_seller_nft_prev_owner_address_address
          new_owner: parsed_seller_nft_new_owner_address_address
          nft_address: address
          col_address: parsed_nft_collection_address_address
          price: parsed_seller_nft_price
        }
      }

Let's satisfy your curiosity - what are the largest deals

dton.io allows you to specify the order by any field, for example, you can sort by price, get the largest deals first:

      {
        raw_transactions(
          parsed_seller_is_closed: 1
          account_state_state_init_code_has_get_nft_data: 1
          workchain: 0
          page: 0
          page_size: 10
          gen_utime__gt: "2024-12-04 00:00:00"
          not__: {parsed_seller_nft_prev_owner_address_address: new_owner}
          order_by: "parsed_seller_nft_price",
          order_desc: true,
        parsed_nft_collection_address_address: "80D78A35F955A14B679FAA887FF4CD5BFC0F43B4A4EEA2A7E6927F3701B273C2"
        ) {
          prev_owner: parsed_seller_nft_prev_owner_address_address
          new_owner: parsed_seller_nft_new_owner_address_address
          nft_address: address
          col_address: parsed_nft_collection_address_address
          price: parsed_seller_nft_price
        }
      }

Convenient features - converting addresses

As in other blockchains, TON has different representations of addresses. To convert the addresses in the request to the most commonly used form, you need to change the fields to the same fields but with the prefix friendly.

In our case, it will look something like this:

      {
        raw_transactions(
          parsed_seller_is_closed: 1
          account_state_state_init_code_has_get_nft_data: 1
          workchain: 0
          page: 0
          page_size: 10
          gen_utime__gt: "2024-12-04 00:00:00"
          not__: {parsed_seller_nft_prev_owner_address_address: new_owner}
          order_by: "parsed_seller_nft_price",
          order_desc: true,
        parsed_nft_collection_address_address: "80D78A35F955A14B679FAA887FF4CD5BFC0F43B4A4EEA2A7E6927F3701B273C2"
        ) {
          prev_owner: parsed_seller_nft_prev_owner_address_address__friendly
          new_owner: parsed_seller_nft_new_owner_address_address__friendly
          nft_address: address__friendly
          col_address: parsed_nft_collection_address_address__friendly
          price: parsed_seller_nft_price
        }
      }

If you are interested in reading about different representations of addresses, TON has a separate standard :https://github.com/ton-blockchain/TEPs/blob/master/text/0002-address.md

Data aggregation

After we have collected a simple query, checked that in the response to the query we receive the transactions that we expected, we can proceed to the assembly of the aggregated query.

For aggregation, you can use the groupTransactions and groupAccountStates endpoints to perform aggregation operations. These endpoints support various aggregation functions and allow you to group, sort and filter the results.

There are quite a few parameters in aggregation queries, which is why I initially advise collecting simple queries and only then simply collecting the aggregation from them.

We put the fields by which we filtered the query in filter. The remaining fields are filled in accordance with the aggregation logic:

{
  groupTransactions(
    by: ["parsed_seller_nft_new_owner_address_address__friendly"],
    aggregations: [
      {field: "parsed_seller_nft_price", operation: "sum"},
      {operation: "count"}
    ],
    order_by: "parsed_seller_nft_price__sum",
    order_desc: true,
    page: 0,
    page_size: 10,
    filter_by: {or__: [
      {not__: {
        parsed_seller_nft_prev_owner_address_address: parsed_seller_nft_new_owner_address_address
      }},
      {and__: {
          parsed_seller_is_closed: 1
          account_state_state_init_code_has_get_nft_data: 1
          workchain: 0
          gen_utime__gt: "2024-12-04 00:00:00"
          parsed_nft_collection_address_address: "80D78A35F955A14B679FAA887FF4CD5BFC0F43B4A4EEA2A7E6927F3701B273C2"
      }}]}
  ) {
    trader_new_address:parsed_seller_nft_new_owner_address_address__friendly
    deal_count: count
    deal_sum: parsed_seller_nft_price__sum
  }
}

Using by, we aggregate by the address of the new owner, in the remaining fields we write the logic of calculation by price. For representativeness, we will calculate both the amount and the number of transactions. More information about aggregation is here - https://docs.dton.io/aggregations.

I will note that to perform large aggregations, a paid dton.io plan is required

Conclusion

I like the TON blockchain for its technical elegance, at least it is not another copy of Ethereum, which is accelerated with the help of large capital without looking back, and in general why the user needs it. If you want to learn more about the TON blockchain, I have open source lessons, thanks to which you will learn how to create full-fledged applications on TON.

https://github.com/romanovichim/TonFunClessons_Eng

I post new tutorials and data analytics here: https://t.me/ton_learn

Looking for memecoins on Stonfi in TON or is there life beyond Notcoin

Ivan Romanovich 🧐 — Mon, 10 Jun 2024 08:33:55 +0000

Memcoins along with NFT are some of my least favorite crypto trends, no technology, just marketing. But the explosive growth in Q1 2024 draws attention to the memecoin narrative. (According to coingecko, this is the most profitable narrative of Q1 2024, but there are questions about the methodology). Therefore, I suggest trying to find memecoins on TON blockchain by trying to put together a simple memecoin dashboard.

In the article, I want to figure out why memecoin analytical platforms look like a live feed of purchases of these very memecoins. Run through the new Stonfi APIs, the data from this DEX is now displayed on Dexscreener, and separate open APIs have appeared for this, which greatly affects the availability of data (and this is a problem in TON). And thirdly, look at some of your statistical hypotheses regarding pools.

What are we going to look for?

If you google the definition of memecoins, the definitions will vary from the pyramids of our time to the digital meme currency, a new word in the field of social tech. Therefore, within the framework of this article, I offer my definition: Memcoin is a dynamically growing token without an initial utility, using Fomo mechanisms to ensure growth. This definition allows me to formulate a hypothesis: Memcoins are fast-growing small projects, which means you can first cut off large projects by the volume of their pools (Total Value Locked), and then consider small projects by the number of swaps for the last, for example, day.

Building a TVL diagram for Stonfi pools - looking for the threshold beyond which memecoins

Let's start with TVL. Stonfi has two APIs v1/pools - returning the current state of pools and v1/stats/pools - statistics on pools for the period. Let's use v1/pools, we need the pool address, the addresses of the tokens that make up the pool and lp_total_supply_usd - the total supply of lp tokens in the pool. Lp tokens are automatically generated by DEX and credited to the liquidity provider for contributing assets to the liquidity pool. These tokens represent a share of the commissions earned by the liquidity pool. Accordingly, their supply in dollar terms will reflect the TVL.



import requests
r = requests.get('https://api.ston.fi/v1/pools')
result = r.json()['pool_list']

temp_pool_list = [{'pool_address': pool['address'], 'token0_address': pool['token0_address'], 'token1_address': pool['token1_address'], 'tvl': round(float(pool['lp_total_supply_usd']),2)} for pool in result]


# Let's sort
sorted_pool_list = sorted(temp_pool_list, key=lambda d: d['tvl'],reverse=True)
sorted_pool_list[0:5]

After sorting we get:

It is easier to perceive information visually, so let's plot a graph:



import matplotlib.pyplot as plt

fig, ax = plt.subplots()

pool_addresses = [d['pool_address'][-4:] for d in sorted_pool_list]
tvl_values = [d['tvl'] for d in sorted_pool_list]

plt.bar(pool_addresses, tvl_values, color='skyblue',log=True)
plt.xlabel('Pool Address')
plt.ylabel('TVL')
plt.title('TVL for Pools')
plt.xticks(color='w')
plt.tight_layout()

Result:

This graphic clearly shows that the first couple of pools cover all the others in volume, it is also clear that there are plateaus, and the volume of locked funds allows to classify tokens into certain leagues, from small change to simply huge pools.

To play with hypotheses, let's make a pie chart, dividing all the pools by a certain threshold, and put the number of pools in the name, then it will be very clear, for example, that the first three pools in volume contain more than half of the locked liquidity.



threshold = 10000000

big_pools = sum(item['tvl'] for item in sorted_pool_list if item['tvl']>=threshold)
small_pools = sum(item['tvl'] for item in sorted_pool_list if item['tvl']<threshold)

big_pools_count = len([item['tvl'] for item in sorted_pool_list if item['tvl']>=threshold])
small_pools_count = len([item['tvl'] for item in sorted_pool_list if item['tvl']<threshold])

labels = 'Big pools', 'Small pools'
sizes = [big_pools, small_pools]

fig, ax = plt.subplots()

ax.pie(sizes, labels=labels)
ax.set_title("Big pools count:{}, Small pools count {} ".format(big_pools_count,small_pools_count))

Result:

There is a pump fun launchpad on the Solana blockchain, it has a limit of 69k dollars, i.e. you can launch your token, but as soon as it grows above 69k dollars it will “go” to a large exchange, let's try this threshold:

But there are some nuances here, pools can contain pairs of any jettons (the standard of tokens on TON) or TON. But for the most part, these are pools:

jetton - TON
jetton - stablecoin
jetton - Notcoin

Notcoin can be called the largest TON memecoin, so our simple dashboard should start with the dominance of Notcoin. Let's check the diagram:



Notcoin = 'EQAvlWFDxGF2lXm67y4yzC17wYKD9A0guwPkMs1gOsM__NOT'


not_pools = sum(item['tvl'] for item in sorted_pool_list if item['token0_address']==Notcoin or item['token1_address']==Notcoin )
notnot_pools = sum(item['tvl'] for item in sorted_pool_list if item['token0_address']!=Notcoin or item['token1_address']!=Notcoin )


not_count = len([item['tvl'] for item in sorted_pool_list if item['token0_address']==Notcoin or item['token1_address']==Notcoin])
notnot_count = len([item['tvl'] for item in sorted_pool_list if item['token0_address']!=Notcoin or item['token1_address']!=Notcoin])

labels = 'Notcoin pools', 'Other pools count'
sizes = [not_pools, notnot_pools]


fig, ax = plt.subplots()


ax.pie(sizes, labels=labels)
ax.set_title("Notcoin pools count:{}, Other pools count {} ".format(not_count,notnot_count))

Result:

As you can see, compared to other tokens, Notcoin is huge and its dominance is worth considering when reviewing the TON memecoin market.

Liquidity problem

Okay, let's say you've chosen some TVL threshold and looked at Notcoin dominance, but there's no point in looking at TVL any further. Small pools suffer from a liquidity problem - they simply don't have enough swaps. And the liquidity locked in them doesn't allow you to compare them.

That's why memecoin analytics platforms often look like a live feed. Applications scan blocks for swaps in small pools. This allows you to see in about a minute which memecoin is currently growing. To make it clearer what it looks like, I've put together something similar for Stonfi pools:
tonhotshot.fun

First, pools are scanned and the largest pool up to 69k TVL is found - the king of the hill
Then each block is scanned for transactions in Stonfi in pools up to 69k.

And the data is taken from the publicly available Stonfi APIs that I mentioned earlier. Let's try to build our own dashboard on this data.

Ranking Memcoins on Stonfi

We will take data on swaps from the new API created for Dexscreener, this is the Events API (export/dexscreener/v1/events). This API returns all DEX Stonfi events between two blocks. Let's say we select a period of a day for our dashboard, where can we get the current block to start from? There are two options:

The first option is to use the neighboring handle export/dexscreener/v1/latest-block, it will return the last block processed by the exchange backend, which indexes the blockchain and allows us to get data in an aggregated form. The advantage of this approach is that we will get the last block processed by the indexer, the disadvantage is that the handle works on average 10 seconds and this is not always convenient.



  ltb = requests.get('https://api.ston.fi/v1/screener/latest-block')
  lastest_block = ltb.json()['block']['blockNumber']

The second option is to simply take the last block from the blockchain, yes, the last block in the blockchain is not equal to the last block processed by the exchange indexer, but it is fast, one of the options for how to do this:



  # RPS 1 sec
  ltb = requests.get('https://toncenter.com/api/v3/masterchainInfo')
  return ltb.json()['last']['seqno']

Let's say we take all the events, how do we remove large blocks? Just like we did at the beginning - get all the pools and remove those we don't need:



def get_block_list(sorted_pool_list,threshold):
  Notcoin = 'EQAvlWFDxGF2lXm67y4yzC17wYKD9A0guwPkMs1gOsM__NOT'
  return [item['pool_address'] for item in sorted_pool_list if item['tvl']>=threshold or item['token0_address'] == Notcoin or item['token1_address'] == Notcoin]

We will also immediately remove pools with notcoin, since we are looking for new memecoins.

After collecting events, we will immediately count the number using Counter:



def count_swap_events(sorted_pool_list,threshold):
  blocker = get_block_list(sorted_pool_list,threshold)


  ltb = requests.get('https://api.ston.fi/v1/screener/latest-block')
  lastest_block = ltb.json()['block']['blockNumber']

  start_block=lastest_block - int(86400/5) # TON "sync" block every 5 second, in a day 86400 second
  payload = {'fromBlock': start_block, 'toBlock': lastest_block}
  r = requests.get('https://api.ston.fi/export/dexscreener/v1/events', params=payload)


  count_arr=[]
  for item in r.json()['events']:
    if(item['eventType']=='swap'):
      if(item["pairId"] not in blocker):
        count_arr.append(item)


  c = Counter()
  for event in count_arr:
    c[event["pairId"]] += 1

To grab a couple more handles, we will enrich our data using only the pool address, for this we will use /export/dexscreener/v1/pair/ to get the token addresses and use /v1/assets/ to get the token names or TON.



def pool_pair(pool_addr):
    p = requests.get('https://api.ston.fi/export/dexscreener/v1/pair/{}'.format(pool_addr))
    try:
      pair=p.json()['pool']
      return jetton_name(pair['asset0Id']) +"-"+ jetton_name(pair['asset1Id'])
    except:
      return pool_addr

Here I will note that this is just a tutorial and all the code is written as primitively as possible so that you can read it diagonally. Let's enrich our Counter and sort it:



def count_swap_events(sorted_pool_list,threshold):
  blocker = get_block_list(sorted_pool_list,threshold)


  ltb = requests.get('https://api.ston.fi/v1/screener/latest-block')
  lastest_block = ltb.json()['block']['blockNumber']


  start_block=lastest_block - int(86400/5) # TON "обновляет" блоки каждые 5 секунд в дне 86400 секунд
  payload = {'fromBlock': start_block, 'toBlock': lastest_block}
  r = requests.get('https://api.ston.fi/export/dexscreener/v1/events', params=payload)


  count_arr=[]
  for item in r.json()['events']:
    if(item['eventType']=='swap'):
      if(item["pairId"] not in blocker):
        count_arr.append(item)


  c = Counter()
  for event in count_arr:
    c[event["pairId"]] += 1


  enriched_arr=[]

  for pool in sorted(list(c.items()), key=lambda d: d[1],reverse=True)[0:30]:
    enriched_arr.append({"pool": pool_pair(pool[0]),'24h_swaps':pool[1]})


  return enriched_arr

Result is something like this:

Of course, there is much that can be improved, but I suggest the reader to complete the dashboard himself, all Stonfi APIs can be viewed here - https://api.ston.fi/swagger-ui/

Conclusion

I admire the TON blockchain for its technical elegance, at least it is not another copy of Ethereum, which is accelerated with the help of large capital without looking back, and in general why does the user need it. If you want to learn more about the TON blockchain, I have open source lessons, thanks to which you will learn how to create full-fledged applications on TON.

https://github.com/romanovichim/TonFunClessons_ENG

I throw new tutorials and data analytics here

Let's understand the stablecoin on TON blockchain or how your funds can be blocked

Ivan Romanovich 🧐 — Fri, 03 May 2024 10:42:39 +0000

On April 19, at the Token2049 conference, after a speech by Pavel Durov, the appearance of a stablecoin from Tether in the TON - USDt network was announced. Since this is a centralized stablecoin, the issuer of such a token must have control over user funds to comply with regulatory requirements, for example, block user funds. In this short article I will tell you how tokens on TON work and what opportunities a stablecoin issuer has.

If you are already familiar with the TON blockchain and the concept of how tokens work, you can immediately skip to the Locking your funds paragraph, which discusses the mechanics of a stablecoin, since we will start in this article with the basics.

How smart contracts work in the TON network

The TON blockchain is asynchronous, smart contracts transmit data to each other by messages. Contracts are written in TACT and FUNC, the logic written in these languages is quite understandable without detailed study, we will use this when reviewing stablecoin contracts. But if you want to understand deeper, there are free open source lessons here.

Smart contracts on TON do not involve storing large dictionaries within themselves; instead, it is supposed to create a master contract that can “recreate” (the address is a hash of the initial information of the smart contract and its code, which makes similar mechanics possible) within itself the addresses of related contracts and thus send them messages, my information was stored in them

This mechanism is called smart contract sharding. Don't worry if this is not very clear, below we will look at how it works using examples. For now, just note that at TON we almost always deal with a group of smart contracts transferring information from each other.

The sharding mechanism allows business logic on smart contracts to be scalable, but this complicates the design of smart contracts. This is worth considering if you plan, for example, to transfer the logic of some EVM-compatible decentralized application to TON.

For each smart contract, there are reserved methods for internal and external messages. Typically, these methods describe the logic of what the smart contract should do, depending on the message payload. It looks something like this:

Like other networks, TON has smart contract standards. The standard describes the required signature of smart contract methods and the required logic. Standards allow decentralized applications, such as wallets, to display information, to understand, for example, what is an NFT contract and what is a token contract.

A stablecoin is a fungible token tied to the price of the token, so before moving on, let's look at the standard fungible token in the TON network - Jetton.

Jettons or how to store data without a dictionary

Tokens in the TON network are fungible tokens. A token is a unit of accounting for some digital asset in some blockchain network. It is important to note that a token does not usually mean a cryptocurrency, but rather a record distributed on a blockchain that is managed using smart contracts. The smart contract records the values of account balances of token holders (the balance of a certain address of the owner), and the smart contract also provides the ability to transfer tokens from one account to another.

Accordingly, a technical problem arises: how to store balances of tokens of different addresses and effectively update these balances. This is where sharding comes in handy.

We will split the storage logic into several contracts, for each token owner, we will issue a separate wallet contract (namely the wallet of this token, not the wallet for Toncoin), which will store information about the balance of only this wallet, and there will also be a master contract that will store information about the token itself, for example, its name, symbol, general offer, etc.

Thus, if our token is owned by 1,000,000 different users, exactly 1,000,001 contracts will be deployed. And with various messages, you can perform certain functions.

Let's look at a simple token transfer. So, in order to transfer tokens from wallet one to wallet two, we will need to send a message with the transfer flag to our token wallet, which in turn, having received such a flag, will send funds to token wallet 2 and here is an interesting point, since the messages use Toncoin as fee for messages and execution of smart contracts (like gas in EVM networks), then in the end the token wallet 2 will return the excess back to Toncoin wallet

Note that the master smart contract is not involved at all in transferring tokens between wallets. But if we didn’t want to transfer the tokens, but, for example, just burn them, then we would have to send a separate message to the master contract, because when burned we would change the total supply of the token.

Of course, these are slightly simplified diagrams, the purpose of which is to give an idea of the architecture of smart contracts on TON. If you want to dive deeper into how Token standard tokens work, I have a full breakdown of the standard here.

Another small nuance: most often it is the master contract that deploys contracts and wallets.

Locking your funds in a stablecoin

Like a regular token, a stablecoin smart contract has a master contract and a wallet contract. And the first thing that distinguishes a regular token from a stablecoin is storage.

A regular token stores the balance, owner's address, master contract address and wallet code (together with the contract storage, the code allows you to obtain the code of a given token wallet). From jetton-wallet.fc (this is the wallet code in FunC language):

But the stablecoin, in addition to this information, also stores the Status, and it is this variable that allows you to block your funds.

Before describing how it works, I’ll make a small clarification for those attentive: if you compare smart contract storages, you will not see cells with code in the stablecoin, this is due to the fact that during the time between the creation of the token standard and the stablecoin token, there were updates to the virtual machine TVM and at the moment you can not transfer a cell with a code, but simply get the current smart contract code through the MYCODE primitive

In the proposed standard, the status variable, if we simplify the logic with bits, can take three values:
0 - unlocked, you can safely use your funds
1 - wallet owner cannot send stablecoin
2 - the wallet owner cannot receive stablecoin
This status can only be changed from the contract master; the following conditions are in the function that processes internal messages:

It works quite simply, the owner of the master contract sends a message with a certain payload to the master contract, and the master contract changes the status of the message.

Thus, the issuer of a stablecoin can block the funds of any wallet with a given stablecoin. As I mentioned at the beginning of the article, this is necessary to comply with the requirements of regulators by the issuer, which are usually funds that provide the stablecoin in a certain ratio with fiat and, of course, such funds cannot ignore the requirements of the state.
But blocking your funds is not the only option for the administrator of the master contract - the issuer. In addition, your funds can be burned or simply transferred.

Burning and Transferring Your Funds

If we go to the master contract jetton-minter.fc (this is the master contract) and see how the status change is implemented, then in the function for processing internal messages we will see the call_to fork, in which you can send a command to any wallet with stablecoins to transfer a stablecoin, burning or just a change in status, which we considered in the previous block of the article.

Thus, the admin of the master contract can easily manage your funds. It is important to note here that if an administrator tries to steal your funds without regulatory pressure, this will discredit the stablecoin, since all transactions in the blockchain are visible and a similar precedent will be replicated by telegram channels in a couple of days to all users who do not want to use such a stablecoin.

Such a system of incentives restrains the stablecoin issuer from abusing technical capabilities.
But that's not all.

Mechanism for replacing code and master contract data

The title of this paragraph reveals its entire essence - yes, this is possible, a complete replacement of the master contract code. It looks something like this:

From the message that comes to the contract, they will simply take new data for the storage and a new code.

This way, the creators of the stablecoin can change almost everything about it. And this is some reality that cryptocurrencies face, forced under pressure from regulators to create blocking mechanics, as well as leave the opportunity to change everything if something goes wrong.

Conclusion

I like the TON blockchain for its technical elegance; at least it’s not another copy of Ethereum, which is being overclocked with the help of a lot of capital without regard to why the user needs it in general.

If you want to learn more about the TON blockchain, I have open source lessons that will teach you how to create full-fledged applications on TON.

https://github.com/romanovichim/TonFunClessons_Eng

I post new tutorials, data analytics and articles here: https://t.me/ton_learn

Stablecoin standart code: https://github.com/ton-blockchain/stablecoin-contract/tree/main

Create a simple dashboard of tokens on TON blockchain using the Stonfi API

Ivan Romanovich 🧐 — Tue, 12 Mar 2024 13:06:02 +0000

Anyone who has planned to purchase tokens or other digital assets on the blockchain has encountered the difficulty of researching such assets. Blockchains are poorly adapted for collecting analytics - nodes and lightnodes provide information only on a specific block of the network, so you have to use blockchain indexers services or API of services running on the blockchain.

As decentralized exchanges strive to have a presence on platforms like coingecko, they create standardized APIs, often making them open source. Such APIs allow you to get a lot of information related to token trading.

Therefore, in this article we will put together a dashboard for tokens on the TON blockchain and see how easy it is to get data from DEX, but in the end we will talk about the problems of this method of data collection. I hope such a simple tutorial will make the steps in the TON blockchain clear and enjoyable.

Before we get into data collection, a few disclaimers:

the code in the tutorial is as simple as possible so that it can be read diagonally
tokens are the standard for fungible tokens on TON

Helper functions

Before calling the API handle, let's write helper functions that will help us with the parameters for the called APIs, namely, determine the dates and put them in the format we need:



def now_utc():
      return datetime.datetime.utcnow().strftime('%Y-%m-%dT%H:%M:%S')

def thirty_days_utc():
      return (datetime.datetime.utcnow() - datetime.timedelta(days=30)).strftime('%Y-%m-%dT%H:%M:%S')

def seven_days_utc():
      return (datetime.datetime.utcnow() - datetime.timedelta(days=7)).strftime('%Y-%m-%dT%H:%M:%S')

def day_utc():
      return (datetime.datetime.utcnow() - datetime.timedelta(days=1)).strftime('%Y-%m-%dT%H:%M:%S')

now_utc()

Let’s call API GET https://api.ston.fi/v1/stats/pool



payload = {'since': day_utc(), 'until': now_utc()}
r = requests.get('https://api.ston.fi/v1/stats/pool', params=payload)
r.json()['stats'][0]

Let's get various pool parameters:

Now let's try to collect the information we need

Take fields

For our dashboard we will need:
Token name and symbol
Latest Jetton price for the selected period
Volume in TON
Swap pool link

Let's select this data using our API:



def take_pool_stats():
  temp_arr=[]
  payload = {'since': day_utc(), 'until': now_utc()}
  # stonfi pools stats
  r = requests.get('https://api.ston.fi/v1/stats/pool', params=payload)
  resp = r.json()['stats']

  for jetton in resp:
    temp_dict = {'coin': jetton['base_name'],'pair': jetton['base_symbol']+'/'+jetton['quote_symbol'],'url': jetton['url'],'price_ton': jetton['last_price'],'volume_ton': jetton['quote_volume']}
    temp_arr.append(temp_dict)

  return temp_arr

take_pool_stats()

We will see these jsons:



{'coin': 'Chow Chow',
  'pair': 'CHOW/TON',
  'url': 'https://app.ston.fi/swap?ft=EQBtWFPgVknfzu6xaVcRBbNP3h_6UJ_xe29sVkiFTyPiv2bq&tt=EQCM3B12QK1e4yZSf8GtBRT0aLMNyEsBc_DhVfRRtOEffLez',
  'price_ton': '0.010646867',
  'volume_ton': '0.000000000'},
 {'coin': 'Tonald Duck',
  'pair': 'TDUCK/TON',
  'url': 'https://app.ston.fi/swap?ft=EQBUGgcu-h4SqMh5hrentq4vE59tBRRfrYE3H_0s6D_1Xzsa&tt=EQCM3B12QK1e4yZSf8GtBRT0aLMNyEsBc_DhVfRRtOEffLez',
  'price_ton': '0.000000363',
  'volume_ton': '0.000000000'},

Don’t be surprised that the volume is zero - anyone can create a pool, so not all of them are in great demand.

Step back

It is inconvenient to look at prices in TON, so to represent prices in dollars, we will get the current value of TON/USD. To do this, we will use tonapi.io for our task, free limits are quite enough:



def take_ton_price():
  resp = requests.get('https://tonapi.io/v2/rates?tokens=ton&currencies=usd')
  # Ex: {"rates":{"TON":{"prices":{"USD":2.1215},"diff_24h":{"USD":"+0.85%"},"diff_7d":{"USD":"-6.04%"},"diff_30d":{"USD":"+2.75%"}}}}
  return resp.json()["rates"]["TON"]["prices"]["USD"]


take_ton_price()

Let's remove the unnecessary

We use the resulting exchange rate to recalculate volume and price. As mentioned above, we give away all pools to the API, this means that unclaimed pools whose volume is zero can get there. As well as pools between tokens, which are not interesting to us, since they do not display the price and trading volume relative to TON or the dollar. Respectively:
choose pools with TON
we will remove pools in which the last price is zero
we will remove pools where the trading volume for a period is less than $1000

And immediately sort by volume:



def take_pool_stats():
  temp_arr=[]
  payload = {'since': day_utc(), 'until': now_utc()}
  # stonfi pools stats
  r = requests.get('https://api.ston.fi/v1/stats/pool', params=payload)
  resp = r.json()['stats']
  # ton_usd
  ton_usd = take_ton_price()

  for jetton in resp:
    temp_dict = {'coin': jetton['base_name'],'pair': jetton['base_symbol']+'/'+jetton['quote_symbol'],'url': jetton['url'],'price_ton': jetton['last_price'],'volume_ton': jetton['quote_volume'], 'price_usd': round(float(jetton['last_price'])*ton_usd,2),'volume_usd': round(float(jetton['quote_volume'])*ton_usd,2)}
    if(jetton['quote_symbol']=='TON' and int(float(jetton['last_price']) != 0) and int(float(jetton['quote_volume']) != 0) and int(temp_dict['volume_usd']) > 1000):
      temp_arr.append(temp_dict)

  return sorted(temp_arr, key=lambda d: d['volume_usd'],reverse=True)

# Coin - base_name
# Pair - base_symbol/quote_symbol url
# Price - last_price * TON in USD price

take_pool_stats()

As result we get json like this:



{'coin': 'STON',
  'pair': 'STON/TON',
  'url': 'https://app.ston.fi/swap?ft=EQA2kCVNwVsil2EM2mB0SkXytxCqQjS4mttjDpnXmwG9T6bO&tt=EQCM3B12QK1e4yZSf8GtBRT0aLMNyEsBc_DhVfRRtOEffLez',
  'price_ton': '2.220362684',
  'volume_ton': '74651.187439449',
  'price_usd': 5.89,
  'volume_usd': 197926.43},

Percentage of total

It is convenient to consider the volume from the total volume on the market, so let’s calculate the amount of volume and add a percentage. (In the code you can see that I left as many as 10 digits after the decimal point, this is due to the fact that TON recently took an initiative to add liquidity to the pools, due to which the volumes were greatly “blurred”. In a normal situation, 4 characters would be enough)



# Percentage from total
def take_pool_stats(payloa):
  temp_arr=[]

  # stonfi pools stats
  r = requests.get('https://api.ston.fi/v1/stats/pool', params=payload)
  resp = r.json()['stats']
  # ton_usd
  ton_usd = take_ton_price()
  # for volume percentage
  all_volume = sum(float(item['quote_volume']) for item in resp)

  for jetton in resp:
    temp_dict = {'coin': jetton['base_name'],'pair': jetton['base_symbol']+'/'+jetton['quote_symbol'],'url': jetton['url'],'price_ton': jetton['last_price'],'volume_ton': jetton['quote_volume'], 'price_usd': round(float(jetton['last_price'])*ton_usd,4),'volume_usd': round(float(jetton['quote_volume'])*ton_usd,2),"volume_perc": round((float(jetton['quote_volume'])/all_volume)*100,10)}
    if(jetton['quote_symbol']=='TON' and int(float(jetton['last_price']) != 0) and int(float(jetton['quote_volume']) != 0) and int(temp_dict['volume_usd']) > 1000):
      temp_arr.append(temp_dict)

  return sorted(temp_arr, key=lambda d: d['volume_usd'],reverse=True)


payload = {'since': day_utc(), 'until': now_utc()}
take_pool_stats(payload)

Let's put the resulting list in a dataframe of the Pandas library for ease of display and get:

Using such a simple example, you can test your hypotheses related to tokens, for example, by looking at the dynamics over several days or by considering different periods. But almost any study leads to the need for detailed information on pools. After all, it is perfectly clear that you can increase the volume from a couple of wallets, thus accelerating the token. To do this, you need to collect information on operations.

Collecting information on operations

The StoneFi API has a separate method that returns operations. Conveniently, for each operation the pool in which the operation was performed is registered. We will use this to count the number of swaps by pool and see how many unique wallets are among these operations.

First, let’s get the operations for the period:



def take_operations(payload):
  r = requests.get('https://api.ston.fi/v1/stats/operations', params=payload)
  return r.json()["operations"]

Operations in pools are different, there are swaps, when users exchange tokens and TON, there is the addition of liquidity to pools, possible operations may differ from exchange to exchange. For our simple example, we will only take swaps.

Let’s assemble a function that, for each pool we select, will return the number of swaps and the number of unique wallets that made these swaps.



def count_pool_operations(operations,addr_str):
    pool_oper_list = list(filter(lambda person: person['operation']['pool_address'] == addr_str, operations))
    unique_counts = collections.Counter(e['operation']['destination_wallet_address'] for e in pool_oper_list )
    return len(pool_oper_list),len(set(unique_counts))

Let's add this information to the dashboard:




def take_pool_stats(payload):
  temp_arr=[]

  # stonfi pools stats
  r = requests.get('https://api.ston.fi/v1/stats/pool', params=payload)
  resp = r.json()['stats']
  # ton_usd
  ton_usd = take_ton_price()
  # for volume percentage
  all_volume = sum(float(item['quote_volume']) for item in resp)
  # take operations
  payload['op_type'] = 'Swap'
  operations = take_operations(payload)
  for jetton in resp:
    temp_dict = {'pool_address': jetton['pool_address'],'coin': jetton['base_name'],'pair': jetton['base_symbol']+'/'+jetton['quote_symbol'],'url': jetton['url'],'price_ton': jetton['last_price'],'volume_ton': jetton['quote_volume'], 'price_usd': round(float(jetton['last_price'])*ton_usd,4),'volume_usd': round(float(jetton['quote_volume'])*ton_usd,2),"volume_perc": round((float(jetton['quote_volume'])/all_volume)*100,2)}
       if(jetton['quote_symbol']=='TON' and int(float(jetton['last_price']) != 0) and int(float(jetton['quote_volume']) != 0) and int(temp_dict['volume_usd']) > 1000):

      temp_pool_count = count_pool_operations(operations,temp_dict['pool_address'])
      temp_dict["count_swaps"] = temp_pool_count[0]
      temp_dict["count_unique"] = temp_pool_count[1]
      temp_arr.append(temp_dict)



  return sorted(temp_arr, key=lambda d: d['volume_usd'],reverse=True)

payload = {'since': day_utc(), 'until': now_utc()}

df = pd.DataFrame(take_pool_stats(payload))

Result:

A spoon of tar

The example we collected above shows how easy it is to collect data on pools and operations, but there are also problems.
Since the API’s task is to provide statistical data, based on the price of tokens we get the last price for the period, which is not very correct from an analytical point of view. Therefore, if the price of a token for a period is important to us, it will be important to calculate the time-weighted price, which will lead to an increase in the number of API requests.
The second problem with such data collection is the time it takes to complete one request; the APIs of decentralized exchanges are not adapted for deep queries, so if you are interested in some kind of live analytics, then the only way out is to collect data directly from smart contracts - calling get methods .

Conclusion

I like the TON blockchain for its technical elegance; at least it’s not another copy of Ethereum, which is being overclocked with the help of a lot of capital without looking back, and in general why the user needs it. If you want to learn more about the TON blockchain, I have open source lessons that will teach you how to create full-fledged applications on TON.

https://github.com/romanovichim/TonFunClessons_Eng

I post new tutorials and data analytics here:

https://t.me/ton_learn

How to rank Fungible Tokens in the TON blockchain by transactions

Ivan Romanovich 🧐 — Tue, 19 Dec 2023 10:30:27 +0000

Introduction

Often in the crypto world, the words decentralization and smart contracts are just a shell for scam or, to put it mildly, dishonest schemes.

Those eager to make money on the waves of hype are faced with the problem of researching and finding projects, which leads them to the need to research the blockchain and collect information.

But the TON blockchain, due to its asynchronous nature and sharding of smart contracts, is difficult to retrieve data; almost any action consists of chains of several interacting smart contracts that send messages to each other. Therefore, using the example of a simple task, I decided to show how the process of searching for the necessary information can be arranged.

At TON, the last couple of months have been all about Tokens, a standard for fungible tokens. Many projects are emerging and some kind of starting point in research is needed.

In this article we will figure out how to find transactions that include the transfer of tokens and rank some of the tokens by the number of transactions. And all this from the perspective of how the process of data excavation can generally be structured.

Disclaimer: This article does not advertise any projects, but only offers insight into collecting data through a blockchain indexer on a small application problem.

Where will we get the data from?- dton.io

Dton.io is a blockchain indexer, which means it collects information from each new block into its database. You can make GraphQL queries into this database and thus collect historical information without parsing the entire blockchain.

The two most important sources of dton.io data are the transaction table and a view of the latest state of wallets/accounts on the network. There are also various auxiliary views.

In this tutorial, we will need a table of transactions and a view with a pre-calculated number of certain transactions.

How to find the necessary transactions

The dton.io indexer allows you to retrieve information using GraphQL queries, and here we have a query fish:

query = """
query MyQuery {

}
"""

What to do next? - there are more than a hundred possible arguments. The algorithm is simple in words, complex in details:
We study smart contracts that interest us
Then capture the message chains between contracts that we need
From the two steps above we get the arguments for our request

Let's start with smart contracts.

What are Jettons?

Jetton is a standard fungible token in the TON network. In order for tokens to be used in other applications (from wallets to decentralized exchanges), standards for smart contract interfaces for tokens are being introduced. The interface in this case is a certain signature - (the syntactic construction of a function declaration.

Smart contract standards require public discussion. In TON it happens here - https://github.com/ton-blockchain/TEPs/blob/master/text/0074-jettons-standard.md

The basic logic is written inside, and there is also some simple implementation.

Consider the Token standard

We will not analyze the Jetton standard in detail, since I have a separate article about it -
https://github.com/romanovichim/TonFunClessons_ru/blob/main/lessons/smartcontract/9lesson/ninthlesson.md. Let us note the key ideas at this stage:

Token standard The token must consist of two types of smart contracts:

master contract
contract wallet

Each Token has a main smart contract (we'll call it a master contract), which is used to mint new Jettons, account for the total supply, and provide general information about the token.

Information about the number of tokens owned by each user is stored in smart contracts called Jetton wallet.

Internal transfer

After the initial familiarization with smart contracts, you need to pay attention to op codes - this is the code that displays what the smart contract should do.

After a quick look at the op codes, you need to identify those that suit your task and draw the process. Let's look at the process:

sender sends an op::transfer message to its jetton wallet
sender_jetton_wallet reduces the token balance
sender_jetton_wallet sends an op::internal_transfer message to the recipient's jetton wallet
recipient_jetton_wallet increases its token balance
recipient_jetton_wallet sends op::transfer_notification to its owner (recipient ton wallet)

there is also an excess return, but we will omit this within the article, I suggest the reader familiarize himself with the standard

After this, for convenience, we will draw the process of sending Jettons:

Judging by this scheme, we need internal transfer; it is messages with internal transfer that will allow us to collect the sending of Jettons from wallet to wallet.

For more complex chains of contracts, it is also important to specify what data is transmitted internally, since not every message carries information about a certain business transaction - more often than not, only the required minimum. This is especially relevant for DEX. Once again, at this stage we are looking for transactions that allow us to collect the necessary information.

Collecting information about one Jetton

After we have determined the necessary transactions, we will reduce the task to the minimum - in our case, this is searching for information about one Jetton.

For your research, I suggest the following rule of thumb - reduce the task until it can be checked in Explorer. That is, when you run a query, you can find the transaction and understand whether you found the correct information.

Let's denote the endpoint dton.io (all the code from this tutorial will be available at the end in Google Colab):

endpoint = 'https://dton.io/graphql/'

Let's create an empty request:

query = """
query MyQuery {

}
"""

response = requests.post(endpoint, json={'query': query})

if response.status_code == 200:
  data = response.json()['data']
  print(data)
else:
  print(response.json())

For our goal - counting the number of Jetton transactions, there is a convenient representation of lastTransactionCountSegments that we will use. Inside we will put the required op_code - internal transfer and the address of the master contract of any token (since this is not an advertising article, I won’t say what kind of token it is, if you want to try some other token, you can get it here - the links immediately contain hex address format https://tonlearn.tools/#/jetton-dash).

In this case, we can take op_code directly from the standard to convert it from bytes to 32-integer, we will use this site: https://cryptii.com/pipes/integer-encoder

По итогу получим:

endpoint = 'https://dton.io/graphql/'

query = """
query MyQuery {
  lastTransactionCountSegments(
    parsed_jetton_wallet_jetton_address_address: "F4BDD480FCD79D47DBAF6E037D1229115FEB2E7AC0F119E160EBD5D031ABDF2E"
    in_msg_op_code: 395134233
      segmentation: "day"
      days: 30) {
    segment
    cnt
  }
}
"""

response = requests.post(endpoint, json={'query': query})

if response.status_code == 200:
  data = response.json()['data']
  print(data)
else:
  print(response.json())

Now that we have learned to collect information from one token, we can move on.

Let's move on to ranking

List of Jettons

To have something to rank, you need some list of master contracts of Jettons; it could be collected by simply making queries into the table of contract states on the network - account_states. But this makes little sense - we will get a huge list of Jettons, some of which will have a maximum of 1 - 2 transactions. Therefore, I propose a simple method, take addresses from the public list of Gettons, which one of the popular wallets on TON displays in its interface:

url = "https://raw.githubusercontent.com/tonkeeper/ton-assets/main/jettons.json"
resp = requests.get(url=url)
jettons_list = resp.json()

jettons_list[0]

It is important to note that this is an open source list and there are errors in it, for example, sometimes addresses are stored in different formats. Therefore, when constructing the next query, we will need to clean out the unnecessary.

Constructing the request

To collect information in the context of Jetton, you just need to concatenate queries:

formatted_query_cnt =""

for token in jettons_list:
  # if not hex skip
  if(token['address'].upper()[0:2]!='0:'):
    #print(token['address'].upper()[0:2])
    continue
  #cut workchain from address
  temp_addr = token['address'].upper()[2:]
  template = string.Template("""
    seq${addr_cut}: lastTransactionCountSegments(
      in_msg_op_code: 395134233
      segmentation: "day"
      days: 30
      parsed_jetton_wallet_jetton_address_address: ${addr_dop}
    ) {
      segment
      cnt
    }
  """)

  formatted = template.substitute(addr_cut = temp_addr,addr_dop= '"'+temp_addr+'"')
  formatted = formatted +"\n"
  formatted_query_cnt += formatted


formatted_query = """query {"""+formatted_query_cnt+"""}"""
response_cnt = requests.post(endpoint, json={'query': formatted_query})
response_cnt

The concatenation limit is 100, but in my experience, it is important to evaluate how complex the computable query is and go from there.

Sort and discuss the results

Afterwards, we will collect and sort the data, I will not add this code to the article, everything is in Google Colab:

https://colab.research.google.com/drive/1h6fFy2OaTntlUXNKJLaVztuOGH-A_NIg?usp=sharing

The result is something like this:

I note that this is a training task; it is quite difficult to rank Jettons by transactions applicable to real tasks, since many Jettons increase the volume of transactions on the DEX and such transactions need to be cleaned out.

Conclusion

https://github.com/romanovichim/TonFunClessons_ru

I post new tutorials and data analytics here: https://t.me/ton_learn

Code for this tutorial:

https://colab.research.google.com/drive/1h6fFy2OaTntlUXNKJLaVztuOGH-A_NIg?usp=sharing

How TON-20 and BRC-20 work and why they create such problems for blockchains

Ivan Romanovich 🧐 — Wed, 13 Dec 2023 15:53:27 +0000

Last week, an incident occurred in TON - a huge load on one of the addresses created a queue of transactions, the blockchain itself withstood the load - the sharding system worked, but nevertheless, due to the queue, as well as as a result of the load on the indexers, the blockchain was paralyzed for a couple of days until the line “diverged”.

Such load was caused by a new type of token TON-20, an analogue of BRC-20 from the Bitcoin network. There is almost no information in the TON-20 docs, except that the message format copies the BRC-20. If you look at the blockchain explorer, a person familiar with smart contracts development will be surprised - a bunch of ordinary transactions with json inside messages.

But since this is some kind of fungible token, then there should be some way to get the balance and check the transfer of tokens and so on?

I will try to answer these questions in this article. We will look at how BRC20 “tokens” work and why they lead to problems for blockchains and blockchain network infrastructure projects.

Place in transaction

BRC-20 is the concept of a fungible token in the Bitcoin network. The concept emerged after the success of Bitcoin Oridinals, a protocol that allows the creation of NFTs on the Bitcoin network. This became possible thanks to:

OP_return - A function that adds an additional output to transactions on the Bitcoin network that contains third-party information (Inscriptions), such as metadata, but does not contain funds. Initially, the data volume in OP_return had a limit of 80 bytes.
An update to the Bitcoin Taproot network that increased the amount of information in OP_returen to 400 KB.

Thus, with a transaction in Bitcoin, it became possible to send enough information to implement the token logic.
Looking ahead, I will say that the author of the BRC-20 concept notes that this is an experiment, and not the best practice, which has the right to be. But as soon as money flowed into such tokens, everyone forgot about it.

BRC-20

The idea behind brc-20 is to send three types of metadata with a transaction:
Deploy - for the initial creation of a brc-20 token
Mint - to receive an already created brc-20 token
Transfer - for transferring such tokens
Metadata example:

And now the most interesting thing is how the “protocol” tracks how many tokens, for example, I can transfer to my friend: A wallet that can process such a standard checks all transactions and thus understands whether the transaction is valid. Thus, this is a certain chain of transactions, within the blockchain there are also transaction chains.

The same is true with the balance; only records are stored in the blockchain itself, and all other logic is moved offchain to wallets or services compatible with brc-20.
It’s difficult to even call such mechanics a digital asset, which is why some developers believe that transactions with brc-20 should be considered spam. And filter them accordingly so that they do not clog the mempool.

So that you understand the whole concept is described in just a couple of pages: https://domo-2.gitbook.io/brc-20-experiment/
At the same time, at the time of writing, in the Bitcoin network there are more than $1 billion of tokens with a total capitalization according to https://www.brc-20.io/. A full-fledged market, that’s the power of marketing.

House of cards

Why does such a primitive concept lead to problems? Let's look at the example of TON with its copy of the TON-20 concept.
As you can understand from the concept, in general, it does not matter where to send transactions, since only the fact of a transaction with the data is important, then everything is offchain. Accordingly, if we want to create a wallet that works with TON-20 or some kind of indexer site for this concept, then we need a powerful blockchain indexer, according to which we will receive the balance and generally count which transactions are valid and which are not.

And here lies an important detail: if we work with ordinary fungible tokens, then the depth of requests (how many blocks ago) is small. The latest states of smart contracts store balances, and the logic inside the smart contract does not require parsing a bunch of addresses to understand whether Petya can transfer tokens to Alice.

In TON-20 and BRC-20 it’s the other way around; constant deep queries are needed for a large number of addresses. All this creates a heavy load on indexers.

You say, well, you can collect your own indexer and immediately present the data in a form convenient for collection - for each block, count all transactions, mints and deployments of tokens. But in reality, creating such an indexer is a difficult and expensive task, so public indexers with APIs are used.

What's a simple way to reduce the load in this configuration? Throw all transactions in the blockchain to one address, because we don’t care where the transaction came from, only what’s in it is important. Thus, to obtain data about a token, you will simply need to request all transactions of one address, and then count them and do not need to index the entire blockchain.

This is exactly what the company did when it transferred the BRC-20 to TON. They suggested sending all transactions from TON-20 to one address - the burning address, it looks like this:

You can watch it here: https://tonscan.org/address/EQAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAM9c
But the problem with this approach is that blockchains are not designed for a huge number of transactions per address, even such as TON. TON holds the record for the number of transactions per second - about 100,000, but not per address, this is an important nuance. Also, some indicators used by many services on TON were not ready. Thus, a queue of transactions was formed, which pulled the services on the blockchain with it.

The last layer of the house of cards, which aggravated the problem when there were many such transactions to one address, were validators, the same ones that support the network; it turned out that many used weak hardware - sometimes it’s easier to pay fines than to constantly rent expensive hardware.

All this led to a transaction queue, yes, the blockchain survived, and the queue resolved quickly, but nevertheless, it’s funny how some messages in transactions around which offchain built logic led to such problems.

Some thoughts

The problems that arise from such technically strange solutions lead to the difficult problem that blockchain creators face. They have to create not just a technical platform that is easy to change manually in case of problems, but an ecosystem where it is difficult to align the incentives of the participants creating their products within.

If you look at the problem that arose in TON, you can see that no one had malicious intent. Everyone acted economically rationally. And in such a situation it is easy to blame that, for example, indexers must carry a huge load. But if we put ourselves in the shoes of the owners, we will understand that the economy simply will not work out if we keep a bunch of “hardware” in reserve just in case.

The answer to the problem of coordinating economic incentives can be found in economic theory - competition, well-organized access to resources, such as grants and investments. Will TON Foundation be able to cope with this? Nobody knows, but one thing is for sure - the competition between blockchains takes place not only in the technical part or in terms of the volume of investments in them, but also in the ability to build a competent ecosystem within.

Conclusion

https://github.com/romanovichim/TonFunClessons_Eng

I will be glad to see your stars on the repository. Also I post new tutorials and data analytics here: https://t.me/ton_learn

Looking for scam tokens using bubble charts in TON blockchain

Ivan Romanovich 🧐 — Thu, 09 Nov 2023 06:06:07 +0000

Often in the crypto world, the words decentralization and smart contracts are just a shell for scam or, to put it mildly, dishonest schemes.

You can hear Do Your Own Research - do research, but an ordinary person, not a venture capital firm with a staff of analysts, cannot spend so much time on analysis.

What can be done?

Without canceling DYOR, we can try to find an indicator/dashboard that, with 20% of the effort to study, will give 80% of the result - information about the token.

What kind of indicator? - I propose to consider the distribution of the supply of tokens.

Let's look at the cases:

1) Manipulations with floor and NFT volume

NFT buyers often focus on floor (the lowest price of a collection) and collection volume - dishonest sellers influence these indicators by reselling NFTs between a pair of wallets.

If you display resales on a dashboard relative to the overall distribution, such manipulations will be immediately visible.

2) Concentration of 25% of fungible tokens in one hand

The creators of tokens often flaunt a large number of token transactions, look, they use our token, although often it is simply transferring tokens between a pair of interconnected wallets - the token ecosystem is active, but the real owner is actually one...

3) Fake DAOs

An important part of a DAO is voting management - it is important to understand whether the voting will be real or just legitimizing the decisions of a couple of people. If a couple of people own almost the entire supply, it is not a DAO.

You will say, this is certainly cool, but going through a bunch of wallets will take a lot of time…

Bubble charts come into the picture

The best way to visualize token distribution is with a bubble chart; this will allow you to quickly see suspicious patterns.

In this tutorial we will put together a script that will chart the top 100 wallets by balance for any Jetton in the TON blockchain!

In the next tutorial we will take these top 100 wallets and find transactions among them, thus highlighting the clusters.

The script will be in python, the data will be taken from the dton.io indexer. We will focus on obtaining data.

The result will be a Google Colab that you can use in your research.

How to collect data from the TON blockchain?

The two most important sources of dton.io data are the transaction table and a view of the latest state of wallets/accounts on the network. Make queries in this view and table, you can collect almost any information.

In this tutorial, the states of the wallets will be useful to us; based on the state, we will determine whether there are tokens

Install dependencies

As we know, GraphQL requests are sent in the body of an HTTP POST request, so let’s install requests and other necessary dependencies:

# install requriments
!pip install requests
!pip install matplotlib 

import requests
import string
from collections import Counter
import string
import json

Let us also denote the endpoint dton.io:

endpoint = 'https://dton.io/graphql/'

Get the contract type and information about the token.

For our script to work, we need a master contract of a fungible token for TON. We will discuss what this is below. But since I want to make Colab easy to use, I first need to learn how to check that the entered address is exactly what is needed.

There are many options for this, you can, for example, take all the information from the network - take the low-level code of the contract and compare it with the requirements of the fungible token standard, but there are simpler options.

The simplest option, where with one request we can check what kind of contract it is by address and immediately get information about the token, is the option to make a request in toncenter - this is the HTTP API for The Open Network. Toncenter has /getTokenData for this:

#Note api has limits
API_ENDPOINT = 'https://toncenter.com/api/v2'
payload = {'address': master_addr}
r = requests.get(API_ENDPOINT+'/getTokenData', params=payload)
#print api response
print(json.dumps(r.json(), indent=2))

# Token supply
total_supply = r.json()['result']["total_supply"]

# Token name for our future diagram
token_name = r.json()['result']["jetton_content"]["data"]["name"]

In the resulting result, from the “contract_type” field you can understand what type of smart contract it is (if such a type is supported by the service).

Constructing a request in dton.io

Now we know for sure that the address entered by the user is the address of the master contract, it’s time to talk about the standard of fungible tokens.

In the TON blockchain there is no single smart contract that stores a list of addresses of the owning tokens and their balances. Instead, sharding of smart contracts is used - there is a master contract that stores information about the token and contract wallets - for each owner of the token, one is created.

We will not dwell in detail on the sharding of tokens in TON in this article; if you are interested in this, I offer you an analysis of smart contracts of the standard - ENG LINK.

Since the use of a smart contract requires “payment for the work” of TVM in gas (part of the TON for any transaction goes as a commission), this makes such a standard extremely economical - there is no need to search in a huge list of addresses.

But if there are pros, then there are also cons.

The problem of smart contract sharding from the point of view of data collection

Sharding saves gas, but there is no list of owners - then how to get all the addresses of the token owners? The only option is to run through all the blocks and find the addresses we need. But fortunately, there are blockchain indexers that allow us to get the addresses we need with a couple of queries.

In the dton.io we already mentioned, there is a representation of the latest state of wallets/accounts on the network. Accordingly, we need to add restrictions to the request of this view that will allow us to display the top 100 wallets.

Collecting a request in dton.io

Dton.io not only indexes the blockchain, but also enriches the data. It is important for us that in the representation of accounts on the network there is a field parsed_jetton_wallet_jetton_address_address - for token wallets this field is filled in with their master contract, which is what we will use. To sort only contracts and token wallets, let’s check that the contract has a get_wallet_data method (this method is required for token wallets according to the standard).

query MyQuery {
  account_states(
    account_state_state_init_code_has_get_wallet_data: 1
    parsed_jetton_wallet_jetton_address_address: "${addr}"
    workchain: 0


  ) {


  }
}

Let's sort by balance and select the first 100 records on the page.

query MyQuery {
  account_states(
    account_state_state_init_code_has_get_wallet_data: 1
    parsed_jetton_wallet_jetton_address_address: "${addr}"
    workchain: 0
    order_by: "parsed_jetton_wallet_balance"
    order_desc: true
    page_size: 100
  ) {
  }
}

All that remains is to specify that we want to get the address of the owner of the token wallet and its balance. We get:

query MyQuery {
  account_states(
    account_state_state_init_code_has_get_wallet_data: 1
    parsed_jetton_wallet_jetton_address_address: "${addr}"
    workchain: 0
    order_by: "parsed_jetton_wallet_balance"
    order_desc: true
    page_size: 100
  ) {
    address
    parsed_jetton_wallet_balance
    parsed_jetton_wallet_owner_address_address
  }
}

How to enrich data

I think you understand that exchanges or service smart contracts can have many outgoing transactions, it would be cool to mark them so as not to consider that anything is a scam, because people simply exchange tokens through the exchange. But where to get the necessary base? Let's start from the problem of who needs such a dataset most of all - of course, wallets, because it will also give them UX - it is convenient for users to see who is sending them funds, for example + plus this is some protection against spam - if the token is not in the whitelist, then perhaps it is a scam or advertising sending of a token (they throw a token and in the message there is a link somewhere).

Therefore, let’s pay attention to the open source lists from the tonkeeper wallet:

https://github.com/tonkeeper/ton-assets

Of course, these are not all possible combinations of wallets and their owners, and one owner may have many wallets, but it’s better than nothing.

After enrichment, we will see that a small share of the token is owned by the MEXC exchange:

Bubble chart

All that remains is to build a diagram, prepare random colors (code below in the link) and get:

But can scammers use multiple wallets?

Yes, and that is why in the second part we will track transactions between the top 100 wallets and collect clusters.

Conclusion

https://github.com/romanovichim/TonFunClessons_Eng

I post new tutorials and data analytics here:
https://t.me/ton_learn

Code from this tutorial: https://colab.research.google.com/drive/10qv2AN68F3IOwfJyLXsDiQenOhBvyN6I?usp=sharing

How to find the richest wallets in the TON blockchain - get data from the dton.io indexer and Python

Ivan Romanovich 🧐 — Thu, 28 Sep 2023 10:26:17 +0000

News about rich people and investments of large funds are always on the front pages. People are interested in what people with large capital buy and invest in.

Blockchain allows you to satisfy curiosity on a new level, because all information is publicly available in full. In this article I will tell you how to get whale wallets of the TON blockchain using a request to the dton.io indexer.

We’ll also talk about how you can simply enrich your data. An example of what we get at the end can be seen here:

https://tonlearn.tools/#/ton-toncoin-whales

What is dton.io?

In this tutorial, we will need the states of wallets; we will take the balance

Install dependencies

As we know, GraphQL requests are sent in the body of an HTTP POST request, so let’s install requests and other necessary dependencies:



# install requriments
!pip install requests

import requests
import string

Let's add the dton.io endpoint:



endpoint = 'https://dton.io/graphql/'

Let's get the bills

Let's use the account status view:



 query = '''
  {
    account_states(
      workchain: 0
      page_size: 100
    ){
      balance: account_storage_balance_grams
      address
    }
  }
  '''

Let's get some addresses and their balances, the question arises: how to get the top 100? To do this, the indexer has an order by field in which you can write a field by which to order the answer.

Order the selection

Since we are interested in balance, let's add sorting by the account_storage_balance_grams field.



#how many sales were there in the last 24 hours
def getTopHundredTonOwners(endpoint):
  query = '''
  {
    account_states(
      order_by: "account_storage_balance_grams"
      order_desc: true
      workchain: 0
      page_size: 100
    ){
      balance: account_storage_balance_grams
      address
    }
  }
  '''

  response = requests.post(endpoint, json={'query': query})

  data = response.json()['data']['account_states']
  return data

These are the top 100 wallets, but what kind of wallets are they, maybe they are exchanges or some kind of technical wallets?

How to enrich data

Of course, most of the top 100 are just wallets, but there are also exchanges and special ones. wallets and it would be cool to compare the address and its name, if available. But where to get the necessary base? Let's start from the problem of who needs such a dataset most of all - of course, wallets, because it will also give them UX - it is convenient for users to see who is sending them funds, for example + plus this is some protection against spam - if the token is not in the whitelist, then perhaps it is a scam or advertising sending of a token (they throw a token and in the message there is a link somewhere).

Therefore, let’s pay attention to the open source lists from the tonkeeper wallet:

https://github.com/tonkeeper/ton-assets

Of course, these are not all possible combinations of wallets and their owners, and one owner may have many wallets, but it’s better than nothing.

We enrich and the explorer trick

It would also be convenient if we had not just an address, but a link to the blockchain explorer with this address. To do this, we need to put the address in another form.

Addresses of smart contracts running in TON are expressed in two main formats:
Raw hex addresses: raw full representation of smart contract addresses
User-friendly addresses: extended addresses for user convenience
More information about addresses in documentation. To convert an address from one type to another, we will use a script, which we will not analyze in this tutorial, but at the very end of the tutorial there will be all links to the code.

Let's put it all together:



url = "https://raw.githubusercontent.com/tonkeeper/ton-assets/main/accounts.json"
resp = requests.get(url=url)
assets_list = resp.json()

for row in top_hundred:
  for asset in assets_list:
    if('0:'+row['address'].lower() == asset['address']):
      row['name'] = asset['name']
      print(asset['name'])
    else:
      pass
      #row['name'] = None

  row['tonscan'] = "https://tonscan.org/address/" + account_forms('0:'+row['address'])['bounceable']['b64url']
  row['normalized_balance'] = round(int(row['balance']) /1000000000)

We get:

What can be improved

In the next tutorial we will add what percentage of TONcoin whales own from the total supply of TONcoin and cumulative.

Conclusion

https://github.com/romanovichim/TonFunClessons_Eng

I post new tutorials and data analytics here: https://t.me/ton_learn

You can see the ranking of collections from this tutorial in a convenient form here: https://tonlearn.tools/#/ton-toncoin-whales

Code from this tutorial: https://colab.research.google.com/drive/16AiDKEwHi_GX0IY3cATjHAZHkO0dHu6o?usp=sharing

How to find popular NFT collections in the TON blockchain - ranking collections using the dton.io indexer and Python

Ivan Romanovich 🧐 — Thu, 21 Sep 2023 09:44:44 +0000

When trading any asset, you need to understand the current state of the market, and for NFTs this is no exception. In this tutorial, I will show you how to collect information on sales volume by collection over the last 24 hours in the TON blockchain.

Why is this necessary?

Before diving into the code and how the dton.io indexer works, let’s talk about why we might need it. Let's say we want to purchase an NFT for further resale, which, roughly speaking, is important to us:

Increasing the price of a collection - in order to resell it for more than what we bought
Liquidity - so that if the price rises, a buyer can be found
A rough understanding that the first two points are not a consequence of fraudulent actions - for example, laundering trading (how to search for laundering transactions in collections was described in another article)

If we move away from general words about liquidity and price growth, then the algorithm could be as follows:

Understand the current general state of the market and specific blockchains
After deciding on the favorable state of the market for purchasing and choosing a blockchain, find out which collections are currently popular on this blockchain
Analysis of certain collections - look at the historical floor, sales history, number of unique owners, etc.
After choosing a collection, look for undervalued items in them to purchase

Of course, this is not the only strategy - you can look at what collections are on the go or what whales are buying, but still, the step of reviewing the state of the market and searching for popular collections is now difficult to skip in order to understand the overall picture.

An example of the result we will get can be seen here:

https://tonlearn.tools/#/ton-nft-sales-volume

How can dton.io help you find information?

It is important to note that the dton.io indexer collects not only raw data from the TON blockchain, but also enriches the data - this is very convenient, as we will see in the example below.

Installing dependencies

As we know, GraphQL requests are sent in the body of an HTTP POST request, so let’s install requests and other necessary dependencies:

# install requriments
!pip install requests

import requests
import string 
import time
from itertools import groupby

Let’s also denote the endpoint dton.io:

endpoint = 'https://dton.io/graphql/'

Let's calculate how many NFT sales there are in total

Since we want to count the number of transactions over a certain time, we will use the lastTransactionCountSegments view:

#how many sales were there in the last 24 hours
def getDaySaleNum(endpoint):
  query = '''
    query {
    lastTransactionCountSegments(
    segmentation: "hour"
    ) { cnt segment}
  }

This view returns the number of transactions per segment, at most an hour. Since data can be received by pages and specify the number of data per page (maximum 150), you can get data for 24 hours by adding page_size: 24 But we don’t need all transactions, but only sales…. .

To understand how to cut off sales, you need to understand the standards of smart contracts, specifically here you need to understand how the sale of NFTs in TON works. I have a separate tutorial with an analysis of smart sales contracts, and in this tutorial I will say that the main thing for us is that sales smart contracts have a special get-method get_sale_data, which will allow us to extract everything from sales related transactions.

Since several transactions are involved in the sale, we need one that kind of closes the sale, and here the fact that dton.io has data enrichment comes to the rescue; the parsed_seller_is_closed: 1 field will help to understand whether a transaction closes the deal. We get:

def getDaySaleNum(endpoint):
  query = '''
    query {
    lastTransactionCountSegments(
      segmentation: "hour"
      account_state_state_init_code_has_get_sale_data: 1
      parsed_seller_is_closed: 1
      page_size: 24
    ) { cnt segment}
  }
  '''

  response = requests.post(endpoint, json={'query': query})

  data = response.json()['data']['lastTransactionCountSegments']
  dayTxes = sum(item['cnt'] for item in data)
  return dayTxes

At the end, we sum it up and get the number of NFT sales in the TON network in 24 hours.

An experienced reader may say that this does not take into account all possible situations of the right to transfer an NFT, since there are three of them:

sale
auction
offer (offer to purchase with subsequent purchase) And indeed, this method covers sales and auctions, but within the framework of a business task, offers should be placed in a separate dashboard, since they do not particularly reflect the picture as a whole and require detailed consideration separately.

Collect all 24 hour sales

Now we know how many transactions we need to get. Let's take into account that since only 150 values can be obtained on the page, we will have to send requests in batches, concatenating them into one request. I’ll say right away that extra sales will not be cut off, so as not to complicate the tutorial:

day_sales_count = getDaySaleNum(endpoint)

page_size = 100

needed_queries = day_sales_count // page_size + 1

# собираем запрос
formatted_query =""
for batch_num in range(needed_queries):
  print(batch_num)
  template = string.Template("""
    q${page}: transactions(
      account_state_state_init_code_has_get_sale_data: 1
      parsed_seller_is_closed: 1
      workchain: 0
      page_size: ${page_size}
      page: ${page}
      ) {
      collection: parsed_seller_nft_collection_address_address
      price: parsed_seller_nft_price
      nft: parsed_seller_nft_address_address
      block_time:gen_utime
      }
  """)

  formatted = template.substitute(page=batch_num,page_size=page_size)
  formatted = formatted +"\n"
  formatted_query += formatted

formatted_query = """query {"""+formatted_query+"""}"""
formatted_query
response = requests.post(endpoint, json={'query': formatted_query})

res_arr = []

for v in response.json()['data'].values():
  for item in v:
    if(item['collection'] is not None):
      res_arr.append(item)

print("Collected txes: ",len(res_arr))

You may notice that we do not take NFTs without a collection (where None), and the NFT standard in TON implies the possibility of NFTs without a collection, since unlike EVM networks, NFT is not one contract storing a certain database of records, but one smart collection contract and separate contracts for each element.

Also, a reader well familiar with blockchain technology can say, why make two requests, first how many, and then transactions, you can get the block generation time with each transaction, and by unloading all transactions in a row, you can break out of the cycle on blocks that were 24 hours. Yes, this is true, but for ease of code readability and to show you more about dton.io, I decided to do it through two requests.

Let's move on.

Group and count the number of transactions and sales volume

Regarding the collection, let’s sum up the sales, this will be the sales volume, and also calculate the number of transactions. Why do we need both quantity and amount? One amount does not reflect well the situation in the collection, one large sale per day is worse for us than many medium ones - liquidity is important for resale.

#We group the data by collection - collection, number of sales, sales volume and of course sort
# define a function for key
def key_func(k):
    return k['collection']

# sort INFO data by 'company' key.
INFO = sorted(res_arr, key=key_func)

count_arr=[]

for key, value in groupby(INFO, key_func):
    #print(key)
    #print(list(value))
    sum=0
    count=0
    for item in list(value):
      count = count + 1
      sum = sum + int(item['price'])
      #print(item['employee'])

    #print("Txes:",count)
    #print("Sum:",sum/1000000000)
    temp_d = {'col': key, 'txes': count, 'sum': round(sum/1000000000)}
    count_arr.append(temp_d)


newlist = sorted(count_arr, key=lambda d: d['sum'], reverse=True)
print("Size of sorted Arr: ",len(newlist))

Let's see what we have:

newlist[0:5]

[{'col': 'B774D95EB20543F186C06B371AB88AD704F7E256130CAF96189368A7D0CB6CCF',
  'txes': 59,
  'sum': 332419},
 {'col': 'E06664290C96CEEA5687BEF89B6976173251006041FA2E752D039275F56D7C74',
  'txes': 2,
  'sum': 100098},
 {'col': '28F760D832893182129CABE0A40864A4FCC817639168D523D6DB4824BD997BE6',
  'txes': 32,
  'sum': 12053},
 {'col': 'D398C94E9AB5BE730AA5E4DBCEF4754F3FB1B6805B61BEB15B31052E6C0E838C',
  'txes': 106,
  'sum': 11737},
 {'col': 'B3B928E4814343EB597B8081F736F55636D674457D61C73D6B5A7989A863E666',
  'txes': 3,
  'sum': 10052}]

It looks like it’s not very clear what kind of collection it is, and what kind of cumbersome address it is.

Addresses in TON and a little trick with TON blockchain explorer

It would also be cool to get information about the collection, the name, a picture of the collection, and generally see what the collection is. You can, of course, get all the data from the blockchain, but the NFT standard in TON is not focused on quick access to the entire collection, so for the first step, I suggest taking advantage of the fact that one of the explorers allows you to view collections by the collection address. Let's take advantage of this.

for row in newlist:
  row['explorer_link'] = "https://tonscan.org/nft/"+account_forms('0:'+row['col'])['bounceable']['b64url']

newlist[0:5]

We get:

[{'col': 'D398C94E9AB5BE730AA5E4DBCEF4754F3FB1B6805B61BEB15B31052E6C0E838C',
  'txes': 108,
  'sum': 289703904,
  'explorer_link': 'https://tonscan.org/nft/EQDTmMlOmrW-cwql5NvO9HVPP7G2gFthvrFbMQUubA6DjH7-'},
 {'col': 'B774D95EB20543F186C06B371AB88AD704F7E256130CAF96189368A7D0CB6CCF',
  'txes': 34,
  'sum': 510053,
  'explorer_link': 'https://tonscan.org/nft/EQC3dNlesgVD8YbAazcauIrXBPfiVhMMr5YYk2in0Mtsz0Bz'},
 {'col': '80D78A35F955A14B679FAA887FF4CD5BFC0F43B4A4EEA2A7E6927F3701B273C2',
  'txes': 20,
  'sum': 301248,
  'explorer_link': 'https://tonscan.org/nft/EQCA14o1-VWhS2efqoh_9M1b_A9DtKTuoqfmkn83AbJzwnPi'},
 {'col': 'A27781584DA7136F20D0D3BC1ECEC5D00BCC82C720E08BF0A81BD3063D704628',
  'txes': 13,
  'sum': 30868,
  'explorer_link': 'https://tonscan.org/nft/EQCid4FYTacTbyDQ07wezsXQC8yCxyDgi_CoG9MGPXBGKLJ_'},
 {'col': '82AADDD0778EE241E576637F17D1DA83E85AA1CAC4D7B47D5B075D6A089CB218',
  'txes': 264,
  'sum': 17350,
  'explorer_link': 'https://tonscan.org/nft/EQCCqt3Qd47iQeV2Y38X0dqD6FqhysTXtH1bB11qCJyyGG8G'}]

By following the link we can view the collection, but what if we want to get the data ourselves.

The easiest way to get data about the collection

The most convenient option is to receive data via the http protocol. Since TON nodes use their own ADNL transport protocol (more about it below), an intermediate service is required for the HTTP connection.

Toncenter is such an intermediate service that receives requests via HTTP; it accesses liteserver’s of the TON network. It has a /getTokenData method that returns token information according to the TON token data standard.

API_ENDPOINT = 'https://toncenter.com/api/v2'

for row in newlist:
  try:
    payload = {'address': '0:'+row['col']}
    r = requests.get(API_ENDPOINT+'/getTokenData', params=payload)
    col_content = r.json()
    print(col_content)
    # limit because we dont use api key 
    time.sleep(1.1)
  except:
    print('some error')

In total, we get ranked collections, ready for viewing, an approximate result can be seen here: https://tonlearn.tools/#/ton-nft-sales-volume

Collection data directly from the blockchain without intermediaries

It was mentioned above that TON has its own transport protocol, but what should we request from Liteserver? To understand this, you need to understand what a collection actually is; there is an analysis here.

Let’s say we figure it out and understand that a smart contract is an NFT collection in TON if it meets some conditions, one of which is that there must be a get_collection_data() method that will return this data.

Thus, we need to call the GET method get_collection_data() on the collection smart contract and thus enrich the data.

Data collection through ADNL is a fairly extensive topic, which I covered here, and after some time I will publish this on dev.to.

Conclusion

https://github.com/romanovichim/TonFunClessons_Eng

I post new tutorials and data analytics here: https://t.me/ton_learn

You can see the ranking of collections from this tutorial in a convenient form here: https://tonlearn.tools/#/ton-nft-sales-volume

Code from this tutorial: https://colab.research.google.com/drive/1Nm6ZJa1I892dq10JKpUuyijlV-SZTSRr

TON Smart Contract Pipeline Part4 - Chatbot Contract + Writing onchain tests on the testnet

Ivan Romanovich 🧐 — Thu, 07 Sep 2023 14:30:16 +0000

Introduction

In this tutorial, we will analyze the chatbot smart contract and then write on-chain tests for it. In this tutorial, we will focus how to inspect transactions in tests and for onchain tests.

This tutorial is part of an open source course that I am currently updating, link to the repository, I will be glad to see your star

About TON

TON is an actor model is a mathematical parallel computing model that underlies TON smart contracts. In it, each smart contract can receive one message, change its own state, or send one or more messages per unit of time.

Most often, to create a full-fledged application on TON, you need to write several smart contracts that seem to communicate with each other using messages. In order for the contract to understand what it needs to do when a message arrives, it is recommended to use op. op is a 32-bit identifier that should be passed in the body of the message.

Thus, inside the message using conditional statements, depending on the smart contract op performs different actions.

Therefore, it is important to be able to test messages, which we will do today.

The chatbot smart contract receives any internal message and responds to it with an internal message with the reply text.

Parsing the contract

Standard Library

The first thing to do is import the standard library. The library is just a wrapper for the most common TVM (TON virtual machine) commands that are not built-in.

#include "imports/stdlib.fc";

To process internal messages, we need the recv_internal() method

() recv_internal()  {

}

External method arguments

Here a logical question arises - how to understand what arguments a function should have so that it can receive messages on the TON network?

According to the documentation of the TON virtual machine - TVM, when an event occurs on an account in one of the TON chains, it triggers a transaction.

Each transaction consists of up to 5 stages. More details here.

We are interested in Compute phase. And to be more specific, what is "on the stack" during initialization. For normal post-triggered transactions, the initial state of the stack looks like this:

5 elements:

Smart contract balance (in nanoTons)
Incoming message balance (in nanotones)
Cell with incoming message
Incoming message body, slice type
Function selector (for recv_internal it is 0)

() recv_internal(int balance, int msg_value, cell in_msg_full, slice in_msg_body)  {

}

But it is not necessary to write all the arguments to recv_internal(). By setting arguments to recv_internal(), we tell the smart contract code about some of them. Those arguments that the code will not know about will simply lie at the bottom of the stack, never touched. For our smart contract, this is:

    () recv_internal(int msg_value, cell in_msg, slice in_msg_body) impure {

    }

Gas to handle messages

Our smart contract will need to use the gas to send the message further, so we will check with what msg_value the message came, if it is very small (less than 0.01 TON), we will finish the execution of the smart contract with return().

#include "imports/stdlib.fc";

() recv_internal(int msg_value, cell in_msg, slice in_msg_body) impure {

  if (msg_value < 10000000) { ;; 10000000 nanoton == 0.01 TON
    return ();
  }

}

Get the address

To send a message back, you need to get the address of the person who sent it to us. To do this, you need to parse the in_msg cell.

In order for us to take the address, we need to convert the cell into a slice using begin_parse:

var cs = in_msg_full.begin_parse();

Now we need to "subtract" the resulting slice to the address. Using the load_uint function from the FunC standard library it loads an unsigned n-bit integer from the slice, "subtract" the flags.

var flags = cs~load_uint(4);

In this lesson, we will not dwell on the flags in detail, but you can read more in paragraph 3.1.7.

And finally, the address. Use load_msg_addr() - which loads from the slice the only prefix that is a valid MsgAddress.

slice sender_address = cs~load_msg_addr();

Code:

#include "imports/stdlib.fc";

() recv_internal(int msg_value, cell in_msg, slice in_msg_body) impure {

  if (msg_value < 10000000) { ;; 10000000 nanoton == 0.01 TON
    return ();
  }

  slice cs = in_msg.begin_parse();
  int flags = cs~load_uint(4); 
  slice sender_address = cs~load_msg_addr(); 

}

Sending a message

Now you need to send a message back

Message structure

The full message structure can be found here - message layout. But usually we don't need to control each field, so we can use the short form from example:

 var msg = begin_cell()
    .store_uint(0x18, 6)
    .store_slice(addr)
    .store_coins(amount)
    .store_uint(0, 1 + 4 + 4 + 64 + 32 + 1 + 1)
    .store_slice(message_body)
  .end_cell();

As you can see, functions of the FunC standard library are used to build the message. Namely, the "wrapper" functions of the Builder primitives (partially built cells, as you may remember from the first lesson). Consider:

begin_cell() - will create a Builder for the future cell
end_cell() - will create a Cell (cell)
store_uint - store uint in Builder
store_slice - store the slice in the Builder
store_coins - here the documentation means store_grams - used to store TonCoins. More details here.

Message body

In the body of the message we put op and our message reply, to put a message we need to do slice.

slice msg_text = "reply";

In the recommendations about the body of the message, there is a recommendation to add op, despite the fact that it will not carry any functionality here, we will add it.

In order for us to create a similar client-server architecture on smart contracts described in the recommendations, it is proposed to start each message (strictly speaking, the message body) with some op flag, which will identify what operation the smart contract should perform.

Let's put op equal to 0 in our message.

Now the code looks like this:

#include "imports/stdlib.fc";

() recv_internal(int msg_value, cell in_msg, slice in_msg_body) impure {

  if (msg_value < 10000000) { ;; 10000000 nanoton == 0.01 TON
    return ();
  }

  slice cs = in_msg.begin_parse();
  int flags = cs~load_uint(4); 
  slice sender_address = cs~load_msg_addr(); 

  slice msg_text = "reply"; 

  cell msg = begin_cell()
      .store_uint(0x18, 6)
      .store_slice(sender_address)
      .store_coins(100) 
      .store_uint(0, 1 + 4 + 4 + 64 + 32 + 1 + 1)
      .store_uint(0, 32)
      .store_slice(msg_text) 
  .end_cell();

    }

The message is ready, let's send it.

Message sending mode(mode)

To send messages, use send_raw_message from the standard library.

We have already collected the msg variable, it remains to figure out mode. Description of each mode is in documentation. Let's look at an example to make it clearer.

Let there be 100 coins on the balance of the smart contract and we receive an internal message with 60 coins and send a message with 10, the total fee is 3.

mode = 0 - balance (100+60-10 = 150 coins), send(10-3 = 7 coins)
mode = 1 - balance (100+60-10-3 = 147 coins), send(10 coins)
mode = 64 - balance (100-10 = 90 coins), send (60+10-3 = 67 coins)
mode = 65 - balance (100-10-3=87 coins), send (60+10 = 70 coins)
mode = 128 -balance (0 coins), send (100+60-3 = 157 coins)

As we choose mode, let's go to documentation:

We're sending a normal message, so mode 0.
We will pay the commission for the transfer separately from the cost of the message, which means +1.
We will also ignore any errors that occur during the processing of this message on the action phase, so +2.

We get mode == 3, the final smart contract:

#include "imports/stdlib.fc";

() recv_internal(int msg_value, cell in_msg, slice in_msg_body) impure {

  if (msg_value < 10000000) { ;; 10000000 nanoton == 0.01 TON
    return ();
  }

  slice cs = in_msg.begin_parse();
  int flags = cs~load_uint(4); 
  slice sender_address = cs~load_msg_addr(); 

  slice msg_text = "reply"; 

  cell msg = begin_cell()
      .store_uint(0x18, 6)
      .store_slice(sender_address)
      .store_coins(100) 
      .store_uint(0, 1 + 4 + 4 + 64 + 32 + 1 + 1)
      .store_uint(0, 32)
      .store_slice(msg_text) 
  .end_cell();

  send_raw_message(msg, 3);
}

hexBoC

Before deploying a smart contract, you need to compile it into hexBoС, let's take the project from the previous tutorial.

Let's rename main.fc to chatbot.fc and write our smart contract into it.

Since we changed the filename, we need to upgrade compile.ts as well:

import * as fs from "fs";
import { readFileSync } from "fs";
import process from "process";
import { Cell } from "ton-core";
import { compileFunc } from "@ton-community/func-js";

async function compileScript() {

    const compileResult = await compileFunc({
        targets: ["./contracts/chatbot.fc"], 
        sources: (path) => readFileSync(path).toString("utf8"),
    });

    if (compileResult.status ==="error") {
        console.log("Error happend");
        process.exit(1);
    }

    const hexBoC = 'build/main.compiled.json';

    fs.writeFileSync(
        hexBoC,
        JSON.stringify({
            hex: Cell.fromBoc(Buffer.from(compileResult.codeBoc,"base64"))[0]
                .toBoc()
                .toString("hex"),
        })

    );

    console.log("Compiled, hexBoC:"+hexBoC);

}

compileScript();

Compile the smart contract with the yarn compile command.

You now have a hexBoC representation of the smart contract.

Check if there is a transaction

Since we are using the draft of the previous tutorial as a template, we already have a test framework, open the main.spec.ts file and remove from there everything related to the GET method:

import { Cell, Address, toNano } from "ton-core";
import { hex } from "../build/main.compiled.json";
import { Blockchain } from "@ton-community/sandbox";
import { MainContract } from "../wrappers/MainContract";
import { send } from "process";
import "@ton-community/test-utils";

describe("test tests", () => {
    it("test of test", async() => {
        const codeCell = Cell.fromBoc(Buffer.from(hex,"hex"))[0];

        const blockchain = await Blockchain.create();

        const myContract = blockchain.openContract(
            await MainContract.createFromConfig({}, codeCell)
        );

        const senderWallet = await blockchain.treasury("sender");

        const sentMessageResult = await myContract.sendInternalMessage(senderWallet.getSender(),toNano("0.05"));

        expect(sentMessageResult.transactions).toHaveTransaction({
            from: senderWallet.address,
            to: myContract.address,
            success: true,
        });

    });
});

We see that at the moment, it is checked whether the transaction has been sent to our smart contract. This is due to the sentMessageResult.transactions object. Let's take a close look at it and see what we can test based on this object.

If we just print this object to the console, it will consist of a lot of raw information, for convenience we will use flattenTransaction from @ton-community/test-utils:

import { Cell, Address, toNano } from "ton-core";
import { hex } from "../build/main.compiled.json";
import { Blockchain } from "@ton-community/sandbox";
import { MainContract } from "../wrappers/MainContract";
import { send } from "process";
import "@ton-community/test-utils";
import { flattenTransaction } from "@ton-community/test-utils";



describe("msg test", () => {
    it("test", async() => {
        const codeCell = Cell.fromBoc(Buffer.from(hex,"hex"))[0];

        const blockchain = await Blockchain.create();

        const myContract = blockchain.openContract(
            await MainContract.createFromConfig({}, codeCell)
        );

        const senderWallet = await blockchain.treasury("sender");

        const sentMessageResult = await myContract.sendInternalMessage(senderWallet.getSender(),toNano("0.05"));

        expect(sentMessageResult.transactions).toHaveTransaction({
            from: senderWallet.address,
            to: myContract.address,
            success: true,
        });

        const arr = sentMessageResult.transactions.map(tx => flattenTransaction(tx));
        console.log(arr)


    });
});

What you see in the console can be used for tests, let's check that the message our chatbot sent is equal to reply.

Let's assemble the message, in accordance with what we collected in the smart contract.

    let reply = beginCell().storeUint(0, 32).storeStringTail("reply").endCell();

Now, using messages, check that there is such a transaction:

import { Cell, Address, toNano, beginCell } from "ton-core";
import { hex } from "../build/main.compiled.json";
import { Blockchain } from "@ton-community/sandbox";
import { MainContract } from "../wrappers/MainContract";
import { send } from "process";
import "@ton-community/test-utils";
import { flattenTransaction } from "@ton-community/test-utils";



describe("msg test", () => {
    it("test", async() => {
        const codeCell = Cell.fromBoc(Buffer.from(hex,"hex"))[0];

        const blockchain = await Blockchain.create();

        const myContract = blockchain.openContract(
            await MainContract.createFromConfig({}, codeCell)
        );

        const senderWallet = await blockchain.treasury("sender");

        const sentMessageResult = await myContract.sendInternalMessage(senderWallet.getSender(),toNano("0.05"));

        expect(sentMessageResult.transactions).toHaveTransaction({
            from: senderWallet.address,
            to: myContract.address,
            success: true,
        });

        //const arr = sentMessageResult.transactions.map(tx => flattenTransaction(tx));

        let reply = beginCell().storeUint(0, 32).storeStringTail("reply").endCell();

        expect(sentMessageResult.transactions).toHaveTransaction({
            body: reply,
            from: myContract.address,
            to: senderWallet.address
        });

    });
});

Run the tests with the yarn test command and see that everything works. Thus, in tests we can collect objects the same as in a smart contract and check that the transaction was.

Onchain tests

Sometimes a situation may arise that you need to run your smart contracts on the test network (a situation where there are a lot of contracts). Let's try this with our example.

In the scripts folder we will make the onchain.ts file, for ease of launch, add to package.json "onchain": "ts-node ./scripts/onchain.ts":

  "scripts": {
    "compile": "ts-node ./scripts/compile.ts",
    "test": "yarn jest",
    "deploy": "yarn compile && ts-node ./scripts/deploy.ts",
    "onchain": "ts-node ./scripts/onchain.ts"
  },

Первое, что нам понадобиться для тестов, это адрес смарт-контракта, соберем его:

import { Cell, beginCell, contractAddress, toNano} from "ton-core";
import { hex } from "../build/main.compiled.json";
import { TonClient } from "ton";

async function onchainScript() {
    const codeCell = Cell.fromBoc(Buffer.from(hex,"hex"))[0];
    const dataCell = new Cell();

    const address = contractAddress(0,{
        code: codeCell,
        data: dataCell,
    });

    console.log("Address: ",address)

}

onchainScript();

The test for the test network will offer us to deploy a transaction via a QR code into our smart contract and check every 10 seconds if the answer has appeared on the network.

Of course, this is a simplification for an example, the essence is just to show the logic.

Let's collect a QR code, by which we will conduct a transaction through Tonkeeper. For our example, it is important that the amount of TON is sufficient so as not to throw an exception written in the contract.

import { Cell, beginCell, contractAddress, toNano} from "ton-core";
import { hex } from "../build/main.compiled.json";
import { TonClient } from "ton";
import qs from "qs";
import qrcode from "qrcode-terminal";

async function onchainScript() {
    const codeCell = Cell.fromBoc(Buffer.from(hex,"hex"))[0];
    const dataCell = new Cell();

    const address = contractAddress(0,{
        code: codeCell,
        data: dataCell,
    });

    console.log("Address: ",address)

    let transactionLink =
    'https://app.tonkeeper.com/transfer/' +
    address.toString({
        testOnly: true,
    }) +
    "?" +
    qs.stringify({
        text: "Sent simple in",
        amount: toNano("0.6").toString(10),
    });

    console.log("Transaction link:",transactionLink);


    qrcode.generate(transactionLink, {small: true }, (qr) => {
        console.log(qr);
    });

}

onchainScript();

In order to receive data from the test network, we need some kind of data source. Data can be obtained via ADNL from Liteservers, but we will talk about ADNL in the following tutorials. In this tutorial, we will use the TON Center API.

    const API_URL = "https://testnet.toncenter.com/api/v2"

We will make requests through the Http client axios, install: yarn add axios.

Among the Toncenter methods, we need getTransactions with the limit 1 parameter, i.e. we will take the last transaction. Let's write two helper functions for requesting information:

// axios http client // yarn add axios
async function getData(url: string): Promise<any> {
    try {
      const config: AxiosRequestConfig = {
        url: url,
        method: "get",
      };
      const response: AxiosResponse = await axios(config);
      //console.log(response)
      return response.data.result;
    } catch (error) {
      console.error(error);
      throw error;
    }
  }

async function getTransactions(address: String) {
  var transactions;
  try {
    transactions = await getData(
      `${API_URL}/getTransactions?address=${address}&limit=1`
    );
  } catch (e) {
    console.error(e);
  }
  return transactions;
}

Now we need a function that will call the API at intervals, for this there is a convenient method SetInterval:

import { Cell, beginCell, contractAddress, toNano} from "ton-core";
import { hex } from "../build/main.compiled.json";
import { TonClient } from "ton";
import qs from "qs";
import qrcode from "qrcode-terminal";
import axios, { AxiosRequestConfig, AxiosResponse } from "axios";


const API_URL = "https://testnet.toncenter.com/api/v2"

    // axios http client // yarn add axios
async function getData(url: string): Promise<any> {
    try {
      const config: AxiosRequestConfig = {
        url: url,
        method: "get",
      };
      const response: AxiosResponse = await axios(config);
      //console.log(response)
      return response.data.result;
    } catch (error) {
      console.error(error);
      throw error;
    }
  }

async function getTransactions(address: String) {
  var transactions;
  try {
    transactions = await getData(
      `${API_URL}/getTransactions?address=${address}&limit=1`
    );
  } catch (e) {
    console.error(e);
  }
  return transactions;
}

async function onchainScript() {
    const codeCell = Cell.fromBoc(Buffer.from(hex,"hex"))[0];
    const dataCell = new Cell();

    const address = contractAddress(0,{
        code: codeCell,
        data: dataCell,
    });


    console.log("Address: ",address)

    let transactionLink =
    'https://app.tonkeeper.com/transfer/' +
    address.toString({
        testOnly: true,
    }) +
    "?" +
    qs.stringify({
        text: "Sent simple in",
        amount: toNano("0.6").toString(10),
        //bin: beginCell().storeUint(1,32).endCell().toBoc({idx: false}).toString("base64"),
    });

    console.log("Transaction link:",transactionLink);


    qrcode.generate(transactionLink, {small: true }, (qr) => {
        console.log(qr);
    });

    setInterval(async () => {
        const txes = await getTransactions(address.toString());
        if(txes[0].in_msg.source === "EQCj2gVRdFS0qOZnUFXdMliONgSANYXfQUDMsjd8fbTW-RuC") {

        }

    },10000)


}

onchainScript();

It is important to note here that the API returns transactions, not messages, so we need to check that IN received the address of our wallet (here I just hardcoded it) and the message (which we put under the QR), and output the message of the first message in OUT. We also display the date, we get:

import { Cell, beginCell, contractAddress, toNano} from "ton-core";
import { hex } from "../build/main.compiled.json";
import { TonClient } from "ton";
import qs from "qs";
import qrcode from "qrcode-terminal";
import axios, { AxiosRequestConfig, AxiosResponse } from "axios";


const API_URL = "https://testnet.toncenter.com/api/v2"

    // axios http client // yarn add axios
async function getData(url: string): Promise<any> {
    try {
      const config: AxiosRequestConfig = {
        url: url,
        method: "get",
      };
      const response: AxiosResponse = await axios(config);
      //console.log(response)
      return response.data.result;
    } catch (error) {
      console.error(error);
      throw error;
    }
  }

async function getTransactions(address: String) {
  var transactions;
  try {
    transactions = await getData(
      `${API_URL}/getTransactions?address=${address}&limit=1`
    );
  } catch (e) {
    console.error(e);
  }
  return transactions;
}

async function onchainScript() {
    const codeCell = Cell.fromBoc(Buffer.from(hex,"hex"))[0];
    const dataCell = new Cell();

    const address = contractAddress(0,{
        code: codeCell,
        data: dataCell,
    });


    console.log("Address: ",address)

    let transactionLink =
    'https://app.tonkeeper.com/transfer/' +
    address.toString({
        testOnly: true,
    }) +
    "?" +
    qs.stringify({
        text: "Sent simple in",
        amount: toNano("0.6").toString(10),
        //bin: beginCell().storeUint(1,32).endCell().toBoc({idx: false}).toString("base64"),
    });

    console.log("Transaction link:",transactionLink);


        qrcode.generate(transactionLink, {small: true }, (qr) => {
            console.log(qr);
        });

        setInterval(async () => {
            const txes = await getTransactions(address.toString());
            if(txes[0].in_msg.source === "EQCj2gVRdFS0qOZnUFXdMliONgSANYXfQUDMsjd8fbTW-RuC") {

                console.log("Last tx: " + new Date(txes[0].utime * 1000))
                console.log("IN from: "+ txes[0].in_msg.source+" with msg: "+ txes[0].in_msg.message)
                console.log("OUT from: "+ txes[0].out_msgs[0].source +" with msg: "+ txes[0].out_msgs[0].message)
            }

        },10000)


    }

    onchainScript();

We launch the yarn onchain command, scan the QR, send the transaction and wait for our transaction to arrive.

Conclusion

I hope you enjoyed the pipeline series. I will be grateful to the asterisk on the repository.I publish tutorials here, if you liked the article, subscribe so as not to miss new ones.

Searching for NFT wash trading in Telegram Usernames in the TON blockchain

Ivan Romanovich 🧐 — Fri, 01 Sep 2023 12:21:10 +0000

Searching for NFT wash trading in Telegram Usernames in the TON blockchain

The lack of normal regulation of the crypto market makes it a testing ground for all possible scam mechanics. 2022 study showed that about 45% of NFT trade volume is part of wash trade.

Given such a large percentage in EVM chains, I became interested in how things are with this in the TON blockchain and I decided to apply the filters from the study above to find laundering transactions in the largest TON NFT collection - Telegram Usernames

Let's take a look at:

what is NFT wash trading
why it is used in NFT space and what led to such a large percentage of laundering in EVM blockchains
methodology for searching for wash deals
wash trading in Telegram Usernames - largest collection in TON blockchain

What is wash trading?

Under the wash trade in traditional financial markets is meant the conduct of fictitious transactions in order to mislead the market. Through wash transactions, fraudsters create the appearance of attractiveness of assets, trying to involve other market participants in trading these assets.

The idea of laundering is not new, as early as 1936 in the United States a law was adopted prohibiting such trade. But there are no such laws for the crypto market, so laundering transactions are used to draw attention to certain tokens.

NFT Wash Trading

When a potential buyer considers an NFT collection, he is trying to understand whether it shows a stable growth so that by buying NFT he can sell it later at a higher price, they usually look at the trading volume indicators. Laundering trading will allow simulating volume growth by reselling an NFT element between multiple wallets.

Another scenario is an attempt to inflate the price of a particular NFT element. A potential buyer looks at the sales history and sees that the element is popular and there are many transactions, but does not realize that all the wallets in the transaction history lead to one wallet, which sponsored the price surge.

Marketing companies/hacks also contribute to the wash trade, in 2022 the NFT marketplace market was overheated, as new platforms were constantly launched. The platforms were compared with each other in terms of trading volume - where there are more, it will be easier to buy and sell your NFT. To stimulate sales Rarible, LooksRare, X2Y2 marketplaces began to reward with a token for transactions, which led to laundering transactions to receive reward tokens and their subsequent resale.

In summary, NFT wash trading is the resale of NFTs between controlled wallets for subsequent profit.

Filters to search for wash trading - hildobby Method

To search for wash trades, we will use the hildobby method, which consists of four filters, which we will consider below.

The data source will be the dton.io indexer, from which we will receive information. I don’t have the capacity to collect information on the entire TON, so let’s consider the largest collection of Telegram Usernames in terms of sales, which allows you to trade usernames in Telegram.

If you want to try to find TON wash trades for some other collection here there is a link to a Python script, with which you can collect transactions and pass them through the filters. I also note that the results in this article are relevant as of 08/24/2023, but they can always be updated through the script.

In total, as of 08/24/2023, there were 109158 sales in the collection with a volume of 57,143,316 TON, which is about $80,572,075 as of the date.

Let's move on to filters

Filter #1 Buyer equals Seller

Reselling a collection element to itself is clearly unnatural behavior. Therefore, the first filter looks for deals where the seller is equal to the buyer when selling NFTs. The results are as follows:

Buyer = Seller sales: 1562

Percentage of total sales: 1.43%

Buyer = Seller Volume: 333267 TON ~ $469,906

Percentage of total sales: 0.58%

Examples of such a transaction in the explorer and on the marketplace (not advertising, it's just convenient to watch there):

Filter #2 Reverse Trades

You can resell items not only between the same wallet, a simple strategy is to trade the same NFT between two different wallets, sending it back and forth.

There are few such deals in the Telegram Username collection:

Reverse sales: 366

Percentage of total sales: 0.34%

Volume of reverse trades: 96275 TON ~ 135747 dollars

Percentage of total sales: 0.17%

Examples of such a transaction in explorer and marketplace) (not advertising, it's easy to see it there.)

Filter #3 Purchase item three times

Since some unscrupulous NFT sellers create chains to resell items, there are money laundering deals that bypass the previous filters. Therefore, it is convenient to consider all transactions where one and the same owner repeats one and the same owner three or more times as money laundering.

The author of the technique emphasizes that this filter is not optimal, but the importance of this filter outweighs its disadvantages. Results:

3x sales: 307

Percentage of total sales: 0.28%

3x trade volume : 3676 TON ~ $5183

Percentage of total sales: 0.01%

Examples of such a transaction in the explorer and on the marketplace (not advertising, it's just convenient to watch there ):

Filter #4 Buyer and seller funded from the same wallet

To save time, wallets used for fictitious trading are often funded by the same wallet or each other. The filter will check each buyer and seller for who sent them the first transaction.

If it is the same wallet (or different), then it is marked as a laundering trade.

Sales with common owner: 1722

Percentage of total sales: 1.58%

Sale volume with common owner : 159462 TON ~ 224841 USD

Percentage of total sales: 0.28%

For example, this NFT had two owners 1 and 2, which were sponsored from one wallet.

Result

The total data is as follows:

Total sale in the collection: 109158

Transactions that fell under the filters: 3957

Percentage from all deals: 3.63 %

Filter trade Volume: 592680 TON ~ 835678 dollars

Percentage from Total Sales: 1.04 %

It is important to note that the filters certainly have errors and it seems to me that the fourth filter could be improved by searching for all transactions between wallets and highlighting some clusters between wallets (they could be displayed in bubble charts).

Also TON blockchain, at the architectural level, is an actor model in which smart contracts exchange messages and this leads to large chains of smart contracts in token transactions. Token standards in the TON network also lead to an elongation of the chains of smart contracts involved in the transfer of ownership. Therefore, filters designed for EVM networks can skip some of the money laundering transactions.

Conclusion

To dive deeper, I invite you to get to know the TON blockchain through my open source tutorials that I publish here. Also, the laundering transactions found above lead to the conclusion that due to the lack of normal regulation of the cryptocurrency market, any purchase of assets requires more research.
P.S If you like it I will be grateful if you like or repost the thread:https://x.com/roma_i_m/status/1697585133870579735?s=46&t=qYsFs6APw12MupUmHzTVCg

TON Smart Contract Pipeline Part3 - Convenient deployment to the test network

Ivan Romanovich 🧐 — Sat, 26 Aug 2023 08:31:34 +0000

Introduction

In the first two parts, we analyzed compiling and testing.In this tutorial, we will make a convenient deployment of our contract to the test network, for this we will generate a link for the deployment, which we will present in the form of a QR code, we will scan the code with a wallet and thus the deployment will take place.

The question may arise, what is the link for deployment - to deploy a smart contract, you need to send a message with data about the contract, we will do it in the following way, we will form a deeplink for the Tonkeeper wallet into this deeplink we will transfer all the data for deployment and then from the link it will be possible make a QR, which, when scanned by a wallet, will deploy a contract, transferring all the information to the blockchain.

Deploy script

Let's create a separate file for deploying to testnet deploy.ts.

StateInit

StateInit serves to delivery inital data to contract and used in contract deployment.

At the top level, to deploy to the testnet or mainnet, we need to send a StateInit message to the network. StateInit consists of a cell with a code and a cell with data, and the smart contract address is roughly equal to: hash(initial code, initial state).

Let's collect the StateInit and send a message for deployment.

In the deploy.ts file, create the deployContract() function:

async function deployContract() {
}

deployContract()

Let's import the necessary additional functions from ton-core, as well as our smart contract in hex format:

import { Cell, StateInit, beginCell, contractAddress, storeStateInit, toNano } from "ton-core";
import { hex } from "../build/main.compiled.json";

async function deployContract() {
}

deployContract()

We will get the cell with the code from the hex representation of the contract, there will be no starting data - we create an empty cell. Now we have data ready to form StateInit

import { Cell, StateInit, beginCell, contractAddress, storeStateInit, toNano } from "ton-core";
import { hex } from "../build/main.compiled.json";

async function deployContract() {
    const codeCell = Cell.fromBoc(Buffer.from(hex,"hex"))[0];
    const dataCell = new Cell();   

    const stateInit: StateInit = {
        code: codeCell,
        data: dataCell,
    };
}

deployContract()

Now we need to build StateInit by first creating Builder(Data bits and references to other cells can be stored in a builder, and then the builder can be finalized to a new cell.) , but we use the helper functions from ton-core storeStateInit, we get:

import { Cell, StateInit, beginCell, contractAddress, storeStateInit, toNano } from "ton-core";
import { hex } from "../build/main.compiled.json";

async function deployContract() {
    const codeCell = Cell.fromBoc(Buffer.from(hex,"hex"))[0];
    const dataCell = new Cell();   

    const stateInit: StateInit = {
        code: codeCell,
        data: dataCell,
    };

    const stateInitBuilder = beginCell();
    storeStateInit(stateInit)(stateInitBuilder);
    const stateInitCell = stateInitBuilder.endCell();

}

deployContract()

Above, we said that you can get the address from the source data, let's do it with the help of contractAddress:

import { Cell, StateInit, beginCell, contractAddress, storeStateInit, toNano } from "ton-core";
import { hex } from "../build/main.compiled.json";

async function deployContract() {
    const codeCell = Cell.fromBoc(Buffer.from(hex,"hex"))[0];
    const dataCell = new Cell();   

    const stateInit: StateInit = {
        code: codeCell,
        data: dataCell,
    };

    const stateInitBuilder = beginCell();
    storeStateInit(stateInit)(stateInitBuilder);
    const stateInitCell = stateInitBuilder.endCell();

    const address = contractAddress(0, {
        code: codeCell,
        data: dataCell,
    });

}

deployContract()

QR code

Let's start forming a string for deploying a contract:

Install the library for convenient string generation

yarn add qs @types/qs --dev

And a library for generating a QR code.

yarn add qrcode-terminal @types/qrcode-terminal --dev

To send a message, we need to use deeplink /transfer:

ton://transfer/<address>
ton://transfer/<address>?amount=<nanocoins>
ton://transfer/<address>?text=<url-encoded-utf8-text>
ton://transfer/<address>?bin=<url-encoded-base64-boc>
ton://transfer/<address>?bin=<url-encoded-base64-boc>&init=<url-encoded-base64-boc>

https://app.tonkeeper.com/transfer/<address>
https://app.tonkeeper.com/transfer/<address>?amount=<nanocoins>
https://app.tonkeeper.com/transfer/<address>?text=<url-encoded-utf8-text>
https://app.tonkeeper.com/transfer/<address>?bin=<url-encoded-base64-boc>
https://app.tonkeeper.com/transfer/<address>?bin=<url-encoded-base64-boc>&init=<url-encoded-base64-boc>

Using the qs library, we will collect the following line:

     let deployLink =
        'https://app.tonkeeper.com/transfer/' +
        address.toString({
            testOnly: true,
        }) +
        "?" +
        qs.stringify({
            text: "Deploy contract by QR",
            amount: toNano("0.1").toString(10),
            init: stateInitCell.toBoc({idx: false}).toString("base64"),
        });

Now QR code:

    qrcode.generate(deployLink, {small: true }, (qr) => {
        console.log(qr);
    });

At the end, we will generate a link to the blockchain explorer in the test network, for our convenience - after the deployment, we will see that the contract is ready.

Final code:

import { Cell, StateInit, beginCell, contractAddress, storeStateInit, toNano } from "ton-core";
import { hex } from "../build/main.compiled.json";
import qs from "qs";
import qrcode from "qrcode-terminal";

async function deployContract() {
    const codeCell = Cell.fromBoc(Buffer.from(hex,"hex"))[0];
    const dataCell = new Cell();   

    const stateInit: StateInit = {
        code: codeCell,
        data: dataCell,
    };

    const stateInitBuilder = beginCell();
    storeStateInit(stateInit)(stateInitBuilder);
    const stateInitCell = stateInitBuilder.endCell();

    const address = contractAddress(0, {
        code: codeCell,
        data: dataCell,
    });


    let deployLink =
        'https://app.tonkeeper.com/transfer/' +
        address.toString({
            testOnly: true,
        }) +
        "?" +
        qs.stringify({
            text: "Deploy contract by QR",
            amount: toNano("0.1").toString(10),
            init: stateInitCell.toBoc({idx: false}).toString("base64"),
        });

    qrcode.generate(deployLink, {small: true }, (qr) => {
        console.log(qr);
    });

    let scanAddr = 
        'https://testnet.tonscan.org/address/' +
        address.toString({
            testOnly: true,
        })

    console.log(scanAddr);

    }

deployContract()

Upgrade the file package.json by adding "deploy": "yarn compile && ts-node ./scripts/deploy.ts" to scripts:

{
  "name": "test_folder",
  "version": "1.0.0",
  "main": "index.js",
  "license": "MIT",
  "devDependencies": {
    "@swc/core": "^1.3.63",
    "@ton-community/func-js": "^0.6.2",
    "@ton-community/sandbox": "^0.11.0",
    "@types/jest": "^29.5.2",
    "@types/node": "^20.3.1",
    "@types/qrcode-terminal": "^0.12.0",
    "@types/qs": "^6.9.7",
    "jest": "^29.5.0",
    "qrcode-terminal": "^0.12.0",
    "qs": "^6.11.2",
    "ton": "^13.5.0",
    "ton-core": "^0.49.1",
    "ton-crypto": "^3.2.0",
    "ts-jest": "^29.1.0",
    "ts-node": "^10.9.1",
    "typescript": "^5.1.3"
 },
  "scripts": {
    "compile": "ts-node ./scripts/compile.ts",
    "test": "yarn jest",
    "deploy": "yarn compile && ts-node ./scripts/deploy.ts"
  },
  "dependencies": {
    "@ton-community/test-utils": "^0.2.0"
  }
}

It remains to run it all:

in the console we type the command yarn deploy
scan the QR code in the Tonkeeper switched to the test network
confirm the transaction
look in the explorer that the contract is deployed

Conclusion

Thank you for your attention, in the next two tutorials we will pay attention to messages, what to build on when writing tests and how to write tests for the test network.P.S. I publish such articles here