Managing Large DataList In Solana

Shaikh Sadi — Fri, 22 Sep 2023 05:00:21 +0000

This is a continuation of my solana journey. Here I will share my experience of using solana.

Everything is stored in Account

In solana everything is stored in account. So a program is stored in account , your wallet data is stored in account, any user data who is interacting with any program that is also stored in account. As of now i used anchor for building solana programs so I will try to explain in that perspective.

Default Way

The most easiest and low maintenance way is to store data in a vec. In this way your account could go up to 32kb. But there is a catch, I used to calculate the space and count how many items i can push into the vec before hitting that 32kb stack size. But unfortunately i was not able to reach that count, it was failing almost in the half way with Memory allocation failed error.
After many finding i came to know this is due to rust, rust doesn't allocate memory on every push, Not sure but I think they try to allocate double of the current size once you reach the current size. Although there is workarounds around for this.

Zero Copy Account

With zero-copy account you can reach 10mb size, this is the maximum account size that solana allows. The biggest issue is you need to pre-define array length , yes array. You can't use vector in zero-copy account. As a result when creating account user will have to pay much more even though they won't use all the space.

Splitting Account

If other two options are not suitable for you , you may consider this way. The idea is you will store every element in its own account(I am currently experimenting with this approach). On positive side , it can grow almost infinitely , only pay for one account at once.
But management can get hard in this approach, like you will have to decide how you will generate the seed for each element, you wanna reuse the space of previously allocated unused account, etc etc....

Third one is more of a structural solution. But you can choose any of them based on your necessity and requirements. I am not an expert , so I would really love to hear any correction or idea. Thank You.

My Web3 Journey with Solana

Shaikh Sadi — Thu, 14 Sep 2023 23:57:15 +0000

Intro

Before joining my current company I had very little to no knowledge about blockchain and nft related stuff. Although it was in my task list of 2022, But not because I wanna learn it , rather I wanted to ignore it 😁.

Let me clear what i mean. I was seeing a lot of hype around blockchain at that time. Just like we are watching around bun currently. So, what i do is I normally ignore the initial hype, the reason behind this is simple as a developer you can't go after every hype because new technology emerges on a regular basis. And you can't learn all of them. I am peaky on new tech to save time and energy.

I remember when i was learning javascript . I was hearing huge hype around nextjs. I was confused because i just learned about versions of javascript. Like ES5, ES6 and ES-Next. So i was like isn't nextjs same as es-next. Why people are talking so much about this 😅. I eventually learned nextjs after react .When initial hype is gone and i have seen the potential of it.

Learning Solana

Well enough gossip , lets come to the point . Although blockchain was not in my list I started learning it because of my job and yes it was solana . As you know solana program (smart-contract) uses rust programing language (there is way to do it in python,c also). At that point I was also beginner of rust, Just completed tutorials from rust book . So, I knew its gonna be hard because of the two unfamiliar technology at the same time.

I was a bit lucky because at the beginning there was another developer (part time) who was doing the program (smart-contract) related development on rust. So my task was to integrate solana program with nextjs.
My initial struggle was to grasp the idea of how the program is working (less important), and how it will work in frontend, what should be the project structure , what is the best practice about managing instructions, transactions all those stuff. Unfortunately resources on internet are not enough about solana and I had to surf a lot find any answer and waste a lot of time.

I was not comfortable with Buffer or Raw Data. But after this initial phase my fear was gone and I was confident parsing accounts . Although we were using anchor , but not sure why i was manually parsing data 😅. Maybe because i wanted to know how its working.

After initial integration , most of the things were working, its was not the best but it was working 😛. But after the review we came to know that there is a big gap between whats been asked and what is delivered , you can't blame me it was the rust program 😂. So we had to update the program and lead programmer was not available. After waiting couple of weeks I started taking on it, And completed the task very smoothly and I started living happily here after. What !! hell no…It was full of struggle and a complete misery 🥺.

Will be continued…

DEV Community: Shaikh Sadi