DEV Community: Tin Chung

Introducing terminal tool to manage embedded database

Tin Chung — Sat, 08 Apr 2023 15:00:33 +0000

Hello everyone, a few months ago I wrote a terminal tool in Rust to help view data stored in embedded databases like RocksDB or Sled. As someone who enjoys coding low-level systems, I found it very difficult when there were almost no tools available to view the byte data of embedded databases. That's why EDMA was created to solve that problem.
The project is still in its early stages, but it already encompasses some of the main features. I hope it will be useful for those who are into low-level or embedded programming.

EDMA is a command-line tool that allows users to interact with embedded databases at a byte level. It provides a simple yet powerful interface to view, modify, and delete data in databases like RocksDB or Sled. The tool is built using Rust, a systems programming language known for its safety, speed, and concurrency.

One of the key features of EDMA is its ability to work with a wide range of data types. Users can view and modify data in the database in different formats, including binary, hexadecimal, and ASCII. They can also specify the type of data stored in the database and EDMA will interpret it accordingly.

Another useful feature of EDMA is its support for filtering and sorting data. Users can search for specific data based on a range of criteria, such as key or value length, and sort the data by key or value. This makes it easy to find and analyze specific data in the database.

EDMA is designed to be easy to use and customizable. Users can configure the tool to work with their specific database and data format requirements. They can also extend the functionality of the tool by creating custom plugins.

Overall, EDMA is a valuable tool for developers working with embedded databases. It simplifies the process of working with low-level data and provides a convenient interface for analyzing and modifying data in embedded databases.

Link to Github repository: https://github.com/nomadiz/edma

Rust Blog Series: What is Rust programming language?

Tin Chung — Fri, 20 Jan 2023 19:30:48 +0000

Rust is a systems programming language that is designed to be safe, concurrent, and fast. It is an open-source programming language that is gaining popularity among developers for its unique approach to memory safety, which eliminates many of the common issues that plague other systems programming languages like C and C++.

One of the most notable features of Rust is its ownership model, which ensures that resources are only accessed in a safe and controlled way. In Rust, each value has a unique owner, and when the owner goes out of scope, the value is automatically cleaned up. This eliminates the need for manual memory management, and greatly reduces the likelihood of common programming errors such as null pointer dereferences and buffer overflows.

Rust also has built-in support for concurrency, which allows for the creation of multi-threaded applications that can take full advantage of modern multi-core CPUs. The language's built-in support for message passing concurrency makes it easy to create concurrent code that is also safe and easy to reason about.

In addition to its focus on safety and concurrency, Rust is also designed to be fast. The language's strict type system and low-level control over memory layout allows for the creation of highly performant code that is on par with C and C++. Rust's built-in support for inline assembly and FFI also makes it easy to interface with existing C and C++ code, allowing for the reuse of existing libraries and frameworks.

Another strong point of Rust is its active and welcoming community, which is made up of both experienced and new developers. The community provides a wide range of resources and support for learning Rust, including documentation, tutorials, and a large number of open-source libraries and frameworks.

Rust is also used in a wide range of applications, from web development to embedded systems, and has been adopted by companies such as Amazon, Dropbox, and Microsoft.

In summary, Rust is a systems programming language that is designed to be safe, concurrent, and fast. Its ownership model and built-in support for concurrency make it easy to create safe and concurrent code that is also highly performant. The language's strong type system, low-level control over memory layout, and active community make it a powerful choice for a wide range of applications.

SolomonDB Weekly Update (#2): Palmon

Tin Chung — Wed, 30 Nov 2022 07:55:47 +0000

Support my project by visiting SolomonDB Github and give me a star ⭐️

Weekly Update #2: Palmon

Date: November 2022 (1-7/11/2022)

Palmon is one of the Kings of Hell, more obedient to Lucifer than other kings are, and has two hundred legions of demons under his rule. He has a great voice and roars as soon as he comes, speaking in this manner for a while until the conjurer compels him and then he answers clearly the questions he is asked. When a conjurer invokes this demon he must look towards the northwest, the direction of Paimon's house, and when Paimon appears he must be allowed to ask the conjurer what he wishes and be answered, in order to obtain the same from him. - Wikipedia

Commits on Nov 1, 2022

Description

The prior implementation of RocksDBDriver was using the database instance initialized using DB object. And all methods are called from the object DB instance too. However, this approach has some limitations in terms of concurrency operations. Briefly, I migrate it from using DB to handling operations using transactions - a better control flow approach. Instead of using TransactionDB, I choose OptimisticTransactionDB which is more efficient for read contention database, correct me if I am wrong.

You can read more about this in RocksDB Wiki

Detail explanation

Even though there is only one commit and only today, this commit has a lot of code refractors. An obvious refractor is migrating from Driver to Adapter which follows adapter pattern.

StorageDriver to StorageAdapter
RocksDBDriver to RocksDBAdapter

Walk through the implementation of StorageAdapter, we have

pub struct StorageAdapter<T> {
    pub name: StorageAdapterName,
    pub db_instance: Pin<Arc<T>>,
    pub variant: StorageVariant,
}

You may notice the use of generic type Pin<Arc<T>>. Explain more about why I wrote this. Pin trait will pin the object in a memory, which means it can't be move to a different location in a memory, for example, using std::mem::swap. Inside the Pin type, we have Arc or Atomically Reference Counted.Arc type shares ownership between threads, which is different from single threaded type Rc. Hence, this is usually used for handling multi-threaded operations and it is suitable for our distributed database design.

New method added to RocksDBAdapter to create a transaction

pub fn transaction(
        self: &'static Self,
        w: bool,
        r: bool,
    ) -> Result<Transaction<TxType>, Error> {
        let inner = self.get_inner();
        let db_instance = &inner.db_instance;
        let tx = db_instance.transaction();

        // The database reference must always outlive
        // the transaction. If it doesn't then this
        // is undefined behaviour. This unsafe block
        // ensures that the transaction reference is
        // static, but will cause a crash if the
        // datastore is dropped prematurely.
        let tx = unsafe {
            std::mem::transmute::<
                rocksdb::Transaction<'_, OptimisticTransactionDB>,
                rocksdb::Transaction<'static, OptimisticTransactionDB>,
            >(tx)
        };

        Ok(Transaction::<TxType>::new(tx, w, r))
    }

There is a head-aching concept in this method, we have an unsafe method use std::mem::transmute. This is not recommended to use as it transform the lifetime of an object. The reason why we use this method here is because, we need to cast the original lifetime of OptimisticTransactionDB to static as we want the transaction remains as long as it can until the program stops. This is referenced from the source code of SurrealDB.

On the other hand, we have an implementation for internal transaction

impl Transaction<TxType>

Any storage adapter can be understand as a bridge to create transaction, it does not store any transaction value. This separation provides the ability to control a single transaction each operation instead of a whole db_instance.

Contributors

Chung Quan Tin (@chungquantin)

Commits on Nov 2, 2022

Description

Enhancing the development experience by configuring the CI/CD pipeline using Github Actions.
Implement basic methods of RocksDB OptimisticDB transaction

Detail explanation

There are two workflows added:

Formatter (check + apply) workflow: Use cargo clippy and cargo fmt to format and lint the repo.
Test: Run cargo test on the workspace whenever there's an update to master branch

Every datastore transaction will be marked generically as DBTransaction or Distributed Database Transaction. This is implied by Solomon DB technical directory. Transaction will implement a trait that requires these below method

// Check if closed
fn closed(&self) -> bool;
// Cancel a transaction
fn cancel(&mut self) -> Result<(), Error>;
// Commit a transaction
fn commit(&mut self) -> Result<(), Error>;
// Check if a key exists
fn exi<K>(&mut self, key: K) -> Result<bool, Error>
where
K: Into<Key>;
// Fetch a key from the database
fn get<K>(&mut self, key: K) -> Result<Option<Val>, Error>;
// Insert or update a key in the database
fn set<K, V>(&mut self, key: K, val: V) -> Result<(), Error>;
// Insert a key if it doesn't exist in the database
fn put<K, V>(&mut self, key: K, val: V) -> Result<(), Error>;
// Delete a key
fn del<K>(&mut self, key: K) -> Result<(), Error>;

Contributors

Chung Quan Tin (@chungquantin)

Commits on Nov 4, 2022

Description

Write a new macro to auto generate test code for datastore adapter. Whenever there's a new datastore implemented, we can add it to the test suite easily by

full_test_impl!(RocksDBAdapter::default());

This code implementation is referenced from IndraDB. On the other hand, these commits add a new feature tag called test-suite which must be declared to allow all test runs.

[features]
default = ["kv-rocksdb"]
kv-rocksdb = ["dep:rocksdb"]
+ test-suite = []

To run cargo test or cargo nextest enabling the test-suite feature, can follow these commands to run

cargo test --features test-suite

cargo nextest run --features test-suite

Detail explanations

The logic behind the new macro is not too complicated, the macro define_test! receive any datastore_adapter as an input along with the name for the test. This name is also a name of methods exported from crate tests. This approach is required to overpass the type strictness of DatastoreAdapter as we will support multiple types of datastore adapter.

/// Defines a unit test function.
#[macro_export]
macro_rules! define_test {
    ($name:ident, $datastore_adapter:expr) => {
        #[tokio::test]
        async fn $name() {
            let datastore_adapter = $datastore_adapter;
            $crate::tests::$name(datastore_adapter).await;
        }
    };
}

/// Use this macro to enable the entire standard test suite.
#[macro_export]
macro_rules! full_test_impl {
    ($code:expr) => {
        #[cfg(test)]
        define_test!(should_delete_key, $code);
    };
}

Contributors

Chung Quan Tin (@chungquantin)

Commits on Nov 5, 2022

Description

Introducing a new struct DatastoreManager which is generated using macro impl_datastore_manager. The idea behinds this implementation is to manage all datastore adapter in the cleanest way.

Adding method to generate a random datastore path

/// Generate a path to store data for RocksDB
fn generate_path(id: Option<i32>) -> String {
     let random_id: i32 = generate_random_i32();
     let id = &id.unwrap_or(random_id).to_string();
- String::from("./.temp/rocks-") + id + ".db"
+ let path = if cfg!(target_os = "linux") {
+    "/dev/shm/".into()
+ } else {
+    temp_dir()
+ }
+ .join(format!("solomon-rocksdb-{}", id));
+
+   path_to_string(&path).unwrap()
}

Adding column family to RocksDB

Adding models for graph related struct: Node - Label - Property - Relationship

Detail explanation

RocksDB Column Family

As being stated in the RocksDB Wiki:

Each key-value pair in RocksDB is associated with exactly one Column Family.

In SolomonDB design, there will be multiple column families specified for different set of data. For example: vertex, vertex-property, edge, edge-property and label. All methods of RocksDB transaction will include a column family attribute. For example, the get method

/// Fetch a key from the database
- async fn get<K: Into<Key> + Send>(&mut self, key: K) -> Result<Option<Val>, Error>;
+ async fn get<K: Into<Key> + Send>(&mut self, cf: CF, key: K) -> Result<Option<Val>, Error>;

Property Graph Design

Not sure but I remember this has been mentioned in a very first commit log, our graph database will follow the structure of Property Graph.

Vertex: Common object in every graph model. In Property Graph, vertex is more versatile. It has multiple properties, it can be considered as a document in NoSQL database. Vertex can also have multiple labels to identify itself.
Relationship: Or edge in single relational database. Indicates the link between two vertex. However, relationship can have properties too. It also has a type, for example, if two LOVER nodes connect, the relationship type should be LOVE. Defined as source_vertex -> relationship -> target_vertex.
Property: Define attribute type and name of object field. For example, vertex (or node) can have name, age, birthday and relationship can also have name. Structure of property is uuid | name | type. Property of each core objects (node and relationship) will be stored in a HashMap<Uuid, Vec<u8>> where Uuid = Property ID and Vec<u8> = Byte value for that property
Label: Vertex can have multiple labels. For example, one vertex can be a Person, Programmer and Employee at the same time. This can be misunderstood with Property. However, they are not the same. Labels are used for marking node instead of defining attributes.

Contributors

Chung Quan Tin (@chungquantin)

Commits on Nov 6, 2022

Description

Write DEVLOG for 2022 October CHANGLOG and November 1st commits

Detail explanation

In facts, I did try to add a Github Actions workflows to auto generate Github CHANGELOG. However, it did not work as expected so I just decide to write CHANGELOG on my own.

name: Changelog
on:
    release:
        types:
            - created
jobs:
    changelog:
        runs-on: ubuntu-20.04
        steps:
            - name: "✏️ Generate release changelog"
              uses: heinrichreimer/github-changelog-generator-action@v2.3
              with:
                  token: ${{ secrets.GITHUB_TOKEN }}

Contributors

Chung Quan Tin (@chungquantin)

Commits on Nov 7, 2022

Description

Updating Rust compiler version to support new feature in Rust 1.65.0: Generic Associated Type.
Adding CassandraDB
Write two new methods for Vertex controller

Detail explanation

Generic Associated Type is not new and it is common in Rust nightly channel. However, it was only officially released to stable channel in the latest version. Based on the given definition:

Generic associated types are an extension that permit associated types to have generic parameters. This allows associated types to capture types that may include generic parameters that came from a trait method. These can be lifetime or type parameters.

Here is how GAT used in SolomonDB code. We no longer have to care about Generic Type passed between structs. On the other hand, GAT Transaction force all implemented Transaction type to be SimpleTransaction.

#[async_trait]
- pub trait DatastoreAdapter<T: SimpleTransaction> {
+ pub trait DatastoreAdapter {
+   type Transaction: SimpleTransaction;
    // # Create new database transaction
    // Set `rw` default to false means readable but not readable
- fn transaction(&self, rw: bool) -> Result<T, Error>;
+ fn transaction(&self, rw: bool) -> Result<Self::Transaction, Error>;

    fn default() -> Self;

With GAT, we can make a fully reusable struct DatastoreAdapter and get rid of declaring Transaction generic type like RocksDBTransaction. The implementation of DatastoreAdapter instance for RocksDB will be

#[async_trait]
- impl DatastoreAdapter<RocksDBTransaction> for RocksDBAdapter {
+ impl DatastoreAdapter for RocksDBAdapter {
+ type Transaction = RocksDBTransaction;

    fn default() -> Self {
        let path = &RocksDBAdapter::generate_path(None);
        RocksDBAdapter::new(path, None).unwrap()

Super clean right? Worth a try if you have not updated yet.

Contributors

Chung Quan Tin (@chungquantin)

SolomonDB Weekly Update (#1): Baal

Tin Chung — Wed, 30 Nov 2022 07:52:00 +0000

Support my project by visiting SolomonDB Github and give me a star ⭐️

Weekly Update #1: Baal

Date: October 2022 (26-28/11/2022)

According to the Grand Grimoire, Baal (or Bael) is the head of the infernal powers. He is also the first demon listed in Wierus' Pseudomonarchia daemonum. According to Wierus, Bael is the first king of Hell with estates in the east. He has three heads: a toad, a man, and a cat. He also speaks in a raucous, but well-formed voice, and commands 66 legions. Bael teaches the art of invisibility, and may be the equivalent of Baal or Baalzebub, one of the Seven princes of Hell. - Wikipedia

Commits on Oct 26, 2022

Description

A soft beginning. I just spend some time to read graph database book and learning concepts, requirements needed to construct Solomon DB.

Detail explanation

There are a few interesting learning outcomes acquired here:

These concepts are not too complicated actually. Not talking about native graph database, non-native graph database is more like a database wrapper which has internals built on top of other databases.

-   Underlying storage
-   Processing engine
-   Graph compute engine

Learnt and noted down these concepts in WIKI.md.

As this is the first commit, it only includes code to initialize a codebase and its basic layout for Rust project using cargo new.

// A simple hello world program (db/src/main.rs)
fn main() {
    println!("Hello, world!");
}

Contributors

Chung Quan Tin (@chungquantin)

Commits on Oct 28, 2022

Description

Add StorageDriver, new type RocksDBDriver, macro impl_driver_core and misc implementation for StorageErr and Status.

Detail explanation

Things getting more exciting when we touch advanced syntaxes in Rust: New Type design pattern and Macro. Before jumping right into the main point of today changes, one of a missing features in Rust compared to other OOP languages must be mentioned first is the ability to do inheritance.

With what Rust provides so far, it is quite tricky to construct an inheritable struct object. At first, you may think doing OOP in Rust is quite bad as inheritance is a core component of any OOP design. However, if you learn more about OOP, you will know Inheritance < Composition. This is quite off topic, but there is a few discussions and articles related to the way Rust do OOP that you should read:

Briefly, even though Rust does not have an obvious OOP feature like inheritance, it still provides several approaches: Trait, New type, Macro or Enum to do thing in a inheritance way but cleaner as things are separated elegantly.

New Type design pattern for RocksDB Driver

"The newtype idiom gives compile time guarantees that the right type of value is supplied to a program" - Rust book.

I apply the design to implement RocksDB Driver. The idea is to have a parent struct called StorageDriver and we will build a new type RocksDBDriver from that.

pub struct StorageDriver<T> {
 pub db_instance: Option<T>,
 pub path: String,
}

pub struct RocksDBDriver(StorageDriver<DB>);

However, NewType design does not exactly like inheritance. For example, if you want to get field db_instance from RocksDBDriver, you can't do self.db_instance but self.0.db.instance. Can imagine db_instance is inside StorageDriver while StorageDriver is wrapped inside new type RocksDBDriver. This would make the syntax looks a bit dirty. The solution I used to handle this case is Macro.

Macro provides an ability to maximize the power of Rust syntax. I wrote a macro to implement all below methods used inside any impl calling to impl_driver_core. Then to get the inner of new type, we just have to use get_core.

macro_rules! impl_driver_core {
 ($DbType: ty) => {
  fn get_core(self: &mut Self) -> &mut StorageDriver<$DbType> {
   &mut self.0
  }
 };
}

pub(crate) use impl_driver_core;

Struct RocksDBDriver now becomes:

impl RocksDBDriver {
 super::core::impl_driver_core!(DB);

 /// # Initializing RocksDB driver
 pub fn initialize(self: &mut Self, cf_name: &String, options: rocksdb::Options) {
  let core = self.get_core();
  ...
 }
}

Contributors

Chung Quan Tin (@chungquantin)

List of No-code tools that will enhance your productivity 10x ⚡️

Tin Chung — Tue, 29 Nov 2022 14:04:14 +0000

Original Twitter thread: https://twitter.com/chasechung111/status/1597466428546121728

What is No-code?

No-code is the next era of the Internet. It enhances the development experience and also reduce cost waste for building unnecessary tools for management or analytics.

Why should you use No-code tools?

No-code tools will be a core infrastructure of every new era business. If you are a developer, don't reinvent the wheel, save your time by using other people products.

Here are 5 no-code tools to try give it a try today:

1. Auth0

Website: https://auth0.com

Try it today for "Sign in with Github" feature. Just a few click and the feature is done. Auth0 is an essential tool for every startup which want to kickstart their login functionality in a few minutes.

2. Novu

Website: https://novu.co

A brand new open-source notification infrastructure which is trending on Github. Implementing notification on your own is a pain in the ass, trust me. Novu is a right painkiller for project to keep user up-to-date

3. Hashnode

Website: https://hashnode.com

As a developer, I used to write a whole website with JAMstack then realized it was a bad decision. Hashnode has good community, analytic tools, rick markdown editor...for every tech guy to start writing blog

4. Mixpanel

Website: https://mixpanel.com

Mixpanel is a powerful UX analytics tool to track the engagement of users with your product. It is sophisticated and easy to embed to your site. This tool will turn your data into profit.

5. Moralis Web3

Website: https://moralis.io

Tasks like connecting to wallet, retrieving data from blockchain, signing a transaction are duplicated across the blockchain industry. Moralis empowers your app with its services to eradicate those.

How did I promote my open source project to get a first star ⭐️?

Tin Chung — Mon, 28 Nov 2022 05:47:35 +0000

Building in public has gained traction in recent years. Instead of building on somebody's dream and earn money, digital nomads start inclining to build on their own dream and make revenue. I am also really into working publicly and contributing to open source projects. Hence, I decided to work on my own project to learn more how this community drives.

What is the project I am working on?

The open source project I mention is SolomonDB: A distributed Gremlin-compatible graph database written in Rust 🦀.

SolomonDB is briefly a graph database that supports Gremlin as a primary query language. The key point of this article is about how did I promote my project SolomonDB to get a first star from stranger on Github. The current status of SolomonDB is that I already get around 4 stars ⭐️ under a week from strange Github users.

How did I get my first star?

Each project can have different approach to gain traction. This is shared based on my personal experience so it might not be suitable to your project but hope it will be helpful.

To me, getting a first step of everything is important, as it shows you are having a progress. For example, building a startup, first step would be first customer. In this case, my very first goal of working on an open source project has been accomplished, getting the first star.

SEO (Search Engine Optimization) for Github repository is quite bad in a first place actually. Not until your project has around couple of hundreds star, it won't be found on Google. So the strategy would be

1. Create a minimal website of your repository

A minimal website can be any kind of website: a landing page, a documentation page or a progress update page. This web page will enhance the SEO of your Github repository as you can give user more information about the repository inside a page which links to your Github repo. In my case, I design a documentation page using Rust mdboook and Github Pages. My page is easily deployed and edited just by using Github Actions and Markdown file. If your project is a Rust app, you can try the stack, it is quite easy to use.

Link to SolomonDB Documentation

2. Write a good README file

Imagine a README is a frontend of your project. If you are a web designer, you definitely understand that a backend only does not make a project works. A README in this case can be considered in a same way. README file gives a clear introduction on what is the project offers and how can user use it for their own purpose. A good README file will bring a good impression to the user that could land you a first Github star easily.

3. Update the CHANGELOG frequently

A CHANGELOG is a diary of your project. It reports an update of the project daily, weekly or monthly. The purpose of writing a CHANGELOG is to keep the user up-to-date with changes made to the repo. CHANGELOG is also very necessary if you want to onboard a new contributor smoothly to the open source project. This is an example of one CHANGELOG.

4. Maximize the benefits of social networking

As a programmer, I can confirm that engineering is usually overvalued and marketing is undervalued in tech community. Normal user or even other developers won't understand you 1000 lines of code as they don't want to waste their time on something that does not bring them any benefit in a first place.

On the other hand, choosing a correct social network to build your project brand gains you a lot of expansion in social branding as well. The main social network I choose to deliver my product update is Twitter.

With SolomonDB, I try to keep have daily tweet about a product with hashtag and tagging to bring it to more targeted user on the platform.

Choosing a right hashtag and right accounts to tag is a strategy. To build your account from scratch is a very challenging task. However, when you reach a certain level, the connection will be automatically expanded due to the power of social network.

5. Active in tech community

Every technology has its own tech community or tech channel (Discord or Reddit). It is fulfilled with people who interested in the technology and its ecosystem. Therefore, announcing your project in those places would improve your project position. In my case, as the main technology I use to build SolomonDB is Rust, I post an announcement in Rust User Forum.

Keep building and connecting

Those above approaches won't bring you an overnight success. It claims to be a long-term journey and persistence is a key. Hence, keep building and connecting with people on social media incrementally. Try to make the loop of feedback and development keeps running and you will see the result.

Thank you for reading until now. Hope this article is beneficial to your open source project. Even though my project SolomonDB is not a successful open source project yet, it is in a beginning and I believe persistence will prevail.

If you are interested with my article.

Please follow my Twitter: https://twitter.com/chasechung111
Give my project a star: https://github.com/nomadiz/solomon-db

Correct me if my new STARTUP idea is WRONG!

Tin Chung — Mon, 21 Nov 2022 09:57:32 +0000

Idea is cheap, execution is everything

This statement is widely popular in a startup world. We all know and understand but it is hard to apply it well in a real world. The ideation stage is the most dreamy and easiest stage kickstarting a startup, no one would want to dig deeper into the later stages. I have a new idea today but tomorrow, it might be obsolete already.

Reflect upon my own experience, there has been thousands of times I surrounded myself with new startup ideas for a few hours and then decide to pick one which I feel most feasible. After a few days coding, I started losing motivation due to the large gap between ideation stage and execution stage. I did not measure thoroughly the workload and essential resources needed for my project. Hence, most of my projects just become unknown repositories and never got released.

Why do I tell you about that shit?

Well, before, whenever I think about doing startup, I envisage something huge and mysterious. Mysterious enough that nobody know the fancy technologies I used for that project so they can't copy it. But it is not a good strategy, what I learnt is that

The more you give, the more you receive

So I think this could be a wise move to share what am I doing with the community and if there's anyone who get interested in the project, I can make a new friend. You know, working on side project feels a little bit lonely.

What am I building?

Enough for beating around the bush, we will jump into the main part now. There's an application on mobile that I am really into called Forest and it is a very good app for tracking productivity. However, it is not free and the feature is quite limited.

And Github is my most favorite platform. The community on Github is so huge and I am also a big fan of it. I started thinking why does not anyone build an app to track the contribution activity on Github. Imaging if you can receive an in-game reward for every contribution on Github, it would motivate programmers a lot. After brainstorming the idea out on paper, I estimate the resources will be spent for this app and it is not much. We have the Github API and all UI libraries in React ecosystem to quick develop this app under two months.

But I want something a little bit adventurous, something like a Pokemon game. Github Octocat can be a base character of the app, instead of planting a tree whenever user finish their goal, we will give user Experience for their Pokemon whenever a contribution goal reached.

Current progress

I kickstarted the project two days ago and already have a very simple version for the app, I named the app GitPet - reference to VoidPet - a very cool Pokemon-like web game.

So what do you feel about this project idea? Your opinion would be a great contribution to the app. I will work on the app and try to deliver it to the community as soon as possible.

Thanks for reading to the end of the blog post.

10 months building a crypto startup and BUMP!

Tin Chung — Sun, 20 Nov 2022 03:23:25 +0000

Let's see where to begin with. I have been working on crypto startup for around 1 year since the great correction happened with a whole crypto market. That is also a period of time which I help to cofound a crypto startup. The startup is in Solana blockchain.

Where does the story begin?

If you follows the market sentiment for Solana blockchain in the upcoming days, it was not in a very good condition, or I can say, extremely negative. It was the worst year for Solana ecosystem since its launch a few years ago. Several large projects got hacked and manipulated by their own DeFi models. Following that, a story of FTX is what pushed the position of Solana in the whole market deviated. FTX was a huge supporter of the Solana ecosystem. With what happened, our Solana believers were devastated. Enough flashback and quick update on Solana current position. However, this incident brought me back to the main track. I gave myself some time to contemplate a right track of my personal career and the direction of a startup.

Career path in crypto

What can I say? Working in crypto does not give you a very good start for your career. I am a final year undergraduate and to be honest, there is too many factors affecting the growth of your career. The good side of crypto is you will gain a lot of money in this field due to the hype of the community in a bull market. But the bad thing that consolidate my point of view is the technical aspect.

There are dozens of blockchain network maintained and released every month. They can be compatible with the existing blockchains virtual machine like EVM-compatible or they will just release a whole new smart contract language and let the marketing team do the rest. As an engineer, you will have to learn several languages if you want to survive in the market. Imagine learning only Solidity and one day Ethereum crashes, your career just become volatile.

As the blockchain is designed to be your project core system, your project relies a lot on how well the core team behinds of that blockchain manage the project. If they just accidentally corrupt the network, you will be affected too. Even though decentralized is an attributed used to grade any blockchain network, the ecosystem is not decentralized on its self.

According to EarthWeb's statistics:

Currently, there are at least 1,000 blockchains with at least four types of blockchain networks. While the idea of blockchain is a singular data transfer type, there are multiple platforms provided in this industry.

Yes, every blockchain will have a specific use cases. But for public blockchains, the way they grow the ecosystem and destroy it with a few clicks is not worth it.

The direction of a startup

What we are facing is unexpected and it is not a fault of the management team or something. We have been building a startup wholeheartedly and candidly. I discussed with my team and what we decided is to keep building on the chain but trying to reach out to a more Web 2.5 user. Web 2.5 user is more off-chain, for example, people who keep their fund on CEX like Binance or OKX. We try to reposition ourselves as a more Web 2 crypto product instead of relying entirely on the onchain technology. From my perspective, there won't be one chain for all. In the end, multi chain should be a right path for every crypto startup. Targeting at one chain is like neglecting the pompousness of other blockchains.

Don't neglect personal branding

Running a startup is actually fun and I learnt a lot along side with my startup founder and the team. However, I realized I had neglected my own startup: My personal brand. There's a good blog I read recently about growing personal brand:
Building in Public - Behind the scenes

The author declared that

But, everything I worked hard for was for the startup, I didn't build up my brand or voice at all in the last couple of years. "Where do I even start?" That was at the top of my mind.

And this is the same thing what I feel when there was a correction in crypto market. It did not only correct the market but also correct my mindset. The trail of most programmers follow is that they think programming is all what their life needed.

However, it's not. Mental health, family, social life and communication is what I would prefer more in life. Programming is still my passion but I truly f*cked up my old life by spending 12 hours a day programming, not until I got some health issues. As a programmer, whatever we do, running a startup or working for a large corporation, does not matter. Both can be destroyed by a financial crisis. But personal branding is what remains after all.

Conclusion

Well, this article is not that I am trying to be a life advisor or something. I hope this can be considered as a self reflection that other pals who in the same situation like I was can take this personally.

Solana DEV #02: Retrieve all Deposit transactions of your wallet

Tin Chung — Mon, 29 Aug 2022 14:07:28 +0000

If you use Phantom wallet to pay for transactions on Solana blockchain, you may recognize that every time you send SOL or SPL tokens to another address, that transaction will be recorded and listed on Phantom wallet interface.

At first, I thought that it is just similar to any other data fetched by using getProgramAccounts with filtering stuffs. However, it does not work that way.

Fetching signatures and transactions

To fetch all deposit transactions of one address, you must get all signatures of that address by using the method getSignaturesForAddress



const signatureInfos = await connection.getSignaturesForAddress(vaultAddress);

And then map the info to get only transaction signatures



const signatures = signatureInfos.map(sig => sig.signature);
const transactions = await connection.getParsedTransactions(signatures);

This approach has a very big problem. It does not allow you to filter out fetched signatures like what you usually do with getProgramAccounts. And in that way, it will just fetch a bulk load of transactions from your address as your address have other wallet interaction beside receive and send coins. To make it more efficient, I try to filter out success signatures from the return data



const filteredSignatureInfos = signatureInfos.filter(sig => !sig.err && !sig.memo);

But unlucky, if your wallet has around 300 or more transactions, the RPC won't allow you to fetch as there's a rate limit. I don't know if there are any other better approaches but this is the one that I found.

How to improve the approach?

This might not be an acceptable answer but there's a few ways that you can improve the case.

Trying to separate a vault from a wallet. A vault will be a PDA that use the wallet address as a seed. In that way, all transactions that the vault includes will be related to tokens and native SOL only. Even the most traffic vault can only have around 50-100 transactions.
Using cache database, partition and event listener: This technique is commonly used in tradition backend development. If you have around 300 transactions for an address, try to partition those into smaller parts: 50 transactions per each. Cache those data and then only update the database if there is a new data coming (use onAccountChange).

Filtering transactions

There will be two types of transactions that you may want to fetch to display a similar result from Phantom: Native SOL deposit transactions and SPL tokens deposit transactions. If you use getParsedTransactions like what I did above, all the data is already converted into human readable language. The raw transactions using getTransactions return only hexadecimal data. I wrote these two methods so that you can reuse it, it's quite simple because all data is parsed already.



// Filter SPL token deposit transaction
isSplTokenDepositTx(tx: any): boolean {
  const ixs = tx.transaction.message.instructions;
  const ix: any = ixs[ixs.length - 1];
  return (
    ix.programId.equals(TOKEN_PROGRAM_ID) &&
    (ix.parsed.type === 'transfer' || ix.parsed.type === 'transferChecked')
  );
}

// Filter Native SOL deposit transaction
isSolDepositTx(tx: any): boolean {
  const ix: any = tx.transaction.message.instructions[0];
  return ix.programId.equals(SYSTEM_PROGRAM_ID) && ix.parsed.type === 'transfer';
}

Solana DEV #01: Fetching onchain data in a right way

Tin Chung — Mon, 29 Aug 2022 12:55:00 +0000

Working with Solana decentralized app from a client side can be a core part of the Solana app development. Before being comfortable with the process, I thought smart contract programming is the one and only concern that we should thoroughly care about. However, similar to traditional website development, data fetching on client side is impactful to the experience delivered to user. In this article, I will list out key notes of data fetching.

Get filtered program accounts

First and foremost best practice is using getProgramAccounts with a filter. This getProgramAccounts method in @solana/web3.js package is extremely useful for fetching all accounts of a provided program ID. However, as in a smart contract, there would be multiple type of accounts, filtering is required for best performance.

/**
 * A filter object for getProgramAccounts
 */
export type GetProgramAccountsFilter = MemcmpFilter | DataSizeFilter;

Now we play with the source code a bit. There are two filtering types that we can reference MemcmpFilter and DataSizeFilter.

DataSizeFilter

/**
 * Data size comparison filter for getProgramAccounts
 */
export type DataSizeFilter = {
  /** Size of data for program account data length comparison */
  dataSize: number;
};

DataSizeFilter is used to filter accounts of a program matched with a declared data size. Simple right? This filtering type is useful for accounts with fixed data size. With accounts contain dynamic size fields such as vector or str.

MemcmpFilter

/**
 * Memory comparison filter for getProgramAccounts
 */
export type MemcmpFilter = {
  memcmp: {
    /** offset into program account data to start comparison */
    offset: number;
    /** data to match, as base-58 encoded string and limited to less than 129 bytes */
    bytes: string;
  };
};

Between the two filtering types, I find this approach is used more as it allows you to filter accounts in a more dynamic and selective way. MemcmpFilter or Memory comparison filter has two fields: offset and bytes. Every account data is a set of bytes, you can use offset to choose specific data field to compare and bytes is a the later part of a comparison.

For example, if I want to get all program accounts with a specific account discriminator, I can set the offset as 0 and get the bytes of the compared account discriminator. This is a trick to get specific type program accounts without using Anchor (as the framework already handle this process under the hood).

I usually use this filtering types to filter out which accounts that a specific wallet owned. To do this, in the Rust program, you must store the wallet address of the owner in the account. For example, I have this account data (using Anchor):

#[account]
pub struct WalletAddressContainer {
  pub address: PubKey
}

To fetch every WalletAddressContainer accounts of a specific wallet address, I would do

// using Anchor
await program.account.flow.all([
 {
  memcmp: {
    offset: 8,
    bytes: publicKey.toBase58(),
  },
 }
]);

// without Anchor
await connection.getProgramAccounts(
 programID,
 {
  ...,
  filters: [
   {
    memcmp: {
     offset: 0,
     bytes: /** Account discriminator bytes of WalletAddressContainer **/,
    }
   },
   {
    memcmp: {
     offset: 8,
     bytes: publicKey.toBase58()
    }
   }
  ]
 }
);

In an Anchor way, as mentioned, Anchor framework handles comparing account discriminator under the hood, so we don't have to compare it again. In that way, we only need one memcmp filter to compare the public key of the wallet but the offset is still have to be 8 because the first 8 bytes are for account discriminator.

On the other hand, getProgramAccounts without Anchor requires two comparison. First the account discriminator and second the wallet address.