DEV Community: Njokom Alain Junior

Improving Bitcoin: My experience contributing to Utreexo for SOB 2024.

Njokom Alain Junior — Thu, 22 Aug 2024 17:38:43 +0000

Over the last 3 months or so, I was privileged to have been selected for the 2024 session of the Summer of Bitcoin (SOB) internship. During this time, I worked on Utreexo project. Utreexo significantly reduces the resources needed to run a fully validating Bitcoin node, by introducing a hash based dynamic accumulator approach for representing the Bitcoin state.

This post documents my experience working on this project, the particular feature I was tasked with, and it's impact to the overall system.

How was the internship carried out?

The internship comprised of several phases. The first phase was the "Learning phase" During this phase, all applicants were provided with resources for learning about Bitcoin. Precisely, we were tasked with studying the book Grokking Bitcoin by Kalle Rosenbaum. This phase consisted of weekly meetups and targets to be met/studied for that week. During the meetings, we were split into several breakout rooms, where, we discuss among ourselves (other prospective interns) about what we understood from the target section for that week, after which we get back in the main Meeting room, where we can further discuss with, and ask questions from the Organizers.

We then proceeded to the next phase, where we were given an assignment to do and submit. The task was to build a simple Bitcoin mining node simulator. We were given sample transactions as json files and tasked with carrying out the steps to validate transactions, assemble a block and successfully mine the block. This was certainly a very challenging task, as I came into this project with no knowledge writing code to perform this. However, the knowledge gathered from studying the above Ebook, and so many other online resources, especially knowledge gotten from learn me a bitcoin were of immense help here. Thankfully, after several weeks of intense hard work and learning, I was able to get a near optimal bitcoin mining simulator. That was one of the proudest moments I have ever had 😁.

Anyways, from here, we had to scan through the available organisations and find one which aligns with what we are looking for and checkout their projects, and write a proposal. Note that it is highly advisable to start searching for organisations and get familiar with them and even start making contributions to them immediately they are added. This will give you a head start and improves your chances of coming up with a great proposal. While I didn't find my preferred organisation early, I certainly did find them on time, and started going through their documentation and codebase to understand what the project is all about and what is required in order to make a contribution.

Finally, I picked two organisations GoCoin and Utreexo. My proposal for Gocoin wasn't selected, however, my proposal for Utreexo, with was about "Implementing new getcfilters message for utreexo roots" was selected. I can't even begin to explain using words how excited I was when I received the mail.

Next up was the contribution phase. Here, we had to actively make contributions to the selected project. This can be considered as the internship proper. Although I got to start exploring the documentation and code base of my project early, I had a hard time getting hold of my mentor, and this set me back a little bit. I eventually got hold of him, and we sped up things. He was really nice and guided me thoroughly throughout the journey.

After working for about a month and a half or so, came time for mid terms evaluation, where progress was evaluated and the first part of the stipend was paid out after passing the evaluation.
We continued working for the rest of the internship period, and eventually, we had the final evaluation towards the end of August.

What exactly was my contribution to the project?

So as earlier mentioned, i was tasked with implementing new getcfilters messages for utreexo roots. These messages were to be implemented as per the BIP157 specification. The aim of this was to allow A light client peer to be able to ask for the utreexo roots at a given height from a peer. Essentially, I had to add a filter type and replace the filter contents with the utreexo roots and serve to a peer when requested. Sounds easy right? So I thought.

I started by defining the protocols for the new message type, adding the new type to the list of supported message types. I called it "UtreexoCFilter". I know,.. not a very convincing name, but well.
Next I wrote the implementation for creating headers for this new filter type
This was followed by the actual implementation of the filter indexer, updating the implementation of the handler, the actual implementation of the handling function and so much more.

I had so many hurdles along the way, making changes that caused the node to panic on start, resolving the panic, but getting unsatisfactory results, initial block download getting stuck, and so many issues that really made me scratch my head. In the end after several days/weeks of debugging, and with the help of my mentor, I was able to resolve all of the issues and get the node fully functional and working properly.

What impact did this have on the project?

The node already supported compact filters. However, adding this new filters gives other Compact state nodes the ability to request utreexo roots from a particular height from a peer. When they receive these nodes, they can easily verify the received roots.

Overall, the internship was an exciting and really resourceful, learning opportunity for me, albeit extremely stressful, but I was very happy to put in the work, and I was glad to see good results. This program has given me a really solid foundation, and confidence to take on any Bitcoin and blockchain related project.
Special thanks to Adi, the lead organizer who was setup everything and made sure I had this wonderful opportunity.
A special thanks too to my mentor Calvin Kim, who was the best mentor anyone could have asked for. He sacrificed his time and energy to make sure he's always available when I need his support, and he also pushed me to my limits and made sure I learnt a lot during this journey.

How to start contributing to Utreexo

Njokom Alain Junior — Thu, 22 Aug 2024 11:03:18 +0000

Have you ever come across an exciting open source project which you would love to make a contribution to, but you have no idea on how to start? Or perhaps you are a relatively experienced developer (Or just someone who’s more verse with open source) who’s looking to contribute to an exciting open source project? Well, I have an answer to your question on “How do I start contributing to open source”.
Technically, I won’t be answering that question in the general sense. This article will be more inclined towards a specific open source project, Utreexo. So, I guess this post aims rather, to answer the question “How do I start contributing to Utreexo?”.

So What if Utreexo in the first place?

Well, if you are here, I’m assuming that you have heard about Utreexo from somewhere, or at least you know that it is a project related to Bitcoin. If you have never heard of it however, that’s still fine, I will try to break it down more, and also provide relevant links, however, it is highly recommended to have a high level knowledge on how Bitcoin works, in order to better understand this article.

Anyways, to answer the question above, let’s give a little background of what Utreexo aims to achieve first.
The Bitcoin network as we know it, (or… don’t know it yet) stores records of all transactions in a digital ledger, known as the Blockchain. This ledger contains records of all the transactions that have ever been carried out on the network. The current state however of the network does not necessarily contain the history of everything that has ever happened on the network, rather, it consists of the set all the Unspent Transaction Outputs (UTXO set). This set holds information about “which wallet has what”. In simpler terms, the UTXO set basically just contains records of how much bitcoin a Bitcoin wallet has in their account. Well, the last sentence isn’t very accurate because our Bitcoin wallets usually have severally Unspent transaction outputs, which when combined, reflects as our wallet balance, and what is available to spend.
The problem is, this Bitcoin state is growing faster and larger as the number of transactions on the network increases. This poses a significant scalability issue, as the larger size requires larger hardware resources, which can therefore dissuade some people from operating Bitcoin nodes among other disadvantages.
That is where Utreexo comes in.

So we ask again, “What is Utreexo”?. Well, Utreexo is a method for greatly reducing the storage needed to run a fully validating node by proposing a hash based dynamic cryptographic accumulator, and introducing a Compact state node which stores only an accumulator representation of the state. These nodes require additional inclusion proofs from the sender, before they are able to verify transactions,

With Utreexo, the current state of the network is represented in a significantly smaller size, which means less resource usage, which will then encourage pretty much anyone to run a Node, conveniently. You can the read more about Utreexo from the following resources ELI5: Utreexo — A scaling solution, or Improving the Bitcoin network using Utreexo, or download the Utreexo white paper from here.

Setting up your development environment
Now that we know a little bit more about Utreexo, let’s dive in to setting up our development environment.

Utreexo is written in Go, so having a basic knowledge and understanding of Go programming language will be an added bonus. However, if you are new to Go, you can check out these resources to know more about the language, and to understand a few basic concepts of the language Effective Go, Learn Go.

I use Visual studio code for this post, but feel free to use any text editor of your choice.
First of all, we will need to install Go programming language. The steps to install will not be covered here, but you can check out this article, detailing how to install and run Go programs. Download and Install Go. Once you’ve finished installing Go, as per the article and ensuring that everything works, by checking the output of

go version

we can proceed to cloning the codebase. Having a basic knowledge of git will be beneficial here. First, we need to have git installed on our computer system. To check if git is installed, you can open a new terminal or powershell window and type

git –version

if git is installed, you will see an output with the version number. If not, you have to proceed to install git using directives from Installing Git.
Afterwards, you can try the the command

git –version

again. If git was successfully installed, we can now proceed to forking and cloning the repository.
Forking the repository basically means creating a copy of the repository on our personal GitHub accounts. Changes we make thereafter will be pushed to our forked repository, and then we can submit a pull request.

To clone the repository, we navigate to the desired location and open in terminal. Or we can navigate directly using the terminal.
For instance, let’s say we have a folder on our desktop called "Projects" which we want our Utreexo code to reside in, we simply navigate to desktop, and then open the “Projects” folder, and inside there, we right click and take “Open in terminal” for windows and Ubuntu. Or we can navigate directly inside our terminal window

First let’s ensure we are in the root directory of our system by navigating there.

Cd ~

And then

cd Desktop/Projects

When in here, we need to clone the repository. There’s a slight consideration to be made here. When you visit the utreexo main organisation on GitHub, we have several repositories. One of which is the utreexo repo, which contains the actual dynamic accumulator implementation, and then there is the utreexod repo, which is a bitcoin full node implementation, that supports Utreexo. We will be focusing on the latter, that is, the bitcoin full node implementation with Utreexo support. Let’s proceed to fork and then clone the repository

To fork the repository, we go to the utreexod repo github page, and on the far right, we will find the “fork” button.

We need to click on this, and follow the onscreen instructions. Once done, our personal github accounts will have the same copy of this reposirory.
Now we can proceed to clone the repository from our personal account

git clone https://github.com/yourgithubusername/utreexod.git

When it’s done, we can open the folder with our text editor, and proceed to install all dependencies as described in the description of the GitHub repo. Now we’re in.

From here, we will want to spend some time, understanding the various sections of the code base, and what each section is responsible for. Once you have a broad idea of the code base, you can go ahead and check the Issues section of the repository to find any issue you would love to tackle, or perhaps a feature suggestion.
Once identified, we create a new branch and name the branch properly

git branch feature1
git checkout feature1

Here, “feature1” is the name of the feature we are going to be working on. Give it a more descriptive name.
Once we are done working on our feature and feel it is ready, we commit and push to github, and then raise a pull request.

To commit, we first of all stage the changes made using

git add .

And then proceed to commit with

git commit -m “Commit message”

The commit message should be a short description of the changes made and it should be inside quotes as mentioned above.
If you are using vscode, the git extension really simplifies all of these processes.

If you need more resources on how to commit your changes and push then to github, check this resource, and for creating a pull request, you can also check here.

After our pull request has been made, we then wait for updates from maintainers, and once everything has been sorted, and probably more changes made requested, the Pull request will be merged, and… there you go. You have Successfully contributed to Utreexo

Open source contributions like this are a driving source of major software, making contributions to open source is very important, as you make changes to software used by tens of thousands of users out there. That contribution brings a high level of satisfaction. From here, it is advisable to continue building relationships with other community members and maintainers and continue contributing to the software on a regular basis.

Improving the Bitcoin network using Utreexo

Njokom Alain Junior — Mon, 29 Jul 2024 20:56:45 +0000

The world is growing ever digital by the day and more and more aspects of our lives have become more and more intangible. Currency hasn’t been left out, from physical cash, to cashless systems and now cryptocurrencies. Since the inception of the first block of Bitcoin in 2009 it’s adoption has grown ever so rapidly and with more adoption comes more stability and trust in the network which will only increase it’s adoption.

First of all, what is Bitcoin, and how does Bitcoin work?

Bitcoin is a decentralized peer to peer network digital currency created in 2008 by a pseudo anonymous entity named Satoshi Nakamoto. The Bitcoin network is a network made up of computers, also referred to as nodes which are interconnected to each other and help propagate transactions and validate transactions throughout the network. This network uses the digital currency bitcoin (BTC). These transactions are stored on a decentralized public ledger which uses blockchain technology.
The Bitcoin blockchain is made up of blocks which are linked to each preceding block, except for the first block also known as the Genesis block which isn’t connected to any preceding block. Blocks are made up of several transactions which have been validated and carefully added to a block by miners, before performing the mining process. A transaction is basically a transfer of value between bitcoin wallets. Miners are responsible for performing the major work on the network which confirm or “mine” new blocks and add them to the blockchain.
There are several resources to learn about Bitcoin available online. These resources give more details about the functioning of bitcoin.
Despite the great things about bitcoin, scalability has always been a major issue. Bitcoin blocks are limited to a size of 1MB, with a block mined roughly every 10 minutes. This has put the current size of the Bitcoin blockchain at around 580 GB as of this writing, which is about 18% up over the last year. For full nodes to join the blockchain and verify networks, they need to download the entire blockchain and start performing performing validation on all old blocks and on newly added blocks. This size is about guaranteed to keep growing by the day and at some point, might become too large, so much so that it dissuades some normal users from joining the network due to it’s resource constraints.
Asides the size of the blockchain, nodes also verify and store the current state of the network. This state is the current Unspent transaction output set (UTXO) which is relatively much smaller in size to the entire blockchain, however, this state as well is guaranteed to keep growing fast as more and more users carry out more transactions on the network. This set is the set of all Unspent transaction outputs in the network.

So what is Utreexo?

Utreexo introduces a hash based dynamic accumulator which allows to significantly reduce the size of the current state. It allows for nodes to fully verify inputs of a transaction without knowing the entire state of the system. It accomplishes this by letting the owner of funds maintain a proof that the funds actually do exist, and they then present this funds when they are about to spend the funds.
Utreexo introduces a new type of node called a Compact State Node. These nodes only store an accumulator representation of the state. In order for these nodes to verify transactions, they need an inclusion proof. This proof is provided by the spending transaction when they are about to spend some inputs.

How does Utreexo improve the Bitcoin network?

As seen above Utreexo allows one to represent the state of the Bitcoin network as a dynamic accumulator, these accumulators are only a few kilobytes in size, as opposed to the current state of Bitcoin, which is over 5GB large.
To understand how Utreexo works, we have to first understand what a cryptographic accumulator is, and how it works. A cryptographic accumulators, allows us to query a set without storing or revealing all members of the set. This accumulator construction approach works great for Bitcoins UXTO set because for each transaction, we would like to query whether the TXOs being spent are indeed members of the UTXO set, and if not, reject the transaction.
Regular nodes when joining the network have to download the entire blockchain history which is over 580GB and verify transactions and build their own copy of the UTXO set. They then have to verify all state changes hitting the node. All of these processes are resource intensive operations, which thereby limit the number of participants in the network which then limits scalability.
This initial synchronization process, also known as the Initial Block Download (IBD) can take a very long time, depending on internet connection and hardware resources. One of the major factors affecting the speed of this IBD operation is the type of storage disk used and the speed of I/O operations, especially, the ability to rapidly perform random access reads. This is why computers using a solid state drive, which normally has far superior random access read times can use over 30 times less time to verify the transactions, as compared to computers with a Hard disk drive.
With Utreexo, the type of disk used wouldn’t make such a massive difference, as we will see just a slight performance difference between SSD computers and HDD computers
Utreexo introduces a hash-based dynamic accumulator with no trusted setup or manager requirements. As mentioned above, accumulators are compact representations of a set, to which elements can be added and proven. A Utreexo accumulator uses a forest of perfect Merkle trees which allows for efficient removals of elements from the accumulator, reducing the total number of leaves in the forest when deletions occur.
Additions are computable without any data beyond the accumulator and the element to be added, and deletions are computable with an inclusion proof of the data to be deleted.
The design of the accumulator is a forest of perfect binary hash trees. The representation of the accumulator which must be stored includes: the number of elements stored, and the root of every tree in the forest.
The logical structure of the perfect binary tree is beyond the scope of this article as this was just an introductory article. However, the full Utreexo paper can be found here.

Conclusion

Utreexo hash based accumulator aims to reduce the size of the bitcoin state to just a few kilobytes, allowing pretty much any device out there to join the bitcoin network and begin verifying transactions with no need for expensive and top shelf hardware. This will greatly increase scalability of the bitcoin network as the size of the accumulator grows very slowly (Onlogn) space complexity.