DEV Community: Noob Science

Understanding Concurrency

Noob Science — Tue, 12 Sep 2023 20:27:47 +0000

Some Background

Well in real life, we are often advised to do one thing at a time and do it perfectly. Did I just use the UNIX philosophy and applied it to real life? Well, yes. I am a nerd.
Programs too, most times are better off doing just one thing and doing it perfectly.
However, it takes a whole 360 degree turn when it comes to computing. The keyword here being computing. Not the fancy CLI's and GUI's we use. Computing is a very broad term and it includes a lot of things. One of them being concurrency.
So let's give a text book definition first shall we:

Concurrency is the ability of different parts or units of a program, algorithm, or problem to be executed out-of-order or in partial order, without affecting the final outcome.

Basically, the program is able to do multiple things at once. So, let's look inside this a bit more. There are two types of concurrency:

Parallelism: This is when the program is able to do multiple things at once. This is the most common type of concurrency. This is what we are going to be talking about in this post.
Asynchronous: This is when the program is able to do multiple things at once, but not exactly at the same time. This is a bit more complicated and we will talk about it in a later post.

So, let's get into the whole thing shall we, but first, let's talk about the anti pattern of concurrency: Synchronous programming.

Jargon Buster and Basics

Before we get into the whole thing, here is a small jargon buster for you:

Blocking: Blocking is when the program is waiting for something to happen. For example, if you are reading from a file, the program will be blocked until the file is read.
Thread: A thread is the most basic unit of execution. A process can have multiple threads. Each and every thread in a memory space has access to the same memory. Hence, they can share state between each other.
Process: A process is a program that is running. A process can have multiple threads. Each process has its own memory space.
Daemon: A daemon is a process that runs in the background.
IPC: IPC stands for Inter Process Communication. It is a mechanism that allows processes to communicate with each other. This can be done using TCP/IP, Sockets, Message Queues, etc.
Operation: An operation is a task that the program has to do. For example, reading from a file is an operation.

Synchronous Programming

Synchronous programming is when the program does one thing at a time.
Let's say we have a very simple web server written in go and it has to do some algorithmic computation. So, it will do the computation and then send the response to the client.
For every request sent, imagine an imaginary plate stack. The program will take a plate from the stack, do the computation and then send the response.

Essentially when you imagine a stack of plates, you put all the plates on the fancy plate stack the pops out the plates one by one. This is how synchronous programming works. It does one thing at a time and only then exposes the next thing to do.

Hence, if you are doing a lot of computation, the program will be blocked until the computation is done.
Let's say that you are reading from an external API, and the API is slow. The program will essentially be blocked until the API responds.
Hence, your program is essentially choke holding itself.
This is where the concept of concurrency comes in.

Concurrency

Now imagine you tilt the plate stack to be horizontal. You can now do multiple things at once. You can take any plate from the stack and enjoy the yummy food (computations) on it.
So, essentially, your program is no longer blocked by a long computation. It can do other computations while the long computation is going on.
In fact, it doesn't matter if the computation is long or short, the program can do other things while the computation is going on.

Hence concurrency sort of does stuff in parallel. It is not exactly parallelism, but it is close enough.
We will be differentiating between concurrency and parallelism in a bit.
Regardless, this pattern of programming is called concurrent programming.

Async and Concurrent are used interchangeably, however, as I mentioned in the beginning, they are not the same thing.
Asynchronous programming is a subset of concurrent programming.
So what are the other subsets of concurrent programming?

Classification of Concurrences

There are boardly three types of concurrent programming:

MultiThreading
MultiProcessing
Asynchronous

Let's talk about each of them one by one.

MultiThreading

MultiThreading is often confused with parallelism. However, it is not the same thing.
MultiThreading is when you have multiple threads of execution in a single process.
A thread is the most basic unit of execution. A process can have multiple threads.
Each and every thread in a memory space has access to the same memory. Hence, they can share state between each other.
This solves the problem of blocking. If one thread is blocked, the other threads can continue to do their work.
We will be talking more about blocking in a bit.

You can think of a thread as a worker. Each worker has a task to do. If one worker is blocked, the other workers can continue to do their work.
Hence, essentially, you are running a main worker called the daemon and the daemon is spawning multiple workers to do the work, while it self is not able to interfere with the worker's state.
You might have heard of the term daemon in UNIX. A daemon is a process that runs in the background.

MultiProcessing

MultiProcessing is when you have multiple processes running at the same time. Each process has its own memory space and hence, they cannot share state between each other.
This is particularly useful for a micro service like architecture. Each process can be a micro service and they can communicate with each other using some sort of IPC (Inter Process Communication) mechanism.
These methods can vary from TCP/IP to Sockets to even message queues.
MultiProcessing finds its use in distributed systems and in micro service architectures.

Asynchronous

An async operation runs on a single thread know as the event loop. This event loop is responsible for running the async operations.
The event loop is non blocking and hence, it can run multiple async operations at the same time.

However, the async operations are not exactly running at the same time. They are running one after the other, but they are not blocking each other.
What I mean by blocking is that the event loop is not waiting for the actual data. It operates on the promise of the data. This might be a bit familiar if you have used JavaScript.
The event loop is not waiting for the data to come back from the API. It is just waiting for the promise of the data to come back. Once the promise is fulfilled, the event loop will run something called a callback function. The promise is called using the await keyword in most languages.

let res = new Promise((resolve, reject) => {
  // do something
  resolve("Hello World");
});

let data = await res;

In the above example, the program interperts data as a promise that will be fulfilled by the variable res. Once the promise is fulfilled, the program will run the next line of code.
So, the program can operate on the principle of promises. This is what async programming is.

This is basically like you using one company's stocks to buy another company's stocks. You are essentially operating on the promise of the first company's stocks inherent value and you will be able to buy the second company's stocks.

Asynchronous is hence the most basic form of concurrency. You don't have something running in parallel technically, but sort of dismissed on the basis of the promise of the data.
This helps the event loop to run multiple operations at the same time.
There is a very famous talk by Philip Roberts on the event loops that gets into the nitty gritty of how the event loop works. I highly recommend you to watch it.

I plan on doing a deep dive into async programming in a later post. So stay tuned for that.

Blocking

Blocking occurs when a program is paused or held up, waiting for a specific event to complete, such as reading data from a file. During this time, the program can't proceed because it relies on the outcome of that event. Concurrency, on the other hand, is a technique that allows a program to continue executing other tasks while one or more operations are blocked, enhancing its overall efficiency.

Think of it like a restaurant scenario. You're a waiter taking orders from customers and passing them to the chef. The chef prepares the dishes and hands them back to you, and you serve them to the customers. It sounds straightforward, right?

However, consider a situation where a customer orders a complex dish that takes a long time to cook. In a blocking scenario, the chef would wait until that dish is ready before starting any others, causing delays. This is similar to how a program can get blocked when waiting for a time-consuming operation to complete.

Now, let's introduce concurrency. If the chef decides to cook the complex dish concurrently, it means they can work on other orders while the challenging dish is being prepared. This parallels how a program can continue performing other tasks even while waiting for certain operations to finish. This ability to multitask or work in parallel, like our chef, is the essence of concurrency, making processes more efficient and responsive.

Understanding through a Go Program

Now, let's understand this through a simple go program.
We want to emulate a web server that some complex algorithmic computation. We will be using the time.Sleep() function to emulate the computation and be calculating the time taken to do the computation.

Below is an example of a synchronous program:

package main

import (
    "fmt"
    "time"
)

func main(){
    start := time.Now()
    fmt.Println("Program started")

    fmt.Println(emulateComputation(1))
    fmt.Println(emulateComputation(2))
    fmt.Println(emulateComputation(3))
    fmt.Println(emulateComputation(4))
    fmt.Println(emulateComputation(5))

    fmt.Println("Program ended in", time.Since(start))
}

func emulateComputation(id int) string{
    time.Sleep(1000 * time.Duration(id) * time.Millisecond)

    return "Computation done for " + fmt.Sprint(id)
}

Notice how we must first wait for the id 1 computation to finish before we can start the id 2 computation.
This is because the program is synchronous and it can only do one thing at a time.

When we run the program, we get the following output:

Program started
Computation done for 1
Computation done for 2
Computation done for 3
Computation done for 4
Computation done for 5
Program ended in 15.0052ms

So, the program took 15.0052ms to complete. This is because the program is doing one thing at a time and hence, it is blocked by the computation.
Finding the time complexity of this program is pretty easy. It is O(n) where n is the number of computations.

Now, let's make the program concurrent. We will be using the go keyword to make the program concurrent.

package main

import (
    "fmt"
    "sync"
    "time"
)

func main() {
    now := time.Now()

    resch := make(chan int, 5)
    wg := &sync.WaitGroup{}

    go emulateAction(1, resch, wg)
    wg.Add(1)
    go emulateAction(2, resch, wg)
    wg.Add(1)
    go emulateAction(3, resch, wg)
    wg.Add(1)
    go emulateAction(4, resch, wg)
    wg.Add(1)
    go emulateAction(5, resch, wg)
    wg.Add(1)

    go func() {
        wg.Wait()
        close(resch)
    }()

    for res := range resch {
        fmt.Println("Result:", res)
    }

    endProgram(&now)
}

func emulateAction(id int, resch chan int, wg *sync.WaitGroup) {
    time.Sleep(1000 * time.Duration(id) * time.Millisecond)
    resch <- id
    wg.Done()
}

func endProgram(n *time.Time) {
    fmt.Println("Program ended in", time.Since(*n))
}

The first one was pretty simple? I know. This one is a bit more complicated. Let's break it down.
So, go has this concept of channels. Channels are a way to communicate between goroutines. Goroutines are like the threads in go.

Essentially a channel is like a way to share state between these different goroutines. It is a type of IPC (Inter Process Communication) mechanism.
I mean, not exactly, but it is close enough. Now, we use a waiting or a blocking mechanism to wait for the goroutines to finish.
The sync.WaitGroup is a way to wait for the goroutines to finish. The rest is handled native to go.

When we run the program, we get the following output:

Program started
Result: 1
Result: 2
Result: 3
Result: 4
Result: 5
Program ended in 5.0120ms

Hence, the total time of execution is the same as the longest computation. This is because the program is doing multiple things at once.
While the timer is counting to 5 seconds on one thread, another thread is already done with counting 2 seconds and is waiting in the waiting group to be closed.
The outputs are systematically put in the channel resch and then the main thread prints them out using the range iterator.

Interesting thing to be noted here is that the order of the results is not the same as the order of the goroutines.
This is because the goroutines are running concurrently and hence, the order of the results is not guaranteed.
One more thing that beginners often get confused about is that the program is not running in parallel.
It is running concurrently. Parallelism is when the program is doing multiple things at the same time.
Concurrency is when the program is doing multiple things at once. The difference is subtle, but it is there, which we will be talking about in a later post, so stay tuned for that.

Where and Why to use Concurrency

Concurrency is a very powerful tool. It can significantly improve the performance of your program.
Concurrency might be a little harder to implement than synchronous programming, but it is worth it.
However, it is not a silver bullet. It is not a tool that you can use everywhere. It is a tool that must be used wisely.
For example, if you want to read from a file, you don't need concurrency. You can just read from the file asynchronously and you will be fine.

Concurrency is specifically used when you have to compute almost parallel. For example, if you have to read from multiple files, you can use concurrency.

With that being said, let's talk about some of the use cases of concurrency:

WebServers: Web servers can use concurrency to handle multiple requests at the same time. This is a very common use case of concurrency. Moreover, you can also handle the loss of a thread by spawning a new thread.

Database Management Systems: MultiThreading can be used to handle multiple queries at the same time. This makes the database more efficient and responsive.
Distributed Systems: A system that can efficiently handle multiple requests at the same time is a distributed system.
Video Streaming: Streaming services use concurrency to serve multiple users simultaneously. Each user's request for video content can be processed concurrently, allowing for efficient streaming.
Video Games: Video games use many concurrency patterns like the actor model to handle multiple players at the same time, lobbies, etc.

Concurrency Patterns

Concurrency patterns are a set of patterns that are used to solve a specific problem. They are a set of best practices that are used to solve a specific problem.
They are some very interesting and powerful patterns that are used in concurrency. I will listing some of them below, they will be explained in a later post.

Actor Model: The actor model is a model of concurrent computation that treats actor as the universal primitive of concurrent computation. In response to a message that it receives, an actor can: make local decisions, create more actors, send more messages, and determine how to respond to the next message received.
Thread Pool: A thread pool is a software design pattern for achieving concurrency of execution in a computer program. Often also called a replicated workers or worker-crew model, a thread pool maintains multiple threads waiting for tasks to be allocated for concurrent execution by the supervising program.
Futures and Promises: Futures and promises originated in functional programming and related paradigms (such as logic programming) to decouple a value (a future) from how it was computed (a promise), allowing the computation to be done more flexibly, notably by parallelizing it.

Executive Summary

If you're time is limited, here is a quick summary of the post:

Concurrency is the ability of different parts or units of a program, algorithm, or problem to be executed out-of-order or in partial order, without affecting the final outcome.
There are three types of basic concurrency:
1. MultiThreading
2. MultiProcessing
3. Asynchronous
MultiThreading is when you have multiple threads of execution in a single process.
MultiProcessing is when you have multiple processes running at the same time.
Asynchronous is when you have a single thread of execution that is non blocking.
Blocking is when the program is waiting for something to happen.
Concurrency is when the program is able to do multiple things at once.
Parallelism is when the program is able to do multiple things at the same time.
Concurrency significantly improves the performance of the program.
However, it is not a silver bullet. It can be a bit complicated to implement and it can be a bit hard to debug.
Moreover, not everything must be concurrent. It is a tool that must be used wisely.

Conclusion

So, this was the primer on concurrency. I hope you enjoyed it. I will be doing a deep dive into async programming in a later post.
I mostly addressed some of the major questions I had when I was learning concurrency. I hope this post helped you.
If you have any questions, feel free to reach out to me on Twitter or contact me through my website.

With that being said, I hope you had a great time reading this post. Have a great day and keep reading Ishan Writes. Bye!

Thanks for the enormous response I got for my posts here ❤️. I hope you enjoyed this style of blog posts, with the illustrations and weird examples. If you did, please let me know 😄.

Grow into Programming

Noob Science — Sat, 09 Sep 2023 18:45:11 +0000

This October marks 3 years since I started programming. Do I feel good? Yeah obviously! I love coding and my passion for programming has only grown since when I started. I love learning new stuff, trying to keep up with what’s new and just over all have a great time.

Talking cool stuff about programming however is not why I am writing this article. The reason I am writing this article is well, I wanted to talk about something that doesn’t get said very often in programming. I know I am not like a senior dev with all this cool experiences and anecdotes I can tell you, but yeah I can tell you are few anecdotes from my own learning experiences. So buckle up my friend.

Programming is not something you learn. In my opinion, it is like an art form that is perfected through smart repetition and having genuine interest. Some of the best programmers I have ever had the pleasure of talking to (chatting with) are some the most humble and amazing people that really love what they do and are pretty good at it. That is something that is taken for granted quite often actually. Programming is not a skill, it is an art. Yes I know I sound like a old couple in a old museum, but I mean it.

Git Checkout

I’ll start with a very good example. If you are a programmer, you know what git is. It is a version control system used for tracking changes in software development and if you are not a programmer, think of it like Google Drive for your code (I know this definition is not right, but it is not wrong as well, so 🙂). I started using GitHub quite early in my programming journey. My first repo was just to store my Firefox config. It’s been nearly 3 years since I am in the code thing and I started using commands like git branch, git checkout and git stash just yesterday when I was working on a rust CLI that wraps around git and makes it easy for beginners to start out. That is how long is took me to even use the most basic of git features.

This might seem so wrong to some of you. I mean, how can I use git without completely knowing everything about it. Here’s the thing. I didn’t learn git, I grew into git. You learn something for the sake of learning it or probably because you want to. Whereas you only grow into something when you need it. It is sort of like “Necessity is the mother of all invention”.

Knowing and Learning

We often think that knowing something as completely learning it. This might be true to other sciences, but programming I feel is different. You can never really learn anything. You only know stuff and have used it enough number of times to accurately reproduce the results. That is why developers write code that is so varied. Some like to use the docs, others like to ask on forums or increase their stack overflow points, even use ChatGPT. All of these methods are valid. Because I feel that you are ultimately learning how to solve a problem. Not a learning a language.

This does not mean that I am asking you to be a script kiddie, the type of programmer that cannot even import a function without autocomplete or “looking it up”. You should obviously know what you are doing and must know a language as much as possible. But here’s the thing. You should not actively seek knowing the language as the only measure of your programming skills. This again ties into the concept of not being only a “React Programmer” or just a “Flutter Dev”. I believe once you know the basics of a programming language, to write code in another one, all you should know is the syntax of that programming language.

One you are familiar with any programming language, trying out a new one should not be that big a deal. Just get to know the basic syntax of the language and implement some programs in it. I have a certain set of programs I write in a language before I am ready to actually use it. Doing this gives me confidence on the language and that is enough for me to start a new project in the language. Do I know everything about the language? No! But I still can solve a problem in it. I use the docs, sometimes stack overflow and ever since I got the student access to GitHub Copilot, I don’t even need to exit my VSCode or Neovim to ask AI something.

If you are wondering why I don’t need to exit, try this. Copilot is technically still GPT. So, if you ask it a question in a comment with q:, it will answer it. So, something like
q: what are rust macros will get it to answer it like normal GPT.

So, all you need to know is why you want to do something,
the rest can be figured out along the way.

The Issues

It is wrong to assume that this is the best way to do programming.
I have never really worked in the industry, so I don’t know how it is there.
However, I don't think it would be financially viable to learn as you go in a company.
I mean, you are getting paid to do something, so you should know how to do it.
However, I am not talking about the industry. Atleast for now, I program for fun, it is a hobby.

So, I don’t really need to know everything about a language to use it.
I just need to know enough to get started.
Moreover, I am mostly a project based learner.
I write code to solve a problem that most times is not mathematical.
So, I can't speak for leetcoders as well.
All these issues are not something that should stop you from learning this way, but rather short comings that should be
addressed by you later on so that you can be a better programmer.

However

I am not saying that you should not learn a language.
You should know the language you are using.
You should try and understand it's inner workings, how it works, what it does etc.
Honestly, it is quite fascinating to know how a language works.
I am currently learning rust and recently wrote my first lexer and parser.
I am quite proud of it and I am sure I will be using it in my future projects.

Hence all I am saying is that start to treat programming like an art form more than a skill.
An art is something you can work on for years and still not be perfect.
An art is something that you can never really learn, you can only grow into it.

How to grow into a language

Well the idea behind it is quite simple. Don't focus on the language, focus on the problem.
If you are trying to solve a problem, you will automatically learn the language.

I mean, let's say that you are writing a parser in rust.
Don't start by learning the whole of rust. Know a little bit about it, just enough to get started.
Then start writing the parser. Oh so what do you need for the parser? You need to know how to read a file.
Okay, cool. Read about the fs module in rust. Then you need to know how to build a Abstract Syntax Tree.
Okay, cool. Read about the enum and struct in rust.
You get the idea. You are learning the language as you go.

You are not opening up the docs and reading the whole thing like reading your favourite novel.
Once you have written the parser, you will know a lot about rust.
This is because you have used it and you have used it enough number of times to know it.
That is how you grow into a language.
You don't learn it, memorize it and then use it.
You use it and then you automatically learn it.

Similarly, you don't need to know every command and every vim movement to start using vim.
I mean, the only thing you need to know is how to exit vim. That too you know only because you entered vim in the first place.
So, don't just be stuck and sucked into these online courses that teach you how to code.
They are useful no doubt, but they are not the only way to learn programming.
An artist can never really learn how to draw. He just draws and draws and draws.
He grows into it. He learns how to draw a face, then a body, then a landscape and so on.

So, to all my fellow code artists (cringey I know), keep on coding and keep on growing.
I know I want to.

The End

So, the next time you see someone who asks you which is the best way to learn a language, tell them that there is no best way.
There is only the way that works for you. If you are a person who likes to learn a language before using it, go ahead.
I am not saying that is wrong. I am just saying that it is not the only way to learn programming.
You can learn programming by just doing it. You can learn programming by just solving problems.

One more small disclaimer. This is purely my opinion and not a fact.
At this point I am just rambling on and on about how I feel about programming.
All these might be wrong and I might be a complete idiot and that's okay. So, yeah.
Take everything I say with a grain of salt. I am just a 18 year old kid who likes to code.
I am not a senior dev or anything. I mean, I want to be one, but I am not one right now.
Maybe when I like 30 or something, I will look back at this article and laugh at myself. Who knows?
Then I'll probably write another article about how I was wrong and how I have grown as a programmer.
Till then, I'll just keep on coding and keep on growing (That should totally be a song or something).

Thanks for reading this article. I hope you liked it. If you did, please consider sharing it with your dev friends or on social media.
It would mean a lot to me. If you have any suggestions or feedback, please let me know in the comments below.
Hope you have a great day :)

Chad-ifying My Web Tools

Noob Science — Wed, 06 Sep 2023 19:28:55 +0000

I am almost certain that if you are a web dev, actually scrap that, if you are a developer, you at some point or the other have
built at least one of the following:

A URL shortener
Paste bin
Markdown editor
Todo list
A CLI (Any CLI, it doesn't matter)

These might seem very superficial and useless, but they are far from that. So this post is to show you how I built my own versions of these tools, and how you can too. Maybe this article will inspire you to finally dig open the grave of that URL Shortener you started 2 years ago and never finished. However, along the way, you might end up making some mistakes that I made, and I hope this article will help you avoid them. With that out of the way, let's get started.

What I have built

Well I have built a lot of things, most of them not worthy of being mentioned here. I am primarily a project based learner, so I learn by building things. I honestly have a folder called "graveyard" where I keep all my failed projects. But I have built a few things that I am proud of, and I will be talking about them here. Most of these tools are public, you can give them a try if you want. All of these are open source of course, and I will link the source code in the respective sections. They are also quite easy to build yourself. I will be mentioning the architecture of these tools as well so you can build your own versions of them. However, you are free to also just fork my repos and build on top of them. I will be happy to see what you build.

NoobShort

NoobShort is one project that I have a lot of memories with. It was first a very simple URL Shortener built using Flask and Postgres. It was a very simple project, but it was one of my
first major projects that I actually deployed. Back when heroku was free, it was called Shortpaw. You can actually still find the original source code [here (https://github.com/newtoallofthis123/shortpaw). When heroku ended their free tier, I sort of abandoned it and started using other URL shorteners. Shortpaw was an excellent learning experience for me. With a lot of hard work I introduced reactiviy to the app, using jquery and ajax. I also learned how to use Postgres and Flask. This was a precursor to all the comforts that React and Next.js provide.

A while ago, when I was learning Next.js, I decided to build a URL shortener again. This time I decided to use Next.js and MongoDB. I also decided to make it a bit more advanced. I coded an API and a beautiful frontend using tailwindcss and that was it for a while. Looking back, it was quite cool that I used a stack that is so easily deployable, nearly free. So when I got my domain noobscience.rocks and started building a portfolio site, I just decided to leave it as a subsection of my portfolio site. I also decided to make it open source. You can find the source code here.

When I migrated my site to Astro, I just split it into API and frontend components. The frontend form is now a component that I can use anywhere. The API is part of Astro's serverless functions.
I decided to not allow anyone to custom slug URLs, because I didn't want to deal with the hassle of moderating them.
I also added a small rust CLI for it which makes it very easy to use. You can find the CLI [here (https://github.com/newtoallofthis123/short_cli").

That is how it looks. It is quite simple, but it works. I have also added a few features like custom slugs, and a few other things. I will be adding more features in the future.

NoobPaste

You know how you have that one project that has so much nostalgia attached to it that you just can't let it go? Well NoobPaste is that project for me. It was one of my first projects that I actually deployed. It is once again built using Flask and Postgres. It is a very simple Paste Bin that allows you to paste code and share it with others. It is very simple, but it works. I have used it a lot of times to share code with my friends. Then original app was a huge flask app with a lot features like a full code editor, syntax highlighting, and a lot of other things. I have since then stripped it down to the bare minimum. You can find the original source code here

Now however, it is a very simple and beautiful frontend built using tailwindcss and a very simple API built using Next.js and MongoDB. I added the ability to handle all the code snippets from my custom built rust CMS which I will talk about in one my previous posts. One of the most nostalgic things about this project is that I used to use it to share code with my friends.
I pretty much still use it for the majority of my code sharing.
This is how it looks:

I hope you are able to see a pattern here. I am a huge fan of simple and minimalistic UIs. I also like to keep my projects as simple as possible. I don't like to add a lot of features to my projects. Why? Well I'll be talking about that in a bit.

HTMLEr

Any markdown app that you use is probably a 100 MB download. I don't know why, but they are. I don't like that. I don't like downloading a 100 MB app just to write markdown! So, I decided to build my own markdown editor. I decided to call it HTMLEr. It is a very simple Markdown Editor built using plain HTML and JS. Okay I added a little of Tailwind CSS, but that's it. I promise. The source code is very simple so if you are interested, you can check it out here.
It uses the marked.js library to convert markdown to HTML. I in fact am using it to write this article. It is very simple, but it works. The best feature? Every click is a save. So you don't have to worry about losing your work. Everything and I mean everything is saved in the local storage. So you can close the tab, close the browser, restart your computer, and your work will still be there.
You can even save it as a file. It is very simple, but it works. This is how it looks:

Quite fun and minimalistic right? I love it.
It is like my go to writing app. Best part? I just have to open a URL. No downloads, no installations, nothing. Just a URL. I love it.

NoobKey

NoobKey is a very simple Key Value store built using Rust.
It wrote a lot about it in my previous post. You can check it out here.

Why I built these tools

The internet is full of tools. There are so many. You have something for everything. Want to unwrap a JSON? There is a tool for that. Want to convert a PDF to a Word Document? There is a tool for that. It is a wonderful eco system that we have built. But there is one problem. Most of these tools are either paid or have a lot of ads. I don't like that. I don't like paying for things that I can build myself. That is just a personal preference.

Keeping aside the fact about paying, there is something else that we need to take into consideration. Privacy. Most of these tools are not open source. So you don't know what they are doing with your data. URL's for example are a very private thing to share. You key value store is also a very private thing. You don't want to share it with anyone. So why would you use a tool that you don't know what it is doing with your data? It is a very different story if the tool you are using is open source. But most of them are not. So I decided to build my own tools. I decided to build tools that I can trust. Tools that I know are not doing anything with my data. Because I own my data. I even took it one step further and even bought my own domain.

I trust my tools because they are after all, my tools. If I didn't like a feature, it is a commit away. If I suddenly find the whole UI ugly, that too, is just a commit away. Moreover, once I figure out the API's and stuff, I can use them in my other projects as well. I can use my URL shortener in my portfolio site. I can use my paste bin to share code with my friends. It just feels so empowering to have your own tools. It is like having your own super powers.

Advice

While building these tools, I have made a lot of mistakes. Here are a few lessons that I learned along the way.

Keep it simple stupid

Keep your tools as minimal as possible. There is no need to add authentication, or a CRUD API, or a lot of other things. Keep it simple. Keep it minimal. You want to build a URL shortener? Just build a URL shortener. Don't unnecessarily add a lot of features. You can always add them later if you want to. For example, my pastebin doesn't need a code editor. It is a pastebin. It is supposed to be minimal and fast. Just a textarea where I can paste my code and share it quickly with a very easy to access URL.
The syntax highlighting too is not very customizable. I just use a single highlight.js theme. I don't want to add a lot of themes either.

The more you try and add features, the more you will get distracted. The point is not to have a sass product that you can sell. The point is to have a tool that you can use. The keyword being just you. You don't need to build a product. You just need to build a tool that you can use. So keep it simple. Keep it minimal.

Don't over engineer

Choose a very simple and easily deployable stack. I chose Next.js and MongoDB. It is very easy to deploy. I can deploy it on Vercel for free and don't have to worry about it. Not need to worry about scaling, misusing resources, or any of the stuff that I would have had to worry about if I had used something like Django or Express.
Similarly, I use Rust for building my CLI tools because it just works. I have never had a rust tool fail on me. It is very easy to build and install. Portable, functional and fast.
That is all I need.

Don't over optimize

It is okay if your tool is not the most fastest, SEO optimized, or the most beautiful.
Remember you are the only one who is going to be using it. Yes other people can, but that is not the point. The point is to build a tool that you can use. If other people start using it, good for you. But that is not the point. The point is to build a tool that you can use. So don't worry about SEO, or performance, or any of that stuff.

Remember it is your tool so you can always optimize it later if you want to, but you don't need to. If you chose a simple stack, you probably have enough server resources to not worry about performance. Remember your tools is not a product. It most probably wont' have a lot of concurrent users and that is okay.

It is for you

Not for anyone else. You are building this tool for yourself. So don't worry about what other people think. If you want to add a feature, add it. If you don't want to, don't.
For example, I always found the QR code feature in URL shorteners to be very useless. I never used it. So I didn't add it. I don't care if other people want it. I don't want it, so my tool doesn't have it. Yes, it really does feel that powerful. You can build your own tools, and you can build them however you want.

Don't be afraid to change

Delete the whole project if you want to. It is your project. Let it grow with you. If you want to change the stack, change it. If you want to change the UI, change it. Let your tools reflect your unique personality. Give them space to breath. They are not putting bread on your table and are not supposed to. They are just tools that you can use.

Some Observations

My work flow with these tools is quite simple actually. I use my URL shortener to shorten all my URLs. I use my paste bin to share code with my friends. The markdown editor is what I use to write all my articles. I use my key value store to store all my keys and secrets. The Custom rust CMS that I built is used to manage the whole MongoDB.
Hence, I don't need to build a lot of features into my tools. I don't need to add authentication, or a lot of other things.
ALl my tools don't even have the option to delete anything because I don't need a bloated UI for that, nor do I need to worry about security of my data. If I want to delete something, I just use the CMS CLI to delete it. It is that simple.

Moderation? What even is that? I obviously won't be sharing any illegal stuff on my tools, but still, the feeling of not being moderated is quite liberating. What about me being a bad actor? Well first up, all of my tools are open source. So you can see what they are doing. Second, I am not a bad actor. I don't want to be one. All this is quite cool and all but what about others being bad actors? Well, the DB is in my control. I can always delete stuff if I want to. In fact, about once every month, I just delete the last 100 entries of my URL shortener. I don't need them. I don't want them. So I just delete them. It is that simple.
Some of the more permanent stuff like my keys and secrets are stored in my key value store. I don't need to delete them. I just update them if I want to.
It is honestly so cool.

Conclusion

Well, this was once again, one of those weird posts that I just wanted to write. I hope you enjoyed it. I hope you are inspired to build your own tools. If you do end up building something, do let me know in the comments. I would love to see what you build. I hope you learned something from this post. It is always liberating to start having your own spaces on the web, be it a blog or a tool. It is a very different feeling. I hope you get to experience it and I hope you enjoy it as well.
So, I hope I have inspired you enough to start digging your old projects and start building your own tools. Remember, it is not about someone else using your tools. It is about you using your tools. So, build it in such a way that you can use it. Making it public is just a bonus. So, go ahead and build your own tools. I hope you have fun doing it. I know I did.
Thanks for reading. Have a nice day. Bye.

PS

Here are the links to all the tools that I mentioned in this post:

Small disclaimer before using them though. All of their content is clearly visible to me. So don't share anything that you don't want me to see. I don't care about your data, but I can see it and I am not comfortable with that.
Care about your data. Be safe. If you want to use them, I recommend you fork them and build your own versions. I will make a future blog post or maybe a doc on how to do that.
Keep upto date with my blog to know when that happens. Thanks for reading. I hope you find my tools useful. I know I do.

Migrating My Site To Astro

Noob Science — Wed, 06 Sep 2023 16:34:45 +0000

It feels good doesn't it? You've just finished building your website and you're ready to show it off to the world. You've got your domain, your hosting, and your site is live. When we as developers build a website, we expect it to work seamlessly for the rest of time. Now this might be true for a static website, which is hosted on a CDN and doesn't have any backend code. However, most sites now a days are dynamic, and have some sort of backend code, which is where the complexities of maintainance come in. Now it is not really the backend which is to blame. Sometimes find even the frontend to be a pain to maintain. Especially the front ends which are built using meta frameworks like Gatsby, Next, Nuxt, etc. Dynamic and interactive websites are great, but they come with a cost.

Why I was Migrating My Site

This was when my site was built using Next.js and I was using a custom CMS I made using MonoDB to manage the content. Now this was a very small dynamicity, but it was still a pain to maintain. I also had to make sure that I don't exceed the free tier of MongoDB Atlas, which was a pain. I had to make all sort of changes to the SWR hooks to make sure that the site doesn't make too many requests to the database. After all this too, the site was rather slow at times and on top of that, the site was 500GB in size.

The thing was, the reason I had to introduce this dynamicity was because I wanted to make the content on my site easily manageable.
I wanted to be able to add new quips (called updates) to my site without having to make a new commit to my site. However, this would be one of the best examples of over engineering.
I didn't have to make a whole CMS for this. Now trust me, it was fun to make the CMS, but it was not worth it. It slowed my site terribly and most of the libraries that were used to render the markdown I was storing in the database were not working properly with Next.js's new server components. Moreover, as I will be talking more about later, writing the code using messy scss and weird React hooks made the code really hard to maintain. I realized that in the age of static site generators, I was using a meta framework to build a simple site, with very little dynamicity.

So, I decided to migrate my site to a static site generator. The only question was which one?

Why I Chose Astro

I had used Gatsby in the past, but I didn't really enjoy it. Gatsby is an excellent for high performance, commercial sites, but it is not really suited for personal sites. I mean, it is, but it was not for me. It was a bit too complex to set up. Powerful, but complex. This would then, once again lead to over engineering. Hence, I wanted to use something which was simple, yet powerful.
I had already learned and proved that I could build a site using Next.js, so I wanted to try something new. I had heard about Astro, and I decided to give it a shot.

Astro is a static site generator which is really unique. It provides a lot of features which are not available in other static site generators. For one, it feels very natural for a Web developer to use Astro. You can write normal HTML, CSS, and JS in Astro. However, what makes Astro unique is that it allows you to use React components in your HTML. Not just react components, but you can use any framework in your HTML. This is because Astro is framework agnostic. This means that you can use any framework you want in your Astro site. Astro uses the concept of islands, which are basically components that can be rendered using a framework of your choice. This is a much cleaner way of doing things, as you don't have to worry about all the messy code.

Hence, Astro seemed like a good choice for me. So, with that, I started migrating my site to Astro. However, before that, I needed to identify the pain points in my site. This would help me not make the same mistakes again. With that, let's talk about the pain points I identified in my site.

The Pain Points

In any analysis, the first step is to identify the pain points. Now, you might multiple pain points in your site, but you need to identify the ones which are the most important. So pain points I found in my site were:

Migrating from page/ to app/ in Next.js
The messy scss
Long repetitive code
Really slow build times
Easily Breakable code

Now these all might sound sort of a given, but let me explain why these were pain points for me. First off, with the release of Next.js 13.4 and the release of the app/ directory, I had to migrate my site from page/ to app/. This was easier said then done, as I had a lot of problems with the messy scss my was using.
This led me to using weird React hooks to make sure that the site was responsive. It was now a pain to migrate to the new server components, as I had to make sure that the site was responsive.
This led to a whole lot of making new directories, layouts and components.
I actually did the whole thing before finally deciding that this site was not worth maintaining. This is what the final result would be if you don't take proper steps to write clean and maintainable code.

You might have different pain points, but these were the ones I had, which turn out to be quite common in Next.js sites. So, let's talk about how I solved these pain points.

How I Solved These Pain Points

The first pain point was the migration from page/ to app/. This was solved de facto by moving to Astro. Astro has a much cleaner syntax and a much cleaner way of doing things. It uses the concept of islands, which are basically components. These islands can be used to create layouts, pages, and components. This is a much cleaner way of doing things, as you don't have to worry about the messy scss. You can just use the built in classes to style your site. This was perfect for me, so I decided to move to Astro.

The second pain point was the messy scss. The main issue I faced was that I had written a lot of weird React Hooks to manage the responsiveness of the site. This was primarily because I was writing bad css and scss. So, for my new site, I decided to use Tailwind CSS. I had used tailwindcss in the past, but I didn't really enjoyed it.
However, I decided to give it another shot and I'm glad I did.
I now use Tailwind CSS as my de facto css framework.

The third pain point was the long repetitive code. This was solved by using the concept of islands in Astro. I now have a layout for my blog posts, a layout for my projects, and a layout for my home page. This has made my code much cleaner and much more maintainable. I also have a layout for my 404 page, which is a nice touch. Tailwind CSS also helps with this, as I don't have to write long repetitive css code. I can just use the built in classes to style my site.

The fourth pain point was the really slow build times. This was solved by using Astro. Astro has a really fast build time, which is a huge plus. It also has a really fast dev server, which is also a huge plus. This has made my development process much faster and much more enjoyable. I can now make changes to my site and see them instantly. The build time was very slow in Next.js, which was a huge pain.

The fifth pain point was the easily breakable code. Till now, Astro has been really good at this. I haven't had any issues with the code breaking. Tailwind CSS also helps with this, as I don't have to worry about the css breaking. All in all, Astro is an amazing framework and I'm glad I decided to use it.

Deploying The Switch

Now that I had decided to move to Astro, I had to figure out how to deploy it. I had initally planned on using netlify and I had even set up a netlify.toml file. But, I decided to go with Vercel, as I found it to be much easier to use.

I had to make a few changes to my site to make it work withVercel. Which was relatively very easy. All you had to do is the vercel's server adapter and you're good to go.
Which you can do by running the following command:

npx astro add vercel

It's that simple.
The vercel server adapter, however, only starts to work when you deploy your site. So, you'll have to deploy your site to see it in action. When you are developing locally, you can use the default server adapter, which is the node server adapter.

So, is Astro it?

Well, it is hard to say. I have been using Astro for a while now and I have to say, I'm really enjoying it. It had been far easier to maintain my site with Astro than it was with Next.js.
However, this is true for a site with very little dynamicity.
When I am working on a site with a lot of dynamicity, I will probably use Next.js. It is just easier to do so.

However, if you are working on a site with very little dynamicity, Astro is the way to go. It is much easier to maintain and it is much easier to deploy. Moreover, if you site has no dynamicity at all, you can also deploy it as a static site. This is a huge plus, as you don't have to worry about the server costs. You can just deploy it as a static site and you're good to go.

However, I can't say that Astro is the best framework out there.
The JavaScript ecosystem is huge and there are a lot of frameworks out there. It is continuously evolving and there are a lot of new frameworks coming out every day. So, my advice would be to try out a few frameworks and see which one works best for you. The thing however, is that with so many frameworks out there, it is quite easy to migrate from one to another. This is because most of the frameworks use the same concepts. So, if you know one framework, you can easily migrate to another.

My site now

So, this site you are currently reading is built using Astro.
I use a few islands to create the layout of the site.
Most importantly the Nav, Footer. I have still kept a few pages from the old site, like the updates and the url shortener.
These have the same database as the old site, so I didn't have to change anything. I just had to make sure that they are migrated to Astro and rewritten using tailwindcss. This site is till in beta, but Astro has helped me add a lot of new features to it.
I was also able to improve it's SEO and performance a lot.
Since most of the site is static, it is also very fast.

You might have also noticed the blog you are currently reading.
I am quite proud of it. I was able to incorporate suggested posts, tags, and a lot of other features. I was also able to improve the SEO of the blog a lot. So, you now get the title and image of the blog when you share it on social media. This is a huge plus, as it helps with the discoverability of the blog. One more thing that Astro helped me is with the code blocks. I was able to add syntax highlighting to the code blocks, which is a huge plus.
This makes the blog much more readable and much more enjoyable.

I am quite happy with the way the site has turned out. I'll write another post about the site when it is out of beta.

Conclusion

So, that's it for this post. I know this post was relatively weird, but I just wanted to share my experience with Astro. The main takeaway from this post is that you should try out a few frameworks and see which one works best for you. Also, I hope my method of identifying pain points and solving them was helpful.
If you have any questions, feel free to reach out to me on Twitter. Also, if you want to check out my site which I built using Astro, you can do so here.

Thanks for reading, Hope you have a great day!

Writing your own CLI in rust

Noob Science — Tue, 05 Sep 2023 12:56:09 +0000

The world of the terminal is beautiful and I always love dwelling in it. Away from the small old experience of the web which I am finding is quite “routine” now a days, you can build a little corner where you are in total control. So, here is a small tutorial on building your own small CLI in rust. I’ll be going through how to prettify the output, connections and more. So, let’s get started.

Disclaimer
This tutorial is by no means to a complete guide. This is just to show you the basic way you can approach making a CLI and how to sort of
go about making it. This article also presumes that you have a good enough knowledge of the rust language. If you don’t, I recommend you check out the official rust book. It is a very good resource for learning rust. You can find it here: https://doc.rust-lang.org/book/

What are we building?

So, what we are going to build is actually quite simple and you might actually find it useful. We are going to build a “Key-Value” Store. So, the concept is quite simple. We have 5 commands

Add a Key-Value [C]
Read a Value from Key [R]
Update a Value from Key [U]
Delete a Value from key [D]
Search a Key / Value [S]

The classic CRUDS App.
So, this is how we are going to do this.

Enter Rust

We will be using rust. Rust is a very simple to use memory and type safe language that is excellent for building cool and reliable CLI’s. In fact it has quickly become the number one tool for building CLI’s. I’ll dive into more on why rust CLI’s are good in a future blog post, so stay tuned for that.
So, with that, let’s get our project set up.
Install rust on your machine if you have not already. You can do so by visiting
Rust Programming Language
and just downloading rust for your platform.
Once that is done, just head over to your terminal. The terminal is an excellent way to interact with your computer. If you want to know the basics, there are plenty of resources on YouTube, below is one I recommend from freeCodeCamp.
With that, cd into a new directory and run

cargo init . --bin --name "nkv"

We are using --bin to signify to cargo that we will be using this as a binary rather than a crate, just to make it clear.
Now, open this up in your favourite text editor. I personally use VSCode (I know shocking), but rust has a very good ecosystem for Neovim and Emacs as well.
Now, create all the files shown below

Don’t worry, you don’t need to have SQLITE3 installed to have this work. We will be directly using rust a crate. Let’s just quickly go over the file structure:

We are going to have four files in the src, including the main.rs:

db.rs: Handles database connections and the CRUD operation’s brain
handler.rs: Essentially acts as a middle ware
utils.rs: For some cool utility functions like copying, hash generation etc.
.env file to store our Database address
main.db, which is our dev database.

Now that you have created these files, it is time to install some crates. Crates are independent rust libraries that provide additional features on top. They are built and maintained by the community and are very useful. It is very easy to install them in your rust project. Just head over to cargo.toml and under [dependencies], just paste the dependency name. I’ll be listing what we are using below, however, if you want to find more rust crates, I recommend you check out the GitHub List linked below.

rust-unofficial / awesome-rust

A curated list of Rust code and resources.

Awesome Rust

A curated list of Rust code and resources.

If you want to contribute, please read this.

View on GitHub

This is going to to our cargo.toml

[package]
name = "nkv"
version = "0.2.0"
edition = "2021"

[dependencies]
clap = {version = "4.3.23", features = ["derive"]}
sqlx = { version = "0.7", features = [ "runtime-tokio", "sqlite" ] }
serde = {version = "*", features=["derive"]}
bunt = "*"
inquire = "*"
tokio = {version="*", features=["full"]}
chrono = "*"
clipboard = { version = "0.5.0"}
rand = "*"
dotenv = {version="*"}
tabled ={ version="*"}
human-panic = "*"

We will be using quite a bit of dependencies to make a pretty CLI. The clap dependency is a very simple to use CLI argument parser that uses a predefined struct to manage. sqlx is a simple SQL binder, not ORM for rust that makes it very easy to execute SQL queries on our database.

serde is a popular JSON library, we use chrono for managing time related stuff. clipboard is an intuitive Clipboard abstraction for rust. rand is random generation, dotenv for reading from the .env file, tabled to style a struct into a table and tokio to make our function asynchronous, so execution will be blocked till our database is queried.
bunt and inquire are CLI formatting and prompting libraries that are similar to the rich library in python
Woah. So we are done with libraries. Be sure to also include the same features I did since we will be using all of them.

Let’s Start

Now, we can finally start the actual coding. Open [main.rs](http://main.rs) file and configure the clap::Parser to a nice struct.

use clap::Parser;
use dotenv::dotenv;
use human_panic::setup_panic;

#[derive(Parser, Debug)]
#[command(name="NoobKey", author="Ishan Joshi", version, about="A Simple Key Value Store", long_about = None)]

//? The Args struct is used to parse the command line arguments
struct Args {
    #[arg(required=false)]
    cmd: Option<String>,

    #[arg(short, long)]
    custom: Option<String>,

    #[arg(short, long)]
    docs: bool,
}

mod db;
mod utils;
mod handler;

The cmd is a Optional argument and we will prompt the user if it is not entered. We want this CLI for personal use, so no need to worry too much.
We also add all the files to the main file as mods.
Now, in the main function, all the predefined dotenv().ok(); setup_panic!()
These will take care of the environment and errors in production. Now let’s parse the arguments and check if the user enter the command. If he has not, let’s ask him

let args = Args::parse();

    let cmd:String;
    if args.cmd.is_some(){
        cmd = args.cmd.unwrap();
    }
    else{
        cmd = inquire::Text::new("Enter Command: ").with_help_message("Enter a valid command").with_autocomplete(
            &utils::suggester,
        ).prompt().unwrap();
    }

The utils::suggester is a simple filter and map function that inquire takes as input, this is the function. You can open up utils and paste this into it

pub fn suggester(val: &str) -> Result<Vec<String>, Box<dyn std::error::Error + Send + Sync>> {
    let suggestions = [
        "get",
        "set",
        "delete",
        "list",
        "search",
        "help",
        "exit",
    ];

    let val_lower = val.to_lowercase();

    Ok(suggestions
        .iter()
        .filter(|s| s.to_lowercase().contains(&val_lower))
        .map(|s| String::from(*s))
        .collect())
}

We basically just add a simple rust filter map function.

The Database

Let’s configure the database connection now. First, open up db.rs and import the necessary modules and define a struct of how our Database query would look like.

use sqlx::Connection;
use serde::{Deserialize, Serialize};
use sqlx::Row;

#[derive(sqlx::FromRow, Serialize, Deserialize)]
pub struct Entry {
    pub id: i32,
    pub key: String,
    pub value: String,
    pub hash: String,
    pub created_at: String,
}

We use serde to Serialize and DeSerialize data, that is derived from sqlx.
Our entry will have a simple id, key, value, hash and timestamp. Quite a simple and easy to work structure.
Now, let’s get the Database connection using sqlx and return a connection to the table I am calling entries

async fn get_db() -> sqlx::Result<sqlx::SqliteConnection> {
    //used to connect to the DATABASE_URL
    // It can be any valid SQLite connection string
    sqlx::SqliteConnection::connect(format!("sqlite:{}", std::env::var("KEY_STORE").unwrap()).as_str()).await
}

pub async fn connect() -> sqlx::SqliteConnection {
    //create table todo if not exists
    let mut conn = get_db().await.expect("Error connecting to db");
    let _ = sqlx::query(
        r#"
        CREATE TABLE IF NOT EXISTS entries (
            id INTEGER PRIMARY KEY AUTOINCREMENT,
            key TEXT NOT NULL,
            value TEXT NOT NULL,
            hash TEXT NOT NULL,
            created_at TEXT NOT NULL
        )
        "#,
    ).execute(&mut conn).await.unwrap();
    conn
}

As you can see, in the connect function, we just use a SQL query. No ORM needed.

Now, we can use the connect() function to just return a connection from the db, without having the need to reconnect everytime.

We now define an add_to_db function, the create part of CRUD

pub async fn add(key: String, value: String, hash: String){
    let mut conn = connect().await;
        let _ = sqlx::query(
        r#"
        INSERT INTO entries (key, value, hash, created_at)
        VALUES (?, ?, ?, ?)
        "#,
    )
        .bind(key)
        .bind(value)
        .bind(hash.clone())
        .bind(chrono::Local::now().to_rfc3339())
        .execute(&mut conn)
        .await.expect("Error adding entry");
}

As you can see, we just used simple SQL. sqlx allows us to bind the ? with a value, which is exactly what we did. We also used the chrono::Local time in a the ISO format as the timestamp.

We similarly write rest of the RUD:

//get from db
pub async fn get(key: String) -> sqlx::Result<Entry> {
    let mut conn = connect().await;
    let row = sqlx::query("SELECT * FROM entries WHERE key = ?")
        .bind(key)
        .fetch_one(&mut conn)
        .await?;
    Ok(Entry {
        id: row.get(0),
        key: row.get(1),
        value: row.get(2),
        hash: row.get(3),
        created_at: row.get(4),
    })
}

//delete from db
pub async fn delete(key: String) -> sqlx::Result<()> {
    let mut conn = connect().await;
    sqlx::query("DELETE FROM entries WHERE key = ?")
        .bind(key)
        .execute(&mut conn)
        .await?;
    Ok(())
}

//list all entries
pub async fn list() -> sqlx::Result<Vec<Entry>> {
    let mut conn = connect().await;
    let mut entries = vec![];
    let rows = sqlx::query("SELECT * FROM entries")
        .fetch_all(&mut conn)
        .await?;
    for row in rows {
        entries.push(Entry {
            id: row.get(0),
            key: row.get(1),
            value: row.get(2),
            hash: row.get(3),
            created_at: row.get(4),
        });
    }
    Ok(entries)
}

//list all keys
pub async fn list_keys() -> sqlx::Result<Vec<String>> {
    let mut conn = connect().await;
    let mut keys = vec![];
    let rows = sqlx::query("SELECT key FROM entries")
        .fetch_all(&mut conn)
        .await?;
    for row in rows {
        keys.push(row.get(0));
    }
    Ok(keys)
}

The Handler

We now define a middleware like file to handle all the operations and display appropriately to the user. For example, we can ask the user to add like this

pub async fn add(){
    bunt::println!("Executing add command...");
    let key = inquire::Text::new("Enter Key: ").with_help_message("Enter any identifier").prompt().unwrap();
    let value = inquire::Text::new("Enter Value: ").with_help_message("Enter any value").prompt().unwrap();
    let hash = super::utils::random_hash();
    super::db::add(key.clone(), value.clone(), hash).await;
    bunt::println!("Added entry: {$green}{}{/$}", key);
    bunt::println!("Value: {$yellow}{}{/$}", value);
}

Now that we have defined the add, we can just use a match statement in the main file to call the handler

match cmd.as_str() {
        "set" => handler::add().await,
        "list" => handler::list().await,
        "delete" => handler::delete().await,
        "get" => handler::get().await,
        "search" => handler::search().await,
        "exit" => {
            bunt::println!("{$red}Exiting...{/$}");
            std::process::exit(0);
        }
        "help" => todo!("Help command not implemented"),
        _ => todo!("Command not found")
    }

Like I defined the functions, similarly, you can define your own as well.

Now, this is where I leave you.

Conclusion

The whole point of this article was to help you take your first steps in writing your own CLI using rust and the whole point would be lost if I impose my own ideas on to you. You are now free to create your own CLI.
Format it as you like. Maybe using a little bit of rust magic, you can make a very pretty CLI.
You are free to use NoobKey for reference, or even build on top of it. It is quite simple and easy to use codebase. This is it’s GitHub repo:

newtoallofthis123 / noob_key

A Simple and efficient Key Value Store

NoobKey

A Simple Key Value Store written in rust

View on GitHub

With that being said, thank you for reading. You can check out this article on my site here. Thanks for reading, hope you have a great day.

🦀 Writing a CMS in Rust

Noob Science — Sat, 02 Sep 2023 16:58:54 +0000

Introduction

So, CMS is something that I always loved using. I just like the idea of having a service that controls your service.
I mean, it's like a service-ception. But, I always hated the idea of having a bloated admin panel.
I mean, why do you need a whole admin panel for a blog that you can manage with a CLI tool?
You can if you want to, but then why not just integrate it with your blog?
If you do that, you would need to do so many things:

Need to configure secure routes
Add authentication
Manage Access Control
Basically a lot of things

So, I thought why not just write a CLI tool that handles all the stuff for you and since I had been learning Rust, I thought why not write it in Rust?
Hence, welcome the Noob Handler.

Why Write in Rust?

I basically love CLI tools. They are a great and easy way to get things done.
They are also so easy to ship. No need to check some weird windows compatibility or anything.
Just compile it and ship it. That's it. And, Rust is a great language for writing CLI tools.
It's fast, it's easy to write, and it's easy to ship.
You get an executable on the spot. Moreover, you get an executable that is fast and small in size.
No weird dependencies folder that needs to be shipped or anything.

Why not python?

Python on the other hand, atleast in my opinion, has the worst executable experience.
To compile a python file, you might end up using a library like pyinstaller or something.
And, even then, you would end up with a huge executable. I mean, I don't want to ship a 100MB executable for a CLI tool.
That's just too much. Moreover, let's actually look at how pyinstaller works.

So, you see, pyinstaller works primarily by creating a virtual environment and then installing all the dependencies in that virtual environment.
Then, it creates all the necessary binding to all the dependencies in cpython. And, then it creates an executable.
This executable is just for name sake. It's just a wrapper around the virtual environment.
So, when you run the executable, it just runs the virtual environment and then runs the python file.
This slows down the execution time and also increases the size of the executable.
That in the long run is not good for a CLI tool. That brings us to an important conclusion.

Python is not meant to be compiled, it is meant to be interpreted.

Don't get me wrong, python has these cool libraries like rich that make writing CLI tools in python a breeze.
I have worked with rich in the past and it is one of my favorite libraries. However, it is best when
it is used as a script. I use rich to style my scripts and it works great. But, when it comes to writing a CLI tool,
I would rather use a language that is meant to be compiled.

Rust is awesome

So, welcome Rust.
Rust has a bunch of libraries that make writing CLI tools a breeze.
You have one for asking questions, one for styling, one for parsing arguments, and so on.
It is python with the speed of C and the safety of Haskell. What more can you ask for?
Actually scrap that. You can't really compare Rust with any other language.
It is it's own thing and it is great at what it does.

Here are some things that made me choose Rust:

Great Type System
Simple to use package manager
Well documented libraries for CLI tools
Single Binary executable
Fast execution
Good error handling
Great community

If you feel that I just listed the most generic reasons to use Rust, you are right.
I mean, there is a reason why so many CLI's are written in Rust.
Infact, there is a whole community for rewriting GNU/Linux tools in Rust.
So, for example, you might have heard of bat which is a replacement for cat.
It has some cool features like syntax highlighting and stuff like that, you have exa which is a replacement for ls with some cool features like icons.
You basically have a sub community in rust that is interested in command line tools.

What is a CMS?

Well, if you are unfamiliar with the term CMS, it stands for Content Management System.
So, the idea behind it is that instead of your content being changed at deployment time, it is changed at runtime.
Let me explain. So, let's say you use Astro to build your website. You have a bunch of markdown files that you use to generate your website.
Now, let's say you want to add a new blog post. You would have to add a new markdown file and then deploy your website.
This is the traditional way of doing things. However, with a CMS, you can just add a new blog post from the admin panel and it would be added to your website.
This is because your website will be reading off a remote database and not a local markdown file.

Now, my current blog is written using markdown files. I have a bunch of markdown files that are run through some remark plugins and then converted to HTML by Astro.
Nothing fancy. This is because my blog posts usually take a lot of time to write and I don't really write them that often.
Hence, when I do write them, I just add a markdown file and then deploy my website. It's not that big of a deal.

However, I have a page called quips where I write little updates about my life. I write them pretty often and I don't really want to deploy my website every time I write a new quip.
Hence, I use a MongoDB database to store my quips. I have a simple API that I use to add new quips and then I have a page that fetches all the quips from the database and displays them.
This is a CMS. I am managing my content from a remote database.

Along with quips, I have two other pages that use a database.

NoobPaste Mini: A simple and minimal pastebin that I use to share code snippets.
NoobShort: A simple URL shortener that I use to shorten URLs.

Both of these use a database to store their data. I have a simple API that I use to add new pastes and new URLs.
I actually have a NoobShort CLI that you can check out at newtoallofthis123/short_cli.
It is quite easy to use and it is one of the first CLI's that I wrote in Rust.
You can just install it using cargo install nshrt and then you can use it to shorten URLs.

Anyways, getting back to the topic at hand, I have a bunch of pages that use a database to store their data.
All this is stored using a MongoDB database. I use MongoDB because it is easy to use and it is easy to setup.
So, MongoDB has a very easy to use rust driver called mongodb. It is well documented and it is easy to use.
I have used it in the past and I have had no issues with it. So, I decided to use it for my CMS.

The Plan

The plan is simple. I want to write a CLI tool that can be used to manage my content.
I had to manage three types of content with the following B-JSON schemas:



// Quips
{
    "_id": "ObjectId",
    "hash": "String",
    "name": "String",
    "content": "String",
    "date": "String",
    "author": "String",
}

// Code
{
    "_id": "ObjectId",
    "hash": "String",
    "title": "String",
    "content": "String",
    "lang": "String",
    "author": "String",
}

// GO
{
    "_id": "ObjectId",
    "url": "String",
    "slug": "String",
}

So, the idea was that each of these had the following operations:

Show
Add
Edit
Delete The CRUD operations if you may. The CLI would have the following commands:



handler __doc__ __hash__

handler new __doc__

handler list __doc__

The doc can be either quips, code, or go. The __hash__ is the hash of the document that you want to edit or delete.

The list was basically show and search combined. It would show all the documents and then you could search for a specific document.
This could be done natively using the inquire crate, but more on that later.

Summary of the Plan

So, let's summarize the plan:

Connect to MongoDB
Enable CRUD operations on the database
Add a CLI interface to the CRUD operations
Add New, Edit, and Delete operations
Add a list operation
Make everything as native and as fast as possible
Make it look pretty

Let's take all these head on and see how we can implement them.

Solving the Problems

First up was actually setting up the database. I already had a database with the given collections.
I just used the URI of it and connected to it using the mongodb crate. It was pretty easy to do.
It looks something like this:



async fn get_connection() -> Result<Client, mongodb::error::Error> {
    let client = Client::with_uri_str(get_mongo_url()).await?;
    Ok(client)
}

The get_mongo_url function just returns the URI of the database. I have it stored in an environment variable so that
I don't have to hardcode it in the code. I just use the dotenv crate to load the environment variables.

Next up was actually enabling CRUD operations on the database. This was pretty easy to do as well.
So, the mongodb crate has a Collection struct that you can use to interact with the database.
I predefine the collections and then I just use the Collection struct to interact with the database.



#[derive(Clone, Debug, Deserialize, Serialize)]
pub struct Page{
    pub _id: ObjectId,
    pub hash: String,
    pub name: String,
    pub content: String,
    pub date: String,
    pub author: String,
}

The collections use serde to automatically serialize and deserialize the data.
Next, I can use the inbuilt functions of the collections struct like insert_one, find_one, find, delete_one, and update_one to interact with the database.



pub async fn get_page(hash: &str)->Page{
    let collection = get_page_conn().await;
    let result = collection.find_one(doc! {"hash": hash}, None).await.unwrap();
    result.expect("Page Not Found")
}

So, that is one of the functions that I use to get a page from the database. I just use the find_one function to find a page with the given hash.
The doc! macro is very handy and is used to create a BSON document. It is basically a JSON object that is used to query the database.
If you are starting to see a pattern here, you are right. The mongodb driver for rust is basically an ORM for MongoDB.
One thing that I forgot to mention, all this set up with tokio and futures so that the whole execution is blocked till the database operation is complete.
This makes it easier to avoid type errors and stuff like that.

CLI and Inquire

Next up was actually adding a CLI interface to the CRUD operations. This was pretty easy to do as well.
I use the clap crate to parse the arguments and then I use the inquire crate to ask questions.
The clap crate is pretty easy to use. You just define the arguments in a struct and then you use the clap macros to parse the arguments.
It looks something like this:



#[derive(Parser, Debug)]
#[command(name="Handler", author="Ishan Joshi", version, about="A Simple CLI to handle my site", long_about = None)]

// The Args struct is used to parse the command line arguments
struct Args {
    #[arg(required=true)]
    option: String,

    #[arg(required=false)]
    hash: Option<String>,
}

That's it! It is that simple. This comes with everything, the help command, the version command, and so on.
You can also add your own commands and subcommands. I just use the arg macro to define the arguments.
We get the arguments by calling the Args::parse() function. This returns a struct with the parsed arguments.

Although, I don't like to impose arguments on the user. I like to ask questions. So, I use the inquire crate to ask questions.
Inquire is a very useful crate that beautifully implements most of standard input and output operations.
So, you can ask for confirmation, ask for a password, ask for a list of options, and so on.
You can even have it open up a text editor for you to write your answer. It is a very useful crate.
So, I used it to write a custom function that takes in a instance of a struct and then asks questions based on the struct.
The quips struct looks something like this:



pub fn ask_page(page: &Page)->Page{
    println!("{}", format!("Editing {}", page.name).as_str());
    let name = inquire::Text::new("Name").with_default(page.name.as_str()).prompt().unwrap();
    let content = inquire::Editor::new("Content").with_file_extension(".md").with_editor_command(std::ffi::OsStr::new("vim")).with_predefined_text(page.content.as_str()).prompt().unwrap();
    let author = inquire::Text::new("Author").with_default(page.author.as_str()).prompt().unwrap();
    let date = inquire::Text::new("Date").with_default(page.date.as_str()).prompt().unwrap();

    let page = Page{
        _id: page._id.clone(),
        hash: page.hash.clone(),
        name,
        content,
        author,
        date,
    };
    page
}

This would give an output like this

Which is pretty cool. I mean, it is a CLI tool, but it looks like a GUI. I love it.

I also had to implement a search function. I used the inquire crate for that as well.
I just used the Select struct to ask for a list of options and passed in the list of pages.
The inquire's Select struct is very powerful. It automatically asks the user to start typing and then it filters the list based on the input.
So, I just passed in the list of pages and then I got a list of pages that I could select from.
Searching was solved.

Compiling

Next, all I had to do was integrate the whole thing together. I just had to call the functions in the right order.
It was quite easy and after a lot of debugging, I finally got it to work. Then, for the moment of truth, I ran cargo build --release and I got a single binary executable.
It was just 914kb in size! I now have a fully functional CMS that is just 914kb in size.

The best part? I don't need to carry or clone anything if I want to use it on another machine.
I only need to download the executable, which I will be hosting on my website, and then I can use it to manage my content.

Conclusion

So, what can you take away from this? Well, you can take away the fact that you can write a CMS in Rust.
Rust is an awesome tool for writing CLI tools. It is fast, it is easy to use, and it is easy to ship.
After the whole thing, I only ended up writing about 600 lines of code. That is just amazing.

This was also a good learning experience for me. I learnt a lot about Rust, tokio and futures.
Overall, it was quite cool. So, for conclusion, should you write a CMS in Rust? Well, if you want to, go ahead.
However, sometimes, just sticking with a good web framework is the best option. I mean, you don't need to write a CMS for everything.
In my case, I had a very small use case and I wanted to write a CLI tool. Hence, I wrote a CMS in Rust.
However, I probably do recommend you writing a CLI CMS, since that teaches you a lot about the language and it's libraries, plus it is fun.

Anyways, I hope you enjoyed this post. I know this was a bit more technical than my other posts, but I hope you enjoyed it.
If you want to check out the code, you can check it out at newtoallofthis123/noob_handler.
You can even clone it, but you can't really use it since it is configured to work with my database.
Thanks for reading and I will see you in the next one. Bye!

DEV Community: Noob Science

Understanding Concurrency

Some Background

Jargon Buster and Basics

Synchronous Programming

Concurrency

Classification of Concurrences

MultiThreading

MultiProcessing

Asynchronous

Blocking

Understanding through a Go Program

Where and Why to use Concurrency

Concurrency Patterns

Executive Summary

Conclusion

Grow into Programming

Git Checkout

Knowing and Learning

The Issues

However

How to grow into a language

The End

Chad-ifying My Web Tools

What I have built

NoobShort

NoobPaste

HTMLEr

NoobKey

Why I built these tools

Advice

Keep it simple stupid

Don't over engineer

Don't over optimize

It is for you

Don't be afraid to change

Some Observations

Conclusion

PS

Migrating My Site To Astro

Why I was Migrating My Site

Why I Chose Astro

The Pain Points

How I Solved These Pain Points

Deploying The Switch

So, is Astro it?

My site now

Conclusion

Writing your own CLI in rust

What are we building?

Enter Rust

rust-unofficial / awesome-rust

A curated list of Rust code and resources.

Awesome Rust

Table of contents

Let’s Start

The Database

The Handler

Conclusion

newtoallofthis123 / noob_key

A Simple and efficient Key Value Store

NoobKey

🦀 Writing a CMS in Rust

Introduction

Why Write in Rust?

Why not python?

Rust is awesome

What is a CMS?

The Plan

Summary of the Plan

Solving the Problems

CLI and Inquire

Compiling

Conclusion