DEV Community: Arnab Sen

Row, Row, Raft Your Nodes: A Guide to Consensus in Distributed Systems

Arnab Sen — Sun, 22 Oct 2023 18:05:23 +0000

Introduction

I have been recently diving into the world of Distributed systems and I came across a rather interesting paper: "In Search of an Understandable Consensus Algorithm". What caught my attention was the paper's primary objective of creating a consensus algorithm that is easy to understand. Typically, in the realm of new research, we tend to emphasize factors like efficiency, correctness, and conciseness while overlooking how accessible a concept is for others to grasp and apply in practical scenarios.

We often neglect how easy it for someone is to grasp the concept and use it to build practical stuff or solve real problems. Sometimes we do the opposite, if something works but is very hard to understand and is confusing we consider it to be very "clever" and end up rewarding complexity. That's why I appreciate the approach taken by the authors, Diego Ongaro and John Ousterhout, in prioritizing the algorithm's comprehensibility.

With many major corporations transitioning to distributed systems, having an algorithm that is both easy to understand and intuitive can empower developers to create systems more seamlessly and devise more efficient implementations."

So, what is Consensus

Let's start with a simple analogy:

You have been invited to a party and you have dressed up all nice and good. Then you ask your 4 friends if you are looking good (if you don't have so many friends just imagine you do, at this point, you must be good at it). Two of them say that you are looking good, and the rest say you are not. What are you supposed to do?

But if 3 of them said that you do look good (quite unusual right?) then you can go with the majority decision.

This is consensus !!

In distributed systems, we know that we have multiple nodes, and consensus refers to coming to an agreement.

Consensus simply means to get all the nodes in distributed computing to agree on a common value or decision, despite the presence of faults, delays, or unreliable communication. This agreement is crucial for ensuring that the distributed system functions correctly and consistently.

Even though it sounds simple, many broken systems have been built in the mistaken belief that this problem is easy to solve.

There is a pattern in most of the fault-tolerant systems. For example, in MapReduce, the computation is replicated but the whole thing is controlled by a single master. Similarly, we have GFS which replicates the data and has a single master to determine who the primary is for the piece of data. The benefit of having such a single master system is that there will be no disagreement. But at the same time, it makes it a single point of failure. So we need some sort of consensus to avoid such failures.

💡

It's fine if you don't have an understanding of MapReduce or GFS, they were just examples. You can read my blog on MapReduce

https://arnabsen.dev/understanding-mapreduce

History of Consensus Protocols

Well, now that I think "history" is an overstatement cause there is just one very popular protocol called "Paxos" which has been the gold standard for a long long time. First submitted in 1989 by the Leslie Lamport , the Paxos protocol is named after a fictional legislative consensus system on the island of Paxos in Greece.

FYI Leslie Lamport was the creator of LaTeX. He also won the Turing Award which is like the Nobel Prize for Computer Science.

https://youtu.be/rkZzg7Vowao

The problem with Paxos was it was really hard to wrap our minds around. It only solved a portion of the problem and hence made it difficult to build systems around Paxos.

Raft Algorithm

As mentioned before the authors' primary design goal when building Raft was "understandability".

🤓

Even the author of Raft found it difficult to understand Paxos and he once said that it took him building a new consensus algorithm to really understand how Paxos worked and its correctness.

Now this doesn't mean that Raft is very very easy to understand, there are still a lot of intricacies and edge cases to cover, but I will try my best to give a brief overview of the entire thing. So let's start.

What are we dealing with?

We have a client, which can be a user or a service etc. And it tries to fetch some data from the server we have which is nothing but a cluster of nodes.

Now, to ensure consensus we need the nodes to agree on a particular value and then send that to the client.

How do we ensure that all the nodes have the same value? Raft achieves that through something called a "Replicated Log".

It ensures that all the commands (or operations) that need to be performed are replicated across the nodes in the same order. So, let's say you have a variable X initially, the value is 0 and then you perform some operations like:

add 5
mul 10
sub 15

If these commands were logged in the same order then the final value of X will be the same for all the nodes assuming the fact that the operations in the nodes (which are also referred to as state machines) are deterministic.

So, when a new command gets proposed, it first gets logged on all the nodes (at least most of the nodes) at the end of their logs. Only when the log gets properly replicated, does the node perform its computation and send the result to the original client.

The idea behind this is if the logs get replicated properly the cluster can continue functioning as long as the majority of the nodes are up (Yes, it's more like democracy, the majority wins).

Who will instruct the nodes on what to do?

Again similar to democracy, here also we have a leader node elected by other nodes. The leader node is responsible for talking to the client, giving commands to the other nodes, etc.

Before diving deep into the functioning of the leader let's see how the nodes become a leader.

Server States

The nodes/server have 3 states:

Follower
Candidate
Leader

In normal service, there will be just one Leader and all the other servers will be Followers.

Followers are passive, they issue no requests and can only respond to requests from Leaders and Candidates.

The Leader handles all the client requests. It also sends regular heartbeats (empty requests) to the Followers to maintain authority. More like it's saying "Don't get too excited, I am still your leader".

If a Follower receives no communication over a time period (called election timeout ) it assumes that no Leader is available and starts the process of Leader Election.

Leader Election

Let's talk about term before diving into Leader Election. To have consensus we need to have a mechanism to detect obsolete information. Like if a particular server was a leader and then it stopped responding, and a new leader was appointed. We need to ensure that the other servers don't take instructions from the old leader anymore. Raft achieves this by dividing the time into terms or arbitrary lengths called term.

Each term starts with an election of the leader and then if the election is successful the leader rules for some period of time which is represented in green in the above diagram.

You can see how this is similar to the administrative process of a country where each leader has a term like 5 years and then post that election happens to elect a new leader.

In Raft, now the interesting thing is there is no global concept of term among the servers. Each server starts with its own terms.

When the election process starts the follower increments its current term index and becomes a candidate. It votes itself and then sends specific requests to the other nodes requesting to be a leader in other words asking for their vote. This happens by a special kind of request called RequestVote RPCs.

Whenever the server communicates with the other it includes what it thinks the current term index is. Then the other server responds by including its term index.

Now if there is a mismatch the server with the lower-term starts having an identity crisis and falls back to being a follower. And the other server just ignores it.

A Server is entitled to vote only one server and that happens on a first come first serve basis. So by the end of the process, the Candidate server with the most votes becomes the leader and then sends heartbeat messages to all of the other servers to establish its authority and prevent new elections.

Here I had a doubt, what if all the server realizes that the leader no longer exists or the current term is over, votes themselves, and requests a vote at the same time? They will just end up waiting indefinitely to get majority votes. Or what if two candidates receive the same number of votes (this is called a split vote)?

Raft solves this by using randomized election timeouts, which means election timeouts are chosen randomly from a fixed interval (e.g., 150-300 ms). This way in most cases only one server will timeout at a time, become a candidate, and will request a vote before the other server times out. Even in case of a split vote, each candidate restarts its randomized election timeout at the start of an election and it waits for that timeout to elapse before starting the next election. This randomness drastically reduces the chances of another split vote.

How does the normal operation happen?

Let's go through how a normal operation will take place with consensus in Raft.

First, the client sends a command to the leader. Wait, how will the client know who the leader is? Well, the client can send it to any server and if the server is a follower it will simply redirect it to the leader.

The Leader appends the command to its log and sends another request (called AppendEntries RPC request to all its followers). The followers on receiving this request just append the new log to their existing logs record. Note, at this point they just append it and don't actually perform any computation. It's the leader who decides when it is safe to apply the log entry i.e. when the entry has to be committed. This is actually called 2PC (Two Phase Commit).

The append request contains two identifiers, one is the term index which I already mentioned before, and another is the log index which is the position of the new entry in the log. The leaders use these two index values to determine if the majority of the servers have the logs up to date and then issue a commit.

In any case, the logs become inconsistent, the leader forces the followers to duplicate the leader's log. To bring a followers log into consistency with its own, the leader finds the latest log entry where the two logs agree, deletes any entries in the followers log after that point, and sends the follower all of the leaders entries after that point.

This is how the Leader ensures that every server (or at least the majority) has the same order of the logs i.e. consensus is reached, performs the computation, and then responds to the client.

This is how we can build a system with high data consistency with Raft and many distributed consistent database like CockroachDB uses Raft.

This was a very brief overview of Raft. There are many more aspects to this that couldn't be covered in a blog and I urge you to go through the original paper. I hope you found my blog useful. If you have any feedback share it in the comments. You can sign up for the Hashnode newsletter to get notified every time I post a blog. Learn more about me at arnabsen.dev/about. Have a nice day.

Understanding MapReduce

Arnab Sen — Sat, 30 Sep 2023 07:25:48 +0000

MapReduce is a programming paradigm that helps us perform large-scale computation across computing clusters.

DISCLAIMER: In this blog, I will take a very simple overview of this topic. If this blog actually spikes your interest and you want to delve into the technical nitty-gritty then I would urge you to read the original paper: "MapReduce: Simplified Data Processing on Large Clusters" (It was written by the Top-Gs in the programming world: Jeff Dean and Sanjay).

Background

Let's start with a little bit of background.

Currently at Google, I am working on a project that requires me to process a large amount of logs, perform some computations, and then write the computed results in a database. This has to happen on a daily basis and the output is then consumed by downstream teams for further operation.

One might think, what's the big deal in that? Just take the log file as input, implement the logic, and write the results to an output file. That's what we have always been doing.

Well, one thing we should take into consideration is the size of the file that we are dealing with. It is actually in the multiple of Terabytes. Taking our traditional path for problems like this would cause a lot of issues like:

Performance Issue: the entire thing would probably take a whole lot of time and it can have a lot of potential bottlenecks.
Memory Issue: How are we even going to load the entire TBs of data into memory? We have to implement some partitioning ourselves to do it effectively.
Scalability Issue: Very difficult to actually scale it to multiple machines.
Fault Tolerance: Let's say you somehow managed to overcome all the problems above, what if at the last moment, something crashes, how are we going to recover?

This is why Google came up with the MapReduce paradigm. There is also an open-source implementation of the same called "Apache Hadoop".

Link: https://github.com/apache/hadoop

What's MapReduce?

You must have already noticed that the name MapReduce is made up of two parts: Map and Reduce.

You might have often used these functions in Python, JavaScript, etc. Something like this:

Map Function

The Map function applies a given function to each element in a collection (such as a list or array) and produces a new collection of transformed values. It is often used to perform element-wise operations on data.

Reduce Function

The Reduce function takes a collection and combines its elements into a single result by repeatedly applying a binary operation. It reduces a collection to a single value.

Here is a visualization to help you understand it better:

These two methods are fundamental to the Functional Programming Paradigm. I came across these when I started learning functional programming languages like OCaml, Lisp, and Rescript. The main idea is to process the data in a functional, declarative manner.

Inside the map function, we provide the transformational logic and in the reduce function we provide the aggregator logic, and no need to specify any logic for the looping, etc.

A large number of problems can actually be split into two broad logic: Map phase and Reduce phase. Some examples mentioned in the paper are:

Distributed Grep
Counting URL access frequency
Reverse Web-Link Graph
Term-Vector per host

Let me give a simple example, let's say you are working on a Music Recommendation Algorithm. Before you can actually use the data for the recommendation algorithm you will have to do some pre-processing. Usually, logs contain everything, and we may not need all of them, so we have to modify the logs to a form that is relevant to us. To further simplify things, let's say we just need the Genre of the music from the log data. We can use a map function for that.

Then let's say our recommendation algorithm also needs the number of music the user has listened to for each genre. A reduce function can do this. Here is a sample Python code to help you understand better:

from functools import reduce

logs = [
  {
    'name': 'Shape of You',
    'singer': 'Ed Sheeran',
    'genre': 'pop'
  },
  {
    "name": "Uptown Funk",
    "singer": "Mark Ronson ft. Bruno Mars",
    "genre": "pop"
  },
  {
    "name": "Bohemian Rhapsody",
    "singer": "Queen",
    "genre": "rock"
  },
  {
    "name": "Friends in Low Places",
    "singer": "Garth Brooks",
    "genre": "country"
  }
]

def update_frequency_count(freq_dict, num):
    if num in freq_dict:
        freq_dict[num] += 1
    else:
        freq_dict[num] = 1
    return freq_dict

# Map operation
# {name, singer, genre} -> genre
genres = list(map(lambda log: log['genre'], logs))
# genres: ['pop', 'pop', 'rock', 'country']

# Reduce operation
frequency_count = reduce(update_frequency_count, genres, {})

print(frequency_count)
# {'pop': 2, 'rock': 1, 'country': 1}

Both of these functions are deterministic. Also, the map operation on one element doesn't depend on the result of another operation. In other words, they are independent of each other as well.

So we can parallelize the operations of Map and Reduce. We can further extend the same thought process by breaking the input data into several chunks of data and having one map function for each chunk of data. This can also solve our Fault Tolerance issue , if something breaks on operation, it won't affect the others cause now they are all running independently in a parallel fashion. We can just re-run the broken process with the small chunk of data. You can see the big picture now as to why the Map and Reduce functions were considered specifically to solve this kind of problem.

To quote the paper:

"Our use of functional model with user specified map and reduce operation allows us to paralelize large computations easily and to use re-execution as the primary mechanism for fault tolerance."

Let's dive into the programming model deeper and then we will take another very popular example to understand the programming model.

With MapReduce, we usually deal with key/value pairs cause it's pretty generic. The Map takes an input pair and produces a set of intermediate key/value pairs. The MapReduce library groups all the intermediate values associated with the intermediate key I and passes them to the Reduce function.

The Reduce function then accepts the intermediate key I and a set of values for that key and then merge together values to form a possibly smaller set of values (it can be one as well).

Stages of MapReduce

Now let's dive into the actual working of MapReduce.

Splitting the Input

The input data is actually partitioned into smaller chunks, typically 16 MB or 64 MB. Let's say the number of chunks/splits is M. Then there will be M map operations in total.

Why 64MB? That can be a whole separate topic of discussion but you can think of it like these large amounts of data are usually stored on local disks of the machine that make up the cluster. Google File System (GFS) divides each file into 64MB blocks, so having the input data of the same size will ensure that the map function just has to read data from one block thus reducing I/O latency.

The Reduce invocations are also distributed by partitioning the intermediate key space into R pieces using a partitioning function (we will discuss this later in this blog). The number R is also set by the user.

Building the army

When we start the MapReduce function, it first starts a special process called "master". The master as you might have already guessed does a lot of management-related work like assigning Map and Reduce operations, handling exceptions, etc.

The program also starts up many copies of the program on the other machines. These are done by the fork system call. All the other programs except the master are called "worker nodes" or simply "workers". Each of the workers will be assigned a task from the M map tasks and R reduce tasks which is determined by the master.

Map Stage

A worker who is assigned a map task reads the contents of the corresponding input split. It parses key/value pairs out of the input data and passes each pair to the user-defined Map function. The intermediate key/value pairs produced by the Map function are buffered in memory. Since the map operation doesn't depend on any other worker, all the map workers can parallelly perform this, thus allowing us to linearly scale the performance of the task of extracting data.

Before the workers with Reduce task assigned can work, these buffered pairs are written to the local disk which is partitioned into R partitions by the partitioning function. Once the partitioning is completed, the master is informed, who then signals the location to the Reduce workers to get the rest of the job done.

Reduce Phase

The Reduce workers then read the intermediate data by using RPC calls. Once it has read all the intermediate data, it performs a sort based on the keys. Why?

So that all the occurrences of the same key are grouped together. The sorting is needed because typically many different keys map to the same reduce task.

Let me be frank, it took me some time to wrap my head around this, and I am still not sure if I got it correctly. So if you think my understanding is wrong please don't hesitate to correct me.

Previously we saw that we have M map operations and R reduce operations. Sometimes the same worker is assigned multiple Reduce jobs. Sorting the data makes it easier for the worker to distinguish between the new intermediate data and the old intermediate data. Once the worker finds that the partition has some new data it starts the reduce job again.

In the reduce step it also performs a merge of all the values corresponding to a particular key. So, sorting the keys makes it easier to merge. This will get clear once we go through the steps again taking the example of a word counter.

Finally, the reduce worker goes through the sorted intermediate data and for each unique intermediate key, it passes the key and the set of intermediate values to the user's Reduce function. The output is then appended to the final output file.

Last stage

When all the Reduce workers have completed execution, the master passes control back to the user program. The output of MapReduce is stored in the R output files that the R reduce workers created.

Partitioning Function

Let's understand what the partitioning function is all about. As a user, you can define the number of output files you want your data to split between. That number is denoted as R and MapReduce will assign the same number of Reduce workers. So if R = 10, it means that MapReduce will partition the data from the map workers into 10 splits. Each split will be assigned to one Reduce worker which means that there will be 10 Reduce workers. And then each Reduce worker will write the output to a file so the user will have their processed data split into 10 files.

By default, the partitioning algorithm that is used is hash(key) mod R. This usually gives a fairly balanced partition. In some situations, you might want to use a custom function as well. Taking the example from the paper, if your keys are a bunch of URLs and you want them to be partitioned based on the hostname, then you can use a function that extracts the hostname and then applies the hash and mod. So something like: hash(getHostName(key)) mod R.

Example

Let's take the example of the Character Count. We have an input text file with a bunch of words and we have to calculate the frequency of all the characters appearing in the input.

So, first, we have the input data being split into M partitions. In our case M = 5. So the master worker will assign 5 map workers to map each of these input partitions using the user-defined map function which can be something like this.

map(String key, String value):
    // key: document name
    // value: document content
    for each char c in value:
        EmitIntermediate(c, "1");

So, for the text "apple" our map function will return:

a: 1
p: 2
l: 1
e: 1

Now, the intermediate output will be "shuffled" by using our partitioning algorithm. Since we are going to consider only the lowercase English alphabet we can have R = 26 for our partitioning algorithm. That way each output will contain the count of one letter. Simply put, our partitioning function will group all the a's and then pass it to our reducer function.

So, in our case, the reducer function will get the following input:

a: [1, 3, 1, 1]

Since we want the total count, we can have a reducer function like this:

reduce(String key, Iterator values):
    // key: a word
    // values: a list of counts
    int result = 0;
    for each v in values:
        result += v;
        Emit(AsString(result));

This will give us the final result: a: 6.

So this was a very basic overview of the MapReduce workflow. Keep in mind, that this is not a very generic parallel programming paradigm but only applies to problems that fit into these map-reduce paradigms. But from my work experience, I have seen that a lot of the data analysis work does actually fall into this category. Otherwise, why would Google come up with this in the first place xD.

There are many more technical concepts of MapReduce that I didn't cover in this blog. Do go through the research paper for an even more in-depth analysis. In case you have any doubts, feel free to ping me or put them in the comments.

Hope you liked my blog, follow me for more such content. You can learn more about me at arnabsen.dev/about. You can find my social links at arnabsen.dev/links. Have a nice day.

A beginner-friendly introduction to Docker and Containers

Arnab Sen — Sun, 28 May 2023 05:54:21 +0000

What is a Container?

You might be familiar with these containers.

These are used for storing stuff and shipping from one place to another. In DevOps, containers have similar applications as well.

A Container is a way to package an application with all the necessary dependencies and configuration.

And that package can be portable easily, which makes the process of development and the job of a developer easier.

What happens in these containers? Nothing much, just a couple of processes run in isolation on a shared kernel.

How is it isolated? The isolation of containers is provided by a Linux feature called namespaces. Namespaces partitions kernel resources such that one set of processes sees one set of resources while another set of processes sees a different set of resources. They give the group of running processes an isolated view of the kernel. For example,

Namespaces	Description
`PID`	process IDs
`USER`	user and group IDs
`UTS`	hostname and domain name
`NS`	mount points
`NET`	Network devices, stacks, ports

You can use the command lsns to get lists of all the currently accessible namespaces.

$ lsns
        NS TYPE NPROCS PID USER COMMAND
4026531835 cgroup 85 1571 seth /usr/lib/systemd/systemd --user
4026531836 pid 85 1571 seth /usr/lib/systemd/systemd --user
4026531837 user 80 1571 seth /usr/lib/systemd/systemd --user
4026532601 user 1 6266 seth /usr/lib64/firefox/firefox [...]
4026532928 net 1 7164 seth /usr/lib64/firefox/firefox [...]

Another feature of the kernel is control groups (a.k.a cgroups) which monitors, limits, accounts for, and isolates the resource usage of a collection of processes (also known as containers)

Virtual Machine VS Containers

An operating system has two main layers:

OS Kernel: It communicates with hardware like memory, CPU .etc.
Applications Layer: They run on the Kernel.

Linux is a kernel. There are many Linux distributions each looking different from one another because the applications are different. But under the hood, they use the same kernel i.e. Linux.

Coming back to the difference between VM and Containers:-

VM runs on something called Hypervisor. Each virtual machine includes a full-blown OS and its process, which are very heavy and slow to start.

Containers on the other hand don't include full-blown OS, only include a set of OS-specific files. They are just processes that share the same kernel with other containers and the isolation by the containers is provided by the Linux namespaces. So since they run on top of the kernel they are very fast and also light-weight. So, we are getting the benefits of isolation by VM without the heaviness that comes with VM

But remember:

Containers don't replace VMs. Both have their own purpose.

What is Docker?

An early implementation of container technology was added to FreeBSD in 2001. Whereas Docker debuted to the public in Santa Clara at PyCon in 2013 and was made open source in March 2013. The tooling for using Linux Containers was really lacking and that's where docker comes into play. Basically, docker is tooling to manage containers.

Docker allows developers to package their applications into containers and directly use them in their CI/CD pipeline.

It helps achieve:

Build once, run everywhere

Why should we at all bother?

There are a number of advantages of using Docker and Containerisation: (No wonder it is very popular)

You can't say "But... It works on my machine" anymore. Good Luck with that. Because we are packaging the application with all the dependencies and the configurations required, even if it is running on a different machine it will still behave the same way.

They are very lightweight and fast. Already discussed this, just adding one thing. The fact that dockers are lightweight is the main reason behind them being so portable. Obvious
Docker has its own ecosystem provided by the community and has many tools that come with it, which helps solve a lot of issues.

Let's get our hands dirty now.

To install docker you can look into the official docs. Else if you want to just try your hands first, then you can use this: https://labs.play-with-docker.com/

Practical 1

Let's go through some popular docker commands. The docker commands are the same for all environments so it won't be an issue.

Run a container

Run this command

docker container run -t ubuntu ls

docker container run will run the image that your provided, in this case, it is Ubuntu. Now if you are running this command for the first time high chances are that the image is not downloaded, so it will show something like this:

Unable to find image 'ubuntu:latest' locally
latest: Pulling from library/ubuntu

If you notice something here we just mentioned ubuntu not ubuntu:latest. If we didn't specify any version it will take the latest version. But if you wanted to use ubuntu 18.04 you can specify it like this:

docker container run -t ubuntu:18.04 ls

It will again show:

Unable to find image 'ubuntu:18.04' locally
latest: Pulling from library/ubuntu

But this time notice the version. Also in both cases you see an output like this

bin dev home lib64 mnt proc run srv tmp var
boot etc lib media opt root sbin sys usr

Basically, it is the output of the ls command.

Now let's run this command

docker container run -t ubuntu top

And open a new shell and continue. If you are using Play with Docker. Then create a new instance, and then ssh into the previous instance.

Now type this command

$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
6d2a990df65d ubuntu "top" 3 seconds ago Up 2 seconds blissful_austin

This command shows you the containers and all the necessary information about it.

Usually, containers are used for running a process, maybe a server or some application, and soon as the main process exits the container will stop too. So let's create a very simple process.

So run this command

docker run -d ubuntu sleep 300

Yeah, this is our process, we will ask the container to sleep for 300 secs.

Every container has a unique container id. So in most of the commands which deal with a particular container, we have to provide the container id. Now let's hop into the sleeping container.

docker exec -it 6d2a990df65d bash

Note: 6d2a990df65d is the docker container id in my case. Just do docker ps look at the container id and paste it there.

It will spawn a bash terminal inside the container. You can run all the basic commands of Ubuntu.

To get the list of all the containers (even the ones that exited) run

docker ps -a

Stop a container

To stop the container we need to use docker stop <container-id-1> <container-id-2>.

docker stop 6d2a990df65d d9da0526d987

To remove containers you can use docker system prune

There is a nice collection of important docker commands by garystafford here which I find very helpful.

Debugging container

Usually, for debugging, we need to look at the logs. For that use the command:

docker logs <container-id>

Also, docker exec sometimes helps in the debugging process.

Remember that containers use kernel-level features to achieve isolation and that containers run on top of the kernel. Your container is just a group of processes running in isolation on the same host, and you can use the command docker exec to enter that isolation with the bash process. After you run the command docker exec, the group of processes running in isolation (in other words, the container) includes sleep and bash.

Where are all these images stored?

Docker maintains a public repository of all the images called Dockerhub.

Also, you can run more than one container simultaneously.

While running containers you can do a lot more stuff, like

--name tag will allow you to name the container.
--detach or -d will allow you to run the docker in detached mode i.e. in the background.
--publish or -p to publish the ports to the host (very important)

You can the list of all the options here

Containers are self-contained and isolated, which means you can avoid potential conflicts between containers with different systems or runtime dependencies. You can run multiple NGINX containers that all have port 80 as their default listening ports. If you're exposing the host by using the --publish flag, the ports selected for the host must be unique. Isolation benefits are possible because of Linux namespaces. Running multiple containers on the same host gives us the ability to use the resources (CPU, memory, and so on) available on a single host. This can result in huge cost savings for an enterprise.

Although running images directly from the Docker Store can be useful at times, it is more useful to create custom images and refer to official images as the starting point for these images.

Docker Images and Docker Containers

Now, if you are confused let's look into the differences between Docker Images and Docker containers. A docker image is a tar file or an archive of the filesystem or the container. It contains the metadata of the filesystem.

Whereas Container is the running environment of the image. It's the process (isolated process to be specific). The filesystem of the container is virtual, i.e. it has its own abstraction.

Images are used to create containers (more than one). You can consider it as a blueprint. We can share our Docker Image and then we can create containers using the image. We can also push images to the Docker Hub.

How to create Docker Image?

To create a Docker image, we use a special file called Dockerfile (no extension), which consists of a list of commands to build our image. After we have created our image we can pass it to docker build which will build the image.

docker build -f Dockerfile

A Docker Image is a set of layers where each layer represents an instruction from the Dockerfile. The layers are stacked on top of each other. Each new layer is only a set of differences from the previous one.

The best part of the image layer is that they are cached. If you change say the 5th line then the docker engine will reuse the first 4 layers and then start building from the 5th line. This improves the time during build and also in the context of CI/CD once the base is pushed, the subsequent pushes will be very fast. To optimize the caching, we need to organize the Dockerfile in a way that the line that will change the most is located at the bottom of the Dockerfile.

Let's dive into this.

Practical 2

Creating flask application

We are going to create a simple Flask application.

Download this Python script https://gist.github.com/arnabsen1729/1fa19228e4451963bbb64563da98f880 and save it as app.py.
Install the package flask by pip3 install flask and run the app python3 app.py.
When the app is running you can visit your 0.0.0.0:5000, you will see hello world!.

Hence we have set up a basic Flask server. To close this process press Ctrl+C.

Now we will dockerize this flask app.

Building Dockerfile

So create a file Dockerfile and open your text editor and paste this

FROM python:3.6.1-alpine
RUN pip install flask
CMD ["python","app.py"]
COPY app.py /app.py

Let's go through this line by line

`FROM python:3.6.1-alpine`

This is the starting point for your Dockerfile. Every Dockerfile typically starts with a FROM line that is the starting image to build your layers on top of. In this case, you are selecting the python:3.6.1-alpine base layer because it already has the version of Python and pip that you need to run your application. The Alpine version means that it uses the Alpine distribution, which is significantly smaller than an alternative flavour of Linux. A smaller image means it will download (deploy) much faster, and it is also more secure because it has a smaller attack surface.

It is highly recommended to only use official images found in the Docker Hub, or noncommunity images found in the Docker Store.

`RUN pip install flask`

The RUN the command executes commands needed to set up your image for your application, such as installing packages, editing files, or changing file permissions. In this case, you are installing Flask. The RUN commands are executed at build time and are added to the layers of your image. Usually for node applications, this will involve installing all the node_modules, if it was a larger Python application, you will have to install those as well.

CMD ["python", "app.py"]

CMD is the command that is executed when you start a container. Here, you are using CMD to run your Python application. There can be only one CMD per Dockerfile.

If you specify more than one CMD, then the last CMD will take effect.

`COPY app.py /app.py`

This line copies the app.py file in the local directory (where you will run the docker image build) into a new layer of the image. This instruction is the last line in the Dockerfile. Layers that change frequently, such as copying source code into the image, should be placed near the bottom of the file to take full advantage of the Docker layer cache. This allows you to avoid rebuilding layers that could otherwise be cached. For instance, if there was a change in the FROM instruction, it will invalidate the cache for all subsequent layers of this image. You'll see this a little later in this lab

But, how can the CMD command run if we are copying the app.py later? CMD is the command that is executed when you start a container , until and unless you run the container it is not executed. Also since the command to run the application will not change, we have placed it higher.

Here is the list of all commands

Now to build the image with the Dockerfile run

Running the container

$ docker build -t flask-app . # if the Dockefile is in that directory
#or
$ docker build -t flask-app -f /path/to/Dockerfile

It will pull the Python:3.6.1-alpine, and go through the process. In the end it will output something like this

Successfully built 38f35dd1d2a4
Successfully tagged flask-app:latest

Now run the command docker images and you will see flask-app or whatever name you mentioned in the list. So now you have successfully created the image.

For the final part let's run the container with this image.

docker run -p5001:5000 -d --name flask-container flask-app

What are we doing here?

docker run to run the container
-p5001:5000 to map port 5000 of our container with 5001 of the host machine
--name flask-container giving the name of my container
flask-app the image we want to build

Now do docker ps you will see

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
940f215ceea0 flask-app "python app.py" 2 seconds ago Up 2 seconds 0.0.0.0:5000->5001/tcp flask-container

Now go to localhost:5001, if everything was correct you will see hello world!

To check the logs we can run docker logs

$ docker logs flask-container
 * Serving Flask app "app" (lazy loading)
 * Environment: production
   WARNING: This is a development server. Do not use it in a production deployment.
   Use a production WSGI server instead.
 * Debug mode: off
 * Running on http://0.0.0.0:5000/ (Press CTRL+C to quit)
172.17.0.1 - - [03/Jan/2021 14:55:58] "GET / HTTP/1.1" 200 -
172.17.0.1 - - [03/Jan/2021 14:55:59] "GET / HTTP/1.1" 200 -

Docker images contain all the dependencies that they need to run an application within the image. This is useful because you no longer need to worry about environment drift (version differences) when you rely on dependencies that are installed on every environment you deploy to. You also don't need to follow more steps to provide these environments. Just one step: install docker, and that's it.

Now if you change the app.py only the last step in Dockerfile needs to be updated the rest is already cached.

And you have successfully dockerized your application.

Docker Compose

Sometimes the docker run command becomes very long and very tedious like sometimes we even need to specify lots of environment variables and stuff. Running those long commands every single time becomes difficult. Also if you are working with let's say 3 or 4 containers then you have to write such long-run commands every single time you want to start the container. So there is a simple way to express the commands in a structured way, save it in a file and simply run that file.

That file is the docker-compose.yml. The file is a YAML file. (Fun fact: full form of YAML is 'YAML Ain't Markup Language' recursive huh!!)

Writing a docker-compose file is not very difficult, you just need to know how to structure it. This article written by Gabriel Tanner explains it nicely.

Link to the article: gabrieltanner.org/blog/docker-compose

Conclusion

In this article, we learned about Docker and how to dockerize a simple application. We also learned about Docker Compose and how to use it.

A really nice collection of Docker study material is available here: Docker Handbook 2021 Edition

Hope you liked my article, do follow me on Dev.to and on Twitter (handle: @arnabsen1729) for updates.

Quick HTML tips: Enhancing UX and Accessibility with `enterkeyhint`.

Arnab Sen — Thu, 25 May 2023 12:25:27 +0000

In the world of web development, creating a delightful user experience (UX) goes hand in hand with ensuring accessibility for all users. Accessibility , often referred to as " a11y" (short for "accessibility" and the 11 letters between the "a" and "y"), focuses on making web content usable by individuals with disabilities. While major accessibility considerations involve things like proper semantic structure and alternative text for images, even small details can significantly impact the overall user experience. One such detail is the enterkeyhint attribute in HTML.

While using various apps on your smartphones, you might have noticed that the button where we usually have "enter", sometimes gets replaced with terms like "search", "next", etc. This small thing might be insignificant but still gives a lot of clarity about what's going to happen next. Ever wondered how that is implemented?

It's through the enterkeyhint attribute in HTML.

The enterkeyhint attribute provides a means for developers to communicate the expected action to the browser when the user presses the Enter key within a form. By utilizing this attribute effectively, developers can enhance both the UX and accessibility of web forms. By guiding the browser to understand the intended action, users can navigate forms more easily, without relying solely on mouse or touch input.

So, let's dive in and explore how this seemingly minor attribute can contribute to a more inclusive and user-friendly web!

What is the `enterkeyhint` attribute?

The enterkeyhint attribute is an HTML attribute that can be added to form input elements such as <input> and <textarea>. It is used to suggest to the browser the type of action that should be taken when the user presses the Enter key while the input element has focus.

How does `enterkeyhint` work?

The enterkeyhint attribute accepts a few predefined values that represent different actions. Here are the possible values:

enter: Indicates that the default action for the Enter key should be performed. This is the default value if the attribute is not specified.
done: Suggests that pressing Enter should submit the form or perform the action that signifies completion.
go: Suggests that pressing Enter should initiate a "go" operation, such as navigating to a URL or starting a search.
next: Indicates that pressing Enter should move the input focus to the next input field or control in the form.
previous: Suggests that pressing Enter should move the input focus to the previous input field or control in the form.
search: Indicates that pressing Enter should initiate a search operation.

The browser may use this hint to display an appropriate keyboard layout or provide other UI cues to the user.

Example usage

Let's consider a simple example of a login form to demonstrate the usage of the enterkeyhint attribute. We have two input fields: one for the username and another for the password. Here's how we can use the enterkeyhint attribute effectively:

<form>
  <label for="username">Username:</label>
  <input type="text" id="username" enterkeyhint="next">

  <label for="password">Password:</label>
  <input type="password" id="password" enterkeyhint="done">

  <input type="submit" value="Login">
</form>

In this example, we set the enterkeyhint attribute for the username input field to "next" and for the password input field to "done". This provides a hint to the browser about the expected action when the user presses the Enter key.

By setting the enterkeyhint to "next" for the username field, we suggest that pressing Enter should move the input focus to the password field. This helps users navigate through the form easily without having to use the mouse or touch input.

For the password field, we set the enterkeyhint to "done". This indicates that pressing Enter should submit the form or perform the action that signifies completion, which, in this case, is logging in.

Browser support

The enterkeyhint attribute is supported by all the browsers.

Source: developer.mozilla.org/enterkeyhint#browser_compatibility

Here is how all of them look:

Conclusion

Hope you learned something new through this blog. Do check out my other blogs and follow me on Twitter at ArnabSen1729 for more such interesting updates. You can also subscribe to my Hashnode newsletter to get updates every time I publish a new article.

Have a nice day 😄 👋.

Breaking Captchas with Golang: Leveraging the Power of 2captcha

Arnab Sen — Mon, 22 May 2023 20:28:52 +0000

Uncover the techniques to break captchas using Golang and leverage the power of 2captcha, empowering your automated systems with advanced capabilities

Understanding the Role of Captchas in Digital Security

In today's digital landscape, where cyber threats lurk around every virtual corner, safeguarding sensitive information is of paramount importance. One crucial tool that plays a pivotal role in enhancing digital security is captchas. Captchas, short for "Completely Automated Public Turing tests to tell Computers and Humans Apart," act as a gatekeeper, effectively distinguishing human users from malicious bots. They were invented in 2000 by Luis von Ahn, Manuel Blum, Nicholas Hopper, and John Langford at Carnegie Mellon University. The original CAPTCHAs were text-based and required users to identify distorted words or letters.

These ingenious puzzles serve a lot of purposes. They help prevent a variety of attacks like:

Spam: Spam is unsolicited electronic messages, typically sent in bulk. CAPTCHAs can be used to prevent spam bots from creating accounts or sending messages by requiring users to solve a CAPTCHA before they can create an account or send a message.
DDoS attacks: A DDoS attack is a distributed denial-of-service attack. This type of attack is designed to overwhelm a website or online service with so much traffic that it becomes unavailable to legitimate users. CAPTCHAs can be used to prevent DDoS attacks by making it more difficult for attackers to generate large numbers of requests.
Credential stuffing: Credential stuffing is a type of attack where attackers try to use stolen login credentials to access multiple accounts. CAPTCHAs can be used to prevent credential stuffing attacks by making it more difficult for attackers to try login credentials on multiple websites.
Phishing: Phishing is a type of attack where attackers send fraudulent emails that appear to be from a legitimate source. The goal of phishing is to trick the recipient into clicking on a malicious link or providing personal information. CAPTCHAs can be used to prevent phishing attacks by making it more difficult for attackers to create realistic-looking phishing emails.
Click fraud: Click fraud is a type of attack where attackers click on ads without human interaction. This type of attack is used to generate revenue for the attacker. CAPTCHAs can be used to prevent click fraud by making it more difficult for attackers to click on ads without human interaction.

By incorporating captchas, website owners can fortify their defense against various cyber threats, such as spam, fraud, and unauthorized access. The utilization of visual or audio-based challenges, often involving distorted characters or logical tasks, impedes automated programs' ability to infiltrate online platforms.

How did Captchas even come into the picture?

Well, this dates back to a paper titled "Computing Machinery and Intelligence" that was published in 1950 by the famous and legendary computer scientist, mathematician, logician, cryptanalyst, philosopher, and theoretical biologist 🤯.

Are you able to guess who I am talking about?

Well here is a hint 💡: He is famously regarded as the father of modern computer science.

Yes, none other than Alan Turing.

He asked a question "Can a computer talk like a human?". This question led to an idea for measuring artificial intelligence that would famously come to be known as the Turing test. Turing proposed the following game. A human judge has a text conversation with unseen players and has to evaluate their responses. To pass the test, a computer must be able to replace one of the players without substantially changing the results. In other words, a computer would be considered intelligent if its conversation couldn't be easily distinguished from a human's.

Believe it or not, this same principle is used today to differentiate between a real human and a bot. You can think of captchas as simplified and automated Turing Test and every time you are able to solve a captcha you are basically passing a mini Turing Test.

P.S: If you are also a movie buff I would highly encourage you to watch the movie The Imitation Game where Benedict Cumberbatch played the role of Alan Turing.

Now, that you know a brief about how captchas came into the picture, let's learn the different types of captchas.

What are the different types of captchas?

You might have across different types of captchas while surfing the internet, here are the broad categories of captchas:

Text CAPTCHA: The oldest and most common type of CAPTCHA, this involves inputting text from a distorted or obscured image. The logic here is that humans can interpret obscured text while bots cannot.
Image CAPTCHA: This type of CAPTCHA involves identifying specific types of objects or patterns in an image. For instance, a user might be asked to select all squares of a grid containing traffic lights or buses.
Audio CAPTCHA: For users with visual impairments, audio CAPTCHAs are used. In this case, a short audio clip is played and the user is asked to type the words or numbers that they hear.

There are a bunch of service providers available who allow you to use their captcha services on your websites. Some of the most popular ones include:

Now, from a personal point of view I kind of hate solving these captchas, and finding the traffic lights.

So, I started looking if there were any ways to bypass these irritating captchas and I landed upon a goldmine 2captcha. Turns out, these captchas weren't as secure as I used to believe. Today we will look at a demo of how I ended up writing a script to bypass these captchas.

But, before that, let's learn about 2Captcha.

What is 2Captcha?

2Captcha operates as a real-time CAPTCHA decoding service, proficient in distinguishing and interpreting captchas with accuracy. It leverages human intellect for image recognition tasks, ensuring high levels of precision. Offering compatibility with a wide range of programming languages through its API, 2Captcha can identify and solve a diverse variety of CAPTCHA types. In fact, 2Captcha can actually solve all the captcha types I mentioned above.

So, without further ado, let's use the Golang module of 2Captcha and build ourselves a CLI tool. You will be amazed to see how easy it is to use this service.

Building the Golang CLI tool

So, the first step would be to create an account in 2Captcha and get our API key.

1. Creating our 2Captcha account

Visit their website https://2captcha.com/ and then sign in with your method of choice. After that, you will land on a page like this:

Select the left option of "I'm a customer" and then you will be redirected to the dashboard where you can see your API key.

Now, to further move ahead with our project make sure that you have Golang installed in your machine. To check that simply run go version in the terminal and it should show the version of Golang that is installed. If it gives an error, you will have to install Golang.

Once you are done with all these we can finally initialize our Golang Project.

2. Initialize our Golang Project

To get started, let's initialize a new Golang project. Open your terminal or command prompt and follow the steps below:

Create a new directory for your project:
```
mkdir captcha-solver
cd captcha-solver
```
Initialize the Go module:
```
go mod init github.com/your-username/captcha-solver
```
Replace your-username with your actual GitHub username or any other relevant identifier.
Create a new Go source file named main.go:
```
touch main.go
```

You have now set up the basic structure for your Golang project.

3. Installing Required Modules

To interact with the 2Captcha service, we need to install the 2captcha-go library. Run the following command to install the required module:

go get github.com/2captcha/2captcha-go

The module will be downloaded and added to your project's go.mod file.

Finally, we will write the code for our captcha solver.

4. Building the Captcha Solver

Now let's proceed with building the captcha solver. Open the main.go file in a text editor and add the following code:

package main

import (
    "flag"
    "fmt"
    "io/ioutil"
    "log"
    "os"

    "github.com/2captcha/2captcha-go"
)

const apiKey = "<API-KEY>" // Replace with your actual API key

func main() {
    // Parse input arguments
    captchaImagePath := flag.String("image", "", "Path to the captcha image file")
    flag.Parse()

    // Verify that the captcha image path is provided
    if *captchaImagePath == "" {
        fmt.Println("Please provide the captcha image path.")
        flag.PrintDefaults()
        os.Exit(1)
    }

    // Load the captcha image
    _, err := ioutil.ReadFile(*captchaImagePath)
    if err != nil {
        fmt.Printf("Failed to read captcha image: %v\n", err)
        os.Exit(1)
    }

    // Initialize the 2Captcha client
    client := api2captcha.NewClient(apiKey)

    // Use the 2Captcha client to solve the captcha
    cap := api2captcha.Normal{
        File: *captchaImagePath,
    }

    captchaText, err := client.Solve(cap.ToRequest())
    if err != nil {
        if err == api2captcha.ErrTimeout {
            log.Fatal("Timeout")
        } else if err == api2captcha.ErrApi {
            log.Fatal("API error")
        } else if err == api2captcha.ErrNetwork {
            log.Fatal("Network error")
        } else {
            log.Fatal(err)
        }
    }

    // Output the solved captcha text
    fmt.Printf("Solved captcha: %s\n", captchaText)
}

Now, let's break down the code and understand each section's functionality.

Parsing Input Arguments

captchaImagePath := flag.String("image", "", "Path to the captcha image file")
flag.Parse()

This code uses the flag package to parse input arguments provided when running the program. We define a flag named image that represents the path to the captcha image file. The flag.Parse() function is then called to parse the input arguments.

Loading the Captcha Image

_, err := ioutil.ReadFile(*captchaImagePath)
if err != nil {
    fmt.Printf("Failed to read captcha image: %v\n", err)
    os.Exit(1)
}

In this section, we use the ioutil.ReadFile() function to load the contents of the captcha image file specified by the user. If there is an error while reading the file, an error message is printed, and the program exits.

Initializing the 2Captcha Client

client := api2captcha.NewClient(apiKey)

Here, we initialize the 2Captcha client by creating a new instance of api2captcha.Client with the provided API key. Remember to replace the API key with the key that you saw in your dashboard.

Solving the Captcha

cap := api2captcha.Normal{
    File: *captchaImagePath,
}

captchaText, err := client.Solve(cap.ToRequest())

To solve the captcha, we create a Normal struct instance from the api2captcha package and pass the captcha image file path to it. Then, we call the Solve() method of the 2Captcha client, passing the ToRequest() method's result from the cap struct as an argument. This method sends the captcha image to the 2Captcha service for solving and returns the solved captcha text.

Outputting the Solved Captcha Text

fmt.Printf("Solved captcha: %s\n", captchaText)

Finally, we output the solved captcha text to the console using fmt.Printf().

Conclusion

Congratulations! You have successfully built a captcha solver using the 2Captcha service in Golang. You can now run the program by executing the following command:

go run main.go --image /path/to/captcha.png

Replace /path/to/captcha.png with the actual path to your captcha image file. The program will send the image to the 2Captcha service, solve the captcha, and display the solved text.

Feel free to explore the 2Captcha Go client library documentation (https://pkg.go.dev/github.com/2captcha/2captcha-go) for more advanced usage and options.

That's it! You can now integrate this captcha solver into your own projects or applications to automate captcha solving using the 2Captcha service.

How to debug buggy code? Exploring the debugger GDB

Arnab Sen — Sun, 21 May 2023 07:15:08 +0000

Introduction

No matter how big or small the codebase is, errors and bugs are inevitable. So as a developer we should know how to fix them. But, even before fixing an error, we have a more daunting task. Can you guess it? Yes, finding where the error is in the first place.

Debugging is the process of detecting and removing existing and potential errors (also called 'bugs') in a software code.

In our day-to-day programming scenarios, we often use the print statements (like console.log()) to debug. But imagine this scenario. You are working on a large codebase and compiling the entire code takes like 3-4 mins. Now if you are stuck somewhere and you add a print statement you have to recompile the code. And if you keep doing that repeatedly, you will have to recompile the code every single time you make a change. This is too inefficient and a huge waste of time.

This is where Debuggers come to the rescue. In this article, we will look at GDB.

What is GDB?

GDB (GNU Debugger) is a debugger for C and C++.

GDB allows you to do things like run the program up to a certain point then stop and print out the values of certain variables at that point, or step through the program one line at a time and print out the values of each variable after executing each line.

GDB uses a command-line interface which might be intimidating at first but once you get used to the commands it becomes very easy to use. We will explore some of the basic commands in this article.

Let's start with installing gdb first.

Installing GDB

If you are using Linux, you probably already have gdb, but if you are using Windows, you will need to install it.

Check if you already have gdb installed by running the following command in your terminal:

gdb --version # if you are in a Unix based system

c:\mingw\bin\gdb.exe --version # if you are in windows

Linux

You can install gdb on Debian-based Linux distro (e.g. Ubuntu, Mint, etc) by the following command.

sudo apt-get update
sudo apt-get install gdb

Windows

If you have MinGW installed then you already have gdb. Otherwise, download the latest MinGW installer from here.

Install MinGW from the installer.
Run the "MinGW Installation Manager" (which might be located in C:\MinGW\libexec\mingw-get\guimain.exe )
Make sure that the mingw32-gdb bin package is installed.

After installation is complete, check the version once again with the commands above. Let's get started with debugging a simple program now.

NOTE: GDB is good but it isn't very fancy, so instead you can use GDB-GEF. It's very easy to install. I will be using GEF throughout this article and you should use it too.

Starting GDB

To debug we need to have some code. Let's write a simple program to calculate factorial in C.

#include <stdio.h>

const int MAX_FACTORIAL = 20;

int factorial(int n) {
  int res = 1;
  for (int i = 1; i <= n; i++) {
    res = res * i;
  }
  return res;
}

int main() {
  int val;
  printf("Enter a number: ");
  scanf("%d", &val);
  printf("Factorial of %d is %d\n", val, factorial(val));
  return 0;
}

Let's compile the code:

gcc main.c -o factorial

Now, let's run gdb on the binary file factorial. We do that by the gdb program command.

gdb factorial

Do you see some errors like this?

Reading symbols from factorial...
(No debugging symbols found in factorial)

To quit gdb press Ctrl+C.

This is because to prepare our program for debugging with gdb, we must compile it with the -g flag. So, let's recompile our program this time with the -g flag.

gcc -g main.c -o factorial
gdb factorial

Now, we don't see the error (No debugging symbols found in factorial) anymore. So now we can use gdb to debug our code.

What gdb did under the hood is that it automatically loaded the symbol table. We will look into Symbol Table later.

Breakpoint

A breakpoint is a spot in your program where you would like to temporarily stop execution in order to check the values of variables, or to try to find out where the program is crashing, etc. To set a breakpoint you use the break command.

Remember: Almost all the commonly used gdb commands have a shorter version and we should use them heavily to improve speed.

break main
# OR a shorter version
b main

Output:

gef b main
Breakpoint 1 at 0x117f: file main.c, line 15.

This means we have set a breakpoint at the main function so that as soon as the instruction pointer reaches the main function it will stop the execution and wait for our commands.

To see the list of all the breakpoints use the command info break. As the command name suggests it will give us information about all the breakpoints we have set.

gef info break
Num Type Disp Enb Address What
1 breakpoint keep y 0x000000000000117f in main at main.c:15

There is an index associated with each breakpoint. These index values will be used if you want to delete a breakpoint or disable it. First, let's run the program with the breakpoint set. To run, use the run command (or just r as a shorter version).

gef run
// OR
gef r

You will see the program execution stopped. Let's just resume the execution with the command continue (or just c).

gef c
Continuing.
Enter a number: 5
Factorial of 5 is 120
[Inferior 1 (process 128334) exited normally]

You will notice it will prompt for the number and then display the factorial of that number and exit normally. Why? Because there were no other breakpoints in our code and execution hence didn't stop anywhere else.

Sometimes, we don't need to stop at a breakpoint so we can disable it. To disable a breakpoint use the disable command.

gef info break
Num Type Disp Enb Address What
1 breakpoint keep y 0x000000000000117f in main at main.c:15
gef disable 1
gef info break
Num Type Disp Enb Address What
1 breakpoint keep n 0x000000000000117f in main at main.c:15

Once, disabled you can notice the value under Enb is not y anymore. It's n representing it's not enabled anymore. If you run the program now, you will notice that it didn't stop at any place. This is because the breakpoint is disabled.

gef r
Starting program: /home/arnab/Desktop/SWE-Lab/ass2/prog/factorial
Enter a number: 5
Factorial of 5 is 120
[Inferior 1 (process 411121) exited normally]

Sometimes, we add a breakpoint to a function inside the loop. And we don't want to stop for every iteration of the loop. In that scenario, we can ignore the breakpoint for a fixed number of times using the ignore command. Let's try that. We have a loop in our factorial() function. Let's add a breakpoint to the loop.

TIP: You can see the code of a function directly from the gdb terminal using the list command. So do list factorial and it will display the source code of factorial().

We want to set the breakpoint at line 8 of this main.c file. So, we will use the command b main.c:8.

gef info break # to list our breakpoints
Num Type Disp Enb Address What
1 breakpoint keep n 0x000000000000117f in main at main.c:15
gef list factorial # to see the source code
1 #include <stdio.h>
2
3 const int MAX_FACTORIAL = 20;
4
5 int factorial(int n) {
6 int res = 1;
7 for (int i = 1; i <= n; i++) {
8 res = res * I; # <-- breakpoint here
9 }
10 return res;
11 }

gef b main.c:8 # add breakpoint to line 8 of main.c
Breakpoint 2 at 0x115c: file main.c, line 8.
gef info break # to check if the breakpoint was added
Num Type Disp Enb Address What
1 breakpoint keep n 0x000000000000117f in main at main.c:15
2 breakpoint keep y 0x000000000000115c in factorial at main.c:8

Now, if you run the program with the r command. It will prompt you for a number (enter something like 5). You will then see it hits the breakpoint at the factorial() function (Remember we disabled the breakpoint at the main() function). Type the command continue or c and it will again hit the same point. Cause not we are now inside a loop. Keep doing this 5 times. Finally, when the loop ends you will notice the factorial result is displayed.

In this scenario, if we want to ignore the breakpoint for the first 4 times maybe. We will use the ignore <index of breakpoint> 4 command.

gef info break
Num Type Disp Enb Address What
1 breakpoint keep n 0x000055555555517f in main at main.c:15
2 breakpoint keep y 0x000055555555515c in factorial at main.c:8
 breakpoint already hit 5 times
gef ignore 2 4
Will ignore next 4 crossings of breakpoint 2.
gef info break
Num Type Disp Enb Address What
1 breakpoint keep n 0x000055555555517f in main at main.c:15
2 breakpoint keep y 0x000055555555515c in factorial at main.c:8
 breakpoint already hit 5 times
 ignore next 4 hits

Let's run the program. This time when it prompts for a number, we will enter 5. After that, it will hit the breakpoint at the factorial() function. Press c to continue, and the program ends. This is because gdb ignored the breakpoint for the first 4 times and only hit the 5-th time we hit the breakpoint.

Now, let's delete that breakpoint using the command delete breakpoints <index>.

gef info break
Num Type Disp Enb Address What
1 breakpoint keep n 0x000055555555517f in main at main.c:15
2 breakpoint keep y 0x000055555555515c in factorial at main.c:8
 breakpoint already hit 5 times
gef delete 2
gef info break
Num Type Disp Enb Address What
1 breakpoint keep n 0x000055555555517f in main at main.c:15
gef delete 1
gef info break
No breakpoints or watchpoints.

Symbol Table

Have you ever wondered how a compiler remembers which data structure is of which type and so on? Well, the answer is Symbol Table. It is a data structure used by a compiler to keep track of semantics of variable names like:

Data type of the variable.
When is used: scope (The effective context where a name is valid).
Where it is stored: storage address.

In gdb, when you run gdb factorial it will load the symbol table. Let's quit the gdb now by pressing Ctrl+C or typing q and Enter. This time just run gdb.

To load a program and its symbol table use the command file <program>.

gef file ./prog/factorial
Reading symbols from ./prog/factorial...

Now that it has read the symbols, we can interact with them. Run the command info address <symbol name> to see the address of that symbol.

gef info address main
Symbol "main" is a function at address 0x1177.
gef info address factorial
Symbol "factorial" is a function at address 0x1145.
gef info address MAX_FACTORIAL
Symbol "MAX_FACTORIAL" is static storage at address 0x2004.

Now since you know the address you can reverse lookup the symbol name with the command info symbol <address>.

gef info symbol 0x1177
main in section .text
gef info symbol 0x1145
factorial in section .text

The best part of this info symbol is that if you don't specify the exact address, show the offset from the beginning of the symbol. So in my case, the main symbol is at address 0x1177. So let's try to see what the symbol name of the address 0x1180.

gef info symbol 0x1180
main + 9 in section .text

Now, remember we had a variable called val in main. Let's see the address of that.

gef info address val
No symbol "val" in current context.

This means we are not in the correct scope yet. Because the variable val was declared inside the main() function, we cannot access it outside the main() function. So let's set a breakpoint at the main and run the program and then we will see the address of val in the symbol table.

gef b main
Breakpoint 1 at 0x117f: file main.c, line 15.
gef r
gef info address val
Symbol "val" is a complex DWARF expression:
     0: DW_OP_fbreg -20
.

To see the list of all the functions in the symbol table use the command info func. You can also filter the functions with regexes. Let's search for our factorial function. The exact regex match will be ^facorial$.

gef info func ^factorial$
All functions matching regular expression "^factorial$":

File main.c:
5: int factorial(int);

We can do the same for global variables using the command info var. Let's look for our global variable MAX_FACTORIAL.

gef info var ^MAX_FACTORIAL$
All variables matching regular expression "^MAX_FACTORIAL$":

File main.c:
3: const int MAX_FACTORIAL;

The symbol table also stores the type of the variable. We can use the command whatis <variable name> to see the type of the variable.

gef whatis MAX_FACTORIAL
type = const int
gef whatis val
type = int

Working with Variables

So far we have seen how to set breakpoints, run the program and get the symbol table. Now let's see how to work with variables. Most of the time we don't need all the global variables rather want to see the local variables. For that, we have the command info locals which will show us the local variables.

First, let's set a breakpoint after the scanf statement (which is line number 17 in main.c file). So run:

gef b main.c:17
Breakpoint 2 at 0x11a8: file main.c, line 17.

If the program is not running press r (to run) otherwise press c (to continue). It will show the prompt Enter a number: to which you should type a value and the program will reach another breakpoint.

Now run the command to get the local variables.

gef info locals
val = 0xa

It says the value of the variable val is 0xa. We can also print the values with the command print (or just p).

gef p val
$1 = 0xa
gef p/d val
$2 = 10

By default, the value will be in hex, but we can specify /d to display in decimal. Here are other formats:

format	description
x	hexadecimal
d	signed decimal
u	unsigned decimal
o	octal
t	binary
a	address, absolute and relative
c	character
f	floating-point

We can also set the variable values, using the set variable command.

gef set variable val=5
gef p/d val
$3 = 5

Functions

Functions play an important role in programs. So, let's see how gdb handles them. In our code, we have two functions main and factorial. So, let's start with setting a breakpoint at main.

gef b main
Breakpoint 1 at 0x117f: file main.c, line 15.
gef info break
Num Type Disp Enb Address What
1 breakpoint keep y 0x000000000000117f in main at main.c:15

Run the program with r. GDB will stop execution at the first line of the function main. Before moving ahead, let's try to understand the difference between this next command and another very similar command called stepi.

next: execute next line, including any function calls.
stepi: step by machine instructions rather than source lines

So, at any point, if we are at a function call, do si (step into) to enter the function.

TIP : You can use a count with next or stepi to perform the command multiple times.

We will perform next 2 i.e next twice. This time you will be prompted for a number. After that, we will step into the instruction that will take us to the factorial function.

gef r
gef n 2 // we will be in that line where factorial is called
gef si 3 // we will be in the factorial function
gef n // the first line of the factorial function

We are doing si 3 because, si steps one instruction at a time, so to go through the function prologue we have to do it thrice.

Now, to see the function arguments we will again use the info command. This time with args. So perform info args.

gef info args
n = 0xa
gef p/d n
$1 = 10

If you know the name of the function argument then we can use the print command to print the value of that argument as well.

Program Stack

Since, we are inside the function factorial() which was called from main() so if we return from factorial() we will be back in main(). For a simple program like this, it's not much of a big deal to remember which function is called by whom. But for a large codebase, it becomes difficult to keep track of it. So we have a command called backtrace that shows us exactly this.

gef backtrace
#0 factorial (n=0xa) at main.c:6
#1 0x00005555555551b2 in main () at main.c:17

This very clearly tells us that we left main() on line 17 and now we are in factorial() on line 6.

Cheatsheet

Command	Description
`gdb program`	to load the symbol table of the program and run the debugger
`break <function name>`	to set a breakpoint at a function
`break *<address>`	to set a breakpoint at a specific address
`info break`	to see the list of all the breakpoints
`delete break <index>`	to delete a breakpoint
`disable break <index>`	to disable a breakpoint
`ignore <index> <count>`	to ignore a breakpoint for count number of times
`run <arglist>`	start your program with arglist
`continue`	continue the program execution
`next`	to execute the next line, including function calls
`stepi`	to step into the next machine instruction
`info address <symbol-name>`	show where symbol s is stored
`info func <regex>`	show names, types of defined functions (all, or matching regex)
`info var <regex>`	show names, types of global variables (all,or matching regex)
`info locals`	show names, types of local variables
`info args`	show names, types of function arguments
`whatis <expr>`	show data type of expr
`p/<format> <expr>`	print the value of expr in the specified format
`backtrace`	show the call stack

Final Notes

Even though there is still a lot more to cover this article should give a basic understanding of how the debugger GDB works. If you are working with a large codebase you will have no option but to use debuggers like GDB. If properly used, these tools make your life a lot easier.

Also, if you are a CTF player you might have come across Binary Exploitation challenges where again the only way to get the flag is by debugging the code and understanding the binary. There are more to GDB, like looking up the registers, finding a variable in a stack, watchpoints, threads, etc but those will be covered in later articles.

I am a college undergrad who is constantly learning. If you have any feedback feel free to put it in the comments. I would really appreciate that. Also, feel free to contact me through any of these social handles (mainly active on Twitter):

Merkle Trees and its role in the decentralized web

Arnab Sen — Wed, 17 May 2023 11:30:00 +0000

Centralised Web

How does the web in general work?

Usually, we have a server where we upload files, and then we can access those files from some other nodes connected to the network the server is connected to.

For example, if you want to host your videos for people to watch, you would rather choose Youtube or Google Drive or similar hosting/file-sharing services. Now, this is very convenient because a user doesn't have to maintain those services or manage them. But it comes at a cost. The data is now concentrated on the infrastructure of a handful of providers.

Content Addressing

But we can avoid this if we use a peer-2-peer system , with the help of content addressing. Before diving into how the p2p system avoids decentralization let's first understand content addressing. So, depending on the value of the content we can generate an address. We can use a cryptographic hash function. One of the popular functions is the SHA-256. In your terminal, you can write this command.

$ sha256sum file # will generate the SHA-256 hash of the file.

Now even if you make a very small change, the sha256 value of the file will change drastically.

Now the output of this function (usually known as hash or digest) is of a fixed length. In the case of SHA-256, it's 32 bytes. Also for each unique data, we will get a unique hash value. Thereby we can use the hash value for integrity check. As an analogy, if Alice and Bob are two different persons then quite naturally they will have different home addresses (not considering the exceptional cases, where Alice and Bob are roommates). So the home address in a way plays the role of identification for Alice and Bob. Similarly, the hash value plays the role of an address of the content (or data).

The exceptional case where Alice and Bob are roommates is actually analogous to something called Hash Collision. Where two different values give the same hash, but it is very rare. So far no one has been able to find such values which give the same sha256 hash.

Peer to Peer Network

Coming back to the peer-to-peer network. Instead of keeping the data in one server, we can share the data across multiple nodes (computers connected to the network are called nodes ).

Now what are the pros:

It requires less disk space since the data is now shared among multiple peers.
If one node gets destroyed/tampered with the data can still be retrieved from the other node. Hence, no single-point-of-failure.

But there are some cons as well. Participants in a peer to peer cannot be trusted. So when we retrieve a file we have to ensure that it is in good condition and is not tampered with or it is not malicious. But how can we do that in an efficient manner? This is where Merkle Trees come into play.

Merkle Trees

Merkle trees are a data structure that allows us to do efficient data verification across networks. It is a binary tree where the nodes store hashes instead of chunks of data.

Merkle trees are named after Ralph Merkle , who proposed them in a 1987 paper titled "A Digital Signature Based on a Conventional Encryption Function." Merkle also invented cryptographic hashing.

The leaf nodes store the hashes of the chunks of the data. And their parent nodes store the hash of concatenating the hashes of left child and right child.

So, if we have 4 chunks of data. We first hash each one of them and get Hash 1, Hash 2, Hash 3, Hash 4. Now we concatenate Hash 1 and Hash 2 and then calculate the hash of that concatenated string. That is stored in B1 (Branch 1), similarly for Hash 3 and Hash 4. We will keep doing this until we come to a single node in the tree which is called the root.

The root node has a special name in a Merkle Tree, we call it the Root Hash. And actually plays a big role in p2p networks.

Checking Equality

Checking for equality becomes easy with Merkle Tree. Why? Let's say one of the leaves has different data (unintended/tampered data). Now the hash for the corresponding leaf will be different. As a result, the parent hash will also be different and so on till the root. So if any data is tampered with, the root hash will give us different values.

So a change in any leaf node will be bubbled to the root hash. Now that reduces the comparisons we need to make to check the equality. And all we need to send to verify the data is the root hash, which is just 256 bits (or 32 bytes).

This ease of verification plays a big role in the p2p network because all the peers are anonymous and are untrusted.

Now, let's say you are supposed to download a file from your peers. And you already have the root hash of the file you were supposed to get. You download the data from your peers. What you can now do is build the Merkle Tree and calculate the Root Hash and check if the calculated Root Hash matches with the one you are expecting. If it matches then we are 100% sure that the data wasn't accidentally corrupted or intentionally tampered with. If it doesn't match then the data is tampered with or it is malicious.

Partial Verification

But why shall we go all the way to make a Merkle Tree and not just concatenate all the data and calculate the hash? The reason is partial verification. Sometimes we don't need the entire data set, only a couple of files are enough. We cannot verify that using the naive approach where we concatenated all the chunks of data and calculated the hash. But with the help of Merkle Trees, we can do it efficiently.

Let's say we have 4 chunks of data. Before uploading to the p2p network I create a Merkle tree and find the root hash.

Now we have saved the root hash safely. Now let's say we need Data 4. And one node says that it has the data. But how do we ensure that the data is not malicious? First, we will get Data 4, so we can just hash it and get Hash 4. Then we will need Hash 3 (remember not Data 3, just the hash). Using Hash 4 and Hash 3 we can calculate hash B2. Then we will need B1 and using B2 and B1 we can calculate root hash, which we already have. So we can easily compare them. So what information did we need?

Data 4 (the chunk of data we need)
Hash 3
B2

That's it. For a bigger tree, we will just need one data and a couple of hashes. The hashes have a fixed size of 32 bytes, so it will save a lot of bandwidth. Also, the number of hashes will be logarithmic to the number of leaves i.e chunks of data. This way Merkle Trees can ease the verification process.

Use cases

Merkle Trees are extensively used in Git, Bitcoin, and many other places. In Bitcoin specifically, it is used to validate transactions in a block. You can read more about it in the book Grokking Bitcoin by Kalle Rosenbaum Section 6.5, the partial verification of Merkle Trees is used to check if a Transaction exists in a block.

Gaurav Sen has also explained the same concept by taking the case of Git. You can watch the video here.

I was a part of the Summer of Bitcoin'21 when I came across the concept of Merkle Trees and understood the important role it plays in Bitcoin and Blockchain. I would like to thank Kalle for explaining these concepts so beautifully. If you are new to bitcoin you should definitely check out his book Grokking Bitcoin. I am also grateful to Adi Shankara, Caralie Chrisco, Adam Jonas for giving us this amazing opportunity.

Google I/O 2023 Highlights: Unveiling Google's Latest Innovations and Improvements

Arnab Sen — Fri, 12 May 2023 06:29:55 +0000

In this blog, I will cover all the latest developments happening within Google, as presented at Google I/O. I won't delve deeply into the workings and technology, as that would make the blog excessively long. Instead, I will provide an overview of the topics discussed and explain them as concisely as possible.

I will mainly cover the new software developments in the field of AI and not the hardware and android updates.

TL;DR

Google I/O 2023 showcased AI advancements such as LaMDA, PaLM2, Imagen, and new AI features in Google Search. They also introduced AI integrations in products like Gmail, Google Maps, and Google Photos, along with the launch of Bard, an AI chatbot. Other announcements include AI tools for developers like Vertex AI and Project Tailwind, an AI-first notebook.

Let's learn what Google I/O is all about.

What is "Google I/O"?

Google I/O is an annual developer conference held by Google in Mountain View, California. The name "I/O" is taken from the number googol , with the "I" representing the 1 in googol and the "O" represents the first 0 in the number.

A googol is the large number 10¹⁰⁰. In decimal notation, it is written as the digit 1 followed by one hundred zeroes: 10,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000,000.

The conference also features a number of educational sessions and workshops, where developers can learn about the latest technologies and best practices.

In addition to being a valuable resource for developers, Google I/O is also a great opportunity for Google to connect with its users and get feedback on its products. The conference is open to the public, and there are a number of activities and events that are available to all attendees.

Google I/O 2023 was a two-day event held on May 11-12, 2023 , at the Shoreline Amphitheatre in Mountain View, California. The event was attended by over 50,000 developers from around the world.

A keynote was addressed by Sundar Pichai , CEO of Google, where he announced new products and features for Google's various platforms.

💡 What were my expectations from Google I/O 2023?

We all know that Google has been a leader in the field of artificial intelligence (AI) for decades. In 2017, they released the revolutionary transformer model, which is now widely used in natural language processing (NLP) tasks such as machine translation, text summarization, and question-answering.

You can checkout the amazing paper by folks at Google titled "Attention is All You Need" from here.

It was therefore no surprise that Google's annual developer conference, Google I/O 2023, was full of announcements about new AI advancements.

Some of the most notable announcements included:

The release of PaLM2 , a 540-billion parameter LLM that can generate text, translate languages, write different kinds of creative content, and answer your questions in an informative way.
The release of Imagen , a new AI model that can generate realistic images from text descriptions.
The announcement of new AI features for Google Search , including the ability to answer your questions in more natural language and to provide more relevant results.
The launch of new AI tools for developers , including the Google AI Platform and the Google AI Test Kitchen.

These are just a few of the many AI advancements that were announced at Google I/O 2023. It is clear that Google is committed to leading the way in the field of AI, and these advancements are sure to have a significant impact on the way we live and work.

A significant part of Google I/O focused on integrating Generative AI into their products. So, what exactly is Generative AI?

🤖 Understanding Generative AI

Generative AI as the name suggests is a type of artificial intelligence (AI) model called Large Language Model (LLM) that can create new content , such as text, images, audio, and video. Generative AI models are trained on large datasets of existing content, and they use this data to learn the patterns and rules that govern how that content is created. Once a generative AI model has been trained, it can use this knowledge to create new content that is similar to the content it was trained on.

Here is a nice video on the recent updates in Generative AI by Google.

📲 AI integrations in Google products

📪 Gmail's new "Help me write"

This AI feature will generate the entire email for you. All you have to do is specify the details of the email in the prompt. It can further pull context from the previous email.

This is something I will personally use quite often. Currently, there are some excellent tools in the form of browser extensions built on OpenAI's GPT model that serve the same purpose. However, it appears that this new feature from Google could be detrimental to those tools.

https://youtube.com/clip/Ugkxf9r4MA7Fztdkgzi6MjvJrh_Baz5ze8dt

🌍 Immersive View for routes in Maps

Google Maps' Immersive View is being expanded to allow users to see their entire route in advance, whether they're walking, cycling, or driving. This will allow users to get a better feel for their route and make informed decisions about where to go. Immersive View will begin to roll out over the summer and will be available in 15 cities by the end of the year.

https://youtube.com/clip/UgkxcxMUAvh_dh2v1vf-QCk_W8bl2pf42mBb

Magic Eraser for Google Photos

It is an AI-powered tool that allows users to remove unwanted distractions from photos. Magic Editor is a new AI-powered tool that will allow users to do much more with their photos, such as repositioning objects, changing the sky, and more. Magic Editor is expected to roll out later this year.

The second-gen PaLM model: PaLM 2

PaLM is a large language model (LLM) from Google AI, but it is not the only LLM that Google has developed. Other LLMs from Google include LaMDA, Meena, and T5. PaLM is the largest and most powerful LLM that Google has developed to date, but it is still under development.

And in this I/O, Google announced a more advanced version of that called PaLM2.

PaLM2 is a second-generation large language model (LLM) from Google AI. It is built on the same architecture as PaLM, but it is trained on a larger dataset of text and code.

PaLM2 is a family of models, each with its own strengths and weaknesses. Here are some of the different models in PaLM2:

Gecko: Gecko is the smallest model in PaLM2. It is fast and efficient, but it is not as powerful as the other models.
Otter: Otter is a medium-sized model in PaLM2. It is more powerful than Gecko, but it is not as powerful as the larger models.
Bison: Bison is a large model in PaLM2. It is more powerful than Otter, but it is not as powerful as the largest model.
Unicorn: Unicorn is the largest model in PaLM2. It is the most powerful model in PaLM2, but it is also the slowest and least efficient.

PaLM2 is particularly good at:

Math: It can solve math problems, such as algebra, calculus, and geometry. It can also generate mathematical proofs.
Coding: PaLM2 can generate code that is both correct and efficient. It can write code in a variety of programming languages, and translate between them.
Learning & Reasoning: PaLM2 is able to learn new tasks and concepts quickly. It can be trained on a new task with just a few examples.
Translating languages: PaLM2 can translate languages with high accuracy. It can translate between any pair of languages that it has been trained on.

PaLM2 models perform even better when fine-tuned for particular applications. Some of my favorite fine-tuned models are Sec-PaLM and Med-PaLM2. Sec-PaLM can explain the behavior of potentially malicious scripts, and better detect which scripts are actually threats to people and organizations. Med-PaML2 can answer questions and summarize insights from a variety of dense medical texts.

Google is also working on another next-generation foundation model called Gemini. Gemini is being designed to be multimodal, which means that it will be able to process and understand information from a variety of sources, such as text, code, images, and audio. This will make Gemini much more versatile than previous AI models, and it will allow it to be used in a wider range of applications.

💬 Bard

The part of Google I/O that I was most excited about was the launch of Bard. It was initially rolled out to only US and UK, but now it's publicly available and I am really loving it.

A very popular alternative that we all know is ChatGPT. Here are some of the features that Bard is providing which is not yet available in ChatGPT:

Information Cutoff: The information cutoff for ChatGPT is September 2021. Whereas Bard can perform an internet search and give a response accordingly. This allows you to get the latest information and updates.
Export options: Bard allows you to export the response as Docs or a Draft in Gmail. If it detects code then it also gives an option to "Export to Colab" but ChatGPT doesn't have any such functionality yet.
Citations: Bard can also cite sources, something which ChatGPT still can't do.

Further, Google is promising that in the next few weeks, Bard will become more visual. Bard will also be integrated with Google Lens, where it can analyze the photos and then respond according to your prompt.

🎨 Bard x Adobe Firefly

This collaboration will allow Bard users to generate images directly from their text descriptions, using Firefly's state-of-the-art technology.

To use this feature, Bard users will simply need to type a description of the image they want to generate into the chatbot. Firefly will then use its AI to create an image that matches the description. Users will be able to edit the image as needed, and they can even share it directly on social media.

💼 Workspaces

Some of the popular tools available in Google Workspaces like Docs, and Sheets will have options for users to prompt specific requirements. From generating job descriptions, and creating tables with data inserted, to creating images with specific styles everything can be done with just a simple prompt directly from these tools. You can do these with a side panel called "sidekick" which will keep track of the context and then provide relevant information. This will make prompting very easy.

All of this will be generally available to business and consumer Workspace users later this year via a new service called Duet AI for Workspace.

🔍 Generative AI in Search

A typical Google Search can be quite challenging, to be honest. To obtain very specific information, you must break down the text you are searching for and then skim through the results to find the precise answer.

But with this new integration of Generative AI, you don't have to do so anymore. It will automatically take care of extracting the information, getting relevant results and then compiling the results to give you the perfect response.

Vertex AI

Vertex AI is a managed machine learning (ML) platform that helps you build, deploy, and scale ML models. It offers a unified experience for managing the entire ML lifecycle, from data preparation to model training and deployment. Vertex AI also provides a variety of tools and services to help you accelerate your ML projects.

Other big techs are already availing this functionality in their apps, like:

replit
Uber
Canva
character.ai

Vertex has the support of 3 new models, in addition to PaLM2.

Imagen: which powers image generation, editing, and customization from text inputs.
Codey: for code completion and generation, which can be trained on code base to help build applications faster.
Chirp: universal speech model which brings speech-to-text accuracy for over 300 languages.

All of these features are already in Preview and can be used.

📔 Project Tailwind

Project Tailwind is an experimental AI-first notebook that learns from your documents and helps you learn faster. You can upload the docs using Google Drive and that creates a custom fine-tuned model for you to interact with.

It can be used:

to create study guides
find sources
generate quizzes
do a quick Google search to get more information.

Conclusion

Google I/O 2023 unveiled groundbreaking AI advancements and integrations, showcasing the power of AI to revolutionize the way we live, work, and interact with technology. These innovations promise to make our lives more efficient, connected, and inspired as we move toward a brighter future.

As we embrace the future of AI, let's harness its power to innovate, improve our lives, and create a world where technology is a catalyst for positive change, empowering us to reach new heights and achieve our greatest potential.

References

If you found this blog useful, do leave a comment and like this blog, it really motivates me to put forward more such content. You can follow me on Twitter for further updates. My Twitter handle: @ArnabSen1729.

You can find the rest of my social links at arnabsen.dev/links. Thank you and have a wonderful day.

Landing a Google SWE Internship: Interview Experience and preparation journey

Arnab Sen — Tue, 04 Apr 2023 19:14:00 +0000

Even though, there is a plethora of articles on Google SWE Internship interview experience online, I am still frequently asked the following questions:

How did you crack Google SWE Internship?
Which resources did you follow?
Can you share some interview tips for my upcoming interview?
Was it on-campus or off-campus?

... and many more. So, I decided to pen down my thoughts so that I can share this article next time someone asks me anything about my SWE Internship.

What can you expect from this blog?

These are the few things that I would like to cover in this blog

How I optimized my resume.
How I approached (or, usually approach) referrals.
Resources I referred to while preparing for DSA.
Interview tips

NOTE: Although this article is about Google, the recommendations will be useful if you apply to other product-based companies.

Introduction

Hi, I am Arnab Sen, a Software Developer from India 🇮🇳. I interned at Google as an SWE for the summer of 2022 and even grabbed a Pre-Placement Opportunity. I will join Google India as a Full Time, in July 2023. When I was in my third year Google, unfortunately, didn't visit our campus. So I had to take the off-campus route to land at Google and here is my journey:

Decoding the Google Application Process

Let's understand how Google usually hires candidates for the Software Engineering Role:

Resume Shortlisting
Online Aptitude Round
Telephonic Round
Technical Interviews
Hiring Committee

I'll go through each round and talk about what happened and how I prepared.

The same rounds are usually followed by most of the Product based companies like Microsoft, Amazon, Meta .etc.

Round 1: Resume Shortlisting

It was in Aug 2021 that I came across the Google India SWE Intern opening in Google's career portal and I applied with a referral from one of my seniors.

Resume building tips

Your resume is the first thing that creates an impression. Even if you are super talented and skilled, a poor resume can negate all your abilities. So, you need to create a kickass resume that showcases your talents and impresses everyone!

Originally I used to create my resume in LaTeX but later on, it became a hassle to maintain and make updates. Hence I moved on to this website called flowcv.com.

Their interface is user-friendly and offers pre-defined templates while allowing ample options for personalizing your resume. Check out the template I personally used.

I prefer a single-column resume. If you want to stick to LaTeX and you are looking for a resume template, my suggestion would be to use DeedyCV. It looks very professional and compact.

Resume tips to keep in mind

Here are some of my tips for your resume:

If you wanna look like a pro, make sure your stuff doesn't have any spelling or grammar mistakes. You can use tools like Google Docs spelling check or Grammarly to help you out.
Just get straight to the point and don't use fancy words. Nobody cares about your language skills, they just want to know what you can do as a developer.
It would be great if you could add some proof to your work. If you have built a project, published a paper, or completed a certification, you must provide a link for the respective work.
Don't stuff everything into one page of your resume. Leave some space to breathe. It's totally fine to use 2 pages if you need to. I checked with some Google peeps and recruiters and they all said it's all good to have a 2-page resume if one page just ain't cutting it.

Now, let me share with you some hacks that can use:

If you don't have achievements, try to take part in Leetcode, GFG, Hackerrank, or HackerEarth contests or contests of college fests. You can put those ranks in your achievements. To know the upcoming contests, check out clist.by.
If you don't have projects, try to take part in hackathons with your peers. "I can't think of a good project" is what I hear from a lot of my juniors. Right now, if I ask you to think of anything, you won't be thinking of anything. But if I asked you to think of a fruit, you might be thinking of a banana, apple, mango, etc. Every hackathon has a unique theme, so by the end of the hackathon you will have a somewhat unique project. You can find such hackathons at Devpost.
If you don't have enough work experience, pick up a good open-source project from GitHub and try to make contributions to that for like 1-2 months. By the end of it, you will have enough experience and contributions that you can put it under the "Open Source Experience" category.

If you are interested to know what my resume looks like, check out: arnabsen.dev/resume.

Elevate your referral game

Companies like Google actually let their own employees refer potential candidates. Although a referral doesn't necessarily guarantee you a job offer, it does mean your application will stand out from the rest of the applicants, which can only mean one thing - your chances of getting hired just got a whole lot higher. In some cases, a Good Resume + Referral can fast forward in the application rounds. In my case, I was directly skipped to the Technical Interview rounds.

Now few things to keep in mind:

The number of referrals doesn't matter. It's the quality of the referral that makes the difference. I'm familiar with Google's internal referral portal. Employees must respond to questions such as, why do you believe this candidate is a good fit, have you previously worked with the candidate, and so on. Hence, if the person referring you understands your strengths and weaknesses, the referral will be stronger, and your chances will be better.
I was lucky enough to have a senior in my college who was ready to refer me. We knew each other since day 1 of my college so he was able to write a good referral for me. Shoutout to Suryansh Gupta. So you can always reach out to your seniors regarding referrals.
If you don't have such seniors in your college, no worries, you can get referrals over LinkedIn, Twitter, etc. When asking for referrals make sure you provide all the necessary information:
- The job id of the role you are applying for.
- Your resume.
- Your achievements and relevant skills.
- Why do you think you are a good fit for the role?

College tip: active interaction with seniors in your college can have a positive effect on your career. So, don't be shy.

Now, let's say your resume got shortlisted. Then what?

Round 2: Online Aptitude Round

Even though I skipped the OA round, here are some preparation tips:

Google usually asks pretty standard questions which are of the Leetcode Medium-Hard difficulty. So don't just solve easy questions. Also, no need to waste time by solving very difficult questions either.
Make sure you know all the standard algorithms, I will share the list after this.
I also prepared by participating in Competitive Programming contests, especially from Atcoder and LeetCode. This was done to ensure that I could answer the questions swiftly and within the time limit, exactly like an actual OA.

My advice would be to go through the past OA rounds questions. Here is the list I referred to for Google's questions:

You can find similar collections of questions for other companies as well.

Standard topics to prepare for OAs

Binary search
BFS/DFS/Flood fill
Tree traversals
Hash tables
Linked list, Stacks, Queues, Two pointers/Sliding window
Binary heaps
Dynamic programming
Union find
Ad hoc/string manipulations
Trie (good to know)
Segment trees/Fenwick trees (good to know)
Bitmasks (good to know)

Once you are done with the OA you will be taken to the 3rd round.

Round 3: Telephonic Round

So, for this round, they're gonna call you up and hit you with some theoretical Data Structures and Algorithms questions. Some might even have multiple-choice answers. They might throw in some programming language questions too. Here are some examples of questions that you can expect to get an idea:

What are the pros and cons of using an adjacency list over an adjacency matrix for a graph problem?
What is the data structure used in a priority queue?
What is the difference between Binary Search Tree and Heap?
How is a C++ map different from unordered_map from an implementation point of view?
Which algorithm would you use for a network flow problem?
What are the time complexities of Bellman-Ford and Dijkstra's algorithm?

If you have a strong fundamental understanding of Data Structures and Algorithms then these won't be very difficult for you.

Resources to study DSA

I studied DSA mainly from these two sources:

Introduction to Algorithm by CLRS (a.k.a the Bible of DSA)

MIT 6.006 Introduction to Algorithms, Fall 2011

So, I made sure to take notes while watching those videos and reading the book chapters. And when I got stuck with any problem, I just Googled it and checked out the first link - be it an article or a video.

Once you complete the telephonic round you will be invited for the technical interview rounds.

Round 4: Interview Rounds

It was on 13th September I received a mail stating that I have my interview lined up after a week. I freaked out a little. Because first, I didn't expect to hear back from them. And second, how am I supposed to prepare for a Google interview in 7 days?

I took a smart approach. I checked out other people's experiences with Google Interviews and saw that they like to ask about DP, Graphs, Binary Search, Lazy Sum, and Sliding Windows stuff. I went back and reviewed all the notes I took from that MIT course and honed in on those types of questions. I practiced topic-wise from LeetCode itself.

Another question I get is if LeetCode premium is really worth it. To be honest, in today's time it isn't anymore. There are tonnes of other platforms which have similar questions for free that are only available in LeetCode premium. But 2 years back, I felt spending Rs 10.5k for 1 year of LC premium might be worth it. Because if I am able to crack a job like Google, that amount invested will be just a fraction of the monthly salary. So, I got 1 year of LC premium.

My style of practicing for interview

After discussing with my seniors I realized that solving a question in an interview setting is completely different than practicing questions by myself. Most of these Google interviews happen on an editor-based portal where you cannot run the code and it also doesn't support intellisense. So, I would create a Google doc where I would solve all the questions and write code. This way I didn't have the privilege of syntax highlighting or intellisense. As they say, "Hope for the best, Prepare for the worst".

The strategy of breaking down Google interviews

A Google Interview happens for 45 mins and they usually ask 2 questions. So, I divided the total time into (20 + 20 + 5) mins. 20 mins for each question and 5 mins as a buffer (for an emergency like network issues or Q&A at the end).

Within those 20 mins, I would spend 3-5 mins understanding the question asking clarifying questions, and explaining my approach. 10 mins for implementing the optimized solution and 5 mins for dry running or debugging. Note that you have to come up with the optimal solution as quickly as possible. So, I would insist on not wasting time trying to implement the brute-force solution.

So, even during practice when I was solving the questions myself I followed the same strategy. I would pretend that I am in an interview setting. I would try to read my thoughts out loud and explain everything. I even made the habit of doing a dry run every time I wrote the final code for a problem.

Mock Interviews are an absolute necessary

To succeed in any field, preparation is key. Just as astronauts undergo rigorous training to prepare for the unique conditions of zero gravity, so too must job seekers prepare for the challenge of a job interview. One effective way to do this is by practicing mock interviews. That way you can know your weakness, and you can prepare for them before the actual interview.

I also watched a lot of Mock Interview videos on Youtube to give me an idea of what it would be like during an interview setting. I created a playlist of all the videos that I referred to.

Link to the playlist 📼.

I gave a bunch of mock interviews (around 5-6) by taking the help of my seniors and using online free platforms like Pramp.

Importance of clarifying questions

The initial question that will be given to you, will be intentionally very vague. They will expect you to ask some clarifying questions to identify the fine details of the question. Let's take a very simple example, you are given an array of n numbers, and you have to find the subarray with the maximum sum.

Now, your first thought might be using Kandane's Algorithm. But wait, what if all the numbers are positive? In that case, the answer is simply the entire array. As you can see one small change in the constraints, had a big effect on the final solution. So always practice asking clarifying questions.

Interview tips to keep in mind

Treat the interview more like a conversation with a fellow software developer and not like a viva. This will also help you tackle your nervousness.
During the interviews, you should also think about other ways to improve the solution. Share the different options or tradeoffs that you're considering.
Think out loud because that way the interviewer will know your thought process and can give you signals if you are going in the wrong direction.
Discuss tradeoffs if possible. How would you improve your solution? How do you make it faster? How to optimize the space complexity? Know the time and space tradeoffs of the solution/data structures that you are picking.
Try to avoid taking hints, but always remember solving with hints is much better than not solving at all.
Avoid syntactical errors like missing semicolons, not closing parenthesis, wrong indentation, etc. Even though they are not deal-breakers, writing high-quality code can showcase your skills as a developer.
Finally, use proper variable names and if required add meaningful comments. But remember you have limited time so just don't spend minutes trying to come up with an appropriate variable name.
Make it a habit to always calculate the Time and Space complexity of your solution.

If you don't remember the exact syntax of a library class or method, that is fine. You can just let the interviewer know, and replace it with a meaningful substitute.

All of these points might be hard to keep in mind during the interview, which is why the more mock interviews you give, the better you get at remembering these subtle points.

Here are some more resources that I referred to:

Big-O-Cheatsheet

For some last-minute revision, I also followed this collection of LC questions.

https://seanprashad.com/leetcode-patterns/

My interview experience

I cannot reveal the actual questions because of NDA, but in my first interview, I had a question on Sliding Windows and Dynamic Programming.

In my second interview which was held the next day, I had 3 questions. One was a Scheduling Problem based on the heap, the second was again a DP question. I was able to quickly answer these 2 questions, and there were around 10-12 mins left. So the interviewer gave me another DP question for which he just asked me to write the recursive function and how I would memoize it. I was able to do that as well.

Both my interviews went well from my end, I didn't take any hints, wrote high code quality, and was able to successfully dry run all my solutions with edge cases.

Round 5: Hiring Committee

After I finished my interviews, I had a long wait. According to what I've heard, there is a hiring committee that comprises four to five persons that have prior interview experience and are familiar with the hiring standards. They would review your resume, your code during interviews, and the feedback from the interviewers before making a final decision.

For me, it was a long long wait. I very clearly remember the day. It was Maha Shasthi (the onset of Durga Puja), I woke up to a mail saying:

"Congratulations on your SWE Summer Intern offer with Google India!"

It was a perfect blessing to me. I was ecstatic and overjoyed. It was an absolute dream come true moment for me and my family. Looking back I remember I felt very depressed when I was rejected by Microsoft twice. I gave my best in the Microsoft Engage Aptitude round yet wasn't even selected for the project building round. Then again when Microsoft came on-campus I was rejected after the OA round which still baffles me. But as they say "Whatever happens, happens for Good".

So whenever something unexpected or bad happens to you, remember these lines:

I hope you liked my blog, do follow me for more such content. I wrote another blog sharing some tips on how to convert your internship to PPO.

You can learn more about me at arnabsen.dev/about. Have a nice day !!

Important tips on bagging PPO during Internship and securing a Full-Time Role

Arnab Sen — Sun, 29 Jan 2023 17:40:49 +0000

If you are here after you have got an internship, then a big congratulations to you for getting an internship. I hope you are enjoying it and learning a lot. I am sure you are. But now you want to make sure that you get a PPO at the end of it. So, let's get started.

I have been wanting to write this blog for a long time, I will try to include all the important points that I followed and have learned during my internship which helped me bag a PPO. I will break them down into 2 parts mainly, Technical and Non-Technical. Before that let me give you a little background.

Background

I applied for Google Summer SWE Intern Opening on the Google's careers portal with a referral. My Resume got selected and I was called for 2 Rounds of Interviews. After that I got the offer and interned at Google for 10 weeks remotely with the Google Ads team. And finally, I was able to crack the PPO and got a full-time offer from Google as an SWE. I am planning to join them in July 2022.

From
// Detect dark theme var iframe = document.getElementById('tweet-1528345010164613123-821'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1528345010164613123&theme=dark" }

To
// Detect dark theme var iframe = document.getElementById('tweet-1551660735440179201-631'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1551660735440179201&theme=dark" }

Conversion Process

Before sharing the pieces of advice, I feel it's important to learn about the conversion steps. This will vary from company to company but overall the process remains the same.

Every Intern is assigned a project and a mentor (at Google we call them host) and a co-host. The project is usually a part of the product that the team is working on. The mentor is a senior engineer who will be your guide throughout the internship. You will be working with your mentor and other engineers on the team.
There will be two evaluations during the internship. The first evaluation called Mid Evaluation happens after 5 weeks (since, my internship period was 10 weeks) and the Final Evaluation happens at the last week of your internship. The Mid Evaluation is more of a check-in to see how you are doing and if you are on track. The Final Evaluation is more of a review of your work and the project. The host and co-host also gets to give their feedback on your performance and your project progress. This feedback plays a huge role in the final conversion.

Technical Advice

1. Learn the basics

You must have strong problem-solving abilities to work on the project that will be assigned to you. There are some basic Computer Science concepts that one should be familiar with. I will list them down below.

a. Version Control Systems

No matter which company you are working at, you will be using a version control system. You must be familiar with the basics of version control systems. I would recommend you learn Git and GitHub. You can learn it from here.

At Google, we didn't use git, instead, we used Mercurial. But the fundamental concepts are the same. If you know how git works, learning a new version control system will just require you to learn the commands specific to that version control system.

b. Basic Commandline

There is a high chance that you will be working in a cloud environment and you must be comfortable with the terminal. You should be able to navigate through the file system, create directories, create files, delete files, etc. You should also be able to use basic commands like cat, grep, sed, awk, etc. You can learn about these commands from the man pages. For example, to learn about the cat command, you can run man cat in your terminal.

c. Data Structures and Algorithms

You should be familiar with the basic data structures and algorithms. You should be able to implement them in your language of choice and most importantly be aware of when to use which data structure and algorithm.

Note: You don't need to be an expert in these topics. You just need to be familiar with them. But most of these companies have DSA questions in their interviews. So, it is important that you are familiar with them.

I love the MIT 6.006 Introduction to Algorithms lectures, they have helped me build the foundation of algorithms. You can find the playlist here

2. Knowing the syntax of the language you will be using

Sometimes you can ask the recruiter about the tech stack that you will be using. But most of the time, you won't be able to know the tech stack beforehand. But, you can still learn the basics of the language that is popularly used in the company. For example, I knew that Google uses C++, Java, and Golang a lot in their backend. I already knew C++ and Golang. So I learned the basics of Java before joining Google. This helped me save a lot of time during the internship picking up the language.

3. Learn Design Patterns

Design patterns are important for developers to understand because they provide proven solutions to common software design problems. By using design patterns, developers can improve the quality, reusability, and maintainability of their code.

While you don't need to know every design pattern in existence, it's helpful to have a good understanding of the most common design patterns and when to use them. This knowledge can save a lot of time and effort by allowing you to reuse tried-and-true solutions to common problems, rather than having to reinvent the wheel every time you encounter a new challenge.

4. Write clean code

I can't stress this enough. Writing clean code is very important especially if you are interning at a big organisation like Google. Every change you make goes through a rigorous review process. If your code is not clean, it will be rejected. So, you must write clean code. Here are some points that you should keep in mind while writing clean code.

Keep it simple: When writing code, it's important to avoid unnecessary complexity. Simple, easy-to-understand code is usually easier to maintain and debug.
Use clear and descriptive names: Choosing clear and descriptive names for variables, functions, and other elements of the code can make it much easier to read and understand. Avoid using abbreviations or single-letter names, unless they are well-known and widely understood.
Use comments: Comments can be a helpful tool for explaining the purpose and functionality of different parts of the code. However, it's important to use comments sparingly and only when they are truly necessary, as too many comments can make the code harder to read.
Follow a consistent style: Consistency is key when it comes to writing clean code. Using a consistent style for indentation, spacing, and other elements of the code can make it easier to read and understand. It's also a good idea to follow established coding standards and guidelines, such as those provided by the programming language's community or the company you work for.

5. Learn testing

Testing is an important part of software development. It's important to test your code to make sure it works as expected and to catch any bugs that might have been introduced during development. Testing can also help you to ensure that your code is clean and easy to understand. There are different types of testing like unit testing, integration testing, etc. Knowing about these testing types will help you a lot.

Here is a free course on Test Driven Development.

6. Know how to write a Design Doc

At Google, we heavily focus on the Design Doc. Even before we write a piece of code for a particular implementation we have to make sure we have analysed every possible edge case, and technical and non-technical requirements. Design documents also help with communication among team members and can serve as a reference for future maintenance and updates to the software.

There are certain key points that need to be covered in a Design Doc like Architecture, Requirements, Design, User Interface, Testing .etc.

Here is a great post that talks in depth about Design Docs.

This is a sample template of a Design Document.

Non-Technical Advice

In addition to the technical advice, I would also like to share some non-technical advice that I followed during my internship.

1. Be proactive

This was something my host pointed out in my mid-evaluation. I was being too dependent initially on my mentor, waiting for her to tell me the next steps. But later I took the advice. So I started planning my work, assigning tasks to myself, and taking the initiative to solve the problems. But obviously, I kept my mentor in the loop. This takes me to the next point.

2. Keep your mentor (host and co-host) in the loop

Remember, your host and co-host will be evaluating you at the end of your internship. Also, they have their deadline and they will be busy with their work. So, you must keep them in the loop. You should be able to communicate with them and tell them what you are working on and what problems you are facing. This will help them to guide you better.

We had a weekly meeting with our host and co-host but even then I would update them on my progress every alternate day so that they are aware of what I am working on. I would also disclose the problems I faced and the solutions I found.

There is a really good article on Writing a brag document that I would recommend you to read.

3. Be open to feedback

No matter how big a company you are interning at, you should remember that you are still very new to this field and there is a lot to improve. Feedback from your host/co-host or other team members can help you identify areas of their work that needs improvement. It might be difficult to accept the feedback at first but you should be open to it. You should be able to take the feedback positively and work on improving yourself.

Not always your mentor will reach out to you to give you feedback. If required you might have to reach out to them and ask for feedback. You can ask them to give you feedback on your code, your communication skills, your problem-solving skills, etc. This also shows that you are proactive and you are willing to make improvements.

A good place to get feedbacks from your host/co-host is through 1:1 meetings. Here is a good video on The Art Of The 1:1 Meeting.

4. Time management

Plan out the tasks that you will have to do, assign priority to them and then start working on them. This will help you to manage your time better. You should also be able to estimate the time that you will need to complete a task. Initially, you may end up overestimating or underestimating the time but as you gain more experience, you will be able to estimate the time better.

5. Ask questions

An internship usually lasts 10-12 weeks, which is not a lot of time. So if you are stuck at something, you can't afford to lose time. You should be able to ask questions to your mentor and other team members. But before that make sure you have done a fair amount of research on your end as well.

You have to ask the right questions to get quick answers. If you came across an error message, explain properly how to reproduce the error, and what the error message is. Share the code snippet that might be causing the error, or share the files you are working on. Further, the approaches you have already taken to solve the problem. This will help the person you are asking the question to understand the problem better. Also, make sure you are asking the question in the right channel. For example, if you are stuck on a coding problem, you should ask your mentor or other team members. But if you are stuck in a process, you should ask your host/co-host.

Here is a great article on How to ask questions the smart way.

Podcast w/ Aniket

I was recently on a podcast where we went through some of these points in more depth. Watch the video here:

Conclusion

These are some of the things that I followed during my internship. I hope this article will help you to prepare for your internship. One thing you should also keep in mind is to enjoy the internship. It is a great opportunity to learn and grow, network with people and build your resume. Don't burnout yourself before or during the internship. Take breaks, go out with your friends, play games, watch movies, etc. This will help you to stay motivated and perform better.

If you have any questions, feel free to ask in the comments. I will try my best to answer them. Also, if you have any other advice that you would like to share, feel free to do so in the comments.

My experience of Summer Of Bitcoin'21

Arnab Sen — Tue, 17 Jan 2023 12:06:14 +0000

On June 26, I received an email saying that I was one of the 50 students selected for the Summer of Bitcoin 2021 program. I was really happy and felt proud of myself because to be very honest I didn't have high hopes at that point. In this blog, I will cover the selection process, and my experience at Summer of Bitcoin and will also share some tips for those who are interested in this program.

About the Summer of Bitcoin

A global program that matches students with open source, free software, and technology-related organizations working on bitcoin to write code and become part of these communities while making some BTC along the way! The organizations provide mentors who act as guides through the entire process, from learning about the community to contributing code. Students get involved in and become familiar with the bitcoin open-source community and put their summer break to good use.

To know more visit the Summer Of Bitcoin website.

🪴 Selection Process

How I came to know about the program

I was scrolling through LinkedIn when I came across a post about the Summer Of Bitcoin. There was a form attached to it. It had some basic questions and a couple of essay questions like why do you want to take part etc. My tips for filling out the form would be to:

Be truthful and original. You have to understand that those who will select you are way more experienced than you so don't just copy-paste from random sites.
Make sure you don't make any grammatical errors. What I usually do is first write it down in Google Docs and use spell-check. You can also install the Grammarly extension to avoid typos and grammatical errors.
Gather information. Go through the official websites, or take help from past participants. In my case, this was the first time, so there wasn't much public information available, but yeah I did visit the official website. The reason I am saying this is you can understand what the motive of the program is and then write your essays on the same line.

1️⃣ Round 1: Problem-Solving

We are not supposed to discuss the problem and our approaches, so I would refrain from giving details

It was not a typical CP question, rather was more real-life and bitcoin-related. We had to find an optimal solution but satisfy the conditions specified. And then submit our code. After spending around 2 days on the problem I was somewhat satisfied with my result, and I thought of submitting it. I also described my approach in a README file.

2️⃣ Round 2: Follow-up question

Then the next day I received a follow-up question in the email where I had to explain in detail a better approach to my previous solution. I had one more approach in my mind which I then explained and also proved how theoretically it is supposed to give a better solution.

And a few days later I received this mail:

Yayy !! My solution was in the top 1% among thousands of applicants. At this point, I was pretty confident.

3️⃣ Round 3: "Navigating Ambiguity"

We were asked to find an issue in the official Bitcoin repo and explain what approach we would take to solve it. So I went to the issues tab and I didn't understand a single word since I was an absolute beginner. But I spent around 2 days trying to figure out the issues, even while doing this I learned a lot about stuff like fuzzing, AFL, and many other concepts. I picked one particular issue, tried to understand it end to end, researched a little bit, and came across an approach that I then explained.

4️⃣ Round 4: Telephonic Interview

2 days later I received a calendly link for a telephone round. In the telephonic round, I was asked to give a brief introduction of myself, my past experiences, and achievements, technical skills, and projects. Also, I was asked some questions like why I wanted to be a part of this program and what my expectations are.

After the call, my interviewer sent me a link for a video interview which was scheduled an hour later.

5️⃣ Round 5: Final Interview

Here I was asked more about my knowledge of Bitcoin and Blockchain, which programming aspects I like to work on, and some more HR questions. I also asked him questions regarding the program.

I prepared myself by researching a little bit on Bitcoin, going through the Bitcoin whitepaper, and watching a couple of introductory videos.

At the end of the conversation, I was pretty confident that I was able to ace it, cause I was getting positive vibes and my interviewer also shared some resources to get started (I have added them below).

📼 Sessions

We had some really amazing sessions with folks who are associated with Bitcoin for years. They have made contributions to Bitcoin Core and other bitcoin-related projects, and every one of them shared their perspective towards Bitcoin and how revolutionary this concept of a "Decentralised monetary system" is.

Here are some of the links to the recorded sessions:

The CS in Bitcoin with Sanket Kanjalkar

Deep Dive Into Bitcoin Mechanics with Matthew Zipkin

Decentralized open-source development with Amiti Uttarwar

Layering to Scale Bitcoin with Will Clark

You will find more on the Summer Of Bitcoin Youtube Channel.

Then we had some lectures which were given by Kalle Rosenbaum, he is the author of the book Grokking Bitcoin. I would suggest this book to any beginner. The book explains how a system of Cookie Token used in a Cafe can be improved step by step and we will end up with a system like Bitcoin. The last day of the first week ended with experimenting with the bitcoin-cli. We created our own bitcoin full node in our machine and created a wallet. For testing purposes, we used the signet. Kalle gave us some coins which we then used to make transactions among our peers.

In between the lectures we used to have Breakout rooms that would give us a chance to discuss with peers and solve questions and resolve each other's doubts.

By the end of the first week, I went from having very little knowledge about Bitcoin to having a good grasp on fundamentals like consensus (proof-of-work), the P2P network, how bitcoin protocol upgrades take place, and innovations taking place in Bitcoin like the SegWit, Taproot, etc.

📦 Project Matching

Everyone was matched with a mentor and was assigned a Project. I was assigned the amazing 0xB10C and my project was to improve the "User-Space, Statically Defined Tracing" (USDT) support for Bitcoin Core. For the next few weeks, we collaborated with our mentors and made contributions to the various projects at the same time reviewing each other's work. I worked on adding some utxocache tracepoints for events like flush, addCoin, spendCoin etc. I learned a lot about the eBPF (extended Berkeley Packet Filter) technology, tracepoints, Linux Observability, navigating through large codebases like Bitcoin, and about open source in general.

🗓️ Weekly Seminars

For the first 5 weeks, we also had weekly seminars on Bitcoin and Lightning Protocol Development. I was in the Bitcoin Protocol Development, so we had these topics:

Week 1: Welcome to Bitcoin Protocol Development

Week 2: Segwit

Week 3: Mining & Network Block Propagation

Week 4: P2P

Week 5: Script & Wallets

Every week, we would be matched with a random peer and assigned a particular question on the topic we had that week. We have to work with our partners to brainstorm answers to that question. Then every week we will meet in a breakout room and discuss our findings with everyone else. It was so much fun learning and sharing our research with our peers.

📚 Resources

PRDV151: Bitcoin for Everybody
CS120: Bitcoin for Developers I
Grokking Bitcoin by Kalle Rosenbaum
Bitcoin and Cryptocurrency Technologies.
Mastering Bitcoin for more technical insights.
Onboarding to Bitcoin Core by Amiti Uttarwar
Bitcoin Whitepaper (must read !!)
Learning-Bitcoin-from-the-Command-Line

After you have a good understanding of the fundamentals you can also dive into https://chaincode.gitbook.io/seminars/ to get into the depth of each concept.

📝 Takeaways

Prior to this Summer Of Bitcoin, I had very little experience with cryptocurrency, blockchain, and also open-source. It was not just a bitcoin learning experience but we had career-building, resume, and interview prep sessions too. I also kept a talk on Bitcoin Script. To make things even more amazing we also got some swags !! T-shirt and a Hardware Crypto Wallet.

But the most important takeaway would be the new connections I made, I made a lot of new friends who are really amazing at what they do, and I am still working with my mentor 0xB10C and I will be joining back this summer as a student coach for the next cohort.

Final Notes

Since I was a part of the first cohort of Summer Of Bitcoin, the procedure was a bit different. This time they have made some updates to their application procedure and have also introduced Designer Track. You will find the new timeline on their website. To keep getting updates follow @summerofbitcoin.

I would love to extend my gratitude to Adi Shankara, Caralie Chrisco, and Adam Jonas for giving us this amazing opportunity and to my mentor 0xB10C for always helping me with my doubts. 0xB10C is very active in the Bitcoin Development Community you should definitely check out his work at b10c.me and follow him @0xB10C.

For those who are looking forward to applying for the next cohort of Summer of Bitcoin, all the very best. If you have any doubts or questions feel free to contact me, my DMs are always open.

(Source: https://www.summerofbitcoin.org/)

I was recently invited to a podcast where I shared my Summer of Bitcoin experience and advice for the application rounds. We also discussed the future of Bitcoin and job opportunities in this space. You can check out the podcast as well 👇

Hello World! in Rescript-React

Arnab Sen — Fri, 05 Mar 2021 04:50:46 +0000

Hello World! in Rescript React

Talking a bit about Rescript, it is a new language that combines the best parts of JS and OCaml. It has a compiler that compiles to JS and leverages OCaml’s static and sound type system. Many people prefer this approach to TypeScript. The compiler was originally named BuckleScript but in 2020, it was rebranded to ReScript because of its Reason-like syntax.

Rescript docs are well written, organized, and maintained actively. Link to their documentation.

If you have a basic knowledge of Rescript syntax and React, then we are good to go.

Setting up the project

Start with the command npm init -y. If you simply do npm init it will ask you to fill in a couple of data, if you want them to be default values add the -y tag.

$ npm init -y

Now, let's install the ReScript compiler locally. The reason we are installing it with --save-dev is because we need it as a dev dependency.The Rescript compiler will compile the *.res files to JS files. This is actually called source to source compilation.

$ npm install --save-dev bs-platform

Before, moving on to the configuration part, let's install the rescript-react plugin.

$ npm install @rescript/react --save

Now, we have to set up a bsconfig.json file. Create this file and then copy and paste the following into it:

bsconfig.json

{
    "$schema": "https://raw.githubusercontent.com/rescript-lang/rescript-compiler/master/docs/docson/build-schema.json",
    "name": "project-name",
    "sources": [
        {
        "dir": "src",
        "subdirs": true
        }
    ],
    "package-specs": [
        {
        "module": "es6",
        "in-source": true
        }
    ],
    "suffix": ".bs.js",
    "namespace": true,
    "bs-dependencies": ["@rescript/react"],
    "ppx-flags": [],
    "reason": {"react-jsx": 3},
    "warnings": {
        "error": "+101"
      },
    "refmt": 3
}

Let's understand the config for a moment, going line by line:

$schema: Editors like VSCode have a feature of schema autocompletion, and to get that we need to specify the schema.
name: It is the name of the library or the main project name in simple words.

Note: the bsconfig.json name should be the same as the package.json name, to avoid confusing corner-cases. However, this means that you can't use a camelCased names such as MyProject, since package.json and npm forbid you to do so (some file systems are case-insensitive).

sources: We have to specify where the source files will reside. Usually, it is the /src directory.
suffix: The file extensions after compilation by the Rescript compiler, can be .js or .bs.js. ( Although, the latter is always preferred )
bs-dependencies: The list of Rescript dependencies, and since we are using the rescript-react we have to specify that here. There is also bs-dev-dependencies. These are very similar to how we mention node dependencies in package.json.
reason: Since we will be using React-JSX, we have to specify {"react-jsx": 3}.

For normal Rescript projects, the bs-dependencies will have [] and reason will be {}. To learn more about the config refer here

To compile the source files we have to add two scripts in our package.json.

package.json

"scripts": {
    "clean": "bsb -clean-world",
    "start": "bsb -make-world -w",
}

start: will compile the res files and will run in watch mode i.e will continuously look for changes and compile it. It is always advised to run the start script and then code the Res files, cause the intelligent rescript compilation helps a lot and makes the coding easier for beginners to the language.
clear: will clean/remove the previously compiled *.bs.jsfiles.

Now, install react and react-dom packages.

$ npm install react react-dom --save

Finally, we will use snowpack to build and bundle the project during development. In snowpack each file needs to be built only once and then is cached forever. When a file changes, Snowpack rebuilds that single file. There’s no time wasted re-bundling every change, just instant updates in the browser (made even faster via Hot-Module Replacement (HMR)). It is really fast. Again we will install snowpack as a dev dependency:

$ npm install --save-dev snowpack

Make a snowpack.config.js file and copy-paste this. Since we are not using any kind of plugins there is nothing to specify over here.

snowpack.config.j

// Example: snowpack.config.js
// The added "@type" comment will enable TypeScript type information via VSCode, etc.

/** @type {import("snowpack").SnowpackUserConfig } */
module.exports = {
    plugins: [
        /* ... */
    ],
};

Now, we have to add a couple of npm scripts for snowpack to start the server and build the project.

package.json

"scripts": {
    "dev": "snowpack dev",
    "build": "snowpack build"
}

At, this point we are done setting up dependencies, and your final package.json should look like this:

package.json

{
  "name": "react-res-test",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "clean": "bsb -clean-world",
    "start": "bsb -make-world -w",
    "dev": "snowpack dev",
    "build": "snowpack build"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "devDependencies": {
    "bs-platform": "^9.0.1",
    "snowpack": "^3.0.13"
  },
  "dependencies": {
    "@rescript/react": "^0.10.1",
    "react": "^17.0.1",
    "react-dom": "^17.0.1"
  }
}

Start writing your Hello World Code!!

First start off with a simple index.html in your root directory. Paste this basic template. One important thing is that there should be a <div> with id="root" inside which the React components will be rendered. This is how React works, if you already have some experience then you would know. Also, we have to include the compiled index.bs.js file. You can simply copy-paste this HTML. If you will use index.css then create a file and add its relative path.

index.html

<!DOCTYPE html>

<html lang="en">
  <head>
    <meta charset="utf-8" />
    <title>ReScript React</title>
    <link rel="stylesheet" href="index.css" />
  </head>
  <body>
    <div id="root"></div>
    <script type="module" src="src/index.bs.js"></script>
  </body>
</html>

Now create a folder src and create a new file src/index.res. Inside the index.res file just copy-paste this for the time being. We will dive into the code later on. Currently, let's test if all the setup we did was a success.

switch ReactDOM.querySelector("#root") {
| Some(root) => ReactDOM.render(<div> {React.string("Hello World!")} </div>, root)
| None => Js.log("Error: could not find react element")
}

NOTE: Here we are querying for #root in the DOM(index.html) i.e looking for an element with id="root". If in the index.html you have created some other id let's say xyz then over here you have to replace #root with #xyz.

Now, in the terminal run the command to compile the index.res file.

$ npm run start

Once you run this, a new file index.bs.js will be created. ( This is why we included the index.bs.js file in <script> tag in index.html ).

To build the react app run this is another terminal window.

$ npm run dev

This will start a snowpack server at localhost:8080 with hot reload. Visit the URL in your browser and if you see Hello World! Congratulations!! You are now ready to create your rescript-react project.

NOTE: Don't keep your index.css file empty if you have linked it to the index.html.

Final folder structure will be like this:

├── bsconfig.json
├── index.css
├── index.html
├── package.json
├── package-lock.json
├── snowpack.config.js
└── src
    ├── index.bs.js
    └── index.res

Was it a long process? Yeah, kind of. But every time you won't have to go through all. Here I have already created a repo with all these steps. Simply clone it and get started:

DEV Community: Arnab Sen

Row, Row, Raft Your Nodes: A Guide to Consensus in Distributed Systems

Introduction

So, what is Consensus

History of Consensus Protocols

Raft Algorithm

What are we dealing with?

Who will instruct the nodes on what to do?

Server States

Leader Election

How does the normal operation happen?

Understanding MapReduce

Background

What's MapReduce?

Map Function

Reduce Function

Stages of MapReduce

Splitting the Input

Building the army

Map Stage

Reduce Phase

Last stage

Partitioning Function

Example

A beginner-friendly introduction to Docker and Containers

What is a Container?

Virtual Machine VS Containers

What is Docker?

Why should we at all bother?

Practical 1

Run a container

Stop a container

Debugging container

Where are all these images stored?

Docker Images and Docker Containers

How to create Docker Image?

Practical 2

Creating flask application

Building Dockerfile

FROM python:3.6.1-alpine

RUN pip install flask

COPY app.py /app.py

Running the container

Docker Compose

Conclusion

Quick HTML tips: Enhancing UX and Accessibility with `enterkeyhint`.

What is the enterkeyhint attribute?

How does enterkeyhint work?

Example usage

Browser support

Conclusion

Breaking Captchas with Golang: Leveraging the Power of 2captcha

Understanding the Role of Captchas in Digital Security

How did Captchas even come into the picture?

What are the different types of captchas?

What is 2Captcha?

Building the Golang CLI tool

1. Creating our 2Captcha account

2. Initialize our Golang Project

3. Installing Required Modules

4. Building the Captcha Solver

Parsing Input Arguments

Loading the Captcha Image

Initializing the 2Captcha Client

Solving the Captcha

Outputting the Solved Captcha Text

Conclusion

How to debug buggy code? Exploring the debugger GDB

Introduction

What is GDB?

Installing GDB

Linux

Windows

Starting GDB

Breakpoint

Symbol Table

Working with Variables

Functions

Program Stack

Cheatsheet

`FROM python:3.6.1-alpine`

`RUN pip install flask`

`COPY app.py /app.py`

What is the `enterkeyhint` attribute?

How does `enterkeyhint` work?