DEV Community: Vignesh Muthukumaran

Bloom Filters and Cuckoo Filters

Vignesh Muthukumaran — Mon, 30 Jun 2025 18:37:52 +0000

Probabilistic data structures are essential tools for efficiently answering questions like “Is this element in my set?” when working with large-scale data. Two of the most popular structures for fast set membership queries, with tunable tradeoffs between space and error rate, are Bloom Filters and Cuckoo Filters.

This article explains both data structures conceptually, and compares their strengths and weaknesses.

Where Are Bloom Filters and Cuckoo Filters Used?

Both Bloom Filters and Cuckoo Filters are widely used in real-world systems where memory efficiency and fast lookups are essential and occasional false positives are tolerable. Some common applications include:

Databases and Caches: Used to quickly check if an item might exist before performing expensive disk or database lookups (e.g., Bigtable, HBase, Cassandra).
Network Systems: Employed in web proxies and routers to filter URLs or packets.
Distributed Systems: Used in distributed hash tables (DHTs), peer-to-peer networks, and blockchain systems to reduce unnecessary data transfers.
Security and Privacy: Utilized in password breach detection, malware detection, and privacy-preserving systems.
Web Analytics and Ad Tech: For deduplication, click fraud detection, and audience segmentation.

Bloom Filters

A Bloom Filter is a space-efficient, probabilistic data structure used to test whether an element is a member of a set. False positives are possible (an element might appear to be in the set when it isn't), but false negatives are not (an element that is in the set will never be missed).

How It Works

Allocate a bit array of size m, all initialized to 0.
Use k independent hash functions.
To add an element, compute its k hashes and set the corresponding bits in the array.
To query membership, compute the k hashes and check those bits. If any bit is 0, the element is definitely not present. If all are 1, it might be present (with a certain false positive probability).

Pros and Cons

Pros: Very space-efficient, simple to implement, fast insert and lookup.
Cons: Cannot delete elements (without counting), false positives possible, no information about element multiplicity.

Cuckoo Filters

A Cuckoo Filter is another probabilistic data structure for set membership tests, inspired by Cuckoo Hashing. It supports efficient additions, deletions, and queries, with lower false positive rates and often better performance than Bloom Filters for some applications.

How It Works

Divide the filter into buckets, each holding a small number of fingerprints (compact representations of set elements).
To add an element, compute two possible bucket locations from its fingerprint.
If either bucket has space, insert the fingerprint.
If both are full, randomly evict one fingerprint ("cuckoo") and try to re-insert it in its alternate location, repeating as necessary.
To query, check if the element’s fingerprint is present in either bucket.

Pros and Cons

Pros: Supports deletion, often lower false positive rates, similar or better space efficiency vs. Bloom Filters, good cache performance.
Cons: More complex, insertions can fail if the filter is too full, slightly higher per-operation overhead.

Comparison Table

Feature	Bloom Filter	Cuckoo Filter
Membership Query	Yes	Yes
Deletion	No (unless counting)	Yes
False Positives	Yes	Yes
False Negatives	No	No
Space Efficiency	High	Very High
Complexity	Simple	Moderate

Conclusion

Both Bloom Filters and Cuckoo Filters are invaluable for high-performance, space-efficient set membership queries where occasional false positives are acceptable. The right choice depends on factors like need for deletions, expected query volume, and tolerance for complexity.

Further Reading:

Building a RAG application using LangChain and LangSmith

Vignesh Muthukumaran — Sun, 21 Jul 2024 15:30:40 +0000

Why RAG?

LLMs have come a long way since their inception. However, one of the main problems with LLMs is that they are trained on publically available data or generic training data. This means they are not able to perform well in domain-specific scenarios. To overcome this we would either need to retrain the model with Domain-specific data which is costly or use something like RAG.

Another important need is that LLMs are known to hallucinate(generate a response that seems true but is not available in training data/factually incorrect). One of the ways to overcome this is by grounding the response with relevant reliable information.

What is RAG?

RAG stands for Retrieval Augmented Generation. The basic principle is that when we query an LLM, we pass in relevant context with the query so that the LLM can easily respond with relevant information. We are able to do this now because the context windows which used to 2048/4096 tokens have now come a long way to around 1 Million for newer models. There are 3 basic steps in RAG,

Indexing
Retrieval
Generation

We will take a simple use case and see LLMs in action. I am taking a blog post I wrote earlier using it for context, and asking a relevant question for the same. Here's a quick diagram explaining the flow.

We will use LangChain, HuggingFace(Free API request), and LangSmith(for tracing) to achieve this. First, install the required modules.



! pip install langchain_community tiktoken langchainhub chromadb langchain langchain_huggingface

Create a .env file and add the API keys used in this article.



LANGCHAIN_API_KEY='<langchain-key>'
HF_TOKEN='<hf-key>'

Initialize a few environment variables, which can be set in the .env file.



import os
os.environ['LANGCHAIN_TRACING_V2'] = 'true'
os.environ['LANGCHAIN_ENDPOINT'] = 'https://api.smith.langchain.com'

Indexing

The first step in RAG is indexing. We basically need to convert the document into a vector representation called embeddings. The reason behind this is that it's easier to find relevance between similar pieces of text when they are in vector format. There are a few steps involved in Indexing the data.

Loading
Splitting
Embedding

Loading

LangChain offers a multitude of ways to load data. We will use a simple WebBaseLoader to load the article into memory.



# Load blog
import bs4
from langchain_community.document_loaders import WebBaseLoader
loader = WebBaseLoader(
    web_paths=("https://vignesh.page/posts/kafka/",),
    bs_kwargs=dict(
        # parse_only=bs4.SoupStrainer(
        #     class_=("main-content")
        # )
    ),
)
blog_docs = loader.load()
blog_docs

Splitting

We need to split the data for Embedding because there is a size limitation to how much data can be embedded in one go. We are using the RecursiveCharacterTextSplitter for our example.



# Split
from langchain.text_splitter import RecursiveCharacterTextSplitter
text_splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(
    chunk_size=300,
    chunk_overlap=50)

# Make splits
splits = text_splitter.split_documents(blog_docs)

Indexing

The last step is to index the data into a Vector Store. For this example, we will use an in-memory instance of Chroma. We are using the HuggingFaceEmbeddings to embed the data. We set the vector store as a retriever with k = 1 (we are using k nearest neighbors algo to find relevant documents, so 1 returns only relevant documents).



# Index
from langchain_huggingface import HuggingFaceEmbeddings
from langchain_community.vector stores import Chroma
vector store = Chroma.from_documents(documents=splits,
                                    embedding=HuggingFaceEmbeddings())

retriever = vectorstore.as_retriever(search_kwargs={"k": 1})

Retrieval

The next step is quite straightforward, we just need to retrieve relevant document chunks whenever we get a query. My query "What is Kafka?" returned 4 documents with relevant data which will be passed as context to the LLM.



docs = retriever.get_relevant_documents("What is Kafka?")
len(docs)

We can see the LangSmith trace letting us know what was retrieved.

Generation

Now, all that is left is to pass the relevant documents to the LLM to get our response to the query.

We are using the Falcon-7B model from HuggingFace in this example, feel free to swap out with any LLM of your choice.



# LLM
from langchain_huggingface import HuggingFaceEndpoint, ChatHuggingFace

llm = HuggingFaceEndpoint(
    repo_id="tiiuae/falcon-7b",
    task="text-generation",
    max_new_tokens=512,
    do_sample=False,
    repetition_penalty=1.03,
)

Next, we create a prompt template to pass the query and the context.



from langchain.prompts import ChatPromptTemplate

# Prompt
template = """Answer the question based only on the following context:
{context}

Question: {question}
"""

prompt = ChatPromptTemplate.from_template(template)

We will use LangChain to chain these together and create a simple chain. Then, we invoke the chain.



# Chain
chain = prompt | llm

# Run
chain.invoke({"context":docs,"question":"What is Kafka?"})

After adding some syntactic sugar the same can be made easier to read. We will also add a default RAG prompt that is available in the LangChain hub.



from langchain import hub
prompt_hub_rag = hub.pull("rlm/rag-prompt")

rag_chain = (
    {"context": retriever, "question": RunnablePassthrough()}
    | prompt_hub_rag
    | llm
    | StrOutputParser()
)

rag_chain.invoke("What is Kafka?")

We can see the trace of the execution of this chain in LangSmith as well.

The code is available at my github. Please do give it a go. All the resources used are under the free tier making it easily accessible.

Thus with a few lines of code, we are able to build a simple RAG application. We have a multitude of tools inside LangChain that can be a separate article.

Originally posted on vignesh.page.

If you found the article useful please share it. Hope this helps in starting you Gen AI application development journey.

tmux 101

Vignesh Muthukumaran — Wed, 17 Jul 2024 15:00:45 +0000

tmux is a terminal multiplexer. We can have multiple terminals inside a single terminal, which is especially useful when we ssh into a remote machine.

tmux has 3 levels of hierarchy,

Sessions - To have completely different work environments for different concerns
Windows - To have multiple tabs for what that window represents, basically one for monitoring logs and performance and another to execute the commands required.
Panes - To have different panes inside a window, like one running top, and another running a couple of tail commands on log files.

To start using tmux, we just need to run the command tmux.

One key concept in tmux is the concept of prefix key. After pressing the prefix key, tmux enters command mode, similar to vim's insert, command, and view modes. The prefix key for tmux by default is Ctrl + B.

Basic commands

Start a new session

tmux

Detach from a session

Ctrl+B d

Attach to the last accessed session

tmux attach

Exit and quit tmux

Ctrl+B &

Session management

Create a named session

tmux new -s <session_name>

Attach to a session

tmux attach -t <session_name>

tmux a -t <session_name>

Switch sessions

tmux switch -t <session_name>

List sessions

tmux ls

Kill sessions

tmux kill-session -t <session_name>

Window management

Key	Operation
Ctrl+B C	Create new window
Ctrl+B N	Move to next window
Ctrl+B P	Move to previous window
Ctrl+B L	Move to last window
Ctrl+B 0-9	Move to window by index number

Pane management

Key	Operation
Ctrl+B %	Vertical split (panes side by side)
Ctrl+B "	Horizontal split (one pane below the other)
Ctrl+B O	Move to other pane
Ctrl+B !	Remove all panes but the current one from the window
Ctrl+B Q	Display window index numbers
Ctrl+B Ctrl-Up/Down	Resize current pane (due north/south)
Ctrl+B Ctrl-Left/Right	Resize current pane (due west/east)
Ctrl+B W	Open a panel to navigate across windows in multiple sessions

Command mode

Some other key points are we can enter command mode by

Ctrl + B :

We can allow access to a mouse by

set -g mouse

Tmux config

We have a tmux config file that is a dot config file. We can create it using the following command.

vi $HOME/.tmux.conf

We can add the above command to allow mouse usage into the tmux conf file.

Tmux References

There is a lot more to tmux than this, which we can check out from the tmux manual.

Originally posted on vignesh.page.

Hope tmux improves your daily workflow as it did mine. Comment out any cool features or tmux plugins that you find useful.

Setup a k3s Cluster at home quickly

Vignesh Muthukumaran — Sun, 14 Jul 2024 13:54:57 +0000

I was setting up a new Fedora Server VM over the weekend and wanted to use it as a kubernetes cluster to run my containerized workloads. I usually install docker and docker-compose and get on with my work. But I wanted to set up a k8s cluster for fun. When looking up online I found a quick and easy way to set it up.

Why k3s over k8s?

Rather than going with a full-blown k8s cluster with multiple VMs, I went with a single node k3s setup. k3s is a lightweight k8s, it has a smaller footprint and is great for a home lab.

I would encourage you to check out their docs for more info based on your requirements, but the gist is as follows.

Requirements

The minimum hardware requirement is

Spec	Min	Recommended
CPU	1 core	2 core
RAM	512 MB	1 GB

I am running this on a Fedora Server VM with 4 cores and 8 GB of RAM, so well above the recommended settings.

Installation

The command to install is,

curl -sfL https://get.k3s.io | sh -

It will ask for the sudo password. Have a look at the script before running it at Installation Script. Also, they have a lot of options to customize the components installed. Check out the full list at Config Options.

Verify the install

If everything goes well, you should be able to check the service through systemctl.

$ systemctl status k3s
● k3s.service - Lightweight Kubernetes
     Loaded: loaded (/etc/systemd/system/k3s.service; enabled; preset: disabled)
    Drop-In: /usr/lib/systemd/system/service.d
             └─10-timeout-abort.conf
     Active: active (running) since Sun 2024-07-14 18:47:30 IST; 21min ago
       Docs: https://k3s.io

Check the default k3s objects

Running the following command gives you all the k3s objects deployed.

$ sudo kubectl get all -n kube-system
NAME                                         READY   STATUS    RESTARTS   AGE
pod/coredns-6799fbcd5-svqqf                  1/1     Running   0          26m
pod/local-path-provisioner-6f5d79df6-d8h5g   1/1     Running   0          26m
pod/metrics-server-54fd9b65b-5xxq4           0/1     Running   0          26m

NAME                     TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)                  AGE
service/kube-dns         ClusterIP   10.43.0.10 <none>        53/UDP,53/TCP,9153/TCP   26m
service/metrics-server   ClusterIP   10.43.198.179 <none>        443/TCP                  26m

NAME                                     READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/coredns                  1/1     1            1           26m
deployment.apps/local-path-provisioner   1/1     1            1           26m
deployment.apps/metrics-server           0/1     1            0           26m

NAME                                               DESIRED   CURRENT   READY   AGE
replicaset.apps/coredns-6799fbcd5                  1         1         1       26m
replicaset.apps/local-path-provisioner-6f5d79df6   1         1         1       26m
replicaset.apps/metrics-server-54fd9b65b           1         1         0       26m

Configuring k3s

Create and edit the following file to configure the k3s setup.

sudo vi /etc/rancher/k3s/config.yaml

Check out all the available config at k3s Config.

Post, the creation/updation of the config file, restart using systemctl.

sudo systemctl restart k3s

Stopping k3s

Just use systemctl to stop the service.

sudo systemctl stop k3s

Unintalling k3s

Uninstalling is also a breeze with k3s, as it generates an uninstall script as soon as you install k3s at /usr/local/bin/k3s-uninstall.sh.

Just run the script to uninstall.

/usr/local/bin/k3s-uninstall.sh

Post uninstall and check the status with systemctl.

systemctl status k3s

Originally published at vignesh.page.
Please comment any workloads you have for a k3s cluster at home.

Improving throughput and latency using Java Virtual Threads in Spring

Vignesh Muthukumaran — Sun, 30 Jun 2024 17:01:29 +0000

One of the major changes that Java 21 brought about is Virtual threads. There is so much hype around it, but let's see a real-world example with some metrics. Spring introduced the ability to create GraalVM native images that use Spring Boot and Java 21's virtual threads(Project Loom).

We will look at virtual threads in detail and the many features that come with them in another article. We will focus on a simple project where we can enable the features of Java Virtual Threads and see the difference in performance gains.

Why virtual threads are faster than normal threads?

Normal threads in Java are tied to OS threads. So, there is a limitation to the actual number of threads it can create. Also, time is lost waiting for blocking I/O Calls. Normal Threads are called Platform Threads. They are an expensive resource that needs to be managed well.

Now, when it comes to virtual threads, they are lightweight constructs. Multiple virtual threads can be tied to a single platform thread which in turn gets tied to an OS thread.

The simple idea behind Virtual Threads

The Spring Project used for benchmarking

The project used in this article can be found here. It has a simple fetch API that returns the Thirukkural based on the ID sent.

Now, by default the http server in Tomcat can run many threads in parallel and this will not let us test out this feature easily. So let's throttle it to 10 max threads by adding the below property in the application.properties file. This will allow Tomcat to only use 10 threads at max.

server.tomcat.threads.max=10

We will mimic a blocking IO call using sleep. We will also log the current thread it's using.

Thread.sleep(1000);
log.info("Running on " + Thread.currentThread());

Benchmarking using hey

We will benchmark this using hey tool.

hey -n 200 -c 30 http://localhost:8080/thirukural/1

Summary:
 Total:        36.1759 secs
 Slowest:      8.0463 secs
 Fastest:      2.0080 secs
 Average:      5.6840 secs
 Requests/sec: 4.9757


Response time histogram:
 2.008 [1]     |
 2.612 [10]    |■■■
 3.216 [0]     |
 3.819 [0]     |
 4.423 [11]    |■■■
 5.027 [0]     |
 5.631 [0]     |
 6.235 [156]   |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
 6.839 [0]     |
 7.442 [0]     |
 8.046 [2]     |■


Latency distribution:
 10% in 4.0305 secs
 25% in 6.0212 secs
 50% in 6.0263 secs
 75% in 6.0340 secs
 90% in 6.0376 secs
 95% in 6.0387 secs
 99% in 8.0463 secs

Details (average, fastest, slowest):
 DNS+dialup:   0.0007 secs, 2.0080 secs, 8.0463 secs
 DNS-lookup:   0.0005 secs, 0.0000 secs, 0.0041 secs
 req write:    0.0000 secs, 0.0000 secs, 0.0003 secs
 resp wait:    5.6832 secs, 2.0079 secs, 8.0419 secs
 resp read:    0.0001 secs, 0.0000 secs, 0.0005 secs

Status code distribution:
 [200] 180 responses

A few log entries are listed below to see that these requests are using Platform Threads.

2024-06-30T22:07:29.659+05:30  INFO 2120 --- [thriukural] [nio-8080-exec-7] p.v.thriukural.web.ThirukuralController  : Running on Thread[#45,http-nio-8080-exec-7,5,main]
2024-06-30T22:07:29.659+05:30  INFO 2120 --- [thriukural] [io-8080-exec-10] p.v.thriukural.web.ThirukuralController  : Running on Thread[#48,http-nio-8080-exec-10,5,main]
2024-06-30T22:07:29.659+05:30  INFO 2120 --- [thriukural] [nio-8080-exec-2] p.v.thriukural.web.ThirukuralController  : Running on Thread[#40,http-nio-8080-exec-2,5,main]

Now, to enable the application to use virtual threads, we just add the below property in application.properties.

spring.threads.virtual.enabled=true

Let's run and see the same benchmark results again.

hey -n 200 -c 30 http://localhost:8080/thirukural/1

Summary:
 Total:        12.2478 secs
 Slowest:      2.1747 secs
 Fastest:      2.0086 secs
 Average:      2.0410 secs
 Requests/sec: 14.6965


Response time histogram:
 2.009 [1]     |
 2.025 [149]   |■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■■
 2.042 [0]     |
 2.058 [0]     |
 2.075 [0]     |
 2.092 [0]     |
 2.108 [0]     |
 2.125 [0]     |
 2.141 [0]     |
 2.158 [0]     |
 2.175 [30]    |■■■■■■■■


Latency distribution:
 10% in 2.0115 secs
 25% in 2.0129 secs
 50% in 2.0158 secs
 75% in 2.0188 secs
 90% in 2.1724 secs
 95% in 2.1739 secs
 99% in 2.1747 secs

Details (average, fastest, slowest):
 DNS+dialup:   0.0015 secs, 2.0086 secs, 2.1747 secs
 DNS-lookup:   0.0014 secs, 0.0000 secs, 0.0093 secs
 req write:    0.0000 secs, 0.0000 secs, 0.0002 secs
 resp wait:    2.0392 secs, 2.0085 secs, 2.1656 secs
 resp read:    0.0003 secs, 0.0000 secs, 0.0026 secs

Status code distribution:
 [200] 180 responses

A few log entries to see them running in virtual threads.

2024-06-30T22:12:38.485+05:30  INFO 17700 --- [thriukural] [mcat-handler-48] p.v.thriukural.web.ThirukuralController  : Running on VirtualThread[#103,tomcat-handler-48]/runnable@ForkJoinPool-1-worker-6
2024-06-30T22:12:38.485+05:30  INFO 17700 --- [thriukural] [mcat-handler-66] p.v.thriukural.web.ThirukuralController  : Running on VirtualThread[#121,tomcat-handler-66]/runnable@ForkJoinPool-1-worker-12
2024-06-30T22:12:38.485+05:30  INFO 17700 --- [thriukural] [mcat-handler-69] p.v.thriukural.web.ThirukuralController  : Running on VirtualThread[#124,tomcat-handler-69]/runnable@ForkJoinPool-1-worker-7

Benchmarking results

As evident from the results, we see an increased throughput and better latency as well once virtual threads are enabled.

Parameter	Without Virtual Threads	With Virtual Threads
Requests/sec	4.9757	14.6965
99% Latency	8.0463 sec	2.1747 sec

Summary

A simple parameter change has enabled us with a lot of improved performance and throughput. We will further explore in future articles some of the considerations that need to be taken before using virtual threads.

Originally published at vignesh.page.
Please let me know if any improvements you have experienced using Java Virtual Threads

React refersher

Vignesh Muthukumaran — Sat, 20 Jan 2024 15:57:20 +0000

React

I recently decided to refresh my React skills, since it has been a very long time since I did something with any frontend technologies. React is one of the most used front-end JS libraries that is used to build user interfaces for web apps and native apps(React Native).

How does React work?

React maintains a Virtual DOM, similar to the actual DOM. DOM is a Document Object Model, which is essentially a tree. React works fast by updating the virtual DOM first which is cheaper compared to updating the actual DOM. Once it does that, it effectively reconstructs the actual DOM to push only the changes.

What is JSX?

React uses JSX which is nothing but syntactic sugar. Everything written in JSX is converted to JS.

const para = <p>Hi World</p>

gets converted to

const tag = React.createElement("p", {}, "Hi World")

One thing we observe with JSX that is different from HTML is that we can't use class, instead we use className. This is because class is a reserved keyword. There are a few subtleties like this in JSX which we need to be careful about.

What are components?

React is a component-based front-end library. This allows us to essentially build our UI as small reusable components. Every React app is a tree of React Components.

Types of Components

React has 2 types of components.

Class-based
Functional

Class Based Components

Let's see how class-based components look like first

import React, {Component} from 'react'
class Test extends Component {
    render() {
        return <div>Hi There!</div>
    }
}

As we can see, we create a separate file for a component. Then we need to import React and Component. We create a class called Test(name of our component). Then we use the render method to define what to display. But this is history, it's legacy code now as functional components have started to take precedence.

Functional Components

import React from 'react'

const Test = () => {
    return <div>Hi There!</div>
}

Here too, we import a React library, but this can be skipped in the latest versions. Then we just need to create a function(can be named or arrow). That's it.

Setting up a project

We can either

setup a project manually by using Webpack and Babel
using create-react-app command

We will go with create-react-app as it's far easier. But if finer control is required better to go with the first approach.

What is create-react-app?

create-react-app is a tool to generate the starter project. It's officially supported by the React team. To create a project just create a project directory and run the below command.

npx create-react-app ./

This creates our React Project. We can start the project by running the below command.

npm start

The project will be hosted on port 3000 on localhost.

React Project structure

Let's look at the directories the create-react-app had created to understand what files are required for React to run.

The first file we need to look at is package.json. Here we will find a list of dependencies for the project. We will find the react libraries as well here.

"dependencies": {
  "@testing-library/jest-dom": "^5.17.0",
  "@testing-library/react": "^13.4.0",
  "@testing-library/user-event": "^13.5.0",
  "react": "^18.2.0",
  "react-dom": "^18.2.0",
  "react-scripts": "5.0.1",
  "web-vitals": "^2.1.4"
}

Next, in the public directory, we find the index.html. The main thing we need to look at is the root div. React injects the entire app inside the root tag.

<div id="root"></div>

This is done in the index.js file found under the src directory. Below is a sample of the file.

import React from "react";
import ReactDOM  from "react-dom";

import App from './App';

ReactDOM.render(<App/>, document.getElementById("root"));

Here the ReactDOM is imported with React library. We use ReactDOM only once in the entire app. We use ReactDOM.render with the component we want to inject and the target DOM element. React injects the App component into the root element. This is the basis of how React works.

The next file is App.js inside the src directory. It's the place where we are defining what to render.

React in action

The main feature of React is the power to write HTML inside Javascript and vice versa. Inside a pair of curly braces, we can put and valid Javascript code.

For instance, we can start with a clean App.js. Remember App.js is a component.

import './App.css';

const App = () => {
  return (
    <div className="App">
      <h1>Hi There</h1>
    </div>
  );
}

export default App;

This renders a simple Hi There on the page. Let's first understand what is happening here.

We are importing a css file for styling.
We are defining a functional component called App using the arrow function.
We export the App function, so it can be used wherever the file gets imported.

Now, we can start playing with it. We can define a variable and display the same to the user.

const App = () => {
  const name = "John Doe";
  return (
    <div className="App">
      <h1>Hi {name}</h1>
    </div>
  );
}

React renders us Hi John Doe. It evaluates the name variable to get the actual value and shows us the result.

We can make the same conditional based on a parameter.

const App = () => {
  const name = "John Doe";
  const isNameEnabled = false;
  return (
    <div className="App">
      <h1>Hi {isNameEnanled ? name : "there"}</h1>
    </div>
  );
}

React renders Hi there. Here we have used a ternary operator to display the name conditonally. Similarly, we can perform calculations, computations, etc.

Now, let's assume we need to render multiple elements, we usually do that with divs. But a cleaner approach to stop the page from having multiple inner divs is using fragments.

import React from 'react';

const MultipleElements = () => (
  <>
    <h1>Hi</h1>
    <p>There</p>
  </>
);

Note this is a newer React feature, so legacy apps might not support it. If we don't nest them, we run into the error "Adjacent JSX elements must be wrapped in an enclosing tag". So, be careful with this.

Moving forward, we will start creating components.

Creating and importing components

One of the selling points of React is that we can create smaller components and embed them into parent components. This way we can reuse the components in multiple places.

We will look at an example below.

const Person = () => {
  return (
    <>
      <h1>First Name: John</h1>
      <h1>Last Name: Doe</h1>
      <h1>Age: Unknown</h1>
    </>
  );
}

const App = () => {
  return (
    <div className="App">
      <Person />
    </div>
  )
}

This will render us with text defined in the Person Component. This is how we import one Component into another.

What are props?

To pass data to our Components we use props. Props are arguments passed into React Components. We pass them as HTML attributes. We use props to pass data from one component to another (Parent to Child). Data passed in props is read-only and cannot be modified by the received component.

We will look at the same example above with props.

const Person = (props) => {
  return (
    <>
      <h1>First Name: {props.firstName}</h1>
      <h1>Last Name: {props.lastName}</h1>
      <h1>Age: {props.age}</h1>
    </>
  );
}

const App = () => {
  return (
    <div className="App">
      <Person firstName='John' lastName='Doe' age={20}/>
      <Person firstName='Jane' lastName='Doe' age={25}/>
    </div>
  )
}

As we can see, in the Child component we get the props argument and use the attributes passed from the parent component.

What is state?

State is a JS object used by React to represent the component's state(maybe a counter instance). It is completely managed by the Component it's present in. Whenever the state changes, the component re-renders. The state object is where we store property values that belong to the component. State data can be modified by its component, but is private.

We will look at a simple counter using state.

import { useState } from 'react';
import './App.css';

const App = () => {
  const [counter, setCounter] = useState(0);
  return (
    <div className='App'>
      <button onClick={() => {setCounter((prevCounter) => prevCounter - 1)}}>-</button>
      <h1>{counter}</h1>
      <button onClick={() => {setCounter((prevCounter) => prevCounter + 1)}}>+</button>
    </div>
  );
}

export default App;

To use state we need to import the react hook called useState. There are a few hooks that React gives, it's better to know of them.

useState accepts an initial state and returns 2 values. They are the current state and a function to update the state. We should never update the state directly and always use this function.

So here we have defined the current state as 0, the current state is counter, and the setCounter is the function to update the state. This helps us keep track of the counter and as and when the counter gets updated the component gets re-rendered. If we don't use state and directly do the same, the component doesn't get re-rendered and the value remains the same.

Another important hook is useEffect. It's used to perform side effects on the components. For instance, if we want to show an alert every time the counter is updated, we can do something like this.

useEffect(() => {
  alert("Counter has been updated to " + counter);
}, [counter])

useEffect takes in 2 arguments. The function it has to call, and what it is dependent on.

If we don't pass the second argument, it gets triggered on every re-render. The below block will always make the counter to 0, whenever it changes. (Just for example)

useEffect(() => {
    setCounter(0);
  });

If we pass an empty array, it gets triggered only on the first render. The below block sets the counter to 100 on the first render.

useEffect(() => {
    setCounter(100);
  }, []);

If we pass the dependent element, it gets triggered when the value gets updated. This block just alerts the counter whenever the counter gets changed.

useEffect(() => {
  alert("Counter has been updated to " + counter);
}, [counter])

There are more hooks that React offers. It's a good idea to be aware of them.

In the next article, I write about React and will look at expanding further on it.

References

Sample project

Movie Search

Kafka, The What, Why and How?

Vignesh Muthukumaran — Fri, 17 Nov 2023 14:16:27 +0000

What is Kafka?

Created by LinkedIn, and was written in Java and Scala, Apache Kafka is a distributed event streaming platform that can scale massive pipelines of real-time data. So, what is Event Streaming? Event streaming is capturing real-time data from event sources(which can be anything from IOT devices, and mobiles to cloud services, software applications, etc); storing them reliably for retrieval; manipulating them if necessary, and routing them to different destinations if necessary. Kafka can do this at scale efficiently. Kakfa is not an in-memory DB like Redis or Memcached, rather it stores the data in the disk.

Why Kakfa?

The main reasons to use Kafka are,

High throughput - Capable of handling high velocity and volume.
High scalability - It can scale to thousands of brokers and scale up or down as required.
Low latency - Can achieve low latency with a cluster setup
High availability - Since we can have multiple clusters across servers, and geographies, it is extremely fault tolerant with very minimal risk of data loss

Just a side note, when high throughput is required we can go for Kafka. Though Kafka maintains ACID properties, it's not advised to be used as a DB. If we need to look at data, it's faster in a DB than in Kafka. Ksql is currently available which will allow us to query the message stream, to have continuously updated derived tables.

How does Kafka work?

Kafka is composed of multiple components. We will list them down first and understand everyone with a sample use case of a food delivery app.

Message

A message is the atomic unit of data in Kafka. It can be a JSON, integer, String, etc. Messages can have a key associated with them, which can be used to determine the destination partition.

For our example, it can be something like this for order info.

{
    "orderId": "1",
    "status": "Cooking",
    "food": [
        { "name": "Pizza",  "qty" : "1"},
        { "name": "Coke",  "qty" : "1"}
    ],
    "city": "Chennai"
}

Topics

Topics are logical partitions of events. We can have separate topics for different types of messages.

For our example, maybe we can have different topics for order status info, delivery partner location info, etc.

Brokers

Kafka instances that store and replicate events. We will try it out with 1 Kafka broker for our example.

Producer

A client app that puts events into Kafka. One more key thing the producer must do is to decide which partition it has to put the data in based on the key.

No key - If no key is specified, a random partition is chosen and tries to balance based on the total number of messages in the partitions
Key specified - When the key is specified, it uses consistent hashing to decide. So, the same key would go to the same partition as consistent hashing ensures the same hash is generated always.
Partition specified - We can hardcode the partition.
Custom - We can write rules as per our requirements.

For our example, we will use location(city) as the key to deciding the position.

Consumer

A client app that consumes the events from Kafka. Each time the consumer processes a record it updates the offset as well.

Partition

Partitions are meant to spread the topics into different buckets. This is similar to sharding in a DB. Instead of vertically scaling in a single partition for a single topic, we can scale horizontally to have more partitions. Note that partitions will not have the same data. Based on the key specified in the message it will choose a partition and write the data.

For our example, we can have 2 partitions for each topic.

Replication factor

The replication factor specifies the copies of partitions that should exist. It's specified at the topic level. So if we have 1 topic, 2 partitions, and 2 replication factors, we would end up with 4 partitions. Note that the replication factor should be less than equal to several brokers.

For our example, we can go with a replication factor of 2.

Offset

To keep track of the messages already processed, we have something called an offset. So, once a message in a partition has been consumed by a consumer, it increments the offset. If a consumer goes down, we can resume work from the same offset.

Zookeeper

Zookeeper is an extra service to track the Kafka brokers, storing offset for all the partitions, etc.

Consumer Group

Consumer groups are created to help attain higher consumption rates if multiple consumer groups are consuming from the same topic.

One thing we need to be clear is that one consumer can consume from multiple partitions but multiple consumers can consume from a single partition, within a consumer group.

Also, consumer groups are self-balancing. Let's understand this with an example.

Exact match - 4 partitions, 4 consumers in a consumer group result in 1 consumer per partition.
Less consumers - 4 partitions, 2 consumers in a consumer group results in 2 partitions per consumer.
More consumers - 4 partitions, 5 consumers in a consumer group results in 1 consumer per partition and 1 idle consumer.

Based on this we might wonder about MQ and Pub-Sub.

MQ - If we want Kafka to act as a MQ, where each event is processed exactly once, we can have the same number of consumers and partitions or a lesser number of consumers than partitions.
Pub-Sub- We can have multiple consumer groups reading from the same topic/partitions.

In our example, we can have a single consumer group.

Example program

We can try out the sample code from Github if you are interested. Link

To run Kafka and Zookeeper locally, we presume docker and docker-compose are installed.

docker-compose -f docker-compose.yaml up

Please refer to this Github project for more alternative setups available.

The sample project has 3 files

admin.js is used to create the topics. We can create as many as required.

node admin.js

producer.js opens up a cli interface where we can give the order ID, status, and location.

node producer.js

consumer.js opens up a consumer and displays the message as and when received. We can mention the group which it should be part of. Since we have 2 partitions, it would be ideal if we create 2 consumers with the same consumer group. This will allow us to see all messages sent with location as Bangalore goes to one partition and the rest all to another partition.

node consumer.js group1

There are further advanced topics to explore such as Ksql, Kafka streams, etc which we can explore in a future article.

How to solve any dynamic programming question?

Vignesh Muthukumaran — Sun, 12 Nov 2023 09:52:38 +0000

So, let's start with a quote from Confucius.

“Study the past if you would define the future.”
- Confucius

The quote let's us know all we need to know about dynamic programming. We need to remember the results computed for previous small problems to solve bigger newer problems.

There are 2 approaches to dynamic programing usually

Tabulation - Bottom Up
Memoization - Top Down

We can take a common problem "Fibonacci Series" to understand the approaches.

The first thing we can try to do is solve it by recursion. So, the recursion solution for Fibonacci is as follows

int fibonacci(int n){
    if (n <= 1) return n;
    return fibonacci(n - 1) + fibonacci(n - 2);
}

If we consider the above solution and compute the recursion tree we get the following tree for n = 5.

fib(5) => 5
1. fib(4) => 3
  1. fib(3) => 2
    1. fib(2) => 1
      1. fib(1) => 1
      2. fib(0) => 0
    2. fib(1) => 1
  2. fib(2) => 1
    1. fib(1) => 1
    2. fib(0) => 0
2. fib(3) => 2
  1. fib(2) => 1
    1. fib(1) => 1
    2. fib(0) => 0
  2. fib(1) => 1

Memoization

As we can see above, we have overlapping subproblems, so we don't need to solve these problems again and again. We can store these solutions, this is memoization. We have a single parameter, so we would need a 1D array.

So how do we apply memoization? We do it with 3 steps.

Declare the DP array.
Start adding the solution for computed sub problems into DP array.
Check if the solutions already exists and use it if it exists.

int fibonacci(int n, int[] dp/* Adding the DP array*/){
    if (n <= 1) return n;
    if (dp[n] != -1) return dp[n]; //Checking if solution exists and using it
    dp[n] = fibonacci(n - 1, dp) + fibonacci(n - 2, dp);//Updating the array if it doesn't exist
    return dp[n];
}

So, the time complexity would be O(n). This is because we don't call the function again for already computed solutions. But, the space complexity would be O(n) + O(n), one for the dp array and another for the extra stack space.

Tabulation

So, now let's try tabulation. As already highlighted above this is a bottom-up approach. For the current problem, let's try to rewrite it with tabulation.

int fibonacci(int n){
    int[] dp = new int[n + 1];
    Arrays.fill(dp, -1);
    dp[0] = 0; 
    dp[1] = 1;
    for(int i = 2; i <= n; i++){
        dp[i] = dp[i - 1] + dp[i - 2];
    }
    return dp[n];
}

We calculated the base cases first and started iterating and solving the bigger solutions with previously computed values.

The time complexity is the same, i.e. O(n). The space complexity comes down here because we are not using the stack trace which is O(n).

Space optimization on Tabulation

In the above tabulation, we can see that we don't need to maintain all the previous values. Just the last two solutions would suffice for this problem.

int fibonacci(int n){
    int a = 0, b = 1;
    for(int i = 2; i <= n; i++){
        int cur = a + b;
        a = b;
        b = cur;
    }
    return b; //since computing till n 
}

So, now we have reduced the space complexity to O(1), while the time complexity stays the same.

Now, we will move on to some more complex problems to apply the above approach in the coming articles.

Using Git 1

Vignesh Muthukumaran — Sat, 12 Aug 2023 16:02:46 +0000

Git

Git is a source control program. It can be used to,

Version control (maintain all version of a project)
Revert to older version if something breaks
Collaborate with people (allowing multiple people work on a single project)
Track all the changes done by multiple people

In this article, we just explore all the features we can use locally.

Using Git

Create a git repo

The way to create a git repo is to run the git init command in the project folder

git init

Git will create a folder called .git in the current working directory.

Start tracking

From now on, all the changes done in the folder will be tracked. Try adding a file and just check it out.

git status

This will show unstaged files i.e files not yet committed in red, committed files in green.

Commiting the changes

If we commit the changes, it will create a snapshot of the version. But before that the changes need to be added to staging.

git add .

To add the files to staging using the above command. Now git status will return the files in green. Now the changes can be committed.

git commit -m "Changes done"

This will now create a snapshot with the commit.

Tracking changes

To checkout the history of the changes, we can use git log.

git log

Reverting a commit

To revert to an older version, we choose the version needed from git log and use git reset to go back to previous commit. This will revert the changes and unstage the changes done as well.

git reset $prev_commit_hash

A git status should show us the changes done after the reverted hash in red.

If we use the flay --hard, it removes the files as well from the working directory, so use with caution if required.

git reset --hard $prev_commit_hash

Stash changes

Suppose we need to temporarily move the changes and get it to previous committed version, we can use git stash.

git stash

To bring back from stash, we can bring it back using git pop

git stash pop

We can stash and remove the changes as well.

git stash clear

To remove just the top of stash changes we can use git stash drop

git stash drop

Getting the changes back after a git reset --hard

There is another set of logs in git called reference logs. The command to access reference logs is reflog

git reflog

We can use this to checkout a branch from here to an older before a hard reset was done to get the changes.

In part 2, we will explore the feature we can use to collaborate.

VS Code Snippets

Vignesh Muthukumaran — Sun, 30 Jul 2023 11:11:48 +0000

VS Code snippets is an easy way to create a template for recurring code blocks or front matter for certain files. We will look at a small example here for a Zettelkasten header.

Here, we have a snippet that will be triggered by "---" followed by Ctrl + Space. Intellisense will also pick it up and suggest the same as well.

Name of the snippet: We have named it Test
Prefix: The trigger that would invoke the snippet. "---"
Body: The conent to be added.
Description: A small description of what the snippet is for.

"FrontMatter": {
    "prefix": "---",
    "body": [
        "---",
        "title: $1",
        "date: \"$CURRENT_YEAR-$CURRENT_MONTH-$CURRENT_DATE\"",
        "type: ${2|Daily,Literature,Permanent,Index|}",
        "description: $3",
        "tags: [$4]",
        "---"
    ],
    "description": "FrontMatter"
    }

Inside the body, we have a lot of control.

Parameters defined by $1, $2, etc, that can be cycled through quickly using tab.
Date or time that can be auto populated.
Set a list of values that can be quickly selected and restricted.

A small gif showing the snippet in action.

Reliable, Scalable and Maintainable Applications 1

Vignesh Muthukumaran — Mon, 16 Jan 2023 13:16:00 +0000

As I started to go through the book Designing Data-Intensive Applications, I thought I would consolidate my notes to provide a quick reference for my future self. In this article, I will consolidate my notes for the first chapter Reliable, Scalable, and Maintainable Applications.

An awesome quote at the start of the chapter is

The Internet was done so well that most people think of it as a natural resource like the Pacific Ocean, rather than something that was man-made. When was the last time a technology with a scale like that was so error-free? - Alan Kay, in an interview with Dr. Dobb’s Journal (2012)

The quote emphasizes the beauty of building Reliable, Scalable and Maintainable applications and what we should strive for as architects and developers.

Broadly applications can be classified into 2 types,

Compute intensive - Requiring a lot of computing resources
Data-intensive - Huge amount of data, complex data, and pace at which these data change is high

Most of today's applications are data-intensive. Typical Data Intensive application has the following components.

Store data (DBs)
Remember the results of expensive operations(Caches)
Allow users to search (Search Indexes)
Send messages to handle asynchronously (Stream Processing)
Process huge chunks of data at once (Batch Processing)

Major concerns when designing a software system are

Reliablilty - System should work correctly even in face of adversity(hardware/software faults, human error, etc)
Scalability - As system grows(in volume, traffic, etc), there should be ways to deal with that growth
Maintainability - Should be easy to maintain the system(make changes to fix bugs, handle more usecases, etc)

Reliability

For software, typical reliability expectations are,

Performs the function user expects
Tolerate user mistakes
Performance is good enough for the use cases, under expected load and data volume
Prevent unauthorized access and abuse

So, the expectation is for the system to work correctly when things go wrong(faults). So, we will be building fault-tolerant (resilient) systems.

Fault vs Failure - A fault is one/more components deviating from expected specifications, whereas a failure is when the system is not able to service the user request.

Counterintuitively we can increase faults, to know how well the system can handle the faults(Eg: Netflix Chaos Monkey). Also, in some cases, we need to stop the faults, like in cases of data breaches where there is no way to fix the fault or tolerate them if the data is breached.

Types of faults are,

Hardware faults
Software faults
Human errors

Hardware Faults

If any hardware components were prone to failure, the natural thing to do would be redundancy. In this approach, we may set up the drives in RAID config, with dual power supplies and hot-swappable CPUs. Though not completely foolproof these approaches can keep machines running uninterrupted for years.

Nowadays, with AWS EC2-like VMs running most of the newer applications, where these systems favor elasticity to single VM reliability, applications need to be able to handle entire machine losses as well.

We go to multi-machine setups which have few operational advantages as well over the traditional systems like rolling upgrades, zero downtime, quick scaling to load, etc.

Software errors

These are errors due to the bugs in the software. There is no quick solution to these errors. Things like thorough testing, process isolation, allowing processes to crash and restart, and monitoring help here. If a system is to provide some guarantee, it can self-check itself constantly and raise an alert in case of discrepancy.

Human errors

Humans are the most unreliable part of any software system. Some ways to keep these errors in check are

Limit chances to introduce these human errors
Decouple places where people make most mistakes (Ex: Test env and prod env)
Testing thoroughly
Allow quick recovery (Ex: Rollback config if any failures)
Detailed monitoring

Even in noncritical applications, the system needs to be reliable. There are cases where cost vs reliability sacrifice arises, but we should be careful of the choices.

Will continue in the next article as I don't want the article to be too long.

Go Scrapper API

Vignesh Muthukumaran — Sun, 03 Jul 2022 09:46:26 +0000

As I was exploring further about Go, I came across APIs in Go, which was very easy to build. So instead of going the traditional way and building a CRUD API, I decided to write a simple scrapper API. The API scraps Goodreads quotes page and returns a JSON of quotes with their authors.

Here I am using 2 modules,

Gin - For quickly building the API
Colly - For web scrapping.

I have built it and deployed it on Heroku.

Commands I ran from start to finish

To initiate the current folder/project as a module

go mod init vigneshm.me/go-quotes-api

This creates a file called go.mod. This has the name of the project and the version of Go it uses.

To import modules

go get github.com/gin-gonic/gin
go get github.com/gocolly/colly

This imports the modules and creates a go.sum file to add checksums for the modules it downloads. Also, the go.mod page gets filled with all the dependent modules info.

Next, I filled in the main.go with my code.

Now, for the part where I deploy the app in Heroku.

Install Heroku CLI

npm install -g heroku

Login to Heroku

heroku login -i

Initate a Git repo and commit the code

git init
git add .
git commit -m "initial commit"

Create a Procfile for Heroku

touch Procfile
echo "web: bin/go-quotes-api" > Procfile

This tells Heroku to run the command on startup.

Create Heroku App

heroku create go-quote-api
git push heroku master

This creates the Heroku app and pushes the code.

Heroku takes this code runs the build and exposes the app at https://app-name.herokuapp.com/

My app is available at https://go-quote-api.herokuapp.com/quotes/dumbledore

The last word can be replaced with any search criteria to search for quotes from your favorite character or book. Code can be found at https://github.com/vigneshm243/go-quotes-api.

Happy Coding!

Thanks for reading. Leave a ❤️ if you liked it and comment and let me know what you think.
Originally published at vigneshm.me