DEV Community: Ishan Mishra

❌ A2A "vs" MCP | ✅ A2A "and" MCP - Tutorial with Demo Included!!!

Ishan Mishra — Fri, 16 May 2025 04:53:23 +0000

Hello Readers!!

You must have heard about MCP an emerging protocol, "razorpay's MCP server out", "stripe's MCP server out"... But have you heard about A2A, a protocol sketched by google engineers and together with MCP these two protocols can help in making complex applications.

Let me guide you to both of these protocols, their objectives and when to use them!

Lets start with MCP first, What MCP actually is in very simple terms?[docs]
Model Context Protocol where protocol means set of predefined rules which server follows to communicate with the client. In reference to LLMs this means if I design a server using any framework(django, nodejs, fastapi...) but it follows the rules laid by the MCP guidelines then I can connect this server to any supported LLM and that LLM when required will be able to fetch information using my server's DB or can use any tool that is defined in my server's route.

Lets take a simple example to make things more clear[See youtube video for illustration]:
I want to make my LLM personalized for myself, this will require LLM to have relevant context about me when needed, so I have defined some routes in a server like /my_location, /my_profile, /my_fav_movies and a tool /internet_search and this server follows MCP hence I can connect this server seamlessly to any LLM platform that supports MCP(like claude desktop, langchain, even with chatgpt in coming future), now if I ask a question like "what movies should I watch today" then LLM can fetch the context of movies I like and can suggest similar movies to me, or I can ask LLM for best non vegan restaurant near me and using the tool call plus context fetching my location it can suggest me some restaurants.

NOTE: I am again and again referring that a MCP server can connect to a supported client (I am not saying to a supported LLM) this is because I cannot say that ~~Lllama-4 supports MCP and Lllama-3 doesn't~~ its just a tool call internally for LLM its the responsibility of the client to communicate with the server and give LLM tool calls in the required format.

Now its time to look at A2A protocol[docs]
Similar to MCP, A2A is also a set of rules, that when followed allows server to communicate to any a2a client. By definition: A2A standardizes how independent, often opaque, AI agents communicate and collaborate with each other as peers. In simple terms, where MCP allows an LLM client to connect to tools and data sources, A2A allows for a back and forth communication from a host(client) to different A2A servers(also LLMs) via task object. This task object has state like completed, input_required, errored.

Lets take a simple example involving both A2A and MCP[See youtube video for illustration]:
I want to make a LLM application that can run command line instructions irrespective of operating system i.e for linux, mac, windows. First there is a client that interacts with user as well as other A2A servers which are again LLM agents. So, our client is connected to 3 A2A servers, namely mac agent server, linux agent server and windows agent server all three following A2A protocols.
When user sends a command, "delete readme.txt located in Desktop on my windows system" client first checks the agent card, if found relevant agent it creates a task with a unique id and send the instruction in this case to windows agent server. Now our windows agent server is again connected to MCP servers that provide it with latest command line instruction for windows as well as execute the command on CMD or powershell, once the task is completed server responds with "completed" status and host marks the task as completed.
Now image another scenario where user asks "please delete a file for me in my mac system", host creates a task and sends the instruction to mac agent server as previously, but now mac agent raises an "input_required" status since it doesn't know which file to actually delete this goes to host and host asks the user and when user answers the question, instruction goes back to mac agent server and this time it fetches context and call tools, sending task status as completed.

A more detailed explanation with illustration and code go through can be found in this youtube video.

I hope I was able to make it clear that its not ~~A2A vs MCP~~ but its A2A and MCP to build complex applications.

Set Up Your Own Free Coding Copilot with Continue, Deepseek & Open Models – No Limits!

Ishan Mishra — Sat, 15 Feb 2025 15:15:08 +0000

Video Tutorial

Intended Audience 👤

Every software engineer!!
Unlimited autocomplete and chat without paying anything extra!!
VRAM or RAM(macos) required: 8GB

NOTE: Click on video link to see comparison between mac and linux as well as hosted deepseek vs qwen-2.5-coder

Intro 👋

We are going to setup our own custom coding copilot in both Linux and Mac(tested on M1 pro) , which will be free unlimited and will be hosted on local machine thus no leakage of sensitive data. We will use an open-source tool called continue for our setup and ollama to run the model.

Installing continue 📥

Continue come with a VS Code and JetBrains extension, simply search continue in extensions bar and install it.(NOTE: disable github copilot or other coding assistants). Below is a vs-code example to install continue.

Installing ollama 📥

Ollama is also very simple to install, just visit the download section, select your OS and get the installer, and is available for Linux and MacOS

Setting up continue 🛠️

As shown in the image above click on the continue icon(if not visible reload the vscode), click on "configure autocomplete options" from the drop-down , this will open the config file for continue. Replace it with the following code snippet

{
  "models": [
    // {
    //   "model": "gpt-4o",
    //   "title": "GPT-4o",
    //   "systemMessage": "You are an expert software developer. You give helpful and concise responses.",
    //   "apiKey": "",
    //   "provider": "openai"
    // },
    // {
    //   "model": "gpt-4o-mini",
    //   "title": "GPT-4o Mini",
    //   "systemMessage": "You are an expert software developer. You give helpful and concise responses.",
    //   "apiKey": "",
    //   "provider": "openai"
    // },
    {
      "model": "qwen2.5-coder:latest",
      "title": "Qwen2.5 coder:7B",
      "systemMessage": "You are an expert software developer. You give helpful and concise responses.",
      "provider": "ollama"
    }
  ],
  "tabAutocompleteModel": {
    "model": "qwen2.5-coder:latest",
    "title": "Qwen 2.5 Coder 7b",
    "provider": "ollama"
  },
  "customCommands": [
    {
      "name": "test",
      "prompt": "{{{ input }}}\n\nWrite a comprehensive set of unit tests for the selected code. It should setup, run tests that check for correctness including important edge cases, and teardown. Ensure that the tests are complete and sophisticated. Give the tests just as chat output, don't edit any file.",
      "description": "Write unit tests for highlighted code"
    }
  ],
  "contextProviders": [
    {
      "name": "code",
      "params": {}
    },
    {
      "name": "docs",
      "params": {}
    },
    {
      "name": "diff",
      "params": {}
    },
    {
      "name": "terminal",
      "params": {}
    },
    {
      "name": "problems",
      "params": {}
    },
    {
      "name": "folder",
      "params": {}
    },
    {
      "name": "codebase",
      "params": {}
    },
    {
      "name": "web",
      "params": {
        "n": 5
      }
    },
    {
      "name": "file"
    },🤖
    {
      "name": "currentFile"
    },
    {
      "name": "open"
    },
    {
      "name": "search"
    },
    {
      "name": "url"
    },
    {
      "name": "clipboard"
    }
  ],
  "slashCommands": [
    {
      "name": "edit",
      "description": "Edit selected code"
    },
    {
      "name": "comment",
      "description": "Write comments for the selected code"
    },
    {
      "name": "share",
      "description": "Export the current chat session to markdown"
    },🤖
    {
      "name": "cmd",
      "description": "Generate a shell command"
    },
    {
      "name": "commit",
      "description": "Generate a git commit message"
    }
  ]
}

This adds qwen-2.5 coder as chat and autocomplete model, there is also commented code to add openAI's gpt as well. To add different model for chat and autocomplete from different providers follow these docs:

Running Qwen2.5 coder(7B) 🤖

After installing ollama, go to your terminal and run ollama pull qwen2.5-coder and wait for it to complete.

Chat and Edit features ✨

Open vs-code and press ctrl+L for linux or cmd+shift+L for mac. this will open the continue window, drag it to right and select the qwen model. It may take longer for the first time since model is getting loaded into memory, now you can ask your queries. Let me walk you through some interesting features:

Use @docs in query box to reference the already provided docs, or you can add a new one as well.
Use @web to search and reference the web(doesn't work very well, instead setup google search)
Other context providers: docs
Use ctrl+I to open the selected code in edit mode.

Autocomplete 🚀

Go to the continue icon at the bottom right and enable the autocomplete option, now simply go to a code or write any comment autocomplete should start to show, press tab to accept whole or ctrl + -> to accept line by line.
For my system with M1 chip(pro) qwen-2.5:7b worked slow, and wasn't very usable for autocomplete, but was for chat. In linux with Rtx4060 both features were fast and usable. Go through the video for comparison.

And done 🥳🥳!! You know have your unlimited free coding copilot running on your local system!! 🥂🥂

Supercharging Deepseek-R1 with Ray + vLLM: A Distributed System Approach

Ishan Mishra — Sun, 02 Feb 2025 19:18:50 +0000

Video Tutorial

Intended Audience 👤

Everyone who is curious and ready to explore extra links OR
Familiarity with Ray
Familiarity with vLLM
Familiarity with kubernetes

Intro 👋

We are going to explore how we can run a 32B Deepseek-R1 quantized to 4 bit model, model_link. We will be using 2 Tesla-T4 gpus each 16GB of VRAM, and azure for our kubernetes setup and vms, but this same setup can be done in any platform or local as well.

Setting up kubernetes ☸️

Our kubernetes cluster will have 1 CPU and 2 GPU modes. Lets start by creating a resource group in azure, once done then we can create our cluster with the following command(change name, resource group and vms accordingly):

az aks create --resource-group rayBlog \  
    --name rayBlogCluster \  
    --node-count 1 \  
    --enable-managed-identity \  
    --node-vm-size Standard_D8_v3 \  
    --generate-ssh-keys

Here I am using Standard_D8_v3 VM it has 8vCPUs and 32GB of ram, after the cluster creation is done lets add two more gpu nodes using the following command:

az aks nodepool add \  
    --resource-group rayBlog \  
    --cluster-name rayBlogCluster \  
    --name gpunodepool \  
    --node-count 2 \  
    --node-vm-size Standard_NC4as_T4_v3 \  
    --labels node-type=gpu

I have chosen Standard_NC4as_T4_v3 VM for for GPU node and kept the count as 2, so total we will have 32GB of VRAM(16+16). Lets now add the kubernetes config to our system: az aks get-credentials --resource-group rayBlog --name rayBlogCluster.
We can now use k9s(want to explore k9s?) to view our nodes and check if everything is configured correctly.

As shown in image above, our gpu resources are not available in gpu node, this is because we have to create a nvidia config, so lets do that, we are going to use kubectl(expore!) for it:
kubectl create -f https://raw.githubusercontent.com/NVIDIA/k8s-device-plugin/v0.17.0/deployments/static/nvidia-device-plugin.yml
Now lets check again:

Great! but before creating our ray cluster we still have one step to do: apply taints to gpu nodes so that its resources are not exhausted by other helper functions: kubectl taint nodes <gpu-node-1> gpu=true:NoSchedule and same for second gpu node.

Creating ray cluster 👨‍👨‍👦‍👦

We are going to use kuberay operator(🤔) and kuberay apiserver(❓). Kuberay apiserve allows us to create the ray cluster without using native kubernetes, so that's a convenience, so lets install them(what is helm?):

helm repo add kuberay https://ray-project.github.io/kuberay-helm/

helm install kuberay-operator kuberay/kuberay-operator --version 1.2.2

helm install kuberay-apiserver kuberay/kuberay-apiserver --version 1.2.2

Lets portforward our kuberay api server using this command: kubectl port-forward <api server pod name> 8888:8888. Now lets create a common namespace where ray cluster related resources will reside k create namespace ray-blog. Finally we are ready to create our cluster!
We are first creating the compute template that specifies the resource for head and worker group.
Send POST request with below payload to http://localhost:8888/apis/v1/namespaces/ray-blog/compute_templates
For head:

{
    "name": "ray-head-cm",
    "namespace": "ray-blog",
    "cpu": 5,
    "memory": 20
}

For worker:

{
    "name": "ray-worker-cm",
    "namespace": "ray-blog",
    "cpu": 3,
    "memory": 20,
    "gpu": 1,
    "tolerations": [
    {
      "key": "gpu",
      "operator": "Equal",
      "value": "true",
      "effect": "NoSchedule"
    }
  ]
}

NOTE: We have added tolerations to out worker spec since we tainted our gpu nodes earlier.
Now lets create the ray cluster, send POST request with below payload to http://localhost:8888/apis/v1/namespaces/ray-blog/clusters

{
   "name":"ray-vllm-cluster",
   "namespace":"ray-blog",
   "user":"ishan",
   "version":"v1",
   "clusterSpec":{
      "headGroupSpec":{
         "computeTemplate":"ray-head-cm",
         "rayStartParams":{
            "dashboard-host":"0.0.0.0",
            "num-cpus":"0",
            "metrics-export-port":"8080"
         },
         "image":"ishanextreme74/vllm-0.6.5-ray-2.40.0.22541c-py310-cu121-serve:latest",
         "imagePullPolicy":"Always",
         "serviceType":"ClusterIP"
      },
      "workerGroupSpec":[
         {
            "groupName":"ray-vllm-worker-group",
            "computeTemplate":"ray-worker-cm",
            "replicas":2,
            "minReplicas":2,
            "maxReplicas":2,
            "rayStartParams":{
               "node-ip-address":"$MY_POD_IP"
            },
            "image":"ishanextreme74/vllm-0.6.5-ray-2.40.0.22541c-py310-cu121-serve:latest",
            "imagePullPolicy":"Always",
            "environment":{
               "values":{
                  "HUGGING_FACE_HUB_TOKEN":"<your_token>"
               }
            }
         }
      ]
   },
   "annotations":{
      "ray.io/enable-serve-service":"true"
   }
}

Things to understand here:

We passed the compute templates that we created above
Docker image ishanextreme74/vllm-0.6.5-ray-2.40.0.22541c-py310-cu121-serve:latest setups ray and vllm on both head and worker, refer to code repo for more detailed understanding. The code is an updation of already present vllm sample in ray examples, I have added few params and changed the vllm version and code to support it
Replicas are set to 2 since we are going to shard our model between two workers(1 gpu each)
HUGGING_FACE_HUB_TOKEN is required to pull the model from hugging face, create and pass it here
"ray.io/enable-serve-service":"true" this exposes 8000 port where our fast-api application will be running

Deploy ray serve application 🚀

Once our ray cluster is ready(use k9s to see the status) we can now create a ray serve application which will contain our fast-api server for inference. First lets port forward our head-svc 8265 port where our ray serve is running, once done send a PUT request with below payload to http://localhost:8265/api/serve/applications/

{
   "applications":[
     {
         "import_path":"serve:model",
         "name":"deepseek-r1",
         "route_prefix":"/",
         "autoscaling_config":{
            "min_replicas":1,
            "initial_replicas":1,
            "max_replicas":1
         },
         "deployments":[
            {
               "name":"VLLMDeployment",
               "num_replicas":1,
               "ray_actor_options":{

               }
            }
         ],
         "runtime_env":{
            "working_dir":"file:///home/ray/serve.zip",
            "env_vars":{
               "MODEL_ID":"Valdemardi/DeepSeek-R1-Distill-Qwen-32B-AWQ",
               "TENSOR_PARALLELISM":"1",
               "PIPELINE_PARALLELISM":"2",
               "MODEL_NAME":"deepseek_r1"
            }
         }
      }
   ]
}

Things to understand here:

ray_actor_options are empty because whenever we pass tensor-parallelism or pipeline-parallelism > 1 then it should either be empty to num_gpus set to zero, refer this issue and this sample for further understanding.
MODEL_ID is hugging face model id, which model to pull.
PIPELINE_PARALLELISM is set to 2, since we want to shard our model among two worker nodes. After sending request we can visit localhost:8265 and under serve our application will be under deployment it usually takes some time depending on the system.

Inference 🎯

After application is under "healthy" state we can finally inference our model. So to do so first port-forward 8000 from the same head-svc that we prot-forwarded ray serve and then send the POST request with below payload to http://localhost:8000/v1/chat/completions

{
    "model": "deepseek_r1",
    "messages": [
        {
            "role": "user",
            "content": "think and tell which shape has 6 sides?"
        }
    ]
}

NOTE: model: deepseek_r1 is same that we passed to ray serve

And done 🥳🥳!!! Congrats on running a 32B deepseek-r1 model 🥂🥂

What is Apache arrow and when to use it?

Ishan Mishra — Sun, 21 Jan 2024 19:32:47 +0000

What is Apache arrow(Bookish definition)? 🤔

Apache Arrow defines a language-independent columnar and in memory format, it also supports modern hardware like CPUs and GPUs. The Arrow memory format also supports zero-copy reads for lightning-fast data access without serialization overhead.

Is it a database? 💾

No, Apache Arrow is not a database. Instead, it's a cross-language development platform for in-memory data. Arrow's primary role is to act as a "universal data language" allowing different systems and programming languages to work with large datasets more efficiently and quickly.

Then what is Apache arrow...

My use-case for apache arrow arrived, when I wanted to create a data streaming pipeline of a very large dataset with almost zero latency for ML related processing.

Why not sql databases(PG, mysql etc..): My primary usecase is to have very low streaming latency and the transactional nature and disk-based storage of many SQL databases can introduce significant latency
Why not a database plus kafka or redis streams: While this setup is good for production applications with distributed system scaling. But for developing a data streaming pipeline which will be used by a processing service for data analysis, setting up kafka and configuring it for required throughput becomes a little complex or overkill. Plus I didn't wanted data to be transferred over network between my compute nodes. So eventually I had to setup kafka on each system, thus giving me no advantage of its distributed architecture.
Why Apache arrow: Apache arrow is in-memory storage and lazily loads data when iterated to it, making latency very small, and its table format storage also allows me to use simple filters. Plus it can interact with GPU memory as well for data processing. And it doesn't have any serialisation/de-serialisation overheads, as compared to redis which sores data in-memory as bytes and for my usecase it required serialisation/de-serialisation overheads.

Conclusion 👋

In conclusion, Apache Arrow emerges as an ideal solution for specific use cases where low-latency and efficient in-memory data processing are paramount. It's particularly well-suited for scenarios, where the goal is to create a high-performance data streaming pipeline for machine learning and data analysis purposes.

Weekly Awesome JS KT Part-2: Promise.all() vs multiple awaits

Ishan Mishra — Sun, 19 Mar 2023 14:43:02 +0000

Imagine you have a case where you want to retrieve some information about a list of users, and for each user you have to hit the endpoint and await a promise, how will you do it? I came across a similar problem(I am giving a simplified version of the problem I came across not the full code), and the approach I took was something like this:

First I created a async function which handles all the logic of updating and getting a particular user from its id

const updateUser = async ({userId}:{userId:string})=>{

await updateUserDetails(userId)

}

userIds = [....]

// loop for each userId and update it
for(const id of userIds)
{
 await updateUser({userId: id})
}

But there is a problem with this code, let me explain with a more simplified example:

const test1 = async () => {
  const delay1 = await Promise.delay(600); 
  const delay2 = await Promise.delay(600); 
  const delay3 = await Promise.delay(600); 
};

const test2 = async () => {
  await Promise.all([
  Promise.delay(600), 
  Promise.delay(600), 
  Promise.delay(600)]);
};

Here test1 takes almost ~1800ms to run since it runs sync each promises whereas test2 takes ~600ms to runs since it runs all promises in parallel

So a faster version of the code snippet one would be something like this:

const updateUser = async ({userId}:{userId:string})=>{

await updateUserDetails(userId)

}

userIds = [....]

await Promise.all(userIds.map((userId)=>updateUser({userId})))

Keep Exploring 🛥️

https://stackoverflow.com/questions/45285129/any-difference-between-await-promise-all-and-multiple-await

Application to make short gifs for your videos

Ishan Mishra — Sat, 18 Mar 2023 11:57:09 +0000

💡 Introduction

This article is a related to my previous article, in this I will discuss and share the code to make a gif maker of your own. If you want to know the idea behind this project do read my previous article.

🤔 Why is it useful or what's the importance of this project?

Have you tried converting a video to a gif using an online tool? even a 5 sec gif can go upto 10 of MBs, and gifs can improve the overall look of the application by replacing the static images with a moving short glimpse of the video or reels.

🏛️ Architecture

Our current architecture for gif maker is, we have a consumer(RabbitMQ) written in python and when a video is uploaded and processes successfully, an event is fired with params "mp4Url" and "s3Destination", then the script downloads the video from the given url convert it to a compressed gif and upload it to the S3 destination.

🥸 Code and Explanation

from moviepy.editor import VideoFileClip
import os
import urllib.request
import uuid
import logging
import boto3
from pathlib import Path
from dotenv import dotenv_values

ROOT_DIR = os.path.dirname(os.path.abspath(__file__))
config = dotenv_values(".env")


logging.basicConfig(
    filename=str(Path(__file__).parents[2]) + "/logs/app.log",
    format="%(name)s - %(levelname)s - %(message)s",
)

mp4UrlPrefix = config["CLOUDFRONT_VIDEO_PREFIX"]


def getGifUrlFromMp4Driver(params):

    logging.info("Started getGifUrlFromMp4Driver")

    if "mp4Url" not in params:
        logging.error("mp4Url not in params")
        return {"gifUrl": None}
    if "s3Destination" not in params:
        logging.error("s3Destination not in params")
        return {"gifUrl": None}

    # get full mp4 url
    mp4Url = mp4UrlPrefix + "/" + params["mp4Url"]
    s3Destination = params["s3Destination"]

    # if tweaking params are not passed, then use the default 
    # value
    fps = params.get("fps", 20)
    fuzz = params.get("fuzz", 1)
    start = params.get("start", 5)
    duration = params.get("duration", 5)
    videoWidth = params.get("videoWidth", 144)
    videoHeight = params.get("videoHeight", 256)

    try:
        # initializing s3 session
        session = boto3.Session(
            aws_access_key_id=config["AWS_KEY"],
            aws_secret_access_key=config["AWS_SECRET"],
        )

        s3 = session.resource("s3")

        mp4_folder = ROOT_DIR + "/videos/"
        gif_folder = ROOT_DIR + "/gifs/"
        # creating a unique name for mp4 video download path 
        # and gif
        name = str(uuid.uuid4())

        # download mp4
        downloadedVideoPath = f"{mp4_folder}{name}.mp4"
        urllib.request.urlretrieve(mp4Url, downloadedVideoPath)

        # to reduce size of gif as well as to not take a long 
        # time we will try 3 times with reduced frame rates 
        # and increased fuzz to reduce size of gif
        counter = 0
        convertedGifPath = f"{gif_folder}{name}.gif"
        while True:
            counter += 1
            videoClip = VideoFileClip(downloadedVideoPath)

            # take a clip of video from x to y
            videoClip = videoClip.subclip(start, start + duration)
            # resizing video dimensions to desired width and 
            # height, this also reduces gif size to choose it 
            # wisely
            videoClip = videoClip.resize((videoWidth, videoHeight))

            # setting video fps, this also reduces gif size
            videoClip = videoClip.set_fps(fps)
            videoClip.write_gif(
                filename=convertedGifPath,
                program="ImageMagick",
                opt="optimizeplus",
                tempfiles=True,
                verbose=False,
                fuzz=fuzz,
                logger=None,
            )

            # get size of converted gif
            file_size = os.path.getsize(convertedGifPath)

            # greater than 500Kb then reduce fps
            if file_size > 500000 and counter <= 3:
                if counter == 1:
                    fps = 15
                elif counter == 2:
                    fps = 10
                elif counter == 3:
                    fps = 5
                continue
            break

        # remove downloaded video from disk
        os.remove(downloadedVideoPath)
        destBucketName = config["AWS_BUCKET_IMAGE_NAME"]

        if s3Destination[-1] != "/":
            s3Destination += "/"

        gifPath = "gif" + ".gif"

        # upload gif to s3 bucket        
       s3.Bucket(destBucketName).upload_file(convertedGifPath, 
       s3Destination + gifPath)

        gifUrl = f"{s3Destination}{gifPath}"
        os.remove(convertedGifPath)

        # return back the uploaded gif url
        return {"gifUrl": gifUrl}

    except Exception as e:
        logging.error(f"Error in getGifUrlFromMp4Driver: {e}")
        os.remove(downloadedVideoPath)
        return {"gifUrl": None}

The above code is executed when the consumer listening to the event is triggered, here's an example of consumer

import json
from dotenv import dotenv_values
import logging
from getGifUrlFromMp4 import getGifUrlFromMp4Driver
import requests

config = dotenv_values(".env")


def gifConsumer(ch, method, properties, body):
    try:
        params = json.loads(body)
        print(f" Received: {params}")

        res = getGifUrlFromMp4Driver(params)
        print(f" Gif created, url: {res}")

        if res["gifUrl"] == None:
            print("Gif url not found")
            ch.basic_ack(delivery_tag=method.delivery_tag)
            return

        res["id"] = res["gifUrl"].split("/")[1]
        mediaResponse = requests.post(
            <url>, data=res
        )

        print(f"Response from media endpoint status code: {mediaResponse.status_code}")
        print(f"Response from endpoint: {mediaResponse.json()}")

        ch.basic_ack(delivery_tag=method.delivery_tag)
    except Exception as e:
        print("Error in gifConsumer: ", e)
        logging.error(f"Error in gifConsumer: {e}")
        ch.basic_ack(delivery_tag=method.delivery_tag)

This consumer calls getGifUrlFromMp4Driver with mp4Url and s3Destination, the code is well explained in the comment, so do read them. After successfull or unsuccessfull conversion of gif controls comes back to the consumer function, now in case there's a gif url we give a api call to our endpoint informing it a successfull conversion of the gif and to save the gif url with the reel.

Here's the gist code in case formatting is messed up here: gist

Keep Exploring 🛥️

From Ideation to Production In Just Four Hours

Ishan Mishra — Fri, 10 Mar 2023 06:03:28 +0000

Second Part: link. This part contains the code and explanation.

During my internship with YellowClass, I came through developing an interesting project. Little did I know that this project would end up feeling like a hackathon. In this post, I'll be sharing the journey I went through while developing this project. From the initial architecture and the thought process behind it, to the challenges I faced and how I overcame them, I'll be detailing my experiences step-by-step. So, buckle up and join me as I take you through my journey of developing this exciting project.

How it started?

It was a friday evening around 7PM when our team manager brought up an interesting idea, the possibility of creating an Instagram-style auto-play section for the reel on our app's home page. Instead of displaying a static image, the reel would showcase a short snippet of the content, enticing the user to click and watch the full reel. This sparked a conversation between the app team and product managers, as they brainstormed and debated the best approach for implementing this new feature. I was sitting and doing the jira's assigned to me as well as listening to the whole conversation(not very focused on work that day 😅).

As everyone discussed the possibility of creating an auto-play section for the reel on our app's home page, the team began brainstorming different ways to display short snippets of content. Someone suggested using GIFs, as it would require no app change, but there was a catch. They had previously tried using GIFs, but found that the size of converting to them was too large for our needs. I also began to explore different libraries in Python to see if I could find a way to create smaller, more compressed GIFs. After some experimentation, I discovered a library that allowed me to tweak certain parameters and achieve the desired result, it was moviepy. My initial code was a simple snippet that would take a video path and create a GIF for the first 5 seconds of the video. I shared this with my manager and the engineering director, and to our delight, the snippet was able to generate GIFs under 1MB for many of our videos. While we still needed to make some tweaks to get under the 500KB limit, we finally had a viable solution. Best of all, since it was a script, we could simply give it the links to all of our reels and upload the resulting GIFs to our AWS S3 bucket without having to involve the content team, this could save their one week work.

Hackathon begins...

With the decision made to use GIFs as our solution, it was time to create a roadmap for the project. It was already 8PM, and most of the office had emptied out for the day. We knew that we needed to act quickly to make progress. The project was divided into two distinct steps. The first step was to replace the existing static images for the reels with the newly generated GIFs. The second step was to automate the process of generating and storing GIFs for any new video that was uploaded to the app. This would be a more complex undertaking.

With step one of the project in motion, our manager and engineering director began to write a query to get the reels and their uploaded paths in a proper CSV format. They also worked on changing the backend to accommodate the new GIF format. Meanwhile, I was tasked with writing a Python script that would read the CSV and generate a GIF for each video, then upload it to our S3 bucket. To ensure that the GIFs were compressed to the desired size, we added an additional step that would retry the process 3-4 times if a particular GIF was larger than 500KB. We also included a feature to reduce the frame rate until the file was within the desired size range. Additionally, we knew that we would need to make the script multithreaded in order to generate GIFs for the approximately 900 reels in a reasonable amount of time. This was a complex task, but we were determined to see it through to the end.

After one hour of hard work, our script, CSV, and backend were finally ready to be put to the test. With a deep breath and a sense of excitement, we ran the script, watching as the timer ticked down to the finish line. Two hours was the estimated time to complete the task, and we eagerly watched the number of generated GIFs climb higher and higher on the large screen in front of us. As we waited for the script to finish, we took a break to order and enjoy a much-needed dinner, our eyes glued to the screen the entire time. The anticipation was almost too much to bear. But finally, the script came to a close, and we could see the results of our hard work displayed in front of us.
With eager anticipation, I opened the app and navigated to the home page, eager to see the results of our hard work. But to my surprise and shock, none of the videos were displaying GIFs. My heart sank, and I was overcome with a sense of disappointment. But then, my manager opened his phone, and to my relief, every video was displaying a GIF. I quickly realized that we were in the midst of an A/B testing, and that our hard work had indeed paid off. We tested everything thoroughly and found that everything was working perfectly. Finally, as the clock approached midnight, we wrapped up our hackathon. Now, we could monitor the click rates on reels during the weekends and make informed decisions based on the data we collected.

This post is already going longer than I expected 😅. In the part-2 of this post I will share the code snippet as well as the working process.

Weekly Awesome JS KT Part-1

Ishan Mishra — Thu, 09 Mar 2023 15:03:19 +0000

1=> ⚡️ Spread Operator(...)

let numbers = [1, 2, 3, 4];
let newArray = [...numbers, 5, 6];
console.log(newArray)

Output: [1,2,3,4,5,6]

2=> ⚡️ Closures in javascript with real world example

Closures: a function that returns another function with access to the parent scope. Example below:
function outerFunction(outerValue) {
  return function innerFunction(innerValue) {
    console.log(`Outer value: ${outerValue}, Inner value: ${innerValue}`);
  };
}
const innerFunc = outerFunction("Hello");
innerFunc("World");
Output: "Outer value: Hello, Inner value: World"

Here we can see that parent function's variables are preserved and still accessible by child function even though parent function has been already executed.

Now, let's see a real world use case:

function createCounter() {
  let count = 0;

  function increment() {
    count++;
    console.log(`Count: ${count}`);
  }

  function decrement() {
    count--;
    console.log(`Count: ${count}`);
  }

  return {
    increment,
    decrement,
  };
}

const counter = createCounter();
counter.increment(); // Output: "Count: 1"
counter.increment(); // Output: "Count: 2"
counter.decrement(); // Output: "Count: 1"

In this example, the createCounter function returns an object with two methods, increment and decrement. The count variable is defined inside the createCounter function and is not accessible outside of it. The increment and decrement methods have access to the count variable due to the closure created by the createCounter function.

This allows us to create an object with private variables and methods that are not accessible from outside the object, providing a way to encapsulate functionality and prevent outside code from interfering with the object's state.

3=> ⚡️ Prototypes in javascript with example

Prototypes: an object that is associated with every function and object in JavaScript. Example:

function Person(name, age) {
  this.name = name;
  this.age = age;
}

Person.prototype.greet = function() {
  console.log(`Hello, my name is ${this.name}`);
};

const john = new Person("John", 30);
const jane = new Person("Jane", 25);

john.greet(); // Output: "Hello, my name is John"
jane.greet(); // Output: "Hello, my name is Jane"

In this example, we define a Person constructor function that takes a name and an age parameter and sets them as properties on the newly created Person object using this. We then define a greet method on the Person.prototype object, which is shared by all Person objects.

By using the prototype object to define methods on the Person class, we can create new Person objects that inherit these methods, without having to define them again for each new object. This makes our code more efficient and easier to maintain, as we can make changes to the Person class and have those changes reflected in all instances of the class.

In addition, we can also use prototypes to add new methods and properties to existing objects, allowing us to extend their functionality dynamically.

Awesome & Productive Terminal For MacOS

Ishan Mishra — Sun, 15 Jan 2023 17:17:10 +0000

1. Intended audience.

This article is for every programmers, who want to make their day to day command line experience cool as well as more productive.

2. Overview.

As a programmer, we always use terminal for git, navigating through directories, running servers etc. Why not take some time out and make this basic tool more productive and cool, thats what we are going to do in this article.

Note: To save in nano hit control+o and hit return, to exit hit control+x

3. Step-1: Download Iterm2 and Oh-My-Zsh.

Download Iterm2 from its official website and install it. iTerm2 is a replacement for MacOS default Terminal

MacOS now by default uses zsh, so we just need to install oh-my-zsh which is a framework for managing zsh configurations. Install it by using entering the following command

sh -c "$(curl -fsSL https://raw.githubusercontent.com/ohmyzsh/ohmyzsh/master/tools/install.sh)"

4. Add VsCode Shortcut.

Run nano ~/.zshrc to open and edit zsh configuration files

To open vscode in PWD using "code ./" add the following line in your .zshrc:

# vscode code () { VSCODE_CWD="$PWD" open -n -b "com.microsoft.VSCode" --args $* ;}

5. Add theme

Theme Credit: https://github.com/stefanjudis/dotfiles/blob/primary/config/oh-my-zsh/stefanjudis.zsh-theme

This theme will have random emoji at prompt section and will also display current git branch.

Make a new theme file using

nano ~/.oh-my-zsh/themes/ishanextreme.zsh-theme

Copy and paste theme code from here to "~/.oh-my-zsh/themes/ishanextreme.zsh-theme" file

Finally run nano ~/.zshrc and set ZSH_THEME="ishanextreme"

Run source ~/.zshrc to reload and apply changes.

6. Plugins

Here's the list of plugins I use

git

zsh-autosuggestions

sudo

zsh-syntax-highlighting

tutorial to add plugins here

7. Style ITerm2

Open ITerm2 and hit ⌘ , go to "Appearance" tab and set theme to "Minimal".

Color theme used: Blue Matrix

Next lets setup status bar, open settings (⌘ ,) then go to profiles tab select "Default" profile select "Session" tab from it and click on enable status bar("status bar enabled"), and finally start configuring it. Here's the list of things we will add to our status bar:

Github current branch: drag "git state" from status bar component menu to active components(down). You can configure colors of component by selecting them and clicking "configure component" option at bottom.

Current Working Directory: drag "Current Directory" component to active components list.

Current conda environment: (If you have anaconda installed) or you can use this method to display any information(like node version, etc).

Enable Shell Integration from ITerm2->Install shell integration. Note-> this will enable a blue prompt by default in terminal you can disable it, search "show mark indicators" setting and disable it.

Now you can create your own variable and display them using "Interpolated Strings".

Run nano ~/.zshrc and add the following
iterm2_print_user_vars() {
  # extend this to add whatever
  # you want to have printed out in the status bar
  # if you don't have anaconda install use it for displaying
  # your node version "echo node -v" or anything you like
  iterm2_set_user_var condaEnv $(echo $CONDA_DEFAULT_ENV)
  iterm2_set_user_var pwd $(pwd)
}
Now use this variable in status bar, again open configuration setting for status bar, drag "Interpolated String", click on configure component and input \(user.condaEnv) in "String value" field.

7. Adding and enabling scripts.

In this step we will add some useful scripts to our ITerm and will also add shortcuts for the same.

Run cd /Users/"your user name"/Library/Application\ Support/iTerm2/Scripts, open any editor in this directory, create a new file "awesome_scripts.py" and copy the code from here to the editor and save the file (take care of identations, its a python code).

This will add globally callable functions which can be added as shortcuts, you need to everytime enable this scripts when you open ITerm window from scripts->awesome_scripts

Now to add schortcut to touchbar, hit ⌘ , got to Keys tab, click add Touch Bar Item.

Give it a label(use fn for emojis), Under actions field choose "Invoke Script Function" now add the function name to "Function Call" field. (See function name and there action below)

To customize touch bar, go to view->Customize Touchbar, you can see all your newly created shortcuts, drag and add them. (Tip: Remove siri icon to make more space in touchbar 😉)

List of function calls and their actions:

clear(): It will run "clear" command on all the sessions in a particular tab.

start_server(): It will run "yarn start:qa" on all the sessions in a particular tab.

stop_servers(): It will run "killall node" and "clear" on all the sessions in a particular tab.

shutdown(): It will kill all processes in every tab and force closes ITerm window.

git_push(): To use this make sure you have the theme in Step-5 installed. It will run "git add -A", "git commit -m'random emoji in your current line of ITerm'" and "git push origin 'your current branch'" in only your current session.

git_pull_staging(): It will run "git pull origin @staging --ff" in all the sessions in your current tab.

Thats It!!!!!

What is Tor and how to use it?

Ishan Mishra — Mon, 31 Oct 2022 11:17:51 +0000

1. Intended audience.

This article is for everyone who browses the internet and wants to learn how to surf the internet anonymously to protect one's identity.

2. Overview.

This article is an introduction to an open-source project "TOR" in this article we will get an overview of how tor works, how it can help protect one's identity while browsing internet and how to download, set it up and use it. I will try to make this article as simple and free of technical terms as possible. This article is for educational purposes only.

3. What is Tor?

The United States Naval Research Laboratory developed The Onion Routing Protocol (Tor) to protect U.S. intelligence communications online. Currently Tor is an open source project maintained by the Tor Project, Inc., nonprofit organization. According to the official website Tor is helpful to "Browse privately, explore freely. Defend yourself against tracking and surveillance. Circumvent censorship."

4. Tor vs VPN.

Normal request
When we visit a website let's say "www.google.com" a packet similar to the one shown in the image below is generated and sent to the requested website's server, now this packet contains our(source) IP address, destination IP address, and the request data. If we are using an HTTPS connection(see the lock icon in url bar) then the data is encrypted and cannot be read by any middle man but the source and destination IP addresses can be sniffed by hackers and other advertisement agencies to show relevant ads to a user based on the website he/she is visiting.

Through VPN(Centralised)
When a request is sent through VPN, the client's IP address is hidden and replaced by the VPN server's IP therefore its not possible for any middleman to get the client's source and destination but the VPN service provider knows everything about the website a user is visiting, so there should be trust between the user and the VPN service provider, plus VPN services are not free of cost.

Through Tor(Decentralised)
When a request is sent through Tor, it goes through Tor Nodes, then finally to a destination, so when source(client) sends a request it will show destination as "Tor Node-1" and when exit node send request to destination it will show source as "Tor node-3(ex)", therefore it is very difficult to track by a middleman as to which website a request is sent and from which user a particular request is coming. Drawback of using Tor is that it's very slow as compared to normal and VPN connection.

5. How to download and setup Tor.

Visit tor project official website, here, then choose your operating system, for android Tor browser can also be downloaded directly from playstore. Tor is currently available for windows, MacOs, linux, and android. This will download a Tor browser now whenever you want to surf the internet anonymously simply open the browser and click the "Connect" button.

6. Is Tor legal?

So, is Tor legal? In short: yes, using Tor is completely legal and is not banned in India. However, if you use Tor for illicit activities, like buying drugs, weapons etc.. then you can face legal scrutiny if your activities are traced back to you.

7. Conclusion.

Tor is a great project to protect one's identity online and can be used like a free and no data limit VPN, if you can tolerate slow loading 😅.

[1]: Tor project official website

[2]: https://vpnoverview.com/privacy/anonymous-browsing/is-tor-legal/

TCP Pros and Cons with working example

Ishan Mishra — Sat, 24 Jul 2021 07:40:30 +0000

Backend Essentials-2 [TCP]

1. Overview

This article covers the basics of TCP(Transmission Control Protocol) in terms of its pros and cons. This article also provides the working example of TCP.

2. TCP

TCP is a transport protocol that is used on top of IP to ensure reliable transmission of packets. TCP includes mechanisms to solve many of the problems that arise from packet-based messaging, such as lost packets, out of order packets, duplicate packets, and corrupted packets.

Pros:

Acknowledgement: To ensure that data is received successfully this header is used. After sending the data sender starts a timer and if receiver doesn't sends back the acknowledgement after the timer ends sender sends the packet again. This ensures no data loss.

Guaranteed delivery.

Connection Based: Client and server needs to establish a unique connection.

Congestion Control: Data waits(if network is busy) and sent when network can handle it.

Ordered Packets: TCP connections can detect out of order packets by using the sequence and acknowledgement numbers.

Cons:

Larger Packets: Because of all the above features.

Slower than UDP.

Stateful.

Server Memory(DOS): Because of statfullness server needs to store the connection information. If some low level client keeps on sending request to the server without doing any acknowledgement, than server keeps waiting and the data is stored in memory(until timeout) so server can eventually die if many of these clients sends request.

3. Code Example

Requirements: nodejs, code editor, telnet

const net = require("net");

const server = net.createServer(socket=>{
    socket.write("Hello\n");

    socket.on("data", data=>{
        console.log(data.toString())
    })
})

server.listen(8080);

Run the script by executing following command

node <filename>.js

Next step is to connect to the TCP server, for this execute the following command in a terminal

telnet localhost 8080

and now a "Hello" message will be printed on terminal screen, type any message in terminal and it will be printed on server's console screen.

[1]: TCP vs UDP Crash Course[Hussein Nasser]

[2]: Transmission Control Protocol[Khan Academy]

Why your data is not secure in public WIFI?-The OSI Model

Ishan Mishra — Fri, 21 May 2021 07:08:57 +0000

Backend Essentials-1 [The OSI Model]

1. Overview

This article gives basic information of the OSI(Open Systems Interconnection) Model. OSI model is conceptual framework used to describe the functions of a networking system, it consists of 7 layers. It was published in 1984 and is still used today as a means to describe Network Architecture.

2. The 7 Layers of OSI Model

To understand the layers of OSI model lets take an example of sending a simple get request to the server.

Layer-7 Application:

This layer contains bunch of strings like header, cookies..., prepared by the network application. It also receives stuff for application to process and display.

Layer-6 Presentation

If using HTTPS, this layer encrypts/decrypts the above data.

Layer-5 Session

Controls connection between different computers. Like for HTTP we require a TCP connection. Assigns a tag which is send with the data.

Layer-4 Transport

On receiving the data from above layers, this layer performs segmentation, i.e. breaks the data into smaller units attaches the target and source port(mostly auto generated by application). It also implements error control.

Layer-3 Network

This layer handles the transmission of data. It doesnot checks for any error, its job is to just send the data to destination IP address.

Layer-2 Data Link

This layers performs node-to-node transmission. It devides data into frames and attaches the MAC address to the frames. It also performs error correction.

Layer-1 Physical

This layer contains information in the form of "bits" and transmits individual bits from one node to another in from of electrical/light/wifi... signals.

Now at receivers end data is processed from Layer-1 to Layer-7.

3. Why Public WIFI is Unsafe?

Frames sent by the client connected to WIFI network goes to every user connected to that network, network card rejects the frames which are not meant to be received by it, but a hacker can design an application to receive all the frames which are coming and if the data is not encrypted(site is not having HTTPS connection for example) the hacker can read all the data. This method is known as "network sniffing".

4. Conclusion

This was an overview of The OSI model in software engineering perspective.

[1]: The OSI Model-Explained by Example[Hussein Nasser]

[2]: Layers of OSI Model

[3]: What is the OSI Model?