DEV Community: Pranjal Sharma

Back-of-the-envelope Estimation System Design

Pranjal Sharma — Thu, 27 Jun 2024 20:27:30 +0000

Back-of-the-envelope estimation is a technique used to quickly approximate values and make rough calculations using simple arithmetic and basic assumptions.

Estimation Techniques

1) Rule of Thumb →

General principals applied to make good estimates. eg : 1 user generates 1MB of data on social media / day.

2) Approximations →

Rounding of complex calculations to powers of 10 or 2 to simply and get to the estimates easily. eg: 1 day = 10^5 seconds.

3) BreakDown and aggregation →

Breaking down bigger problems to smaller components and estimating them individually along with aggregating or combining them to reach the results. eg: Social media data = User Data + Multimedia Data + Metadata .

4) Sanity check →

Just having an overall check over the possibility of the estimates not varying a lot from reality is needed at last . For eg : The numbers achieved should match the original real life data.

Types of Estimations

1) Load Estimations

Designing a post generation social media platform.
Daily Active Users ( DAU ) → 100 Million
Avg. Posts → 10 per user per day
Total posts → 100 M * 10 = 1B post/day

Hence Request rate per second = 1B / 10^5 requests/second = 10000 req/sec.

2) Storage Estimations

Twitter Storage
DAU → 500 M
1 user = 3 tweets (avg)/day
1 tweet text ~ 250B
1 photo ~ 200KB [10% contain photo]
1 video ~ 300MB [5% contain video]

Total storage/day ~ 1500M * (250B + 20KB + 15KB)
~ 375 GB + 30TB + 225TB ~ 255TB

3) Bandwidth requirements

Estimate the daily amount of incoming data to the service.
Estimate the daily amount of outgoing data from the service.
Estimate the bandwidth in Gbps (Gigabits per second) by dividing the incoming and outgoing data by the number of seconds in a day.

4) Latency Estimation

For eg. API consist of RestCall 1 , Rest Call 2 , Rest Call 3

Total Latency → 50ms + 100ms + 150ms ~ 300ms [ if it is sequential ]
→ max(50,100,150) ~ 150ms [ if it is parallel ]

5) Resource Estimation

1 req ~ 10ms of CPU
total req ~ 10000req/sec
total cpu time ~ 10000 * 10 = 100000 ms/sec.
1 CPU can handle 1000ms/sec
Total CPU core = 100000 / 1000 = 100

Notion for NextJS CMS

Pranjal Sharma — Fri, 21 Jun 2024 09:18:02 +0000

Notion has become a popular productivity tool for individuals and teams. Did you know that Notion can also serve as a backend for your web applications? In this article, we’ll explore the benefits of using Notion as a backend for a Next.js application and demonstrate how to do it using the Notion API and TypeScript.

Prerequisites

Before we get started, make sure you have the following:

A Notion account
A Notion API key
A Next.Js project set up with TypeScript
The notion-client package installed

To get your Notion API key, go to your Notion integrations page, create a new integration, and copy the API key. To install the notion-client package, run the following command:

npm install notion-client

1) Create a Notion database

First, let’s create a Notion database to store our blog posts. To do this, go to your Notion dashboard and create a new page. In the page properties, click the “Add a database” button, and select “Blog Posts” as the database template.

This will create a database with the following properties:

Date
Tags
Title
Content Feel free to customize the properties to suit your needs.

Step 2: Fetch data from Notion

Next, let’s fetch the data from Notion using the Notion API. Create a new file called notion.ts in your Next.js project, and add the following code:

import { Client } from '@notionhq/client';


const notion = new Client({ auth: process.env.NOTION_API_KEY });
export async function getBlogPosts() {
  const databaseId = process.env.NOTION_DATABASE_ID;
  const response = await notion.databases.query({
    database_id: databaseId,
  });
  return response.results.map((page) => ({
    id: page.id,
    title: page.properties['Title'].title[0].text.content,
    date: page.properties['Date'].date.start,
    tags: page.properties['Tags'].multi_select.map((tag) => tag.name),
    content: page.properties['Content'].rich_text[0].text.content,
  }));
}

In this code, we’re creating a new Client object from the notion-client package, using the NOTION_API_KEY environment variable to authenticate. We’re also defining a function called getBlogPosts that retrieves the blog posts from Notion. We’re using the database_id environment variable to specify the ID of the Notion database we created earlier. Then, we’re using the databases.query method to retrieve the data from the database. The databases.query method returns an array of pages, where each page represents a blog post. We’re mapping over the results and extracting the relevant properties (title, date, tags, and content) from each page.

Step 3: Create a Next.js API route

Next, let’s create a Next.js API route to serve our blog posts. Create a new file called blog.ts in the pages/api directory, and add the following code:

import { NextApiRequest, NextApiResponse } from 'next';
import { getBlogPosts } from '../../lib/notion';

...

export default async function handler(req: NextApiRequest, res: NextApiResponse) {
  const posts = await getBlogPosts();
  res.status(200).json(posts);
}

This code defines a new API route that retrieves the blog posts using the getBlogPosts function we defined earlier. We’re using the NextApiRequest and NextApiResponse types from Next.js to ensure type safety.

Step 4: Display the blog posts on the frontend

Finally, let’s display the blog posts on the frontend. Create a new file called index.tsx in the pages directory, and add the following code:

import { GetStaticProps } from 'next';
import { getBlogPosts } from '../lib/notion';

export const getStaticProps: GetStaticProps = async () => {
  const posts = await getBlogPosts();
  return {
    props: {
      posts,
    },
  };
};
interface Post {
  id: string;
  title: string;
  date: string;
  tags: string[];
  content: string;
}
interface Props {
  posts: Post[];
}
export default function Home({ posts }: Props) {
  return (
    <div>
      {posts.map((post) => (
        <div key={post.id}>
          <h2>{post.title}</h2>
          <p>{post.date}</p>
          <ul>
            {post.tags.map((tag) => (
              <li key={tag}>{tag}</li>
            ))}
          </ul>
          <p>{post.content}</p>
        </div>
      ))}
    </div>
  );
}

In this code, we’re using the GetStaticProps function from Next.js to fetch the blog posts at build time. We’re defining a new interface called Post to represent a single blog post, and another interface called Props to represent the props of our Home component. In the Home component, we’re using the map method to render each blog post as a div element. We’re displaying the title, date, tags, and content of each post.

Step 5: Run the application

That’s it! You can now run your application using the following command:

npm run dev

MsSQL on MacOs

Pranjal Sharma — Thu, 20 Jun 2024 17:08:24 +0000

MSSql database is easy to configure on a Windows System . For MacOs we need to take care of few steps to get it installed and run properly .

Lets see what all steps we need to follow →

01 : Download Docker

Docker is a set of platform as a service products that use OS-level virtualization to deliver software in packages called containers. We would need it to run Microsoft SQL on Mac.

→ Check docker version
$ docker --version

→ Download and install docker from here 👉 Docker Desktop

02 : Download the MS SQL Server Image to Docker

→ After that, you need to pull the SQL Server 2019 Linux container image from Microsoft Container Registry.

[ Make sure docker is running in background ]

$ sudo docker pull mcr.microsoft.com/mssql/server:2019-latest

→ Then you can run the docker images command and verify whether the docker image has been pulled successfully.

03 : Run the docker container

→ Command to run the docker container.

🔥 Command to run the container 
docker run -d --name sql_server_demo -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=reallyStrongPwd123' -p 1433:1433 mcr.microsoft.com/mssql/server:2019-latest

🔥 Command for M1 Chip, please try this
docker run -e "ACCEPT_EULA=1" -e "MSSQL_SA_PASSWORD=reallyStrongPwd123" -e "MSSQL_PID=Developer" -e "MSSQL_USER=SA" -p 1433:1433 -d --name=sql mcr.microsoft.com/azure-sql-edge

→ Make sure to put you own password in SA_PASSWORD.
→ You can name your container after the --name flag.
→ -d flag represents the detach mode that releases the terminal after you run the above command.

→ Then run the docker ps command to verify whether your container has started to run

→ If your container stops after a few seconds it started, run docker ps -a command & > docker logs to check what’re the errors.

04 : Install the MS SQL CLI

→ Next, you need to install sql-cli via npm.

$ npm install -g sql-cli
OR
$ sudo npm install -g sql-cli

→ Link to install npm if not present .

05 : Test the Installation by Login In

→ Testing the mssql integration by logging in

$ mssql -u sa -p <Your Pass word>

→ If correctly done : mssql> prompt will come up.

→ Then run select @@version to verify the connectivity.

$ mssql -u sa -p reallyStrongPwd123
Connecting to localhost...done
sql-cli version 0.6.2
Enter ".help" for usage hints.
mssql> select @@version
--------------------------------------------------------------------
Microsoft SQL Server 2019 (RTM-CU15) (KB5008996) - 15.0.4198.2 (X64)
Jan 12 2022 22:30:08
Copyright (C) 2019 Microsoft Corporation
Developer Edition (64-bit) on Linux (Ubuntu 20.04.3 LTS) <X64>
1 row(s) returned
Executed in 1 ms
mssql>

06 : [OPTIONAL] Download and install the GUI application - Azure Data Studio

Azure Data Studio

07 : 😊 We Are Done ! Stop the services once completed with the work

→ docker stop <container-id> to stop the docker container.

ChatGPT Slack Bot

Pranjal Sharma — Thu, 20 Jun 2024 16:38:20 +0000

Slack is one of the most widely used communication tools for teams, and with the integration of OpenAI’s ChatGPT, it becomes an even more powerful tool. ChatGPT is a highly advanced language model that can generate human-like responses to a given prompt. In this blog, we will show you how to integrate ChatGPT with Slack and use it to answer questions and have conversations with your team members.

Note: the API has a higher uptime compared to the ChatGPT UI 😄

1. Register an app with Slack and gather all the tokens

First step is to register a new app on Slack and obtain the Slack Bot Token and Slack API Token.

Log in to your Slack workspace and Go to Slack API website.
Click on “Create an app” and select “From scratch”
Give your app a name, select your Slack workspace.
In Basic information > Add features and functionality. Click on “Permissions” and in Scopes add in Bot Token Scopes: app_mentions:read ; channels:history ; channels:read ; chat:write.
In settings, click on “Socket Mode”, enable it and give the token a name. Copy the Slack Bot App Token.
In Basic information > Add features and functionality. Click on “Event Subscriptions” and enable it. Furthermore in “Subscribe to bot events” select “app_mention”. Save changes.
Go to the “OAuth & Permissions” section and install your app to your workspace.
Copy the Slack Bot Token.

2. Get the OpenAI API key . [ Valid for a month for free users ]

Need to have a OpenAI API key to integrate ChatGPT.

Go to OpenAI website.
Go to API ket section and create a new API key after you login.
Copy the API key.

3. Install necessary dependencies

pip install openai
pip install slack-bolt 
pip install slack

Install these dependencies . Slack - bolt is a bunch of tools and libraries that allow developers to easily create Slack applications.

Run the application

Fill in the first 3 tokens in this script with your tokens and run the application.

SLACK_BOT_TOKEN = "xoxb-2196501177986-5475158173799-DTxoGAJMjSrqZ1UbKJQDRkYq"
SLACK_APP_TOKEN = "xapp-1-A05DZ02E7JT-5502278073649-6d0d2eabadfa2388189e2bd414393d764d87de68f4df7234f2e87a421eba9440"
OPENAI_API_KEY  = "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"
import os
import openai
from slack_bolt.adapter.socket_mode import SocketModeHandler
from slack import WebClient
from slack_bolt import App
# Event API & Web API
app = App(token=SLACK_BOT_TOKEN) 
client = WebClient(SLACK_BOT_TOKEN)
# This gets activated when the bot is tagged in a channel    
@app.event("app_mention")
def handle_message_events(body, logger):
    # Log message
    print(str(body["event"]["text"]).split(">")[1])
    # Create prompt for ChatGPT
    prompt = str(body["event"]["text"]).split(">")[1]
    # Let the user know that we are busy with the request 
    response = client.chat_postMessage(channel=body["event"]["channel"], 
                                       thread_ts=body["event"]["event_ts"],
                                       text=f"Hello LazyPay junkies !! :robot_face: \nThanks for your request, I'm on it!")
    # Check ChatGPT
    openai.api_key = OPENAI_API_KEY
    response = openai.Completion.create(
        engine="text-davinci-003",
        prompt=prompt,
        max_tokens=1024,
        n=1,
        stop=None,
        temperature=0.5).choices[0].text
    # Reply to thread 
    response = client.chat_postMessage(channel=body["event"]["channel"], 
                                       thread_ts=body["event"]["event_ts"],
                                       text=f"Here you go: \n{response}")
if __name__ == "__main__":
    SocketModeHandler(app, SLACK_APP_TOKEN).start()

Add here we go !! Add your bot to a channel in the integration tab. 🚀

>1 RDBMS in Spring Data JPA

Pranjal Sharma — Thu, 20 Jun 2024 15:34:35 +0000

This document deals with building the backend application that uses Spring Data JPA with multiple relational databases.
For an example we will connect to MySQL + MSSQL database.

Main task here is to seperate properties and configurations for all the multiple databases that have to integrated.

Other JPA layers in code remain the same as for single integration. [ Repository + Entity]

[ Point to remember : Define these in different packages for different databases as we would need them when defining configs. ]

For specific code refer this.

Sample Configurations in application properties

spring.datasource.url=jdbc:mysql://127.0.0.1/heimdall_db?useSSL=false
spring.datasource.username=root
spring.datasource.password=pranjal
spring.datasource.driverClassName=com.mysql.cj.jdbc.Driver

##SQL Server
sqlserver.datasource.url=jdbc:sqlserver://localhost;databaseName=jpa_test
sqlserver.datasource.username=sa
sqlserver.datasource.password=reallyStrongPwd123
sqlserver.datasource.driverClassName=com.microsoft.sqlserver.jdbc.SQLServerDriver

spring.jpa.database=default

Don't define other hibernate configurations specific to database here.

Defining Separate Config Classes for all the databases

package com.sma.backend.multidb.config;

import org.springframework.beans.factory.annotation.Qualifier;
import org.springframework.boot.autoconfigure.jdbc.DataSourceProperties;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.boot.orm.jpa.EntityManagerFactoryBuilder;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.jpa.repository.config.EnableJpaRepositories;
import org.springframework.orm.jpa.JpaTransactionManager;
import org.springframework.orm.jpa.LocalContainerEntityManagerFactoryBean;
import org.springframework.transaction.PlatformTransactionManager;
import org.springframework.transaction.annotation.EnableTransactionManagement;


import javax.persistence.EntityManagerFactory;
import javax.sql.DataSource;

@Configuration
@EnableTransactionManagement
@EnableJpaRepositories(entityManagerFactoryRef = "sqlServerEntityManagerFactory",
        transactionManagerRef = "sqlServerTransactionManager",
        basePackages = "com.sma.backend.multidb.database.sqlserver.repository")
public class SqlServerConfig {

    @Bean
    @ConfigurationProperties(prefix = "sqlserver.datasource")
    public DataSourceProperties sqlServerDataSourceProperties() {
        return new DataSourceProperties();
    }
    @Bean
    public DataSource sqlServerDataSource(@Qualifier("sqlServerDataSourceProperties") DataSourceProperties dataSourceProperties) {
        return dataSourceProperties.initializeDataSourceBuilder().build();
    }

    @Bean(name = "sqlServerEntityManagerFactory")
    public LocalContainerEntityManagerFactoryBean sqlServerEntityManagerFactory(@Qualifier("sqlServerDataSource") DataSource sqlServerDataSource, EntityManagerFactoryBuilder builder) {

        return builder.dataSource(sqlServerDataSource)
                .packages("com.sma.backend.multidb.database.sqlserver.domain")
                .persistenceUnit("sqlserver")
                .build();

    }

    @Bean
    public PlatformTransactionManager sqlServerTransactionManager(@Qualifier("sqlServerEntityManagerFactory")
                                                                              EntityManagerFactory factory) {
        return new JpaTransactionManager(factory);
    }
}

MySqlConfig

package com.sma.backend.multidb.config;

@Configuration
@EnableTransactionManagement
@EnableJpaRepositories(entityManagerFactoryRef = "mysqlEntityManagerFactory", transactionManagerRef = "mysqlTransactionManager", basePackages = {"com.sma.backend.multidb.database.mysql.repository"})

public class MySqlConfig {

    @Primary
    @Bean
    @ConfigurationProperties(prefix = "spring.datasource")
    public DataSourceProperties mysqlDataSourceProperties() {
        return new DataSourceProperties();
    }

    @Primary
    @Bean
    public DataSource mysqlDataSource(@Qualifier("mysqlDataSourceProperties") DataSourceProperties dataSourceProperties) {
        return dataSourceProperties.initializeDataSourceBuilder().build();
    }

    @Primary
    @Bean
    public LocalContainerEntityManagerFactoryBean mysqlEntityManagerFactory(@Qualifier("mysqlDataSource") DataSource hubDataSource, EntityManagerFactoryBuilder builder) {
        return builder.dataSource(hubDataSource).packages("com.sma.backend.multidb.database.mysql.domain")
                .persistenceUnit("mysql").build();
    }

    @Primary
    @Bean
    public PlatformTransactionManager mysqlTransactionManager(@Qualifier("mysqlEntityManagerFactory") EntityManagerFactory factory) {
        return new JpaTransactionManager(factory);
    }

}

POINTS TO REMEMBER :

hibernate.dialect → The dialect specifies the type of database used in hibernate so that hibernate generate appropriate type of SQL statements. For connecting any hibernate application with the database, it is required to provide the configuration of SQL dialect.

Hence to specify which language to use we have to define seperate values for this.

We can do that by passing this and all other properties which are specific to the databases in a map tagged as properties in EntityManagerFactoryBuilder

If running on Mac local you have to keep different ports for running both the databases on localHost as MSSQL needs Docker to run .

I hope that this Blog Post helped you! If you have any questions, feel free to use the comment section! 💬

Oh and if you want more content like this, follow me: