DEV Community: Neeraj Kumar

Day 2: Advanced SQL Preparation Guide

Neeraj Kumar — Thu, 05 Feb 2026 01:30:08 +0000

Objectives

Select and download suitable datasets
Set up PostgreSQL environment
Practice basic queries as foundation
Prepare for CTEs, window functions, and query optimization

Part 1: Dataset Selection

Recommended Datasets

Dataset 1: E-commerce Sales Data (Primary)

Source: Kaggle - "Superstore Sales Dataset"

URL: https://www.kaggle.com/datasets/vivek468/superstore-dataset-final
Why it's ideal:
- Multiple tables (orders, customers, products)
- Time-series data (perfect for window functions)
- Sales metrics (great for aggregations and CTEs)
- ~10,000 rows (manageable size)

Key columns: Order ID, Customer ID, Product, Category, Sales, Profit, Quantity, Order Date, Ship Date

Advanced SQL use cases:

Running totals and moving averages (window functions)
Customer cohort analysis (CTEs)
Sales ranking by region/category (RANK, DENSE_RANK)
Year-over-year comparisons

Dataset 2: Employee/HR Data (Secondary)

Source: Kaggle - "HR Analytics Dataset"

URL: https://www.kaggle.com/datasets/pavansubhasht/ibm-hr-analytics-attrition-dataset
Why it's useful:
- Employee hierarchy (good for recursive CTEs)
- Salary and performance data
- Attrition tracking

Advanced SQL use cases:

Employee tenure analysis
Salary percentiles by department
Retention rate calculations
Manager-employee relationships

Alternative Dataset Options:

Option A: COVID-19 Data

Source: Our World in Data
URL: https://github.com/owid/covid-19-data/tree/master/public/data
Time-series perfect for window functions
Country comparisons

Option B: Airbnb Listings

Source: Inside Airbnb
URL: http://insideairbnb.com/get-the-data/
Price analysis, availability patterns
Geographic data for spatial queries

Part 2: PostgreSQL Setup

Installation

On macOS:

# Using Homebrew
brew install postgresql@15

# Start PostgreSQL service
brew services start postgresql@15

# Verify installation
psql --version

On Ubuntu/Debian:

# Update package list
sudo apt update

# Install PostgreSQL
sudo apt install postgresql postgresql-contrib

# Start service
sudo systemctl start postgresql
sudo systemctl enable postgresql

# Switch to postgres user
sudo -u postgres psql

On Windows:

Download installer from: https://www.postgresql.org/download/windows/
Run the installer (includes pgAdmin 4)
Remember your superuser password
Add PostgreSQL bin to PATH

Initial Configuration

-- Connect to PostgreSQL
psql -U postgres

-- Create a database for practice
CREATE DATABASE data_engineering_practice;

-- Connect to the new database
\c data_engineering_practice

-- Create a user (optional)
CREATE USER data_engineer WITH PASSWORD 'your_password';
GRANT ALL PRIVILEGES ON DATABASE data_engineering_practice TO data_engineer;

Useful PostgreSQL Commands

-- List all databases
\l

-- List all tables in current database
\dt

-- Describe table structure
\d table_name

-- List all schemas
\dn

-- Quit psql
\q

-- Execute SQL from file
\i /path/to/file.sql

-- Show query execution time
\timing on

Part 3: Loading Your Data

Method 1: Using COPY (Fastest for CSV)

-- Create table structure first
CREATE TABLE superstore_sales (
    row_id SERIAL PRIMARY KEY,
    order_id VARCHAR(50),
    order_date DATE,
    ship_date DATE,
    ship_mode VARCHAR(50),
    customer_id VARCHAR(50),
    customer_name VARCHAR(100),
    segment VARCHAR(50),
    country VARCHAR(50),
    city VARCHAR(100),
    state VARCHAR(50),
    postal_code VARCHAR(20),
    region VARCHAR(50),
    product_id VARCHAR(50),
    category VARCHAR(50),
    sub_category VARCHAR(50),
    product_name VARCHAR(255),
    sales DECIMAL(10,2),
    quantity INTEGER,
    discount DECIMAL(4,2),
    profit DECIMAL(10,2)
);

-- Load CSV data
COPY superstore_sales(order_id, order_date, ship_date, ship_mode, 
                      customer_id, customer_name, segment, country, 
                      city, state, postal_code, region, product_id, 
                      category, sub_category, product_name, sales, 
                      quantity, discount, profit)
FROM '/path/to/superstore.csv'
DELIMITER ','
CSV HEADER;

Method 2: Using pgAdmin GUI

Right-click on table → Import/Export
Select file and format
Map columns
Import

Method 3: Using Python (for data cleaning)

import pandas as pd
from sqlalchemy import create_engine

# Read and clean data
df = pd.read_csv('superstore.csv')

# Connect to PostgreSQL
engine = create_engine('postgresql://username:password@localhost:5432/data_engineering_practice')

# Load to database
df.to_sql('superstore_sales', engine, if_exists='replace', index=False)

Part 4: Basic SQL Practice Queries

Query Set 1: Basic SELECT and Filtering

-- 1. View first 10 rows
SELECT * FROM superstore_sales LIMIT 10;

-- 2. Count total orders
SELECT COUNT(DISTINCT order_id) as total_orders 
FROM superstore_sales;

-- 3. Filter by category
SELECT * FROM superstore_sales 
WHERE category = 'Technology' 
LIMIT 20;

-- 4. Multiple conditions
SELECT customer_name, sales, profit 
FROM superstore_sales 
WHERE region = 'West' 
  AND sales > 1000 
  AND order_date >= '2017-01-01';

-- 5. Pattern matching
SELECT DISTINCT product_name 
FROM superstore_sales 
WHERE product_name LIKE '%Chair%';

Query Set 2: Aggregations

-- 6. Total sales by category
SELECT 
    category,
    SUM(sales) as total_sales,
    SUM(profit) as total_profit,
    COUNT(*) as order_count
FROM superstore_sales
GROUP BY category
ORDER BY total_sales DESC;

-- 7. Average discount by segment
SELECT 
    segment,
    AVG(discount) as avg_discount,
    MIN(discount) as min_discount,
    MAX(discount) as max_discount
FROM superstore_sales
GROUP BY segment;

-- 8. Monthly sales trend
SELECT 
    DATE_TRUNC('month', order_date) as month,
    SUM(sales) as monthly_sales,
    COUNT(DISTINCT order_id) as order_count
FROM superstore_sales
GROUP BY DATE_TRUNC('month', order_date)
ORDER BY month;

-- 9. HAVING clause
SELECT 
    customer_name,
    SUM(sales) as total_spent
FROM superstore_sales
GROUP BY customer_name
HAVING SUM(sales) > 5000
ORDER BY total_spent DESC;

Query Set 3: JOIN Preparation

-- Create dimension tables for practice

-- Customer dimension
CREATE TABLE customers AS
SELECT DISTINCT
    customer_id,
    customer_name,
    segment,
    country,
    city,
    state,
    region
FROM superstore_sales;

-- Product dimension
CREATE TABLE products AS
SELECT DISTINCT
    product_id,
    product_name,
    category,
    sub_category
FROM superstore_sales;

-- Fact table (orders)
CREATE TABLE orders AS
SELECT 
    row_id,
    order_id,
    order_date,
    ship_date,
    ship_mode,
    customer_id,
    product_id,
    sales,
    quantity,
    discount,
    profit
FROM superstore_sales;

-- 10. INNER JOIN
SELECT 
    o.order_id,
    c.customer_name,
    p.product_name,
    o.sales
FROM orders o
INNER JOIN customers c ON o.customer_id = c.customer_id
INNER JOIN products p ON o.product_id = p.product_id
LIMIT 100;

-- 11. LEFT JOIN with NULL check
SELECT 
    c.customer_name,
    COUNT(o.order_id) as order_count,
    COALESCE(SUM(o.sales), 0) as total_sales
FROM customers c
LEFT JOIN orders o ON c.customer_id = o.customer_id
GROUP BY c.customer_name
ORDER BY order_count DESC;

Query Set 4: Subqueries (Foundation for CTEs)

-- 12. Subquery in WHERE
SELECT * FROM superstore_sales
WHERE sales > (SELECT AVG(sales) FROM superstore_sales);

-- 13. Subquery in FROM
SELECT 
    category,
    avg_sales,
    CASE 
        WHEN avg_sales > overall_avg THEN 'Above Average'
        ELSE 'Below Average'
    END as performance
FROM (
    SELECT 
        category,
        AVG(sales) as avg_sales
    FROM superstore_sales
    GROUP BY category
) cat_avg
CROSS JOIN (
    SELECT AVG(sales) as overall_avg 
    FROM superstore_sales
) overall;

-- 14. Correlated subquery
SELECT 
    s1.category,
    s1.product_name,
    s1.sales
FROM superstore_sales s1
WHERE s1.sales = (
    SELECT MAX(s2.sales)
    FROM superstore_sales s2
    WHERE s2.category = s1.category
);

Part 5: Preview of Advanced Concepts

Window Functions Teaser

-- Running total of sales over time
SELECT 
    order_date,
    sales,
    SUM(sales) OVER (ORDER BY order_date) as running_total
FROM superstore_sales
ORDER BY order_date
LIMIT 50;

-- Rank products by sales within each category
SELECT 
    category,
    product_name,
    SUM(sales) as total_sales,
    RANK() OVER (PARTITION BY category ORDER BY SUM(sales) DESC) as rank_in_category
FROM superstore_sales
GROUP BY category, product_name
ORDER BY category, rank_in_category
LIMIT 30;

CTEs (Common Table Expressions) Teaser

-- Using WITH clause
WITH monthly_sales AS (
    SELECT 
        DATE_TRUNC('month', order_date) as month,
        SUM(sales) as total_sales
    FROM superstore_sales
    GROUP BY DATE_TRUNC('month', order_date)
)
SELECT 
    month,
    total_sales,
    LAG(total_sales) OVER (ORDER BY month) as previous_month,
    total_sales - LAG(total_sales) OVER (ORDER BY month) as month_over_month_change
FROM monthly_sales
ORDER BY month;

Part 6: Practice Exercises

Exercise 1: Data Exploration

How many unique customers are there?
What date range does the dataset cover?
Which region has the highest total sales?
What's the average order value?

Exercise 2: Business Questions

Find the top 10 customers by total profit
Which product category has the highest profit margin?
What's the average delivery time (ship_date - order_date)?
Which month had the highest sales in 2017?

Exercise 3: Complex Queries

Find customers who made purchases in all four regions
Calculate the percentage of sales each category contributes
Find products that were never discounted
Identify customers whose average order value is above the overall average

Exercise 4: Data Quality Checks

-- Check for NULL values
SELECT 
    COUNT(*) as total_rows,
    COUNT(order_id) as non_null_orders,
    COUNT(customer_id) as non_null_customers,
    COUNT(sales) as non_null_sales
FROM superstore_sales;

-- Check for duplicates
SELECT 
    order_id, 
    product_id, 
    COUNT(*) 
FROM superstore_sales
GROUP BY order_id, product_id
HAVING COUNT(*) > 1;

-- Check data ranges
SELECT 
    MIN(order_date) as earliest_order,
    MAX(order_date) as latest_order,
    MIN(sales) as min_sale,
    MAX(sales) as max_sale,
    AVG(sales) as avg_sale
FROM superstore_sales;

Part 7: Performance Tips

Creating Indexes

-- Create indexes on frequently queried columns
CREATE INDEX idx_order_date ON superstore_sales(order_date);
CREATE INDEX idx_customer_id ON superstore_sales(customer_id);
CREATE INDEX idx_category ON superstore_sales(category);
CREATE INDEX idx_region ON superstore_sales(region);

-- Composite index for common query patterns
CREATE INDEX idx_customer_date ON superstore_sales(customer_id, order_date);

EXPLAIN and Query Analysis

-- See query execution plan
EXPLAIN SELECT * FROM superstore_sales WHERE order_date > '2017-01-01';

-- See actual execution stats
EXPLAIN ANALYZE 
SELECT category, SUM(sales) 
FROM superstore_sales 
GROUP BY category;

Part 8: Next Steps Checklist

[ ] Install PostgreSQL
[ ] Download 1-2 datasets from recommended sources
[ ] Create database and load data
[ ] Complete all basic query exercises
[ ] Create indexes on key columns
[ ] Try the window function and CTE teasers
[ ] Document any questions for the advanced session

Resources

Documentation

PostgreSQL Official Docs: https://www.postgresql.org/docs/
PostgreSQL Tutorial: https://www.postgresqltutorial.com/
Mode Analytics SQL Tutorial: https://mode.com/sql-tutorial/

Practice Platforms

SQLZoo: https://sqlzoo.net/
LeetCode SQL: https://leetcode.com/problemset/database/
HackerRank SQL: https://www.hackerrank.com/domains/sql

Tools

pgAdmin: PostgreSQL GUI client
DBeaver: Universal database tool
DataGrip: JetBrains SQL IDE

Quick Reference Card

-- CONNECTION
psql -U username -d database_name

-- COMMON AGGREGATES
COUNT(), SUM(), AVG(), MIN(), MAX()

-- DATE FUNCTIONS
DATE_TRUNC('month', date_column)
EXTRACT(YEAR FROM date_column)
AGE(date1, date2)

-- STRING FUNCTIONS
CONCAT(), UPPER(), LOWER(), SUBSTRING()
LIKE, ILIKE (case-insensitive)

-- CONDITIONAL
CASE WHEN ... THEN ... ELSE ... END
COALESCE(value1, value2, default)
NULLIF(value1, value2)

-- GROUPING
GROUP BY, HAVING
GROUP BY ROLLUP, CUBE, GROUPING SETS

Good luck with your Day 2 preparation! 🚀

Data Engineering Fundamentals & Roadmap (2026 Edition)

Neeraj Kumar — Wed, 04 Feb 2026 02:07:36 +0000

📌 Today's Objective

Build a strong understanding of data engineering fundamentals, its scope, and a clear career roadmap for aspiring data engineers.

🔍 1. What is Data Engineering?

Definition:

Data Engineering is the practice of designing, building, and maintaining systems that collect, store, process, and analyze data at scale.

Key Differentiators:

Data Science → Asking questions from data and building models
Data Engineering → Building the infrastructure for data to flow reliably
Data Analytics → Transforming data into actionable business insights

🎯 2. The Data Engineering Hierarchy of Needs

       ┌─────────────────────┐
       │   Data Products     │ ← Machine Learning, Analytics, BI
       └─────────────────────┘
                ↓
       ┌─────────────────────┐
       │   Analytics / ML    │ ← Aggregations, Training, Predictions
       └─────────────────────┘
                ↓
       ┌─────────────────────┐
       │    Transform        │ ← Cleaning, Enrichment, Validation
       └─────────────────────┘
                ↓
       ┌─────────────────────┐
       │      Store          │ ← Databases, Data Lakes, Warehouses
       └─────────────────────┘
                ↓
       ┌─────────────────────┐
       │      Move           │ ← Pipelines, Ingestion, ETL/ELT
       └─────────────────────┘
                ↓
       ┌─────────────────────┐
       │     Collect         │ ← APIs, Databases, Streaming, Files
       └─────────────────────┘

⚙️ 3. Core Pillars of Data Engineering

A. Data Storage & Databases

OLTP Databases: PostgreSQL, MySQL, SQL Server
OLAP Databases: ClickHouse, Apache Druid, DuckDB
Data Warehouses: Snowflake, BigQuery, Redshift, Databricks SQL
Data Lakes: Amazon S3, Azure Data Lake Storage (ADLS) with Delta Lake, Apache Iceberg, Apache Hudi
NoSQL: MongoDB, Cassandra, DynamoDB, Redis

B. Data Processing

Batch Processing: Apache Spark, AWS Glue, Google Dataflow, Databricks
Stream Processing: Apache Kafka, Apache Flink, Spark Streaming, AWS Kinesis
Orchestration: Apache Airflow, Dagster, Prefect, Mage

C. Data Modeling

Star Schema & Snowflake Schema (dimensional modeling)
Data Vault 2.0 (enterprise data warehousing)
Medallion Architecture (Bronze → Silver → Gold layers)
One Big Table (OBT) approach for analytics

D. Infrastructure & DevOps

Cloud Platforms: AWS, Azure, GCP
Infrastructure as Code (IaC): Terraform, Pulumi, CloudFormation
Containers: Docker, Kubernetes
CI/CD: GitHub Actions, Jenkins, GitLab CI, CircleCI

📊 4. Data Engineer Role Types

Role Type	Focus Area	Key Technologies
Pipeline Engineer	ETL/ELT, Data Movement	Airflow, dbt, Fivetran, Airbyte
Platform Engineer	Infrastructure & Tooling	Kubernetes, Terraform, AWS/GCP
Analytics Engineer	Data Modeling & Transformation	SQL, dbt, Looker, Tableau
MLOps Engineer	ML Pipelines & Serving	Kubeflow, MLflow, SageMaker, Vertex AI

🎯 5. 30-Day Learning Roadmap

Week 1: Foundations

Day 1: Core Concepts & Roadmap Overview
Day 2: Advanced SQL (CTEs, Window Functions, Optimization)
Day 3: Python for Data Engineering (Pandas, APIs, Data Structures)
Day 4: Linux & Shell Scripting Basics
Day 5: Git & Version Control Best Practices
Day 6: Docker Fundamentals & Containerization
Day 7: Cloud Basics (AWS/GCP/Azure Introduction)

Week 2: Storage & Processing

Days 8-9: Databases, Data Warehousing, and Data Lakes
Days 10-11: PySpark & Batch Processing
Days 12-14: Kafka & Real-time Streaming Basics

Week 3: Orchestration & Pipelines

Days 15-16: ETL vs ELT Patterns
Days 17-19: Apache Airflow (Basics → Advanced DAGs)
Days 20-21: dbt & Modern Data Stack Integration
Day 21: Data Quality, Monitoring & Alerting (Great Expectations, Soda)

Week 4: Advanced Topics & Projects

Days 22-23: Data Modeling & Query Optimization
Days 24-25: Cost Optimization & Infrastructure as Code
Days 26-27: CI/CD for Data Pipelines
Days 28-30: End-to-End Project & Interview Preparation

💼 6. Industry Expectations (Entry-Level)

Technical Skills

SQL: Window functions, CTEs, query optimization, indexing strategies
Python: Pandas, data manipulation, API integration, OOP concepts
Cloud: S3, IAM, Lambda/Cloud Functions, basic networking
Big Data: Spark fundamentals, distributed computing concepts
Version Control: Git workflows, branching strategies, pull requests

Conceptual Knowledge

Data modeling principles and normalization
ETL/ELT pipeline design patterns
Data quality and testing frameworks
Distributed systems fundamentals
Data governance and security basics

✅ 7. Day 1 Action Items

Immediate Setup

[ ] Install Python 3.10+ (latest stable version)
[ ] Install Docker Desktop
[ ] Create GitHub account and configure SSH keys
[ ] Set up VS Code with extensions: Python, Docker, SQL, GitLens
[ ] Install PostgreSQL locally or use Docker

Learning & Career Positioning

[ ] Watch: "What is Data Engineering?" overview (15-20 mins)
[ ] Read: Fundamentals of Data Engineering – Chapter 1
[ ] Update LinkedIn headline: "Aspiring Data Engineer | Learning Python, SQL & Cloud"
[ ] Follow 5 data engineering professionals on LinkedIn/Twitter
[ ] Join data engineering communities: Data Engineering Slack, Reddit r/dataengineering

Documentation

[ ] Create /data-engineering-30days project folder
[ ] Start Day 1 learning notes in Markdown format
[ ] Initialize Git repository with proper .gitignore
[ ] Set up a learning journal template

📚 8. Recommended Resources

Free Learning Platforms

Courses: DataCamp Data Engineer Track, Coursera Data Engineering, freeCodeCamp
Books:
- Fundamentals of Data Engineering by Joe Reis & Matt Housley
- Designing Data-Intensive Applications by Martin Kleppmann
Practice: LeetCode (SQL), HackerRank (Python), StrataScratch

Certifications (Optional but Valuable)

AWS Certified Data Analytics – Specialty
Google Professional Data Engineer
Azure Data Engineer Associate (DP-203)
Databricks Certified Data Engineer Associate

Communities & Blogs

Seattle Data Guy (YouTube)
Data Engineering Weekly Newsletter
Locally Optimistic Blog
dbt Community Slack

🚨 9. Common Pitfalls to Avoid

Tool obsession over fundamentals – Master SQL and Python first
Ignoring SQL – Still represents 70%+ of daily work
Delaying cloud platform learning – Cloud skills are essential today
Theory without projects – Build real pipelines, not just tutorials
Learning in isolation – Engage with communities and seek feedback
Skipping data modeling – Understanding schemas is crucial

📈 10. Success Metrics

Week	Target Outcome
Week 1	Local development environment, strong SQL & Python
Week 2	First ETL pipeline with cloud storage integration
Week 3	Orchestrated Airflow pipeline with data quality checks
Week 4	Deployed end-to-end project with GitHub portfolio

➡️ Next Steps (Day 2)

Complete all Day 1 action items
Prepare for Advanced SQL session (CTEs, window functions, query optimization)
Select 1–2 datasets from Kaggle, Google Dataset Search, or public APIs
Set up PostgreSQL and practice basic queries

💡 Key Takeaway

"Data engineering isn't about knowing every tool—it's about understanding which tool solves which problem and why."

Remember: Consistency beats intensity.

2 focused hours daily > 8 hours of weekend cramming.

📅 Document Version

Last Updated: February 2026
Next Review: May 2026
Maintained By: Neeraj Kumar

Good luck on your data engineering journey! 🚀

golang

Neeraj Kumar — Wed, 04 Dec 2024 02:20:03 +0000

Two Sum Problem’ on LeetCode

Neeraj Kumar — Tue, 12 Nov 2024 18:20:38 +0000

Problem Description
Given an array of integers nums and an integer target, return the indices of the two numbers that add up to the target.

Go Function Signature:
func twoSum(nums []int, target int) []int
Example 1:
Input: nums = [2,7,11,15], target = 9
Output: [0,1]
Explanation: Because nums[0] + nums[1] == 9, we return [0, 1].
Example 2:

Input: nums = [3,2,4], target = 6
Output: [1,2]
Example 3:

nput: nums = [3,3], target = 6
Output: [0,1]
Solution 1: Brute Force Approach

Solution 1: Brute-Force Approach (Nested Loops)
In this approach, you check each pair of elements to see if they sum up to the target. This involves iterating through the array with two nested loops.

Code

func twoSum(nums []int, target int) []int {
    for i := 0; i < len(nums); i++ {
        for j := i + 1; j < len(nums); j++ {
            if nums[i] + nums[j] == target {
                return []int{i, j}
            }
        }
    }
    return nil
}

Complexity Analysis

Solution 2: Two-Pass Hash Table
This solution improves on the brute-force approach by using a hash map to store each element's value and index in the first pass. In the second pass, you check if the complement (i.e., target - num) exists in the hash map.

Code

func twoSum(nums []int, target int) []int {
    numMap := make(map[int]int)
    // First pass: populate the map with each element's index
    for i, num := range nums {
        numMap[num] = i
    }
    // Second pass: check for the complement
    for i, num := range nums {
        if j, ok := numMap[target - num]; ok && i != j {
            return []int{i, j}
        }
    }
    return nil
}

Solution 3: One-Pass Hash Table (Optimal Solution)

This approach combines both insertion and lookup in a single pass. As you iterate through the array, you:
Check if the complement (i.e., target - num) exists in the hash map.
If the complement is found, return the indices.
If not, add the current element and its index to the hash map for future lookups.
Code

func twoSum(nums []int, target int) []int {
    numMap := make(map[int]int)
    for i, num := range nums {
        if j, ok := numMap[target - num]; ok {
            return []int{j, i}
        }
        numMap[num] = i
    }
    return nil
}

Complexity Analysis

Time Complexity: 𝑂(𝑛)
- Only one pass through the array is required, making this approach linear in time complexity.
Space Complexity:o(n)
- The hash map stores each element and its index.

Pros and Cons
Pros: The most efficient approach for time complexity, with a clean and compact implementation.
Cons: None, as it achieves optimal time and space complexity for this problem.

Work with Apache Kafka in Spring Boot

Neeraj Kumar — Sat, 31 Aug 2024 19:07:03 +0000

Step 1: Create a New Spring Boot Starter Project

creating a new Spring Boot Starter Project using STS. While configuring the project, select Spring Web, Spring for Apache Kafka, and Spring Boot DevTools as dependencies
Step 2: Enable Kafka in the Main Class

To integrate Apache Kafka with Spring Boot,

package com.dev.spring.kafka;

import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.kafka.annotation.EnableKafka;

@SpringBootApplication
@EnableKafka
public class SpringBoot2ApacheKafkaTestApplication {
    public static void main(String[] args) {
        SpringApplication.run(SpringBoot2ApacheKafkaTestApplication.class, args);
    }
}

Step 3: Create a Custom MessageRepository Class

Next, create a MessageRepository class to store incoming messages.

package com.dev.spring.kafka.message.repository;

import java.util.ArrayList;
import java.util.List;
import org.springframework.stereotype.Component;

@Component
public class MessageRepository {

    private List<String> list = new ArrayList<>();

    public void addMessage(String message) {
        list.add(message);
    }

    public String getAllMessages() {
        return list.toString();
    }
}

Step 4: Create a MessageProducer Class

Create a MessageProducer class to send messages to the Kafka topic.

package com.dev.spring.kafka.sender;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.stereotype.Component;

@Component
public class MessageProducer {

    private Logger log = LoggerFactory.getLogger(MessageProducer.class);

    @Autowired 
    private KafkaTemplate<String, String> kafkaTemplate;

    @Value("${myapp.kafka.topic}")
    private String topic;

    public void sendMessage(String message) {
        log.info("MESSAGE SENT FROM PRODUCER END -> " + message);
        kafkaTemplate.send(topic, message);
    }
}

Step 5: Create a MessageConsumer Class

Now, create a MessageConsumer class to consume messages from the Kafka topic.

package com.dev.spring.kafka.consumer;

import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Component;
import com.dev.spring.kafka.message.repository.MessageRepository;

@Component
public class MessageConsumer {

    private Logger log = LoggerFactory.getLogger(MessageConsumer.class);

    @Autowired
    private MessageRepository messageRepo;

    @KafkaListener(topics = "${myapp.kafka.topic}", groupId = "xyz")
    public void consume(String message) {
        log.info("MESSAGE RECEIVED AT CONSUMER END -> " + message);
        messageRepo.addMessage(message);
    }
}

Step 6: Create a KafkaRestController Class

Finally, create a KafkaRestController class to handle REST requests for sending and retrieving messages.

package com.dev.spring.kafka.controller;

import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.GetMapping;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;
import com.dev.spring.kafka.message.repository.MessageRepository;
import com.dev.spring.kafka.sender.MessageProducer;

@RestController
public class KafkaRestController {

    @Autowired
    private MessageProducer producer;

    @Autowired
    private MessageRepository messageRepo;

    // Send message to Kafka
    @GetMapping("/send")
    public String sendMsg(@RequestParam("msg") String message) {
        producer.sendMessage(message);
        return "'" + message + "' sent successfully!";
    }

    // Read all messages
    @GetMapping("/getAll")
    public String getAllMessages() {
        return messageRepo.getAllMessages();
    }
}

Step 7: Create the application.yml File

Lastly, configure your application by creating an application.yml

server:
  port: 9090

spring:
  kafka:
    producer:
      bootstrap-servers: localhost:9092
      key-serializer: org.apache.kafka.common.serialization.StringSerializer
      value-serializer: org.apache.kafka.common.serialization.StringSerializer

    consumer:
      bootstrap-servers: localhost:9092
      key-deserializer: org.apache.kafka.common.serialization.StringDeserializer
      value-deserializer: org.apache.kafka.common.serialization.StringDeserializer

myapp:
  kafka:
    topic: myKafkaTest

Kubernetes Cluster Step By Step

Neeraj Kumar — Mon, 01 Jan 2024 13:31:28 +0000

A Kubernetes cluster is a group of nodes or machines running together. At the highest level of Kubernetes, there are two types of servers: a Master node and Worker nodes. These servers can be either Virtual Machines (VMs) or physical servers (Bare metal). Together, these servers form a Kubernetes cluster, and they are controlled by the services that make up the Control Plane.

Prerequisites:

Choose a Cloud Provider: Popular choices include AWS, Google Cloud Platform (GCP), Microsoft Azure, and others. Alternatively, you can use on-premises solutions like VMware or tools like Minikube for local development.

Install Necessary Tools: kubectl: Kubernetes command-line tool kubeadm, kubelet, and kubernetes-cni: Install these on each cluster node.

Step 1: Provision the Infrastructure
On Cloud Providers:

Create VM instances or nodes based on your chosen provider.

Ensure that each node has a compatible OS (Ubuntu, CentOS, etc.).

On-Premises or Local:

```
Set up physical or virtual machines.
```

Ensure network connectivity between nodes.

Step 2: Install Docker (or Another Container Runtime)
Install Docker on each node or use an alternative container runtime:

# For Ubuntu
sudo apt update
sudo apt install docker.io
sudo systemctl enable --now docker

Step 3: Install kubeadm, kubelet, and kubectl

# For Ubuntu
sudo apt-get update && sudo apt-get install -y apt-transport-https curl
curl -s https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
echo "deb https://apt.kubernetes.io/ kubernetes-xenial main" | sudo tee /etc/apt/sources.list.d/kubernetes.list
sudo apt-get update
sudo apt-get install -y kubelet kubeadm kubectl
sudo systemctl enable --now kubelet

Step 4: Initialize the Master Node

sudo kubeadm init --pod-network-cidr=10.244.0.0/16

Step 5: Set Up Cluster Networking
Choose a network plugin for your cluster. For example, Calico or Flannel. Install the chosen plugin on the master node:

kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml

Step 6: Join Worker Nodes
run the kubeadm join command provided at the end of the master node initialization.

sudo kubeadm join <master-node-ip>:<master-node-port> --token <token> --discovery-token-ca-cert-hash <hash>

Step 7: Verify Cluster Setup

On the master node, run:

kubectl get nodes
kubectl get pods --all-namespaces

Step 8: Deploy an Application
Deploy a sample application to test your cluster:

kubectl create deployment nginx --image=nginx
kubectl expose deployment nginx --port=80 --type=NodePort

Step 9: Access Your Application
Retrieve the NodePort and access the deployed application:

kubectl get svc

Visit http://node-ip:node-port in your web browser.

Pagination in Golang and MongoDB

Neeraj Kumar — Wed, 23 Aug 2023 19:02:23 +0000

MongoDB is a document based data store and hence pagination is one of the most common use case of it. So when do you paginate the response? The answer is pretty neat; you paginate whenever you want to process result in chunks. Some common scenarios are

Batch processing
Showing huge set of results on user interfac

Paginating on client and server side are both really very expensive and should not be considered. Hence pagination is generally handled at database level and databases are optimized for such needs to

2 approaches through which you can easily paginate your MongoDB responses. Sample Document

 {
        "_id" : ObjectId("6936d17263623919cd5145db"),
        "name" : "Neeraj Kumar",
        "age" : 25
    }

Approach 1: Using cursor.skip and cursor.limit
MongoDB cursor has two methods that makes paging easy; they are

cursor.skip()
cursor.limit()

skip(n) will skip n documents from the cursor while limit(n) will cap the number of documents to be returned from the cursor. Thus combination of two naturally paginates the response.
In Mongo Shell your pagination code looks something like

 // Page 1
    db.students.find().limit(10)

    // Page 2
    db.students.find().skip(10).limit(10)

    // Page 3
    db.students.find().skip(10).limit(10)

implement pagination:

func GetPagination(limit, page int) error {
  ctx, cancel := context.WithTimeout(context.Background(), 12*time.Second)
  defer cancel()
   coll := o.db.Database(mongoDatabaseName).Collection(offerCollectionName)

   l := int64(limit)
     skip := int64(page * limit - limit)
     fOpt := options.FindOptions{Limit: &l, Skip: &skip}

     curr, err := coll.Find(ctx, bson.D{{}}, &fOpt)
   if err != nil {
      return result, err
   }

   for curr.Next(ctx) {
      var el Offer
      if err := curr.Decode(&el); err != nil {
         log.Println(err)
      }

      result = append(result, el)
   }

   return result, nil
}

Approach 2: Using _id and limit

This approach will make effective use of default index on _id and nature of ObjectId. I bet you didn’t know that a Mongodb ObjectId is a 12 byte structure containing

Using this property of ObjectId and also taking into consideration the fact that _id is always indexed, we can devise following approach for pagination:

Fetch a page of documents from database
Get the document id of the last document of the page
Retrieve documents greater than that id

    // Page 1
    db.students.find().limit(10)

    // Page 2
    last_id = ...  # logic to get last_id
    db.students.find({'_id': {'$gt': last_id}}).limit(10)

    // Page 3
    last_id = ... # logic to get last_id
    db.students.find({'_id': {'$gt': last_id}}).limit(10)

func GetPagination(limit, page int)  error {
    ctx, cancel := context.WithTimeout(context.Background(), 12*time.Second)
  defer cancel()

   coll := o.db.Database(mongoDatabaseName).Collection(offerCollectionName)
   ctx, _ := context.WithTimeout(context.Background(), contextTimeout)

   curr, err := coll.Find(ctx, bson.D{{}}, newMongoPaginate(limit,page).getPaginatedOpts())
   if err != nil {
      return result, err
   }

   for curr.Next(ctx) {
      var el Offer
      if err := curr.Decode(&el); err != nil {
         log.Println(err)
      }

      result = append(result, el)
   }

   return result, nil
}

Hello World in Golang

Neeraj Kumar — Sat, 25 Feb 2023 18:05:11 +0000

First add the package main in your program:

package main

Every Go file must start with a package name statement. Packages provide code encapsulation and reuse. The name of the package here is main.

After adding main package import “fmt” package in your program:

import(
"fmt"
)

Import the fmt package, which will be used to print text to the standard output in the main function.

Now write the code in the main function to print hello world in Go language:

func main() {  
    fmt.Println("Hello World") 
}

func main():

The main function is a special function that is the entry point of a Go program. The main function must be included in the main package. { and } denote the start and end of the main function.

fmt.Println(“Hello World”):

The Println function in the fmt package is used to print text to standard output.

Introduction and Installation

Neeraj Kumar — Fri, 24 Feb 2023 19:08:22 +0000

What is golang ?
Golang (Go) is a statically and compiled language created by google.
Go is mostly used to develop highly scalable and fast web apps.

Benefits of using go for your rest api

There are so many programming languages to choose from when you want to make a rest api.Python(flask),node or even java. However go has some benefits which makes it a good choice whenever you are making a rest api.

some of the benefits of using go:

Quick compilation and execution speed.
its very easy and fast to learn -go has a very simple
syntax.This makes it very easy to learn.
Lightweight goroutines that support concurrency.
Extensive and very well documented built-in libraries.

Why learn Golang?
Now you still might be asking why should I learn Golang with all the other alternatives out there.
list of some of the pros of the Go programming language:

Simple syntax - Go has concise and straightforward syntax that makes writing readable and maintainable code easy.
Compiled language
Static linking - The compiler supports static linking which means you can statically link your project into one massive binary and then simply deploy it to the cloud or a server.
Open source - Go is open-source so you can read the source code and contribute to the repository.

Installing / Updating Go
If you’re updating a previously installed Go version, you must first delete the older one:

sudo rm -r /usr/local/go
Download the latest version of Go archive from https://golang.org/dl/ and extract it:

sudo tar -C /usr/local -xzf go1.7.1.linux-amd64.tar.gz
Note: some people use the Go version manager (GVM) to install and set up Go, a tool similar to NVM which I highly recommend for Node.js, but for Go I prefer to set up everything myself.

Next, set the PATH environment variables by adding the following to .bashrc:

export PATH=$PATH:/usr/local/go/bin export GOPATH=$HOME/go
Here I use home/neeraj/go as my workspace (which is $HOME/go), but feel free to change it according to your preference.

And, create your workspace:

mkdir -p ~/go/src/github.com/neeraj

Golang Study Point

How to Use the Call, Apply, and Bind Functions in JavaScript .

Neeraj Kumar — Sun, 08 Jan 2023 17:08:00 +0000

Both call and apply invoke a function. Their only difference is that call accepts arguments in a comma-separated fashion while apply requires arguments to be passed as an array or an array-like object.

Call ( )
Call invokes the function immediately and allows you to pass in arguments one by one.

Call() Syntax

function.call(thisArg, arg1, agr2, ...)

Call() Parameters

The call() method can take two parameters:

thisArg - The thisArg is the object that the this object references inside the function func.
arg1, ... argN (optional) - Arguments for the function func.

Note: By default, in a function this refers to the global object i.e, window in web browsers and global in node.js.

call() Return Values
Returns the result obtained from calling the function with the specified this value and arguments.

Note: By using call(), we can use the functions belonging to one object to be assigned and called for a different object.

Example:1

let lastName = { name: 'Yadav' };
function fullName(greeting){
  console.log(`${greeting}${this.name}`)
}
fullName('Neeraj')
fullName.call(lastName,"Neeraj")

Example:2

let name =  {
    firstname : "Neeraj",
    lastname : "Kumar",
}
fullDetails =function(home){
    console.log(this.firstname + " " + this.lastname +", " + home)
}

fullDetails.call(name,"Bihar")

the first argument e.g name inside call method is always a reference to (this) variable and latter will be function variable

Apply ( )

The apply() method calls a function with the passed this keyword value and arguments provided in the form of an array.

apply invokes the function on given context and allows to pass arguments as an array
The apply() method is similar to call(). The difference is that the apply() method accepts an array of arguments instead of comma separated values.

apply() Syntax

function.apply(thisArg, [argumentsArr])

apply() Parameters
The apply() method can take two parameters:

thisArg - The value of this which is provided while calling func.
argsArray (optional) - An array containing the arguments to the functio

Example1: Append two Arrays

let backend = ["Neeraj", "Shashank", "Ramesh"];
let frontend = ["Parveen", "Ashish"];

// appending 
console.log(backend.push.apply(backend, frontend))

bind()
bind returns a new function by setting the provided context, and allows to pass arguments one by one

The bind() method allows an object to borrow a method from another object without copying.

bind() Syntax

func.bind(thisArg, arg1, ... argN)

bind() Parameters

The bind() can take two parameters:

thisArg - The value provided as this parameter for func.
arg1, ... argN (optional) - The value of arguments present inside func. Notes: If thisArg is not specified, the this of the executing scope is treated as thisArg.

bind() Return Value

Returns a copy of the given function with the specified this value, and initial arguments (if provided).

// object definition
const student1 = {
  name: "Neeraj",
  introduction: function (score) {
    console.log(this.name + "scored " + score + " in an exam.");
  },
};

// object definition
const student2 = {
  name: "Sonu",
};

let result = student1.introduction.bind(student2, 95);

// invoking result() function
result();

Call and Apply are interchangeable. You can decide whether it’s easier to send in an array or a comma separated list of arguments. Bind is different. It always returns a new function.

javascript interview questions

Thanks for reading.

Fibonacci Sequence in(JS and Golang)

Neeraj Kumar — Sat, 10 Dec 2022 19:43:04 +0000

The Fibonacci sequence is the integer sequence where the first two terms are 0 and 1. After that, the next term is defined as the sum of the previous two terms.

0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, ...

Fibonacci number using the following formula:

Fibonacci(n)=Fibonacci(n-1)+Fibonacci(n-2)

Fibonacci Series Up to n Terms(JS)

const number = parseInt(prompt('Enter the number of terms: '));
let n1 = 0, n2 = 1, nextTerm;

console.log('Fibonacci Series:');

for (let i = 1; i <= number; i++) {
    console.log(n1);
    nextTerm = n1 + n2;
    n1 = n2;
    n2 = nextTerm;
}

Fibonacci Series Up to n Terms(Golang)

package main

import (
  "fmt"
)

func Fibonacci(n int) {

  var n3, n1, n2 int = 0, 0, 1

  for i := 1; i <= n; i++ {

    fmt.Println(n1)

    n3 = n1 + n2

    n1 = n2

    n2 = n3

  }
}

func main() {

    Fibonacci(10)

  }

Using Recursion(Golang)

package main

import (
  "fmt"
)

func Fibonacci(n int) int {

  if n <= 1 {

    return n

  }

  return Fibonacci(n-1) + Fibonacci(n-2)

}

func main() {

  for i := 1; i <= 10; i = i + 1{

    fmt.Println(Fibonacci(i))

  }

}

Fibonacci Sequence Up to a Certain Number(JS)

const number = parseInt(prompt('Enter a positive number: '));
let n1 = 0, n2 = 1, nextTerm;

console.log('Fibonacci Series:');
console.log(n1); // print 0
console.log(n2); // print 1

nextTerm = n1 + n2;

while (nextTerm <= number) {

    // print the next term
    console.log(nextTerm);

    n1 = n2;
    n2 = nextTerm;
    nextTerm = n1 + n2;
}

Find the Factorial of a Number(JS AND Golang)

Neeraj Kumar — Sat, 10 Dec 2022 12:02:38 +0000

factorial of a non-negative integer n, denoted by n!, is the product of all positive integers less than or equal to n. example:

5! = 5 * 4 * 3 * 2 * 1 = 120

simple method: 1(JS)

function factorial(number) {
  let result = 1;

  for (let i = 2; i <= number; i += 1) {
    result *= i;
  }

  return result;
}
console.log(factorial(5))

simple method: 1(Golang)

var factVal uint64 = 1 // uint64 is the set of all unsigned 64-bit integers. 
var i int = 1
var n int

func factorial(n int) uint64 {   
    if(n < 0){
        fmt.Print("Factorial of negative number doesn't exist.")    
    }else{        
        for i:=1; i<=n; i++ {
            factVal *= uint64(i) 
        }   
    }    
    return factVal 
}

method:(Using Recursion in JS)

function factorialRecursive(number) {
  return number > 1 ? number * factorialRecursive(number - 1) : 1;
}
console.log(factorialRecursive(5))

method:(Using Recursion in Golang)

func factorial(n int) int {
    if n == 0 {
        return 1
    }
    return n * factorial(n-1)
}