DEV Community: Vikas Prabhu

RAG Explained for SQL Developers: Think of It as SELECT, But for Meaning

Vikas Prabhu — Sat, 28 Feb 2026 11:06:42 +0000

If you've spent years writing SELECT statements, tuning execution plans, and building B-Tree indexes, you already understand 80% of what RAG does — you just don't know it yet.

RAG (Retrieval-Augmented Generation) is essentially a retrieval pipeline, just like SQL. The difference? Instead of WHERE category = 'Small Cap', it does WHERE meaning ≈ 'what are the risks'. Instead of returning rows, it returns document passages. And instead of your application rendering a table, an LLM synthesizes a human-readable answer.

This article maps every RAG concept to the SQL equivalent you already know — from storage to indexing to query execution — so it clicks immediately.

The Mental Model

Here's the mapping that'll make everything click:

You Know This (SQL)	Now Learn This (RAG)
`CREATE TABLE` + `INSERT` rows	Chunk documents + Store embeddings
`CREATE INDEX` (B-Tree)	Vector index (HNSW)
`WHERE column = value`	Cosine similarity search
ROWID → fetch row from data block	Chunk ID → fetch text passage
Execution plan (CBO optimizer)	Embed query → Top-K nearest neighbors
ResultSet → render table in app	Retrieved chunks → LLM generates answer

Now let's walk through each phase.

Phase 1: Data Sources — What You're Storing

🟡 SQL — You know this

You work with structured, schema-defined data. Every row has the same columns, every column has a defined type:

NAV tables — fund_id, nav_date, nav_value (NUMBER)
Portfolio holdings — security_id, weight_pct, market_value
Transaction ledgers — txn_id, amount, txn_type, timestamp
Client master — client_id, name, PAN, risk_profile
Fund performance — fund_id, period, return_pct, benchmark_pct

Clean, typed, relational. This is your comfort zone.

🔵 RAG — Now imagine this

But what about all the data that doesn't fit into rows and columns?

SEBI circulars & regulatory filings — PDF documents with legal text
Fund factsheets & offer documents — Multi-page PDFs with tables mixed into prose
Research reports & analyst notes — Word docs, emails, presentations
Board meeting minutes — Free-form narrative text
Annual reports & MD&A sections — Long-form financial commentary

You can't SELECT * FROM sebi_circular WHERE topic = 'liquidity risk' — because there's no table, no columns, no schema. This is exactly the problem RAG solves. It makes unstructured documents queryable, just like SQL makes structured tables queryable.

Phase 2: Data Ingestion & Storage

🟡 SQL — You know this

You define a schema, create tables, and insert rows. Each row gets a physical address (ROWID) in Oracle's data blocks.

-- Step 1: Define schema (DDL)
CREATE TABLE funds (
    fund_id      NUMBER        PRIMARY KEY,
    fund_name    VARCHAR2(200) NOT NULL,
    category     VARCHAR2(50),
    risk_rating  VARCHAR2(20),
    aum_crores   NUMBER(15,2),
    inception_dt DATE
) TABLESPACE fund_data;

-- Step 2: Insert structured rows
INSERT INTO funds VALUES (
    1001,
    'Axis Small Cap Fund',
    'Small Cap',
    'Very High',
    18542.30,
    DATE '2013-11-05'
);

-- Step 3: Data stored in Oracle data blocks
-- Each row → ROWID (physical address)
-- e.g., AAABnPAAEAAAAJHAAA
-- Format: Object | File | Block | Row

Summary: Schema → typed rows → physical storage in data blocks → each row addressable by ROWID.

🔵 RAG — Now here's the equivalent

RAG does the same thing, but for documents. Instead of rows, you store chunks. Instead of ROWIDs, you store vectors.

# Step 1: Load the document (like reading a CSV into a staging table)
doc = load_pdf("SEBI_circular_2024.pdf")

# Step 2: Split into chunks (like breaking one big row into normalized pieces)
chunks = text_splitter(
    doc,
    chunk_size=512,       # ~500 tokens per chunk
    chunk_overlap=50      # overlap so context isn't lost at boundaries
)
# Result: ["Small-cap funds carry higher...",
#          "SEBI mandates stress testing...",
#          "Liquidity risk management..." ]

# Step 3: Convert each chunk into a vector — THIS IS THE KEY STEP
# Think of it as: computing a "semantic ROWID" for each chunk
vectors = embedding_model.encode(chunks)
# Each chunk → [0.023, -0.187, 0.441, ...] (1536 floating-point numbers)

# Step 4: Store in vector database (like INSERT INTO ... VALUES)
vector_db.upsert(
    ids=["chunk_001", "chunk_002", ...],     # like primary keys
    embeddings=vectors,                       # like the "semantic ROWID"
    metadata=[{"source": "SEBI_2024.pdf", "page": 3}, ...]  # like extra columns
)

The SQL developer's translation:

SQL Concept	RAG Equivalent
`CREATE TABLE` with typed columns	Define chunk size + embedding model
`INSERT INTO` a row	`upsert()` a chunk + its vector
ROWID (physical disk address)	Vector embedding (semantic address in meaning-space)
Data block in tablespace	Vector stored in Pinecone / ChromaDB / pgvector
Extra columns (category, date)	Metadata fields (source file, page number)

Key insight: A ROWID tells Oracle where a row lives on disk. A vector embedding tells the vector DB what a chunk means. Both are addresses — one physical, one semantic.

Phase 3: Indexing — Making It Searchable

Without indexes, both systems do full scans. You know this pain. Let's see how each solves it.

🟡 SQL — You know this

-- B-Tree Index (your bread and butter)
CREATE INDEX idx_funds_category
ON funds(category)
TABLESPACE fund_idx;

-- Composite Index (multi-column)
CREATE INDEX idx_perf_fund_date
ON fund_performance(fund_id, period_type, period);

-- Bitmap Index (for low-cardinality columns)
CREATE BITMAP INDEX idx_funds_risk
ON funds(risk_rating);
-- Great for: 'Very High', 'High', 'Moderate' (few distinct values)

-- Function-Based Index
CREATE INDEX idx_funds_upper_name
ON funds(UPPER(fund_name));
-- Speeds up: WHERE UPPER(fund_name) LIKE 'AXIS%'

How B-Tree works: Root → branch → leaf → ROWID → fetch row. For 1 million rows, it takes ~3-4 hops. Deterministic, exact, fast.

🔵 RAG — Now here's the equivalent

Instead of a B-Tree that organizes values in sorted order, RAG uses a vector index that organizes embeddings by semantic proximity.

# HNSW Index (the "B-Tree of the vector world")
# — Most popular, best recall/speed tradeoff
index = create_index(
    type="hnsw",           # Hierarchical Navigable Small World
    metric="cosine",       # similarity metric (like ASC/DESC but for meaning)
    ef_construction=200,   # build quality (like PCT_FREE for B-Trees)
    M=16                   # connections per node (like branching factor)
)

# Other index types:
# IVF  — Clusters vectors first, searches nearby clusters (like partitioning)
# FLAT — Brute force, checks ALL vectors (like a full table scan)

How HNSW works: Imagine a multi-layer graph. Top layer = few nodes (coarse navigation). Bottom layer = all nodes (fine detail). Search starts at top → hops to closest node → drops a layer → repeats → arrives near the target.

The SQL developer's translation:

SQL Index Concept	RAG Vector Index Equivalent
B-Tree (sorted values)	HNSW (semantic proximity graph)
Bitmap (low-cardinality)	IVF (clustered vectors)
Full Table Scan	FLAT index (brute-force vector scan)
`INDEX RANGE SCAN`	Approximate Nearest Neighbor (ANN) search
100% exact match	95-99% approximate match (tradeoff for speed)
`ANALYZE TABLE` for CBO stats	Index tuning: `ef_construction`, `M` params

Key difference: Your B-Tree finds WHERE category = 'Small Cap' — an exact match. A vector index finds WHERE meaning ≈ 'risks of small-cap investing' — a semantic match. "Risk" matches "volatility", "danger", "exposure" — even though the words are different.

Phase 4: Query Execution

A user asks a question. Let's trace the execution path.

🟡 SQL — You know this

-- User wants: "Show me small-cap funds with low liquidity"
-- Translated to SQL:
SELECT fund_name, liquidity_ratio, avg_volume
FROM   funds f
JOIN   fund_liquidity fl ON f.fund_id = fl.fund_id
WHERE  f.category = 'Small Cap'
  AND  fl.liquidity_ratio < 0.5
ORDER BY fl.liquidity_ratio ASC;

-- Oracle CBO generates execution plan:
----------------------------------------------
-- | Id | Operation               | Cost |
-- |----|-------------------------|------|
-- |  0 | SELECT STATEMENT        |   15 |
-- |  1 |  SORT ORDER BY          |   15 |
-- |  2 |   NESTED LOOPS          |   12 |
-- |  3 |    INDEX RANGE SCAN     |    3 |
-- |    |     idx_funds_category  |      |
-- |  4 |    TABLE ACCESS ROWID   |    1 |
-- |    |     funds               |      |
-- |  5 |    INDEX RANGE SCAN     |    2 |
-- |    |     idx_liq_fund_ratio  |      |
-- |  6 |    TABLE ACCESS ROWID   |    1 |
-- |    |     fund_liquidity      |      |
----------------------------------------------

Flow: Parse SQL → CBO picks cheapest plan → index scan → get ROWIDs → fetch rows → sort → return ResultSet. Deterministic. Same query, same result, every time.

🔵 RAG — Now here's the equivalent

# User asks (in plain English, no SQL needed):
# "What are the liquidity risks of small-cap funds?"

# Step 1: Embed the question — like the CBO "parsing" your query
# But instead of parsing SQL syntax, it converts MEANING into a vector
query_vec = embed("What are the liquidity risks of small-cap funds?")
# → [0.034, -0.221, 0.187, ...] (1536-d)

# Step 2: Search the vector index — like INDEX RANGE SCAN
# But instead of matching column values, it matches semantic similarity
results = vector_db.query(
    vector=query_vec,
    top_k=5,                           # like FETCH FIRST 5 ROWS ONLY
    filter={"source": "regulatory"},   # like adding a WHERE clause
    include_metadata=True
)

# Step 3: Results come back ranked by similarity score
# (like ORDER BY relevance DESC)
# chunk_042  score: 0.93  "liquidity risk in small-cap..."
# chunk_187  score: 0.89  "market depth concerns for..."
# chunk_015  score: 0.85  "redemption pressure during..."
# chunk_221  score: 0.81  "SEBI stress testing norms..."
# chunk_098  score: 0.78  "exit load and lock-in..."

Notice: the query doesn't contain the word "market depth" or "redemption pressure" — but those chunks scored high because the meaning is related. A B-Tree could never do this. It only matches exact strings.

The SQL developer's translation:

SQL Query Concept	RAG Query Equivalent
`SELECT ... FROM ... WHERE`	`vector_db.query(vector, top_k, filter)`
CBO parses SQL → execution plan	Embedding model converts question → vector
`INDEX RANGE SCAN` on B-Tree	ANN search on HNSW graph
`WHERE category = 'Small Cap'`	Cosine similarity ≈ "small-cap risks"
`ORDER BY column ASC`	Results ranked by similarity score (0.0 → 1.0)
`FETCH FIRST 5 ROWS ONLY`	`top_k=5`
`AND source = 'regulatory'`	`filter={"source": "regulatory"}`

Phase 5: Retrieval — What Comes Back

🟡 SQL — You know this

You get back a typed ResultSet — exact rows, exact columns, exact values:

+------------------+-------+----------+
| FUND_NAME        | LIQ_R | AVG_VOL  |
+------------------+-------+----------+
| Quant Small Cap  | 0.31  | 1,24,500 |
| Nippon Small Cap | 0.38  | 2,87,300 |
| Axis Small Cap   | 0.42  | 3,45,100 |
| HDFC Small Cap   | 0.47  | 4,12,800 |
+------------------+-------+----------+
4 rows returned in 3ms

Exact. Typed. Deterministic. Run it again → same 4 rows.

🔵 RAG — Now here's the equivalent

Instead of typed rows, you get back ranked text passages with relevance scores:

[
  {
    "id": "chunk_042",
    "score": 0.93,
    "text": "Small-cap funds face significant liquidity risk
      due to lower market depth. During market stress,
      bid-ask spreads can widen 3-5x compared to
      large-cap securities...",
    "metadata": {
      "source": "SEBI_circular_2024.pdf",
      "page": 7,
      "date": "2024-03-15"
    }
  },
  {
    "id": "chunk_187",
    "score": 0.89,
    "text": "Redemption pressure in mid and small-cap
      categories reached ₹8,200 Cr in Q4, forcing
      fund managers to sell positions at unfavorable
      prices...",
    "metadata": { "..." }
  }
]

The SQL developer's translation:

SQL ResultSet	RAG Retrieval
Rows with typed columns	Text chunks with similarity scores
`rs.getString("FUND_NAME")`	`result["text"]` (free-form prose)
`rs.getDouble("LIQ_RATIO")`	`result["score"]` (0.0 to 1.0 relevance)
Always returns exact matches	May return "near misses" (related, not exact)
Deterministic — same query, same result	Probabilistic — may vary slightly
You render it in a table/chart	You feed it to an LLM for synthesis

Phase 6: Response — The Final Output

🟡 SQL — You know this

Your application takes the ResultSet and renders it:

ResultSet rs = stmt.executeQuery(sql);

while (rs.next()) {
    String name = rs.getString("FUND_NAME");
    double liq  = rs.getDouble("LIQ_RATIO");
    long   vol  = rs.getLong("AVG_VOLUME");
    // render in UI table, chart, or report
}

Output: A data table, a chart, a CSV export. The numbers, not the narrative. The human analyst interprets what the data means.

🔵 RAG — Now here's the extra step SQL doesn't have

RAG has one more step: an LLM reads the retrieved chunks and generates a human-readable answer. Think of it as an AI analyst who reads the data and writes the report for you.

# The retrieved chunks become "context" for the LLM
# Think of it like: the LLM is a very smart stored procedure
# that reads text input and generates text output

prompt = f"""Based on the following documents,
answer the user's question.

Context:
{chunk_042.text}
{chunk_187.text}
{chunk_015.text}

Question: What are the liquidity risks?

Answer:"""

response = llm.generate(prompt)

Output:

Small-cap funds face significant liquidity risks across three dimensions:

1. Market depth: Lower trading volumes mean bid-ask spreads can widen 3-5x during stress, per SEBI's 2024 circular.

2. Redemption pressure: Q4 saw ₹8,200 Cr in outflows from mid/small-cap categories, forcing managers to liquidate at unfavorable prices.

3. Exit constraints: Funds may impose exit loads or gate redemptions during severe stress, limiting investor flexibility.

Sources: SEBI Circular (March 2024, p.7), Fund Industry Report Q4

The SQL developer's translation:

SQL Response	RAG Response
ResultSet → app renders table/chart	Retrieved chunks → LLM generates prose
Human interprets the numbers	LLM interprets AND explains
Shows "what" (the data)	Shows "what" + "why" + "so what"
`rs.next()` loop in Java	`llm.generate(prompt)` in Python
Output: rows & columns	Output: paragraphs & citations

The big "aha" moment: RAG is like SQL where the ResultSet gets passed to an incredibly smart stored procedure that reads the data, understands it, and writes a report — automatically.

The Complete Pipeline — One Table to Rule Them All

Phase	SQL (You Know This)	RAG (Now You Know This Too)
📦 Data	Structured tables, typed columns	Unstructured docs, PDFs, emails
📥 Store	`CREATE TABLE` → `INSERT` rows → data blocks	Chunk → `embed()` → store vectors + metadata
🗂️ Index	B-Tree / Bitmap (exact value lookup)	HNSW / IVF (semantic proximity graph)
🔍 Query	Parse SQL → CBO plan → `INDEX RANGE SCAN`	Embed question → cosine similarity → Top-K ANN
📤 Retrieve	Exact typed rows via ROWID (deterministic)	Ranked text chunks via score (probabilistic)
💬 Respond	App renders table/chart (human interprets)	LLM synthesizes answer (AI interprets)

When to Use Which

Question Type	Use	Why
"Show me funds with Sharpe > 1.0"	SQL	Exact numeric filtering on structured data
"What's the NAV of Axis Small Cap today?"	SQL	Single exact value lookup
"Top 10 holdings by weight in Fund X"	SQL	Ordered result set from structured tables
"Why did our fund underperform?"	RAG	Needs synthesis across reports and commentary
"Explain SEBI's new stress testing norms"	RAG	Requires reading and summarizing regulatory PDFs
"What risks should I tell my client about?"	RAG	Open-ended, needs contextual intelligence

The Real Power: Combine Both

The most powerful systems use both pipelines together:

Receive a natural language question
Route it — SQL for data questions, RAG for document questions
Execute the appropriate pipeline
Combine structured data + document context into a single response

Example: "Why did our mid-cap fund underperform last quarter?" triggers SQL for the performance numbers AND RAG for the attribution reports and CIO commentary — giving the user both the data and the explanation in one answer.

TL;DR

If you know SQL, you already know the shape of RAG:

SQL:  Table → INSERT row   → B-Tree index  → WHERE exact match → ResultSet    → App renders table
RAG:  Doc   → Embed chunk  → HNSW index    → WHERE meaning ≈   → Text chunks  → LLM writes answer

Both are retrieval systems. SQL retrieves data. RAG retrieves meaning. The best architectures use both.

If this helped RAG "click" for you, drop a ❤️ — and if you're building systems that combine both, I'd love to hear about your architecture in the comments.

Harnessing the Power of Amazon S3 as a Mounted Drive: Benefits, Costs, and How-To

Vikas Prabhu — Mon, 06 May 2024 15:58:29 +0000

Amazon S3 (Simple Storage Service) by Amazon Web Services (AWS) offers a scalable, reliable, and low-latency data storage solution. While typically used for backup and recovery, data archiving, and as a web application storage layer, an innovative approach is to mount S3 as a drive on a server, such as an Ubuntu AWS instance. This post explores the benefits, costs, and provides a step-by-step guide on how to set this up.

Benefits of Using S3 as a Mounted Drive

Scalability and Elasticity: Virtually unlimited storage means you can scale up as your data grows without worrying about physical storage limits.
Durability and Availability: High durability and availability, with data automatically replicated across multiple devices in multiple facilities.
Cost-Effectiveness: Only pay for the storage you use, with no upfront costs or additional expenses for maintaining physical hardware.
Accessibility: Simplifies workflows by making it accessible as part of the local file system.
Security: Secure data storage with AWS’s robust security model, enhanced by additional encryption and access controls when mounted.
Integration and Automation: Seamless integration with other AWS services and automation capabilities.

Costs of Using S3 as a Mounted Drive

Storage Costs: Based on the amount of data stored per month and the chosen S3 storage class.
Request Costs: Charges for PUT, GET, and other requests, which can add up with frequent file accesses.
Data Transfer Costs: Charges for data transferred out of S3, especially significant if transferring to the internet or across AWS regions.
Management Overhead: Requires expertise to manage and maintain the setup, consuming administrative resources.

How to Mount an S3 Bucket on Ubuntu

Update Your System:

   sudo apt update && sudo apt upgrade -y

Install Required Packages: Install s3fs, a FUSE-based file system:

   sudo apt install s3fs -y

Set Up AWS Credentials: Generate access key from aws console with full s3 access.Securely store your AWS credentials:

   mkdir -p $HOME/.aws
   echo YOUR_ACCESS_KEY_ID:YOUR_SECRET_ACCESS_KEY > $HOME/.aws/credentials
   chmod 600 $HOME/.aws/credentials

Create a Mount Point:

   sudo mkdir /mnt/s3bucket

Mount the S3 Bucket:

   sudo s3fs your-bucket-name /mnt/s3bucket -o passwd_file=$HOME/.aws/credentials -o allow_other -o umask=0022

Verify the Mount and Set Automatic Mount on Boot: Ensure the mount is successful and configure it to mount automatically on system reboot:

   ls /mnt/s3bucket
   sudo nano /etc/fstab

Add the following line to /etc/fstab:

   your-bucket-name /mnt/s3bucket fuse.s3fs _netdev,allow_other,passwd_file=/home/ubuntu/.aws/credentials,umask=0022 0 0

Conclusion

Mounting an S3 bucket as a drive on an Ubuntu AWS instance can greatly enhance how you manage and interact with cloud storage. It not only brings the benefits of S3 into a more accessible and traditional file system format but also necessitates careful management to optimize costs. By following the above steps, organizations can effectively integrate Amazon S3 with their existing infrastructure and enjoy the myriad benefits it offers.

Ensuring Reliability in Web Services: Mastering Idempotency in Node.js and JavaScript

Vikas Prabhu — Sun, 03 Mar 2024 08:36:19 +0000

This is how I explained it to my 9 year old kid :P

Imagine you have a magical notebook that creates a toy every time you write down what toy you want. If you write "I want a skateboard," the notebook creates a skateboard for you. Now, imagine if you accidentally write "I want a skateboard" again in the notebook, but you don't get another skateboard because the notebook already knows you have one. The notebook is smart and remembers what you asked for before, so it doesn't make the same toy again unless you really need it. This way, no matter how many times you write the same thing, you still end up with just one skateboard, not a mountain of skateboards.

This magical notebook is like a computer trick called "idempotency" used in computer programs and the internet. It makes sure that if something is asked to be done more than once, by mistake or on purpose, it doesn't do it again and again but only does it once. This helps prevent messes, like having too many of the same toy, or in the computer world, stopping the same message from being sent out many times or creating too many of the same thing in a computer system. It's a way to keep things neat and avoid unnecessary repeats.

But, What is Idempotency Technically :)

Making your REST service idempotent is essential for ensuring reliability and consistency, especially in distributed systems where the same request might be received multiple times due to network retries or other issues. Idempotency means that making multiple identical requests has the same effect as making a single request.

Understanding Idempotency in HTTP Methods

First, it's important to understand that some HTTP methods are inherently idempotent. GET, PUT, DELETE, HEAD, OPTIONS, and TRACE methods are supposed to be idempotent, meaning that no matter how many times the request is made, the server's state remains the same after the first request. POST methods are generally not considered idempotent, as they are used to create or update resources and can change the server's state with each request.

Strategies to Achieve Idempotency

Use HTTP Methods Appropriately: Adhere to the standard usage of HTTP methods. For operations that retrieve data without changing the server's state, use GET. For operations that update a resource in a way that multiple requests don't change the outcome, use PUT. For creating resources where subsequent requests might create duplicates, consider using PUT with a unique identifier or POST with additional mechanisms to ensure idempotency.
Idempotency Keys: For operations where idempotency is crucial, such as creating a new resource with POST, you can implement idempotency keys. Clients generate a unique key for each operation and send it with the request. The server then checks if this key was used before; if it was, it returns the result of the previous operation instead of creating a new resource. This approach is commonly used in APIs for payment processing or other financial transactions.

Implementing Idempotency on the Server with Node.js and Express

Lets implement idempotency keys with an example of express application. To achieve idempotency in POST operations, we use a combination of idempotency keys and middleware in a Node.js Express application. Here is how you can implement it:

const express = require('express');
const app = express();
const bodyParser = require('body-parser');
app.use(bodyParser.json());

let idempotencyDB = {};

const idempotencyMiddleware = (req, res, next) => {
    if (req.method === 'POST') {
        //get the key from the header
        const idempotencyKey = req.headers['Idempotency-Key'];
        if (idempotencyKey) {
            if (idempotencyDB[idempotencyKey]) {
                return res.status(200).json(idempotencyDB[idempotencyKey]);
            } else {
                res.on('finish', () => {
                    idempotencyDB[idempotencyKey] = res.locals.response;
                });
            }
        }
    }
    next();
};

app.use(idempotencyMiddleware);

app.post('/data', (req, res) => {
    const response = { success: true, data: req.body };
    res.locals.response = response;
    res.status(201).json(response);
});

app.listen(3000, () => {
    console.log('Server is running on port 3000');
});

Test Your Implementation

To test your implementation, you'll need to make HTTP POST requests to your /data endpoint, including an Idempotency-Key header with a unique value for each new request. You can use tools like Postman, curl, or write a client script using Axios or the native fetch API in Node.js.

Here's how you might use curl to test your endpoint:

curl -X POST http://localhost:3000/data \
-H 'Content-Type: application/json' \
-H 'Idempotency-Key: your-unique-key-here' \
-d '{"name":"Test Resource"}'

Repeat the request with the same Idempotency-Key to see that it returns the same resource without creating a new one.

The example above uses an in-memory object (idempotencyDB) to store idempotency keys and responses. For production use, replace this with a persistent storage solution, such as Redis, which is well-suited for this kind of key-value storage with expiry.

Implementing a Idempotency on the Client-Side

The client-side example generates keys using timestamps for simplicity. In a real application, you might want to use a more robust method, such as UUIDs, to ensure uniqueness across requests.

Axios Interceptors

If you're using Axios in a JavaScript application, you can add an interceptor to attach an idempotency key to certain requests.

const axios = require('axios');

// Create or use an existing Axios instance
const instance = axios.create();

instance.interceptors.request.use(config => {
    // Add an idempotency key header to POST requests
    if (config.method === 'post') {
        // This is a simplistic approach for generating keys
        // Consider a more robust method for key generation
        config.headers['Idempotency-Key'] = `idemp-${Date.now()}`;
    }
    return config;
}, error => {
    return Promise.reject(error);
});

// Example of using the Axios instance to make a POST request
instance.post('http://localhost:3000/data', { name: 'Test' })
    .then(response => console.loPg(response.data))
    .catch(error => console.error(error));

Global Interceptors for `fetch` and `XMLHttpRequest`

Achieving idempotency without changing existing code requires intercepting global HTTP requests. Here's how you can override the fetch API and XMLHttpRequest to automatically include an idempotency key in every request.

Overriding the Fetch API

const originalFetch = window.fetch;
window.fetch = async (input, init = {}) => {
    init.headers = {
        ...init.headers,
        'Idempotency-Key': `idemp-${Date.now()}`
    };
    return originalFetch(input, init);
};

Overriding the XMLHttpRequest

const originalOpen = XMLHttpRequest.prototype.open;
XMLHttpRequest.prototype.open = function(method, url, async, user, password) {
    this.addEventListener('readystatechange', () => {
        if (this.readyState === 1) {
            this.setRequestHeader('Idempotency-Key', `idemp-${Date.now()}`);
        }
    }, false);
    originalOpen.call(this, method, url, async, user, password);
};

These interceptors ensure that every outgoing HTTP request, whether made via fetch or XMLHttpRequest, includes a unique idempotency key, improving the robustness and reliability of your application without requiring changes to existing request-making code.

Considerations for Production

Persistence: Use a persistent storage system like Redis, which is well-suited for this kind of key-value storage and has built-in expiration capabilities.
Cleanup: Implement a mechanism to clean up old keys to prevent the storage from growing indefinitely.
Security: Ensure that idempotency keys are securely generated and stored, and consider any privacy implications of storing request data.
Performance: Monitor the performance impact of your idempotency mechanism, especially how it handles storage operations and response caching.

Conclusion

Implementing idempotency in your RESTful services is a best practice that can significantly enhance the reliability and consistency of your application. By using idempotency keys and implementing server-side middleware in Node.js, along with client-side interceptors, you can ensure that your services handle repeated requests gracefully, providing a better experience for your users.

Docker Learning Path - learn from best YouTube videos

Vikas Prabhu — Tue, 20 Jun 2023 15:22:23 +0000

Install Docker

Install Docker on your system by following the official Docker documentation for your specific operating system.
YouTube link: How to Install Docker on Windows 10 | Install Docker on macOS | Install Docker on Ubuntu

Docker Basics

Learn the basic concepts of Docker, such as containers, images, and Dockerfile.
YouTube link: Docker Tutorial for Beginners - Full Course (9M+ views)

Working with Docker Images

Understand how to pull, build, and push Docker images.
YouTube link: Docker Images Explained with Examples (1.4M+ views)

Docker Containers

Explore how to run, manage, and interact with Docker containers.
YouTube link: Docker Tutorial - Running Containers (3.2M+ views)

Docker Compose

Learn how to define and manage multi-container applications using Docker Compose.
YouTube link: Docker Compose in 5 Minutes (1.7M+ views)

Docker Networking

Understand how to connect containers, expose ports, and work with network configurations in Docker.
YouTube link: Docker Networking Basics (480K+ views)

Docker Volumes

Explore Docker volumes and how to persist data between containers.
YouTube link: Docker Volumes Explained with Examples (209K+ views)

Dockerizing an Application

Learn how to containerize an application by creating a Dockerfile and building an image.
YouTube link: Dockerize Your App in 5 Minutes (380K+ views)

Docker Registry

Set up a private Docker registry to store and distribute Docker images.
YouTube link: Creating Your Own Docker Registry (180K+ views)

Docker Swarm (Optional)

If you're interested in orchestration, learn how to create a Docker Swarm cluster and deploy services.
YouTube link: Docker Swarm - Introduction and Demo (310K+ views)

WSL 2.0 - Linux On Windows 10

Vikas Prabhu — Tue, 20 Jun 2023 15:09:39 +0000

Have kept this article simple and to the point , please share you feedback and love <3

Run all commands in Windows PowerShell with admin rights

Enabling the Virtual Machine Platform

dism.exe /online /enable-feature /featurename:VirtualMachinePlatform /all /norestart

Enabling the Windows Subsystem for Linux

dism.exe /online /enable-feature /featurename:Microsoft-Windows-Subsystem-Linux /all /norestart

Restart the machine even if it was mentioned /norestart
After restart reopen powershell with admin rights
Set default WSL version to 2.0

wsl --set-default-version 2

It will give you upgrade message like this

WSL 2 requires an update to its kernel component. For information please visit https://aka.ms/wsl2kernel
For information on key differences with WSL 2 please visit https://aka.ms/wsl2

Download the standalone update from this site
Install the update by double click
Search for Ubuntu 20 LTS from app store
After installation you should be able to see ubuntu in the start menu
Clicking it will open a bash shell
All your windows drives will be under /mnt/
Search for Windows Terminal from app store for better terminal experience
Upgrade your shell to ZSH (not mandatory but just for the love of ZSH)

sh -c "$(curl -fsSL https://raw.github.com/ohmyzsh/ohmyzsh/master/tools/install.sh)"

Python : Virtual Environment + requirments.txt

Vikas Prabhu — Tue, 20 Jun 2023 15:01:46 +0000

Python projects must always be developed in isolated environments or virtual environment so that all our dependencies are locked to that single project only.

requirements.txt in Python is package.json of NodeJS

It contains all the packages required for running the python project.

Lets see how we can create a virtual environment in Python and generate a requirements.txt file

Creating & Managing Virtual Environment

Install

virtualenv is a tool to create isolated environment in Python. Install this tool using apt-get

fire the following command on bash terminal

$ sudo apt-get install python3-venv

Initialize

traverse to the project folder and run the following command

$ virtualenv venv

this creates a copy of Python in whichever directory you ran the command in, placing it in a folder named venv

Activate

to begin using the virtual environment, it needs to be activated using the following command

$ source venv/bin/activate

install any new modules without affecting system default Python or other virtual environments using pip/pip3

$ pip3 install numpy scipy matplotlib

these packages will be installed in the project folder only under venv folder

now we can generate requirements.txt using following command

$ pip3 freeze --local > requirements.txt

--local argument ensures only virtual environment packages are written to requirements.txt

Deactivate

once your work is completed you stop the virtual environment

$ deactivate

Delete

There might be a case where you want to remove the virtual environment. You can simple do this my deleting the venv folder.

Gitlab connect using SSH from Macbook

Vikas Prabhu — Tue, 20 Jun 2023 14:56:12 +0000

To connect GitLab from your MacBook, you can follow these step-by-step commands:

Open the Terminal application on your MacBook. You can find it in the "Utilities" folder within the "Applications" folder or use the Spotlight search to find it quickly.
Configure your Git identity by running the following commands:
```
git config --global user.name "Your Name"
git config --global user.email "your.email@example.com"
```
Replace "Your Name" with your actual name and "your.email@example.com" with your email address. These commands set your global Git configuration, which will be used for all your repositories.
Generate an SSH key pair by running the following command:
```
ssh-keygen -t ed25519 -C "your.email@example.com"
```
This command generates an Ed25519 type SSH key pair. When prompted for a file to save the key, you can press Enter to accept the default location (~/.ssh/id_ed25519).
Start the SSH agent by running the following command:
```
eval "$(ssh-agent -s)"
```
Add your private key to the SSH agent by running the following command:
```
ssh-add ~/.ssh/id_ed25519
```
If you saved your key with a different filename or location, adjust the command accordingly.
Copy your public key to the clipboard by running the following command:
```
pbcopy < ~/.ssh/id_ed25519.pub
```
This command copies the contents of your public key to the clipboard.
Log in to your GitLab account in a web browser.
Navigate to your GitLab account settings and find the "SSH Keys" section.
Click on "Add SSH Key" or a similar button.
Paste the public key from your clipboard into the "Key" field.
Provide a title for the key (e.g., "MacBook SSH Key").
Click on "Add Key" or a similar button to save the SSH key in your GitLab account.

Now you have successfully connected GitLab from your MacBook. You can clone repositories, push changes, and interact with GitLab using the Git commands in the Terminal. Remember to adjust the repository URLs in the Git commands to match the repository you want to work with.

DEV Community: Vikas Prabhu

RAG Explained for SQL Developers: Think of It as SELECT, But for Meaning

The Mental Model

Phase 1: Data Sources — What You're Storing

🟡 SQL — You know this

🔵 RAG — Now imagine this

Phase 2: Data Ingestion & Storage

🟡 SQL — You know this

🔵 RAG — Now here's the equivalent

Phase 3: Indexing — Making It Searchable

🟡 SQL — You know this

🔵 RAG — Now here's the equivalent

Phase 4: Query Execution

🟡 SQL — You know this

🔵 RAG — Now here's the equivalent

Phase 5: Retrieval — What Comes Back

🟡 SQL — You know this

🔵 RAG — Now here's the equivalent

Phase 6: Response — The Final Output

🟡 SQL — You know this

🔵 RAG — Now here's the extra step SQL doesn't have

The Complete Pipeline — One Table to Rule Them All

When to Use Which

The Real Power: Combine Both

TL;DR

Harnessing the Power of Amazon S3 as a Mounted Drive: Benefits, Costs, and How-To

Benefits of Using S3 as a Mounted Drive

Costs of Using S3 as a Mounted Drive

How to Mount an S3 Bucket on Ubuntu

Conclusion

Ensuring Reliability in Web Services: Mastering Idempotency in Node.js and JavaScript

This is how I explained it to my 9 year old kid :P

But, What is Idempotency Technically :)

Understanding Idempotency in HTTP Methods

Strategies to Achieve Idempotency

Implementing Idempotency on the Server with Node.js and Express

Implementing a Idempotency on the Client-Side

Axios Interceptors

Global Interceptors for fetch and XMLHttpRequest

Overriding the Fetch API

Overriding the XMLHttpRequest

Considerations for Production

Conclusion

Docker Learning Path - learn from best YouTube videos

Install Docker

Docker Basics

Working with Docker Images

Docker Containers

Docker Compose

Docker Networking

Docker Volumes

Dockerizing an Application

Docker Registry

Docker Swarm (Optional)

WSL 2.0 - Linux On Windows 10

Python : Virtual Environment + requirments.txt

Creating & Managing Virtual Environment

Install

Initialize

Activate

Deactivate

Delete

Gitlab connect using SSH from Macbook

Global Interceptors for `fetch` and `XMLHttpRequest`