DEV Community: Ananya S

FastAPI for AI Engineers - Part 2: Building Your First CRUD API

Ananya S — Mon, 01 Jun 2026 17:06:06 +0000

In the previous article, we explored why FastAPI has become one of the most popular backend frameworks for modern AI applications.

If you haven't read the previous post, check it out: https://dev.to/zeroshotanu/fastapi-for-ai-engineers-part-1-why-every-ai-backend-is-moving-toward-fastapi-45fg

Now it's time to build something practical.

Most backend applications revolve around four basic operations:

Create
Read
Update
Delete

Together, these operations are known as CRUD.

Whether you're building:

a social media application,
an e-commerce platform,
a chatbot,
or an AI agent,

CRUD operations are the foundation of backend development.

In this article, we'll build a simple Student Management API while learning:

Path Parameters
Query Parameters
GET Requests
POST Requests
PUT Requests
DELETE Requests

Creating Sample Data

Let's start with a small dataset.

from fastapi import FastAPI

app = FastAPI()

students = [
    {
        "id": 1,
        "name": "Ananya",
        "department": "CSE",
        "cgpa": 8.9
    },
    {
        "id": 2,
        "name": "Rahul",
        "department": "ECE",
        "cgpa": 8.4
    },
    {
        "id": 3,
        "name": "Priya",
        "department": "IT",
        "cgpa": 9.1
    }
]

Run the application:

uvicorn main:app --reload

Open Swagger UI:

http://127.0.0.1:8000/docs

Path Parameters

A path parameter is part of the URL itself.

/student/2

Here, 2 is the path parameter.
Think of path parameters as:

"I know exactly which resource I want."

Examples:

/users/10
/products/25
/orders/1001
/student/2

Let's fetch a specific student using their ID.

@app.get("/student/{id}")
def get_student_info(id: int):

    for user in students:
        if user["id"] == id:
            return user

    return {"message": "Student not found"}

Request:

/student/2

Response:

{
    "id": 2,
    "name": "Rahul",
    "department": "ECE",
    "cgpa": 8.4
}

Query Parameters

A query parameter appears after the ? in a URL.

/student?department="CSE"

They are commonly used for:

filtering
searching
sorting
pagination

Let's implement the same endpoint using a query parameter.

@app.get("/students")
def get_students(department: str):

    filtered_students = []

    for student in students:
        if student["department"] == department:
            filtered_students.append(student)

    return filtered_students

Request:

/student?department="CSE"

Response:

{
    "id": 1,
    "name": "Ananya",
    "department": "CSE",
    "cgpa": 8.9
}

All students in CSE department would be filtered.
Query parameters are often optional and are used to modify, filter, or search results.

Path vs Query Parameters

Path Parameter	Query Parameter
Part of URL path	Appears after ?
Identifies a resource	Filters or searches
`/student/1`	`/student?id=1`

GET Request

GET requests are used to retrieve data.

@app.get("/students")
def get_all_students():
    return students

Response:

[
    {
        "id": 1,
        "name": "Ananya",
        "department": "CSE",
        "cgpa": 8.9
    },
    {
        "id": 2,
        "name": "Rahul",
        "department": "ECE",
        "cgpa": 8.4
    },
    {
        "id": 3,
        "name": "Priya",
        "department": "IT",
        "cgpa": 9.1
    }
]

Request Bodies with Pydantic

When users send data to our API, FastAPI needs a way to validate that the incoming data has the correct structure.

This is where Pydantic comes in.

Pydantic allows us to define the expected shape of incoming data using Python classes.

For example, every student should have:

an ID
a name
a department
a CGPA

We can define this structure using a Pydantic model.

from pydantic import BaseModel

class Student(BaseModel):
    id: int
    name: str
    department: str
    cgpa: float

Now FastAPI automatically validates incoming requests.

For example, this request is valid:

{
"id": 4,
"name": "Karthik",
"department": "AI",
"cgpa": 8.8
}

But if someone sends:

{
"id": "four",
"name": "Karthik"
}

FastAPI will automatically return a validation error because:

id should be an integer
required fields are missing

This saves us from writing validation code manually.
We'll explore Pydantic, validation, optional fields, custom validators, and advanced request handling in a dedicated article later in this series.

POST Request

POST requests are used to create new resources.

from pydantic import BaseModel

class Student(BaseModel):
    id: int
    name: str
    department: str
    cgpa: float

@app.post("/student")
def add_student(student: Student):

    students.append(student.dict())

    return {
        "message": "Student added successfully",
        "student": student
    }

Request Body:

{
    "id": 4,
    "name": "Karthik",
    "department": "AI",
    "cgpa": 8.8
}

PUT Request

PUT requests are used to update existing resources.

@app.put("/student/{id}")
def update_student(id: int, updated_student: Student):

    for index, user in enumerate(students):

        if user["id"] == id:

            students[index] = updated_student.dict()

            return {
                "message": "Student updated successfully",
                "student": updated_student
            }

    return {"message": "Student not found"}

Request:

PUT /student/2

DELETE Request

DELETE requests are used to remove resources.

@app.delete("/student/{id}")
def delete_student(id: int):

    for index, user in enumerate(students):

        if user["id"] == id:

            deleted_student = students.pop(index)

            return {
                "message": "Student deleted successfully",
                "student": deleted_student
            }

    return {"message": "Student not found"}

Request:

DELETE /student/3

CRUD Summary

Operation	HTTP Method
Create	POST
Read	GET
Update	PUT
Delete	DELETE

CRUD operations form the foundation of almost every backend application you'll build.

What's Next?

Right now, our data exists only in memory.

If the server restarts, everything disappears.

In the next article, we'll connect FastAPI with SQLite and MySQL so our application can store data permanently, just like real-world production systems.

FastAPI for AI Engineers — Part 1: Why Every AI Backend Is Moving Toward FastAPI

Ananya S — Fri, 29 May 2026 17:57:34 +0000

You open ChatGPT.

You type a prompt.

Within seconds:

your request reaches a backend server,
the backend communicates with an LLM,
retrieves memory,
queries vector databases,
processes context,
and streams responses back to you in real time.

Modern AI applications are no longer just “apps.”

They are systems made up of multiple services constantly communicating with each other through APIs.

And one framework has quietly become the default choice for building these modern AI backends:

FastAPI.

In this article, we’ll understand:

why APIs are essential,
why modern AI systems depend heavily on them,
what FastAPI actually is,
and why it became the preferred backend framework for AI engineers.

Modern Applications Are API Systems

Most applications today are distributed systems.

Your frontend, backend, database, authentication service, payment gateway, and AI models continuously exchange data with one another.

When you order food online:

Frontend → Backend API → Database → Response

When you use an AI chatbot:

User → FastAPI Backend → LLM → Vector DB → Response

Without APIs:

frontend applications would directly access databases,
systems would become tightly coupled,
security would become difficult,
scaling would become messy,
and AI applications would be extremely difficult to maintain.

APIs act as communication bridges between systems.

They define:

how requests are sent,
what data is expected,
and what responses should be returned.

Modern software runs on APIs.

Modern AI systems depend on them even more.

What Exactly Is an API?

API stands for Application Programming Interface.

In simple terms:

An API allows two software systems to communicate with each other.

For example:

a frontend sends a request,
the backend processes it,
and returns a response (usually JSON).

Example:

{
    "message": "Hello World"
}

Every major application you use today relies heavily on APIs:

Instagram
Netflix
Uber
Spotify
ChatGPT
AI agents
recommendation systems
RAG applications

APIs are the foundation of modern backend engineering.

Why AI Applications Changed Backend Development

Traditional web applications were already API-heavy.

But AI applications introduced entirely new backend challenges.

Modern AI systems constantly:

communicate with LLM APIs,
query vector databases,
retrieve embeddings,
stream responses,
interact with external tools,
and handle concurrent requests.

This created a need for backend frameworks that were:

lightweight,
fast,
asynchronous,
scalable,
and developer-friendly.

That’s where FastAPI entered.

What Is FastAPI?

FastAPI is a modern Python framework designed specifically for building APIs.

It became popular because it combines:

high performance,
async support,
automatic validation,
clean developer experience,
and excellent scalability.

FastAPI is built on top of:

Starlette → provides ASGI and async capabilities
Pydantic → handles data validation
Uvicorn → runs FastAPI applications efficiently

Together, this stack became perfect for modern AI systems.


        Client Request
               │
               ▼
         ┌─────────┐
         │ FastAPI │
         └────┬────┘
              │
     ┌────────┼────────┐
     ▼                 ▼
 Starlette         Pydantic
 (ASGI/Async)     (Validation)
              │
              ▼
           Uvicorn
        (ASGI Server)

Why FastAPI Became the Standard for AI Backends

1. Async Support

This is one of the biggest reasons FastAPI exploded in popularity.

AI applications constantly wait for:

LLM responses,
vector database retrieval,
external APIs,
embeddings,
cloud services.

FastAPI supports asynchronous programming using Python’s async and await.

Example:

async def generate_response():
    return {"message": "Async response"}

Instead of blocking the server while waiting for responses, FastAPI can efficiently handle multiple requests concurrently.

For AI systems, this matters a lot.

2. Built on Starlette

FastAPI uses Starlette underneath.

Starlette provides:

ASGI support,
middleware,
WebSockets,
background tasks,
async request handling.

This makes FastAPI much better suited for modern real-time AI applications compared to older synchronous architectures.

3. Powered by Uvicorn

FastAPI applications are commonly run using Uvicorn.

Start a FastAPI server using:

uvicorn main:app --reload

Here:

main → filename
app → FastAPI instance
--reload → automatically reloads during development

Uvicorn is an ASGI server optimized for high-performance asynchronous applications.

4. Automatic Swagger UI Documentation

One of FastAPI’s most loved features is automatic API documentation.

The moment you create routes, FastAPI automatically generates interactive API documentation for you.

Visit:

http://127.0.0.1:8000/docs

You can:

test endpoints,
send requests,
inspect responses,
and debug APIs directly from the browser.

This becomes incredibly useful when:

working with frontend developers,
building AI APIs,
or testing backend systems quickly.

5. Automatic Data Validation Using Pydantic

FastAPI uses Python type hints for validation.

Example:

from pydantic import BaseModel

class User(BaseModel):
    name: str
    age: int

If invalid data is sent, FastAPI automatically validates and rejects it.

This removes a huge amount of manual validation code developers previously had to write themselves.

Installing FastAPI

Install FastAPI and Uvicorn:

pip install fastapi uvicorn

Your First FastAPI Application

Create a file called main.py

from fastapi import FastAPI

app = FastAPI()

@app.get("/")
def home():
    return {"message": "Welcome to Dev.io"}

Run the server:

uvicorn main:app --reload

Open:

http://127.0.0.1:8000/docs

And you’ll see FastAPI’s automatically generated Swagger UI.

At this point, you already have:

a running backend server,
a working API,
and interactive API documentation.

With surprisingly little code.

Why FastAPI Matters for AI Engineers

FastAPI became extremely popular because modern AI applications are fundamentally API systems.

It is heavily used for:

RAG pipelines,
AI agents,
chatbot backends,
LangChain applications,
vector database APIs,
recommendation systems,
and model-serving APIs.

Modern AI engineering is not just about building models anymore.

It’s also about building scalable systems around those models.

And FastAPI fits perfectly into that ecosystem.

Final Thoughts

FastAPI didn’t become popular accidentally.

It became the framework of choice for AI engineers because modern AI systems are:

asynchronous,
API-driven,
performance-sensitive,
and highly modular.

Whether you're building:

AI agents,
chat systems,
RAG applications,
or production AI platforms,

FastAPI provides the exact architecture modern AI applications need.

What’s Next?

Right now, our API returns data, but it doesn’t actually store anything permanently.

In the next article, we’ll build real CRUD APIs using FastAPI and understand:

GET requests,
POST requests,
PUT requests,
DELETE requests,
and how backend applications manage data.

Then we’ll move toward integrating databases like SQLite and MySQL in the following parts of this series.

Check out the next post here:
https://dev.to/zeroshotanu/fastapi-for-ai-engineers-part-2-building-your-first-crud-api-lpl

How I Built an AI-Powered Incident RCA Platform with LangGraph and RAG

Ananya S — Tue, 26 May 2026 03:23:00 +0000

It’s 2:13 AM.

A payment API suddenly starts failing in production.

Customers can’t complete transactions. Alerts begin firing everywhere. Dashboards turn red. Kubernetes pods restart unexpectedly. Database connections start timing out.

And somewhere, an exhausted engineer opens Datadog and starts scrolling through thousands of logs trying to answer a single question:

“What actually broke?”

Modern systems generate enormous amounts of telemetry:

logs
alerts
traces
metrics
infrastructure events

The problem isn’t the lack of monitoring anymore.

The problem is:

making sense of the chaos quickly enough during an outage.

That idea became the starting point for OpsMind AI — an AI-powered incident root cause analysis platform inspired by real-world DevOps and Site Reliability Engineering workflows.

The goal was ambitious but simple:

Upload observability logs → identify probable root cause → generate remediation recommendations automatically.

The Core Problem

In modern distributed systems, a single failure rarely stays isolated.

A database lock might cause:

API latency spikes
gateway timeouts
downstream service crashes
Kubernetes restarts

During incidents, engineers manually jump between:

Grafana dashboards
Datadog alerts
New Relic traces
raw log streams

trying to correlate failures across services.

This process is:

time-consuming
mentally exhausting
highly dependent on experience

I wanted to explore whether multi-agent AI systems could assist in this process.

Not just summarizing logs.

But actually:

retrieving similar historical incidents
classifying incident severity
reconstructing event timelines
identifying affected services
generating RCA explanations
suggesting remediation steps

Enter OpsMind AI

OpsMind AI simulates an AI-driven observability assistant for SRE and DevOps teams.

The platform processes observability logs through a LangGraph-based multi-agent workflow that orchestrates specialized agents for different operational tasks.

Instead of relying on a single monolithic LLM prompt, the system breaks incident investigation into multiple coordinated reasoning stages.

System Architecture

The workflow begins by ingesting logs from simulated monitoring platforms such as:

Datadog
Grafana
New Relic

The logs are normalized and passed into a multi-agent orchestration pipeline.

The architecture consists of:

Retrieval Agent

Searches historical incidents using FAISS vector similarity search.

Incident Classification Agent

Identifies:

incident type
severity level
monitoring source

RCA Agent

Performs root cause analysis and generates remediation recommendations using LLM reasoning.

Timeline & Impact Analysis

Reconstructs operational event sequences and identifies affected downstream services.

Evaluation Layer

Measures:

retrieval accuracy
RCA quality
latency
incident correlation confidence

The frontend dashboard was built using Streamlit to simulate an operational observability console.

Why RAG Was Important Here

One of the most interesting parts of the project was integrating retrieval-augmented generation.

Production incidents often repeat patterns:

database pool exhaustion
API rate limiting
Kubernetes OOM crashes
retry storms
deadlocks

Instead of asking the LLM to reason from scratch every time, OpsMind AI retrieves semantically similar historical incidents from a FAISS vector database and uses them as contextual memory during RCA generation.

This significantly improved the consistency of generated analyses.

Building the Multi-Agent Workflow

The orchestration layer uses LangGraph to model incident analysis as a graph of specialized AI agents.

This made the workflow:

modular
explainable
easier to visualize

One thing I particularly enjoyed was building the animated agent execution dashboard where each agent executes sequentially:

Retrieval Agent
Classification Agent
RCA Agent
Timeline Agent
Impact Analysis Agent

Watching the workflow execute in real time made the system feel much closer to an actual operational AI assistant rather than just another chatbot interface.

Simulating Real Production Incidents

Since real enterprise observability data isn’t publicly available, I generated synthetic production-style incident logs for:

Kubernetes CrashLoopBackOff failures
database connection exhaustion
API rate limiting failures
downstream gateway crashes

The architecture was intentionally designed so that simulated connectors can later be replaced with real monitoring APIs.

Evaluation Was Surprisingly Hard

One unexpected realization during development:

Building the RCA pipeline was easier than evaluating it.

It’s very easy to generate convincing AI explanations.

It’s much harder to measure:

whether the RCA is actually correct
whether retrieval is meaningful
whether severity classification is reliable

That’s why I added an evaluation layer measuring:

Retrieval Accuracy
RCA Match Accuracy
Severity Accuracy
Average Latency
Correlation Confidence

Adding evaluation made the project feel significantly more engineering-focused rather than simply prompt-driven.

Tech Stack

Python
Streamlit
LangGraph
FAISS
SentenceTransformers
Groq LLM API
Pandas

Building Under Hackathon Constraints

OpsMind AI was originally built during a short-duration engineering hackathon focused on AI agents and developer infrastructure workflows.

One interesting challenge was balancing:

ambitious system design ideas
realistic implementation scope
evaluation reliability
UI polish
deployment constraints

I wanted the project to feel less like a simple LLM wrapper and more like an actual operational intelligence platform, which is why I focused heavily on:

multi-agent orchestration
retrieval systems
evaluation metrics
workflow visualization
observability-inspired architecture

Even within a constrained timeline, building the system end-to-end — from synthetic telemetry generation to agent orchestration and evaluation — was an incredibly valuable learning experience.

What I Learned

This project taught me a lot about:

observability systems
multi-agent orchestration
RAG pipelines
AI evaluation strategies
operational intelligence workflows

More importantly, it changed how I think about AI systems.

The interesting challenge wasn’t generating text.

It was designing systems that:

reason through operational data
coordinate specialized agents
retrieve contextual memory
produce actionable outputs

That feels much closer to how real-world AI systems will evolve.

Demo & Repository

GitHub Repository

https://github.com/Anucool419/OpsMind-AI

Demo Video

Future Improvements

Some things I’d love to explore next:

real-time telemetry ingestion
live Datadog/New Relic integrations
Slack incident alerting
autonomous remediation workflows
distributed tracing support
long-term incident memory systems

Conclusion

What started as a simple idea — “Can AI help investigate production incidents faster?” — turned into a much deeper exploration of how intelligent systems can assist engineering operations.

The most interesting part of building OpsMind AI wasn’t the UI or even the LLM integration.

It was understanding how modern operational systems actually behave:

cascading failures
noisy telemetry
infrastructure dependencies
repeated incident patterns
operational uncertainty

This project made me realize that the future of AI in engineering is not just about chat interfaces.

It’s about building systems that can:

reason over complex environments
retrieve operational memory
coordinate specialized agents
assist humans during high-pressure decision making

OpsMind AI is still a prototype, but building it gave me a much deeper appreciation for:

observability engineering
SRE workflows
AI orchestration systems
evaluation-driven AI development

And honestly, that combination of AI + systems engineering is one of the most exciting areas to explore right now. Do suggest any improvements you think I should make or share your experiences.

Thanks for reading.

🚀 This AI Tells You What to Study for Exams

Ananya S — Mon, 04 May 2026 17:45:51 +0000

Consider this:

It’s 3 days before your exam.
You have 5 units. 2 are huge. 1 is confusing.
And you’re thinking:

“What do I actually study?”

So you open past papers.

You start spotting patterns… maybe.

But it’s slow. Inconsistent. And honestly — a bit of guessing.

💡 What if that entire process was automated?

What if you could:

Upload past papers 📄
Instantly see what matters most
Identify high-weightage topics
Know what’s missing from your prep
Get a day-wise study plan

And that's the solution I built and prototyped in 6 hours during a GenAI hackathon

🎯 Introducing: AI Exam Strategist

A system that analyzes past question papers and turns them into actionable study strategy.

Not just summaries. Not just answers.

👉 Actual decision-making support for exams.

🧠 What It Does

📂 Multi-Paper Analysis

Upload multiple past papers → the system processes them together and extracts meaningful patterns.

🔍 Pattern Detection

Finds frequently asked topics
Classifies difficulty levels
Identifies year-wise trends

👉 Helps you focus on topics with the highest exam impact

📚 Syllabus Mapping

Upload your syllabus → instantly see:

✅ Topics already appearing in exams
❌ Topics not covered (potential blind spots)

📊 Visual Insights

Topic frequency charts
Difficulty distribution
Topic vs difficulty breakdown
Year-wise trends

👉 Patterns become obvious at a glance

🧠 Smart Study Planner

Generates a day-wise plan based on:

available time
topic importance

👉 Designed for maximum ROI under time constraints

📝 Practice Question Generator

Select a topic → generate relevant practice questions instantly.

💬 AI Assistant

Ask:

“What should I prioritize?”

Get answers grounded in your own analyzed data.

🏗️ Tech Stack

FastAPI → backend APIs
Streamlit → interactive UI
Groq API (LLM) → classification & generation
LangGraph → structured workflow orchestration
Pandas → data processing

⚙️ How It Works

Upload Papers → Extract Questions → Classify (Topic + Difficulty)
→ Analyze Patterns → Map with Syllabus → Generate Insights
→ Create Study Plan → Practice + AI Chat

🤔 Did I Use RAG?

Not in this version.

Since the dataset is relatively small, I used:
👉 context injection (passing structured analysis directly to the LLM)

This keeps the system fast and simple.

For larger-scale usage, this can evolve into a RAG-based system with vector search.

📏 Evaluation (Keeping It Real)

I added a basic evaluation layer to understand how the system behaves.

Used a small, manually created dataset
Measured:
- Topic classification
- Difficulty classification

⚠️ Important:

Accuracy may appear low if you try it yourself
Because:
- dataset is small
- matching is strict (semantic matches may be marked wrong)

👉 The goal wasn’t perfect scoring —
but to validate the system’s reasoning and consistency

🧠 What I Learned

Building GenAI systems is more about pipelines than prompts
LLM outputs are messy — normalization is critical
Evaluation in AI is not straightforward
Simple approaches (like context injection) can outperform complex ones for MVPs
Speed + clarity > overengineering

🔮 Future Improvements

OCR for scanned PDFs
Semantic topic matching using embeddings
Persistent memory across sessions
Scalable deployment

🎥 Demo & Links

🔗 GitHub: https://github.com/Anucool419/AI-Exam_Strategist
🎥 Demo Video: https://www.loom.com/share/04005565701e45d1855d1fa13bcee73a
🌐 Live App: https://ai-examstrategist-ryjarq6usrfbsd85gipexy.streamlit.app/

⚠️ Note: The live demo UI is deployed, but the backend runs locally. Full functionality is shown in the demo video.

🏁 Final Thoughts

Exams aren’t just about how much you study.

They’re about:

what you choose to study
how you prioritize
how well you use limited time

And right now, students are expected to figure that out manually.

This project explores a simple idea:

What if AI could guide those decisions?

Not replace studying.
Not shortcut learning.

But make preparation more focused, more intentional, and more efficient.

Because sometimes, the smartest move…
is knowing what not to study.

In ~6 hours, this went from an idea to a working system.

It’s not perfect — but it solves a real problem:

Maximizing study ROI when time is limited

Would love your thoughts 👇

🚀 I Built an AI-Powered Fest Assistant with Agents, RAG & Planning (Pragyan @ NITT)

Ananya S — Tue, 14 Apr 2026 17:57:01 +0000

I deleted Instagram more than a year ago, and honestly, it saved me from a lot of distractions.

But there was an unexpected downside.

A lot of informal, real-time information — especially during college events — still lives there.

During our college fest, for example:

Event schedules
Last-minute updates
Food stall announcements
Informal activities

…everything gets posted on Instagram.

At the same time:

Fest details and significance are on the official website
Food stall info is on a separate app
The entire 3-day schedule is compressed into a few posts

There’s no single place to get a clear, structured view of everything.

And that’s when it hit me:

Most college fests have websites.
Some even have apps.

But none of them actually help you navigate the fest intelligently.

They give information.
They don’t give guidance.

But I wanted to build something smarter —
an AI assistant that actually understands queries, plans your day, and even helps you find teammates.

So, I built Pragyan Mentor Assistant — an AI-powered system for navigating a techno-managerial fest.

🎯 Problem

During college fests like Pragyan (NIT Trichy):
There are

There are dozens of events, workshops, and shows
Information is scattered across PDFs, sites, and posters
Users don’t know:
- what to attend
- what matches their interests
- how to plan their time
- who to team up with

👉 Traditional apps = static information
👉 I wanted intelligent interaction

💡 Solution

I built a multi-tool AI assistant that can:

🎉 Answer questions about events, workshops, proshows
🍔 Show food stalls & mess menu
🧠 Recommend activities based on user intent
📅 Plan your schedule
🤝 Match you with like-minded participants/Suggest potential teammates (prototype)
📚 Answer fest-related questions using RAG

🧠 System Design

Instead of a simple chatbot, I designed it as a tool-using agent system.

🔹 Tools

fetch_events
fetch_workshops
fetch_food_stall
fetch_mess_menu
pragyan_bot (RAG-based)
smart_recommender
planner
buddy_matcher

🔹 Agent Flow

User query
LLM decides:

Which tool to call
1. Tool executes
2. Response is generated in natural language

📚 Retrieval Approach

This system uses a hybrid retrieval strategy at the system level:

Structured retrieval (keyword-based)
- Direct tool calls for events/workshops
- Fast and deterministic
Semantic retrieval (RAG)
- Vector search over fest documents
- Handles open-ended queries

👉 This combination allows both precision and flexibility

📚 RAG (Retrieval Augmented Generation)

To handle fest knowledge:

Used:
- Text files (events, shows, lectures, FAQs)
Built:
- FAISS vector store
Retrieval:
- Semantic search on query
Response:
- Context-aware answers

🧠 Memory

Using:

InMemorySaver() (LangGraph)

👉 Enables:

remembering user preferences
better recommendations
conversational continuity

🤖 Smart Features

🎯 Recommendations

Understands intent like:

"What should I attend if I like tech and fun?"

📅 Planner Agent

"Plan my next 3 hours"

Generates a structured schedule based on:

time
interests
available events

🤝 Buddy Matching (Prototype)

Matches based on:

interests
level
context (e.g. case study competitions)

Uses a small dataset to demonstrate logic

🖥️ UI

Built with Streamlit:

Chat-based interface
Quick action buttons
Structured responses

🚀 Deployment

Deployed on Render (free tier)
Environment variables for API security

🎥 Demo

👉 https://www.loom.com/share/13f87025a9154a55b80fc240bfc91ba2

🛠️ Tech Stack

Python
LangChain
OpenAI API
FAISS
Streamlit
Render

⚠️ Challenges Faced

RAG retrieval quality (chunking + parsing issues)
Tool selection accuracy
Structuring multi-agent workflow
Deployment + API key handling

🔄 Ongoing Improvements

Some features I’m actively working on:

Adding database-backed user profiles for real buddy matching
Improving RAG with better retrieval and evaluation
Expanding dataset coverage for more complete fest information
Exploring true hybrid retrieval + reranking

📈 What I Learned

Building agents > building chatbots
RAG needs data structuring, not just embeddings
UI matters a lot for perceived intelligence
Deployment and debugging are part of the real challenge

🔗 Links

🔒 Live demo available on request

💭 Final Thoughts

This project made me realize:

👉 The future isn’t just about LLMs
👉 It’s about systems built around them

If you have suggestions or ideas to improve this, I’d love to hear them!

Stop Calling FAISS a Database: The VectorStore vs. VectorDB Showdown🧠⚡

Ananya S — Tue, 17 Mar 2026 16:40:58 +0000

If you’ve been building with LangChain, you’ve probably used Chroma or FAISS and called them "databases." But in a production environment, that distinction could be the difference between a smooth app and a total system crash.

As AI Engineers, we need to know when to use a lightweight VectorStore and when to upgrade to a full Vector Database.

What is a VectorStore? (The Engine)

A VectorStore is a specialized data structure or a local library. Its primary job is simple: Calculate the distance between vectors as fast as possible.

Best for: Prototypes, local research, and small datasets.
Pros: Zero latency (runs in-process), easy to set up, free.
Cons: If your app restarts, your data might vanish (if not saved to disk). It doesn't scale across multiple servers easily.

Popular Choice: FAISS (by Meta). It's incredibly fast but lacks "database" features like user authentication or real-time updates.

from langchain_community.vectorstores import FAISS
from langchain_openai import OpenAIEmbeddings

# 1. Initialize Embeddings
embeddings = OpenAIEmbeddings()

# 2. Create the VectorStore (In-memory)
texts = ["AI is transforming civil engineering", "LangChain is a framework for LLMs"]
vector_store = FAISS.from_texts(texts, embeddings)

# 3. Search (Fast, but only local)
query = "What is LangChain?"
results = vector_store.similarity_search(query)

# 4. Persistence (Manual step required)
vector_store.save_local("my_faiss_index") 
# To use it later, you must load_local() manually

What is a Vector Database? (The Full System)

A Vector Database is a production-ready management system. It uses a vector store under the hood but wraps it in the features we expect from enterprise software.

Best for: Production apps, multi-user systems, and massive datasets (millions of vectors).

The "Extras" you get:

Persistence: Your data lives on a server, not just in your RAM.
Metadata Filtering: The ability to say "Find similar vectors, but only for documents created in 2024."
Scalability: It can handle billions of vectors by spreading them across different "pods" or nodes.

Popular Choice: Pinecone or Weaviate.

from langchain_pinecone import PineconeVectorStore
from pinecone import Pinecone

# 1. Initialize Cloud Client
pc = Pinecone(api_key="YOUR_PINECONE_API_KEY")
index_name = "my-production-index"

# 2. Connect to the Index (Data lives on Pinecone's servers)
vector_db = PineconeVectorStore.from_texts(
    texts=["B.Tech students at NITT are building AI agents"],
    embedding=OpenAIEmbeddings(),
    index_name=index_name
)

# 3. Search (API Call to the cloud)
# Anyone with the API key can now query this from any device
results = vector_db.similarity_search("Who is building agents?")

Key Observation: There is no "save" step. The moment you run from_texts, the data is permanently stored in the cloud. You can delete your local code, and the data remains accessible.

Feature	VectorStore (e.g., FAISS, Chroma)	Vector Database (e.g., Pinecone, Milvus)
Architecture	A library that runs inside your application code.	A standalone distributed system running on a server.
Data Persistence	Mostly In-Memory. Data is lost when the script ends.	Persistent by default. Data is stored on cloud/disk.
Scalability	Limited by your machine's RAM/Disk. Hard to scale.	Built for Horizontal Scaling. Handles billions of vectors.
Multi-tenancy	No built-in support for isolated users.	High. Supports multiple users and isolated indexes.
CRUD Operations	Hard to update specific vectors without rebuilding.	Full Create, Read, Update, Delete support via API.
Metadata	Basic filtering capabilities.	Advanced Metadata Filtering (e.g., Filter by date).
Cost	Free (Uses your local resources).	Tiered. Free tiers available, then paid.

Let's Discuss!
Are you currently using a local store like Chroma or have you made the jump to a cloud database? What's the biggest challenge you've faced with vector scaling? Drop a comment below! 👇

Decoding Embedding Models: Why Your RAG Is Only as Good as Your Vectors 🚀

Ananya S — Mon, 09 Mar 2026 17:36:43 +0000

As an AI Engineer, the first major decision you make in a RAG (Retrieval-Augmented Generation) pipeline isn't which LLM to use—it's which Embedding Model will represent your data. If your vectors are low-quality, your retrieval will fail, and even a top-tier LLM can't save a response based on the wrong context.

🏗️ What exactly is an Embedding?

Embedding models take text tokens and map them into a multi-dimensional coordinate system (vectors).

Dimensions: These represent the "features" the model understands. Different models represent words in vectors of different dimensions.

Semantic Proximity: In a good model, the vector for "King" and "Queen" will be mathematically closer than "King" and "Keyboard."

🌟 Popular Embedding Models: Hugging Face

Hugging Face models are the go-to for privacy, local deployment, and cost-efficiency. Here are the top picks:

1. all-MiniLM-L6-v2

Dimensions: 384
Description: Fast, efficient, and good quality.
Use Case: General purpose; ideal for real-time applications.

2. all-mpnet-base-v2

Dimensions: 768
Description: The best quality in the MiniLM family, though slower than L6. More dimensions lead to an improvement in accuracy.
Use Case: When quality matters more than speed.

3. all-MiniLM-L12-v2

Dimensions: 384
Description: Slightly better than L6 but a bit slower.
Use Case: A solid balance of speed and quality.

4. multi-qa-MiniLM-L6-cos-v1

Dimensions: 384
Description: Optimized specifically for question-answering.
Use Case: Q&A systems and semantic search.

5. paraphrase-multilingual-MiniLM-L12-v2

Dimensions: 384
Description: Supports 50+ languages.
Use Case: Global and multilingual applications.

💰 The OpenAI Standard

If you need massive scale and the highest "reasoning" in your vectors without managing infrastructure:

text-embedding-3-small

Dimensions: 1536
Cost: ~$0.02 per 1M tokens.
Description: Highly cost-effective with improved accuracy over older models. It also supports Matryoshka Representation Learning, allowing you to trim dimensions (e.g., to 512) to save storage costs without losing much performance.

2. text-embedding-3-large

Dimensions: 3072
Cost: ~$0.13 per 1M tokens.
Description: The most powerful model available. It captures incredibly fine-grained nuances in text.
Feature: Like the "small" version, it supports Matryoshka Representation Learning, which means you can shorten the vector to 256 or 1024 dimensions to save database space while keeping most of the accuracy.
Use Case: Enterprise-level research, legal document analysis, or complex medical data.

3. text-embedding-ada-002

Dimensions: 1536
Cost: ~$0.10 per 1M tokens.
Description: The previous industry standard. While still reliable, it is now considered legacy compared to the "v3" family.
Use Case: Mostly seen in older "legacy" AI systems. For any new project in 2026, you should skip this and go straight to text-embedding-3-small.

⚖️ How to Choose?

Strict Privacy/On-Prem? ➔ Hugging Face (Local).

Real-time/Low Latency? ➔ all-MiniLM-L6-v2.

Multilingual Data? ➔ paraphrase-multilingual-MiniLM-L12-v2.

Enterprise Scale & Accuracy? ➔ text-embedding-3-small.

🚀 Conclusion

In 2026, picking the right embedding model is about balancing latency, cost, and accuracy. Don't just pick the one with the most dimensions—pick the one that fits your specific data and hardware.

What's your go-to embedding model for production? Let's discuss in the comments!

🧠 Your LLM Isn’t an Agent — Until It Has Tools, Memory, and Structure (LangChain Deep Dive)

Ananya S — Mon, 02 Mar 2026 17:09:48 +0000

Most “AI apps” today are just:

Prompt → LLM → Text Response

That’s not an agent.

That’s autocomplete with branding.

A real AI agent can:

🛠 Use tools
🧠 Remember context
📦 Return structured outputs
🔁 Reason across multiple steps

With modern LangChain, building this is surprisingly clean.

Let’s build one properly.

🚀 The Architecture of a Real Agent

A production-ready AI agent has four core components:

Model – the brain
Tools – capabilities
Structured outputs – reliability and formatting
Memory – continuity

If you’re missing one of these, you’re not building a system — you’re running a demo.

1️⃣ The Brain: Modern Agent Setup

We start with create_agent() — the current way to build agents in LangChain.

from langchain.agents import create_agent
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(model="gpt-4o-mini", temperature=0)

Low temperature = more deterministic reasoning.

Now let’s give it capabilities.

2️⃣ Tools: Giving the Agent Superpowers

Tools are just Python functions with clear docstrings.
The docstring matters — it’s how the model decides when to use the tool.

from langchain.tools import tool

@tool
def calculate_revenue(price: float, quantity: int) -> float:
    """Calculate total revenue given price per unit and quantity sold."""
    return price * quantity


@tool
def get_exchange_rate(currency: str) -> float:
    """Get the USD exchange rate for a given currency code."""
    rates = {"EUR": 1.1, "GBP": 1.25}
    return rates.get(currency.upper(), 1.0)

Now we assemble the agent:

agent = create_agent(
    model=llm,
    tools=[calculate_revenue, get_exchange_rate],
    system_prompt="You are a financial analysis assistant."
)

That’s it.

The agent now:

Decides when math is needed
Calls tools autonomously
Observes results
Produces a final answer

No manual routing logic.

3️⃣ Structured Outputs: Stop Parsing Strings

If you're still doing regex on LLM responses, stop.

Modern agents can return structured data using schemas.

from pydantic import BaseModel

class FinancialReport(BaseModel):
    revenue: float
    currency: str
    usd_value: float

Now we enforce structure:

structured_agent = create_agent(
    model=llm,
    tools=[calculate_revenue, get_exchange_rate],
    response_format=FinancialReport,
)

Now when you invoke:

response = structured_agent.invoke({
    "messages": [
        {"role": "user", "content": "I sold 120 units at 50 EUR each. Convert to USD."}
    ]
})

print(response)

You don’t get text.

You get validated data.
Using Pydantic structured output parsers ensures the datatype of the fields is based on how we defined it.

4️⃣ Memory: Making the Agent Stateful

Without memory, every request is a new message.

With memory, your agent becomes a collaborator.

In LangChain, memory can be plugged in via message history.

Example pattern:

chat_history = []

response = agent.invoke({
    "messages": chat_history + [
        {"role": "user", "content": "My product costs 20 USD."}
    ]
})

chat_history.extend(response["messages"])

response = agent.invoke({
    "messages": chat_history + [
        {"role": "user", "content": "Now calculate revenue for 300 units."}
    ]
})

Now the agent remembers:

Product price
Prior discussion
Contextual decisions

Memory transforms isolated responses into evolving workflows.

🧠 What’s Actually Happening Internally?

When you call:

agent.invoke(...)

The agent:

Reads conversation + system prompt
Plans next action
Chooses a tool (if needed)
Executes tool
Feeds result back into reasoning
Produces structured final output

This loop is grounded in tool-calling rather than fragile prompt tricks.

⚠️ Common Mistakes

Things beginner devs get wrong:

❌ Adding too many tools
❌ Writing vague tool descriptions
❌ Not enforcing structured outputs
❌ Forgetting observability/logging
❌ Letting the agent free-run without constraints

Agents are probabilistic planners — not deterministic scripts.

Design them intentionally.

🏗 The Big Shift in How We Build Software

Before agents:

APIs returned static responses
Business logic was deterministic
LLMs were “smart text generators”

After agents:

LLMs orchestrate execution
Tools become capabilities
Structure guarantees reliability
Memory enables continuity

You're no longer building chat interfaces.

You're building goal-driven systems.

🎯 Final Take

If your AI system:

Doesn’t use tools
Doesn’t enforce structure
Doesn’t maintain memory

It’s not an agent.

It’s autocomplete with better marketing.

With modern LangChain, the barrier to real agents is gone.

The question isn’t “Can we build agents?”

It’s:

What workflows are we ready to automate?

Do comment how you build agents and regarding any interesting types of agents you've built!

Understanding LangChain and Vector Embeddings: The Power Duo of Modern AI Applications

Ananya S — Mon, 23 Feb 2026 17:23:19 +0000

Introduction

In the rapidly evolving landscape of artificial intelligence and natural language processing, two technologies have emerged as fundamental building blocks for creating intelligent applications: LangChain and vector embeddings. Together, they form a powerful combination that enables developers to build sophisticated AI systems capable of understanding, reasoning, and generating human-like responses. This post explores both concepts and demonstrates how they work together to create the next generation of AI applications.

What is LangChain?

LangChain is an open-source framework designed to simplify the development of applications powered by large language models (LLMs). It provides a comprehensive set of tools, components, and abstractions that help developers:

Chain together multiple LLM calls and other components
Integrate with external data sources and APIs
Implement memory to maintain context across interactions
Create agents that can make decisions and take actions
Handle complex workflows with ease

At its core, LangChain acts as a bridge between raw LLM capabilities and real-world applications, providing structure and patterns for building production-ready AI systems.

Understanding Vector Embeddings

Vector embeddings are numerical representations of data (typically text, but also images, audio, etc.) in a high-dimensional space. These representations capture semantic meaning and relationships between items:

Semantic similarity: Items with similar meanings have similar vector representations
Dimensionality: Typically 1536, 768, or other dimensions depending on the model
Distance metrics: Cosine similarity or Euclidean distance can measure relatedness

For example, the words "king" and "queen" would have vector embeddings that are close to each other in vector space, while "king" and "banana" would be farther apart.

Common Embedding Models:

OpenAI's text-embedding-ada-002
Sentence-BERT (SBERT)
Hugging Face's all-MiniLM-L6-v2

How LangChain and Vector Embeddings Work Together

The true power emerges when LangChain integrates vector embeddings into its architecture. Here's how they complement each other:

1. Retrieval-Augmented Generation (RAG)

This is perhaps the most impactful combination. LangChain uses vector embeddings to:

Convert documents into vector representations
Store these vectors in vector databases (like Pinecone, Chroma, or FAISS)
Retrieve relevant context when a user asks a question
Pass the retrieved context to an LLM for generating accurate, grounded responses

from langchain_community.vectorstores import Chroma
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain.chains import RetrievalQA

# Create embeddings
embeddings = OpenAIEmbeddings()

# Store documents in vector database
vectorstore = Chroma.from_documents(documents, embeddings)

# Create RAG chain
qa_chain = RetrievalQA.from_chain_type(
    llm=ChatOpenAI(),
    retriever=vectorstore.as_retriever(),
    return_source_documents=True
)

# Get answer with context
result = qa_chain.invoke("What is vector similarity search?")

2. Semantic Search and Filtering

Vector embeddings enable LangChain to perform semantic searches rather than just keyword matching:

Find documents that are conceptually similar to a query
Filter results based on semantic relevance scores
Group similar content automatically
Identify duplicate or near-duplicate content

3. Memory and Context Management

LangChain uses embeddings to:

Store conversation history as vectors
Retrieve relevant past interactions based on current context
Maintain long-term memory across sessions
Recognize when similar situations have occurred before

Practical Use Cases

Enterprise Knowledge Base

A company can use LangChain with vector embeddings to create an intelligent knowledge base:

Ingest internal documents, manuals, and policies
Convert all content to vector embeddings
Allow employees to ask natural language questions
Retrieve relevant information and generate comprehensive answers

Customer Support Chatbot

Build a chatbot that:

Understands customer queries semantically
Retrieves relevant support articles and FAQs
Provides accurate, context-aware responses
Learns from past interactions to improve over time

Research Assistant

Create a tool that:

Analyzes academic papers and research documents
Finds connections between different research areas
Summarizes complex topics based on relevant sources
Recommends related papers based on semantic similarity

Implementation Considerations

Choosing the Right Embedding Model

Consider these factors:

Accuracy vs. Cost: OpenAI embeddings are highly accurate but costly; open-source models like all-MiniLM-L6-v2 are free but less accurate
Dimensionality: Higher dimensions capture more nuance but require more storage and computation
Language support: Some models work better for specific languages or domains

Vector Database Selection

Popular options include:

Chroma: Lightweight, easy to set up, great for development
Pinecone: Fully managed, scalable, production-ready
FAISS: High performance, optimized for similarity search

Performance Optimization

Chunking strategy: How you split documents affects retrieval quality
Indexing techniques: HNSW, IVF, or other indexing methods impact speed and accuracy
Hybrid search: Combine vector search with keyword filtering for better results
Caching: Store frequent queries and results to reduce latency

Code Example: Building a Simple RAG System

Here's a complete example showing how to build a basic RAG system with LangChain and vector embeddings:

import os
from langchain_community.document_loaders import TextLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_community.vectorstores import Chroma
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain.chains import RetrievalQA

# Set up environment
os.environ["OPENAI_API_KEY"] = "your-api-key"

# Load and process documents
loader = TextLoader("knowledge_base.txt")
documents = loader.load()
text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=0)
texts = text_splitter.split_documents(documents)

# Create embeddings and vector store
embeddings = OpenAIEmbeddings()
vectorstore = Chroma.from_documents(texts, embeddings)

# Create QA system
llm = ChatOpenAI(temperature=0)
qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    retriever=vectorstore.as_retriever(),
    return_source_documents=True
)

# Query the system
query = "Explain vector embeddings in simple terms"
result = qa_chain.invoke(query)
print(f"Answer: {result['result']}")
print(f"Sources: {[doc.metadata for doc in result['source_documents']]}")

Best Practices

Data Preparation

Clean and preprocess text before embedding
Remove noise, standardize formatting
Consider domain-specific preprocessing

Evaluation Metrics

Recall@k: Percentage of relevant documents in top k results
Mean Reciprocal Rank (MRR): Quality of ranking
Precision: Relevance of retrieved results
End-to-end accuracy: How well the final answer addresses the query

Security and Privacy

Be mindful of sensitive data in vector databases
Implement proper access controls
Consider data retention policies
Be aware of embedding model biases

Future Directions

The intersection of LangChain and vector embeddings is rapidly evolving:

Multimodal embeddings: Combining text, images, and audio embeddings
Real-time indexing: Near-instantaneous updates to knowledge bases
Cross-lingual capabilities: Seamless understanding across languages
Personalized embeddings: Tailored to individual users or organizations
Edge deployment: Running embedding models on devices

Conclusion

LangChain and vector embeddings represent a paradigm shift in how we build AI applications. By combining the power of large language models with semantic understanding through vector representations, developers can create systems that truly understand context, retrieve relevant information, and generate meaningful responses.

The beauty of this combination lies in its accessibility – with the right tools and understanding, developers can build sophisticated AI applications without needing deep expertise in machine learning. As these technologies continue to evolve, we can expect even more powerful and intuitive applications that bridge the gap between human intention and machine capability.

Whether you're building an enterprise knowledge base, a customer support system, or a research assistant, the LangChain + vector embeddings combination provides a robust foundation for creating intelligent, context-aware applications that deliver real value to users.

Getting Started Resources

LangChain Documentation: https://python.langchain.com
Hugging Face Embeddings: https://huggingface.co/models?library=sentence-transformers
Pinecone: https://www.pinecone.io

What use cases are you exploring with LangChain and vector embeddings? Share your experiences in the comments below!

🚀 Stop Hallucinating! Build a RAG Chatbot in 5 Minutes with LangChain

Ananya S — Mon, 09 Feb 2026 18:03:48 +0000

Ever asked an AI about something that happened yesterday, only for it to confidently lie to your face? That’s because LLMs are frozen in time—limited by their training data.

Enter RAG (Retrieval-Augmented Generation). It’s like giving your AI an open-book exam. Instead of guessing, it looks up the answer in your documents first.

In this post, we’re building a simple RAG pipeline using LangChain. Let’s dive in! 🏊‍♂️

🔥 The "Big Idea"

RAG works in three simple steps:

Index: Chop your documents into small "chunks" and turn them into math (vectors).

Retrieve: When a user asks a question, find the chunks that match best.

Augment: Stuff those chunks into the prompt and let the AI summarize them.

🛠️The Setup
You'll need a few libraries. Open your terminal and run:

pip install langchain langchain-openai langchain-community chromadb pypdf

💻 The Code
Here is a complete, minimal script to chat with a PDF. Replace "your_api_key" with your actual OpenAI key.

import os
from langchain_community.document_loaders import PyPDFLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings, ChatOpenAI
from langchain_community.vectorstores import Chroma
from langchain.chains import RetrievalQA

# 1. Set your API Key
os.environ["OPENAI_API_KEY"] = "sk-..."

# 2. Load your data (Change this to your PDF path!)
loader = PyPDFLoader("my_awesome_doc.pdf")
data = loader.load()

# 3. Chop it up! (Chunking)
# We split text so the AI doesn't get overwhelmed.
text_splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=100)
chunks = text_splitter.split_documents(data)

# 4. Create the "Brain" (Vector Store)
# This turns text into vectors and stores them locally.
vectorstore = Chroma.from_documents(
    documents=chunks, 
    embedding=OpenAIEmbeddings()
)

# 5. Build the RAG Chain
llm = ChatOpenAI(model_name="gpt-4o", temperature=0)
rag_chain = RetrievalQA.from_chain_type(
    llm=llm,
    chain_type="stuff", # "Stuff" all chunks into the prompt
    retriever=vectorstore.as_retriever()
)

# 6. Ask away!
question = "What is the main conclusion of this document?"
response = rag_chain.invoke(question)

print(f"🤖 AI: {response['result']}")

🤔 Why did we do that?
RecursiveCharacterTextSplitter: Why not just feed the whole PDF? Because LLMs have a "context window" (limit). Chunking keeps the info bite-sized and relevant.

ChromaDB: This is our temporary database. It stores the "meaning" of our text so we can search it numerically.

chain_type="stuff": This is the funniest name in LangChain. It literally means "stuff all the retrieved documents into the prompt."

🌟 Pro-Tips for the Road
Overlap matters: Notice chunk_overlap=100? This ensures that if a sentence is cut in half, the context lives in both chunks.

Local Models: Don't want to pay for OpenAI? Swap ChatOpenAI for Ollama and run it 100% locally!

Garbage In, Garbage Out: If your PDF is a messy scan, your RAG will be messy too. Clean your data!

🎁 Wrapping Up

You just built a production-grade logic loop. RAG is the backbone of almost every AI startup today. Whether it's a legal bot, a medical assistant, or a "Chat with your Resume" tool—you now have the blueprint.

What are you planning to build with RAG? Let me know in the comments! 👇

From Words to Meaning: Core NLP Concepts Every Beginner Must Know

Ananya S — Tue, 27 Jan 2026 17:48:10 +0000

In the previous post, we covered the basics of NLP such as tokenization, stemming, lemmatization, and stop words.

In this continuation, we will understand how machines extract meaning from text and represent language numerically.

1. Named Entity Recognition (NER)

Named Entity Recognition (NER) is an NLP technique used to identify and classify real-world entities in text.

Common entity types:

Person
Organization
Location
Date
Time
Money
Percentage

Example sentence:

Elon Musk is the CEO of Tesla and lives in the USA.

NER output:

Elon Musk → PERSON

Tesla → ORGANIZATION

USA → LOCATION

Why NER is important:

Helps extract structured information from unstructured text
Used in resume parsing and document processing
Widely applied in medical and legal NLP systems
Improves search engines and chatbots

2. Bag of Words (BoW)

Bag of Words is one of the simplest techniques to convert text into numbers.

Core idea:

Word order is ignored
Grammar is ignored
Only word frequency matters

Example:

Sentence 1: I love NLP  
Sentence 2: I love AI

Vocabulary:

[I, love, NLP, AI]

Vector representation:

Sentence 1 → [1, 1, 1, 0]

Sentence 2 → [1, 1, 0, 1]

Advantages:

Very easy to implement
Works well for small datasets
Useful as a baseline model

Limitations:

No understanding of context
No semantic meaning
Treats all words as equally important

3. TF-IDF (Term Frequency – Inverse Document Frequency)

TF-IDF improves Bag of Words by assigning importance scores to words.

TF-IDF= TF*IDF

TF(t,d)= Total number of terms in document d/Number of times term t appears in document d

IDF(t)=log( Number of documents containing term t/Total number of documents)

Intuition:

Words that occur frequently in a document are important.
Words that occur in many documents are less important.

Components:

Term Frequency (TF): frequency of a word in a document

Inverse Document Frequency (IDF): rarity of the word across documents

Why TF-IDF is better than BoW:

Reduces importance of common words like the and is
Highlights meaningful words

Performs well in:

Search engines
Spam detection
Document similarity tasks

Limitations:

Does not capture semantic meaning
Synonyms are treated as different words

4. Word2Vec

Word2Vec represents words as dense numerical vectors that capture meaning and context.

Key idea:

Words used in similar contexts have similar meanings. The vectors used to represent King, Queen, Man and Woman when undergo arithmetic operations, give results as below.

Famous examples:

King − Man + Woman ≈ Queen

Paris − France + Italy ≈ Rome

Word2Vec has 2 components:

1. CBOW (Continuous Bag of Words)

Predicts a word using surrounding context.

Sentence: "Raj went to school yesterday"
Window size: 1

Input: [Raj, to] → Output: went
Input: [went, school] → Output: to
Input: [to, yesterday] → Output: school

Working:

The context words are converted to one-hot vectors.
These vectors are summed or averaged
They are passed through the hidden layer.
The model predicts the target word
Error is calculated
Weights are updated using backpropagation

2. Skip-Gram

Predicts surrounding words using a target word.
For same sentence,

Target word: went
Context words: Raj, to

Training pairs:
Input: went → Output: Raj
Input: went → Output: to

Target = to
Context words: went, school

Training pairs:
Input: to → Output: went
Input: to → Output: school

Target = school
Context words: to, yesterday

Training pairs:
Input: school → Output: to
Input: school → Output: yesterday

Working

The target word is converted to a one-hot vector
Passed through the hidden layer
The model predicts each context word
Error is calculated
Weights are updated using backpropagation

👉 The hidden layer weights become the word embeddings

Advantages:

Captures semantic relationships
Produces dense and meaningful embeddings
Useful for clustering and similarity tasks

Limitation:

Same word has the same vector in all contexts

Example:
bank (river) and bank (money)

This limitation is addressed by contextual models like BERT.

When to Use Each Technique?

Use Bag of Words when:

Building simple text classifiers
Creating baseline NLP models

Use TF-IDF when:

Working on search systems
Performing document similarity
Detecting spam

Use Word2Vec when:

Semantic similarity is important
Building recommendation systems
Clustering text data

Final Thoughts

These techniques show the evolution of NLP from counting words to weighting word importance to understanding semantic meaning.
They form the foundation for modern NLP and Generative AI systems.

🌱 NLP for Beginners: Understanding the Basics of Natural Language Processing

Ananya S — Thu, 22 Jan 2026 16:03:32 +0000

Natural Language Processing (NLP) is one of the most exciting areas of Artificial Intelligence today. From chatbots and search engines to spam detection and sentiment analysis, NLP helps machines understand human language.

If you’re just starting out and feel confused by terms like tokenization or lemmatization, this post will give you a clear and gentle introduction.

📌 What is Natural Language Processing (NLP)?

Natural Language Processing (NLP) is a subfield of Artificial Intelligence that enables computers to understand, analyze, and generate human language.

In simple terms:

NLP allows machines to work with text and speech in a meaningful way.

Real-world applications of NLP

Chatbots and virtual assistants
Google Search and autocomplete
Spam email detection
Sentiment analysis of reviews
Language translation

🗺️ A Beginner-Friendly Roadmap to Learn NLP

Before diving into complex models, it’s important to understand how text is processed.

A simple conceptual roadmap

Text Preprocessing
- Tokenization
- Stop words removal
- Stemming
- Lemmatization
Text Representation
- Bag of Words
- TF-IDF
- Word Embeddings
Classical NLP Tasks
- Text classification
- Sentiment analysis
- Named Entity Recognition
Advanced NLP (Later Stage)
- Transformers
- BERT
- GPT
- Large Language Models

🧹 Why Text Preprocessing is Important

Machines don’t understand language like humans do.

Example sentence: "I am learning Natural Language Processing!"

To a machine, this is just a sequence of characters.

Text preprocessing helps convert raw text into a format that machine learning models can understand.

✂️ Tokenization

Tokenization is the process of breaking text into smaller units called tokens.

Example

Sentence:

"I love learning NLP"

After tokenization:

["I", "love", "learning", "NLP"]

Types of tokenization

Word tokenization
Sentence tokenization
Subword tokenization (used in transformers)

🛑 Stop Words

Stop words are commonly used words that usually don’t add much meaning to the text.

Examples:

is, am, are, the, a, an, in, on, and

Why remove stop words?

They add noise
They increase dimensionality
They often don’t help in tasks like classification

🌿 Stemming

Stemming reduces words to their root form by removing suffixes.

Fast
Not always linguistically correct

Common stemming algorithms:

PorterStemmer() : just removes suffix or prefix without context understanding.
SnowballStemmer() : better than PorterStemmer and supports many languages.
RegexStemmer() : removes prefix or suffix based on given expression to be removed.

words=['eating','eaten','eat','write','writes','history','mysterious','mystery','finally','finalised','historical']
from nltk.stem import PorterStemmer
stemming=PorterStemmer()
for word in words:
    print(word+"------>"+ stemming.stem(word))

OUTPUT:
eating------>eat
eaten------>eaten
eat------>eat
write------>write
writes------>write
history------>histori
mysterious------>mysteri
mystery------>mysteri
finally------>final
finalised------>finalis
historical------>histor

Stemming just removes prefixes or suffixes and doesn't give meaning words.

🍃 Lemmatization

Lemmatization converts words into their dictionary base form, called a lemma.
NLTK provides WordNetLemmatizer class which is a thin wrapper around the wordnet corpus

from nltk.stem import WordNetLemmatizer
## WordNet is a dictionary dataset which has words with their base form.We need to download this dataset to use WordNetLemmatizer
import nltk
nltk.download('wordnet')
lemmatizer=WordNetLemmatizer()
lemmatizer.lemmatize('going') #output: going
lemmatizer.lemmatize('going', pos='v') #ouput : go
#This lemmatize command we can add pos_tags that identify the word as verb, noun, adjective, etc. to help decide how to go to root word.
## Parts of Speech: Noun -n, Verb-v, adverb-r, adjective-a. Default pos tag is 'n'

Considers grammar and context
Produces meaningful words
More accurate but slower than stemming

⚖️ Stemming vs Lemmatization

Feature	Stemming	Lemmatization
Speed	Fast	Slower
Accuracy	Lower	Higher
Output	May not be a real word	Always a valid word
Grammar-aware	❌	✅

🧠 Final Thoughts

NLP is not magic — its structured text processing combined with machine learning.
Which is your favorite concept in NLP?
Drop a comment down below!