DEV Community: Nagachinmay KN

Sentence, Word & Subword Tokenisation Explained

Nagachinmay KN — Mon, 08 Dec 2025 18:35:39 +0000

What is Tokenisation? (How Machines Break Text into Pieces)

Before any machine learning model can understand text, it must first break the text into smaller units.
This process is called Tokenisation.

In simple words:

Tokenisation = Converting text into meaningful numerical pieces (tokens)

Because machines cannot understand raw text, they only understand numbers.
✅ 1. Sentence-Level Tokenisation

This method splits a paragraph into individual sentences.

Example:

Input:

I love machine learning. It is powerful. It is the future.

Output Tokens:

I love machine learning.

It is powerful.

It is the future.

✅ Used in:

Document summarization

News classification

Chatbots
✅ 2. Word-Level Tokenisation

This splits a sentence into individual words.

Example:

Input:

I love machine learning

Output Tokens:

love

machine

learning

✅ This was used in:

Traditional NLP

RNNs

LSTMs

Early translation models

⚠️ But it has problems:

Huge vocabulary

Unknown words (OOV problem)

Spelling variations break models

✅ 3. Subword-Level Tokenisation (Modern AI Uses This)

This splits words into smaller meaningful parts.

Example:

Word:

Unbelievable

Subword Tokens:

belie

vable

Or:

Playing → Play + ing

✅ Used in:

GPT

BERT

ChatGPT

Google Translate

✅ Solves:

Unknown words

Large vocabulary issues

Better generalization

Common subword algorithms:

BPE (Byte Pair Encoding)

WordPiece

Unigram LM

🔁 Why Tokenisation Is Essential for Sequential Models

Remember:

Sequential Models work only on sequences.

Tokenisation is what creates the sequence.

➡️ Text → Tokens → Numbers → Model → Output

Without tokenisation:
❌ No RNN
❌ No LSTM
❌ No Transformers
❌ No ChatGPT

✅ Final Tokenisation Summary Table

Level	Input	Output	Used In
Sentence	Paragraph	Sentences	Summarization
Word	Sentence	Words	Classic NLP
Subword	Word	Sub-parts	Transformers

“From sequential data to tokenisation, this is how machines slowly learn to read, remember, and predict — just like us.”

🧠 How Machines Learn from Order: A Simple Guide to Sequential Models

Nagachinmay KN — Mon, 08 Dec 2025 18:15:42 +0000

🧠 A Simple Guide to Sequential Models— A Beginner-Friendly Introduction to How Machines Learn from Order

When we humans read a sentence, listen to music, or observe the weather, order matters.
Machines also need to understand this order — and that’s exactly where Sequential Data and Sequential Models come into play.

In this beginner-friendly article, I’ll explain:

What sequential data is
Why order is important
Types of sequence models
Markov and Autoregressive ideas
And why modern AI uses Transformers today

All in a simple, story-like way.

🔹 What is Sequential Data?

Sequential data is data where the order of values really matters.

Let’s take a famous sentence:

“The quick brown fox jumps over the lazy dog”

This sentence contains all 26 English alphabets.
Now if we change the order randomly:

“Dog the over jumps fox brown quick lazy”

The same words exist, but the meaning is destroyed.
That’s the power of sequence.

✅ Examples of Sequential Data:

Weather records over time
Stock prices
Sentences and speech
Music
DNA sequences

🔹 Where Is Sequential Data Used?

Machines use sequential data in real life for:

🌦 Weather forecasting
📈 Stock market prediction
🗣 Speech recognition
🌍 Language translation
📩 Spam detection
🤖 Chatbots

All of these depend on previous data to predict the future.

🔹 What Are Sequential Models?

Sequential models are ML models that take ordered data as input and/or output.

There are three main types:

✅ 1. Many-to-One (Sequence → One Output)

Input: A full sequence
Output: A single label

🧾 Example:

Email text → Spam / Not Spam
Review → Positive / Negative

✅ 2. One-to-Many (One Input → Sequence Output)

Input: One value
Output: A sequence

🖼 Example:

Image → Caption
Topic → Generated Story

✅ 3. Many-to-Many (Sequence → Sequence)

Input: A sequence
Output: A new sequence

🌍 Example:

English → French Translation
Chatbot responses

This is called Sequence-to-Sequence (Seq2Seq).

🔹 What is an Autoregressive Model?

Autoregressive models predict the next value using previous values.

For example:

Given previous stock prices → predict next price
Given previous words → predict next word

These models:
✅ Work step-by-step
✅ Are time efficient
✅ Use finite past data

⚠️ But they struggle when:

Long context is required
Very deep meaning is needed This is called the long-term dependency problem.

🔹 What is a Markov Model?

A Markov Model works on one powerful idea:

“The future depends only on the present, not the entire past.”

🌦 Weather Example:

If today is sunny, then:

Tomorrow has:
- 70% chance sunny
- 20% chance rainy
- 10% chance windy

It
does not look at last week — only today.

✅ Advantages:

Simple
Memory efficient
Fast

❌ Limitations:

Oversimplifies complex problems
Not suitable for language or deep reasoning

🔹 Then Why Do We Need Memory Models?

Some problems need:

Context from far back
Meaning across long sentences
Emotional tone
Story flow

For this, we use:

RNN (Recurrent Neural Networks)
LSTM & GRU → solve memory loss
Transformers with Attention → modern solution

🔹 Why Transformers Changed Everything

Older models processed data step-by-step, which was slow.

Transformers use:
✅Self-Attention
✅Parallel processing
✅Long-range memory

This is what powers:

ChatGPT
Google Translate
BERT
GPT
Modern NLP systems

✅ Final Summary

Concept	Key Idea
Sequential Data	Order matters
Many-to-One	Sequence → One output
One-to-Many	One → Sequence
Seq2Seq	Sequence → Sequence
Autoregressive Model	Predict next using previous
Markov Model	Only depends on present
Transformer	Uses attention for deep understanding

🚀 What’s Next?

This is just Day 1 of my ML learning journey.
In upcoming posts, I’ll cover:

RNN vs LSTM vs GRU
Attention mechanism deep dive
How Chat GPT actually works
Beginner ML projects

If you’re also learning ML — follow along. Let’s grow together. 💪

Bridging Ayurveda and AI: A Preprint Journey to Modernize Traditional Healthcare

Nagachinmay KN — Thu, 12 Jun 2025 18:55:55 +0000

For most, Ayurveda is tradition. For me, it’s transformation.

I’ve grown up watching my family preserve a 100-year-old Ayurvedic store, Sri Nanjundeshwara Mart. While the world went digital, we struggled to adapt. That’s when I began building SN Mart Insight Engine — a platform combining AI and Ayurveda to make traditional wellness more accessible and intelligent.

I recently published a preprint that proposes:
• Intelligent tracking of Ayurvedic product consumption.
• AI-driven insights for manufacturers like BV Pundit and Hamdard.
• A roadmap to digitize legacy health systems for the next generation.

🌐 Why It Matters:
• Ayurveda is a 5000-year-old science that deserves technological upliftment.
• AI can personalize wellness by understanding real-time consumption patterns.
• This project bridges cultural heritage with cutting-edge tech.

🔍 My Preprint:

📄 Title: [SN Mart Insight Engine: A No-Code Al Platform for Consumer Intelligence in Ayurveda Retail]
📥 Link: [DOI: 10.13140/RG.2.2.27664.72967 ]

✨ What’s Next:

I’m now looking to:
• Collaborate with researchers who believe in blending tradition with technology.
• Connect with university groups or student communities interested in health-tech, AI, or cultural preservation.

If you’re from a university, research lab, or even an independent community:
👉 Feel free to re-publish, share, or build on this work. Full credits welcomed. Let’s make Ayurveda intelligent together.

Let’s talk!
💬 [chinmaynataraj@gmail.com] |

🔗[http://www.linkedin.com/in/nagchinmaykapini]

Cross posted on Medium : https://medium.com/@chinmaynataraj/bridging-ayurveda-and-ai-a-preprint-journey-to-modernize-traditional-healthcare-5b322ebc6a55

Python Variables Explained for Beginners

Nagachinmay KN — Fri, 16 May 2025 20:45:16 +0000

Hey folks, welcome!

Welcome to the first article in my new series:
Python – Made Simpler

Why Python?

Unlike C, Python is widely used—and one big reason is libraries.

Python comes with many built-in libraries, and you can import even more depending on what you need. Think of a library as a toolkit that helps you do things faster and easier.

A Real-Life Example: Randomness

Let’s say I’m playing a card game with friends. I never know which card I’ll get next—why? Because it’s shuffled. It’s random!

In Python, we can use something similar: import random as rand

Just like shuffling cards gives you unpredictable results, the random library in Python helps you generate random numbers or make random choices.

In real life, randomness is unpredictable.
In Python, we simulate that randomness using code.

Cool, right? That’s the magic of Python!

Introduction to Variables

Now let’s talk about variables.

Imagine a kitchen shelf with jars:
One jar has sugar, another has salt, another coffee powder.

In programming, think of those jars as variables. Each jar (or variable) stores something inside it—just like a variable stores data.

Syntax Example:

Let’s say we name our jars like this in Python:

jar1 = "Sugar"
jar2 = "Salt"
jar3 = "Coffee powder"

Now, if we want to see what’s inside, we can do:
print(jar1) # Output: Sugar
print(jar2) # Output: Salt
print(jar3) # Output: Coffee powder

Easy, right?
A variable = a label + value. You store something and use it later.

That’s a wrap!

In the next article, we’ll look at types of variables and best practices for naming them.

Happy coding!
If you found this interesting or have suggestions, feel free to comment — I’d love your feedback!

You can refer my book : https://chinmaynataraj.gitbook.io/chinmaynataraj/