📌 Day 13: 21 Days of Building a Small Language Model: Positional Encodings📌

Today’s blog was simple but eye-opening: transformers don’t understand word order on their own.

Without positional information, a model sees text as a bag of tokens.

So in a sentence like “The student asked the teacher about the student’s project”, both “student” tokens look the same unless we tell the model where they appear.

Adding position sounds easy, but it isn’t.

Naive approaches either overpower meaning or make training harder. This is why positional encodings exist, they quietly teach models that order matters.

And without them, a transformer can’t reliably tell the difference between

✅ “The algorithm processes the data” and

✅ “The data processes the algorithm.”

Small detail. Huge impact.

🔗 Blog link: https://www.linkedin.com/pulse/day-13-21-days-building-small-language-model-prashant-lakhera-spo5c

I’ve covered all the concepts here at a high level to keep things simple. For a deeper exploration of these topics, feel free to check out my book "Building A Small Language Model from Scratch: A Practical Guide."

✅ Gumroad: https://plakhera.gumroad.com/l/BuildingASmallLanguageModelfromScratch

✅ Amazon: https://www.amazon.com/dp/B0G64SQ4F8/

✅ Leanpub: https://leanpub.com/buildingasmalllanguagemodelfromscratch