My Journey with Sentence Transformers, OpenAI Embeddings, and Vector Databases

#ai #vectordatabase #openai #beginners

When I started my Python learning journey, I’ll be honest I didn’t feel very comfortable with it. Even writing simple scripts felt like a challenge.

But one of my projects pushed me into a completely new world:

AI embeddings
Transformers
Vector databases
Dimensions

All new concepts, all at once. At first, I felt overwhelmed. But while struggling to deploy my application, I slowly built a better understanding of how it all actually works.

This post is a recap of that journey.

Starting with Sentence Transformers

My first step was using Sentence Transformers.

These models take a sentence and turn it into a vector a long list of numbers that represent the meaning of the sentence.

Example:

“How old are you?” → [0.12, -0.98, 0.56, …]
“What is your age?” → [0.11, -0.95, 0.58, …]

Even though the wording is different, the vectors are close. That’s how machines know two sentences are similar.

This idea amazed me. But when I tried to deploy it, I hit a wall.

The Limitation I Faced

Running a Sentence Transformer locally is heavy:

The model files are big.
It eats memory.
It needs good CPU/GPU power.

When I deployed my app on a normal server, it just crashed.

That was frustrating. I was still learning Python, and now I was stuck with GPU/infra problems.

Discovering OpenAI Embeddings

The breakthrough came when I tried OpenAI embeddings.

Instead of me running the heavy model, I just send my text to OpenAI’s API.

OpenAI runs the Transformer model on their powerful servers.
They send back the vector (embedding).
My app just stores it and uses it.

No crashes. No GPU setup. It just worked.

👉 For me, this was like moving from running my own generator (loud, heavy, unreliable) to simply plugging into the electricity grid.

Where Do These Vectors Go?

Once I had embeddings, the next question was: Where do I store them?

That’s when I discovered Qdrant, a vector database.

A normal database is good at numbers and text, but not at searching long vectors.
Qdrant is built exactly for this. It can:

Store embeddings.
Search for “nearest neighbors” quickly (find similar sentences or documents).

With this, I could actually build semantic search and other AI-powered features.

The “Dimension” Concept

One confusing part was dimensions.

OpenAI embeddings (like text-embedding-3-small) produce 1536-dim vectors. Some Sentence Transformers produce 768 or even 3072 dimensions.

What does this mean?

A 2D map has 2 numbers (latitude, longitude).
A 3D object has 3 numbers (x, y, z).
A 1536-dim embedding has 1536 numbers, each capturing a tiny piece of meaning.

The higher the dimension, the more detail the model can capture.
But higher dimensions also mean bigger storage and slower search.

Once I understood this, embeddings stopped looking “magical” they became just another way of describing information, but in a very detailed mathematical form.

What I Learned

This project was special in my Python journey because:

I started uncomfortable with Python.
I suddenly faced Transformers, embeddings, and vector databases all new concepts.
I failed first (Sentence Transformer was too heavy).
I found a practical solution (OpenAI embeddings + Qdrant).
And along the way, I gained a deeper understanding of how these pieces fit together.