Embedding Selection for Turkish: Semantic Search with Multilingual

#rag #sistemmimarisi #software

What is Embedding and Why is it Important?

Embedding is a function that converts text into a high-dimensional vector; sentences with the same meaning are located close to each other in this vector space. In semantic search systems, the similarity between query and document vectors directly determines the quality of the search. Comparing millions of documents in a few milliseconds is only possible with the right embedding selection. In this section, I will summarize the basic logic of embedding and its role in search performance.

How to Select Embedding for Turkish in Multilingual Models?

When selecting a multilingual model for Turkish, first, look at the model's vocabulary and pre-training dataset's representation of Turkish. If the model tokenizes Turkish characters (ç, ğ, ı, ö, ş, ü) correctly, it produces consistent vectors for the same word in different forms (e.g., "kredi" vs "krediği"). Using the sentence-transformers package from HuggingFace, you can download and test a model with just a few lines of code.

from sentence_transformers import SentenceTransformer

model_name = "sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2"
model = SentenceTransformer(model_name)

sentences = [
    "Müşteri siparişini onayladı.",
    "Müşteri siparişini onayladı."
]
embeddings = model.encode(sentences, normalize_embeddings=True)
print(embeddings.shape)  # (2, 384)

This code snippet generates 384-dimensional vectors for two identical sentences and, with normalize_embeddings=True, the cosine similarity directly approaches 1.0. In a real environment, running different models with the same test set and comparing cosine similarity values is the primary criterion for the selection process.

Impact of Language Features and Tokenization

Turkish, being an agglutinative language, has long word forms; tokenization errors can negate this. For example, BPE-based tokenizers split the word "çalıştırma" into multiple tokens, while a word-based tokenizer might keep it as a single token. This difference can produce vectors of different lengths and, consequently, different similarity scores. When selecting a model, consider how the tokenizer splits words into subword units like sentence-piece or WordPiece, especially for long sentences, to increase consistency.

The example below compares the same sentence tokenized by two different tokenizers:

python -c "from transformers import AutoTokenizer;
t1 = AutoTokenizer.from_pretrained('bert-base-multilingual-cased');
t2 = AutoTokenizer.from_pretrained('xlm-roberta-base'); # Using XLM-R to demonstrate different tokenization behavior
print('BERT tokens:', t1.tokenize('çalışma ortamı güvenli mi?'));
print('XLM-R tokens:', t2.tokenize('çalışma ortamı güvenli mi?'))"

The output shows that BERT splits the sentence into 6 parts using WordPiece-based tokenization, while XLM-R splits it into 5 parts. This difference can affect the sentence's position in the vector space, particularly critical in retrieval-augmented generation (RAG) scenarios where correct tokenization is key.

Model Comparison: mBERT vs XLM-R vs LaBSE vs Sentence-Transformer

Model	Size (Parameters)	Output Dimension	Turkish Pre-training Data
`bert-base-multilingual-cased`	110M	768	104-language Wikipedia
`xlm-roberta-base`	~270M	768	100+ language CommonCrawl
`LaBSE`	~800M	768	Texts in 109 languages
`paraphrase-multilingual-MiniLM-L12-v2`	~22M	384	Multilingual texts (50+ languages)

The parameter counts in the table are approximate and based on information from model cards or related publications. In a production environment, a lighter model like MiniLM offers a balance between low latency and sufficient accuracy. This decision is made based on the need to run on CPU-only servers or edge devices.

💡 Model Selection: Memory and Latency

Smaller models typically require less RAM, while larger models demand more. For instance, some MiniLM models can use under 50 MB of memory for inference, while larger models may require hundreds of MB or even GB of memory. Memory usage depends on the model size, precision used (FP32, FP16, INT8), and batch size. Measure latency using time python script.py and compare results directly.

Performance Measurement and Evaluation Methodology

When measuring performance, two primary metrics are used: Recall@k and Mean Reciprocal Rank (MRR). Recall@k gives the rate of finding the correct document within the first k results; MRR provides an average score by taking the reciprocal of the correct document's rank. A sample evaluation pipeline includes:

Preparing a suitable-sized Turkish query-document test set.
Generating model embeddings for each query and searching for the corresponding document in the FAISS index.
Retrieving the top k results and comparing them with the ground truth.

Below is a simple evaluation code using faiss:

import faiss, numpy as np
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2")
queries = [...]          # Query sentences
docs = [...]             # Document sentences
doc_embeddings = model.encode(docs, normalize_embeddings=True)
index = faiss.IndexFlatIP(doc_embeddings.shape[1]) # IndexFlatIP is used for inner product similarity
index.add(np.array(doc_embeddings, dtype='float32'))

def recall_at_k(k=10):
    correct = 0
    for q, true_id in zip(queries, range(len(queries))):
        q_emb = model.encode([q], normalize_embeddings=True)
        _, I = index.search(np.array(q_emb, dtype='float32'), k)
        if true_id in I[0]:
            correct += 1
    return correct / len(queries)

print("Recall@10:", recall_at_k(10))

This script prints the Recall@10 value directly to the terminal; saving the output and comparing it with different models makes the decision process objective. In a real project, adding GPU acceleration and reporting measurement time can further enhance the evaluation.

Production Environment: Embedding Service and Query Optimization

In a production environment, serving the embedding calculation via a REST or gRPC API provides a scalable solution. The following Docker-Compose example runs a sentence-transformers service with uvicorn:

version: "3.8"
services:
  embed-api:
    image: python:3.11-slim
    working_dir: /app
    volumes:
      - ./:/app
    command: >
      sh -c "pip install sentence-transformers fastapi uvicorn &&
             uvicorn embed_api:app --host 0.0.0.0 --port 8000" # Uvicorn command line arguments
    ports:
      - "8000:8000"

# embed_api.py
from fastapi import FastAPI
from pydantic import BaseModel
from sentence_transformers import SentenceTransformer

app = FastAPI() # Creating a FastAPI application
model = SentenceTransformer("sentence-transformers/paraphrase-multilingual-MiniLM-L12-v2")

class TextPayload(BaseModel): # Using Pydantic BaseModel
    text: str

@app.post("/embed")
def embed(payload: TextPayload):
    vec = model.encode([payload.text], normalize_embeddings=True)[0].tolist()
    return {"embedding": vec}

This service can handle batch requests to process multiple queries simultaneously; monitoring CPU usage with top or htop is crucial to determine resource limits. For query optimization, configuring an index like FAISS IVF+PQ provides millisecond response times for large collections. Additionally, adding a caching layer (e.g., Redis) to store embeddings for frequently queried texts can significantly reduce latency.

curl -X POST http://localhost:8000/embed -H "Content-Type: application/json" \
     -d '{"text":"How does semantic search work in Turkish texts?"}'

This command returns an embedding in response to a single HTTP request; measuring the response time with time curl ... can verify performance targets.

Conclusion

Selecting an embedding for Turkish text depends on the model's representation of Turkish in its pre-training dataset, tokenizer compatibility, and resource constraints in the production environment. MiniLM-like lightweight models offer a balance between low latency and sufficient accuracy, while larger models like LaBSE provide higher recall but require more memory and GPU resources. By measuring Recall@k and MRR with a real test set, you can objectively determine the most suitable model for your needs. The next step involves integrating the selected model into a CI/CD pipeline and monitoring its performance continuously with tools like Prometheus and Grafana.