DEV Community: Ritesh Totlani

You Cloned the Aggregate. Why Is Your Read-Only Operation Still Changing State?

Ritesh Totlani — Wed, 23 Sep 2026 07:20:58 +0000

Imagine you have a domain operation that answers a very common question:

“What would happen if we continued this trip from here?”

You don't want to actually continue the trip.

You only want to preview the result.

Maybe you're calculating:

the estimated arrival time,
the remaining driving time,
the stops that would be reached,
or the driver's remaining compliance time.

A natural solution is to clone the aggregate and run the real domain logic against the clone.

That sounds like a very good design.

And it usually is.

But there is a subtle problem:

What if the domain logic changes something that isn't part of the aggregate?

That's the problem we're going to walk through in this article.

The requirement: “Run the real logic, but don't change anything”

Let's say we have a Trip aggregate:

public class Trip
{
    public Guid Id { get; }
    public Guid TruckId { get; }

    private readonly List<Stop> _stops = new();

    public IReadOnlyCollection<Stop> Stops => _stops;

    public double DistanceTravelledSoFar { get; private set; }
    public TimeSpan TimeElapsedSoFar { get; private set; }

    // ...
}

The trip contains its stops and its own progress.

Now imagine we already have a domain service that knows how to move the trip forward:

public void AdvanceRoute(
    Trip trip,
    DriverComplianceLedger ledger)
{
    foreach (var stop in trip.Stops)
    {
        // business rules...

        stop.MarkReached();

        trip.AddDistance(...);

        ledger.AddDrivingTime(...);
    }
}

Notice something important.

The operation receives two objects:

AdvanceRoute(
    Trip,
    DriverComplianceLedger
)

And both can change.

The trip changes.

The driver's compliance record changes.

Now let's build our preview.

First attempt: just clone the Trip

The first implementation almost writes itself:

public TripPreview Preview(
    Trip trip,
    DriverComplianceLedger ledger)
{
    var clone = trip.Clone();

    AdvanceRoute(clone, ledger);

    return CreatePreview(clone);
}

At first glance, this looks perfectly reasonable.

We're not passing the real Trip.

We're passing a clone.

Therefore:

The real trip cannot be changed.

Right?

Not quite.

Let's look at what actually happens

Suppose our production state is:

Trip #123
    ├── Stop A
    ├── Stop B
    └── Stop C

Driver Ledger
    └── 6 hours driving time remaining

We clone the trip:

Original state

Trip #123
    ├── Stop A
    ├── Stop B
    └── Stop C


Simulation state

Trip #123'
    ├── Stop A'
    ├── Stop B'
    └── Stop C'

So far, everything looks safe.

But the ledger hasn't been cloned:

Original state

Trip #123
    ├── Stop A
    ├── Stop B
    └── Stop C

Driver Ledger  ← REAL OBJECT
    └── 6 hours remaining


Simulation state

Trip #123'
    ├── Stop A'
    ├── Stop B'
    └── Stop C'

Now the engine starts executing.

It does:

stop.MarkReached();
trip.AddDistance(...);
ledger.AddDrivingTime(...);

The first two operations modify the clone.

The third modifies the real ledger.

Our "read-only" preview just changed production state.

But isn't `Trip.Clone()` a deep clone?

Yes.

And this is where the problem becomes interesting.

Here's a perfectly good implementation:

public Trip Clone()
{
    var clone = new Trip(
        Id,
        TruckId,
        TruckingCompanyId,
        StartedAt)
    {
        CompletedAt = CompletedAt,
        DistanceTravelledSoFar = DistanceTravelledSoFar,
        TimeElapsedSoFar = TimeElapsedSoFar,
    };

    foreach (var stop in _stops)
    {
        clone._stops.Add(stop.Clone());
    }

    return clone;
}

This is a proper deep clone of the Trip.

Every stop is cloned.

Every relevant piece of trip state is copied.

So what's wrong?

Nothing is wrong with the clone.

The mistake is assuming that cloning Trip automatically isolates the entire operation.

It doesn't.

The important distinction

This is the key idea of the whole article:

The object you clone and the state your operation can mutate are not necessarily the same thing.

When we look at Trip.Clone(), we're asking:

What belongs to the Trip object graph?

But when we build a simulation, we need to ask:

What can this operation change?

Those questions happen to have the same answer in simple cases.

They don't always have the same answer.

Object graph vs. mutation boundary

Consider the object graph:

Trip
 ├── Stop
 │    └── ...
 ├── Stop
 │    └── ...
 └── Stop
      └── ...

A deep clone correctly isolates that graph.

But our operation actually works with:

AdvanceRoute()
    │
    ├── Trip
    │    └── Stops
    │
    └── DriverComplianceLedger

The ledger isn't part of Trip.

There is no reference like:

public DriverComplianceLedger Ledger { get; }

inside Trip.

So you won't discover this dependency by inspecting the aggregate.

You discover it by inspecting the behavior.

The method signature gives us the first clue

Whenever I see a simulation or preview implementation, I start with the method being executed.

For example:

public void AdvanceRoute(
    Trip trip,
    DriverComplianceLedger ledger)

Immediately, I have two potential sources of mutation.

Then I look inside the method.

If I find:

trip.AddDistance(...);

and:

ledger.AddDrivingTime(...);

the mutation boundary becomes obvious:

                AdvanceRoute
                     │
          ┌──────────┴──────────┐
          │                     │
        Trip                 Ledger
          │                     │
       mutable              mutable

Both objects participate in the state transition.

Therefore both need to be isolated for a true simulation.

The fix

The fix is actually very small.

Clone both objects:

public TripPreview Preview(
    Trip trip,
    DriverComplianceLedger ledger)
{
    var simulatedTrip = trip.Clone();
    var simulatedLedger = ledger.Clone();

    AdvanceRoute(
        simulatedTrip,
        simulatedLedger);

    return CreatePreview(simulatedTrip);
}

Now the execution looks like this:

REAL STATE
─────────────────────────
Trip
Ledger


        │
        │ Clone
        ▼


SIMULATION STATE
─────────────────────────
Trip Clone
Ledger Clone


        │
        │ AdvanceRoute()
        ▼


Simulation Result

The real state is never given to the mutating operation.

That is what actually makes the preview read-only.

Why I prefer this over writing another calculator

At this point, you might ask:

Why not just create a separate method that calculates the result without changing anything?

For example:

public TripPreview EstimateTrip(Trip trip)
{
    // duplicate the route calculation
    // duplicate compliance rules
    // duplicate stop logic
}

This looks cleaner.

But now we have two implementations of the business rules:

Production Engine
       │
       └── Business rules


Preview Engine
       │
       └── Business rules

Imagine someone changes the production rule next month:

if (driver.HasExceededDailyLimit())
{
    // new business rule
}

The production engine gets updated.

Someone now has to remember to update the preview engine too.

If they don't, the system can produce:

Production:  08:42 arrival

Preview:     09:15 arrival

Neither implementation necessarily has a technical bug.

The problem is architectural:

The same business rule now exists in two places.

Clone-and-replay avoids that duplication.

We execute the real domain behavior.

We just execute it against isolated state.

So what exactly should we clone?

This is where the original rule:

"Clone the aggregate."

isn't quite strong enough.

A better rule is:

Clone every mutable state that belongs to the execution boundary of the operation.

That might be one object.

It might be two.

It might be considerably more.

For example:

Simple operation

Calculate(Trip)
     │
     └── Trip

Here:

var simulatedTrip = trip.Clone();

may be enough.

But:

More complex operation

Calculate(
    Trip,
    DriverLedger,
    MutableContext)
       │
       ├── Trip
       ├── DriverLedger
       └── MutableContext

requires a larger isolation boundary.

A practical way to find the boundary

When you're implementing a preview or simulation, don't start by asking:

"What should I clone?"

Start with:

"What does the operation write to?"

Then work through the operation systematically.

1. Look at the parameters

Calculate(
    Trip trip,
    DriverLedger ledger)

You already have two candidates.

2. Look for mutations

Search for operations such as:

trip.Update(...);
ledger.Add(...);
state.Set(...);
collection.Add(...);

3. Follow mutable references

A parameter might itself contain mutable objects:

Context
 ├── Rules
 ├── State
 │    └── MutableState
 └── Cache

Don't stop at the top-level parameter.

4. Check services involved in the operation

This is particularly important if a domain service or application service maintains mutable state.

5. Define the simulation boundary

Only after understanding the writes should you decide what must be copied or isolated.

There is an even deeper lesson here

This isn't really a Clone() problem.

It's a boundary problem.

DDD gives us useful boundaries such as aggregates and bounded contexts.

But an aggregate boundary answers a domain question:

Which state belongs together for consistency?

A simulation asks a different question:

Which state must be isolated while this behavior executes?

Those boundaries can be identical.

But they don't have to be.

That's why this assumption is dangerous:

Aggregate boundary
        =
Mutation boundary

Sometimes it does.

Sometimes it doesn't.

You have to examine the behavior.

What about external dependencies?

There is another interesting consequence.

Suppose your operation eventually does this:

AdvanceRoute(
    trip,
    ledger,
    weatherService);

Now cloning the first two objects isn't necessarily enough.

What if weatherService maintains mutable state?

Or:

AdvanceRoute(
    trip,
    ledger,
    repository);

What if the operation writes something through the repository?

For example:

repository.Save(ledger);

Now our "simulation" has escaped its isolation boundary entirely.

This is why a truly read-only execution mode often requires thinking about more than object cloning.

You need to understand the side effects of the whole execution path.

For a pure in-memory domain operation, cloning mutable state may be sufficient.

For an operation involving persistence, messaging, external services, or caches, the isolation strategy may need to be different.

For example:

                  Simulation
                      │
       ┌──────────────┼──────────────┐
       │              │              │
   Domain state    Persistence    Messaging
       │              │              │
      clone       suppress/mock   suppress

The important point is that clone-and-replay is a technique, not a guarantee.

The rule I now use in code reviews

When I see something like:

var clone = aggregate.Clone();

RunBusinessLogic(clone);

I don't immediately ask:

"Is this clone deep enough?"

I ask:

"Is this clone the complete mutation boundary of RunBusinessLogic?"

Then I inspect the method.

That small change in the question is surprisingly powerful.

Because the bug isn't necessarily:

Shallow clone ❌

It can be:

Perfect deep clone
        +
Incomplete isolation boundary
        =
Unsafe simulation

A small checklist

If you're implementing a preview, simulation, dry run, or what-if operation, check:

□ What operation am I executing?
□ What parameters does it receive?
□ Which parameters can be mutated?
□ What nested objects can be mutated?
□ Are there mutable fields or collections?
□ Do called services maintain mutable state?
□ Can the operation write to a repository?
□ Can it publish messages or events?
□ Can it modify shared caches/state?
□ Have all required mutable participants been isolated?

The first few questions are usually enough for a simple domain operation.

The later questions become important as the operation crosses architectural boundaries.

The takeaway

The original idea behind clone-and-replay is still a good one.

If you want a preview to behave exactly like the real operation, executing the same business logic is often preferable to maintaining a second implementation.

But don't confuse:

"I cloned the aggregate."

with:

"I isolated the operation."

Those are different statements.

A deep clone tells you that the aggregate's object graph is isolated.

It doesn't tell you that the execution is isolated.

For that, you need to understand the operation's mutation boundary.

So the next time you implement a "read-only" simulation, don't ask:

What object should I clone?

Ask:

What can this operation change?

Then make that entire state part of your simulation boundary.

That's the difference between a preview that merely looks read-only and one that is actually read-only.

The example comes from a freight-marketplace side project https://github.com/RTO-The-Coder/freight-marketplace. The projection walk is implemented in RouteEtaCalculator, with the aggregate and related domain code under Freight.Domain/Fleet.

German Sentiment Analysis with BERT: nlptown vs oliverguhr on 20 Real Sentences

Ritesh Totlani — Fri, 18 Sep 2026 07:50:02 +0000

A support team reading reviews in five different languages just marked an angry customer as "satisfied."

Not because anyone was careless. Because the word that flagged it — "super," "genial," "toll" — looked positive to whatever was scoring it. Sarcasm doesn't translate through a keyword, and a dissatisfied customer doesn't always spell it out directly either.

So let's figure out how AI can actually help here — and whether running real models on real sentences gives us something we can trust, or just a faster way to get it wrong. I took two German sentiment models and ran them head-to-head on the same twenty real-world sentences. Here's what happened.

The models

nlptown/bert-base-multilingual-uncased-sentiment — multilingual, trained on product reviews, outputs 1–5 stars. I mapped 1–2 stars to negative, 3 to neutral, 4–5 to positive.
oliverguhr/german-sentiment-bert — German-specific, outputs positive/negative/neutral natively.

The eval set

20 German sentences, hand-written to cover the cases that actually break sentiment models: clear positive/negative/neutral, negation, sarcasm, mixed sentiment, and colloquial/regional phrasing. A sample:

{
  "text": "Nicht schlecht, aber auch nicht besonders gut.",
  "expected": "neutral",
  "category": "negation"
},
{
  "text": "Der Support war ja wirklich der Hammer, drei Wochen keine Antwort auf meine Anfrage.",
  "expected": "negative",
  "category": "sarcasm"
},
{
  "text": "Also ehrlich, des Ding is a Schrott, funktioniert von Anfang an net richtig.",
  "expected": "negative",
  "category": "colloquial"
}

Full 20-sentence eval set (german_sentiment_eval_set.json)

[
  {
    "id": 1,
    "text": "Der Kundenservice war schnell und hat mein Problem sofort gelöst.",
    "expected": "positive",
    "category": "clear_positive"
  },
  {
    "id": 2,
    "text": "Ich bin sehr zufrieden mit der Lieferung, alles kam pünktlich und unbeschädigt an.",
    "expected": "positive",
    "category": "clear_positive"
  },
  {
    "id": 3,
    "text": "Endlich mal ein Produkt, das genauso funktioniert wie beschrieben. Top!",
    "expected": "positive",
    "category": "clear_positive"
  },
  {
    "id": 4,
    "text": "Das Gerät ist nach zwei Wochen kaputt gegangen und der Support antwortet nicht.",
    "expected": "negative",
    "category": "clear_negative"
  },
  {
    "id": 5,
    "text": "Absolute Zeitverschwendung, ich fordere eine vollständige Rückerstattung.",
    "expected": "negative",
    "category": "clear_negative"
  },
  {
    "id": 6,
    "text": "Die Verpackung war beschädigt und der Inhalt ist zerbrochen angekommen.",
    "expected": "negative",
    "category": "clear_negative"
  },
  {
    "id": 7,
    "text": "Die Bestellung wurde am Dienstag aufgegeben und am Donnerstag versendet.",
    "expected": "neutral",
    "category": "clear_neutral"
  },
  {
    "id": 8,
    "text": "Ich habe eine Frage zur Rechnungsstellung für meinen letzten Einkauf.",
    "expected": "neutral",
    "category": "clear_neutral"
  },
  {
    "id": 9,
    "text": "Können Sie mir bitte mitteilen, wann das Produkt wieder verfügbar ist?",
    "expected": "neutral",
    "category": "clear_neutral"
  },
  {
    "id": 10,
    "text": "Nicht schlecht, aber auch nicht besonders gut.",
    "expected": "neutral",
    "category": "negation"
  },
  {
    "id": 11,
    "text": "Das Produkt ist nicht so schlecht, wie ich erwartet hatte.",
    "expected": "positive",
    "category": "negation"
  },
  {
    "id": 12,
    "text": "Ich kann nicht behaupten, dass ich begeistert bin.",
    "expected": "negative",
    "category": "negation"
  },
  {
    "id": 13,
    "text": "Der Support war ja wirklich der Hammer, drei Wochen keine Antwort auf meine Anfrage.",
    "expected": "negative",
    "category": "sarcasm"
  },
  {
    "id": 14,
    "text": "Toll, das dritte Ersatzgerät und wieder ein Defekt. Genau das, was ich wollte.",
    "expected": "negative",
    "category": "sarcasm"
  },
  {
    "id": 15,
    "text": "Wow, eine Wartezeit von 45 Minuten in der Hotline, das nenne ich mal Kundenservice.",
    "expected": "negative",
    "category": "sarcasm"
  },
  {
    "id": 16,
    "text": "Die Lieferung war super schnell, aber die Qualität hat mich leider enttäuscht.",
    "expected": "mixed",
    "category": "mixed_sentiment"
  },
  {
    "id": 17,
    "text": "Der Mitarbeiter war sehr freundlich, konnte mein Problem aber nicht lösen.",
    "expected": "mixed",
    "category": "mixed_sentiment"
  },
  {
    "id": 18,
    "text": "Guter Preis, aber die Verarbeitung wirkt ziemlich billig.",
    "expected": "mixed",
    "category": "mixed_sentiment"
  },
  {
    "id": 19,
    "text": "Also ehrlich, des Ding is a Schrott, funktioniert von Anfang an net richtig.",
    "expected": "negative",
    "category": "colloquial"
  },
  {
    "id": 20,
    "text": "Läuft bei euch, echt top gemacht, weiter so!",
    "expected": "positive",
    "category": "colloquial"
  }
]

The code

from transformers import pipeline

NLPTOWN_MODEL = "nlptown/bert-base-multilingual-uncased-sentiment"
OLIVERGUHR_MODEL = "oliverguhr/german-sentiment-bert"

def map_nlptown_stars_to_label(raw_label: str) -> str:
    stars = int(raw_label[0])
    if stars <= 2:
        return "negative"
    if stars == 3:
        return "neutral"
    return "positive"

nlptown_pipe = pipeline("sentiment-analysis", model=NLPTOWN_MODEL)
oliverguhr_pipe = pipeline("sentiment-analysis", model=OLIVERGUHR_MODEL)

for item in eval_set:
    nt_raw = nlptown_pipe(item["text"])[0]
    og_raw = oliverguhr_pipe(item["text"])[0]
    nt_label = map_nlptown_stars_to_label(nt_raw["label"])
    og_label = og_raw["label"].lower()
    # compare nt_label, og_label against item["expected"]

A couple of scoring calls worth flagging: results are reported as raw counts (e.g. "9/13"), not percentages — with 20 sentences, a percentage implies more precision than the sample supports. Neutral is scored separately from positive/negative, since oliverguhr predicts it natively while nlptown only reaches it via the star mapping. And "mixed" sentiment sentences are reported qualitatively only, because neither model has a mixed label to be right or wrong against.

Full script (sentiment_comparison.py)

"""
German sentiment analysis: head-to-head comparison of two HuggingFace models.

Models compared:
  - nlptown/bert-base-multilingual-uncased-sentiment
    Multilingual, outputs 1-5 stars. Mapped to labels as:
    1-2 stars -&gt; negative, 3 -&gt; neutral, 4-5 -&gt; positive.
  - oliverguhr/german-sentiment-bert
    German-specific, outputs positive/negative/neutral natively.

Scoring notes:
  - Results are reported as raw counts ("9/13 correct"), not percentages,
    since the eval set is small (~20 sentences) and percentages would
    imply more precision than the sample size supports.
  - Neutral-labeled sentences are scored separately from positive/negative,
    because oliverguhr predicts "neutral" natively while nlptown only
    reaches it via the star mapping above.
  - "Mixed" sentiment sentences are reported qualitatively only (not
    scored correct/incorrect), since neither model has a "mixed" label.

Usage:
    python sentiment_comparison.py
"""

import json
import sys
import time
from pathlib import Path

from transformers import pipeline

if sys.platform == "win32":
    sys.stdout.reconfigure(encoding="utf-8")

EVAL_SET_PATH = Path(__file__).parent / "german_sentiment_eval_set.json"
RESULTS_PATH = Path(__file__).parent / "results_table.md"

NLPTOWN_MODEL = "nlptown/bert-base-multilingual-uncased-sentiment"
OLIVERGUHR_MODEL = "oliverguhr/german-sentiment-bert"

NLPTOWN_LABEL = "nlptown (multilingual)"
OLIVERGUHR_LABEL = "oliverguhr (German-specific)"


def map_nlptown_stars_to_label(raw_label: str) -&gt; str:
    """Fixed mapping, decided before any run and never adjusted afterward."""
    stars = int(raw_label[0])
    if stars &lt;= 2:
        return "negative"
    if stars == 3:
        return "neutral"
    return "positive"


def load_eval_set() -&gt; list[dict]:
    with open(EVAL_SET_PATH, encoding="utf-8") as f:
        return json.load(f)


def run_model(pipe, sentences: list[dict]) -&gt; list[dict]:
    """Run a pipeline over all sentences, returning per-sentence raw output + latency."""
    results = []
    for item in sentences:
        start = time.perf_counter()
        raw = pipe(item["text"])[0]
        elapsed = time.perf_counter() - start
        results.append({"raw_label": raw["label"], "score": raw["score"], "latency": elapsed})
    return results


def build_rows(eval_set: list[dict], nlptown_raw: list[dict], oliverguhr_raw: list[dict]) -&gt; list[dict]:
    rows = []
    for item, nt, og in zip(eval_set, nlptown_raw, oliverguhr_raw):
        nt_mapped = map_nlptown_stars_to_label(nt["raw_label"])
        og_label = og["raw_label"].lower()
        rows.append({
            "id": item["id"],
            "text": item["text"],
            "category": item["category"],
            "expected": item["expected"],
            "nlptown_mapped": nt_mapped,
            "nlptown_latency": nt["latency"],
            "oliverguhr_label": og_label,
            "oliverguhr_latency": og["latency"],
            "models_agree": nt_mapped == og_label,
        })
    return rows


def print_legend():
    print("Legend:")
    print(f"  nlptown    = {NLPTOWN_MODEL} (multilingual, stars mapped to labels)")
    print(f"  oliverguhr = {OLIVERGUHR_MODEL} (German-specific, native labels)")
    print()


def print_results_table(rows: list[dict]):
    print(f"{'ID':&lt;3} {'Category':&lt;16} {'Expected':&lt;9} {NLPTOWN_LABEL:&lt;22} {OLIVERGUHR_LABEL:&lt;28} {'Agree?':&lt;7}")
    print("-" * 95)
    for r in rows:
        agree = "yes" if r["models_agree"] else "no"
        print(f"{r['id']:&lt;3} {r['category']:&lt;16} {r['expected']:&lt;9} "
              f"{r['nlptown_mapped']:&lt;22} {r['oliverguhr_label']:&lt;28} {agree:&lt;7}")


def compute_summary(rows: list[dict]) -&gt; dict:
    posneg_rows = [r for r in rows if r["expected"] in ("positive", "negative")]
    neutral_rows = [r for r in rows if r["expected"] == "neutral"]
    mixed_rows = [r for r in rows if r["expected"] == "mixed"]

    return {
        "posneg_total": len(posneg_rows),
        "nlptown_posneg_correct": sum(1 for r in posneg_rows if r["nlptown_mapped"] == r["expected"]),
        "oliverguhr_posneg_correct": sum(1 for r in posneg_rows if r["oliverguhr_label"] == r["expected"]),
        "neutral_total": len(neutral_rows),
        "nlptown_neutral_correct": sum(1 for r in neutral_rows if r["nlptown_mapped"] == r["expected"]),
        "oliverguhr_neutral_correct": sum(1 for r in neutral_rows if r["oliverguhr_label"] == r["expected"]),
        "mixed_rows": mixed_rows,
        "agreement_count": sum(1 for r in rows if r["models_agree"]),
        "nlptown_avg_latency_ms": sum(r["nlptown_latency"] for r in rows) / len(rows) * 1000,
        "oliverguhr_avg_latency_ms": sum(r["oliverguhr_latency"] for r in rows) / len(rows) * 1000,
    }


def print_summary(rows: list[dict], summary: dict):
    print("\n--- Summary ---")

    print(f"Positive/negative accuracy ({summary['posneg_total']} sentences):")
    print(f"  {NLPTOWN_LABEL}: {summary['nlptown_posneg_correct']}/{summary['posneg_total']} correct")
    print(f"  {OLIVERGUHR_LABEL}: {summary['oliverguhr_posneg_correct']}/{summary['posneg_total']} correct")

    print(f"\nNeutral accuracy ({summary['neutral_total']} sentences, reported separately):")
    print(f"  {NLPTOWN_LABEL}: {summary['nlptown_neutral_correct']}/{summary['neutral_total']} correct")
    print(f"  {OLIVERGUHR_LABEL}: {summary['oliverguhr_neutral_correct']}/{summary['neutral_total']} correct")

    print(f"\nAgreement rate: {summary['agreement_count']}/{len(rows)} sentences")

    print("\nAvg latency:")
    print(f"  {NLPTOWN_LABEL}: {summary['nlptown_avg_latency_ms']:.1f} ms")
    print(f"  {OLIVERGUHR_LABEL}: {summary['oliverguhr_avg_latency_ms']:.1f} ms")

    mixed_rows = summary["mixed_rows"]
    print(f"\n--- Mixed-sentiment cases ({len(mixed_rows)}, qualitative only — no model has a 'mixed' label) ---")
    for r in mixed_rows:
        print(f'  [{r["id"]}] "{r["text"]}"')
        print(f"      {NLPTOWN_LABEL}: {r['nlptown_mapped']}  |  {OLIVERGUHR_LABEL}: {r['oliverguhr_label']}")

    disagreements = [r for r in rows if not r["models_agree"]]
    print(f"\n--- Disagreement cases ({len(disagreements)}) ---")
    for r in disagreements:
        print(f'  [{r["id"]}] "{r["text"]}"')
        print(f"      expected: {r['expected']}  |  {NLPTOWN_LABEL}: {r['nlptown_mapped']}  |  "
              f"{OLIVERGUHR_LABEL}: {r['oliverguhr_label']}")


def write_markdown_table(rows: list[dict]):
    lines = [
        f"**nlptown** = `{NLPTOWN_MODEL}` (multilingual, stars mapped to labels)  ",
        f"**oliverguhr** = `{OLIVERGUHR_MODEL}` (German-specific, native labels)",
        "",
        "| ID | Category | Expected | nlptown (multilingual) | oliverguhr (German-specific) | Agree? |",
        "|---|---|---|---|---|---|",
    ]
    for r in rows:
        agree_icon = "✅" if r["models_agree"] else "❌"
        lines.append(
            f"| {r['id']} | {r['category']} | {r['expected']} | {r['nlptown_mapped']} | "
            f"{r['oliverguhr_label']} | {agree_icon} |"
        )

    RESULTS_PATH.write_text("\n".join(lines), encoding="utf-8")
    print(f"\nMarkdown results table written to {RESULTS_PATH}")


def main():
    eval_set = load_eval_set()

    print("Loading models...")
    nlptown_pipe = pipeline("sentiment-analysis", model=NLPTOWN_MODEL)
    oliverguhr_pipe = pipeline("sentiment-analysis", model=OLIVERGUHR_MODEL)

    print(f"Running {len(eval_set)} sentences through both models...\n")
    nlptown_raw = run_model(nlptown_pipe, eval_set)
    oliverguhr_raw = run_model(oliverguhr_pipe, eval_set)

    rows = build_rows(eval_set, nlptown_raw, oliverguhr_raw)

    print_legend()
    print_results_table(rows)

    summary = compute_summary(rows)
    print_summary(rows, summary)

    write_markdown_table(rows)


if __name__ == "__main__":
    main()

The results

Positive/negative (13 sentences):

nlptown: 9/13 correct
oliverguhr: 9/13 correct

Neutral (4 sentences, scored separately):

nlptown: 2/4 correct
oliverguhr: 2/4 correct

Overall agreement between the two models: 10/20 — a coin flip.

The tie on accuracy is itself the finding: paying for German-specific fine-tuning bought zero net accuracy over a general multilingual model on this set. But which sentences each model got wrong is where they diverge, and that's the part a single accuracy number hides.

Sarcasm (3 sentences, all expected negative):

Sentence (gist)	nlptown	oliverguhr
"Support was such a joy, three weeks no reply"	negative ✅	negative ✅
"Great, third replacement unit, another defect"	positive ❌	positive ❌
"Wow, 45-min hotline wait, that's service"	positive ❌	negative ✅

Both models caught 1 of 3. Both missed the same one identically. Neither model has any mechanism for sarcasm — they're scoring surface-level positive words ("Hammer," "Wartezeit... Kundenservice") without the pragmatic flip.

Negation (3 sentences):

Sentence (gist)	Expected	nlptown	oliverguhr
"Not bad, but not great either"	neutral	neutral ✅	negative ❌
"Not as bad as I expected"	positive	neutral ❌	positive ✅
"Can't say I'm thrilled"	negative	positive ❌	positive ❌

Both models got exactly one right — different ones. On the double-negative case ("can't say I'm thrilled"), both flipped to positive, apparently anchored on "thrilled" and missing the negation scope entirely.

Mixed sentiment (3 sentences, qualitative — no ground truth to score against):

Sentence (gist)	nlptown	oliverguhr
"Delivery was fast, but quality disappointed me"	negative	negative
"Employee was friendly, but couldn't solve my problem"	positive	negative
"Good price, but the build feels cheap"	neutral	negative

oliverguhr called all three negative; nlptown scattered across all three labels. Neither behavior is "wrong" since there's no single correct answer for genuinely mixed sentiment — but if your pipeline needs one label per ticket, oliverguhr's consistent pessimism and nlptown's inconsistency are different failure modes to plan around.

Verdict

For straightforward positive/negative German text, either model works — they tied at 9/13, so the German-specific model isn't earning its specialization on the easy cases. Where they differ is on the messy 40% of real customer language: sarcasm fools both equally, negation trips up both in different, unpredictable ways, and on genuinely mixed sentiment oliverguhr defaults to negative while nlptown just guesses. If your pipeline touches sarcastic or backhanded feedback at any real volume, neither model is safe to trust alone — you need a rule-based or LLM-based fallback layer, not a better BERT model.

Entities Are Not Rows: A Domain Model That Behaves

Ritesh Totlani — Wed, 09 Sep 2026 08:38:42 +0000

Most codebases I've worked in call the classes in the middle of the diagram "entities" and call the layer they live in the "domain." Both words are doing less work than they look like they are. You can have a folder called Domain, a dependency rule that points every arrow inward, and still have a model that is a database schema with getters.

The example here is driving-hours compliance for a fleet of trucks — rest-and-break rules, not CRUD.

The word "entity" is doing two different jobs

In the onion (or clean, or hexagonal) diagram, Entities are the innermost ring and every other ring depends on them. That's a dependency rule, and it's a good one. But it says nothing about what those entities are.

In practice, they tend to become this: a class of public getters and setters, no methods, and every rule that governs them living in a service one or more rings out. Nothing stops you writing a driver's hours-driven-today straight past the legal limit — the object itself has no idea what a legal value looks like. Whether that number is allowed is DriverComplianceService's problem, somewhere else, if anyone remembers to ask it.

The other reading of "entity" — the one from Evans' book that the architecture diagram borrowed the word from — is that the entity is the behaviour. You cannot put a Driver into a state the business would reject, because Driver will not let you. There is no setter to call.

Same word. Two completely different objects. The dependency rule is satisfied by both.

The domain: driving-hours compliance

The system simulates drivers over time. On a regular cadence it has to answer one question per driver: may this driver legally keep driving right now, or must they stop — and for how long?

The real regulation this is modeled on — EU drivers'-hours rules — is dense: caps on continuous driving before a break, caps per day and per week, rolling multi-week limits, rest periods with their own variants. None of the specific numbers matter for what follows. What matters is the shape: a driver accumulates driving time, hits limits, and owes breaks or rest before continuing.

Two objects carry this. Driver is the identity — who they are, and the driving/break/rest cadence they're subject to. DriverComplianceState is the ledger — the running record of what they've actually driven, and how close they are to each limit. Driver owns the ledger; nothing outside Driver is allowed to create one.

The words in that relationship — ledger, break, daily cap, two-week cap — are the same words the regulation uses and the same words that appear in the class and method names. That correspondence is the ubiquitous language, and it's the thing the rest of this post is really about.

An aggregate you can't corrupt

Here is the Driver aggregate, trimmed (full file):

public sealed class Driver
{
    public Guid Id { get; private set; }
    public string FirstName { get; private set; } = null!;
    public string LastName { get; private set; } = null!;

    // The driving/break/rest cadence this driver is subject to.
    public DrivingRules Rules { get; private set; } = null!;

    // This driver's driving-time compliance ledger. Null until the driver
    // first starts driving — see ResetComplianceForNewTrip.
    public DriverComplianceState? ComplianceState { get; private set; }

    private Driver() { }  // EF Core materializer only

    private Driver(Guid id, string firstName, string lastName, DrivingRules rules)
    {
        Id = id; FirstName = firstName; LastName = lastName; Rules = rules;
    }

    public static Driver Create(string firstName, string lastName, DrivingRules rules)
    {
        if (string.IsNullOrWhiteSpace(firstName))
            throw new ArgumentException("First name is required.", nameof(firstName));
        if (string.IsNullOrWhiteSpace(lastName))
            throw new ArgumentException("Last name is required.", nameof(lastName));
        ArgumentNullException.ThrowIfNull(rules);

        return new Driver(Guid.NewGuid(), firstName, lastName, rules);
    }

    // The only state change application code can ask for. The driver builds
    // its own ledger — you cannot hand it one from outside.
    public void ResetComplianceForNewTrip(DateTime tripStartedAt)
    {
        ComplianceState = new DriverComplianceState(Id, tripStartedAt);
    }
}

Every property is private set. There is no public constructor — the only way to make a Driver from application code is Create, which validates its inputs and throws on a blank name or null rules. (The parameterless constructor exists solely so EF Core's materializer can rehydrate one from a row; it is not a construction path anyone else can use.)

And there is exactly one mutation the outside world can request: ResetComplianceForNewTrip. Note what it does not do — it doesn't accept a DriverComplianceState. The driver constructs its own compliance ledger, anchored to the trip start. You cannot inject a ledger, cannot set one to null, cannot put the driver into a half-initialised state.

Compare that to the anemic version: no Driver { get; set; } soup, and no DriverValidator.Validate(driver) sitting in another assembly hoping every call site remembers to invoke it. The invariant and the data it protects are the same object.

One more thing worth noticing: Driver doesn't hold its compliance data as fields of its own. It holds a separate object, DriverComplianceState, and it's the only thing allowed to create one. The identity and the ledger are two objects, deliberately — but the ledger only ever comes into existence through the aggregate that owns it.

The rules live in the model, not a service layer

If Driver guards its own invariants, where does the actual regulation live — the logic that decides a driver has hit a limit and must now take a break or rest?

In a DriverRuleEngine (interface, implementation). Here is its core surface:

public interface IDriverRuleEngine
{
    // Pure. Answered from the ledger and the driver's limits alone — no Driver, no DB.
    DriverEligibility IsEligibleToDriveNow(
        DriverComplianceState ledger,
        RestRuleLimits limits);   // the caps and break/rest durations, as one object

    // Rolls the ledger forward one tick: accrues driving, begins the required
    // break or rest when a limit is hit. Returns a domain outcome + events.
    RestRuleOutcome Advance(
        DriverComplianceState ledger,
        TimeSpan elapsedTick,
        DateTime simulatedNow,
        DrivingRules rule,
        RestRuleLimits limits);
}

public sealed record DriverEligibility(
    bool IsEligible,
    IneligibilityReason? Reason,       // which limit was hit, if any — OnBreak, DailyCapReached, ...
    int? MinutesUntilEligible);

Every parameter is a domain type — a ledger, a limits table, a rules variant, an elapsed tick. Never a Driver itself. Never a DTO, never a DbContext. And every return is a domain type: a DriverEligibility record whose Reason is a typed IneligibilityReason, not a bare status code; a RestRuleOutcome that carries domain events like TruckWentIntoRest and TruckResumedDriving.

Inside, every branch reads in the regulation's own language — hit the two-week cap, and the reason says so:

if (ledger.WeeklyDrivingMinutesThisWeek + ledger.WeeklyDrivingMinutesPriorWeek
        >= limits.MaxTwoWeekDrivingMinutes)
    return new DriverEligibility(false, IneligibilityReason.TwoWeekCapReached, null);

if (ledger.ContinuousDrivingMinutesSinceBreak
        >= limits.MaxContinuousDrivingMinutesBeforeBreak)
    return new DriverEligibility(false, IneligibilityReason.OnBreak, null);

This is the "use-case" logic that a clean-architecture diagram would put in an outer ring. But it isn't a service reaching into a bag of fields and writing them back — it's a domain service in the DDD sense: behaviour that doesn't belong to any single entity, still expressed entirely in domain terms, operating on domain objects, emitting domain events. IsEligibleToDriveNow is a pure function of (ledger, limits). IsEligibleToDriveFuture — "will this driver still be legal a while from now?" — replays the deterministic drive/break/rest sequence forward on a private clone of the ledger and never touches the real one.

Same computation, either way. One version you can read aloud to the person who wrote the regulation. The other you cannot.

The take

Onion architecture gives you a dependency rule. It does not give you a domain model. You can follow every arrow inward, keep your Domain project free of framework references, and still land on anemic entities pushed around by fat services — the structure is right and the model is hollow.

DDD's building blocks are what make the centre of the diagram mean something: an aggregate that guards its own invariants, a value object that travels between contexts but stays immutable, a domain service written in the model's own language.

The test isn't "do the dependencies point inward." It's: can I put this object into a state the business would reject? If yes, it's a row with extra steps.

The model above is from a freight-marketplace side project — github.com/RTO-The-Coder/freight-marketplace. The domain lives under backend/src/Freight.Domain; Fleet and Tracking are the two subdomains in this post.

I Tested 4 RAG Chunking Strategies Everyone Recommends. 2 Were Quietly Broken.

Ritesh Totlani — Thu, 27 Aug 2026 07:06:49 +0000

If you've researched how to build a RAG system, you've seen the advice: use recursive splitting, or better yet semantic chunking, or better yet hierarchical chunking for the tricky cases. Every article explains why each one should work. Almost none of them show what happens when you actually point these techniques at real documents and check the output.

I did that. Took 140 real pages from Microsoft's Azure Service Bus documentation, ran them through four chunking strategies — recursive splitting, semantic chunking, hierarchical (parent-child) chunking, and Contextual Retrieval — and looked, chunk by chunk, at what came out the other side.

Setup: fully local stack — LM Studio for inference, nomic-embed-text for embeddings, LangChain + ChromaDB for orchestration.

Two of the four had genuine bugs. One had a subtle flaw that persisted even after I tried to prompt my way out of it. Here's what that taught me, with the numbers, the fix, and the code.

All four strategies follow the same basic pattern — load the docs, hand them to a splitter, get chunks back:

from langchain_text_splitters import RecursiveCharacterTextSplitter
# swap this import for SemanticChunker, ParentDocumentRetriever's splitters,
# or the Contextual Retrieval prompt step below — everything else stays the same

splitter = RecursiveCharacterTextSplitter.from_tiktoken_encoder(
    chunk_size=450,
    chunk_overlap=0,
)
chunks = splitter.split_documents(documents)

What differs between the four is what came out the other side. That's what this post is actually about.

The chunk that wasn't a chunk

The numbers: 1,330 chunks from 140 documents, average 266.2 tokens, 69% landed in the healthy 200-399 token range, max never exceeded the 450-token budget. Solid on paper.

Then I pulled a sample chunk to eyeball the output, expecting a paragraph of Azure documentation. Instead I got this:

title: Advanced Features in Azure Service Bus Messaging
description: This article provides a high-level overview...
ms.topic: concept-article

Metadata. Every single document in the corpus starts with a YAML header like this, and the very first "chunk" the system produced was just that header — no actual content, nothing a user's question could ever meaningfully match against.

No chunking algorithm on earth fixes this. It's not a splitting problem, it's a data problem — and it's exactly the kind of thing that never shows up in a benchmark paper, because benchmark papers use pre-cleaned datasets. Real corpora aren't pre-cleaned.

FRONTMATTER_PATTERN = re.compile(r"^---\n.*?\n---\n", re.DOTALL)
doc.page_content = FRONTMATTER_PATTERN.sub("", doc.page_content, count=1)

Chunk count before the fix: 1,366. After: 1,330. This is the first thing I'd check on any real client corpus — production RAG failures trace back to ungoverned source data more often than to chunking algorithm choice.

The 5,000-token "chunk"

The numbers, same corpus, semantic chunking instead of recursive:

	Recursive	Semantic
Chunks	1,330	692
Average	266.2 tokens	527.8 tokens
Min / Max	3 / 443	0 / 4,972
% in 200-399 range	69.0%	21.2%

Semantic chunking has a good pitch: instead of cutting text at arbitrary character counts, cut it wherever the meaning shifts. Sounds strictly better than mechanical splitting. Those min/max numbers say otherwise.

One document — a technical reference page listing various messaging protocol operations — came out as a single chunk. Not a paragraph. Not a page. The entire document, nearly 5,000 tokens, because the algorithm never found a strong enough topic shift to justify a cut. Reference documentation reads as "the same topic" to a similarity algorithm even when a human would clearly want it broken into digestible pieces.

Alongside it: 4 chunks containing literally nothing — empty strings, silently sitting in what would have become the search index. And the healthy-range rate tells the rest of the story: only 21.2% of semantic chunks landed in the 200-399 token range, against 69.0% for plain recursive splitting on the same corpus.

Neither failure is visible if you only check "did chunking run successfully" and "how many chunks did I get." Both are only visible if you check the extremes, not just the middle of the distribution.

chunks = [c for c in chunks if c.page_content.strip()]

for chunk in chunks:
    if len(encoding.encode(chunk.page_content)) > MAX_CHUNK_SIZE:
        final_chunks.extend(fallback_splitter.split_documents([chunk]))
    else:
        final_chunks.append(chunk)

After the fix: max dropped from 4,972 to 999 tokens, and the healthy-range rate more than doubled to 46.2%. The underlying lesson holds regardless: semantic chunking can fail badly on structurally repetitive content, and reference-style documentation is full of it.

The right answer, ranked dead last

The numbers: query "what are message sessions used for?" against a hierarchical (parent-child) index returned 4 results, each 2,300–4,400 characters — complete, well-formed sections, nothing came back as an orphaned fragment. Full-context return worked exactly as designed.

Here's what actually came back, in order:

Message sequencing and timestamps — a related but different feature. Similar-sounding, wrong concept.
An SDK/API reference table — mentions session-related method names, but it's a code reference, not an explanation.
Request-queue routing with header parameters — not about sessions at all.
"Enable FIFO with Sessions" — opens with "Azure Service Bus sessions enable joint and ordered processing of unbounded sequences of related messages. Use sessions in first in, first out (FIFO) and request-response patterns." That's a direct, textbook answer to the query. It came back last.

The best-matching document in the entire corpus for this question wasn't just outranked — it was buried behind three progressively less relevant results. Dense embedding similarity latched onto surface-level word overlap ("sessions," "sequencing," "FIFO") instead of which document actually answered the question.

This one stung a little, because hierarchical chunking is supposed to solve exactly this class of problem: search on small, precise pieces of text, but hand back the full surrounding section so nothing gets returned context-free. It did that — every result was complete, nothing was an orphaned fragment. It just didn't fix ranking.

Worth noting: the frontmatter bug from earlier shows up here too — the raw preview of ranks 1 and 4 both start with the YAML title/description block, the exact same data-hygiene issue that broke the very first recursive chunk.

retriever = ParentDocumentRetriever(
    vectorstore=vector_store,
    docstore=docstore,
    parent_splitter=parent_splitter,   # 1000 tokens — what the LLM sees
    child_splitter=child_splitter,     # 200 tokens — what gets searched
)
retriever.add_documents(documents)
results = retriever.invoke("what are message sessions used for?")

The lesson isn't "hierarchical chunking doesn't work." It's that completeness and correctness of ranking are two different problems, solved by two different techniques, and fixing one tells you nothing about whether you've fixed the other. Completeness needed parent-child chunking. Ranking needs hybrid search (BM25 + dense), reranking, or metadata-aware filtering — a different fix entirely. A lot of RAG advice conflates the two.

The context that leaked

The numbers: across 3 test documents, the LLM generated 13 topic statements describing chunk 1 of each. Checked against the actual source text: 12 correct, 1 leaked — a point about avoiding stored credentials via OAuth 2.0 that actually belonged to the next chunk over, not the one it was describing.

The technique — Contextual Retrieval — asks an LLM to write a short blurb situating each chunk within its source document, so an isolated piece of text doesn't lose its meaning once separated from its neighbors:

prompt = CONTEXT_PROMPT.invoke({"document": full_document, "chunk": chunk.page_content})
generated_context = chat_model.invoke(prompt).content
chunk.page_content = generated_context + "\n\n" + chunk.page_content

I tightened the prompt, told the model explicitly not to describe content from elsewhere in the document. The leak got smaller — 1 out of 13, not 1 out of 5 — but it didn't disappear.

That's not a bug I could code my way out of — it's structural. The technique works by showing the model the whole document for context. That same visibility is what let it blur a boundary. You can't fully remove the risk without removing the thing that makes the technique useful in the first place. Knowing that trade-off exists — and being honest about it instead of claiming a technique is flawless — is worth more than pretending it isn't there.

Why this is the actual point

None of these are exotic failures. They're the kind of thing that ships quietly in a lot of RAG systems, because "it ran without an error" and "it looks fine in a quick demo" get mistaken for "it works." A system that returns a metadata header instead of an answer, or ranks the wrong document first, doesn't crash — it just quietly gives worse answers, and nobody notices until a user complains.

The fix isn't a smarter algorithm. It's a habit: don't trust a technique's reputation, verify it against your own data, and specifically go looking at the edge cases — the shortest result, the longest result, the ones that don't fit the average. That's what caught every failure in this post. Not one of them showed up in the headline numbers.

If there's a single thing worth taking from this: the strategy that wins on a benchmark isn't the strategy that's right for your documents. The only way to know which one is right for yours is to actually test it — and actually look.

What's the weirdest thing you've found when you actually inspected your own RAG chunks? Drop it below.

Not All Multilingual Embedding Models Are Equal — Testing 5 of Them

Ritesh Totlani — Thu, 20 Aug 2026 07:51:45 +0000

Part of the "From Hallucination to Precision" series.

If your RAG documents are in English and your users search in German, French, or Spanish, you already know you need a "multilingual" embedding model. But which one? "Multilingual" isn't one thing — it's a label that covers models of very different sizes, ages, and training approaches, and they don't all perform the same.

I tested five multilingual embedding models against the same task: a small English knowledge base built entirely around one subject — a cat — with several near-duplicate sentences on purpose, keyword queries in three other languages, and one question — did each model retrieve the exact right sentence? All local, via sentence-transformers, no API keys.

This post shows the exact code and the exact numbers it produced.

The five models

Model	Dim	What it is
`paraphrase-multilingual-MiniLM-L12-v2`	384	Smaller, older (2021), covers 50+ languages
`paraphrase-multilingual-mpnet-base-v2`	768	Larger version of the same family
`intfloat/multilingual-e5-small`	384	Newer, competitive multilingual retrieval model
`intfloat/multilingual-e5-base`	768	Same E5 family, bigger
`distiluse-base-multilingual-cased-v2`	512	Compact, fast, 50+ languages

All five are multilingual. None are English-only. The question isn't "does multilingual beat English-only" — that's obvious. The question is: which multilingual model actually works, and how much do they differ from each other?

The setup

The knowledge base: 10 English sentences, all about a cat, on purpose.

6 near-duplicates, including two intentionally close pairs: "sat on the mat" vs. "sat on the windowsill," and "knocked over a vase" vs. "knocked over a lamp." These sit close together in meaning — a query has to land on the exact right sentence, not just "something about a cat sitting somewhere" or "something the cat knocked over."
4 distinct-context sentences: won an award at a pet show, needs a vet checkup, was adopted from a shelter, had an empty food bowl. Still about the same cat, but different enough in topic that they're an easier case.

One subject, varying difficulty — closer to how a real knowledge base often looks: many entries about the same product or feature, worded slightly differently.

Instead of full sentences, queries are short keywords — because that's how people actually search. Four keyword queries per language: German, French, Spanish, plus English as a same-language control. Several deliberately target one half of a close pair (e.g. "knocked over a lamp," not "a vase") to see whether the model finds the exact match or grabs its near-twin.

For every query: embed it, compare it against all 10 corpus sentences with cosine similarity, and see which one wins. Whichever sentence scores highest is what a real RAG pipeline would hand to the LLM as context. Wrong match, wrong context, wrong answer — silently, no crash, no error.

The code

"""
Embedding Model Comparison — 5 multilingual models, cross-lingual retrieval test.
No API keys. Runs fully local. Requires: pip install sentence-transformers
"""

from sentence_transformers import SentenceTransformer
from sentence_transformers.util import cos_sim

MODEL_NAMES = [
    "paraphrase-multilingual-MiniLM-L12-v2",
    "paraphrase-multilingual-mpnet-base-v2",
    "intfloat/multilingual-e5-small",
    "intfloat/multilingual-e5-base",
    "distiluse-base-multilingual-cased-v2",
]

LANGUAGE_ORDER = ["German", "French", "Spanish", "English"]

# Knowledge base: 10 sentences, all about a cat. 6 are near-duplicates,
# including two intentionally close pairs (sat-on-X, knocked-over-X).
# 4 are still cat-related but describe clearly different situations.
CORPUS = [
    "The cat sat on the mat.",                                # 0 - tight cluster
    "The cat sat on the windowsill.",                         # 1 - tight cluster (close to 0)
    "The cat knocked over a vase.",                           # 2 - tight cluster
    "The cat knocked over a lamp.",                           # 3 - tight cluster (close to 2)
    "The cat is sleeping in the sun.",                        # 4 - tight cluster
    "The cat scratched the sofa.",                            # 5 - tight cluster
    "The cat won an award at the pet show.",                  # 6 - distinct context
    "The cat needs to see the vet for a checkup.",            # 7 - distinct context
    "The cat was adopted from a shelter last year.",          # 8 - distinct context
    "The cat's food bowl was empty.",                         # 9 - distinct context
]

# Keyword queries: 4 per language (German, French, Spanish, English control).
# Several deliberately target one half of a close pair (e.g. "knocked over a
# lamp") to see if the model finds the exact match or grabs its near-twin.
QUERIES = [
    {"answer_index": 1, "language": "German", "text": "Katze Fensterbank gesessen"},
    {"answer_index": 3, "language": "German", "text": "Katze Lampe umgeworfen"},
    {"answer_index": 7, "language": "German", "text": "Katze Tierarzt Untersuchung"},
    {"answer_index": 9, "language": "German", "text": "Katze Napf leer"},

    {"answer_index": 0, "language": "French", "text": "chat assis tapis"},
    {"answer_index": 2, "language": "French", "text": "chat renversé vase"},
    {"answer_index": 6, "language": "French", "text": "chat prix concours"},
    {"answer_index": 8, "language": "French", "text": "chat adopté refuge"},

    {"answer_index": 1, "language": "Spanish", "text": "gato sentado ventana"},
    {"answer_index": 5, "language": "Spanish", "text": "gato arañó sofá"},
    {"answer_index": 7, "language": "Spanish", "text": "gato veterinario revisión"},
    {"answer_index": 9, "language": "Spanish", "text": "plato comida vacío"},

    {"answer_index": 3, "language": "English", "text": "cat knocked lamp"},
    {"answer_index": 4, "language": "English", "text": "cat sleeping sun"},
    {"answer_index": 6, "language": "English", "text": "cat award pet show"},
    {"answer_index": 8, "language": "English", "text": "cat adopted shelter"},
]


def get_embedding_dim(model):
    # sentence-transformers renamed this method; support both.
    if hasattr(model, "get_embedding_dimension"):
        return model.get_embedding_dimension()
    return model.get_sentence_embedding_dimension()


def test_one_model(model_name):
    print("=" * 78)
    print(f"MODEL: {model_name}")
    print("=" * 78)

    model = SentenceTransformer(model_name)
    dim = get_embedding_dim(model)
    print(f"Embedding dimension: {dim}\n")

    corpus_embeddings = model.encode(CORPUS, convert_to_tensor=True)
    results = {lang: {"hits": 0, "total": 0} for lang in LANGUAGE_ORDER}

    for query in QUERIES:
        query_embedding = model.encode(query["text"], convert_to_tensor=True)
        scores = cos_sim(query_embedding, corpus_embeddings)[0]
        top_match_index = int(scores.argmax())
        top_match_score = float(scores[top_match_index])

        is_correct = (top_match_index == query["answer_index"])
        language = query["language"]
        results[language]["total"] += 1
        if is_correct:
            results[language]["hits"] += 1

        status = "HIT " if is_correct else "MISS"
        match_percent = (top_match_score + 1) / 2 * 100
        print(f"  [{status}] ({language}) \"{query['text']}\"  ->  \"{CORPUS[top_match_index]}\"  ({match_percent:.0f}% match)")

    print()
    accuracy_by_language = {}
    for language in LANGUAGE_ORDER:
        hits = results[language]["hits"]
        total = results[language]["total"]
        accuracy = hits / total if total else 0.0
        accuracy_by_language[language] = accuracy
        print(f"  {language} accuracy: {hits}/{total} = {accuracy:.0%}")
    print()

    return dim, accuracy_by_language


def print_summary_table(all_results):
    print("=" * 78)
    print("SUMMARY — Retrieval Accuracy by Model and Query Language")
    print("=" * 78)

    header = f"{'Model':32}{'Dim':>6}"
    for language in LANGUAGE_ORDER:
        header += f"{language:>12}"
    print(header)
    print("-" * len(header))

    for model_name, (dim, accuracy_by_language) in all_results.items():
        row = f"{model_name:32}{dim:>6}"
        for language in LANGUAGE_ORDER:
            row += f"{accuracy_by_language[language]:>12.0%}"
        print(row)


def main():
    all_results = {}
    for model_name in MODEL_NAMES:
        dim, accuracy_by_language = test_one_model(model_name)
        all_results[model_name] = (dim, accuracy_by_language)

    print_summary_table(all_results)


if __name__ == "__main__":
    main()

Run it yourself:

pip install sentence-transformers
python3 embedding_comparison.py

The results

Here's the real output, one model at a time.

`paraphrase-multilingual-MiniLM-L12-v2`

Embedding dimension: 384

  [HIT ] (German) "Katze Fensterbank gesessen"  ->  "The cat sat on the windowsill."  (89% match)
  [HIT ] (German) "Katze Lampe umgeworfen"  ->  "The cat knocked over a lamp."  (93% match)
  [HIT ] (German) "Katze Tierarzt Untersuchung"  ->  "The cat needs to see the vet for a checkup."  (90% match)
  [MISS] (German) "Katze Napf leer"  ->  "The cat was adopted from a shelter last year."  (83% match)
  [HIT ] (French) "chat assis tapis"  ->  "The cat sat on the mat."  (62% match)
  [HIT ] (French) "chat renversé vase"  ->  "The cat knocked over a vase."  (93% match)
  [HIT ] (French) "chat prix concours"  ->  "The cat won an award at the pet show."  (87% match)
  [HIT ] (French) "chat adopté refuge"  ->  "The cat was adopted from a shelter last year."  (86% match)
  [HIT ] (Spanish) "gato sentado ventana"  ->  "The cat sat on the windowsill."  (94% match)
  [HIT ] (Spanish) "gato arañó sofá"  ->  "The cat scratched the sofa."  (95% match)
  [HIT ] (Spanish) "gato veterinario revisión"  ->  "The cat needs to see the vet for a checkup."  (88% match)
  [HIT ] (Spanish) "plato comida vacío"  ->  "The cat's food bowl was empty."  (78% match)
  [HIT ] (English) "cat knocked lamp"  ->  "The cat knocked over a lamp."  (97% match)
  [HIT ] (English) "cat sleeping sun"  ->  "The cat is sleeping in the sun."  (96% match)
  [HIT ] (English) "cat award pet show"  ->  "The cat won an award at the pet show."  (96% match)
  [HIT ] (English) "cat adopted shelter"  ->  "The cat was adopted from a shelter last year."  (93% match)

  German accuracy: 3/4 = 75%
  French accuracy: 4/4 = 100%
  Spanish accuracy: 4/4 = 100%
  English accuracy: 4/4 = 100%

One miss: "Katze Napf leer" (cat's bowl empty) matched to "adopted from a shelter" instead of "food bowl was empty."

`paraphrase-multilingual-mpnet-base-v2`

Embedding dimension: 768

  [HIT ] (German) "Katze Fensterbank gesessen"  ->  "The cat sat on the windowsill."  (88% match)
  [HIT ] (German) "Katze Lampe umgeworfen"  ->  "The cat knocked over a lamp."  (94% match)
  [HIT ] (German) "Katze Tierarzt Untersuchung"  ->  "The cat needs to see the vet for a checkup."  (94% match)
  [MISS] (German) "Katze Napf leer"  ->  "The cat sat on the mat."  (84% match)
  [MISS] (French) "chat assis tapis"  ->  "The cat sat on the windowsill."  (61% match)
  [HIT ] (French) "chat renversé vase"  ->  "The cat knocked over a vase."  (94% match)
  [HIT ] (French) "chat prix concours"  ->  "The cat won an award at the pet show."  (80% match)
  [HIT ] (French) "chat adopté refuge"  ->  "The cat was adopted from a shelter last year."  (89% match)
  [HIT ] (Spanish) "gato sentado ventana"  ->  "The cat sat on the windowsill."  (95% match)
  [HIT ] (Spanish) "gato arañó sofá"  ->  "The cat scratched the sofa."  (89% match)
  [HIT ] (Spanish) "gato veterinario revisión"  ->  "The cat needs to see the vet for a checkup."  (95% match)
  [HIT ] (Spanish) "plato comida vacío"  ->  "The cat's food bowl was empty."  (80% match)
  [HIT ] (English) "cat knocked lamp"  ->  "The cat knocked over a lamp."  (98% match)
  [HIT ] (English) "cat sleeping sun"  ->  "The cat is sleeping in the sun."  (98% match)
  [HIT ] (English) "cat award pet show"  ->  "The cat won an award at the pet show."  (97% match)
  [HIT ] (English) "cat adopted shelter"  ->  "The cat was adopted from a shelter last year."  (97% match)

  German accuracy: 3/4 = 75%
  French accuracy: 3/4 = 75%
  Spanish accuracy: 4/4 = 100%
  English accuracy: 4/4 = 100%

Two misses: the same "bowl empty" query missed again, plus a French miss where "chat assis tapis" (cat sat mat) — which should have gone to the exact "sat on the mat" sentence — went to "sat on the windowsill" instead. Notably, that's the near-duplicate pair the corpus was designed to test, and the largest model in the lineup got it wrong.

`intfloat/multilingual-e5-small`

Embedding dimension: 384

  [MISS] (German) "Katze Fensterbank gesessen"  ->  "The cat knocked over a vase."  (92% match)
  [HIT ] (German) "Katze Lampe umgeworfen"  ->  "The cat knocked over a lamp."  (94% match)
  [HIT ] (German) "Katze Tierarzt Untersuchung"  ->  "The cat needs to see the vet for a checkup."  (94% match)
  [MISS] (German) "Katze Napf leer"  ->  "The cat needs to see the vet for a checkup."  (90% match)
  [HIT ] (French) "chat assis tapis"  ->  "The cat sat on the mat."  (89% match)
  [HIT ] (French) "chat renversé vase"  ->  "The cat knocked over a vase."  (94% match)
  [HIT ] (French) "chat prix concours"  ->  "The cat won an award at the pet show."  (92% match)
  [HIT ] (French) "chat adopté refuge"  ->  "The cat was adopted from a shelter last year."  (94% match)
  [HIT ] (Spanish) "gato sentado ventana"  ->  "The cat sat on the windowsill."  (92% match)
  [HIT ] (Spanish) "gato arañó sofá"  ->  "The cat scratched the sofa."  (94% match)
  [HIT ] (Spanish) "gato veterinario revisión"  ->  "The cat needs to see the vet for a checkup."  (93% match)
  [HIT ] (Spanish) "plato comida vacío"  ->  "The cat's food bowl was empty."  (90% match)
  [HIT ] (English) "cat knocked lamp"  ->  "The cat knocked over a lamp."  (98% match)
  [HIT ] (English) "cat sleeping sun"  ->  "The cat is sleeping in the sun."  (97% match)
  [HIT ] (English) "cat award pet show"  ->  "The cat won an award at the pet show."  (97% match)
  [HIT ] (English) "cat adopted shelter"  ->  "The cat was adopted from a shelter last year."  (97% match)

  German accuracy: 2/4 = 50%
  French accuracy: 4/4 = 100%
  Spanish accuracy: 4/4 = 100%
  English accuracy: 4/4 = 100%

The worst German showing of the five: 2 misses out of 4, including "Katze Fensterbank gesessen" (cat sat on windowsill) landing on a completely unrelated action — "knocked over a vase" — at a confident 92%.

`intfloat/multilingual-e5-base`

Embedding dimension: 768

  [HIT ] (German) "Katze Fensterbank gesessen"  ->  "The cat sat on the windowsill."  (92% match)
  [HIT ] (German) "Katze Lampe umgeworfen"  ->  "The cat knocked over a lamp."  (93% match)
  [HIT ] (German) "Katze Tierarzt Untersuchung"  ->  "The cat needs to see the vet for a checkup."  (93% match)
  [MISS] (German) "Katze Napf leer"  ->  "The cat needs to see the vet for a checkup."  (90% match)
  [HIT ] (French) "chat assis tapis"  ->  "The cat sat on the mat."  (89% match)
  [HIT ] (French) "chat renversé vase"  ->  "The cat knocked over a vase."  (93% match)
  [HIT ] (French) "chat prix concours"  ->  "The cat won an award at the pet show."  (91% match)
  [HIT ] (French) "chat adopté refuge"  ->  "The cat was adopted from a shelter last year."  (92% match)
  [HIT ] (Spanish) "gato sentado ventana"  ->  "The cat sat on the windowsill."  (93% match)
  [HIT ] (Spanish) "gato arañó sofá"  ->  "The cat scratched the sofa."  (92% match)
  [HIT ] (Spanish) "gato veterinario revisión"  ->  "The cat needs to see the vet for a checkup."  (92% match)
  [HIT ] (Spanish) "plato comida vacío"  ->  "The cat's food bowl was empty."  (91% match)
  [HIT ] (English) "cat knocked lamp"  ->  "The cat knocked over a lamp."  (97% match)
  [HIT ] (English) "cat sleeping sun"  ->  "The cat is sleeping in the sun."  (96% match)
  [HIT ] (English) "cat award pet show"  ->  "The cat won an award at the pet show."  (96% match)
  [HIT ] (English) "cat adopted shelter"  ->  "The cat was adopted from a shelter last year."  (95% match)

  German accuracy: 3/4 = 75%
  French accuracy: 4/4 = 100%
  Spanish accuracy: 4/4 = 100%
  English accuracy: 4/4 = 100%

One miss, the same "bowl empty" query that tripped up three of the five models — this time landing on "needs to see the vet" instead.

`distiluse-base-multilingual-cased-v2`

Embedding dimension: 512

  [HIT ] (German) "Katze Fensterbank gesessen"  ->  "The cat sat on the windowsill."  (88% match)
  [HIT ] (German) "Katze Lampe umgeworfen"  ->  "The cat knocked over a lamp."  (93% match)
  [HIT ] (German) "Katze Tierarzt Untersuchung"  ->  "The cat needs to see the vet for a checkup."  (86% match)
  [HIT ] (German) "Katze Napf leer"  ->  "The cat's food bowl was empty."  (86% match)
  [HIT ] (French) "chat assis tapis"  ->  "The cat sat on the mat."  (75% match)
  [HIT ] (French) "chat renversé vase"  ->  "The cat knocked over a vase."  (86% match)
  [HIT ] (French) "chat prix concours"  ->  "The cat won an award at the pet show."  (74% match)
  [HIT ] (French) "chat adopté refuge"  ->  "The cat was adopted from a shelter last year."  (71% match)
  [HIT ] (Spanish) "gato sentado ventana"  ->  "The cat sat on the windowsill."  (96% match)
  [HIT ] (Spanish) "gato arañó sofá"  ->  "The cat scratched the sofa."  (92% match)
  [HIT ] (Spanish) "gato veterinario revisión"  ->  "The cat needs to see the vet for a checkup."  (84% match)
  [HIT ] (Spanish) "plato comida vacío"  ->  "The cat's food bowl was empty."  (79% match)
  [HIT ] (English) "cat knocked lamp"  ->  "The cat knocked over a lamp."  (96% match)
  [HIT ] (English) "cat sleeping sun"  ->  "The cat is sleeping in the sun."  (96% match)
  [HIT ] (English) "cat award pet show"  ->  "The cat won an award at the pet show."  (92% match)
  [HIT ] (English) "cat adopted shelter"  ->  "The cat was adopted from a shelter last year."  (87% match)

  German accuracy: 4/4 = 100%
  French accuracy: 4/4 = 100%
  Spanish accuracy: 4/4 = 100%
  English accuracy: 4/4 = 100%

Zero misses — the only model that got every single query right, including the "bowl empty" query that tripped up three of the other four models. Its match scores run noticeably lower than the others (71–96% vs. the high-80s/90s elsewhere), but the ranking was always correct, which is what actually matters for retrieval.

Summary table

Model                                   Dim      German      French     Spanish     English
--------------------------------------------------------------------------------------
paraphrase-multilingual-MiniLM-L12-v2   384         75%        100%        100%        100%
paraphrase-multilingual-mpnet-base-v2   768         75%         75%        100%        100%
intfloat/multilingual-e5-small          384         50%        100%        100%        100%
intfloat/multilingual-e5-base           768         75%        100%        100%        100%
distiluse-base-multilingual-cased-v2    512        100%        100%        100%        100%

Note: this is a small demonstration — 16 queries across 5 models — meant to show the direction of the differences, not a production benchmark. With only 4 queries per language, each hit/miss swings a language's accuracy by 25 percentage points. If you're choosing an embedding model for a real multilingual RAG system, test it against your own corpus and real user queries first.

Code walkthrough

The full script above is runnable as-is. Here's what the key steps actually do:

Build the corpus and embed it once

corpus_embeddings = model.encode(CORPUS, convert_to_tensor=True)

This runs once per model. In a real system, this is the step that happens when you ingest documents into your vector store.

Embed each keyword query and compare it against every corpus sentence

query_embedding = model.encode(query["text"], convert_to_tensor=True)
scores = cos_sim(query_embedding, corpus_embeddings)[0]
top_match_index = int(scores.argmax())

cos_sim returns one similarity score per corpus sentence. argmax() picks the index of the highest one — the sentence the model thinks is the closest match.

Check whether the top match was actually correct

is_correct = (top_match_index == query["answer_index"])

Every query already knows which corpus sentence it should match, since the queries were written by hand. This is a simple hit/miss check, rolled up per language into an accuracy percentage.

Rescale the raw score into a readable "match %"

match_percent = (top_match_score + 1) / 2 * 100

Cosine similarity runs from -1 to 1; this rescales it to 0–100% for readability. It's a relative strength score, not a probability — worth keeping in mind given what follows.

Why the same query gets different scores — and different answers

Look at the query that tripped up the most models: "Katze Napf leer" (German, "cat bowl empty"), which should match "The cat's food bowl was empty."

Model	Top match	Score
`paraphrase-multilingual-MiniLM-L12-v2`	"...adopted from a shelter" (wrong)	83%
`paraphrase-multilingual-mpnet-base-v2`	"...sat on the mat" (wrong)	84%
`intfloat/multilingual-e5-small`	"...needs to see the vet" (wrong)	90%
`intfloat/multilingual-e5-base`	"...needs to see the vet" (wrong)	90%
`distiluse-base-multilingual-cased-v2`	"...food bowl was empty" (correct)	86%

Unlike a case where every model converges on the same wrong answer, here four different models missed in three different ways — no single "close neighbor" was consistently confused with the target. That's a different, and arguably more concerning, failure mode: it's not one predictable blind spot, it's inconsistent behavior on the exact same short query across models. The only model that got it right, distiluse-base-multilingual-cased-v2, also had the lowest raw scores across the board (dropping as low as 71% on a correct match) — proof that a lower confidence number doesn't mean a worse model. Score magnitude and correctness are simply not the same thing.

This is the practical trap: cosine similarity scores are not on a universal scale, even across models that all carry the "multilingual" label. A similarity threshold you tune for one model is not portable to another — you have to calibrate on your own data, model by model.

The core finding: which multilingual model actually held up

Ranked by overall accuracy across German, French, and Spanish:

distiluse-base-multilingual-cased-v2 — 100% across every language. The only model with zero misses anywhere, despite being the "compact, fast" option in the lineup and consistently producing the lowest raw similarity scores.
paraphrase-multilingual-MiniLM-L12-v2 and intfloat/multilingual-e5-base — 75% German, 100% everywhere else. Both missed once, on different queries.
paraphrase-multilingual-mpnet-base-v2 — 75% German, 75% French, 100% Spanish/English. The largest model tested, and the only one to miss the "sat on the mat vs. sat on the windowsill" distinction — exactly the kind of fine-grained pair this test was built to catch.
intfloat/multilingual-e5-small — 50% German, the worst score of the five, missing two of four German queries.

The standout lesson here is that the compact, "budget" model won, and the largest model in the set (mpnet-base-v2, 768 dimensions) came in last. Dimension count and model size told you almost nothing about which model would actually retrieve correctly. This directly contradicts an earlier, smaller-scale run of this same idea, where mpnet-base-v2 was the clean winner — a useful reminder on its own: with only 4 queries per language, a single hit or miss can flip which model looks best, and that instability is itself information, not noise. It means none of these models is decisively, reliably better at this specific hard case — the winner changes depending on exactly which near-duplicate pairs you happen to test.

The practical takeaway

"Multilingual" on a model card is a label, not a guarantee, and neither is "bigger" or "newer." In this run, the smallest, lowest-confidence model won outright, and the largest model made the most mistakes. That's not a universal claim that small models are better — it's a demonstration that you cannot predict ranking from parameter count or release date, and you have to actually test.

Two things follow from that:

Benchmark multilingual models against each other on your own languages and your own corpus, with enough queries that one lucky or unlucky match doesn't flip the result. A model that's "multilingual" on paper — even a newer or larger one — can still underperform a smaller model on your specific vocabulary.
Thresholds and top-k tuning are model-specific. A 71% match from one model can be correct, while a 92% match from another can be wrong. Don't set a single similarity cutoff and assume it transfers across models.

Try it yourself

The full script is in this post — swap in your own corpus, your own languages, your own models, and see what actually happens before you pick a multilingual embedding model on faith.

This post is part of the "From Hallucination to Precision" series on RAG and embeddings. Full code, no API keys, fully reproducible locally.

Building a RAG System From Scratch — Four Components, One Working Pipeline

Ritesh Totlani — Wed, 12 Aug 2026 11:40:08 +0000

Most RAG tutorials explain the concept. This one shows the code — a complete working pipeline using LangChain, ChromaDB, and a local LLM via LM Studio. No OpenAI API key. No cloud costs.

The business problem: A company has hundreds of pages of HR documentation. Employees ask questions. An AI answers accurately from the actual documents in seconds.

Stack: Python, LangChain, ChromaDB, nomic-embed-text, Qwen 9B via LM Studio

pip install langchain langchain-community langchain-chroma langchain-openai langchain-text-splitters langchain-core openai requests

Make sure LM Studio is running with both models loaded before running the code.

Component 1 — Document Processor

Splits raw text into chunks. Chunk size and overlap determine everything downstream — cut a paragraph wrong and retrieval suffers.

from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_core.documents import Document
from typing import List

class DocumentProcessor:
    def __init__(self, chunk_size: int = 500, chunk_overlap: int = 50):
        self.splitter = RecursiveCharacterTextSplitter(
            chunk_size=chunk_size,
            chunk_overlap=chunk_overlap,
            separators=["\n\n", "\n", ". ", " ", ""]
        )

    def process(self, text: str, source: str = "") -> List[Document]:
        chunks = self.splitter.split_text(text)
        return [
            Document(
                page_content=chunk,
                metadata={"source": source, "chunk_index": i}
            )
            for i, chunk in enumerate(chunks)
        ]

⚠️ Changing chunk size later means reembedding everything from scratch. Your chunks and embeddings are tightly coupled. Choose carefully the first time.

Component 2 — Embedding Service

Converts text into numerical vectors. LM Studio needs a direct requests call — the standard LangChain OpenAI wrapper sends the wrong input format and throws a 400 error.

from langchain_core.embeddings import Embeddings
import requests

LM_STUDIO_URL = "http://localhost:1234/v1"
EMBED_MODEL   = "nomic-embed-text"

class EmbeddingService(Embeddings):
    def __init__(self):
        self.url   = f"{LM_STUDIO_URL}/embeddings"
        self.model = EMBED_MODEL

    def _embed(self, text: str) -> List[float]:
        response = requests.post(
            self.url,
            json={"model": self.model, "input": text},
            headers={"Authorization": "Bearer lm-studio"}
        )
        response.raise_for_status()
        return response.json()["data"][0]["embedding"]

    def embed_documents(self, texts: List[str]) -> List[List[float]]:
        return [self._embed(text) for text in texts]

    def embed_query(self, text: str) -> List[float]:
        return self._embed(text)

⚠️ Same model must be used for both indexing and querying. They must share the same vector space. Switching models means rebuilding the entire index.

Component 3 — Vector Store

Stores embeddings and retrieves closest matches by meaning — not by keyword. ChromaDB persists to disk so your index survives restarts without reprocessing documents.

from langchain_chroma import Chroma
from typing import Tuple

CHROMA_DIR      = "./chroma_db"
TOP_K           = 4
SCORE_THRESHOLD = 0.3

class VectorStore:
    def __init__(self, embedding_service: EmbeddingService):
        self.store = Chroma(
            collection_name="rag_collection",
            embedding_function=embedding_service,
            persist_directory=CHROMA_DIR
        )

    def add_documents(self, documents: List[Document]) -> None:
        self.store.add_documents(documents)

    def similarity_search(self, query: str) -> List[Tuple[Document, float]]:
        results = self.store.similarity_search_with_relevance_scores(
            query=query,
            k=TOP_K,
        )
        return [(doc, score) for doc, score in results if score >= SCORE_THRESHOLD]

💡 Score threshold of 0.3 works well for small document sets. In production with larger knowledge bases tune this to 0.6–0.75 to filter out loosely relevant chunks.

Component 4 — RAG Pipeline

Orchestrates the full flow. Query in, embed it, retrieve closest chunks, build context, generate grounded answer, return sources.

The most important line in the entire pipeline is the system prompt. Without the strict instruction to answer only from context, the LLM falls back on its general training knowledge — defeating the entire purpose of RAG.

from langchain_openai import ChatOpenAI
from langchain_core.messages import HumanMessage, SystemMessage
from typing import Dict

CHAT_MODEL = "qwen/qwen3.5-9b"

class RAGPipeline:
    def __init__(self, vector_store: VectorStore):
        self.vector_store = vector_store
        self.llm = ChatOpenAI(
            model=CHAT_MODEL,
            openai_api_base=LM_STUDIO_URL,
            openai_api_key="lm-studio",
            max_tokens=1000,
            temperature=0.1
        )

    def _build_context(self, documents: List[Tuple[Document, float]]) -> str:
        parts = []
        for doc, score in documents:
            parts.append(
                f"Source: {doc.metadata.get('source', 'unknown')}\n"
                f"Relevance: {score:.2f}\n"
                f"Content: {doc.page_content}"
            )
        return "\n\n---\n\n".join(parts)

    def query(self, question: str) -> Dict:
        results = self.vector_store.similarity_search(question)

        if not results:
            return {
                "answer": "I could not find relevant information to answer this question.",
                "sources": [],
                "chunks_used": 0
            }

        context = self._build_context(results)

        messages = [
            SystemMessage(content=(
                "You are a precise assistant that answers questions "
                "based solely on the provided context. "
                "If the answer is not in the context, say so clearly. "
                "Do not use your general knowledge to supplement the context."
            )),
            HumanMessage(content=(
                f"Context:\n{context}\n\n"
                f"Question: {question}\n\n"
                f"Answer based only on the context above:"
            ))
        ]

        response = self.llm(messages)

        return {
            "answer": response.content,
            "sources": list(set(doc.metadata.get("source") for doc, _ in results)),
            "chunks_used": len(results)
        }

💡 temperature=0.1 keeps answers factual and consistent. Low temperature means the model stays close to what the context says rather than being creative with it.

Run It

if __name__ == "__main__":

    SAMPLE_DOCUMENT = """
    Employee Leave Policy

    Sick Leave:
    Employees are entitled to 10 days of paid sick leave per calendar year.
    Sick leave resets on January 1st each year.
    Unused sick leave cannot be carried over to the next year.
    To apply for sick leave, submit a request through the HR portal.
    A medical certificate is required for sick leave exceeding 3 consecutive days.

    Annual Leave:
    Employees receive 25 days of annual leave per year.
    Annual leave must be approved by the line manager at least 2 weeks in advance.
    Up to 5 unused annual leave days can be carried over to the following year.

    Remote Work Policy:
    Employees may work remotely up to 3 days per week.
    Remote work requires a stable internet connection and a dedicated workspace.
    Core hours of 10:00 to 16:00 must be maintained regardless of location.

    Expense Policy:
    Business travel expenses must be approved before travel.
    Receipts are required for all expenses above 25 euros.
    Expense reports must be submitted within 30 days of the expense.
    Maximum meal allowance is 50 euros per day during business travel.
    """

    # Initialise all four components
    processor  = DocumentProcessor()
    embeddings = EmbeddingService()
    store      = VectorStore(embedding_service=embeddings)
    pipeline   = RAGPipeline(vector_store=store)

    # Index documents — happens once
    documents = processor.process(SAMPLE_DOCUMENT, source="company_policy")
    store.add_documents(documents)

    # Query — happens live for every user question
    questions = [
        "How many sick days am I entitled to per year?",
        "Can I carry over unused annual leave?",
        "How many days can I work remotely?",
        "What is the maximum meal allowance during business travel?",
        "Do I need a medical certificate for sick leave?",
    ]

    for q in questions:
        result = pipeline.query(q)
        print(f"Q: {q}")
        print(f"A: {result['answer']}")
        print(f"   Sources: {result['sources']}")
        print(f"   Chunks used: {result['chunks_used']}")
        print("-" * 60)

Real Output

This is the actual output from running this pipeline locally:

Q: How many sick days am I entitled to per year?
A: Based on the provided context, employees are entitled to 10 days
   of paid sick leave per calendar year.
   Sources: ['company_policy'] · Chunks used: 4

Q: Can I carry over unused annual leave?
A: Yes, up to 5 unused annual leave days can be carried over to
   the following year.
   Sources: ['company_policy'] · Chunks used: 4

Q: How many days can I work remotely?
A: Based on the provided context, employees may work remotely up
   to 3 days per week.
   Sources: ['company_policy'] · Chunks used: 4

Q: What is the maximum meal allowance during business travel?
A: The maximum meal allowance during business travel is 50 euros
   per day.
   Sources: ['company_policy'] · Chunks used: 4

Q: Do I need a medical certificate for sick leave?
A: A medical certificate is required for sick leave exceeding
   3 consecutive days.
   Sources: ['company_policy'] · Chunks used: 4

Every answer accurate. Every answer sourced. No hallucination. No API key. No cloud cost.

Replace SAMPLE_DOCUMENT with your own content and you have a working RAG system on your documents in minutes.

Moving to Azure

Every component maps directly to an Azure service. The pipeline logic stays identical — only the initialisation changes.

Local	Azure Equivalent
nomic-embed-text via LM Studio	AzureOpenAIEmbeddings
Qwen 9B via LM Studio	AzureChatOpenAI
ChromaDB	Azure AI Search

What to Tune Next

This implementation is intentionally minimal. In production add:

Higher score threshold — 0.6 to 0.75 for larger document sets
Reranking — a second model pass to reorder chunks by true relevance
Query expansion — generate multiple question phrasings before searching
Conversation history — maintain context across multiple turns
Observability — log every retrieval and generation for debugging

These are the patterns covered in the next post — where RAG goes from working to production-ready.

DEV Community: Ritesh Totlani

You Cloned the Aggregate. Why Is Your Read-Only Operation Still Changing State?

The requirement: “Run the real logic, but don't change anything”

First attempt: just clone the Trip

Let's look at what actually happens

But isn't Trip.Clone() a deep clone?

The important distinction

Object graph vs. mutation boundary

The method signature gives us the first clue

The fix

Why I prefer this over writing another calculator

So what exactly should we clone?

A practical way to find the boundary

1. Look at the parameters

2. Look for mutations

3. Follow mutable references

4. Check services involved in the operation

5. Define the simulation boundary

There is an even deeper lesson here

What about external dependencies?

The rule I now use in code reviews

A small checklist

The takeaway

German Sentiment Analysis with BERT: nlptown vs oliverguhr on 20 Real Sentences

The models

The eval set

The code

The results

Verdict

Entities Are Not Rows: A Domain Model That Behaves

The word "entity" is doing two different jobs

The domain: driving-hours compliance

An aggregate you can't corrupt

The rules live in the model, not a service layer

The take

I Tested 4 RAG Chunking Strategies Everyone Recommends. 2 Were Quietly Broken.

The chunk that wasn't a chunk

The 5,000-token "chunk"

The right answer, ranked dead last

The context that leaked

Why this is the actual point

Not All Multilingual Embedding Models Are Equal — Testing 5 of Them

The five models

The setup

The code

The results

paraphrase-multilingual-MiniLM-L12-v2

paraphrase-multilingual-mpnet-base-v2

intfloat/multilingual-e5-small

intfloat/multilingual-e5-base

distiluse-base-multilingual-cased-v2

Summary table

Code walkthrough

Why the same query gets different scores — and different answers

The core finding: which multilingual model actually held up

The practical takeaway

Try it yourself

Building a RAG System From Scratch — Four Components, One Working Pipeline

Component 1 — Document Processor

Component 2 — Embedding Service

Component 3 — Vector Store

Component 4 — RAG Pipeline

Run It

Real Output

Moving to Azure

What to Tune Next

But isn't `Trip.Clone()` a deep clone?

`paraphrase-multilingual-MiniLM-L12-v2`

`paraphrase-multilingual-mpnet-base-v2`

`intfloat/multilingual-e5-small`

`intfloat/multilingual-e5-base`

`distiluse-base-multilingual-cased-v2`