DEV Community: Filipe Neves

The engineering behind a search bar. Is it worth reinventing the wheel?

Filipe Neves — Fri, 31 Jul 2026 19:30:29 +0000

This article it is about the follow repository. I hope you all have a moment to understanding more about it and feel the fun of knowledge.

Have you ever stopped to think about how the Google search bar works? It is something we use every day, millions or even billions of users per day, writing letter by letter and almost with no response time to see the possible searches, words being corrected, suggestions happening, mentions of nearby events, suggestions based on your history, it is a magic of technology if you stop to think and try to understand it. I asked myself this question and started to list features, functionalities, that I could notice. It is like everything is connected, but all of this in one search bar? How is this possible?

I started trying to understand how a robust search bar really works, not an autocomplete that we use in grids or even the smart filters of our screens, I am talking about a search bar that really goes after information, items, records in a database based on everything that is happening in the world, real time.

This study journey took me to some points: inverted index, BM25, Trie, BK-tree, language detection. All of this to understand in the end that there is also a super famous library that does exactly the ranking of records based on your search and is present in many of the search methods we have. By the way, the Atlas Search from MongoDB uses this library. The name of what I found is Apache Lucene. An Open Source library with years of existence and maintenance and adaptations, facilities we can not even imagine.

The thing is that until I got to know Atlas Search I was already developing my own search bar and I wanted to finish many of the technologies and methods I found to really understand what was happening, and after that I wanted to create a new project using Atlas Search and compare the development time, the results, lines of code, to really understand the advantage of using it. With both projects done, I want to tell more about everything I learned.

To have the closest experience to the day to day of Google, I defined a functional Crawler. This Crawler is basically a robot configured to wander the internet and access pages autonomously to capture the information and keep saving all of this in our database, MongoDB. Google has a gigantic fleet of robots like this and they can treat much more the information on the sites to respect some restrictions or secrecy. I chose a public site of our interest to be the starting point and let the Crawler work, the configured one was the Microsoft documentation: https://learn.microsoft.com/pt-br/dotnet/.

In my studies I saw I was circling around a keyword: Apache Lucene. I started trying to understand it, I saw it is an open source framework specialized in searches just like the ones I was building, it is something where everything already exists, ready and validated by the market. Looking at our MongoDB database I noticed there is the Atlas Search which basically runs on top of Lucene. That is when I saw the perfect study scenario, to satisfy a curiosity, what is the real difference of implementing it or not? What runs under the hood of this technology?

Let's divide the text just like the division of the project, a Shared folder where important things for any version happen, the FromScratch version which brought me the practical and implementation knowledge about each thing, and finally the AtlasSearch which answered me about the difference and if all this effort pays off.

src/
├── Shared/            crawler, WebPage entity, DTOs, Blazor UI
├── 01-FromScratch/
└── 02-AtlasSearch/

Crawler

The way the Crawler processes these pages is very simple. It reads every link that is in a queue, it captures the information of the pages by downloading them and saving the useful information, when a link is found inside the page, first it checks if we already processed it and then, if it was not processed yet, it saves this link to the queue where it will be processed too.

private async Task ProcessUrlAsync(string url, HttpClient client, CancellationToken ct)
{
    var html = await client.GetStringAsync(url, ct);
    var document = await _browsing.OpenAsync(req => req.Content(html), ct);

    var page = new WebPage
    {
        Url = url,
        Title = document.Title?.Trim() ?? string.Empty,
        Content = ExtractCleanText(document),
        CrawledAtUtc = DateTime.UtcNow
    };

    await _pages.AddAsync(page, ct);

    foreach (var indexer in _indexers)
        await indexer.IndexAsync(page, ct);

    foreach (var link in ExtractLinks(document))
        if (_visited.TryAdd(link, 0))
            _channel.Writer.TryWrite(link);
}

Inverted index

Now a question: how do you imagine saving these pages the Crawler finds in the database? I imagine the first idea that comes to mind is to save one document per page where we will have the link and its content in a single record. But this means that, for a scenario where we are searching for a word ".NET", we will be accessing all the records and checking one by one if they contain this word. With this scenario described it is clear that it is not interesting to search like this, right?

One solution would be to save it as Inverted Indexes. It works the following way. Each document will use the word as a key and it will point to all the pages where it appears. The records look like this:

{
  "_id": ".NET",
  "Postings": [
    {
      "DocId": "6650f2...",
      "TfTitle": 1,
      "TfContent": 14,
      "PositionsTitle":   [ { "Start": 0,  "End": 7 } ],
      "PositionsContent": [ { "Start": 231, "End": 238 } ]
    }
  ]
}

This way, even if we have a database with millions of pages the cost of finding the word will always be the same. The document now keeps a list of IDs of the pages and this is what will direct us to the correct pages. We will already have the "Term frequency" of the titles and of the contents of the pages, all of this is the result of a processing that does everything to normalize each word and detect this information so that at search time it can just access this information in an easy way.

Tokenization and accent normalization

For a search to happen, we need to normalize the words and define if we are going to consider accents or not. In my case I chose "começar" (portuguese word for "to start") to be considered "comecar", ignoring accents. Not all words were saved in the database because I used a strategy to ignore StopWords and focus only on words that bring meaning to the search.

private static string ExtractWordChars(Rune rune)
{
    var decomposed = rune.ToString().Normalize(NormalizationForm.FormD);

    var sb = new StringBuilder();
    foreach (var dRune in decomposed.EnumerateRunes())
    {
        var category = Rune.GetUnicodeCategory(dRune);
        if (category is UnicodeCategory.NonSpacingMark)
            continue;

        foreach (var ch in dRune.ToString())
            sb.Append(char.ToLowerInvariant(ch));
    }

    return sb.ToString();
}

BM25 Ranking

Knowing we have a database with all the words, we need to rank them all so our search can prioritize the correct or close terms in cases where the search was written in a wrong way.

We need to guarantee that a rare term is worth more than a common term;
We need to guarantee that the tenth occurrence of the term is worth less than the first one;
A long document can not win points just for being long;
If what was searched is in the title of some site, it must be worth more than normal;

public double Score(int tf, long df, long totalDocs, int docLength, double avgDocLength)
{
    var idf = Math.Log((totalDocs - df + 0.5) / (df + 0.5) + 1);
    var lengthNorm = 1 - _options.B + _options.B * docLength / avgDocLength;
    var saturation = tf * (_options.K1 + 1) / (tf + _options.K1 * lengthNorm);
    return idf * saturation;
}

All of this must be saved in the database and analysed page by page to rank.

Phrase boost

And what if you search an exact phrase from some article or site? It can get lost between other higher ranked items. Think about the Microsoft dotnet documentation. We search "programming language" and we can end up finding articles where the words "programming" and "language" are there many times in the article but in a separated way. We need to guarantee that sites that have this exact phrase are privileged in our search.

Did-you-mean

I really stopped for days to go after understanding what happens behind the "Did you mean" field and how it brings exactly what I want even when I write everything wrong. Basically the coolest part of the project for me is here, and that is why I will explain more about this section.

Let's start with the scenario: I search "polymorphsim" and the bar gives me back "Did you mean: polymorphism". For this to happen, the engine needs to answer three questions: how wrong is the word? Which known words are "close" to it? And between those, which one to suggest?

The first question is answered by the Damerau-Levenshtein distance. It measures how many operations are needed to transform one word into another, considering four types of errors: missing letters, extra letters, swapped letters, inverted letters.

Example of the Damerau Levenshtein formula in C#:

public static int DL_Distance(string a, string b)
{
    var da = new Dictionary<char, int>();

    foreach (char c in a)
        da[c] = 0;

    foreach (char c in b)
        if (!da.ContainsKey(c))
            da[c] = 0;

    int maxDist = a.Length + b.Length;

    var d = new int[a.Length + 2, b.Length + 2];

    d[0, 0] = maxDist;

    for (int i = 0; i <= a.Length; i++)
    {
        d[i + 1, 0] = maxDist;
        d[i + 1, 1] = i;
    }

    for (int j = 0; j <= b.Length; j++)
    {
        d[0, j + 1] = maxDist;
        d[1, j + 1] = j;
    }

    for (int i = 1; i <= a.Length; i++)
    {
        int db = 0;

        for (int j = 1; j <= b.Length; j++)
        {
            int k = da[b[j - 1]];
            int l = db;

            int cost = a[i - 1] == b[j - 1] ? 0 : 1;

            if (cost == 0)
                db = j;

            d[i + 1, j + 1] = Math.Min(
                Math.Min(
                    d[i, j] + cost,
                    d[i + 1, j] + 1),
                Math.Min(
                    d[i, j + 1] + 1,
                    d[k, l] + (i - k - 1) + cost + (j - l - 1)));
        }

        da[a[i - 1]] = i;
    }

    return d[a.Length + 1, b.Length + 1];
}

OSA (Optimal String Alignment) version of the formula in C#:

public static int OSA_Distance(string a, string b)
{
    var d = new int[a.Length + 1, b.Length + 1];

    for (int i = 0; i <= a.Length; i++)
        d[i, 0] = i;

    for (int j = 0; j <= b.Length; j++)
        d[0, j] = j;

    for (int i = 1; i <= a.Length; i++)
    {
        for (int j = 1; j <= b.Length; j++)
        {
            int cost = a[i - 1] == b[j - 1] ? 0 : 1;

            d[i, j] = Math.Min(
                Math.Min(
                    d[i - 1, j] + 1,
                    d[i, j - 1] + 1),
                d[i - 1, j - 1] + cost);

            if (i > 1 &&
                j > 1 &&
                a[i - 1] == b[j - 2] &&
                a[i - 2] == b[j - 1])
            {
                d[i, j] = Math.Min(
                    d[i, j],
                    d[i - 2, j - 2] + cost);
            }
        }
    }

    return d[a.Length, b.Length];
}

For our search, users usually miss the position of the letters. The transposition offered by this metric is the differential compared to the pure Levenshtein. Just imagine if while searching, the user inverts two neighbour letters. This is probably the most common mistake of who types fast. Without it, dotnte (dotnet) would cost 2 operations and could lose to a wrong candidate that costs 1.

Levenshtein version:
dotnte
↓
dotntt (replaces 'e' with 't')
↓
dotnet (replaces the last 't' with 'e')

Used version Damerau Levenshtein:
dotnte
↓
dotnet (transposition between 't' and 'e')

Between the two versions, the project uses the complete Damerau-Levenshtein. The reason appears right after: the structure that speeds up this search prunes candidates using the triangular inequality, and the complete version is a true metric that guarantees this property, which allows us to prune with confidence.

The next point to think is that my vocabulary is all the words the crawler found in the pages. Calculating the distance of the query, on every search, against all the words is scanning everything to find very little. We need a structure that discards most of the vocabulary without even looking at it. This structure is the BK-Tree.

In this BK-Tree structure, each node keeps a word and the children are grouped by the exact distance to the parent. If the distance from the query to the current node is 5, and I only accept suggestions with up to 2 edits, the triangular inequality guarantees that any useful word is in children with distance between 3 and 7 from the current node. All the other branches are discarded whole, without calculating anything.

private void Visit(Node node, string query, int maxDistance, List<(string, int)> matches)
{
    var d = _distance(query, node.Word);
    if (d <= maxDistance) matches.Add((node.Word, d));

    foreach (var (k, child) in node.Children)
        if (k >= d - maxDistance && k <= d + maxDistance)
            Visit(child, query, maxDistance, matches);
}

Then the third question remains, the one almost nobody comments: the search returns many candidates, and I can only suggest one. Here came a heuristic that defines, the fewer edits the better the score, and the tie is decided by the size of the common prefix with what was typed. Basically because who makes mistakes typing almost always gets the beginning of the word right, the first letters.

foreach (var (word, distance) in _vocabulary.Current.BkTree.Search(query, _maxDistance))
{
    if (distance == 0) continue;

    var prefix = CommonPrefixLength(word, query);
    var score = distance * 10 - prefix * 2;

    if (score < bestScore)
    {
        best = word;
        bestScore = score;
    }
}

Remember that the vocabulary of the BK-Tree is a dictionary that stores the words extracted from the indexed pages themselves. This allows the corrector to suggest known terms, basically it will only know the world analysed by the crawler.

Autocomplete

Besides the "Did-you-mean" we also have the word suggestions and autocompletes that can happen through the Trie. The Trie is also a data structure for words, but its construction is built for agility with exact words. If I start searching "Lang" it can start suggesting me "Language". The usage flow of the Trie is constantly updated by a frequency check, the more a specific word is searched, the higher it appears in the suggestions ranking. This is what makes a show of the day or some news always be at the top of the rankings of your searches.

public IReadOnlyList<string> Autocomplete(string prefix)
{
    var node = _root;
    foreach (var ch in prefix.Trim().ToLowerInvariant())
    {
        if (!node.Children.TryGetValue(ch, out var next))
            return Array.Empty<string>();
        node = next;
    }
    return node.TopSuggestions.ToArray();
}

Proximity

If you go back and observe the structure of the saved document, of the record of each word, you will see there are positions and this is used here, in a ranking method based on the position and proximity of each searched term. It was used here a two pointers algorithm over the position lists where the smallest window between two words is worth more points than distant words.

while (true)
{
    var minList = 0;
    var minVal = sortedLists[0][pointers[0]];
    for (var i = 1; i < k; i++)
        if (sortedLists[i][pointers[i]] < minVal)
            (minVal, minList) = (sortedLists[i][pointers[i]], i);

    var window = currentMax - minVal;
    if (window < best)
        (best, bestFrom, bestTo) = (window, minVal, currentMax);

    pointers[minList]++;
    if (pointers[minList] == sortedLists[minList].Length) break;

    if (sortedLists[minList][pointers[minList]] > currentMax)
        currentMax = sortedLists[minList][pointers[minList]];
}

Terms

As we were searching about dotnet information, it was needed specific treatments for each unique term, a manual work on the treatment, because this varies according to the context of who is implementing.

Now that we understand the basic technologies through a search bar, let's understand how was the implementation of all of this with Atlas Search.

The first surprise was to understand that it has no indexers. There was no need to use the Core project, many things we built in the FromScratch version do not exist as code here, most of it was implemented just by defining configurations.

This happens because Atlas Search runs a process called mongot next to the database, inside the cluster itself. This process is the Apache Lucene watching the "pages" collection through a MongoDB mechanism to notify changes in real time called change streams and this keeps the Lucene index synchronized in a continuous way, a write can take some instants until it appears in the search. The crawler follows with the same goal but with a small economy. It was configured to follow the strategy of downloading, extracting text, saving the page, tokenizing all the words, writing the whole structure and always updating the statistics, right? For the AtlasSearch project we will have a cut of almost half of this process stopping where we save the page as a whole.

All the definition of tokens, indexing of the fields, performance, is not in our hands but in a declarative JSON document that defines which fields to index and how we are going to analyse this information.

{
  "mappings": {
    "fields": {
      "title": [
        { "type": "string", "analyzer": "folding" },
        { "type": "autocomplete" }
      ],
      "content": { "type": "string", "analyzer": "folding" },
      "language": { "type": "token" }
    }
  },
  "analyzers": [
    {
      "name": "folding",
      "tokenizer": { "type": "standard" },
      "tokenFilters": [
        { "type": "lowercase" },
        { "type": "asciiFolding" }
      ]
    }
  ]
}

Analyzer

All the process and structure to define the searches does not exist as code but as the Analyzer. The pipeline breaks the text into tokens and normalizes them, running both at indexing and at query time. We created a whole normalization process so words like "começar" would be understood as "comecar" but here just an asciiFolding filter already does the same job.

It is important to understand each normalization case to define the correct configurations. The Analyzer exists with a default but allows you to define customizations to attend many cases through the configurations.

The search

All our search process that used BM25, title boost, phrase boost, language boost, becomes a single aggregation stage with the $search. The BM25 we implemented formula by formula is the native ranking of Lucene: it is already there, calibrated by decades of use, and we just say what is worth more.

var clauses = new List<SearchDefinition<WebPageDataModel>>
{
    search.Text(x => x.Title, query, score: score.Boost(_options.WTitle)),
    search.Text(x => x.Content, query),

    search.Text(x => x.Title, query, fuzzy: fuzzy,
        score: score.Boost(_options.WTitle * _options.FuzzyFallbackBoost)),
    search.Text(x => x.Content, query, fuzzy: fuzzy,
        score: score.Boost(_options.FuzzyFallbackBoost)),

    search.Phrase(x => x.Title, query, new SearchPhraseOptions<WebPageDataModel>
    {
        Slop = _options.PhraseSlop,
        Score = score.Boost(_options.WTitle * _options.PhraseBoost)
    })
};

var definition = search.Compound()
    .Must(search.Compound().Should(clauses))
    .Should(search.Equals(x => x.Language, queryLanguage, score.Boost(_options.LanguageBoost)));

Notice that each section of the FromScratch has a correspondent here:

Field boost became score.Boost in the title clause.
Phrase boost became the phrase operator with slop. Basically it is the allowed position gap between the words, which covers the "programming language" case without needing a collection of n-grams.
Typo tolerance became the fuzzy option in the text clause itself: Lucene expands each word of the query to the close variations by edit distance, directly in the index. All that BK-Tree and Damerau-Levenshtein work is solved here too, just through the Lucene way, with automata over the term dictionary of the index.
Language boost reused our own n-gram detector from the Shared folder serving as one more proof that customizations are accepted here in an easy way.

Two details of this code deserve attention. Notice that each field is searched twice, one exact clause with the full boost and one fuzzy clause with a fraction of the boost, the FuzzyFallbackBoost of 0.3. This makes the fuzzy work as a plan B: if you typed everything right, the exact clauses dominate the ranking and the precision stays intact; if you typed wrong, only the fuzzy clause matches and it still finds the document. Notice also the nested Compound: the text clauses stay inside a Must, so a result only exists if there was a text match, and the language equals stays in an external Should, just adding points on top of who already matched. The language reinforces the order of the results, it never creates a result by itself.

Did-you-mean

In the FromScratch, the "Did-you-mean" follows a whole flow before searching. First we check the written words in our vocabulary and understand if it is known or not. We correct this word through the BK-Tree and only then we do some search. In Atlas we do not have the vocabulary exposed, it makes the search with the wrong word work the same way.

What raises the discussion in this step is that our API defines a contract of informing the user what was the correction. We went for the solution of defining the corrected word after the search by observing the results. Atlas has highlight fields that return the exact excerpts where there was a match and through them we define the words used as the marked search.

foreach (var (candidateFold, candidate) in hitWordsByFold)
{
    var distance = DamerauLevenshtein.Distance(folded, candidateFold);
    if (distance < bestDistance)
    {
        bestDistance = distance;
        best = candidate;
    }
}

if (best is not null && bestDistance <= 2 && bestDistance < folded.Length)
    words[i] = best;

So, reviewing: the FromScratch looks at the vocabulary before searching and corrects the word based on what it knows, mapped words. In Atlas the search happens with the incorrect word anyway and only through the highlights we define what was the exact word for the search.

In the end we had one more customization happening in Atlas, the DamerauLevenshtein calculation in the Shared folder.

Autocomplete

In the FromScratch, the autocomplete is a Trie in memory, rebuilt periodically. In Atlas, it is a field type in the index. The autocomplete mapping over the title makes Lucene index edge-grams. Basically the prefixes of each word of the title, like "documentation" becomes "do", "doc", "docu", "docum", (...). By default the smallest indexed prefix has 2 letters (minGrams) and the biggest has 15 (maxGrams). Suggesting something becomes a common search:

var definition = Builders<WebPageDataModel>.Search
    .Autocomplete(x => x.Title, prefix, fuzzy: fuzzy);

The operator accepts fuzzy too: type "document" and it finds the titles with "Documentação" (documentation). To keep the same experience of the FromScratch, which suggests words and titles, the endpoint still synthesizes word suggestions from the matched titles, ranked by distance and frequency.

Numbers

With both projects following the same structure, responding the same way, it is possible to measure the results of both projects and analyse response time, when one or the other project can be more advantageous. There were 16 fixed queries, chosen by phenomenon (portuguese phrase, accentuation, technical terms, typos, english), executed 20 times on each engine, with the calls alternated between them to distribute any oscillation of the cluster between the two. The measured time is the end to end HTTP request (p50 and p95), with the local APIs talking to the same free Atlas M0 cluster through the internet.

Phenomenon	Query	From-Scratch p50/p95 (ms)	Atlas Search p50/p95 (ms)
pt phrase	`linguagem de programação`	270 / 276	113 / 150
pt phrase no accent	`linguagem de programacao`	270 / 288	104 / 156
pt phrase	`sistema operacional`	300 / 307	133 / 197
pt phrase	`biblioteca de classes`	255 / 257	91 / 100
pt accented	`máquina virtual`	138 / 146	77 / 100
pt	`tutorial introdutório`	277 / 282	106 / 119
pt question	`como instalar o dotnet`	425 / 431	96 / 146
en	`what is dotnet`	353 / 359	88 / 99
en	`cloud infrastructure`	135 / 136	65 / 78
en	`generative ai`	478 / 511	144 / 149
technical term	`C#`	344 / 349	134 / 151
technical term	`.NET`	389 / 408	84 / 97
technical term	`asp.net core`	356 / 366	110 / 122
single term	`dotnet`	260 / 266	87 / 103
pt typo	`lingagem`	262 / 265	82 / 92
accent typo	`programaçao`	159 / 174	80 / 87
aggregate	16 queries × 20 iterations	276 / 474	97 / 148

Atlas was almost 3× faster on the p50 (97ms against 276ms). And this happens because of the data access design. The FromScratch makes many trips to the database per search, trips to get the postings of the terms, then the lengths of the candidates, then the documents of the top. While the $search solves everything in a single aggregation, with the index living next to the data. Both projects need to go to the database, the difference is that one of them needs to go only once.

And about the quality of the results? I also compared the pages returned by the two engines: on average, almost 80% of the top 10 matches between them. On the more direct searches and on the typing errors both return practically the same pages, each one getting there by its own way, and the difference only appears on the more ambiguous searches, where a single answer does not exist. In other words, equivalent results with a fraction of the code.

Is it worth reinventing the wheel?

After everything, is it worth reinventing the wheel? Even with Atlas having better performance results, I could say that understanding what is happening and how the searches happen is the biggest performance gain here, it is what allowed me to know this is possible and to be able to create my own customizations. Remember that attending the client is our focus as system developers and each client will need a unique customization. Knowing how to prepare well each environment is the biggest gain we can have.

Start studying, build to learn, analysing the FromScratch project, understand the concepts and then get ready for the build to earn using the rhythm of the AtlasSearch.

Clean Architecture in MongoDB + C#: Why is the Repository Pattern Alone Not Enough?

Filipe Neves — Tue, 26 May 2026 18:48:43 +0000

This article is based on a public repository I created. You can check the full project and follow here: MongoIntegrationAPI

Ever since I started using C# and writing my first lines of code, I was trained to apply the Clean Architecture pattern. Following DDD principles, I focused on the Domain layer, keeping it separated from Infrastructure and completely unaware that entities could represent a database table.

When I got my first MongoDB project, I had several problems. The MongoDB driver expects some specific methods to search and insert data, instances are built differently, and all of this made my first experience a traumatic one.

It turns out that stepping outside my comfort zone gives me motivation and energy to study and understand things. I started building Proofs of Concept (POCs), trying the connection between C# .NET and MongoDB in different scenarios. Working on these projects, I started to feel something was wrong again, I missed the Generic Repository Pattern and all the ease of dependency injection.

I also noticed another point about MongoDB: because it is a document-oriented database, it structures its “tables” differently. We can make a simple analogy where Tables are Collections and Rows are Documents. A document looks a lot like JSON, but it’s not. Actually, it’s BSON, a binary version.

The difference is that BSON is designed for storage efficiency. It stores metadata such as length and type, because of that it can be parsed and analyzed much more quickly by machines than JSON text. BSON supports basic types such as strings, bools, and numbers, and also some extra types such as dates, binaries, and exclusive identifiers like ObjectIds.

Book:

{
  "_id": { "$oid": "69bc53b77b02eb421c958688" },
  "title": "Dune"
}

In software, we deal with data that is frequently accessed together. In situations like this, we can analyze the case and check if it’s worth using an Embedded Document to improve performance and simplify reading all this information. Embedding is basically placing one document, or an array of documents, inside a field of another parent document.

There are a lot of embedded data patterns such as the Subset Pattern, Extended Reference Pattern, and Computed Pattern. All of them fit different scenarios.

The same book, “Dune”, but now as an Embedded Document:

{
  "_id": { "$oid": "69bc53b77b02eb421c958688" },
  "title": "Dune",
  "publisherId": { "$oid": "69bc53957b02eb421c958687" },
  "authors": [
    {
      "name": "Frank Herbert",
      "bibliography": "[...]"
    }
  ],
  "categories": [
    {
      "name": "SciFi",
      "description": "Science fiction exploring the future and technology."
    },
    {
      "name": "Adventure",
      "description": "Stories focused on journeys, challenges, and discoveries."
    }
  ]
}

In C#, our classes require specific annotations which, although functional, introduce coupling with the Infrastructure layer. This reduces portability and violates the separation of concerns proposed by Clean Architecture.

Let’s talk more about how we learn to build systems by checking the TreeView of the project.

3 folders, no separation between domain and storage

C:\...\MongoIntegrationAPI\
├───Controllers\            <-- Depend directly on the concrete repository classes
├───Models\                 <-- Entities decorated with [BsonId], [BsonRepresentation], [BsonElement]
├───Repositories\           <-- Use IMongoCollection<> directly — the MongoDB driver leaks in here
├───MongoContext.cs         <-- Exposes IMongoCollection<T> to the whole application
└───MongoSettings.cs        <-- Connection settings

Taking a closer look, focusing on a small entity shown in the BSON, the Book. This Book class is a main entity which has its own attributes. For the examples, we'll use a simplified version.

Let’s see how our Book is translated into C#:

using MongoDB.Bson;
using MongoDB.Bson.Serialization.Attributes;

public class Book
{
    [BsonId]
    [BsonRepresentation(BsonType.ObjectId)]
    public string Id { get; set; } = string.Empty;

    [BsonElement("title")]
    public string Title { get; set; } = string.Empty;

    [BsonElement("publisherId")] 
    [BsonRepresentation(BsonType.ObjectId)]
    public string PublisherId { get; set; } = string.Empty;

    [BsonElement("authors")]
    public List<AuthorInBook> Authors { get; set; } = new();

    [BsonElement("categories")]
    public List<Category> Categories { get; set; } = new();
}

public class AuthorInBook
{
    [BsonElement("name")]
    public string Name { get; set; } = string.Empty;

    [BsonElement("bibliography")]
    public string Bibliography { get; set; } = string.Empty;
}

public class Category
{
    [BsonElement("name")]
    public string Name { get; set; } = string.Empty;

    [BsonElement("description")]
    public string Description { get; set; } = string.Empty;
}

Adding it to the context:

using MongoDB.Driver;

public class MongoContext
{
    private readonly IMongoDatabase _database;

    public MongoContext(MongoSettings settings)
    {
        var client = new MongoClient(settings.ConnectionString);
        _database = client.GetDatabase(settings.DatabaseName);
    }

    public IMongoCollection<Book> Books =>
        _database.GetCollection<Book>("books");
}

Being used in the Repository:

using MongoDB.Driver;

public class BookRepository
{
    private readonly IMongoCollection<Book> _collection;

    public BookRepository(MongoContext context)
    {
        _collection = context.Books; 
    }

    public async Task<List<Book>> GetAllAsync()
    {
        return await _collection.Find(_ => true).ToListAsync();
    }

    public async Task<Book?> GetByIdAsync(string id)
    {
        return await _collection.Find(x => x.Id == id).FirstOrDefaultAsync();
    }

    public async Task CreateAsync(Book book)
    {
        await _collection.InsertOneAsync(book);
    }
}

Analyzing the code, it’s possible to identify what is described as “infrastructure infection”. Some layers of this project are infected by the MongoDB driver. Entities and the repository import it, and the controller/service layer depends on these classes.

Although the implementation is simple and functional at first glance with our Book model, it introduces some unnecessary coupling that will make the system harder to evolve.

The entity now depends on MongoDB attributes such as ObjectId, which means that business rules become tied to persistence details, reducing Domain independence and making changes in drivers or storage impact parts of the system that, in theory, should be stable.

The repository, in turn, fails to fully fulfill its role as an abstraction. Instead of isolating data access, it only wraps direct calls to the MongoDB driver, making the code tightly coupled to the driver’s API.

private readonly IMongoCollection<Book> _collection;

The Repository is not just using the database anymore. The MongoDB driver is shaping everything and forcing the Book class to satisfy the driver.

It limits the flexibility of the entire application. This coupling directly impacts testability, since the driver's dependency on interfaces makes creating mocks more complex and, in many cases, leads to an overreliance on integration tests.

Can you imagine what could happen if the developer received a task to switch databases or export all entities to a library? It would not be a refactor, it would be a rewrite.

This type of approach, even being common in introductory examples, puts the database as one of the central elements of our software while the Domain starts to play a secondary role.

All of this goes against Clean Architecture principles, which propose exactly the opposite: a system driven by behavior and business rules, with infrastructure being only an external detail.

A Way Out: Translation Layer and Atomic Operations

To guarantee everyone can understand how the proposed isolation works and materializes in practice, it’s essential to understand the project organization.

The folder structure is not just a style choice, but a direct representation of our architectural decisions, establishing clear boundaries between business rules and persistence details:

C:\...\MongoIntegrationAPI\
├───Domain\
│   ├───Entities\           <-- Pure POCOs (Author.cs, Book.cs)
│   ├───Enums\
│   └───Interfaces\         <-- Contracts (IBookRepository.cs)
├───Infrastructure\
│   ├───DataModels\         <-- BSON-decorated models (BookDataModel.cs)
│   ├───Repositories\       <-- Implementations and mappings
│   ├───Daos\               <-- MongoDB-specific operations
│   └───MongoDbSetup.cs     <-- Configuration
└───Controllers\            <-- Application entry point

Inside Infrastructure, we’re going to have three types of classes:

Specific repositories such as BookRepository to receive Domain entities, transform them into DataModels, and move them to database actions.
A Generic Repository to provide easier access and reusable CRUD methods, receiving DataModels as parameters and never knowing about the Domain.
Specific DAOs such as BookDao, a precision tool for MongoDB atomic operations.

Inside the Domain directory lies the core of our system, composed exclusively of essential business elements. The entities are defined as simple objects without any dependency on external libraries, while interfaces guarantee clear contracts about what the system needs to do.

This layer represents the application’s behavior and must stay isolated from any technical detail.

In this model, there is no indication of how the data will be persisted in the database. The focus is exclusively on representing the business.

// Domain layer
public class Book
{
   public string Id { get; set; } = string.Empty;
   public string Title { get; set; } = string.Empty;
   public string PublisherId { get; set; } = string.Empty;
   public List<AuthorInBook> Authors { get; set; } = new();
   public List<Category> Categories { get; set; } = new();
}

public class AuthorInBook
{
    public string Id { get; set; } = string.Empty;
    public string Name { get; set; } = string.Empty;
}

You can notice that our Book entity carries embedded information from our Author, just like the BSON shown earlier, but now it’s not a database decision anymore, it comes from business rules.

The Book entity is loaded, validated, and persisted together with categories and authors, so we create it this way in the Domain.

The difference now is that business operations made us design the entity with these attributes, and MongoDB just respects this with embedding patterns.

If you would keep this shape even with another database, it’s a Domain choice. Otherwise, the database is still in charge.

On the other hand, the Infrastructure folder focuses on all persistence decisions. This is where DataModels live, responsible for handling specific MongoDB features.

The concrete repositories make the bridge between Domain and Database. This organization ensures the complexity stays contained, preventing it from “leaking” into the rest of the system:

using MongoDB.Bson;
using MongoDB.Bson.Serialization.Attributes;
using MongoIntegrationAPI.Infrastructure.Attributes;

namespace MongoIntegrationAPI.Infrastructure.DataModels
{
    // Infrastructure layer: storage decisions
    [CollectionName("books")]
    public class BookDataModel
    {
        [BsonId]
        [BsonRepresentation(BsonType.ObjectId)]
        public string? Id { get; set; }

        public string Title { get; set; } = string.Empty;

        [BsonRepresentation(BsonType.ObjectId)]
        public string PublisherId { get; set; } = string.Empty;

        // The database stores nested objects to optimize reads
        public List<AuthorEmbeddedModel> Authors { get; set; } = new();

        public List<CategoryEmbeddedModel> Categories { get; set; } = new();
    }
}

DataModel is where we isolate all persistence-related decisions, such as the use of ObjectId and embedding structures. This allows us to optimize persistence without affecting the Domain.

Reading our DataModel, it’s possible to notice [CollectionName("books")]. It’s not from MongoDB, it’s from our architecture to make it easier to use our Generic Repository pattern.

We’ll use MongoDB, but how can we use just one class to control the commands here?

Let’s start by looking into how these classes work together, beginning with the new attribute in the DataModel.

The interface:

namespace MongoIntegrationAPI.Infrastructure.Repositories
{
    public interface IGenericRepository<TDocument> where TDocument : class
    {
        Task AddAsync(TDocument document);
        Task<TDocument?> GetByIdAsync(string id);
        Task<IEnumerable<TDocument>> GetAllAsync();
        Task ReplaceAsync(string id, TDocument document);
        Task DeleteAsync(string id);
    }
}

The implementation:

using MongoDB.Bson;
using MongoDB.Driver;
using MongoIntegrationAPI.Infrastructure.Attributes;
using System.Reflection;

namespace MongoIntegrationAPI.Infrastructure.Repositories
{
    public class GenericRepository<TDocument> : IGenericRepository<TDocument> where TDocument : class
    {
        private readonly IMongoCollection<TDocument> _collection;

        public GenericRepository(IMongoDatabase database)
        {
            var collectionName = typeof(TDocument)
                .GetCustomAttribute<CollectionNameAttribute>()?.Name
                ?? typeof(TDocument).Name;

            _collection = database.GetCollection<TDocument>(collectionName);
        }

        public async Task AddAsync(TDocument document)
        {
            await _collection.InsertOneAsync(document);
        }

        public async Task<IEnumerable<TDocument>> GetAllAsync()
        {
            return await _collection.Find(Builders<TDocument>.Filter.Empty).ToListAsync();
        }

        public async Task<TDocument?> GetByIdAsync(string id)
        {
            var filter = Builders<TDocument>.Filter.Eq("_id", new ObjectId(id));
            return await _collection.Find(filter).FirstOrDefaultAsync();
        }

        public async Task ReplaceAsync(string id, TDocument document)
        {
            var filter = Builders<TDocument>.Filter.Eq("_id", new ObjectId(id));
            await _collection.ReplaceOneAsync(filter, document);
        }

        public async Task DeleteAsync(string id)
        {
            var filter = Builders<TDocument>.Filter.Eq("_id", new ObjectId(id));
            await _collection.DeleteOneAsync(filter);
        }
    }
}

The constructor of our Repository asks for an IMongoDatabase to get the Collection found by the collectionName, which is the result of our CollectionNameAttribute on the DataModel.

namespace MongoIntegrationAPI.Infrastructure.Attributes
{
    [AttributeUsage(AttributeTargets.Class, Inherited = false)]
    public class CollectionNameAttribute : Attribute
    {
        public string Name { get; }

        public CollectionNameAttribute(string name)
        {
            Name = name;
        }
    }
}

The main point of this architecture is the role of the specific repository as a translator whenever the Domain and DataModel shapes diverge, which is almost always the case in real systems.

It’s no longer a passive intermediary. It explicitly converts the data between the two models and calls the generic version to make updates:

// Infrastructure
public async Task AddAsync(Book book)
{
   var dataModel = new BookDataModel
   {
       Title = book.Title,
       PublisherId = book.PublisherId,
       Authors = book.Authors
           .Select(a => new AuthorEmbeddedModel
           {
               Id = a.Id,
               Name = a.Name
           })
           .ToList()
   };

   await _genericRepository.AddAsync(dataModel);

   // We adopt the native ObjectId pragmatically,
   // reflecting it back into the Domain after creation
   book.Id = dataModel.Id!;
}

This section works as a barrier against the corruption of responsibilities, acting as an anti-corruption layer.

Database changes stop here in these repositories, allowing us to maintain the stability of our business rules while the Generic Repository only receives DataModel objects.

These repositories work well for translating documents. Now let’s analyze performance scenarios in MongoDB.

Loading a massive embedded document into memory just to add a single field into an array is a waste of resources. This is the point where we need a new layer responsible for leveraging specific MongoDB features.

The new layer is going to be the Data Access Object (DAO).

Think of it like a hospital. The Repository is the reception desk. You go there to find a patient, schedule an appointment, or retrieve a record.

The DAO is the specialist who walks into the archives room, knows exactly which folder to pull, and updates a single page without taking the whole file off the shelf.

// Infrastructure layer: BookDao.cs (atomic performance)
public async Task<bool> AddAuthorToBookAtomicAsync(string bookId, AuthorEmbeddedModel author)
{
    var filter = Builders<BookDataModel>.Filter.Eq(b => b.Id, bookId);

    var update = Builders<BookDataModel>.Update.Push(b => b.Authors, author);

    var result = await _collection.UpdateOneAsync(filter, update);

    return result.MatchedCount > 0;
}

In MongoDB design, there exists a rule that we've talked already:

“Data that's accessed together should be stored together.”

Embeddings help us follow this rule and create denormalization that is very useful for reads and performance in such cases.

Let's think about some ones where this idea is useful. Just one Collection handling all the information about an user it's a good idea helping us to open Profile page and check all the data with just one query.

But now let's think not on queries, but about updates. It's not easy to orchestrate and commit big documents.

By coupling this logic into the DAO, we solve this problem.

Instead of loading the entire Book into the application, changing it with C#, and rewriting the whole document, we use the $push operator to alter only the necessary fragment directly in the database.

The Repository orchestrates the business flow, but only the DAO knows the syntax and atomic operations of the MongoDB driver.

The final result is an architecture where Embedding becomes a clear storage decision and the DAO becomes a precision tool to manipulate data.

In the project, it becomes evident that the BookRepository exposes a clear method to add an Author, while internally using the DAO to submit the surgical update of the BSON document.

With this, we can reach the best of both worlds: a behavior-focused Domain supported by an Infrastructure layer capable of extracting all the performance MongoDB can offer, without either side needing to know the other’s technical details.