DEV Community: Aasawari Sahasrabuddhe

Enhancing MongoDB's Performance With Horizontal Scaling

Aasawari Sahasrabuddhe — Mon, 30 Jun 2025 12:53:00 +0000

Imagine you work for a small business that just boomed with that one TikTok or Instagram reel. The application is flooded with requests, while the company was already planning on adding some new products to the application.

As a team maintaining databases for your organization, you will likely encounter an inevitable challenge when your single database instance that once handled everything smoothly comes crashing down because of increased demand. As your product catalog and customer base expand exponentially, your single MongoDB instance starts to struggle. Query response times increase, impacting user experience, and scaling vertically (adding more resources to a single server) proves to be unsustainable and costly. With the growing infrastructure demand, you as the tech decision maker need a more efficient solution to address the performance bottlenecks and deliver a better user experience.

This is where scaling your application through MongoDB’s sharding transforms the architecture, helps increase in throughput, provides better performance, and much more.

In this article, we will gain a deeper understanding of MongoDB sharding capabilities, explore the difference between horizontal and vertical scaling, and discuss how you can scale your application better.

The difference between horizontal and vertical scaling

Before we get into the details of the horizontal scaling approach, it is equally important to understand the difference between both types of scaling strategies.

Parameter	Horizontal scaling	Vertical scaling
Load distribution	This distributes the load across multiple servers.	This improves the performance by upgrading the hardware being utilised.
Example	Adding VMs in a cluster	Adding memory capacity of existing VM
Workload distribution	Distributed across different nodes	Single node handles the workload
Concurrency	Since the workload is distributed, it becomes easy to handle concurrent requests.	Multi-threading is responsible for concurrency.
Complexity	High	Low
Costs	Optimal over time	Less cost-effective
Failure resilience	Lower as other nodes have the backup	Single point of failure

From the above table, we see that in certain scenarios, vertical scaling proves inefficient because, among other things:

The CPU limit is reached after a point and no further vertical scaling is possible.
The single point of failure would result in a crash and application downtime.
The interrelation between performance and cost becomes exponentially worse.
Setting up global users from a single server location introduces latency issues.

Based on parameters like cost, future growth, reliability, flexibility, and the complexity of the application, you have enough information to choose between the horizontal or vertical approach.

Understanding sharding in databases

Sharding in a database is the concept of distributing the data across different machines. To avoid excessive load on a single machine and improve the retrieval process, the database administrator (DBA) makes the decision to divide the data into small, manageable pieces from the existing database. These pieces are known as shards. This helps in:

Managing only a subset of reads and writes on the complete application.
Allowing queries in different shards to operate parallelly.
Providing better scalability.

To address the system growth, MongoDB supports horizontal scaling using sharding.

In the next section, we will get into a deeper understanding of MongoDB’s sharded clusters.

A MongoDB sharded cluster

In MongoDB, when the data is distributed into different shards, there are a few terms which are important:

Shard: The small pieces of the larger dataset are known as chunks. These chunks are distributed across various shards and each shard is typically a replica sets in themselves.
Mongos: When a request is received, a mongos is responsible to route the query to the responsible shard.
Config server: These maintain the metadata and configuration settings for the cluster.
Shard keys: This is the key parameter to divide the dataset into smaller shards. Choosing the right shard key is the most essential and important step in creating the shards. Therefore, this decision should be made wisely.

The below diagram is a representation of sharded cluster architecture used in production:

Creating a sharded cluster

To create a sharded cluster on a local deployment, follow the steps in the official MongoDB documentation. Maintaining this is not as daunting as it seems, but if you are looking for a more streamlined process, MongoDB Atlas offers a fully managed sharded cluster setup.

With MongoDB Atlas, you can deploy a sharded cluster in just a few clicks, and the platform handles all aspects of maintenance, including monitoring, backups, and automatic scaling. This allows you to focus on developing your application without worrying about the underlying infrastructure.

Monitoring and balancing a sharded cluster

Managing a sharded cluster is necessary to get optimal performance from the application. After a collection is sharded, MongoDB will try to distribute it across the shards created based on the selection of the shard key. Therefore, it becomes important to analyse the shards and understand how data is being distributed across. To help monitor the shards better, MongoDB provides basic commands like:

sh.status()

And:

db.printSghardingStatus()

These commands provide basic information about the shards and existing chunks in the cluster.

It is also important to note that selecting the right shard key plays an important role in creating hot shards and jumbo chunks. To help with managing these chunks, the MongoDB Sharded Cluster Balancer plays an important role. This monitors the amount of data in each shard for each collection.

In essence, effective shard key selection and regular monitoring with the balancer are vital to maintain a well-distributed and high-performing MongoDB sharded cluster.

To shard or not to shard?

When to shard

MongoDB supports horizontal scaling using sharding. The question here is: What are the ideal scenarios when sharding is a better approach?

Increasing data size: When the data can no longer fit into single server storage space, sharding is what one should consider as it horizontally scales and distributes the data across different shards.
Increase in performance: When the application is required to have high read and write rates, a single replica set would not be sufficient to do this, hence spreading across reduces the contention and improves the performance of the database.
Scalability requirements: If the application is expected to grow in terms of users and traffic, and it is always good to shard to handle the load in a manageable way.

While knowing when to shard your database is important, it is equally important to understand when not to shard.

When not to shard

As a software engineer, knowing the best fit is the most crucial step before we reach a decision. To decide whether to shard or not to shard, it is important to understand:

If the database is relatively small and fits within the storage, there is probably no requirement to shard the collection.
Selection of the shard key is the most important step for sharding the collection. Messing up the selection of the shard key can result in jumbo chunks and might not help improve the application.
Selection of the wrong shard key could also result in operating on scatter gather queries and could therefore impact the performance of the application.
Managing the backups and restores in a sharded collection is more complex compared to a non-sharded collection. Therefore, maintaining the backups could be difficult to manage.
Finally, sharding in a multi-cloud environment can result in latency problems between the shards.

Sharding best practices

If you have a sharded cluster locally or on-prem, there are a few considerations:

Selecting the right sharding key helps in even distribution of data and read and write operations.
Avoid scatter gather queries.
Use hash based sharding to help in uniform distribution of reads and writes.
Add a new shard before the current shards gets overloaded, as a precautionary measure.

Conclusion

As modern day applications scale rapidly, ensuring the database can keep up is an important step. Horizontal sharding in MongoDB offers a powerful way to scale, distribute storage, and ease the compute across multiple machines. This article aims to give you a clear and concise understanding of sharding in MongoDB, what it is, how it works, and why it matters. As your application scales and your data grows, knowing when to move from a single-node setup to a sharded cluster can make all the difference in performance and reliability.

To know more about the technical concepts, visit MongoDB’s official documentation.

Building a Chatbot With Symfony and MongoDB

Aasawari Sahasrabuddhe — Wed, 28 May 2025 12:30:00 +0000

We are living in the age of AI. Almost every modern application or website offers some level of AI integration. Whether it is Google Docs, WhatsApp chat, or Zoom, you will come across an AI integration, especially chatbots, which have become essential tools for enhancing user engagement and support. So why not build one tailored for your own application?

A well-designed chatbot doesn't just answer questions—it creates interactions that enhance user experience and build loyalty. The convergence of advanced AI technologies with robust web frameworks has opened new possibilities for businesses to deploy intelligent conversational agents that truly understand user intent.

In this tutorial, we will walk through the steps to build a chatbot application for the Symfony Documentation along with Doctrine ORM and MongoDB Doctrine ODM documentation pages. This application would help you answer Symfony-related questions related to ORM or ODM with structured, accurate, and context-aware responses. A chatbot, in general, uses the powerful AI techniques called retrieval augmented generation, or RAG.

What is retrieval augmented generation (RAG)?

RAG is a technique that enhances traditional language models by combining them with a retrieval system. This is how it works:

Retrieval: When a user prompts a question, the system retrieves information from the knowledge base or database.
Augmentation: The retrieval document is then passed to the large language models, or LLMs, to generate the response.
Generation: The final response is crafted by the LLMs, but it’s based on the real-world content that was fetched during the retrieval step.

The below diagram provides you with more context on how the RAG applications are created.

In this comprehensive guide, we will walk through all the steps required for building a chatbot application using the Symfony framework. We are using the source files of the documentation of Symfony and Doctrine from their GitHub repositories to clone all the files on a local system and then import them into MongoDB in the form of chunks.

The chunks are basically partitions of a complete document to be stored in the database. We will be using LLPhant, a comprehensive PHP generative AI framework. Further, we are making use of Voyage AI to generate the embeddings and later using the OpenAI LLM model for formatting our responses.

Let’s understand each of these steps in detail.

Project overview

This chatbot application uses the RAG architecture to provide accurate responses based on Symfony documentation. Key components include:

Symfony back end: Manages API communication, handles user queries, and integrates with MongoDB, Voyage AI, and OpenAI
MongoDB: Stores chunked Symfony documentation and corresponding vector embeddings; uses Atlas's vector search for retrieving relevant content
Voyage AI: Converts documentation chunks into semantic vector embeddings for efficient similarity search
OpenAI: Generates context-aware responses using retrieved documentation snippets and user queries
Twig front end: Provides a simple web interface for user interaction and response display

Setting up the development environment

To start setting up the environment for the project, we need to have a few things ready:

Create an Atlas cluster: Create your first free cluster. Follow the MongoDB documentation page, Deploy a Free Cluster, or make use of the MongoDB Model Context Protocol (MCP) Server to create your free cluster. You can follow the documentation on the MCP market for detailed steps.
PHP 8 and above: Download the latest PHP version from the official site's downloads page.
Symfony version 7 and above: Get the steps to install Symfony from the documentation.
Creating a Voyage AI API key: Get your Voyage AI API key from the Voyage AI documentation page. Also, make sure to set the pricing so that embeddings are being created.
OpenAI API key: Create your OpenAI key from the API key page.

Creating the Symfony chatbot application

Creating the Symfony project

To create the Symfony project, you need to follow the steps below:

composer create-project symfony/skeleton SymfonyDocsChatBot

Once the project is created, a few dependencies need to be downloaded. These steps will be addressed sequentially as required.

Setting up project dependencies

To build the chatbot application, there are a few dependencies we need to install.

We will be using LLPhant to perform the chunking of the documents. To install in your Symfony application:

composer require theodo-group/llphant

In order to store the chunks into MongoDB, we need to have Doctrine MongoDB ODM installed with the Symfony application. To do so:

Install the MongoDB extension using PECL:

pecl install mongodb

Once installed, install the Doctrine Bundle. The bundle integrates the Doctrine MongoDB ODM into Symfony, helping you to configure and use it in your application. To do so:

composer require doctrine/mongodb-odm-bundle

To create the front end, we will be making use of Twig. It is a fast, secure, and flexible templating engine for PHP used to build clean and structured HTML views in Symfony applications.

composer require symfony/twig-bundle

In this application, we are sending HTTP requests to Voyage AI and OpenAI to perform embedding and formatting of the response. To make use of HTTP requests, install the guzzlehttp package repository.

composer require symfony/http-client

Finally, to have a better look and feel of the response being generated, we will make use of CommonMark, which converts the Markdown into HTML format. To do so:

composer require "league/commonmark”

Finally, at the end, the composer.json will look like:

"require": {
        "php": ">=8.2",
        "ext-ctype": "*",
        "ext-iconv": "*",
        "doctrine/mongodb-odm-bundle": "^5.3",
        "league/commonmark": "^2.7",
        "symfony/console": "7.2.*",
        "symfony/dotenv": "7.2.*",
        "symfony/flex": "^2",
        "symfony/framework-bundle": "7.2.*",
        "symfony/http-client": "7.2.*",
        "symfony/runtime": "7.2.*",
        "symfony/twig-bundle": "7.2.*",
        "symfony/yaml": "7.2.*",
        "theodo-group/llphant": "^0.10.0",
        "twig/extra-bundle": "^3.21",
        "twig/markdown-extra": "^3.21",
        "twig/twig": "^2.12|^3.0"
    }

At this point, we are all set to start building the application.

Building the chatbot application

In this section, we will walk through building a chatbot using Symfony, MongoDB, and LLPhant. We’ll go through each step—from ingesting raw docs to returning intelligent answers.

Step 1: Copy the RST files into Symfony project’s public directory

To create the chatbot application, the first step is to get the raw files on which the processing has to happen. To do that, clone the Symfony Docs GitHub repository. Perform the following steps:

Once this is cloned, it will create a folder for all the RST files for the documentation pages. The next step here is to store the raw files inside the Symfony project.

The Symfony documentation repository has now been cloned to have better and more polished Symfony answers to the questions asked to the chatbot.

Perform the same steps for Doctrine ODM and Doctrine ORM documentations.

The documentation's raw files are available in the docs folder.

Once done, move the RST files into the public directory of the Symfony project. After this, you should be able to see your project structure like this:

Step 2: Create chunks and store them in MongoDB

Generally, most RAG applications are centered around using LLMs to answer questions about large repositories, which, in our case, are the Symfony documentation pages. Processing larger files for the LLMs becomes expensive, and chunking addresses this by breaking large texts into smaller segments. Also, in RAG, embedding these chunks allows the system to retrieve only the most relevant parts for a query, reducing token usage and improving answer accuracy.

To perform the chunking of the RST files, we are creating a Symfony Command class file that uses the LLPhant’s Document Splitter function. This function splits the documents based on the number of characters and returns one chunk at a time for the complete document.

In the below CreateChunks.php class file, this function will create the chunks and store them in MongoDB.

<?php

namespace App\Command;

use App\Document\ChunkedDocuments;
use LLPhant\Embeddings\DataReader\FileDataReader;
use Symfony\Component\Console\Attribute\AsCommand;
use Symfony\Component\Console\Command\Command;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Output\OutputInterface;
use Symfony\Component\Console\Style\SymfonyStyle;
use LLPhant\Embeddings\DocumentSplitter\DocumentSplitter; 
use Doctrine\ODM\MongoDB\DocumentManager;

#[AsCommand(
    name: 'app:create-chunks',
    description: 'This command will generate chunks from the rst files and store into MongoDB',
)]
class CreateChunks extends Command
{
    protected static $defaultName = 'app:create-chunks';
    private DocumentManager $documentManager;

    public function __construct(DocumentManager $documentManager)
{
    parent::__construct();
    $this->documentManager = $documentManager;
}

protected function execute(InputInterface $input, OutputInterface $output): int
{
    $io = new SymfonyStyle($input, $output);
    $io->title("Chunking all .rst files and storing them into MongoDB");

    $directory = '../../public/';
    if (!is_dir($directory)) {
        throw new \Exception("Directory not found: " . $directory);
    }

    $totalChunks = 0;

    foreach (new \DirectoryIterator($directory) as $fileInfo) {
        if ($fileInfo->isFile() && $fileInfo->getExtension() === 'rst') {
            $filePath = $fileInfo->getPathname();
            $io->section("Processing file: " . $fileInfo->getFilename());

            $dataReader = new FileDataReader($filePath);
            $documents = $dataReader->getDocuments();

            $splittedDocuments = DocumentSplitter::splitDocuments($documents, 1000, '.', 20);

            foreach ($splittedDocuments as $doc) {
                $chunk = new ChunkedDocuments();
                $chunk->setContent($doc->content);

                $chunk->setSourceName($doc->sourceName);
                $chunk->setCreatedAt(new \DateTime());

                $this->documentManager->persist($chunk);
                $totalChunks++;
            }

            $this->documentManager->flush();
            $this->documentManager->clear(); 
        }
    }

    $io->success("Successfully stored $totalChunks chunks in MongoDB.");
    return Command::SUCCESS;
}

}

To execute the command, use the below command:

php bin/console app:create-chunks

Step 3: Create an embedding for the chunk text that has been stored in MongoDB

Once the chunks have been stored, the next step is to create the embeddings. An embedding is a mathematical representation of data in the high dimensional space known as vector space. These are the vector representations of the words, phrases, or, at times, complete texts or images. These numerical representations provide more contextual awareness of the words.

To create these embeddings, we will be making use of the Voyage AI embedding model and storing them in MongoDB. To do so, we will create another Symfony command to create those embeddings and flush them into the database.

The EmbeddedChunks.php command class is responsible for generating vector embeddings for document chunks stored in MongoDB. It contains two main functions:

The execute() method retrieves all chunked documents from the ChunkedDocuments collection and processes them in batches. Each batch of content is then passed to the Voyage AI API for embedding.
The embedAndPersist() method handles the actual REST API call to Voyage AI. It sends the batch of texts, receives the generated embeddings, and stores them back into the corresponding documents in the database.

<?php

declare(strict_types=1);

namespace App\Command;

use App\Document\ChunkedDocuments;
use Doctrine\ODM\MongoDB\DocumentManager;
use Symfony\Contracts\HttpClient\HttpClientInterface;
use Symfony\Component\Console\Attribute\AsCommand;
use Symfony\Component\Console\Command\Command;
use Symfony\Component\Console\Input\InputInterface;
use Symfony\Component\Console\Output\OutputInterface;
use Symfony\Component\Console\Style\SymfonyStyle;
use Symfony\Component\DependencyInjection\Attribute\Autowire;

#[AsCommand(
    name: 'app:embed-chunks',
    description: 'This command will create embedding for the stored chunks using Voyage AI API key and store into MongoDB database',
)]
class EmbeddedChunks extends Command
{

    public function __construct(
        private readonly DocumentManager $documentManager,
        #[Autowire(env: 'VOYAGE_API_KEY')]
        private readonly string $voyageAiApiKey,

        #[Autowire(env: 'VOYAGE_ENDPOINT')]
        private readonly string $voyageEndpoint,

        #[Autowire(env: 'BATCH_SIZE')]
        private readonly int $batchSize,

        #[Autowire(env: 'MAX_RETRIES')]
        private readonly int $maxRetries,

        private HttpClientInterface $client,
    ) {
        parent::__construct();
    }

    protected function execute(InputInterface $input, OutputInterface $output): int
    {
        $io = new SymfonyStyle($input, $output);
        $io->title('Batch embedding documents from MongoDB using Voyage AI');

        $documents = $this->documentManager
            ->getRepository(ChunkedDocuments::class)
            ->findAll();

        if (empty($documents)) {
            $io->warning('No documents found.');
            return Command::SUCCESS;
        }


        $batchedContent = [];
        $originalDocs = [];
        $embeddedCount = 0;

        foreach ($documents as $doc) {
            $text = trim($doc->getContent());
            if (empty($text)) continue;

            $text = mb_substr($text, 0, 4000); 

            $batchedContent[] = $text;
            $originalDocs[] = $doc;

            if (count($batchedContent) >= $this->batchSize) {
                $embeddedCount += $this->embedAndPersist($this->client, $batchedContent, $originalDocs, $io);
                $batchedContent = [];
                $originalDocs = [];
                gc_collect_cycles();
            }
        }

        if (!empty($batchedContent)) {
            $embeddedCount += $this->embedAndPersist($this->client, $batchedContent, $originalDocs, $io);
        }

        $this->documentManager->flush();

        $io->success("Embedded {$embeddedCount} documents.");
        return Command::SUCCESS;
    }

    private function embedAndPersist(HttpClientInterface $client, array $inputTexts, array $originalDocs, SymfonyStyle $io): int
{
    $payload = [
        'input' => $inputTexts,
        'model' => 'voyage-3',
        'input_type' => 'document',
    ];

    $embedded = 0;
    $attempt = 0;
    $success = false;

    while ($attempt < $this->maxRetries && !$success) {
        try {
            $attempt++;

            $response = $client->request('POST', $this->voyageEndpoint, [
                'headers' => [
                    'Authorization' => 'Bearer ' . $this->voyageAiApiKey,
                    'Content-Type' => 'application/json',
                ],
                'json' => $payload,
            ]);

            $data = json_decode($response->getContent(), true);

            if (!isset($data['data'])) {
                throw new \RuntimeException('Missing "data" in Voyage AI response');
            }

            foreach ($data['data'] as $index => $embeddingResult) {
                $embedding = $embeddingResult['embedding'] ?? null;
                if ($embedding === null) continue;

                $originalDoc = $originalDocs[$index];
                $originalDoc->setcontentEmbedding($embedding);
                $originalDoc->setcreatedAt(new \DateTime());
                $this->documentManager->persist($originalDoc);
                $embedded++;
            }

            $success = true;
        } catch (\Symfony\Contracts\HttpClient\Exception\TransportExceptionInterface |
                 \Symfony\Contracts\HttpClient\Exception\ClientExceptionInterface |
                 \Symfony\Contracts\HttpClient\Exception\ServerExceptionInterface $e) {
            $io->warning("Attempt $attempt failed: " . $e->getMessage());
            sleep(2 * $attempt);
        } catch (\Throwable $e) {
            $io->error('Unexpected error: ' . $e->getMessage());
            break;
        }
    }

    return $embedded;
}
}

Before the REST API call is made, it is important to store the API key and URL in the .env file as follows:

VOYAGE_API_KEY=<VoyageAPI_KEY>
VOYAGE_ENDPOINT=https://api.voyageai.com/v1/embeddings
BATCH_SIZE=32
MAX_RETRIES=3

To run the above command to create an embedding:

php -d memory_limit=2G bin/console app:embed-chunks

It is important to note here that this step works in batches and hence, this requires some time to create the embedding and store it in the database.

At this point, you should be able to see chunks and their embedding being stored inside the MongoDB database.

Step 4: Create vector search index

Vector search in MongoDB is a search method that allows you to perform semantic searching. It searches based on the vector embeddings that are close to the search query. It makes use of the vector search index to perform the vector search.

To create a vector search index, navigate to the Search Tab of the Atlas UI. Click on the Vector Search tab and name the index, and select the right collection name on which the index is to be created. Click “Next” and select the path and the similarity method for the index. The below picture gives a reference for how the fields are to be filled.

Once done, click “Next,” and you should have your first vector index created.

To know more about creating an index, follow the documentation, How to Index Fields for Vector Search.

At this point, we are all set to create the responses.

Step 5: Create a vector search response into LLM formed response

This is the last and final step to perform the vector search on the stored, chunked data.

To do this, we first need to create the ChatController.php, which will hit the response call to generate the response.

Let’s first create the ChatController as:

public function index(Request $request): Response
{
    $question = null;
    $answer = null;

    if ($request->isMethod('POST')) {
        $question = trim(strtolower($request->request->get('question')));

        $answer = $this->getPredefinedResponse($question);

        if ($answer === null) {
            $answer = $this->responseService->getResponseForQuestion($question);
        }
    }

    return $this->render('chat.html.twig', [
        'question' => $question,
        'answer' => $answer,
    ]);
}

This controller will be hit from the front-end twig page created. This chat.html.twig renders the below page:

When a user submits a question via the form and clicks the "Send" button, the controller processes the input.

It first checks for any predefined answers. If none are found, it delegates the query to ResponseService.php, which performs the vector search and returns the most relevant answer based on the indexed content.

The response service file has three functions:

getResponseForQuestion() is the main entrypoint for generating the response. This generates an embedding using the generateEmbedding function and then performs the $vectorSearch aggregation.

This further passes the query response to the formatResponse() function to generate an LLM formatted response.

public function getResponseForQuestion(string $question): ?string
{
    $embeddedquestion = $this->generateEmbedding($question);

    if (!$embeddedquestion) {
        return "Sorry, couldn't understand the question.";
    }

    $collection = $this->documentManager->getDocumentCollection(ChunkedDocuments::class);

    $pipeline = new Pipeline(
        [Stage::vectorSearch(index: 'vector_index', path: 'contentEmbedding', queryVector: $embeddedquestion, limit: 5, numCandidates: 100)],
        [Stage::project(_id: 0, content: 1)]
    );

    $cursor = $collection->aggregate($pipeline)->toArray();
    $contents = array_map(fn($item) => $item['content'], $cursor);

    return $this->formatResponse($contents, $question);
}

generateEmbeddings() sends the user's question to Voyage AI to generate a vector embedding.

private function generateEmbedding(string $text): ?array
{
    $client = new \GuzzleHttp\Client();

    try {
        $response = $client->post($this->voyageEndpoint, [
            'headers' => [
                'Authorization' => 'Bearer ' . $this->voyageAiApiKey,
                'Content-Type' => 'application/json',
            ],
            'json' => [
                'input' => [$text],
                'model' => 'voyage-3',
                'input_type' => 'query'
            ],
            'timeout' => 20,
        ]);

        $data = json_decode($response->getBody()->getContents(), true);
        return $data['data'][0]['embedding'] ?? null;
    } catch (\Throwable $e) {
        return null;
    }
}

And finally, the formatResponse()** **sends the retrieved context and user question to OpenAI's API to format the response in a structured and helpful way. It creates a structured message payload expected by OpenAI's chat completion endpoint.

public function formatResponse(array $contents, string $query): string
{

    $client = new \GuzzleHttp\Client();
    $combineResponse = implode("\n", array_map('trim', $contents));

    $messages = [
        [
            'role' => 'system',
            'content' => 'You are a Symfony chatbot. You help users by answering questions based on Symfony documentation.'
        ],
        [
            'role' => 'user',
            'content' => "Using the following context from Symfony documentation:\n\n" . $combineResponse
        ]
    ];

    if ($query) {
        $messages[] = [
            'role' => 'user',
            'content' => "User query: " . $query
        ];
    }
    try {
        $response = $client->post($this->openAiApiUrl, [
            'headers' => [
                'Authorization' => 'Bearer ' . $this->openAiApiKey,
                'Content-Type' => 'application/json',
            ],
            'json' => [
                'model' => 'gpt-4.1',
                'messages' => $messages
            ],
            'timeout' => 30,
        ]);

        $data = json_decode($response->getBody()->getContents(), true);
        $OpenAiResponse = $data['choices'][0]['message']['content'] ?? 'No reply from OpenAI.';

        return $OpenAiResponse;

    } catch (\Throwable $e) {
        return "Error generating response: " . $e->getMessage();
    }
}

Again, before using these functions, it is important to add the OPENAI_KEY and URL added in the .env file as follows:

OPENAI_API_KEY=<OPENAI_KEY>
OPENAI_API_URL=https://api.openai.com/v1/chat/completions

Finally, the getPredefinedResponse() method is used in the ChatController to handle simple, commonly used conversational phrases like "hi," "thanks," and "goodbye" with instant, hardcoded replies. This approach offers several key benefits:

Performance: It prevents the need to call the external APIs, resulting in faster response.
Cost-efficient: Since this reduces the number of requests being sent to Voyage AI and OpenAI, it helps largely in reducing the costs.
Resource optimization: This prevents unnecessary computational overhead, reserving the more complex and expensive AI operations for actual Symfony-related technical queries.

To do so, we have added an extra function that takes care of the predefined questions:

private function getPredefinedResponse(string $message): ?string
{
    $responses = [
        'hi' => 'Hello! How can I help you with Symfony today?',
        'hello' => 'Hello! What can I do for you today? We recommend asking a Symfony based question',
        'hey' => 'Hey there! You can ask me anything about Symfony',
        'how are you' => 'I am doing fine, how are you? Are you looking for Symfony questions',
        'good morning' => 'Good morning! Do you have any Symfony related queries?',
        'good evening' => 'Good evening! Need help with something Symfony-related?',
        'bye' => 'Goodbye! ',
        'goodbye' => 'Bbye',
        'thank you' => 'You are welcome! Please feel free to ask any further questions',
        'thanks' => 'Glad to be helpful',
        'good night' => 'Good night',
    ];

    foreach ($responses as $key => $response) {
        if (strpos($message, $key) !== false) {
            return $response;
        }
    }
    return null;
}

At this stage, your application is fully set up and ready to be deployed as a chatbot for the Symfony Documentation page. To launch the application locally and start interacting with the chatbot, simply run the following command:

symfony server:start

This will start the Symfony development server and make your chatbot accessible for use.

Testing

Once the application is started, you can start by asking questions like:

What are the key differences between Doctrine ORM and ODM?

You should get a response like:

Or ask:

How does Symfony support different persistence layers like Doctrine ORM vs ODM?

The application replies:

Similarly, let’s also check how the application responds to preformatted questions like “hello,” “bye,” etc.

After this, you can play around with the questions for chatbot and see the response being received.

Conclusion

Finally, building intelligent chatbots is no longer reserved for complex, cloud-heavy architectures. With tools like Symfony, MongoDB Atlas, and vector search, we now have a clean, elegant, and developer-friendly way to create search-driven, conversational agents that are fast, context-aware, and scalable.

In this tutorial, we have explored how to build a powerful chatbot application using Symfony and MongoDB’s vector search. By integrating natural language processing, embeddings, and MongoDB Atlas, we created a bot that can intelligently search through technical documentation and return accurate responses in real time. Symfony's modularity and MongoDB's rich document model make this a scalable and extensible solution for developer support, internal tools, and customer service platforms.

If you have any further questions related to Symfony or any tech stack that has been mentioned, please refer to our official documentation page for more understanding.

Data Modeling for Java Developers: Structuring With PostgreSQL and MongoDB

Aasawari Sahasrabuddhe — Thu, 20 Mar 2025 23:00:00 +0000

Application and system designs have always been considered the most essential step in application development. All the later steps and technologies to be used depend on how the system has been designed. If you are a Java developer, choosing the right approach can mean distinguishing between a rigid, complex schema and a nimble, scalable solution. If you are a Java developer who works with PostgreSQL or other relational databases, you understand the pain of representing the many-to-many relationships between the tables.

This tutorial will ease your pain with these or other relationships defined in databases by making use of a document database, MongoDB.

In this article, we’ll understand both approaches, contrasting PostgreSQL’s relational rigour with MongoDB’s document-oriented simplicity. You’ll learn how to weigh trade-offs like ACID compliance versus scalability and discover why MongoDB’s design might save you time.

Relationships in databases

Relationships in databases introduce the connection between the entities of two or more different collections or tables. These relationships help the applications to have the dependencies interlinked with each other. To understand this in brief, let’s look at an example of a library management system that has tables for authors, books, issue details, etc. inside the database.

One-to-one relationship

Each record in Table A is related to only one record in Table B. Example: A Library Member table and an Issue Details table can have a one-to-one relationship, where each member can have only one active book issued at a time.

One-to-many relationship

A single record in Table A can be related to multiple records in Table B. For instance, a Book can have many Reviews or a user can have many contact addresses.

Many-to-many relationship

In this type of relationship, multiple records in Table A can relate to various records in Table B, requiring a junction (or bridge) table. For example, a Book can be issued to multiple Members over time, and a Member can borrow numerous Books.

Relational vs. document databases

A relational database is based on rigid predefined schemas and emphasises relationships. These work well when the structure is strict and is not subjected to much change in the future. SQL or MySQL has been the backbone of relational databases for many years now.

Whether you’re organizing customer contacts or powering complex financial systems, SQL’s straightforward yet powerful commands have made it a go-to tool for businesses worldwide—and it’s been that way for decades. Relational databases thrive on flexibility, adapting to everything from small-scale apps to enterprise-level solutions. That’s why industries of all kinds, from retail to healthcare, rely on them to store, manage, and make sense of their data. Think of SQL as the quiet workhorse behind the scenes, keeping everything running smoothly.

A document database, on the other hand, is meant for a more flexible structure. They work well with modern applications where a variety of data is stored and processed from the applications. A document database represents data as documents, typically using formats like JSON (JavaScript object notation) or BSON (Binary JSON). A document database is often referred to as a NoSQL database.

MongoDB is the most popular database used in modern application development. The flexible storage capability allows the application to scale horizontally and grow efficiently with the rapid changes.

Postgres implementation with Java

To understand how data in a Postgres database would be implemented with Java, let us consider an example ER diagram for a library management system consisting of Books, Authors, and Issue Details.

The ER diagram below depicts the one-to-many relationship between Books and Issue Details and the many-to-many relationship between Author and Books using the Author_Books table created using join operations.

Fig: ER diagram for the library management system using four different tables

Now, let us try to understand how these entities will be represented using the Java Hibernate code and the relationship annotations.

Let's first understand the Author representing the Author entity:

@Entity
public class Author {
    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Long authorID;

    private String name;
    private String email;
    private String nationality;

    @ManyToMany(mappedBy = "authors", cascade = CascadeType.ALL)
    private Set<Book> books;
}

Similarly, the Books entity would be represented as:

@Entity
public class Book {
    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Long bookID;

    private String title;
    private String isbn;
    private String genre;
    private int publishedYear;

    @ManyToMany(cascade = CascadeType.ALL)
    @JoinTable(
        name = "author_books",
        joinColumns = @JoinColumn(name = "bookID"),
        inverseJoinColumns = @JoinColumn(name = "authorID")
    )
    private Set<Author> authors;

    @OneToMany(mappedBy = "book", cascade = CascadeType.ALL)
    private Set<IssueDetails> issueDetails;

    //Getters and Setters
}

Now, the Author_books table has been created using the join methods and hence, we do not need to define a separate entity class.

The relationship between the entities has been established using the relationship annotations like @ManytoMany.

Similarly, to represent the one-to-many relationship between Books and the Issue Details using the @OneToMany annotation, we can write the entity class as:

@Entity
public class IssueDetails {
    @Id
    @GeneratedValue(strategy = GenerationType.IDENTITY)
    private Long issueID;

    @ManyToOne
    @JoinColumn(name = "bookID", nullable = false)
    private Book book;

    private Long userID;
    private Date issueDate;
    private Date returnDate;
    private String status;

    // Getters and Setters
}

The relationships implemented in the above code snippet could be very simple. If you are wondering how, the answer to this is in the next section where we will understand how simple it can be to use MongoDB as the database.

MongoDB implementation with Java

Let us understand the representation in a simpler way using the MongoDB document model and its data modelling techniques.

For example, representing the many-to-many relationship between the authors and books can be done using embedding or referencing data modelling techniques.

It is also important to note that if authors and books are frequently queried together, embedding author references within books or vice versa might be efficient. However, since an author can write many books and a book can have multiple authors, embedding might lead to duplication. So maybe using references with arrays of author IDs in books and book IDs in authors would be better. But that could complicate updates if an author's information changes. Alternatively, just using an array of author IDs in the book documents and not maintaining the inverse in authors might suffice, depending on query needs.

Depending on the modelling technique chosen, the documents inside in the collection would change. For example:

{
  _id: ObjectId("60d5ec9f4b1a8e2a1c8f7a1"), 
  name: "J.K. Rowling",
  email: "jkrowling@example.com",
  nationality: "British",
  books: [
    ObjectId("60d5ec9f4b1a8e2a1c8f7a2"), 
    ObjectId("60d5ec9f4b1a8e2a1c8f7a3")
  ]
}

The book collection would look like the below:

{
  _id: ObjectId("60d5ec9f4b1a8e2a1c8f7a2"), 
  title: "Harry Potter and the Sorcerer's Stone",
  isbn: "9780590353427",
  genre: "Fantasy",
  publishedYear: 1997,
  authors: [
    ObjectId("60d5ec9f4b1a8e2a1c8f7a1") 
  ],
  issues: [
    {
      userId: ObjectId("60d5ec9f4b1a8e2a1c8f7a4"), 
      issueDate: ISODate("2023-10-01"),
      returnDate: ISODate("2023-10-15"),
      status: "Returned"
    }
  ]
}

As you can see from the above example, the document model of MongoDB is similar to the POJOs representation in Java. Therefore, the representation of the model class becomes simpler with Java.

For example, the author class in Java would be written as:

@Document(collection = "authors")
public class Author {
    @Id
    private ObjectId id;
    private String name;
    private String email;
    private String nationality;
    private List<Book> bookIds; 
    //Getters and Setters
}

The book class can be written as:

@Document(collection = "books")
public class Book {
    @Id
    private ObjectId id;
    private String title;
    private String isbn;
    private String genre;
    private int publishedYear;
    private List<Author> authorIds;
    //  Getters and Setters
}

From the above code snippet, you can see cleaner and less complex code without any complicated annotations to represent the relationships.

Scalability and performance

After the demonstration of a few small code snippets from both databases, it is also important to understand the scalability and the performance considerations while creating a Java application.

We have all heard by now that Postgres is a powerful relational database that handles the structured query, but what does it do when an application needs to be scaled? Postgres scales vertically, which involves upgrading the hardware of a single server (e.g., adding more CPU, RAM, or storage). This approach is ideal for applications with predictable growth and moderate traffic. Still, it comes with limitations as hardware upgrades can become expensive, and there’s a physical ceiling to how much a single server can handle.

On the other hand, MongoDB is designed for scalability and high performance, especially regarding unstructured data and modern applications. MongoDB performs scalability using horizontal scaling, which means distributing data across multiple servers. This process in MongoDB is known as sharding, and the shards are created by selecting the right and appropriate shard key. Additionally, MongoDB optimizes read and write throughput through features like replica sets (for read scaling) and batch operations (for write scaling).

The performance of Java applications with MongoDB is optimised using demoralizations and indexing. Embedding and referencing the data reduces the need for expensive joins and enhances the query and application performance.

Migration considerations

When you migrate the data from Postgres to MongoDB, the process involves more than just moving the data. It specifically requires you to think about the structuring of the data and also how to manage the data. In the next section, we’ll talk about those briefly.

Rethinking schema design

In a relational database like Postgres, data is often normalised across multiple tables—in our case, between Authors, Books, and Issue Details. In MongoDB, we need to combine the related data into embedded documents. For example, instead of storing authors and books in separate tables, you can embed author information directly within a book document.

@Document(collection = "books")
public class Book {
    private String title;
    private List<Author> authors; // Embedded authors
    private List<IssueDetails> issues; // Embedded issues
}

Replacing joins with aggregations

Postgres makes use of joins to retrieve the related data. On the other hand, MongoDB relies mostly on writing aggregation pipelines to retrieve the related information. For example, a Postgres query might be:

SELECT books.title, authors.name 
FROM books 
JOIN author_books ON books.book_id = author_books.book_id 
JOIN authors ON author_books.author_id = authors.author_id;

This could simply be done using the Java code using the aggregation as:

Document lookupStage = new Document("$lookup", new Document("from", "authors") .append("localField", "authors") .append("foreignField", "_id") .append("as", "authorDetails") ); 
AggregateIterable<Document> result = booksCollection.aggregate(Arrays.asList(lookupStage));

Conclusion

Data modelling is a crucial step in application development and selecting the right database can significantly impact the scalability, performance, and maintainability of the system. For Java developers, identifying the strengths and weaknesses of the relational and non-relational databases is essential to correctly structure and manage the data.

From this tutorial, we have understood that PostgreSQL, with its relational model, excels in scenarios requiring complex queries, ACID compliance, and structured data. Its ability to handle relationships through joins and enforce constraints makes it ideal for applications like financial systems, ERP platforms, and reporting tools. However, its reliance on vertical scaling can become a limitation as applications grow.

On the other hand, MongoDB’s data document model offers greater flexibility, scalability, and higher performance, which makes it a great fit for modern applications. Its ability to handle unstructured data and scale horizontally allows developers to build nimble, scalable solutions without the rigidity of a predefined schema.

MongoDB’s document model becomes a natural fit for the unstructured data in modern applications, offering Java developers the ability to store, query, and manipulate complex data without rigid schemas. With powerful indexing, aggregation pipelines, and native support for JSON-like BSON documents, MongoDB simplifies data access, transformations, and manipulation easier. Whether handling product catalogues, customer interactions, or AI-driven insights, MongoDB empowers Java developers to build scalable, high-performance applications with minimal friction.

Building a Spring Boot CRUD Application Using MongoDB’s Relational Migrator

Aasawari Sahasrabuddhe — Tue, 18 Feb 2025 14:00:00 +0000

Imagine this: You’re working with a relational database that’s served you well for years, but now your applications demand flexibility, scalability, and faster development cycles. Transitioning from structured tables to MongoDB’s dynamic, schema-less JSON documents might be daunting. That’s where MongoDB’s Relational Migrator becomes your secret weapon.

The Relational Migrator tool built by MongoDB no longer only supports migrating data from relational databases to MongoDB. Rather, it supports features like code generation and also a query converter that converts a relational database, such as a Postgres query, or queries from views and stored procedures, into a MongoDB query.

Relational Migrator is a free tool that leverages intelligent algorithms and Gen AI to solve the most common data modeling, code conversion, and migration challenges, reducing the effort and risk involved in migration projects.

In this tutorial, we’ll explore how to harness the power of Spring Boot to build a fully functional CRUD application on top of your modernized database using the Relational Migrator.

This guide will help modernize a legacy application built with relational databases into a complete application using MongoDB.

Let's get started!

Pre-requisites

Before we get into the actual implementation to build the application, below are the pre-requisites that you need to have:

Relational Migrator—you can download the tool available for free using the MongoDB Tools download page
The SQL schema script to create the Postgres database and tables
Java 21 or above
The IDE of your choice
A free Atlas cluster—create your first Atlas cluster for free

Relational Migrator

The Relational Migrator is a tool developed by MongoDB that helps you migrate data from a relational database into MongoDB. It is an application with a UI that helps you map the data from the relational schema to the MongoDB schema.

Along with providing migration capability, the tool offers more than the name suggests.

Generate the application code: After the data has been migrated into the Atlas cluster, the developer asks how we begin the text. Well! The code generation feature of the RM tool is the answer to the questions. This allows you to get the boilerplate code for performing the basic CRUD operations and also expand the project to perform other complex functionality based on the project requirements.
Query converter: The tool's query converter feature helps you convert your SQL queries, views, and stored procedures into equivalent MongoDB queries and validate them. The query converter uses the relational schema, the MongoDB schema, and the mapping rules in your current project to determine how the queries should be converted. Conversions may fail or be incorrect if the queries reference tables that are not in your relational schema or if they are not mapped to MongoDB collections. This query converter also works for views and stored procedures.

Migrating the data from the PostgreSQL schema to MongoDB

Once your free Atlas cluster is ready, download the SQL schema script, which has a sample Postgres schema with data inserted into the tables. Use the PgAdmin application to upload the schema and explore the created schema. The sample schema created nine different tables and their relationships for this tutorial. We will use the same schema to generate the equivalent and simpler MongoDB schema, following the different design patterns. First, let's examine both schemas in detail and then understand how to convert them to the equivalent MongoDB schema for further application development.

Analysing the Postgres schema

In this section, we will understand an example of a library management system where we have nine different tables, and the relationship between them is shown in the diagram below:

The ER diagram above also depicts the relationship between these tables. Relational databases often require intricate joins, nested subqueries, and multiple layers of aggregation to retrieve even moderately complex data relationships. As a result, what might seem like a simple query can quickly escalate into a large, complex, and unwieldy SQL statement. This not only increases the cognitive load for developers but also makes the code harder to maintain and prone to errors.

For example, let’s consider a scenario where we need to list all authors with their aliases.

To do so, we would write the SQL query as:

SELECT a.name, aa.alias
FROM library.authors a
LEFT JOIN library.author_alias aa ON a.id = aa.author_id;

This leads to the joining of two different tables using the foreign key constraint. In the next section, we will understand how this query would look in MongoDB.

Creating mappings to generate the equivalent MongoDB schema

The nine different tables in the relational schema need to be converted into five different collections using the mapping technique given by MongoDB. These techniques follow the design patterns followed by MongoDB in data modelling concepts.

After the mappings have been applied to the tables, the MongoDB schema would like the following:

Let's try to convert the above SQL query into the equivalent MongoDB query:

db.collection.aggregate([
{
$project: {
"name": 1,
"aliases": 1,
"_id": 0
}
}
]
)

This is one such example where a huge ER diagram with nine different tables can be combined to form five different collections and make the query simpler.

After applying the mapping techniques, once the desired MongoDB schema is achieved, the next step is to migrate the data into the MongoDB Atlas cluster.

Migrating the data into MongoDB

To migrate the data into the cluster, click on “Data Migration” and select “Create Job Migration.”

To begin the migration, you should have the host as the relational database connection secure with you, and the destination is where you can apply the MongoDB connection string. Once both the source and the destination databases are all set up, the next step would be to select the type of migrations.

In MongoDB, Relational Migrator offers two types of migrations:

Continuous migrations: Continuous migration jobs cover new incoming data with a zero-downtime change data capture (CDC) migration strategy.
Snapshot migration: This only runs once and usually does a point-in-time migration.

After selecting the appropriate migration job based on your requirements, the next step is to click on “Review Summary” and then “Start.” The migration process will take some time, depending on the size of your data.

Once the migration is complete, you'll find your data seamlessly transferred to the MongoDB Atlas cluster. However, this is often the point where developers encounter uncertainty about what to do next.

But don’t worry—MongoDB Relational Migrator eliminates that confusion.

The tool’s Code Converter feature is designed to simplify your journey further by providing boilerplate code in the programming language of your choice. This makes it easy to jumpstart your application development, so you can focus on building great features without worrying about the groundwork.

Code generation with Relational Migrator

To create an application with migrated data using Relational Migrator, you should click on the “Code Generation” tab and select the language and template as Java and Spring Data, respectively. The below screenshot from the Relational Migrator tools explains the steps for the code generation.

Once you select the framework, you are all set to download/copy the code in the editor of your choice for further processing.

Building Spring Boot application

At this point, building the application becomes very easy, as all the essential code is provided. You just need to perform the business logic, and the application is all set to fulfil the requirements.

In this tutorial, we are using IntelliJ as the IDE and copying all the code provided by the Relational Migrator into a new Spring Boot project created using the Spring Initializr.

Depending on the data model that has been selected, the Relational Migrator tools create the entity and the repository files for the application.

Once the entity and the repository files are all copied/downloaded, all you need to do is create the service and the controller to perform the business logic of the applications. After all the files are copied, the project will look like this:

The controller has all the REST API calls, and the service files are where we need to write the aggregate pipelines.

Let us understand a few examples in the next section.

Examples of aggregation pipelines

At this point in the application development, we have all the entity and repository files required for building the complete application.

Let us take an example where you need to find all books reviewed within a specific date range. To perform so in the Postgres query, you would have the query as:

SELECT 
    r.timestamp,
    u.name AS reviewer_name,
    b.title AS book_title,
    r.rating,
    r.text
FROM 
    library.reviews r
JOIN 
    library.books b ON r.book_id = b.id
JOIN 
    library.users u ON r.user_id = u.id
WHERE 
    r.timestamp BETWEEN '2022-01-01' AND '2024-12-31'
ORDER BY 
    r.timestamp ASC;

The equivalent MongoDB Java query in the schema can simply be written as:

public List<ReviewsEntity> getBookReviews() {
        Aggregation aggregation = Aggregation.newAggregation(
                Aggregation.match(Criteria.where("timestamp")
                        .gte(new java.util.Date(1640995200000L))
                        .lte(new java.util.Date(1735603200000L))),
                Aggregation.lookup("books", "_id.bookId", "_id", "book"),
                Aggregation.unwind("book"),
                Aggregation.project()
                        .and("timestamp").as("timestamp")
                        .and("book.title").as("bookTitle")
                        .and("rating").as("rating")
                        .and("text").as("text"),
                Aggregation.sort(Sort.by(Sort.Direction.ASC, "timestamp"))
        );

        AggregationResults<ReviewsEntity> result = mongoTemplate.aggregate(
                aggregation,
                "reviews",
                ReviewsEntity.class
        );
        return result.getMappedResults();
    }

Similarly, a simple query can get an alias name for the author, for which the Postgres query can be written as below:

SELECT a.name, aa.alias
FROM library.authors a
LEFT JOIN library.author_alias aa ON a.id = aa.author_id;

This Postgres query can be simply used in MongoDB using the $project stage as:

public List<AuthorAlias> getAuthorAliases() {
        Aggregation aggregation = Aggregation.newAggregation(
                Aggregation.project()
                        .andInclude("name", "authorAliases")
                        .andExclude("_id")
        );
        AggregationResults<AuthorAlias> result = mongoTemplate.aggregate(
                aggregation,
                "authors",
                AuthorAlias.class
        );
        return result.getMappedResults();
    }

More such examples are provided in the GitHub repository, which you can use and extend to perform more functionalities.

This tutorial therefore explains how to leverage the Relational Migrator and its functionality to create applications with no complexity and where the focus is more on the business logic and less on the boilerplate repetitive code.

Conclusion

This article shows how transitioning from traditional relational databases to MongoDB’s dynamic and schema-less architecture can be overwhelming, but MongoDB’s Relational Migrator simplifies the process. This tool facilitates seamless data migration and offers advanced features like SQL-to-MongoDB query conversion and automatic boilerplate code generation. In a practical tutorial, the article demonstrates how to modernize a legacy relational database-driven application into a Spring Boot-based CRUD application using MongoDB and Relational Migrator. By converting a sample relational schema into an optimized MongoDB schema, the tool reduces the complexity of joins and nested queries, paving the way for faster and more intuitive development cycles.

The guide further explores the step-by-step implementation, from migrating data into MongoDB Atlas using Snapshot or Continuous Migration strategies to generating ready-to-use application code for CRUD operations. The Relational Migrator provides a streamlined workflow for developers by offering code templates compatible with Java and Spring Data, making it easy to build on top of the migrated database. Additionally, the article showcases how MongoDB’s aggregation framework simplifies queries, such as filtering reviews within a date range or retrieving authors with aliases, compared to SQL. With examples and practical insights, this tutorial empowers developers to create scalable, modern applications effortlessly.

If you have any questions related to the article or you wish to learn more about MongoDB concepts, please visit the MongoDB community forums for the latest updates.