DEV Community: MongoDB

Test Password Strength and Password History with TypeScript and MongoDB

Nic Raboy — Fri, 15 May 2026 13:55:05 +0000

Just about every application you build is going to need some form of user authentication, and the moment you have user accounts, you have passwords to manage. Storing them safely is only part of the job. You also need to make sure those passwords are strong enough to be worth protecting in the first place, and in many cases, you need to make sure the same password isn't recycled every time it’s rotated.

Password reuse is a bigger deal than it sounds. Compliance frameworks frequently require that the last several passwords cannot be used again, and even outside of compliance, it is good hygiene. If a password leaks today and a user just rotates back to it six months from now, your rotation policy did not actually protect anyone.

MongoDB happens to be a really good fit for this kind of feature. A user document can store the current password hash at the root of the document, along with a growing array of previous hashes, all in the same record. There is no separate join table to set up and no migration to run when a user changes their password for the tenth time.

In this tutorial, we'll see how to build a small TypeScript API using Express Framework, Zod, and the MongoDB Node.js driver. This application registers users with strong password rules, authenticates them, and rejects any password change that has been seen before.

The Prerequisites

Prior to starting this tutorial, you'll need a few things in place:

A MongoDB instance, either a local server or an Atlas cluster on the free tier
Node.js 22+

The expectation is that you can already connect to MongoDB with a connection string. If you need help getting an Atlas cluster running, check out the MongoDB documentation for getting started.

We're working from an empty project, so the first thing to do is create the directory and initialize it:

mkdir typescript-password-history-example
cd typescript-password-history-example
npm init -y

The above commands create the project directory and a starter package.json file.

Next, we'll install the runtime dependencies:

npm install express mongodb bcrypt zod dotenv

We're pulling in Express Framework as the HTTP layer, the official MongoDB Node.js driver, bcrypt for hashing passwords, zod for schema validation, and dotenv for loading configuration from an environment file.

For TypeScript and the type definitions, run:

npm install --save-dev typescript ts-node @types/express @types/node @types/bcrypt

The ts-node package lets us run TypeScript directly during development without a separate compile step, keeping the feedback loop tight.

Because this is a TypeScript project, we need a tsconfig.json at the root of the project:

{
    "compilerOptions": {
        "target": "ES2020",
        "module": "commonjs",
        "lib": ["ES2020"],
        "outDir": "./dist",
        "rootDir": "./",
        "strict": true,
        "esModuleInterop": true,
        "skipLibCheck": true,
        "forceConsistentCasingInFileNames": true,
        "resolveJsonModule": true
    },
    "include": ["./**/*.ts"],
    "exclude": ["node_modules", "dist"]
}

The configuration emits CommonJS to a dist/ directory when we run tsc, and as strict mode is turned on, the compiler actually enforces type safety rather than letting things slide.

It also helps to update the scripts block of package.json, so we have a development command and a production command:

"scripts": {
    "build": "tsc",
    "start": "node dist/main.js",
    "dev": "ts-node main.ts"
}

The dev script will run the TypeScript entry point directly. The build script compiles to dist/, and start runs the compiled output.

By the time we're done, the project will have the following shape:

.
├── .env
├── main.ts
├── tsconfig.json
├── package.json
├── libs/
│   ├── mongodb.ts
│   └── validate.ts
└── routes/
    └── passwords.ts

Each of those files will get built up as we work through the tutorial. Go ahead and create the libs and routes directories now so you have somewhere to drop code as we go.

The last piece of setup is the environment file. Create a .env file at the root of the project with the following:

MONGODB_URI=ADD_YOUR_ATLAS_URI_HERE
MONGODB_DB_NAME=password_history_demo
PORT=3000

Replace the MONGODB_URI with the connection string for whatever MongoDB instance you're using. The password_history_demo database specified in MONGODB_DB_NAME does not need to exist ahead of time. MongoDB will create it lazily the first time we write data. If you're using version control, make sure .env is listed in your .gitignore so the connection string does not get committed.

Establish an API Foundation with Express and TypeScript

Before we get into password logic, we need an Express application that listens for HTTP requests and a clean way to start and stop it. The entry point lives in main.ts at the root of the project.

Add the following to main.ts:

import * as dotenv from "dotenv";
dotenv.config();

import express from "express";
import { getDb, closeConnection } from "./libs/mongodb";
import passwordRouter from "./routes/passwords";

const app = express();
const PORT = process.env.PORT ?? 3000;

app.use(express.json());

app.use("/users", passwordRouter);

async function start() {
    const db = await getDb();
    await db.collection("users").createIndex({ username: 1 }, { unique: true });

    const server = app.listen(PORT, () => {
        console.log(`Server running on http://localhost:${PORT}`);
    });

    process.on("SIGTERM", async () => {
        console.log("SIGTERM received, shutting down gracefully...");
        server.close(async () => {
            await closeConnection();
            process.exit(0);
        });
    });

    process.on("SIGINT", async () => {
        console.log("SIGINT received, shutting down gracefully...");
        server.close(async () => {
            await closeConnection();
            process.exit(0);
        });
    });
}

start();

That file is doing a lot, so let's pull it apart.

At the top, we have two dotenv lines followed by the rest of the imports:

import * as dotenv from "dotenv";
dotenv.config();

import express from "express";
import { getDb, closeConnection } from "./libs/mongodb";
import passwordRouter from "./routes/passwords";

The order matters here. dotenv.config() runs before the MongoDB module is imported, which means the connection string and database name will already be present in process.env by the time libs/mongodb.ts is evaluated.

Next, we have the Express application setup:

const app = express();
const PORT = process.env.PORT ?? 3000;

app.use(express.json());

app.use("/users", passwordRouter);

The express.json() middleware parses incoming JSON bodies so we can read req.body as a plain object later. The router is mounted under /users, so every endpoint in routes/passwords.ts will be reachable beneath that prefix. We'll write that router shortly.

Then we have the start function, which is the part that actually brings the server online:

async function start() {
    const db = await getDb();
    await db.collection("users").createIndex({ username: 1 }, { unique: true });

    const server = app.listen(PORT, () => {
        console.log(`Server running on http://localhost:${PORT}`);
    });

    // ... signal handlers shown below
}

The first thing it does is grab a database reference and create a unique index on the username field of the users collection. Doing it at startup means we don't have to check for duplicate usernames before every insert. MongoDB will raise an error with code 11000 if anyone tries to register a username that already exists, and we'll catch that error in the route handler. It is a much cleaner pattern than running a findOne followed by an insertOne, which is also racy.

After the index is ready, we call app.listen and hold onto the returned server reference so we can close it later.

The final piece is the shutdown handling, which lives inside the same start function:

async function start() {
    // ... database, index, and app.listen above

    process.on("SIGTERM", async () => {
        console.log("SIGTERM received, shutting down gracefully...");
        server.close(async () => {
            await closeConnection();
            process.exit(0);
        });
    });

    process.on("SIGINT", async () => {
        console.log("SIGINT received, shutting down gracefully...");
        server.close(async () => {
            await closeConnection();
            process.exit(0);
        });
    });
}

When the process receives either SIGTERM (which most container orchestrators send) or SIGINT (which is what Ctrl+C sends), we stop accepting new connections, close the MongoDB client, and exit. Without this, you'd leave an open client connection behind every time you restart the server.

Of course, none of this will compile yet because libs/mongodb.ts does not exist. That's our next step.

Create a Singleton MongoDB Class to Manage Database Connections

You could open a new MongoClient on every incoming request, but it is almost never what you want. Connecting is expensive, it adds latency, and it makes graceful shutdown much harder than it needs to be. A long-running API like ours should connect once when the process boots and reuse the same client for every request after that.

Open libs/mongodb.ts and add the following:

import { MongoClient, Db } from "mongodb";

const uri = process.env.MONGODB_URI as string;
const dbName = process.env.MONGODB_DB_NAME as string;

if (!uri) {
    throw new Error("MONGODB_URI is not defined in the environment variables");
}

if (!dbName) {
    throw new Error("MONGODB_DB_NAME is not defined in the environment variables");
}

let client: MongoClient | null = null;

export async function getClient(): Promise<MongoClient> {
    if (!client) {
        client = new MongoClient(uri, {
            connectTimeoutMS: 5000,
            socketTimeoutMS: 30000,
            serverSelectionTimeoutMS: 5000,
            appName: "devrel-tutorial-typescript-passwordhistory"
        });
        await client.connect();
    }
    return client;
}

export async function getDb(): Promise<Db> {
    const mongoClient = await getClient();
    return mongoClient.db(dbName);
}

export async function closeConnection(): Promise<void> {
    if (client) {
        await client.close();
        client = null;
    }
}

The top of the file reads the two environment variables we set earlier and fails fast if either of them is missing. There is no point in starting the server if it cannot reach the database.

The interesting bit is getClient:

export async function getClient(): Promise<MongoClient> {
    if (!client) {
        client = new MongoClient(uri, {
            connectTimeoutMS: 5000,
            socketTimeoutMS: 30000,
            serverSelectionTimeoutMS: 5000,
            appName: "devrel-github-typescript-passwordhistory"
        });
        await client.connect();
    }
    return client;
}

The module-level client variable starts as null. The first call to getClient instantiates the MongoClient, awaits connect, and caches it. Every subsequent call returns the cached instance immediately. That is the entire singleton pattern. We did not bother setting a maxPoolSize because this is a traditional long-running OLTP server, and the default value of 100 is appropriate for that workload.

The three timeout options are worth a quick mention. connectTimeoutMS caps how long a single socket connection attempt can take, socketTimeoutMS caps how long a socket can sit idle, and serverSelectionTimeoutMS caps how long the driver will spend looking for a suitable server before giving up. The appName value is cosmetic but useful. It shows up in MongoDB Atlas logs and metrics, so you can tell which application is generating which queries.

Then we have the two helpers that the rest of the application actually uses:

export async function getDb(): Promise<Db> {
    const mongoClient = await getClient();
    return mongoClient.db(dbName);
}

export async function closeConnection(): Promise<void> {
    if (client) {
        await client.close();
        client = null;
    }
}

The getDb function returns a Db reference scoped to the database name from the environment file. Anywhere in the codebase that needs to talk to MongoDB will go through this function. The closeConnection function shuts the client down and resets the module-level variable. This is what gets called by the SIGTERM and SIGINT handlers back in main.ts.

Test Password Strength with Regular Expressions and Zod

A password is only as good as the rules you enforce around it. For this tutorial, we're going to require a minimum of eight characters, at least one uppercase letter, at least one lowercase letter, at least one digit, and at least one character that is not a letter or a number. None of these rules is particularly exotic, but together they keep users from registering with something like password or 12345678.

We're going to enforce them with Zod schemas wired into Express as middleware. That way, the route handlers themselves stay focused on database work and never have to inspect raw request bodies.

Open libs/validate.ts and add the following:

import { Request, Response, NextFunction } from "express";
import { ZodSchema } from "zod";

export function validate(schema: ZodSchema) {
    return (req: Request, res: Response, next: NextFunction): void => {
        const result = schema.safeParse(req.body);
        if (!result.success) {
            res.status(400).json({ errors: result.error.flatten().fieldErrors });
            return;
        }
        req.body = result.data;
        next();
    };
}

The validate function is a middleware factory. You hand it a Zod schema, and it hands you back an Express middleware function that runs that schema against the incoming req.body. If validation fails, the middleware sends back a 400 with a flattened map of field errors, and the route handler is never called. If validation succeeds, it replaces req.body with the parsed output before calling next().

Replacing req.body with result.data is intentional. Zod will strip unknown keys and apply any transformations defined on the schema, so by the time the route handler runs, the body is guaranteed to look exactly like the schema declared. There is no risk of a stray field sneaking through.

Now we can define the actual schemas. These live alongside the route handlers in routes/passwords.ts, but we'll write that file in pieces throughout the rest of the tutorial. For now, here's the schema that encodes the password strength rules:

import { z } from "zod";

const strongPasswordSchema = z
    .string()
    .min(8, "Password must be at least 8 characters")
    .regex(/[A-Z]/, "Password must contain at least one uppercase letter")
    .regex(/[a-z]/, "Password must contain at least one lowercase letter")
    .regex(/[0-9]/, "Password must contain at least one digit")
    .regex(/[^A-Za-z0-9]/, "Password must contain at least one special character");

Each rule is its own .regex call with a custom error message. If a user submits something that breaks two rules, Zod will report both errors at once, which makes for a much better registration experience than trickling them out one at a time.

The character class [^A-Za-z0-9] is doing the work for the special character rule. It matches anything that is not an uppercase letter, a lowercase letter, or a digit. We're not picky about which special character. Punctuation, symbols, and even whitespace will satisfy it.

With the strong password schema in place, we can compose the three request schemas the API needs:

const registerSchema = z.object({
    username: z.string().min(1, "Username is required").trim(),
    password: strongPasswordSchema,
});

const loginSchema = z.object({
    username: z.string().min(1, "Username is required").trim(),
    password: z.string().min(1, "Password is required"),
});

const changePasswordSchema = z.object({
    current_password: z.string().min(1, "Current password is required"),
    new_password: strongPasswordSchema,
});

Registration uses the strict rules on password because that's a brand new password being committed to the database. The change-password schema uses the strict rules on new_password for the same reason.

Notice that loginSchema and the current_password field in changePasswordSchema do not use the strict rules. That's deliberate. The strength rules are about what we are willing to store, not what we are willing to compare against. A user who registered before you tightened your password policy still needs to be able to log in with their old, weak password so they can rotate to a stronger one. If the strong rules applied to the login flow, you would lock those users out permanently.

Store Passwords Safely in MongoDB with Password Change History

Now we get to the part this whole tutorial is named after. The shape of a user document is going to look something like this:

{
    "_id": "ObjectId(...)",
    "username": "nraboy",
    "password": "BCRYPT_HASH",
    "password_history": [
        { "password": "BCRYPT_HASH", "created_at": "ISODate(...)" }
    ],
    "created_at": "ISODate(...)",
    "updated_at": "ISODate(...)"
}

The current password hash lives at the root of the document. That's the value we compare against during login, and it is the one we update during a password change. Alongside it, we keep a password_history array where each entry is a previous hash with the timestamp it was set. When a user first registers, the history starts with a single entry that mirrors the root password. Every password change appends another entry. To check if a candidate password has been used before, we walk the history and compare against each entry.

Let's start building routes/passwords.ts. The top of the file pulls in the dependencies, defines the data shapes, and declares a salt rounds constant:

import { Router, Request, Response } from "express";
import { z } from "zod";
import bcrypt from "bcrypt";
import { getDb } from "../libs/mongodb";
import { MongoServerError } from "mongodb";
import { validate } from "../libs/validate";

const SALT_ROUNDS = 12;

type PasswordHistoryEntry = {
    password: string;
    created_at: Date;
};

type UserDocument = {
    username: string;
    password: string;
    password_history: PasswordHistoryEntry[];
    created_at: Date;
    updated_at: Date;
};

The two types describe the user document and a single history entry. We're using them as generic parameters on db.collection<UserDocument>("users") calls below so TypeScript can give us reasonable autocomplete when we read fields from the results.

Twelve salt rounds is a sensible default for bcrypt. It is slow enough to make brute-force attacks expensive and fast enough that a legitimate login does not feel sluggish on modern hardware.

Next, define three small helper functions for working with bcrypt:

async function hashPassword(plaintext: string): Promise<string> {
    return bcrypt.hash(plaintext, SALT_ROUNDS);
}

async function verifyPassword(plaintext: string, hash: string): Promise<boolean> {
    return bcrypt.compare(plaintext, hash);
}

async function isPasswordInHistory(
    plaintext: string,
    history: PasswordHistoryEntry[]
): Promise<boolean> {
    for (const entry of history) {
        const match = await bcrypt.compare(plaintext, entry.password);
        if (match) return true;
    }
    return false;
}

The first two are thin wrappers around the bcrypt API. They exist mostly to keep the route handlers readable.

The third one, isPasswordInHistory, is where the history check actually happens. It loops through every entry in the array and calls bcrypt.compare against each one. You cannot just hash the candidate password and look for an exact string match because bcrypt mixes a fresh random salt into every hash. Two calls to bcrypt.hash("Nic12345$", 12) will produce two different hash strings. The only way to know if a plaintext matches a stored hash is to feed both into bcrypt.compare.

The loop short-circuits the moment it finds a match, so in the common case where the user picks something genuinely new, the cost is proportional to the size of the history. For typical history sizes of five to ten entries, that overhead is negligible.

With the schemas from the previous section and the helpers above, we can start defining routes. Initialize the router:

const router = Router();

And start with the demo endpoint that lists every user document. This one only exists so you can poke at the database during development:

router.get("/", async (_req: Request, res: Response): Promise<void> => {
    try {
        const db = await getDb();
        const users = await db.collection<UserDocument>("users").find({}).toArray();
        res.status(200).json(users);
    } catch (err) {
        console.error(err);
        res.status(500).json({ error: "Internal server error" });
    }
});

This endpoint returns the full user document, including the bcrypt hash and the entire history array. You would never ship something like this in production. It is genuinely useful while you're developing, though, because it lets you see exactly what the password history looks like as it grows.

The registration endpoint is where new users get created:

router.post("/", validate(registerSchema), async (req: Request, res: Response): Promise<void> => {
    const { username, password } = req.body;

    try {
        const db = await getDb();
        const hashed = await hashPassword(password);
        const now = new Date();

        await db.collection("users").insertOne({
            username,
            password: hashed,
            password_history: [{ password: hashed, created_at: now }],
            created_at: now,
            updated_at: now,
        });

        res.status(201).json({ message: "User created successfully" });
    } catch (err) {
        if (err instanceof MongoServerError && err.code === 11000) {
            res.status(409).json({ error: "Username already exists" });
            return;
        }
        console.error(err);
        res.status(500).json({ error: "Internal server error" });
    }
});

There are a couple of things worth pointing out here. First, validate(registerSchema) runs before the handler. If validation fails, the handler never executes, so by the time we destructure username and password from req.body, we already know they're well-formed.

Second, the new document seeds password_history with a single entry that contains the same hash we just stored at the root. That way, the very first password is part of the history from day one. If we did not do that, the user could "rotate" their password back to the original on their first password change, which is exactly the behavior we're trying to prevent.

Third, instead of running a findOne to see if the username is taken and then doing an insertOne, we just attempt the insert and catch MongoServerError with code 11000. That's the duplicate-key error code, and it is the cleanest way to enforce uniqueness because MongoDB itself is the only thing that needs to coordinate. There is no window between the check and the insert during which two requests could both decide the username is free.

The login endpoint is much simpler:

router.post("/login", validate(loginSchema), async (req: Request, res: Response): Promise<void> => {
    const { username, password } = req.body;

    try {
        const db = await getDb();
        const users = db.collection<UserDocument>("users");

        const user = await users.findOne({ username });
        if (!user) {
            res.status(401).json({ error: "Invalid credentials" });
            return;
        }

        const valid = await verifyPassword(password, user.password);
        if (!valid) {
            res.status(401).json({ error: "Invalid credentials" });
            return;
        }

        res.status(200).json({ message: "Login successful" });
    } catch (err) {
        console.error(err);
        res.status(500).json({ error: "Internal server error" });
    }
});

Notice that both failure cases return the same 401 with the same generic message. There is a temptation to send back "User not found" when the lookup misses and "Wrong password" when the comparison fails, but doing so leaks information. An attacker who can tell those two states apart can enumerate which usernames exist in your system, which is the first half of a credential stuffing attack. Returning a single opaque error keeps both branches indistinguishable.

This is also why a real production version would want some kind of rate limiting on this endpoint. We're not adding that here, but it is worth keeping in mind.

The change-password endpoint is the one where the history check actually pays off:

router.put("/:username/password", validate(changePasswordSchema), async (req: Request, res: Response): Promise<void> => {
    const { username } = req.params;
    const { current_password, new_password } = req.body;

    try {
        const db = await getDb();
        const users = db.collection<UserDocument>("users");

        const user = await users.findOne({ username });
        if (!user) {
            res.status(404).json({ error: "User not found" });
            return;
        }

        const currentValid = await verifyPassword(current_password, user.password);
        if (!currentValid) {
            res.status(401).json({ error: "Current password is incorrect" });
            return;
        }

        const history = user.password_history ?? [];

        const alreadyUsed = await isPasswordInHistory(new_password, history);
        if (alreadyUsed) {
            res.status(409).json({ error: "New password has been used before. Please choose a different password." });
            return;
        }

        const hashed = await hashPassword(new_password);
        const now = new Date();

        await users.updateOne(
            { username },
            {
                $set: {
                    password: hashed,
                    updated_at: now,
                },
                $push: {
                    password_history: { password: hashed, created_at: now },
                },
            }
        );

        res.status(200).json({ message: "Password updated successfully" });
    } catch (err) {
        console.error(err);
        res.status(500).json({ error: "Internal server error" });
    }
});

The handler runs through four checks in order. First, we look up the user by the username from the route parameter, returning a 404 if the user does not exist. Then we verify the supplied current_password against the stored hash. A mismatch returns a 401, which is appropriate because the caller is, in effect, failing to authenticate.

After that, we run the history check:

router.put("/:username/password", validate(changePasswordSchema), async (req: Request, res: Response): Promise<void> => {
    // ... user lookup and current password verification above

    const history = user.password_history ?? [];

    const alreadyUsed = await isPasswordInHistory(new_password, history);
    if (alreadyUsed) {
        res.status(409).json({ error: "New password has been used before. Please choose a different password." });
        return;
    }

    // ... update logic below
});

The nullish coalescing operator gives us a safe default for any users that might somehow be missing the password_history field, though in practice, the registration handler always sets it. If the candidate password matches any entry in the history, we send back a 409 and explain the situation to the caller.

If we make it past all four checks, the update itself uses two operators in one call:

router.put("/:username/password", validate(changePasswordSchema), async (req: Request, res: Response): Promise<void> => {
    // ... lookup, verification, and history check above

    const hashed = await hashPassword(new_password);
    const now = new Date();

    await users.updateOne(
        { username },
        {
            $set: {
                password: hashed,
                updated_at: now,
            },
            $push: {
                password_history: { password: hashed, created_at: now },
            },
        }
    );

    res.status(200).json({ message: "Password updated successfully" });
});

The $set operator overwrites the root password field with the new hash and bumps updated_at. At the same time, $push appends the new hash to the password_history array. Because both operators are part of the same update, the change is atomic from MongoDB's perspective. The current password and the history can never disagree about the user's most recent password.

Finally, the bottom of the file exports the router so main.ts can mount it:

export default router;

That completes the route layer. Time to try it out.

Testing the API Endpoints with cURL

We'll exercise the API from the command line with curl. The repository also includes a Bruno collection in the bruno/ directory if you'd rather click through requests in a GUI, but curl is faster to demonstrate in a tutorial.

Start the server in development mode:

npm run dev

You should see Server running on http://localhost:3000 in the terminal. Leave it running and open a second terminal for the requests.

First, register a new user:

curl -s -X POST http://localhost:3000/users \
    -H "Content-Type: application/json" \
    -d '{"username":"nraboy","password":"Nic12345$"}'

A successful registration responds with a 201 and {"message":"User created successfully"}. If you try the exact same request again, you'll get a 409 because the unique index on username rejects the duplicate.

To prove the validation middleware is doing its job, try sending a weak password:

curl -s -X POST http://localhost:3000/users \
    -H "Content-Type: application/json" \
    -d '{"username":"weakuser","password":"password"}'

The response is a 400 with a per-field map of the rules that were broken. No insert ever reaches MongoDB.

Now try logging in with the user we just created:

curl -s -X POST http://localhost:3000/users/login \
    -H "Content-Type: application/json" \
    -d '{"username":"nraboy","password":"Nic12345$"}'

A correct password returns {"message":"Login successful"}. Get one character wrong, and you'll see {"error":"Invalid credentials"} with a 401.

The interesting endpoint is the password change. Rotate to a brand new password:

curl -s -X PUT http://localhost:3000/users/nraboy/password \
    -H "Content-Type: application/json" \
    -d '{"current_password":"Nic12345$","new_password":"Nic67890$"}'

That responds with a 200 and {"message":"Password updated successfully"}. Now try to rotate back to the original password:

curl -s -X PUT http://localhost:3000/users/nraboy/password \
    -H "Content-Type: application/json" \
    -d '{"current_password":"Nic67890$","new_password":"Nic12345$"}'

This time the response is a 409 and {"error":"New password has been used before. Please choose a different password."}. The history check did exactly what we wanted.

You can verify what's actually in the database by hitting the demo list endpoint:

curl -s http://localhost:3000/users

The response includes the password_history array with two entries, one for the original password and one for the rotation, each with its own timestamp and bcrypt hash. Two different hashes for the same logical password value, which is the whole point of using bcrypt.compare rather than string equality during the history check.

Conclusion

You just saw how to combine TypeScript, Express Framework, Zod, and the MongoDB Node.js driver to enforce password strength and prevent password reuse. Zod schemas plugged into Express as middleware handle the strength rules, bcrypt handles the hashing, and a password_history array on each user document keeps a running record of every hash that user has ever used. The singleton MongoDB client keeps the connection layer simple, and a unique index on username lets MongoDB itself enforce uniqueness without us having to coordinate it in application code.

What we built is good for a demo, but a few things would need to change before this could be a production system. There is no rate limiting on the login or password change endpoints, both of which are attractive targets for brute-force attempts. There is no session or token mechanism, so right now a successful login just sends back a message. The GET /users endpoint exposes every hash in the database and would need to be removed entirely. And depending on your compliance requirements, you may want to cap the size of password_history so it does not grow without bound.

As a next step, try adding a password expiration field that forces a rotation after some number of days, or capping password_history so only the last ten entries are retained.

The full source for this project is available on GitHub.

Whistleblower for Database: Set up an internal informant who exposes performance

MongoDB Guests — Thu, 07 May 2026 10:44:20 +0000

This tutorial was written by Darshan Jayarama.

As DBAs, waking up at 2 AM because production is down or slow is a nightmare, and none of us like it. However, we all know that it is not instant karma; it is like a pressure cooker where the database has reached its breaking point. Database whistleblowers constantly give signals that we ignore before it explodes into a Production Issue.

Whistleblower 1: Performance Informant

High CPU utilization?

CPU utilization is not just an alert, it is a future prediction with regard to the database. When you get constant High CPU utilization alerts, think of it as a Whistleblower reviewing the database logs, metrics, and processes that are consuming CPU. If possible, shift those processes to a different server and let your Database breathe without pressure. Ignoring high CPU utilization can lead to high latency.

Query <5 seconds?

A query that was blazing fast is now causing a slow query alert since the last data load. It is a clear indication of a Missing or Unoptimized index. If an index is missing, create one. If unoptimized, then redesign a better index. Still facing the same? It needs a deeper and wider investigation.

Cache hit ratio <80%?

A lower cache hit ratio means higher disk reads, which tends to cause high IO Wait, and an Increased disk queue depth. It can happen if your query is unoptimized (we had addressed this earlier), or if lower memory/cache is assigned. Increase the memory if there is nothing pending in Database tuning.

Whistleblower 2: Cost Informant

Storage >80%:

Set your whistleblower when storage utilization hits >80% its your duty now to analyze storage to archive any old logs, old data, or remove unwanted processes that consume your space.

IOPS > Budget?

We usually get an alert for a high document-scanned-to-returned ratio, which indicates that queries are performing more IO than required. Optimize them. Reduce IOPS consumption by rewriting queries, redesigning the schema, and implementing a better Index strategy.

Unexpected auto scale?

MongoDB Atlas has a feature that monitors system load and automatically scales up or down as needed. Implement the solution. In case of a scale down, it does require 24-hour monitoring, but if you are sure there are no more surprises, you can scale down yourself.

Whistleblower 3: Reliability Informant

Replication lag >10 Seconds?

Investigate where it is lagging. Is it in primary? Must be loaded from writes, which lowers the read request priority. It usually triggers the built-in flow-control logic. If secondary is the issue, then is it at writing or reading? A detailed understanding of flow can help resolve this data durability. Ask developers to set writeConcern to match the application's requirements.

Oplog <24Hours?

A lower oplog size means lower retention of replication events. Set at least 24 hours, resize as per the application write load minOplogRetention.

Connection >80%?

Each connection can consume 1 MB of memory; the more connections, the more memory is consumed. Maintain connection hygiene, use a connection pool, and close connections when no longer needed.

Conclusion: Your production incident is one missed alert away; hire your whistleblower. Choose: Pressure cooker explosion → OR → Proactive DBA legend?

Vacuum Seal Your MongoDB: Cuts Cost Down 50% Today

MongoDB Guests — Tue, 05 May 2026 10:10:10 +0000

This article was written by Darshan Jayarama.

I was recently packing for a short trip, and I personally hate carrying 2–3 pieces of luggage. One for my laptop, another for clothes. So a person like me has definitely used a “vacuum sealer,” allowing me to pack as many clothes as I want.

During the trip, I had an idea. Which vacuum sealers do we have to help store efficiently and reduce billing? Here, I have jotted down some…

Online Archive

Does your dataset contain cold data? For a supply chain company, data older than three months is considered infrequently accessed. For an e-commerce website or a courier company, data that has already been delivered is accessed by users very rarely. We can group them using a query, such as lastAccess > 30, and create the Online archive store in cloud object storage. Worried about how to access this when needed? Relax — it’s all covered, there are no access restrictions. Storage costs drop from $4/GB/Month to $0.02/GB/Month.

Table Compact

In a write-intensive application where you frequently delete data, storage watermarks always stay high. You can run the below command to check:

db.COLLNAME.stats().freeStorageSize

The above command outputS how many bytes are available to be reused. Even though wiredTiger uses this space organically over time, if the storage utilization is effectively essential, you can run compact on the collection.

We can run compact with the following syntax:

use mydb // switch to the database 
db.runCommand( { compact: "myCollection" } ) // mention the collection name

As it also has many other options, I would recommend visiting the official documentation of compact.

Resync Secondary:

Running compact is fine if you’ve got just a handful of collections. But when you’re dealing with many collections with high watermarks, especially large ones, running compact isn't ideal.

You can resync a member in rolling fashion so that all members can release that space back to the operating system and utilize the storage efficiently.

TTL Index:

So far, we have discussed the reactive approach for storage utilization. We are preparing the plan once the storage hits the ceiling. But do we have any such proactive approach? Yes, here is where TTL comes into play. If you believe the data is no longer needed after a certain time, set the TTL index to expire, and the record will be deleted.

This approach will be useful for the application that generates large logs. Set the TTL index for 3 days, and the log entry will be removed from the collection.

Query Optimization:

Being a query enthusiast, I circle back to query optimization, as a single bad query can eat up your IOPS, Network, and Compute.

Tune the query.
Create a compound index covering the query and the projection
Reduce the network round-trip and IOPS
If possible, precompute the value and store it in a collection.

Don't know where to start? Use/Abuse the Atlas Performance Advisor for index recommendations, inefficient queries, and inefficient schema advice. Or visit my previous blogs:

Flex Cluster:

Planning to perform some dev/API test? Go with the Flex cluster rather than the M10+ cluster for non-prod deployment. Same API, essentially Pause/Resume when not needed.

Conclusion:

MongoDB gives tons of storage-saving options — but only smart DBAs use them. Your cluster is bleeding $$$ right now. Online Archive, TTL indexes, compaction, proper sizing = 60% instant savings.

Pick 3 techniques → Deploy → Watch bill shrink. Simple as that. 💰

MongoDB Vector Search in Laravel: Finding the Unqueryable

Hubert Nguyen — Thu, 30 Apr 2026 19:57:50 +0000

Simple, keyword-based database queries are often inadequate for user searches because they struggle with complexities such as synonyms, slang, and relevance judgments. They potentially also suffer from slow performance on large datasets due to inefficient indexing methods. Consequently, these basic queries fail to provide users with a helpful, relevant, or nuanced list of results, leading to a less-than-ideal user experience.

Vector search is fundamentally better than basic keyword database queries because it searches based on semantic meaning rather than exact text matches, and it is built to scale efficiently.

A more comprehensive explanation of vector search is out of the scope of this article, but here's a quick overview to establish a baseline: Vector search is a technique that uses numerical representations, called vectors or embeddings, to find items that are semantically similar to a query, meaning you find things based on their meaning, not the keywords used to describe them.

The heavy lifting of creating these dense, high-dimensional vectors from text, images, or other data is done by existing embedding models. Vector search works by calculating the distance or similarity between the query's vector and the vectors in a database, quickly returning the most relevant items.

If you want to know more about the vector search concepts, I recommend watching our videos on vectors and embedding fundamentals and the future of data querying, or visit MongoDB's resources for a more thorough explanation of vector search.

Laravel vector search implementation

In this article, we'll use a GitHub code repository to illustrate how MongoDB Vector Search works in Laravel. I created a tag, as the repo will evolve in the future.

The repo's README.md has more information and is structured to mirror the tutorial's natural progression: Each section includes the "why" behind configuration choices, example commands with expected outputs, and a troubleshooting guide that explains what went wrong and how to fix it.

Prerequisites to run the repo

Basic understanding of Vector Search
Start a free MongoDB Atlas cluster with the mflix_sample database loaded
- Here's how to start a free cluster (in the "Atlas UI" tab) and how to load the sample databases.
A free Voyage AI API key
A functioning PHP/Laravel development environment. We recommend using our pre-built container environment on GitHub Codespaces (free for individual accounts).

Voyage AI was recently acquired by MongoDB. Its models have been recognized as some of the highest-performing by industry-leading benchmarks like the Hugging Face MTEB Leaderboard. That said, MongoDB's Vector Search is compatible with nearly all embedding models on the market, so you have ample control and choices for your specific use case.

Connect the Laravel app to the MongoDB database

If you haven't connected to MongoDB from Laravel before, we have a more detailed tutorial on how to build a back-end service with Laravel and MongoDB. In this article, we'll emphasise only the main points related to using MongoDB Vector Search in Laravel.

At this point, we'll assume that your MongoDB Atlas cluster is running and that you have loaded the sample data, especially the sample_mflix database, which we will be using. You can use the MongoDB Atlas GUI to do that (Database > Clusters > Browse Collections). Alternatively, there's MongoDB Compass, a native app that offers a more responsive and user-friendly experience.

The sample_mflix database contains two movie collections, "movies" and "embedded_movies." We are going to work on the "movies" collection as it does not contain vector embeddings. Our Laravel app will create the embeddings.

Connect to the database

Using the code repo, let's first connect to the MongoDB database.

Cluster network access: Make sure your current IP address is allowed through the cluster's firewall by adding it to the allowed list. If you are working on public WiFi (hotel, convention center, airport, etc.) adding the IP won't be enough and you may need to allow all IPs to have access, which is not recommended for security. Follow the instructions on the official documentation page (jump to "Add IP Access List Entries" and choose the "Atlas GUI" tab).

Connect from Laravel: Create an .env file, based on the .env.example file, and look for DB_CONNECTION=mongodb. Right below that, you have to update the DB_DSN entry with the actual connection string of your live cluster that includes username and password. Here's a tutorial that shows how to get the connection string in Atlas (jump to "In Atlas, go to the Clusters page for your project").

Our CodeSpaces environment will run the three commands below at startup. If you prefer your own setup, don't forget to initialize the repo with the commands below:

# create a new .env file
cp .env.example .env

# download libraries required by our app
composer install

# generate the keys for this app
php artisan key:generate

In the .env file, replace the sample DB_DSN MongoDB connection string with your own. Now would be a good time to add the Voyage AI API key as well.

DB_CONNECTION=mongodb
DB_DSN=mongodb+srv://USERNAME:PASSWORD@cluster.mongodb.net/sample_mflix?retryWrites=true&w=majority
DB_DATABASE=sample_mflix

VOYAGE_AI_API_KEY=YOUR_API_KEY_HERE

Note that MongoDB's schema flexibility allows us to remain migration-free for now, so we won't have to execute "php artisan migrate". Once your credentials are in, run this command to launch the app. Locally, it runs at http://localhost:8000, but it may be different depending on how you configured your PHP/Laravel development environment. Adapt the subsequent URL references to your situation.

In the Codespaces environment, you can find the URL by hovering above the "globe" icon in the Ports tab.

We're going to build some API endpoints, so in CodeSpaces, port 80 is made "public" to facilitate access. If you see a warning message, just click on Continue.

php artisan serve

There are some API endpoints that have been created for testing the app, and the connection to MongoDB.

Remember, in Codespaces, the URL format is {friendly-name}-{random-hash}-{port}.app.github.dev and can be obtained in the Ports tab.

# returns {"response":"hello world"} if the app is up and running
curl http://localhost:8000/api/hello

# returns {"status":"success","connection":"MongoDB connection successful"...
curl http://localhost:8000/api/mongodb-test

If both API calls are successful, our connection is solid, and we are ready for the next step: Use Vector Search!

3 steps to your MongoDB Vector Search in Laravel

If your data is already in MongoDB, performing a vector search takes only three steps, and we'll show you in detail below.

Step 1: Generate vector embeddings for your data

Before anything else, we need to have vectors if we want to search based on them! We will use an embedding model from Voyage AI to generate embeddings via their API.

The Voyage AI API access is implemented as a service (in app/Services/VoyageAIService.php), and the most interesting function is generateEmbeddings(), which takes an array of text inputs and returns the model's vector representations for those texts.

This service connects to the Voyage AI API via a REST API call in the makeRequest() function. If successful, the Voyage API returns the following for each datapoint:

vector embedding
embedding model name
number of dimensions
number of tokens used by the model

// makes a request to the external vector embedding model
$response = $this->makeRequest($texts);

if ($response['success']) {
    $embeddings = $response['data']['data'] ?? [];

    return [
        'success' => true,
        'embeddings' => $embeddings,
        'count' => count($embeddings),
        'usage' => $response['data']['usage'] ?? null
    ];
}

return $response;

Vector embeddings for our database records tend to be built at a low frequency: at the initial model selection, when the vectorized data changes, or if there's a new model selection. We've added a console command in app/Console/Commands/GenerateEmbeddings. There's also a command to delete the embeddings in the same directory.

From the terminal, you can generate embeddings for the data, by using this command:

php artisan embeddings:generate

After using it, you can return to the MongoDB Atlas GUI and look at your documents. You should see a new "embedding" field, with an array of 512 floats. That is the vector embedding for that document. We now have documents that are ready to be indexed, well done!

Note: By default, there's a hard limit of 100 total embeddings created by this command. This is so you don't use too many tokens, as the database has tens of thousands of records. Comment out the blocker or raise the pre-set value if you want to experiment with more.

Step 2: Create a vector index

Vector search requires searching an index—a vector index, to be precise. The interesting code is shown below, and you'll have to provide three critical pieces of information:

path: the field name containing the vector data
numDimensions: the number of dimensions of the vector, how many numbers are in the vector array
similarity: the similarity function to be used for the distance calculation

path is something you decide as a developer, and you can name that field anything you want. numDimensions and similarity should be provided by the embedding model creators, so check their documentation. In our case, we're using the Voyage AI voyage-3-lite model, and the embeddings documentation shows 512 dimensions.

To create a vector search index, use the CLI command that is implemented in /app/Console/Commands/CreateVectorIndex.php using the createSearchIndex() function.

$connection = DB::connection('mongodb');
$collection = $connection->getCollection($collectionName);
// ...
$result = $collection->createSearchIndex(
    [
        'fields' => [
            [
                'type' => 'vector',
                'path' => config('vector.field_path'),
                'numDimensions' => $vectorDimensions,
                'similarity' => $vectorSimilarity
            ]
        ]
    ],
    [
        'name' => $indexName,
        'type' => 'vectorSearch'
    ]
);

php artisan vector:create-index

If there's already a vector index, you can delete it and create a new one, using:

php artisan vector:create-index --force

After that, you can check that the index has been created and is ready. In our code repo, the initial index build appears almost instantaneous because we have only 100 vector embeddings to index. However, a database with a few thousand vectors would take some seconds, and the index build time may scale linearly from there. We're suggesting using a free instance here, but there are ways to greatly scale in production, including with the dedicated Search Nodes.

For that, we've added a CLI command you can run (implemented in /app/Console/Commands/CheckVectorIndex.php). That function lists all the indexes for a specific collection and checks for a naming convention in this case.

$indexes = iterator_to_array($collection->listSearchIndexes());

// Look for the specific vector index
$vectorIndex = null;
foreach ($indexes as $index) {
    if ($index['name'] === $indexName) {
        $vectorIndex = $index;
        break;
    }
}

php artisan vector:check-index

Why we check: If you run a vector search query without creating the index first, you won't get an error. Your query executes successfully, the API returns a 200 status code, but the results array is empty. This can be confusing for developers who are new to vector search.

Alternatively, you can also look in the Atlas GUI. Go to the "movies" collection (Database > Clusters > Browse Collections). Select the "sample_mflix.movies" collection, and click on the "Search Indexes" tab. You should see your index as "ready."

Step 3: Perform a query

With a vector index ready to be searched, the last thing we need is to launch a search query! This involves two main steps. First, we need to receive a query from a user, and for that, there's an /api/movie-search-vector API endpoint.

The most interesting part of that code is the text query vector embedding generation and the vector search query itself:

// $query is the user text input
// $result is a multi-dimensions vector
$result = $voyageAI->generateEmbeddings([$query]);

if (!$result['success']) {
    return response()->json([
        'error' => 'Failed to generate query embedding',
        'message' => $result['error']
    ], 500);
}

// Voyage AI returns an array of vectors
// because we use a batch embedding function
$queryVector = $result['embeddings'][0]['embedding'];

// Perform vector search using Eloquent
$results = Movie::vectorSearch(
    index: config('vector.index.name'),
    path: config('vector.field_path'),
    queryVector: $queryVector,
    limit: config('vector.search.limit'), // # of ranked results returned
    numCandidates: config('vector.search.num_candidates')
);

MongoDB's vector search uses two parameters that work together: numCandidates controls how many approximate matches (the candidate pool) MongoDB examines during the HNSW vector index data structure traversal in memory. Limit controls how many final results are returned to the user. The recommended minimum ratio is 20:1, so if you want 10 results (limit: 10), MongoDB should search through 200 candidates (numCandidates: 200) to ensure it finds the best matches.

The endpoint expects a string input as a URL parameter and you can use CURL as follows:

curl -X POST http://localhost:8000/api/movie-search-vector \
  -H "Content-Type: application/json" \
  -d '{"query": "outlaws on the run from law enforcement"}'

This returns a response like the one below. We've chosen to return only a few fields, like title and plot, to make things more readable and allow you to check how relevant the search is.

{
  "query": "outlaws on the run from law enforcement",
  "results": [
    {
      "_id": {"$oid": "573a1390f29313caabcd42e8"},
      "title": "The Great Train Robbery",
      "plot": "A group of bandits stage a brazen train hold-up...",
      "score": 0.8234567
    }
  ],
  "count": 10,
  "embedding_model": "voyage-3-lite",
  "vector_dimensions": 512
}

The "score" field is a very interesting piece of metadata from the vector search engine. It allows you to see the degree of relevance of the results.

Vector search will always return results based on the calculated similarity score, even if the closest match is conceptually unrelated or irrelevant to your query. A very high score (for distance metrics) or a very low score (for similarity metrics) indicates that the returned item is likely not a good match, even though it was the "closest" mathematically.

Now that our vector search is functional, you can try more searches. Remember, vector search can find things based on semantics (meaning) and you do not need to know the exact keywords present in the database records.

Try some of these and see if the results are relevant, and there are more suggestions in the repo's README.md file in the "Semantic Search Query Suggestions" section—have fun!

"prehistoric creature comes alive"
"rags to riches criminal empire"
"sacrifice for a greater cause"

Use cases

MongoDB Vector Search provides a necessary architectural layer for building modern, intelligent Laravel applications. Think of it as replacing flaky, slow, text-based or regex-based database query lookups or complex fuzzy search packages with a specialized search you need to install, manage, secure, and learn.

By sending data to an embedding service and storing the resulting vectors, you can instantly power features that were previously impossible or performance nightmares: semantically relevant search across millions of records, context-aware chatbot retrieval (RAG), and real-time personalized recommendations using a simple, powerful search query via Eloquent.

A diverse range of apps can benefit from vector search, including document search, multimodal (text/image/video) search, recommendations ("more like this"), "similar past events" in logs or telemetry, and more!

Common questions

What happens if the model is updated, or if I change the embedding model?

Generally, if you use a different (updated or new) embedding model, you should expect to have to regenerate the embeddings for all your data. Even small changes in a model can create shifts in the vector space, and this could affect the outcome of your search queries if using previously generated vectors. For model changes, many people run two sets of embeddings and search indexes during the transition period.

Will vector search introduce high latency (slow down my queries) compared to my existing MySQL/full-text search?

At runtime, the vector search query latency will often be faster than a text or regex search on an equally large dataset. It is common for vector queries to run at <200ms, although this number can be affected by the number of candidates selected.

People using vector search in production with very large datasets more often run into cost issues, as the vector index data structure needs to fit in RAM. To mitigate cost (and improve speed), look at using vector quantization.

Conclusion

MongoDB Vector Search is a fundamental upgrade over traditional text query methods because it operates on the principle of semantic meaning rather than keyword matching. This is essential for finding results that would be difficult or impossible to locate with standard database queries, especially when a user's query employs abstract themes, synonyms, or a non-specialized lexicon. By understanding the intent behind the query, vector search can surface the most relevant information, effectively finding what we've called the "unqueryable."

Vector Search is readily available within MongoDB, even in MongoDB Community Edition. This translates directly into flexibility: Leverage MongoDB for unmatched ease of use as your primary database, or deploy it as a secondary, highly scalable vector search database. One thing to consider is that MongoDB also offers full-text search, which can be combined with vector search to unlock the power of hybrid search—something that many alternatives don't have.

MongoDB lets you choose the architecture that best fits your needs, all while keeping advanced vector capabilities native.

My first “local” Vector Search: MongoDB community edition

MongoDB Guests — Thu, 30 Apr 2026 10:56:11 +0000

This article was written by Darshan Jayarama.

Ever since I received an email about the vector search auto-Embedding feature released in MongoDB community edition, my palms have been itching to test it out. And now.. I’m happily writing this blog after witnessing its power firsthand.

Vector Search is one of the most admired, powerful, and amazing products of MongoDB. I felt pity for MongoDB Community/On-prem customers, as they were unable to use this because Vector search and Atlas search were only available in MongoDB Atlas. But this is no longer the case!

How did I test this?

I am sure by now you might be thinking, “Enough of this product endorsement, show me the code!”

I’m getting there :). Initially, I followed what the MongoDB documentation suggests, but I struggled a lot as there is a disconnection when setting up the Docker containers:

As per the Installation documentation, select Docker as your operating system.
When the MongoDB community server starts, it starts with hostname as mongod.search-community:27017. Make a note of this.
When we asked to start the MongoDB Search container, in that yml file, it sets the syncsource as mongot-community.search-community:27017
- Don't just copy/paste the yml from there. Make sure you are setting the correct syncSource, which should be the Community server container name mongod.search-community:27017. To ensure your container hostname, run db.isMaster().me, it should print you the hostname.
Next, there is mongot username and password entry that should be made in the yml file. Whileyou are creating a user, the instructions specify mongot user as mongot, but in the yml file, it is mentioned as mongotUser. If you created it with another name, correct the section.

After these corrections, you can see the 2 containers running happily (I literally jumped out of my chair over joy). Next, connect to the mongod using mongosh to test the power.

I created an index on the favorite movies collection, plot field:

db.movies.createSearchIndex("vector_index", "vectorSearch", {
 "fields": [
   {
     "type": "autoEmbed",
     "modality": "text",
     "path": "plot",
     "model": "voyage-4"
   },
   {
     "type": "filter",
     "path": "year"
   }
 ]
})

You can use voyage-4-lite, but as per the voyage usage stats, both voyage-4 and voyage-4-lite both have the same 10000TPM 3RPM limitation. My preferred choice is voyage-4.

Then I wanted to see the results based on the context. So I ran the below query:

db.movies.aggregate([
 {
   "$vectorSearch": {
     "index": "vector_index",
     "path": "plot",
     "query": {
       "text": "bullied boy learns karate"
     },"model": "voyage-4-lite" ,
           "numCandidates": 10000,
     "limit": 10
   }
 },
 {
   "$project": {
     "_id": 0,
     "title": 1,
     "year":1,
     "plot": 1,
     "score": { $meta: "vectorSearchScore" }
   }
 }
])

The above query should retrieve movies with context of “bullied boy learns karate” (here we expect ‘karate-kid’):

The answers amused me:

[
 {
   plot: 'A love-struck weakling must pretend to be boxer in order to gain respect from the family of the girl he loves.',
   title: 'Battling Butler',
   year: 1926,
   score: 0.6800339221954346
 },
 {
   plot: 'Two young brothers become the leaders of a gang of kids in their neighborhood. Their father is an office clerk who tries for advancement by playing up his boss. When the boys visit the boss...',
   title: 'I Was Born, But...',
   year: 1932,
   score: 0.6692402362823486
 },
 {
   plot: 'A living puppet, with the help of a cricket as his conscience, must prove himself worthy to become a real boy.',
   title: 'Pinocchio',
   year: 1940,
   score: 0.6628834009170532
 },
 {
   plot: 'An idealistic adolescent, suffering under the thumb of a sadistic schoolmaster, falls in love with a loose girl who is bullied and tormented by another lover.',
   title: 'Torment',
   year: 1944,
   score: 0.6620646715164185
 },
 {
   plot: `Against all odds Father Flanagan starts "Boys' Town" after hearing a convict's story. Whitey Marsh comes there. He runs away but, hungry, returns. He runs away again but, when friend Pee ...`,
   title: 'Boys Town',
   year: 1938,
   score: 0.6570459604263306
 },
 {
   plot: 'When three thuggish men are responsible for the death of his father and the crippling of his brother, young David must choose between supporting his family or risking his life and exacting vengeance.',
   title: "Tol'able David",
   year: 1921,
   score: 0.6559526920318604
 },
 {
   plot: "Fight promoter Nick Donati grooms a bellhop as a future champ, but has second thoughts when the 'kid' falls for his sister.",
   title: 'Kid Galahad',
   year: 1937,
   score: 0.6525536179542542
 },
 {
   plot: 'In a repressive boarding school with rigid rules of behavior, four boys decide to rebel against the direction on a celebration day.',
   title: 'Zero for Conduct',
   year: 1933,
   score: 0.651709794998169
 },
 {
   plot: 'While at a ski lodge, Larry Blake sees instructor Karin Borg and decides to sign up for private lessons. The next thing he knows, she is Mrs. Blake. When he announces that he is going back ...',
   title: 'Two-Faced Woman',
   year: 1941,
   score: 0.6513179540634155
 },
 {
   plot: 'To reconcile with his girlfriend, a bookish college student tries to become an athlete.',
   title: 'College',
   year: 1927,
   score: 0.6459328532218933
 }
]

Even though the expected result was not there, all the results related to karate, boxing, or being bullied.

Below are the commands I have used to complete the setup of this.

Pulling docker images;

docker pull mongodb/mongodb-community-search:latest
docker pull mongodb/mongodb-community-server:latest

Create internal docker network to communicate

docker network create search-community

Starting community server container

echo '
net:
  port: 27017
  bindIpAll: true  # Equivalent to --bind_ip_all

replication:
  replSetName: rs0
systemLog:
  destination: file
  path: "/var/log/mongodb/mongod.log"
  logAppend: true

setParameter:
  searchIndexManagementHostAndPort: mongot-community.search-community:27028
  mongotHost: mongot-community.search-community:27028
  skipAuthenticationToSearchIndexManagementServer: false
  useGrpcForSearch: true

# Security configuration
security:
  authorization: enabled  # Equivalent to --auth
  keyFile: /keyfile' > mongod.conf

mkdir ./data/db

openssl rand -base64 756 > keyfile
chmod 400 keyfile

docker run -d --rm --name mongod -e MONGODB_INITDB_ROOT_USERNAME=root -e MONGODB_INITDB_ROOT_PASSWORD=rootpass -v ./mongod.conf:/etc/mongod.conf:ro -v ./data/db:/data/db -v ./keyfile:/keyfile -p 27017:27017 --network search-community mongodb/mongodb-community-server:latest --config /etc/mongod.conf

Initiate replication

mongosh -u root -p rootpass --eval 'rs.initiate(); sleep(10); rs.status()'
mongosh -u root -p rootpass --eval "db.getSiblingDB('admin').createUser(
 {
   user: 'mongot',
   pwd: 'mongotPassword',
   roles: ['searchCoordinator']
 }
)"

Load sample movies collection data
Downloaded movies from https://github.com/neelabalan/mongodb-sample-dataset/blob/main/sample_mflix/movies.json

mongoimport  -d sample_mflix -c movies /Users/darshanj/Downloads/movies.json -u root -p rootpass --authenticationDatabase admin

Prepare search process

mkdir mongot_data

Creating mongot config file

hostname=`mongosh -u root -p rootpass --eval "db.isMaster().me"`
cat << EOF > mongot.config
syncSource:
  replicaSet:
     hostAndPort: "$hostname" 
     username: "mongot"
     passwordFile: "/passwordFile"
     authSource: "admin"
     tls: false
     readPreference: primaryPreferred
storage:
  dataPath: "/data/mongot"
server:
  grpc:
     address: "mongot-community.search-community:27028"
     tls:
        mode: "disabled"
metrics:
  enabled: true
  address: "mongot-community.search-community:9946"
healthCheck:
  address: "mongot-community.search-community:8080"
logging:
  verbosity: INFO
embedding:
  queryKeyFile: /etc/mongot/voyage-api-query-key
  indexingKeyFile: /etc/mongot/voyage-api-indexing-key
  providerEndpoint: https://api.voyageai.com/v1/embeddings
  isAutoEmbeddingViewWriter: true

echo -n "mongotPassword" > passwordFile
chmod 400 passwordFile

Go to https://dashboard.voyageai.com/organization/api-keys API key, copy secret

printf "<your-voyage-api-query-key>" > voyage-api-query-key

Repeat one more time for index API. Go to https://dashboard.voyageai.com/organization/api-keys API key, copy secret

printf "<your-voyage-api-index-key>" > voyage-api-indexing-key

Starting MongoDB Search

docker run -d --rm --name mongot-community -v ./mongot_data:/data/mongot -v ./mongot.config:/mongot-community/config.default.yml -v ./passwordFile:/passwordFile:ro -v ./voyage-api-indexing-key:/etc/mongot/voyage-api-indexing-key:ro -v ./voyage-api-query-key:/etc/mongot/voyage-api-query-key:ro --network search-community -p 8080:8080 -p 9946:9946 mongodb/mongodb-community-search:latest --internalListAllIndexesForTesting=true

Check the docker status

docker ps

Now connect to the mongosh and create the index and run the test query

mongosh -u root -p rootpass

use sample_mflix
db.movies.createSearchIndex("vector_index", "vectorSearch", {
 "fields": [
   {
     "type": "autoEmbed",
     "modality": "text",
     "path": "plot",
     "model": "voyage-4"
   },
   {
     "type": "filter",
     "path": "year"
   }
 ]
})

db.movies.aggregate([
 {
   "$vectorSearch": {
     "index": "vector_index",
     "path": "plot",
     "query": {
       "text": "bullied boy learns karate"
     },"model": "voyage-4-lite" ,
           "numCandidates": 10000,
     "limit": 10
   }
 },
 {
   "$project": {
     "_id": 0,
     "title": 1,
     "year":1,
     "plot": 1,
     "score": { $meta: "vectorSearchScore" }
   }
 }
])

I know we can make one single Docker compose file to make it easier, but I prefer this way to make debugging easier, and the understanding of each step will be clearer.

Congratulations! You have successfully created your first local vector search.

PS: To be safe, delete the API keys when no longer needed, just like I did :).

Real-Time Fraud Detection in Java with Kafka Streams and Vector Similarity

Ricardo Mello — Wed, 29 Apr 2026 13:51:37 +0000

This content is based on a talk that my colleague Tim Kelly and I presented at DevNexus 2026 in Atlanta, one of the largest Java conferences in the world.

Imagine you’re at a supermarket or shopping online. You tap your card or click “Buy Now”. In that exact moment, your bank has to make a decision: is this transaction legitimate, or is it fraud? And it has to do that in milliseconds, and get it right. If it blocks a real transaction, it frustrates the customer. If it approves a fraudulent one, it leads to financial loss. So the question is: how do banks actually do this? What’s happening behind the scenes? In this article, we’ll explore a real-world approach to solving this problem using Java, Apache Kafka, Kafka Streams, and vector similarity with MongoDB.

What We’re Building

At a high level, a payment system works like this: a transaction is created, it goes through a validation step, and then a final decision is made to approve or decline:

Now, let’s zoom in on that validation step, because this is where fraud detection actually happens:

In our approach, this validation is not a single check but a pipeline composed of multiple stages working together:

Guardrails: Rule-based checks that catch obvious and well-known fraud patterns immediately, like impossible travel or unusual transaction velocity.
Vector Scoring: AI-powered similarity matching that compares the transaction against known patterns to detect more subtle or behavioral fraud.
Decision: The final verdict is based on the combined output of both layers: route the transaction to the approved_transactions collection or to the suspicious_transactions collection for further review or definitive blocking.

The decision flow is pretty straightforward.

If a transaction is flagged at any point, either by the guardrails or by the vector scoring stage, we treat it as suspicious and store it in the suspicious_transactions collection. From there, it can be reviewed, enriched with more context, or even blocked depending on the scenario.

On the other hand, if it passes both layers without any issues, we consider it safe and store it in the approved_transactions collection.

The Challenge: Millisecond Decisions with Resilience Under Failure

In fraud detection systems, especially in financial applications, speed is not just important, it is critical. Every transaction must be analyzed in real time, often under heavy load, with thousands of events happening every second. At the same time, the system cannot afford to break if a single component fails.

This leads to two non-negotiable constraints:

Decisions must be made in milliseconds
The system must remain resilient even when individual components fail

Before we get into how each layer works, it is important to understand how these constraints shape the entire architecture.

A natural first instinct is to build this using a simple synchronous architecture.

The payment service calls the fraud service, which then queries the database, and the flow continues to the notification service. This works fine at the beginning. It’s simple and easy to understand.

But the problem is that everything is tightly connected. One service depends on the next.

Now imagine the database takes a bit longer to respond.

That delay starts to affect the whole flow. The fraud service slows down, the payment service has to wait, and the notification service also gets delayed.

At scale, this becomes a real issue. One slow component can impact everything else.

So how do we break this dependency and make each service more independent?

This is where event-driven architecture comes in.

The Strategy: Decoupling the System with Event-Driven Architecture

Instead of having services call each other directly, we introduce an event log as the backbone of communication.

Apache Kafka allows the payment service to publish a transaction event to a topic, without knowing who will consume it. The fraud service, the notification service, and any other interested component can subscribe to that topic independently:

This fundamentally changes how the system behaves.

Services are no longer tightly coupled
Each service can fail or restart without affecting the others
The system becomes more resilient and easier to scale

At this point, we have removed the direct dependency between services and solved the resilience problem.

But we still have another challenge.

Stateful Checks Without a Database Round-Trip

Many fraud detection techniques depend on understanding recent activity.

For example, the system may need to evaluate how a card has been used within a short time window before deciding whether a new transaction is suspicious.

In a traditional architecture, this usually requires querying a database for every transaction.

At high throughput, those round-trips add latency that quickly becomes a bottleneck. In a system where decisions must be made in milliseconds, this approach does not scale.

So the question becomes: How do we perform these kinds of checks without hitting the database every time?

Solving It with Kafka Streams

Kafka Streams is a lightweight Java library that runs inside your application, processing events as they flow through Kafka topics. More importantly, it allows us to maintain state locally.

Instead of querying a remote database for every transaction, we keep the necessary data inside the same process that handles the events, using an embedded store backed by RocksDB.

In our example, we want to analyze transaction activity per card within a short time window. To do that, we group events by card number. Each card number becomes a key, and for each key, the system maintains a running view of recent activity within a defined time window. This means we eliminate network calls and database round-trips from the critical path, relying only on local lookups. For our use case, this makes a big difference.

We avoid unnecessary latency and keep everything fast enough for real-time decisions, without depending on external systems. You can see where this happens directly in the code:

return stream
    .filter((key, tx) -> tx != null)
    .selectKey((key, tx) -> tx.cardNumber())
    .groupByKey(Grouped.with(Serdes.String(), transactionSerde))
    .aggregate(
        FraudDetectionState::empty,
        (cardNumber, newTransaction, currentState) -> { countTransactionsInWindow() },
        Materialized.as("fraud-detection-store")...
    )
    .toStream()

This line is key:

Materialized.as("fraud-detection-store")

It tells Kafka Streams to keep the state locally using an embedded store.

As new transactions arrive, the state for each card is updated right there in the application, so the system can keep track of recent activity in real time.

At this point, we can evaluate transaction activity in real time without relying on database calls.

But that raises an important question: What exactly are we measuring?

Which patterns or signals should trigger a fraud decision?

This is where guardrails come in.

Stage 1: Guardrails, Fast Rule-Based Validation

Guardrails are the first layer in our flow.

They’re just simple checks based on rules, used to catch the most obvious fraud cases as early as possible.

In practice, guardrails can be any business or risk rule that makes sense for your domain. They are fully customizable and evolve over time as new fraud patterns emerge.

In our case, we will implement a couple of common examples to illustrate the approach.

Impossible Travel Check

This rule checks if the same card shows up in places that don’t make sense given the time between transactions.

For example, if a card is used in New York and then shows up in São Paulo less than an hour later, that’s not realistic. Something is clearly wrong, and we treat that as a strong fraud signal.

public boolean isImpossibleTravelFraud(Transaction previous, Transaction current) {
    double distanceKm = calculateDistanceKm(
        previous.latitude(), previous.longitude(),
        current.latitude(), current.longitude()
    );
    Duration timeBetween = Duration.between(
        previous.transactionTime(),
        current.transactionTime()
    ).abs();
    // block if physically impossible
}

Velocity Check

The velocity check looks at how many times a card is used in a short period of time. A very common pattern in fraud is when someone tests a stolen card by making several small transactions before trying something bigger.

In our case, if the same card is used more than three times within one minute, we flag it and block the transaction.

public boolean isVelocityFraud(long txCountInWindow, long maxAllowedPerWindow) {
    return txCountInWindow > maxAllowedPerWindow;
}

These are just examples. In a real system, guardrails can include a wide range of checks depending on your use case, risk tolerance, and business rules.

Where Rule-Based Detection Falls Short

Guardrails are fast and effective, but they have blind spots. Consider a transaction like this:

Amount: $2.00
City: same as the cardholder’s usual city
Card: same as always
Activity: nothing unusual on the surface

This transaction would pass every guardrail check.

But what if $2 micro-charges at gas stations are a known fraud pattern?

This is a classic card-testing technique used before a larger fraudulent purchase.

Rules alone cannot catch this, because the system would need to know the pattern in advance.

This is where a different approach becomes necessary.

Stage 2: Vector Scoring, Behavioral Fraud Detection

At this point, the question changes. Instead of asking: “Does this transaction violate a rule?”

We ask: “Does this transaction behave like something we have seen before?”

To answer that, we need a way to compare transactions based on behavior, rather than exact values.

For example:

A $12 purchase at a coffee shop
A $15 purchase at a bakery

These are not identical, but behaviorally they are very similar.

On the other hand:

A $10,000 crypto transaction at 3 AM

This is clearly very different.

So how do we represent this idea in a way that a system can understand?

From transactions to Vectors

To compare behavior, we need a consistent way to represent a transaction.

Instead of working with raw fields such as amount, merchant, or location separately, we transform the entire transaction into a single structured representation. One simple way to think about it is this:

We take a transaction like:

{ "amount": 15.90, "merchant": "Gas Station", "city": "New York" }

And turn it into a descriptive text:

String textToEmbed = "merchant=%s, city=%s, amount=%s";

This text is then passed to an embedding model, which produces a vector like:

[0.0392, 0.9323, -0.0323, ...]

In our case, we are using the voyage-finance-2 model, which is specialized for financial data from Voyage AI.

This model generates a vector with 1024 dimensions, where each value contributes to capturing a different aspect of the transaction’s behavior.

In the application, this embedding generation step can be implemented with a simple HTTP client in Spring. For example:

@HttpExchange(
url = "/v1/embeddings",
contentType = MediaType.APPLICATION_JSON_VALUE,
accept = MediaType.APPLICATION_JSON_VALUE
)
public interface VoyageEmbeddingsClient {
@PostExchange
EmbeddingsResponse embed(@RequestBody EmbeddingsRequest body);
}

The role of this client is to send the transaction text to the embedding API and receive back the generated vector.

The model maps that description into a vector space where similar behaviors are placed closer together. This means that transactions that behave similarly will generate vectors that are close to each other.

To make this more intuitive, you can imagine a simplified 2D space.

In this space:

Everyday purchases like coffee or groceries tend to cluster together
Similar types of transactions stay close
Unusual patterns, like a late-night crypto transaction, appear far apart

In reality, this space is not 2D, but has hundreds or thousands of dimensions.

That is what allows the model to capture subtle patterns that are not obvious when looking at raw transaction data.

The result is a numerical representation of the transaction that preserves its behavior, making it possible to compare transactions efficiently using vector similarity.

Building the Baseline: Seeding the Fraud Pattern Collection

Now that we can convert transactions into vectors, we need something to compare them against. Vector similarity only works if we have a reference dataset.

This means we need a collection of known patterns that represent both normal and fraudulent behavior. To do that, we create a fraud_patterns collection in MongoDB. Each document in this collection contains:

The original transaction data
A fraud label (true or false)
The corresponding embedding

A normal pattern might look like this:

{ 
"fraud": false, 
"transaction": {
"amount": 13.45, 
"merchant": "Coffee Shop" 
},
"embedding": [ ... ]
}

A fraudulent pattern might look like this:

{ 
"fraud": true, 
"transaction": {
"amount": 87422.45, 
"merchant": "Crypto Exchange" 
},
"embedding": [ ... ]
}

This dataset acts as the baseline for comparison.

Instead of asking “Is this transaction fraudulent?”, we ask: “Does this transaction look like something we already know?”

How This Collection Evolves

This collection does not need to be perfect from the start. It typically begins with a small, curated set of known patterns. As the system runs in production:

Confirmed fraud cases can be added
New patterns can be introduced immediately
The system improves over time

One important advantage here is that we do not need to retrain a model. We simply add new examples to the dataset.

Running Vector Search with MongoDB

To run a vector search, we need to create a vector index. This index allows MongoDB to efficiently search for similar vectors.

{
"fields": [
  {
"type": "vector",
"path": "embedding",
"numDimensions": 1024,
"similarity": "cosine"
   }
]
}

java

Each field here matters.

type: "vector" tells MongoDB that this field contains embeddings
path: "embedding" defines where the vector is stored in the document
numDimensions: 1024 must match the size of the vectors produced by the model
similarity: "cosine" defines how similarity between vectors is calculated

If the number of dimensions does not match the embedding model, the index will not work correctly.

Performing the Search

Once the index is in place, MongoDB can efficiently find the nearest vectors. From Spring Data MongoDB, this becomes very straightforward:

@Repository
public interface FraudPatternRepository extends MongoRepository<FraudPattern, String> {

@VectorSearch(
indexName = "fraud_patterns_vector_index",
limit = "10",
numCandidates = "200"
)
SearchResults<FraudPattern> searchTopFraudPatternsByEmbeddingNear(
Vector vector,
Score score
);
}

This method takes the transaction embedding as input and returns the most similar patterns stored in the collection.

Not All Matches Are Equal: Introducing the Threshold

After running the search, MongoDB returns the closest matches along with similarity scores. For example:

Result 1 = score: 0.94
Result 2 = score: 0.91
Result 3 = score: 0.72
Result 4 = score: 0.60

Each score tells us how similar a stored pattern is to the transaction we are analyzing. In practice, vector search returns the closest matches available, even when the similarity is relatively low.

That is why we introduce a threshold.

A threshold defines the minimum similarity score required for a result to be considered relevant.

Any result below this value is ignored. Choosing the right threshold is critical:

Too low → many false positives
Too high → missed fraud cases

The ideal value depends on your domain and risk tolerance.

The Full Architecture

Putting it all together, the flow looks like this:

Client App produces a transaction event to a Kafka topic.
**FraudDetectorProcessor **consumes the event via Kafka Streams.
Guardrails run first: velocity check using local state store and impossible travel check. If triggered, the transaction is immediately inserted into the suspicious_transactions collection.
If it passes guardrails, it proceeds to Vector Scoring.
The transaction is converted into an embedding and compared against fraud patterns in MongoDB using cosine similarity.
If the score exceeds the threshold against a fraud-labeled pattern, the transaction is flagged as suspicious. Otherwise, it is approved.
Results flow to downstream services via Kafka.

Beyond Rules and Similarity

What we built here is just one way to approach fraud detection.

We combined simple rule-based checks with vector similarity to detect patterns in real time. Guardrails work really well for obvious cases, while vector search helps catch things that don’t follow clear rules.

But fraud detection is not the same for everyone.

Something that looks suspicious for one user might be totally normal for another. For example, a late-night crypto transaction might be unusual for most people, but not for someone who trades crypto regularly.

That’s where things can evolve.

One next step would be to include some form of user profiling, where the system learns how each user behaves over time and uses that as part of the decision.

Instead of only comparing against global patterns, we could also compare against the user’s own history. That adds more context and helps reduce false positives.

At the end of the day, it’s about combining different approaches: simple rules for quick decisions, vector similarity for behavior, and more context to better understand each user.

Source Code

The full project is available on GitHub. You can check out the code, run it locally, and see how everything fits together, from Kafka Streams and guardrails to vector scoring and MongoDB.

You’ll also find all the setup instructions in the README, including how to start Kafka, configure the embedding generator, and run simulations to test different scenarios.

How schema anti-patterns in MongoDB can cost you $$$$

MongoDB Guests — Tue, 28 Apr 2026 13:38:52 +0000

This article was written by Darshan Jayarama.

MongoDB is a flexible schema database. This means you have the flexibility to modify the structure of the data store. One collection can have 5 fields in one row(document), 10 in another row, 1000 in another row, and 1 in another row. It doesn't matter how the structure has been aligned in the collection.

If you are a developer with a SQL background, you may find this to be a fantasy, like a science fiction movie. However, that is the flexibility you can achieve with the help of MongoDB. A good percentage of people choose MongoDB for this flexible schema, but let it be known that abusing this feature could cost you big time.

I would like to share some of the story from when I was working with customers to tune the application's performance.

Massive boundless array

Problem: One customer had ping data from an IOT device stored in an array field. In the first few hours of the day, document writes were working fine, but as time passed, writes started to become expensive.

Reason: As the document gets inflated with the growing array field, the findOneAndUpdate() method gets heavy in the network for each document to fetch →update →Save them.

Solution: As the ping frequency is every minute for 1000s of IoT devices, we created a new collection that maps device and ping data to deviceID, hence throughout the day, write was stabilized.

Timeseries saved time

Problem: For the same customer, write was stabilized with a referential model, but each read started costing them time as it had to map the same device ping throughout the day while creating the report.

Reason: Each deviceID data was spread across storage in different pages, so for 100 records, it was technically fetching 100 pages, keeping the required data, and throwing away the rest. Which increased the pagein-pageout.

Solution: MongoDB offers a timeseries collection, which utilizes a bucketing pattern, it has been explained in detail in the video series.

We have created a timeseries collection to store the ping data with deviceID as the metaField and timestamp as the timeField.

When multitenant became the villain

Problem: A customer from a webhosting platform was using a different DB for a different customer with a customer-identification suffix like billing_cust1, data_cust2… Started growing a customer base, became a pain in their database.

Reason: Customer was using MongoDB ChangeStream, which increased the number of databases and caused a delay in the start of their application collections. As explained in the documentation, changestream uses oplog collection in which we cannot utilize the indexes, hence, opening a high number of targeting streams can impact the performance.

Solution: We converted multi-tenancy to a monolithic database structure, storing all customer data we stored in a single collection with customerID in each document. So whenever we query for a specific customer, we append another filter with customerID to perform a targeted query and ensure data safety. This reduced 10000s of collections into 5–10 collections.

$lookup is designed badly

Problem: The application slowed down during the delivery time while verifying the product to be delivered.

Reason: A famous e-commerce company was storing productID in the orders collection. When a scan was performed via the QR code, it was performing multiple lookups in the products collection to fetch the product details.

Solution: Because the number of products was limited, we remodeled the table’s schema and stored the product details in the order collection. As the order details would no longer be accessed regularly after delivery, we archive orders that are 3–4 months old.

Closing thoughts: Performance tuning is a never-ending task, analyze the queries → tune → repeat.

How MongoDB Executes a find() Query: A Complete Lifecycle Guide

MongoDB Guests — Thu, 23 Apr 2026 08:30:25 +0000

This article was written by Darshan Jayarama.

When you type something like db.orders.find({ status: "pending", customerId: 1042 }) and the results come back in milliseconds, it feels simple… almost instant.

But behind that one line, MongoDB is doing a lot more than just “searching a collection.”

During my time as a Senior TSE at MongoDB, I spent most of my days deep in query performance and indexing issues, working closely with both customers and the engineering team. It may look simple, but internally, a lot is happening when a find() executes.

Most engineers understand it at a surface level. But once you start digging into what actually happens under the hood, that’s when things change. You begin to see why some queries are fast, why others are painfully slow, and how indexes truly make or break performance.

So let’s break it down — step by step — what really happens inside MongoDB when a query runs. No shortcuts, nothing skipped.

The full journey of a `find()` query (quick overview)

Before we go step by step, here’s the big picture. Every find() query goes through a series of steps in the mentioned order.

There are also two “fast paths” (shown as green arrows):
• When MongoDB can reuse a plan from the plan cache
• When the data is already in memory (WiredTiger cache)

Getting your queries to use these fast paths is what good MongoDB performance tuning is all about.

Stage 1: The Client Sends a Query Over the Wire

Before MongoDB even sees your query, your driver prepares it.

Whether you’re using PyMongo, the Node.js driver, or mongosh, your query is converted into BSON (a binary version of JSON) and sent over a network connection to MongoDB.

MongoDB uses the Wire Protocol to communicate, which is basically a structured way of sending messages between your app and the database.

Since MongoDB 3.6, most operations use a format called OP_MSG.

This message includes:
• which database and collection you’re querying
• the actual filter (your query conditions)
• extra options like projection, sort, limit, skip, hints, and read preferences

# PyMongo serializes this to BSON OP_MSG internally

db.orders.find(
   { "status": "pending", "customerId": 1042 },
   projection={"_id": 0, "amount": 1},
   sort=[("createdAt", -1)],
   limit=10
)

The query lands at the mongod process. The network listener accepts the connection and hands it off to a worker thread from the connection pool.

Stage 2: Authentication Check

The first thing MongoDB does — even before looking at any data — is check who is making the request.

MongoDB supports different ways to authenticate users, but in most real-world cases, this step is already handled. Drivers usually keep connections open and authenticated, so this check happens almost instantly

MongoDB just verifies the session linked to that connection.

If authentication fails, the query stops right there.
You’ll see an error like: “Authentication failed”, and the query never even reaches the database engine.

Stage 3: Authorization — Can You Read This Collection?

Authentication tells MongoDB who you are. Authorization decides what you’re allowed to do. MongoDB uses role-based access control (RBAC). It checks whether your user has permission to run a find() on that specific database and collection.

This could come from built-in roles like read or readAnyDatabase, or from custom roles with specific permissions.

If you don’t have access, the query stops there. You’ll see an error like: “not authorized to execute command”. The good part? This check is extremely fast — it’s just a quick in-memory lookup, not a database scan.

Stage 4: Query Parsing and BSON Validation

At this point, your query is sitting in memory as a BSON document. Now MongoDB starts understanding it.

It does two main things:

1. Checks if the query is valid

MongoDB makes sure your query is written correctly — valid operators like $eq, $in, $gt, $elemMatch, proper structure, and no invalid combinations. If something is wrong, the query fails right here with an error.

2. Converts it into an internal format:

MongoDB rewrites your query into a standard internal structure (think of it like a tree). This is why the order of fields doesn’t matter.

For example, these two are treated exactly the same:
{ a: 1, b: 2 }
{ b: 2, a: 1 }

Internally, MongoDB sees it more like:

AND
├── status = "pending"
└── customerId = 1042

In short, MongoDB first checks that your query is valid, then rewrites it into a format it can efficiently work with.

Stage 5: The Query Planner Enumerates Candidate Plans

This is where MongoDB actually starts making smart decisions. The query planner takes your parsed query and determines the best way to get the data. It doesn’t just pick one way — it tries out multiple options.

For every useful index, MongoDB creates a possible plan. It also always keeps one backup option: scanning the whole collection (COLLSCAN).

Each plan is basically a step-by-step approach to fetch the data.

For example:

Using index on status:

 FETCH
  └─ scan index { status: 1 }
Using index on customerId:
 FETCH
  └─ scan index { customerId: 1 }
Using compound index (status + customerId):
 scan index { status:1, customerId:1 }
 (no FETCH needed — everything is in the index)
No index:
 scan entire collection

MongoDB also thinks about a few important things here:
• Can it avoid sorting by using an index?
• Can it return results directly from the index (covered query)?
• Can it combine multiple indexes if needed?

In short, MongoDB tries different ways to run your query and prepares multiple plans before choosing the best one.

Stage 6: Plan Cache Lookup — The Fast Path

Before MongoDB tries out different plans, it first checks something called the plan cache.

The plan cache stores the best plan from previous runs of similar queries. So if MongoDB has already seen a query like yours before, it can skip all the extra work and reuse the same plan.

What matters here is the query shape — basically the structure of the query, not the actual values.

For example, these two queries are treated the same:

db.orders.find({ status: "pending", customerId: 1042 })

db.orders.find({ status: "shipped", customerId: 9999 })

Because structurally, they are identical:
{ status: <eq>, customerId: <eq> }

Now two things can happen:
• Cache hit → MongoDB already knows the best plan, so it skips all the trial work and runs it directly (this is the fast path)
• Cache miss → MongoDB doesn’t have a saved plan, so it tries multiple options to find the best one

A few important things to know about the plan cache:
• It’s stored in memory (not on disk)
• It’s maintained per collection
• It gets cleared when MongoDB restarts
• It’s also reset if the indexes change or the collection changes a lot

You can even inspect or clear it manually:

// See cached plans
db.orders.getPlanCache().list()
// Clear cache (forces MongoDB to re-evaluate plans)
db.orders.getPlanCache().clear()

In short, if your query has been seen before, MongoDB can skip straight to execution — which is why repeated queries are usually much faster 🚀

Stage 7: Multi-Plan Trial — The Index Race

If MongoDB doesn’t find a plan in the cache, it tries something really interesting.

Instead of guessing the best plan, it tests all possible plans simultaneously. This is called the multi-plan stage.

Each plan is given a chance to run for a few steps, one after the other — kind of like a race.

For example:
Round 1:
Plan A → small progress
Plan B → small progress
Plan C → small progress

Round 2:
Plan A → more progress
Plan B → more progress
Plan C → more progress

This continues until one plan clearly outperforms the others.

The winner is the plan that:
• returns the first ~100 results fastest, or
• finishes scanning the data quickest

Once a plan wins, MongoDB:
→ uses it for the current query
→ saves it in the plan cache for next time

What about full collection scans (COLLSCAN)?

They usually lose… but not always.

• If there are no useful indexes → COLLSCAN wins
• If the collection is very small → COLLSCAN can actually be faster than using an index

If you want to see how this decision was made, you can run:

db.orders.find({ status: "pending", customerId: 1042 })
        .explain("allPlansExecution")

Stage 8: Index Scan (IXSCAN) or Collection Scan (COLLSCAN)

Now that MongoDB has picked the best plan, it finally executes the query.

If an index is used (IXSCAN)

MongoDB uses indexes that work like a tree structure. It quickly navigates this tree to find matching entries.

Once it finds a match, it uses a reference (called RecordId) to go and fetch the actual document from the collection.

Think of it like:
— find the entry in the index
— then go grab the full document

There’s also something called a covered query: If all the fields you need are already in the index, MongoDB doesn’t even need to look at the actual documents.

It returns results directly from the index
This is the fastest possible way to read data

If no index is used (COLLSCAN)

If there’s no useful index, MongoDB has no choice — it reads every document one by one.

So if your collection has 10 million documents, it will scan all 10 million.

That’s definitely slow.

How to spot a problem

When you run explain(), watch for this:

db.orders.find({ status: "pending" }).explain("executionStats")
// Red flags to look for:
// "stage": "COLLSCAN"          ← no index used
// "totalDocsExamined": 9847321  ← scanned 9.8M docs
// "nReturned": 142              ← returned only 142
// Ratio: 69,000:1               ← extremely inefficient

Red flags:

"stage": "COLLSCAN" → no index is being used
Very high "totalDocsExamined"
Very low "nReturned"
scanned ~9.8 million documents
returned only 142

That’s a huge gap, and a clear sign your query is inefficient

Simple rule
If MongoDB is reading way more documents than it returns, you probably need a better index.

Stage 9: The Storage Layer — WiredTiger Cache and Disk

Once MongoDB knows which documents it needs, the next question is: Where does it actually read the data from?

There are three possible sources, and the speed depends on where the data is found.

1. WiredTiger cache (in-memory — fastest)
MongoDB uses WiredTiger as its storage engine, which keeps frequently accessed data in memory.

By default, it uses about 50% of the available RAM.

If the required data is already in this cache, MongoDB can return it almost instantly. This is the ideal scenario.

2. OS page cache (still fast)
If the data is not in MongoDB’s own cache, it checks the operating system’s page cache.

The OS may already have the data in memory from recent reads. Since MongoDB memory-maps its data files, this check is efficient.

This is slightly slower than the WiredTiger cache, but still very fast.

3. Disk (slowest)
If the data is not present in either cache, MongoDB has to read it from disk.

The performance here depends on the type of storage:

NVMe SSD: fastest among disks
SATA SSD: moderate
HDD: significantly slower

At scale, disk access becomes the main bottleneck, especially when queries access data randomly across large datasets. If your working data set fits in memory, queries remain fast. If MongoDB frequently needs to read from disk, query performance drops significantly. This is why working set size is critical in MongoDB performance tuning — your frequently accessed data should ideally fit in RAM.

Checking cache performance

You can inspect cache behavior using:

db.serverStatus().wiredTiger.cache
// 'pages read into cache'       ← total cache misses (disk reads)
// 'pages requested from cache'  ← total requests
// Hit ratio = 1 - (read/requested)
// Target: > 95%

Key metrics:

pages read into cache → disk reads (cache misses)
pages requested from cache → total requests

A good system typically has a cache hit ratio above 95%.

Stage 10: Results Returned to the Client

Here’s what happens:

Applies projection: Removes any fields you didn’t ask for and keeps only what’s needed
Applies skip and limit: Skips the first N documents (if specified) and limits how many results are returned
Converts back to BSON: Turns the in-memory document into a format that can be sent over the network
Creates response batches: MongoDB doesn’t send everything at once. The first batch contains up to 101 documents or 16MB of data (whichever comes first)
Sends the response: The data is sent back to your application over the same connection

What if there’s more data?

If your query returns more than one batch, MongoDB doesn’t send everything in one go.

Instead, it returns a cursor ID.

Your driver then automatically requests the next batch using getMore commands. This happens behind the scenes, so you usually don’t notice it.

The Complete Picture — All 10 Stages

Closing notes: Key Takeaways for Production

Always check your explain() output - Use explain("executionStats") to see exactly how your query ran — which plan was used, how many documents were scanned vs returned, and whether the plan came from cache.
Design indexes based on query patterns, not just fields -Think about how your queries are written. For example, a compound index like { status: 1, customerId: 1 } is usually much more effective than having separate indexes on each field. It can even avoid extra steps like fetching documents if the query is covered.
Keep an eye on cache efficiency - Monitor your WiredTiger cache hit ratio. If it drops below ~90%, it usually means your frequently accessed data no longer fits in memory. At that point, you may need to add more RAM, shard the data, or rethink how your application accesses data.
Be aware of plan cache issues - If a query suddenly becomes slow after something like a bulk insert, the plan cache might be the cause. MongoDB may have picked a suboptimal plan after re-evaluating. In such cases, clear the cache and check the query again with explain().

// Your daily driver for query diagnostics
db.orders.find({ status: "pending" }).explain("executionStats")

// Key fields to read:
// executionStats.executionTimeMillis    ← total time
// executionStats.totalDocsExamined      ← docs touched
// executionStats.totalKeysExamined      ← index keys scanned
// executionStats.nReturned              ← docs returned
// queryPlanner.winningPlan.stage        ← IXSCAN or COLLSCAN

Introduction to Events using MongoDB

MongoDB Guests — Wed, 22 Apr 2026 21:52:27 +0000

This article was written by Otavio Santana.

Keeping software simple is one of the hardest challenges for any software architect or senior engineer. Not because of technology, but because of decisions we often overlook—especially coupling. Many systems start simple, but as new requirements emerge, the same code begins to orchestrate multiple responsibilities. What once felt clean becomes fragile, and small changes require touching multiple parts of the system. This is how architectures silently degrade. The good news is that there are ways to address this problem. In this tutorial, we will explore an architectural approach that helps reduce coupling and improve extensibility, and how MongoDB can effectively support this style.

In this tutorial, you’ll:

Model a simple payment system
Write events sync and async events
Explore how MongoDB can help you on archive Event-driven design.

You can find all the code presented in this tutorial in the GitHub repository:

git clone git@github.com:soujava/helidon-mongodb-event-driven.git

Prerequisites

For this tutorial, you’ll need:

Java 21.
Maven.
A MongoDB cluster.
MongoDB Atlas (Option 1)
Docker (Option 2)

You can use the following Docker command to start a standalone MongoDB instance:

docker run --rm -d --name mongodb-instance -p 27017:27017 mongo

When we discuss event-driven design, we change the approach or the question: Instead of asking “who should I call next?”, we start asking: “What just happened?”

This change allows modules to interact without direct dependencies. Different parts of the system can react independently—either synchronously or asynchronously—without modifying the original flow.
As a first step, we need to understand what an event is. An event represents a fact that has already happened in the system. It is immutable and carries enough information for other components to react. Instead of telling the system what to do, events describe what occurred, allowing behavior to emerge through reactions.

In Domain-Driven Design (DDD), events take on a more specific role. They represent meaningful events in the business domain, using the same language as the business itself. These domain events can also act as integration points, enabling different parts of the system, including across bounded contexts—or even external systems—to respond to business changes. Event-driven goes beyond DDD, allowing facts on both the technical and the domain side.

In this tutorial, we will create an over-simplified payment system where, given a payment for a product, we will issue a payment request to an external provider, which will start an async event. Based on that, we will generate an event depending on whether the payment succeeded or failed. The whole process starts at the JAX-RS endpoint, generating a sequence of events and states inside the payment.

In this tutorial, we will simplify using CDI events, but you can naturally enhance it or use another service to handle those events, such as Kafka or a JMS Broker. Thus, we will use MongoDB and Helidon. Creating a Helidon project is simple: use the Helidon Starter, and in the wizard progress flow, you will select Helidon MP, QuickStart, and JSON-B. Feel free to define the groupid, artifactId, version, and package name as you wish. After downloading, the next step is including the MongoDB integration, in this case, Eclipse JNoSQL, in the pom.xml file in the root of the downloaded project:

<dependency>
   <groupId>org.eclipse.jnosql.databases</groupId>
   <artifactId>jnosql-mongodb</artifactId>
   <version>1.1.13</version>
</dependency>

With the project defined, the next step is to set the database configuration. You can either use a local database or explore MongoDB Atlas; either is fine. We will use locally, thus, run this Docker command to start a MongoDB instance, include those new properties:

# configure the MongoDB client for a replica set of two nodes
jnosql.mongodb.url=mongodb://localhost:27017
# mandatory: define the database name
jnosql.document.database=ecommerce

Step 1: Create the entities

With the project defined, the next step is to create the entities and repositories that will support it.

In the src/main/java directory, create a Product class:

package com.acme;

import jakarta.nosql.Column;
import jakarta.nosql.Embeddable;

import java.util.UUID;

@Embeddable(Embeddable.EmbeddableType.GROUPING)
public record Product(@Column UUID code, @Column String name) {

}

Create a PaymentCounter class:

package com.acme.statistics;

import com.acme.Product;
import com.acme.infraestructure.JsonFieldStrategy;
import jakarta.json.bind.annotation.JsonbVisibility;
import jakarta.nosql.Column;
import jakarta.nosql.Entity;
import jakarta.nosql.Id;

import java.util.UUID;

@Entity
@JsonbVisibility(JsonFieldStrategy.class)
public class PaymentCounter {

    @Id
    private UUID productCode;

    @Column
    private Product product;

    @Column
    private int successfulPayments;

    @Column
    private int failedPayments;

    PaymentCounter(Product product) {
        this.productCode = product.code();
        this.product = product;
    }

    PaymentCounter() {
    }

    public void paymentFailed() {
        this.failedPayments++;
    }

    public void paymentSucceeded() {
        this.successfulPayments++;
    }

    @Override
    public String toString() {
        return "PaymentCounter{" +
                "productCode=" + productCode +
                ", successfulPayments=" + successfulPayments +
                ", failedPayments=" + failedPayments +
                '}';
    }
}

And the Payment and PaymentStatus classes:

package com.acme.payment;

import com.acme.Product;
import com.acme.infraestructure.JsonFieldStrategy;
import jakarta.json.bind.annotation.JsonbVisibility;
import jakarta.nosql.Column;
import jakarta.nosql.Entity;
import jakarta.nosql.Id;

import java.math.BigDecimal;
import java.util.Objects;
import java.util.UUID;

@Entity
@JsonbVisibility(JsonFieldStrategy.class)
public class Payment {

    @Id
    private UUID id;

    @Column
    private BigDecimal amount;

    @Column
    private PaymentStatus status;

    @Column
    private Product product;

    Payment() {
    }

    Payment(Product product, BigDecimal amount, PaymentStatus status) {
        this.id = UUID.randomUUID();
        this.amount = amount;
        this.status = status;
        this.product = product;
    }

    public UUID getId() {
        return id;
    }

    @Override
    public boolean equals(Object o) {
        if (!(o instanceof Payment payment)) {
            return false;
        }
        return Objects.equals(id, payment.id);
    }

    @Override
    public int hashCode() {
        return Objects.hashCode(id);
    }

    public Product getProduct() {
        return product;
    }

    @Override
    public String toString() {
        return "Payment{" +
                "id='" + id + '\'' +
                ", amount=" + amount +
                ", status=" + status +
                ", product=" + product +
                '}';
    }

    public void failed() {
        this.status = PaymentStatus.FAILED;
    }

    public void confirmed() {
        this.status = PaymentStatus.CONFIRMED;
    }
}


//different file

package com.acme.payment;

public enum PaymentStatus {
    PENDING,
    CONFIRMED,
    FAILED
}

Explanation of annotations:

@entity: Marks the Room class as a database entity for management by Jakarta NoSQL.
@id: Indicates the primary identifier for the entity, uniquely distinguishing each document in the MongoDB collection.
@column: Maps fields (roomNumber, type) for reading from or writing to MongoDB.

With the entities created, the next step is creating the repositories for each entity. In our tutorial, we simplified Product as a Value Object.

package com.acme.payment;

import jakarta.data.repository.BasicRepository;
import jakarta.data.repository.Repository;

import java.util.UUID;

@Repository
public interface PaymentRepository extends BasicRepository<Payment, UUID> {
}

//different file
package com.acme.statistics;

import jakarta.data.repository.BasicRepository;
import jakarta.data.repository.Repository;

import java.util.UUID;

@Repository
public interface PaymentCounterRepository extends BasicRepository<PaymentCounter, UUID> {
}

With both entities and repositories defined, the next step is defining the events that we will handle on this system.

Step 2: Create the events

In this sample, we will have three events to represent three states of the payment: when started (pending) ,because it requires operation on an external payment service; Payment failed; or confirmed based on the response from this external payment provider.

In the src/main/java directory, create a Product class:

package com.acme.payment;

public record PaymentFailedEvent(Payment payment) {
}
//different file
package com.acme.payment;

public record PaymentConfirmedEvent(Payment payment) {
}
//different file
package com.acme.payment;

public record PaymentRequestedEvent(Payment payment) {
}

Step 3: Generate the listeners

At this event, we will have listeners or classes that respond to some facts or events in our system. Using CDI that is pretty simple, we just need to have a parameter with the class to be observed and put either Observes or ObservesAsync to observe synchronous or asynchronous simultaneously.

We will start creating a service to update the status based on both confirmed or failed payment. This service will find the payment by id and then update the status.

In the src/main/java directory, create a PaymentStatusService class:

package com.acme.payment;

import jakarta.enterprise.context.ApplicationScoped;
import jakarta.enterprise.event.Observes;
import jakarta.inject.Inject;

import java.util.logging.Logger;

@ApplicationScoped
class PaymentStatusService {

    private static final Logger LOGGER = Logger.getLogger(PaymentStatusService.class.getName());

    private final PaymentRepository repository;

    @Inject
    PaymentStatusService(PaymentRepository repository) {
        this.repository = repository;
    }

    PaymentStatusService() {
        this.repository = null;
    }

    void payed(@Observes PaymentConfirmedEvent event) {
        LOGGER.info("Payment " + event.payment().getId() + " was payed");
        var payment = repository.findById(event.payment().getId()).orElseThrow();
        payment.confirmed();
        repository.save(payment);
        LOGGER.info("Payment " + payment.getId() + " was confirmed");
    }

    void errorOnPayment(@Observes PaymentFailedEvent event) {
        LOGGER.info("Payment " + event.payment().getId() + " failed");
        var payment = repository.findById(event.payment().getId()).orElseThrow();
        payment.failed();
        repository.save(payment);
        LOGGER.info("Payment " + payment.getId() + " was failed");
    }
}

The next step is the counter that will register success and failure based on the product code.

In the src/main/java directory, create a PaymentCounterService class:

package com.acme.statistics;

import com.acme.Product;
import com.acme.payment.Payment;
import com.acme.payment.PaymentFailedEvent;
import com.acme.payment.PaymentConfirmedEvent;
import jakarta.data.Order;
import jakarta.data.Sort;
import jakarta.data.page.Page;
import jakarta.data.page.PageRequest;
import jakarta.enterprise.context.ApplicationScoped;
import jakarta.enterprise.event.Observes;
import jakarta.inject.Inject;

import java.util.List;
import java.util.function.Consumer;
import java.util.logging.Logger;

@ApplicationScoped
public class PaymentCounterService {

    private static final Order<PaymentCounter> PRODUCT_CODE = Order.by(Sort.asc("productCode"));

    private static final Logger LOGGER = Logger.getLogger(PaymentCounterService.class.getName());

    private final PaymentCounterRepository repository;

    @Inject
    PaymentCounterService(PaymentCounterRepository repository) {
        this.repository = repository;
    }

    PaymentCounterService() {
        this.repository = null;
    }

    void success(@Observes PaymentConfirmedEvent event) {
        LOGGER.info("Payment successful, incrementing counter: " + event.payment().getId());
        execute(PaymentCounter::paymentSucceeded, event.payment().getProduct());
    }


    void success(@Observes PaymentFailedEvent event) {
        LOGGER.info("Payment failed, incrementing counter: " + event.payment().getId());
        execute(PaymentCounter::paymentFailed, event.payment().getProduct());
    }

    void execute(Consumer<PaymentCounter> action, Product product) {
        PaymentCounter counter = repository.findById(product.code()).orElseGet(() -> new PaymentCounter(product));
        LOGGER.info("Counter found: " + counter);
        action.accept(counter);
        LOGGER.info("Counter incremented: " + counter);
        repository.save(counter);
    }

    public List<PaymentCounter> findAll(PageRequest pageRequest) {
        LOGGER.info("Finding all counters, page: " + pageRequest.page());
        Page<PaymentCounter> counters = repository.findAll(pageRequest, PRODUCT_CODE);
        return counters.content();
    }
}

Finally, we will have the service that will handle an external provider. To simplify, we will set a wait of 2 seconds, and then based on the counter defined as either success or failure, this one uses an asynchronous event, as you can see in the annotation.

In the src/main/java directory, create a PaymentProvider class:

package com.acme.payment.provider;

import com.acme.payment.PaymentFailedEvent;
import com.acme.payment.PaymentRequestedEvent;
import com.acme.payment.PaymentConfirmedEvent;
import jakarta.enterprise.context.ApplicationScoped;
import jakarta.enterprise.event.Event;
import jakarta.enterprise.event.ObservesAsync;
import jakarta.inject.Inject;

import java.util.concurrent.atomic.AtomicLong;
import java.util.logging.Logger;

@ApplicationScoped
class PaymentProvider {

    private static final Logger LOGGER = Logger.getLogger(PaymentProvider.class.getName());

    private final AtomicLong counter = new AtomicLong();

    private final Event<PaymentConfirmedEvent> successEvent;

    private final Event<PaymentFailedEvent> errorEvent;

    @Inject
    PaymentProvider(Event<PaymentConfirmedEvent> successEvent, Event<PaymentFailedEvent> errorEvent) {
        this.successEvent = successEvent;
        this.errorEvent = errorEvent;
    }

    PaymentProvider() {
        this.successEvent = null;
        this.errorEvent = null;
    }

    void receivePayment(@ObservesAsync PaymentRequestedEvent event) {
        LOGGER.info("Processing payment: " + event.payment().getId());
        try {
            Thread.sleep(2000);
        } catch (InterruptedException e) {
            Thread.currentThread().interrupt();
        }
        if(counter.get() % 2 == 0) {
            LOGGER.warning("Payment failed: " + event.payment().getId());
            errorEvent.fire(new PaymentFailedEvent(event.payment()));
        } else {
            LOGGER.info("Payment successful: " + event.payment().getId());
            successEvent.fire(new PaymentConfirmedEvent(event.payment()));
        }
        LOGGER.info("Payment processed: " + event.payment().getId() + " - Confirmation #" + counter.incrementAndGet());
    }
}

Step 4: Generating the rest endpoint

The observers are done, but we need one way to start the whole process. In this case, we will use a rest end-point. This resource will have two methods, one to create a payment and another to paginate the current payments.

At src/main/java, create the PaymentResource class.

package com.acme.payment;

import jakarta.data.page.PageRequest;
import jakarta.inject.Inject;
import jakarta.ws.rs.DefaultValue;
import jakarta.ws.rs.GET;
import jakarta.ws.rs.POST;
import jakarta.ws.rs.Path;
import jakarta.ws.rs.QueryParam;
import jakarta.ws.rs.core.Response;

import java.util.List;

@Path("/payments")
public class PaymentResource {


    private final PaymentService service;

    @Inject
    public PaymentResource(PaymentService service) {
        this.service = service;
    }

    PaymentResource() {
        this.service = null;
    }

    @POST
    public Response create(PaymentRequest request) {
        return Response.accepted(service.create(request)).build();
    }

    @GET
    public List<Payment> getAll(@QueryParam("page") @DefaultValue("1") int page) {
        var pageRequest = PageRequest.ofPage(page);
        return service.findAll(pageRequest);
    }
}

The resource itself needs the PaymentService that will create a payment as pending status, it will start an asynchronous event, and also, it will provide the payment status.

At src/main/java, create the PaymentService class.

package com.acme.payment;

import com.acme.Product;
import jakarta.data.Order;
import jakarta.data.Sort;
import jakarta.data.page.Page;
import jakarta.data.page.PageRequest;
import jakarta.enterprise.context.ApplicationScoped;
import jakarta.enterprise.event.Event;
import jakarta.inject.Inject;

import java.math.BigDecimal;
import java.util.List;
import java.util.logging.Logger;

@ApplicationScoped
public class PaymentService {

    private static final Logger LOGGER = Logger.getLogger(PaymentService.class.getName());
    private static final Order<Payment> PAYMENT_ORDER = Order.by(Sort.asc("id"));

    @Inject
    private PaymentRepository repository;

    @Inject
    private Event<PaymentRequestedEvent> event;

    public Payment create(PaymentRequest request) {

        LOGGER.info("Creating payment for " + request.productCode());
        Product product = new Product(
                request.productCode(),
                request.productName()
        );

        BigDecimal total = request.unitPrice()
                .multiply(BigDecimal.valueOf(request.quantity()));

        Payment payment = new Payment(
                product,
                total,
                PaymentStatus.PENDING
        );

        repository.save(payment);
        LOGGER.info("Payment created: " + payment);
        event.fireAsync(new PaymentRequestedEvent(payment));
        return payment;
    }

    public List<Payment> findAll(PageRequest pageRequest) {
        LOGGER.info("Finding all payments, page: " + pageRequest.page());
        Page<Payment> payments = repository.findAll(pageRequest, PAYMENT_ORDER);
        return payments.content();
    }
}

The last resource for today is the one to show the statistics of our payment based on the product code.

At src/main/java, create the PaymentCounterResource class.

package com.acme.statistics;

import com.acme.payment.Payment;
import jakarta.data.page.PageRequest;
import jakarta.inject.Inject;
import jakarta.ws.rs.DefaultValue;
import jakarta.ws.rs.GET;
import jakarta.ws.rs.Path;
import jakarta.ws.rs.QueryParam;

import java.util.List;

@Path("/payment-counter")
public class PaymentCounterResource {

    private final PaymentCounterService service;

    @Inject
    public PaymentCounterResource(PaymentCounterService service) {
        this.service = service;
    }

    PaymentCounterResource() {
        this.service = null;
    }

    @GET
    public List<PaymentCounter> getAll(@QueryParam("page") @DefaultValue("1") int page) {
        var pageRequest = PageRequest.ofPage(page);
        return service.findAll(pageRequest);
    }

}

Conclusion

We got it! In this tutorial, we learned a new software architecture pattern and its motivation: the event-driven design, which changes our way of thinking about the system by focusing on the events that occur within it, rather than worrying about orchestrating multiple services and eventually handling coupling. We could explore Domain-Driven design concepts with a lightweight event-driven approach, separating responsibilities, handling asynchronous processes naturally, and keeping the core logic simple and extensible.

Ready to explore the benefits of MongoDB Atlas? Get started now by trying MongoDB Atlas.
Access the source code used in this tutorial.
Any questions? Come chat with us in the MongoDB Community Forum.

References:

Source code

Detective mode: Fixing the MongoDB Aggregation pipeline

MongoDB Guests — Tue, 21 Apr 2026 09:32:31 +0000

This article was written by Darshan Jayarama.

It was a Friday, and I was quickly wrapping up my week, finishing the tasks that were assigned. Suddenly, the board became red — another S1. I was like, not now!!!

The application team started complaining about the API responses from the database, going from 600ms to 17 seconds. Booooom!!! That’s insane. No other choice — since it was production, I had to jump in. The war room was ready. The monitoring dashboard was nothing but a heart patient’s ECG — spikes, timeouts, and the main app was choking.

What's the Mystery?

From the dashboard, we identified a query that was asked by the VP to add a breakdown by region to the dashboard. It was just one more field. But it was ONE.MORE.FIELD.

Innocent query, which was harmless for years:

db.orders.aggregate([
 {
   $match: {
     status: "completed",
     created_at: { $gte: new Date("2025-01-31") }
   }
 },
 {
   $group: {
     _id: "$customer_id",
     total_orders: { $sum: 1 },
     total_revenue: { $sum: "$amount" },
     avg_order_value: { $avg: "$amount" }
   }
 },
 {
   $sort: { total_revenue: -1 }
 },
 {
   $limit: 100
 }
])

Simple, right? top 100 customer revenue dashboard.

Execution time 500–700ms, Docs Examined ~2.5M.

We then added just one more field query:

db.orders.aggregate([
 {
   $match: {
     status: "completed",
     created_at: { $gte: new Date("2024-01-01") }
   }
 },
 {
   $lookup: { 
     from: "customers",
     localField: "customer_id",
     foreignField: "_id",
     as: "customer"
   }
 },
 {
   $unwind: "$customer"
 },
 {
   $group: {
     _id: {
       customer_id: "$customer_id",
       region: "$customer.region"
     },
     total_orders: { $sum: 1 },
     total_revenue: { $sum: "$amount" },
     avg_order_value: { $avg: "$amount" }
   }
 },
 {
   $sort: { total_revenue: -1 }
 },
 {
   $limit: 100
 }
])

Execution time >15000ms, Docs Examined 2.5M X 2.5M, memory exploded.

The Investigation: Agent Darshan reporting

Evidence #1:

MongoDB profiler, setting the status to 2, helps us collect all the queries that are being run on the given database. Set this level only when needed, else you’re gonna blow up the log file:

db.setProfilingLevel(2, { slowms: 100 })
db.system.profile.find().sort({ ts: -1 }).limit(1).pretty()

I could see no mercy in our query, brutally attacking the MongoDB cluster:

{
 "op": "command",
 "ns": "production.orders",
 "command": { /* our aggregation */ },
 "keysExamined": 2489234,      // ✅ Used index on status
 "docsExamined": 2489234,      // But still examined all matched docs
 "numYield": 18943,
 "nreturned": 100,
 "responseLength": 8234,
 "millis": 647,
 "planSummary": "IXSCAN { status: 1 }",  // ✅ Used status index
 "execStats": {
   "stage": "FETCH",
   "filter": {
     "created_at": { "$gte": ISODate("2025-01-31") }  // 👈 Filter applied AFTER index
   },
   "inputStage": {
     "stage": "IXSCAN",
     "keyPattern": { "status": 1 },
     "indexName": "status_1",
     "keysExamined": 2489234,
     "seeks": 1,
     "dupsTested": 0,
     "dupsDropped": 0,
     "direction": "forward"
   },
   "totalDocsExamined": 2489234,
   "totalKeysExamined": 2489234,
   "executionTimeMillisEstimate": 447
 }
}

Red Flags:

DocsExamined = ~2.5M, whereas nReturened 100
PlanSummary: Single key index on boolean status_1.
High yield.

Evidence #2:

Need to review and explain the plan to make the query confess to the brutal attack:

{
 "stages": [
   {
     "$cursor": {
       "queryPlanner": {
         "winningPlan": {
          "stage": "IXSCAN",
           "keyPattern": { "status": 1 },
           "filter": {
             "status": { "$eq": "completed" },
             "created_at": { "$gte": new Date("2025-01-31") }
           }
         }
       },
       "executionStats": {
         "executionTimeMillis": 647,  // $match stage: 647ms
         "totalDocsExamined": 2489234,
         "totalKeysExamined": 2489234,  //
         "nReturned": 2489234  // Passed 2.48M docs to next stage
       }
     }
   },
   {
     "$lookup": {  // 💀💀💀 THE SERIAL KILLER
       "from": "customers",
       "as": "customer",
       "executionTimeMillis": 13934,  // 13.9 seconds just for $lookup!
       "totalDocsExamined": 2489234,  // Looked up 2.48M customers
       "stage": "NESTED_LOOP_JOIN",  // No index on customers._id
       "indexesUsed": []  // 👈 DISASTER
     }
   },
   {
     "$unwind": {
       "executionTimeMillis": 823
     }
   },
   {
     "$group": {
       "executionTimeMillis": 1156,
       "usedDisk": true,  // 💀 Spilled to disk (>100MB memory)
       "spilledRecords": 847293
     }
   },
   {
     "$sort": {
       "executionTimeMillis": 87,
       "sortPattern": { "total_revenue": -1 },
       "usedDisk": false  // Lucky, only 100 docs to sort after $limit
     }
   }
 ],
 "executionTimeMillis": 16847
}

Time breakdown:

$match: 647ms → $lookup: 13.9 seconds → $unwind: 823ms → $group: 1156ms → $sort: 87ms → Total 16.9 seconds

Suspects list

Suspect #1:

Unoptimized index for $match. Even though we had a single index on status:1 and created_at:1, we didn't have a compound index on both.

Motive: Index inefficiency
Proof: was high nReturned to the next stage.

Suspect #2:

The $lookup brutalness, for each customer_id from the previous stage, finding the join on the customers collection with an absent element in indexUsed. Isn’t that brutal?

Motive: just one more field request
Proof: empty indexUsed and 13.9s execution time.

Suspect #3:

$group surrendered with memory hog, spilled ~847k documents to disk as it exceeded 100MB per stage MongoDB limit.

Motive: Grouping 2.4M records with useDisk true
Proof: 847k spilledRecords.

Suspect #4:

We were performing an expensive $lookup operation before we could filter down to 100 records.

Motive: Not thinking about pipeline optimization
Proof: Processing 2.48M docs when we only needed 100

Recapping all mistakes to fix!

We were doing $match: 647ms(inefficient index)→ $lookup: 13.9 seconds(massive 2.4M join) → $unwind: 823ms → $group: 1156ms(memory overflow) → $sort: 87ms → Total 16.9 seconds

We redesigned as,

$match with compound index for efficient filtering
$group by customer_id reduce the dataset
$sort and $limit top 100 customer and we have only 100 records now.
$lookup for the 100 customer data we wanted.
$project to add region information.

Does it fix the issues?

The Solution:

2.5M orders
↓ $match (INDEXED — fast!)
2.48M orders
↓ $group (by customer_id only)
~500K customer groups
↓ $sort + $limit
100 top customers
↓ $lookup (only 100 lookups!) ← ✅ FIXED
100 results with customer data
↓ $project (extract region)
100 final results

The best method of using $lookup is not using it!

db.orders.aggregate([
 {
   $match: {
     status: "completed",
     created_at: { $gte: new Date("2024-01-01") }
   }
 },
 {
   $group: {
     _id: "$customer_id",  // Group BEFORE lookup
     total_orders: { $sum: 1 },
     total_revenue: { $sum: "$amount" },
     avg_order_value: { $avg: "$amount" }
   }
 },
 {
   $sort: { total_revenue: -1 }
 },
 {
   $limit: 100  // Reduce to 100 BEFORE lookup
 },
 {
   $lookup: {  // Now only looking up 100 customers
     from: "customers",
     localField: "_id",
     foreignField: "_id",
     as: "customer"
   }
 },
 {
   $unwind: "$customer"
 },
 {
   $project: {
     customer_id: "$_id",
     region: "$customer.region",
     total_orders: 1,
     total_revenue: 1,
     avg_order_value: 1
   }
 }
])

Curious to know the metrics now? I am excited to display as well:

Execution time 847ms, docsExamined 2.5M, lookup performed 100 documents.

✅ Issue resolved

Root cause: $lookup before aggregation.
Fix: Aggregate first, then join the top 100
Performance: 17s → 0.8s (20x improvement)
Added indexes: (status, created_at)

The Costly Lessons Learned:

One field can kill you. That “simple” region addition cost us $24,000 in downtime.
$lookup is not free. It’s actually very expensive. Treat it like a loaded gun.
Order matters. Aggregate → Limit → Join. Not Join → Aggregate → Limit.
Indexes aren’t optional. They’re the difference between 1.8 seconds and 0.15 seconds.
Explain is your friend. If you’re not running, explain, you’re flying blind.
The 100MB limit is real. When $group says “usedDisk: true”, you’re already in trouble.
Test with production data volumes. 1,000 docs is not the same as 2,500,000 docs.

FastAPI With LangChain and MongoDB

MongoDB Guests — Fri, 17 Apr 2026 08:48:54 +0000

This article was written by Carlos Barboza.

I'm happy to welcome you to this tutorial! This demonstration provides a fully functional example of integrating FastAPI, LangChain, and MongoDB, and I'm eager to share it with all technology enthusiasts.

The value of this guide is that it provides a working example—you can copy and paste the code, and it will run seamlessly without any further configuration. It will also give you a clear, practical overview of the kinds of applications you can build using these powerful tools.

I am developing a suggestion application designed to assist customers globally in generating new ideas. This involves providing personalized recommendations and checking for existing companies that offer comparable services.

Prerequisites

Python, virtualenv basics
Basic understanding of REST APIs
MongoDB basics (collections, documents)
Accounts/keys
- MongoDB Atlas or local MongoDB
- LLM provider API key

Project Architecture

FastAPI is a Python framework for building APIs quickly, efficiently, and with very little code.

It’s known for being fast (as the name implies), easy to use, and perfect for modern applications like AI, microservices, and backend APIs. In this example, it will handle the API queries for the backend.

MongoDB is a NoSQL, document-based database that stores data in flexible JSON-like documents instead of rigid tables like SQL. This makes it fast to develop, easy to scale, and great for evolving data structures. For our case, it will store all the information.

LangChain is a framework for building applications powered by LLMs (like GPT, LLaMA, Ollama, and Mistral) that lets you connect models with tools, memory, external data, and multi-step reasoning. It provides the necessary tools in Python for utilizing AI models.

Project Setup

MongoDB Atlas

The simplest way to obtain a MongoDB instance for this guide is to register on the official site. This will provide you with an M0 instance, which is entirely sufficient for executing this guide.

Docker

Should you not have Docker installed, the official link provides installation instructions based on your operating system.

The docker-compose file is configured to set up a MongoDB container. This container uses the official version 7.0, is named mongodb, and uses the credentials root as the username and example as the password. Furthermore, a volume is defined to ensure data persistence across container reloads.

version: "3.3"
services:
  mongo:
    image: mongo:7.0
    container_name: mongodb
    restart: unless-stopped
    ports:
      - "27017:27017"
    environment:
      MONGO_INITDB_ROOT_USERNAME: root
      MONGO_INITDB_ROOT_PASSWORD: example
    volumes:
      # Named volume for persistent data
      - mongo_data:/data/db

volumes:
  mongo_data:

Execute the following commands to execute the docker:

docker compose up -d

Finally, you can connect to the MongoDB instance using the connection string. In your case, it’s mongodb://root:example@localhost:27017.

LLMs

Large language models (LLMs) are AI models trained on massive amounts of text that can understand language, generate text, write code, answer questions, and hold conversations like a human.

They don’t think—they predict the next word intelligently based on patterns they've learned. For our case, we are going to use Ollama so you can execute the example on your local machine.
Alternatively, you may choose to use a different LLM, such as OpenAI, though this may incur additional costs.

Ollama installation

Depending on your operating system, you must follow the official instructions. At the end of the day, you should be able to execute Ollama commands in your terminal.

ollama list # Shows all the models available in your terminal
ollama pull llama3.2 # Download the llama3.2 model

Development

Requirements.txt

These are all the libraries needed for the project:

aiohappyeyeballs==2.6.1
aiohttp==3.13.2
aiosignal==1.4.0
annotated-doc==0.0.4
annotated-types==0.7.0
anyio==4.12.0
attrs==25.4.0
bcrypt==3.2.2
certifi==2025.11.12
cffi==2.0.0
charset-normalizer==3.4.4
click==8.3.1
colorama==0.4.6
cryptography==46.0.3
dataclasses-json==0.6.7
distro==1.9.0
dnspython==2.8.0
ecdsa==0.19.1
email-validator==2.3.0
fastapi==0.123.0
frozenlist==1.8.0
greenlet==3.2.4
h11==0.16.0
httpcore==1.0.9
httpx==0.28.1
httpx-sse==0.4.3
idna==3.11
jiter==0.12.0
jsonpatch==1.33
jsonpointer==3.0.0
langchain==1.1.0
langchain-classic==1.0.0
langchain-community==0.4.1
langchain-core==1.1.0
langchain-openai==1.1.0
langchain-text-splitters==1.0.0
langgraph==1.0.4
langgraph-checkpoint==3.0.1
langgraph-prebuilt==1.0.5
langgraph-sdk==0.2.10
langsmith==0.4.49
marshmallow==3.26.1
motor==3.7.1
multidict==6.7.0
mypy_extensions==1.1.0
numpy==2.3.5
openai==2.8.1
orjson==3.11.4
ormsgpack==1.12.0
packaging==25.0
passlib==1.7.4
propcache==0.4.1
pyasn1==0.6.1
pycparser==2.23
pydantic==2.12.5
pydantic-settings==2.12.0
pydantic_core==2.41.5
pymongo==4.15.4
python-dotenv==1.2.1
python-jose==3.5.0
python-multipart==0.0.20
PyYAML==6.0.3
regex==2025.11.3
requests==2.32.5
requests-toolbelt==1.0.0
rsa==4.9.1
six==1.17.0
sniffio==1.3.1
SQLAlchemy==2.0.44
starlette==0.50.0
tenacity==9.1.2
tiktoken==0.12.0
tqdm==4.67.1
typing-inspect==0.9.0
typing-inspection==0.4.2
typing_extensions==4.15.0
urllib3==2.5.0
uvicorn==0.38.0
xxhash==3.6.0
yarl==1.22.0
zstandard==0.25.0

Environment file

It’s recommended to have an .env file with sensitive information like the MongoDB URI, JWT Secret Key, and other settings. This file should be excluded from the git.ignore and must be configured on the main location.

MONGO_URI=mongodb+srv://admin:admin@atlascluster.f3spasq.mongodb.net/
MONGO_DB_NAME=fastapi_demo
JWT_SECRET_KEY=super-secret-key
JWT_ALGORITHM=HS256
ACCESS_TOKEN_EXPIRE_MINUTES=60
OLLAMA_HOST=http://host.docker.internal:11434 # Only if fastAPI is running with docker

.gitignore

# ---------------------
# Environment variables
# ---------------------
.env
.env.local
.env.*.local

# ---------------------
# Python
# ---------------------
__pycache__/
*.pyc
*.pyo
*.pyd
*.pdb
*.pkl
*.db

# Virtual environments
venv/
.venv/

# Byte-compiled
*.py[cod]
*$py.class

# ---------------------
# FastAPI / Uvicorn logs
# ---------------------
*.log

# ---------------------
# IDE & editor files
# ---------------------
.vscode/
.idea/
*.swp

# ---------------------
# OS files
# ---------------------
.DS_Store
Thumbs.db

# ---------------------
# Docker
# ---------------------
/data/
docker-data/
*.pid

# ---------------------
# Optional cache folders
# ---------------------
cache/
logs/

Schema folder

chat.py
from pydantic import BaseModel

class ChatRequest(BaseModel):
    question: str

class ChatResponse(BaseModel):
    answer: str
    used_context: bool

item.py
from pydantic import BaseModel

class Item(BaseModel):
    id: str
    name: str
    price: float

user.py
from pydantic import BaseModel, EmailStr

class UserCreate(BaseModel):
    email: EmailStr
    password: str

class UserPublic(BaseModel):
    id: str
    email: EmailStr

class Token(BaseModel):
    access_token: str
    token_type: str = "bearer"

class UserInDB(BaseModel):
    id: str
    email: EmailStr
    hashed_password: str

notes.py
from pydantic import BaseModel

class NotesCreate(BaseModel):
    text: str

class NotesInDB(BaseModel):
    id: str
    user_id: str
    text: str

Core folder

config.py
import os

MONGO_URI = os.getenv("MONGO_URI", "mongodb+srv://admin:admin@atlascluster.f3spasq.mongodb.net/")
MONGO_DB_NAME = os.getenv("MONGO_DB_NAME", "fastapi_demo")

JWT_SECRET_KEY = os.getenv("JWT_SECRET_KEY", "change-me")
JWT_ALGORITHM = os.getenv("JWT_ALGORITHM", "HS256")
ACCESS_TOKEN_EXPIRE_MINUTES = int(os.getenv("ACCESS_TOKEN_EXPIRE_MINUTES", "60"))

llm.py
import os
from langchain_community.chat_models import ChatOllama

OLLAMA_HOST = os.getenv("OLLAMA_HOST", "http://localhost:11434")

llm = ChatOllama(
    model="llama3.2"
    ,base_url=OLLAMA_HOST,
)

security.py
from datetime import datetime, timedelta, timezone
from typing import Optional

from fastapi.security import OAuth2PasswordBearer
from jose import jwt
from passlib.context import CryptContext

from app.core.config import JWT_SECRET_KEY, JWT_ALGORITHM, ACCESS_TOKEN_EXPIRE_MINUTES

pwd_context = CryptContext(schemes=["bcrypt"], deprecated="auto")
oauth2_scheme = OAuth2PasswordBearer(tokenUrl="auth/login")

def hash_password(password: str) -> str:
    return pwd_context.hash(password)

def verify_password(plain_password: str, hashed_password: str) -> bool:
    return pwd_context.verify(plain_password, hashed_password)

def create_access_token(
    data: dict,
    expires_delta: Optional[timedelta] = None,
) -> str:
    to_encode = data.copy()
    expire = datetime.now(timezone.utc) + (
        expires_delta or timedelta(minutes=ACCESS_TOKEN_EXPIRE_MINUTES)
    )
    to_encode.update({"exp": expire})
    encoded_jwt = jwt.encode(to_encode, JWT_SECRET_KEY, algorithm=JWT_ALGORITHM)
    return encoded_jwt

db folder

from motor.motor_asyncio import AsyncIOMotorClient
from app.core import config as core_config

client = AsyncIOMotorClient(core_config.MONGO_URI)
db = client[core_config.MONGO_DB_NAME]

users_collection = db["users"]
notes_collection = db["notes"]
items_collection = db["items"]

depsFolder

auth.py
from typing import Optional, Annotated

from fastapi import Depends, HTTPException, status
from jose import JWTError, jwt

from app.core.config import JWT_SECRET_KEY, JWT_ALGORITHM
from app.core.security import oauth2_scheme, hash_password, verify_password
from app.db.mongo import users_collection
from app.schemas.user import UserInDB, UserCreate

async def get_user_by_email(email: str) -> Optional[UserInDB]:
    doc = await users_collection.find_one({"email": email})
    if not doc:
        return None
    return UserInDB(
        id=str(doc["_id"]),
        email=doc["email"],
        hashed_password=doc["hashed_password"],
    )

async def create_user(user: UserCreate) -> UserInDB:
    existing = await get_user_by_email(user.email)
    if existing:
        raise HTTPException(
            status_code=status.HTTP_400_BAD_REQUEST,
            detail="Email already registered",
        )
    hashed = hash_password(user.password)
    res = await users_collection.insert_one(
        {"email": user.email, "hashed_password": hashed}
    )
    return UserInDB(id=str(res.inserted_id), email=user.email, hashed_password=hashed)

async def get_current_user(
    token: Annotated[str, Depends(oauth2_scheme)]
) -> UserInDB:
    credentials_exception = HTTPException(
        status_code=status.HTTP_401_UNAUTHORIZED,
        detail="Could not validate credentials",
        headers={"WWW-Authenticate": "Bearer"},
    )
    try:
        payload = jwt.decode(token, JWT_SECRET_KEY, algorithms=[JWT_ALGORITHM])
        email: str = payload.get("sub")
        if email is None:
            raise credentials_exception
    except JWTError:
        raise credentials_exception

    user = await get_user_by_email(email)
    if user is None:
        raise credentials_exception
    return user

Routers folder

auth.py
from typing import Annotated

from fastapi import APIRouter, Depends, HTTPException, status
from fastapi.security import OAuth2PasswordRequestForm

from app.core.security import create_access_token, verify_password
from app.deps.auth import create_user, get_user_by_email, get_current_user
from app.schemas.user import UserCreate, UserPublic, Token, UserInDB

router = APIRouter(prefix="/auth", tags=["auth"])

@router.post("/register", response_model=UserPublic)
async def register(user: UserCreate):
    new_user = await create_user(user)
    return UserPublic(id=new_user.id, email=new_user.email)

@router.post("/login", response_model=Token)
async def login(form_data: Annotated[OAuth2PasswordRequestForm, Depends()]):
    # I am using email as the username field in the OAuth2 form.
    user = await get_user_by_email(form_data.username)
    if not user or not verify_password(form_data.password, user.hashed_password):
        raise HTTPException(
            status_code=status.HTTP_401_UNAUTHORIZED,
            detail="Incorrect email or password",
        )
    access_token = create_access_token(data={"sub": user.email})
    return Token(access_token=access_token)

@router.get("/me", response_model=UserPublic)
async def get_me(
    current_user: Annotated[UserInDB, Depends(get_current_user)]
):
    return UserPublic(id=current_user.id, email=current_user.email)

chat.py
from typing import Annotated

from fastapi import APIRouter, Depends
from langchain_core.messages import HumanMessage, SystemMessage

from app.core.llm import llm
from app.db.mongo import notes_collection
from app.deps.auth import get_current_user
from app.schemas.user import UserInDB
from app.schemas.chat import ChatRequest, ChatResponse

router = APIRouter(prefix="/chat", tags=["chat"])

@router.post("/", response_model=ChatResponse)
async def chat_with_ai(
    body: ChatRequest,
    current_user: Annotated[UserInDB, Depends(get_current_user)],
    # Uses LangChain + OpenAI + some context from MongoDB.
):
    print(current_user)

    notes_cursor = (
        notes_collection.find({"user_id": current_user.id})
        .sort("_id", -1)
        .limit(3)
    )
    notes = await notes_cursor.to_list(length=3)

    context_text = "\n".join(
        note.get("text", "") for note in notes if note.get("text")
    )
    used_context = bool(context_text)

    system_prompt = (
        "You are an assistant helping the user with their notes."
        " "
        "Use the following context if relevant; if not, just answer normally.\n\n"
        f"Context:\n{context_text or '(no context available)'}"
    )
    messages = [
        SystemMessage(content=system_prompt),
        HumanMessage(content=body.question),
    ]

    response = await llm.ainvoke(messages)

    return ChatResponse(answer=response.content, used_context=used_context)

items.py
from typing import List, Annotated

from fastapi import APIRouter, Depends

from app.db.mongo import items_collection
from app.schemas.item import Item
from app.schemas.user import UserInDB
from app.deps.auth import get_current_user

router = APIRouter(prefix="/items", tags=["items"])

@router.get("/", response_model=List[Item])
async def list_items(
    current_user: Annotated[UserInDB, Depends(get_current_user)]
):
    # Example async Mongo query using Motor.
    # Demo: seed items if collection is empty
    count = await items_collection.count_documents({})
    if count == 0:
        await items_collection.insert_many(
            [
                {"name": "Book", "price": 10.5},
                {"name": "Laptop", "price": 999.99},
            ]
        )

    docs = await items_collection.find().to_list(length=100)
    return [
        Item(id=str(doc["_id"]), name=doc["name"], price=float(doc["price"]))
        for doc in docs
    ]

notes.py
from typing import Annotated

from fastapi import APIRouter, Depends
from langchain_core.messages import HumanMessage, SystemMessage

from app.core.llm import llm
from app.db.mongo import notes_collection
from app.deps.auth import get_current_user
from app.schemas.user import UserInDB
from app.schemas.chat import ChatRequest, ChatResponse
from app.schemas.notes import NotesCreate, NotesInDB

router = APIRouter(prefix="/notes", tags=["notes"])

@router.post("/", response_model=NotesInDB)
async def create_note(body: NotesCreate,current_user: Annotated[UserInDB, Depends(get_current_user)]):
    print(current_user)

    note = await notes_collection.insert_one({"text": body.text, "user_id": current_user.id})
    return NotesInDB(id=str(note.inserted_id), text=body.text, user_id=current_user.id)

Main.py

from fastapi import FastAPI

from app.routers import auth, items, chat, notes

app = FastAPI(title="FastAPI + MongoDB + JWT + LangChain Example")

@app.get("/")
async def root():
    return {"message": "FastAPI + MongoDB + JWT + LangChain example"}

# Include routers
app.include_router(auth.router)
app.include_router(items.router)
app.include_router(chat.router)
app.include_router(notes.router)

Application execution

On the main folder, you can execute the following command:

uvicorn app.main:app --reload
INFO: Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit)

If you access the page http://127.0.0.1:8000/docs#, you can find the Swagger documentation created by default.

Users creation

You can execute the following command to create a new user:

curl -X 'POST' \
  'http://127.0.0.1:8000/auth/register' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/json' \
  -d '{
  "email": "user@example.com",
  "password": "string" 
}'

Or if you have vscode Rest Api Extension you can execute
POST http://127.0.0.1:8000/auth/register
Accept: application/json
Content-Type: application/json

{
  "email": "user@example.com",
  "password": "string"
}

The above command will create a new user with hashed password:

Users authentication

In order to authenticate the user and retrieve the bearer token, please execute the following command:

curl -X 'POST' \
  'http://127.0.0.1:8000/auth/login' \
  -H 'accept: application/json' \
  -H 'Content-Type: application/x-www-form-urlencoded' \
  -d 'grant_type=password&username=user%40example.com&password=string&scope=&client_id=string&client_secret=string'
Rest API extension
POST http://127.0.0.1:8000/auth/login
Content-Type: application/x-www-form-urlencoded
accept: application/json

grant_type=password&username=test@example.com&password=123&scope=&client_id=string&client_secret=string

After a successful authentication, you will receive the following output. The token is important to authenticate on the remaining routes.

HTTP/1.1 200 OK
date: Tue, 02 Dec 2025 17:40:11 GMT
server: uvicorn
content-length: 180
content-type: application/json
Connection: close

{
  "access_token": "eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJzdWIiOiJ0ZXN0QGV4YW1wbGUuY29tIiwiZXhwIjoxNzY0NzAwODExfQ.riXj1vvLZSyzyU4T7kG3ygYaUU8sgvgCjK_Os_uggvo",
  "token_type": "bearer"
}

Idea creation

The main purpose of the application is to make suggestions based on ideas that the end users have. The following request will create those notes.

POST http://127.0.0.1:8000/notes/
Content-Type: application/json
Authorization: Bearer {{token}}

{
  "text": "testing123"
}

Chat interaction

The application now offers advice based on the notes stored by users in the database. To enable interaction with the LLM, we define the chat's behavior in the chat.py file. This definition is an example of prompt engineering, and this is the section where you can modify the prompt's language and content, if needed.

system_prompt = (
        "You are an assistant helping the user with their notes."
        " "
        "Use the following context if relevant; if not, just answer normally.\n\n"
        f"Context:\n{context_text or '(no context available)'}"
    )

Execution example:

POST http://127.0.0.1:8000/chat/
Content-Type: application/json
Authorization: Bearer {{token}}

{
  "question": "What i need to do"
}

Execution output:

{
  "answer": "To help you achieve your goals of saving more money and finishing your project, here are some actionable steps:\n\n**Saving More Money:**\n\n1. **Track your expenses**: Write down every single transaction, no matter how small, for a week or two to understand where your money is going.\n2. **Create a budget**: Based on your income and expenses, allocate a specific amount for savings each month.\n3. **Automate your savings**: Set up an automatic transfer from your checking account to your savings account.\n4. **Cut back on unnecessary expenses**: Identify areas where you can cut back on unnecessary spending, such as dining out or subscription services.\n5. **Consider a savings challenge**: Try a savings challenge like the \"52-week savings challenge\" where you save an amount equal to the number of the week (e.g., Week 1: Save $1, Week 2: Save $2 etc.).\n\n**Finishing Your Project:**\n\n1. **Break down your project into smaller tasks**: Divide your project into manageable tasks to make it feel less overwhelming.\n2. **Create a schedule**: Set a specific deadline for each task and stick to it.\n3. **Prioritize your tasks**: Focus on the most critical tasks first, and then move on to less important ones.\n4. **Remove distractions**: Identify potential distractions (e.g., social media, email, phone notifications) and eliminate them while you work.\n5. **Take regular breaks**: Take short breaks to recharge and maintain productivity.\n\nWhich of these steps do you want to start with?",
"used_context": true
}

Docker (optional)

Our application is currently functional. However, to execute it in a different environment, we must install the necessary dependencies. Docker can facilitate and streamline this deployment process.

Create Dockerfile and paste the following code:

# Dockerfile
FROM python:3.11-slim

# Set work directory inside the container
WORKDIR /app

# Install system dependencies (optional but useful for many libs)
RUN apt-get update && apt-get install -y --no-install-recommends \
    build-essential \
    && rm -rf /var/lib/apt/lists/*

# Copy only requirements first (for better Docker cache)
COPY requirements.txt .

# Install Python dependencies
RUN pip install --no-cache-dir -r requirements.txt

# Copy the rest of your project
COPY . .

# Expose the FastAPI port
EXPOSE 8000

# Default command: run uvicorn
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8000"]

Create the docker-compose.yml:

version: "3.8"
services:
  fastapi:
    build: .
    container_name: fastapi-langchain
    ports:
      - "8000:8000"
    env_file:
      - .env # optional: if you have one
    extra_hosts:
      - "host.docker.internal:host-gateway"
    restart: unless-stopped

Now, we can build and execute our docker:

docker-compose build
docker-compose up -d

Conclusion

This tutorial has provided a complete, practical demonstration of building a modern, data-driven application by seamlessly integrating FastAPI, LangChain, and MongoDB.

We have successfully covered:

FastAPI for creating a robust, fast, and documented REST API, including user registration and JWT-based authentication.
MongoDB (using Motor) for asynchronous and scalable document storage, handling user data and application-specific notes.
LangChain (with Ollama) for enabling powerful AI capabilities, using the stored user notes as dynamic context to generate personalized, helpful suggestions and answers.

The example application, an idea generation assistant, serves as a clear blueprint for how to leverage this stack for complex projects requiring data persistence, secure user access, and cutting-edge large language model integration. By using a local LLM via Ollama, we also ensured the entire stack is executable on a local machine, making it accessible for rapid prototyping and development.

This integration strategy—combining the speed of FastAPI, the flexibility of MongoDB, and the intelligence of LangChain—is a highly effective pattern for developing the next generation of intelligent web applications. We hope this guide serves as a solid foundation for your future AI-powered projects.

Async PyMongo in FastAPI

MongoDB Guests — Tue, 14 Apr 2026 14:53:07 +0000

This tutorial was written by Gourav Bhabesh.

1. Introduction

Why modern APIs need asynchronous I/O

Modern APIs need to be robust and be able to serve calls on the go, and minimize the instances of timeouts and server overloads while providing a good user experience during long-running jobs. Asynchronous I/O can solve one of the key issues making modern APIs: staying fast and reliable. It lets the user know the progress of their task while the API handles the requests in the background and is available for other requests. One of the key challenges in the API calls to the database is that some queries can be returned in a few milliseconds but others can take seconds (even with well indexed queries). Without asynchronous I/O in the API, this can lead to long request queues.

You can imagine this as a bakery storefront serving different goods at the same time to multiple customers. We want the orders to be filled asap without the queue becoming too big. In our example, we have multiple servers (scalable infrastructure) who take the orders which can take varying time to process and each customer with longer order times is given a status bar to know where their order’s current status. So a customer with an order that can be instantly filled (picking up a cookie) does not have to wait behind such a customer with a long processing order (baking a cheese cake).

FastAPI’s design philosophy around async

The building block of the FastAPI is ASGI (Asynchronous Server Gateway Interface) standard, unlike the older WSGI (Web Server Gateway Interface) which was synchronous in nature by default. In short, it uses Python’s async/await syntax, yielding control to the event loop, allowing servers to process other requests in the meantime, rather than blocking an entire thread. Using an event loop lets FastAPI handle thousands of concurrent requests with fewer resources. This leads to higher throughput per thread and lower latency. This automatically encourages the developer to use the async ecosystem to fully leverage the performance benefits of async programming.

Database bottlenecks & why async DB drivers matter

One of the key challenges that any database faces is the uncertainty in response times, in which the queries will be answered by the database server. Some queries can take milli-seconds and some can take seconds. If the architecture is not well built to handle this uncertainty, it can cause severe database bottlenecks. To handle this challenge, async database drivers ensure that longer requests do not become a bottleneck for other incoming requests. Initially, the async MongoDB driver was built in the library Motor. But in late 2023, Pymongo team officially decided to build the native async solution, with the goal to replace Motor as the long-term solution.

Introducing the new Async API in PyMongo

The Pymongo team has been working on the goal to replace the Motor library to give native async support in the pymongo library itself. This gives confidence on the client side to use the official library with async support. The async support was first provided in the version 4.9 in Pymongo and it has been further stabilised to the current version of 4.15.0. The new API capabilities bring the asyncio support directly to the official MongoDB driver, allowing developers to use awaitable database operations without external libraries like Motor. Combined with the FastAPI framework, which already supports await/async capability, Pymongo can now be natively used to build high-throughput, non-blocking services.

2. What is PyMongo?

Overview of the official MongoDB Python driver

MongoDB has provided Pymongo as the official Python driver to allow the applications to communicate with MongoDB clusters, providing production-tested API to perform CRUD operations, manage indexes, run aggregation pipelines and handle advanced features such as transactions and change streams. Pymongo was traditionally built on a synchronous I/O model, meaning every database call made will block the executing thread until the MongoDB server does not return a response leading to database bottlenecks.

Traditional synchronous PyMongo behavior

The key behaviours of the traditional Pymongo library, which we should be aware of, to appreciate the addition of the native support of asynchronous, are that synchronous operations block Python threads until the operation completes or a network error or timeout occurs.. This leads to another behaviour - sequential execution, blocking all requests until and unless the previous request is not complete. This made the MongoClient “Thread-Safe” allowing for multi-threading applications to be written, but each thread still performs blocking database operations.

In our example, of the baking shop, it would be like each server is concurrently serving the customers but each server is waiting for the order to complete rather than handing a token to wait, while serving the next customer request which might be served in much less time.

Blocking vs non-blocking I/O explained simply

Blocking I/O means the application must wait for the database operation to finish or network timeout before it will allow anything else to continue. On the other hand, non-blocking I/O allows requests to continue flowing to the database while previous requests are still being processed. In addition, the async drivers provide a way to receive the status of the request, giving power to the developer to handle long operations and create a better user-experience.

Limitations of synchronous DB operations in async frameworks

Using synchronous DB operations in the async framework is one of the key mistakes made by many clients of MongoDB, part of the reason was the non-availability of official Python drivers that actually supported asynchronous behavior. If the synchronous Pymongo driver is used inside an async framework such as FastAPI, these blocking calls freeze the event loop, preventing the server from processing other incoming requests and significantly reducing concurrency. As a result, applications that rely on synchronous database drivers inside async frameworks often experience increased latency, request queuing, and poor scalability under load, even if the rest of the code is written using async and await.

Imagine our baking shop upgraded its entire framework to be non-synchronous. The cooks can cook multiple dishes at the same time, but the servers are still the same, processing sequential orders and not taking new orders until the previous order has been delivered. The cook is having an easy day!!

3. What is Async PyMongo?

Overview of PyMongo’s new async API (`pymongo.async_`)

The official MongoDB team started working on the native-async support in the Pymongo driver in version4.9, where it was first released as beta version in late 2023. As of writing this article, the pymongo version 4.15 has been released and has been stable since version 4.13. With the pymongo.With async support, we can create an AsyncMongoClient object similar to the MongoClient object we created in the previous synchronous call. The Client provides fully awaitable database operations such as insert_one, find_one, update_one, and aggregation pipelines, allowing MongoDB I/O to integrate natively with Python’s event loop.

Key advantages over synchronous API

Compared to the traditional synchronous API, async PyMongo significantly improves scalability and responsiveness under concurrent workloads. Because database calls no longer block the event loop, asynchronous framework applications can continue handling new requests while waiting for MongoDB responses, resulting in lower latency, higher throughput, and better CPU utilization—especially in high-traffic microservices

Relationship to Motor (Motor’s role before async PyMongo existed)

Before async Pymongo, motor was the de facto library to be used when developers needed asynchronous behavior when communicating with the MongoDB clusters. Motor did this by exposing a coroutine-based API, allowing applications to switch context and perform other tasks while waiting for I/O operations. However, Motor was not a separate driver from scratch; it was designed as a wrapper around the core synchronous PyMongo library. It used Tornado’s or asyncio’s event loop, wrapping Pymongo’s operations asynchronously, effectively allowing the main event loop to remain unblocked. Motor was specifically created to support asynchronous web frameworks, initially Tornado and later adding support for asyncio. Performance comparison between motor and async pymongo can be found on the official documentation here.

How to migrate from motor to async-pymongo and sync-pymongo to async-pymongo

Migrating from the motor-based asynchronous library to the async-pymongo library is simple and is well documented by MongoDB here. In simple terms, one needs to change the driver call to the database using the AsyncMongoClient and add the keyword await in front of the database operations.

When migrating from sync Pymongo to async Pymongo, keep the following important points from the official MongoDB documentation in mind:
· To convert an AsyncCursor to a list, you must use the asynchronous cursor.to_list() method.
· The AsyncCollection.find() method in the PyMongo Async API is synchronous, but returns an AsyncCursor.
To iterate through the cursor, you must use an async for loop.
· The AsyncMongoClient object does not support the connect keyword argument.
· You cannot share AsyncMongoClient objects across threads or event loops.
· To access a property or method of a result returned by an asynchronous call, you must properly wrap the call in parentheses, as shown in the following example:

id = (await posts.insert_one(doc)).inserted_id

4. Key Features of Async PyMongo

True async interface with awaitable operations

The biggest advantage of the async Pymongo library for developers is its awaitable operations. Functions such as insert_one(), find_one(), update_one() and aggregate() return objects that can wait rather than blocking the results. This fundamentally decouples the event loop from the operation, meaning the database operation running on the database side yields the control back to the event loop, allowing FastAPI to process the other incoming requests without blocking the thread.

In our baking shop example, the server has been decoupled from the chef, while the previous order is getting cooked in the kitchen, the server takes in a new order without ceasing the operation and blocking the thread and allowing for incoming requests to be fulfilled !! This model of working is fundamentally different from the traditional synchronous PyMongo, where every database call ties up a worker thread until completion, blocking the thread from serving the next call when under load.

Connection pooling behavior

The connection pool behaviour is quite an interesting one here. On the surface it appears both the synchronous and the asynchronous Pymongo connection-poling implementation look the same (parameters available - maxPoolSize, minPoolSize, waitQueueTimeoutMS etc.) and indeed rely on the same MongoDB C driver. However, async applications are actually able to spawn thousands of concurrent operations and grab many connections from the pool at once, whereas, in the synchronous application each request occupies a python worker thread, holding a connection from the pool waiting for MongoDB. This naturally limits concurrency as the application is limited by the number of workers, effectively meaning the full pool size can be never used optimally.

Imagining in our world of baking shop, we have an army of chefs to process the orders, the async model allows the server to grab hold of any available chef to accept a new order and move on to the next one, whereas in the synchronous setup the server will wait for the chef to finish the previous order before moving to the new order.

Async cursors and non-blocking iteration

One of the primary challenges that any database faces is returning results to query with a large result set. The traditional PyMongo driver already returns a cursor object rather than loading all the documents into the memory at once. In async PyMongo, these cursors are fully asynchronous in nature as well, resulting in iteration without blocking the event loop. Developers now have the choice of using the await cursor.to_list() or iterating over the cursor object using the async for. This will allow data to be streamed in batches while the application is handling incoming requests. This feature is of significant importance when we have endpoints that return large datasets, background processing jobs and real-time analytics workloads.

5. What is FastAPI?

FastAPI’s async-first architecture

FastAPI is a high-performance web framework that is production-ready and designed for building APIs in python, with roots embedded in asynchronous programming. It embraces Python’s asyncio model as its core principle rather than treating it as optional. This design choice gives FastAPI leverage for I/O bound workloads such as db access, external API calls and real-time data streaming.

Other traditional web frameworks default to synchronous request handling, whereas FastAPI encourages using asynchronous endpoints (async def) wherever possible. When an endpoint is declared using the async keyword, FastAPI uses the event loop execution rather than a worker thread. Now as long as those requests involve the non-blocking I/O operations, a single worker process can handle many requests concurrently. This is where FastAPI’s async-first architecture shines. When the entire request path - including the database access - is non blocking, each request becomes super efficient. Any stack in the request that blocks the event loop will make the FastAPI’s concurrency model diminish significantly.

Event loop and concurrency model

At the heart of FastAPI’s concurrency model is Python’s asyncio event loop. The event loop manages the execution of many coroutines, switching between them whenever one is waiting for I/O, such as a network response from MongoDB. This cooperative multitasking enables a single process to efficiently handle thousands of simultaneous requests without spawning thousands of threads. However, this model only works as intended when all long-running operations are awaitable. If a synchronous operation blocks the event loop — for example, a traditional synchronous PyMongo database call — the entire server becomes unresponsive to other requests until that operation completes.

Why FastAPI pairs naturally with non-blocking database drivers

As explained above the core principle of FastAPI is to yield the control back to the event loop and any blocking I/O operation in the chain of the operations sent by the request will be like throwing a wrench on the gears. With the pairing of the non-blocking db driver such as the async pymongo, when a FastAPI endpoint awaits a MongoDB operation, control is yielded back to the event loop, allowing for the next set of requests to be processed in parallel. With the alignment between the db driver and the base principle of FastAPI to be async, we eliminate the friction and make the full request truly asynchronous in nature and hence enable FastAPI to scale efficiently under high concurrency.

Surprising performance differences when mixing sync DB code into async routes

A common pitfall for developers writing code in FastAPI is mixing the synchronous PyMongo calls inside the async def endpoints. On the surface it may appear to work, but it breaks the core principle of the FastAPI to be truly asynchronous. One single blocking db call and the entire event loop gets stalled as other requests will have to queue behind it - even when the server has many workers available.

In practice, this means that an API written with FastAPI but using synchronous PyMongo can perform worse under load than a fully synchronous application with multiple worker threads. By contrast, pairing FastAPI with Async PyMongo preserves the framework’s intended concurrency benefits, leading to better throughput, lower latency for typical requests, and more predictable scaling behavior. In large applications, having the capability to predict the scalability of infrastructure and have a control on the concurrency is of paramount importance.

6. Why Use Async PyMongo with FastAPI?

Avoid event-loop blocking

Using FastAPI with Asynchronous Pymongo correctly is essential if we want to avoid event-loop blocking. The requests from the async-first framework like FastAPI, share the same event-loop within each worker process. Meaning any operation in this request if becomes synchronized will lead to blocking of the entire event-loop, which high-concurrency systems want to avoid at all cost.

By using Async Pymongo, each db operation becomes awaitable rather than blocking in nature, ensuring a slow db operation does not freeze the entire application, allowing other requests to get processed.

Handle thousands of concurrent requests

FastAPI is capable of handling thousands of concurrent requests per worker process, but designing the entire chain of operations in an asynchronous manner is a must to achieve this. One of the primary mistakes that are made in the coding side is missing the async and await keywords in the db operation call, this breaks the core principle of FastAPI leading to slowness. With correct implementation we can make the infrastructure more cost-effective and efficient as compared to a purely synchronous system which would not be scalable in the long run as more threads are required to handle the huge volume of requests.

Reduce latency under load

With modern tech stacks, both horizontal and vertical scaling is a must and systems built with such capabilities can withstand the test of time. And today's system should be capable enough to handle varying loads. With FastAPI - async nature is built in-core principle of the library itself and when combined with Async Pymongo it allows for the potential blocking db operations to be handled asynchronously. This leads to lower median latency for requests, especially I/O intensive ones. In section 8, we will see how the sync/async jobs specifically prove this point, when we compare the median latency of thousands of concurrent requests.

Improve throughput for high-traffic microservices

In the world of microservices where responsiveness and scalability are critical Async Pymongo dramatically improves the performance by removing the blocker of such requests to the db layer. Combined with async frameworks such as FastAPI, Async Pymongo is superbly placed to serve real-time applications, analytics dashboards or user-facing services.

Better CPU utilization and scaling on limited infrastructure

One of the vital nerves of any IT infrastructure is cost-efficiency and with Devops team’s mantra - “doing more with less”, fits perfectly with the Async Pymongo. Making the most of the available resources, async architecture uses the same thread without blocking incoming requests by handing over the control back to the single event loop. Instead of maintaining hundreds or thousands of idle threads waiting on the I/O, FastAPI with Async Pymongo exploits this single event loop allowing it to switch between tasks as needed. In cloud based infrastructure, where compute resources are directly tied to cost, it can lead to meaningful savings without sacrificing performance.

7. Async CRUD Operations with Examples

This section demonstrates the common MongoDB operations asynchronously using the pymongo.async_ inside FastAPI.

All examples assume:

from pymongo.async_ import AsyncMongoClient 
from bson import ObjectId 
client = AsyncMongoClient (MONGO_URI) db = client["appdb"] 
users = db["users"]

Insert one / insert many (async versions)

Insert one document example:

@app.post("/users") 
async def create_user(user: dict):
  result = await users.insert_one (user) 
  return {"inserted_id": str(result.inserted_id)}

Insert many documents example:

@app.post("/users/bulk") 
async def create_many(users_in: list[dict]): 
  result = await users.insert_many (users_in) 
  return {"inserted_ids": [str(i) for i in result.inserted_ids]}

Find one / find many with async cursors

Find one document example:

@app.get("/users/{email}")
async def get_user(email: str): 
 return await users.find_one ({"email": email})

Find many documents example:

@app.get("/users") 
async def list_users():
  cursor = users.find ({}) 
  return await cursor.to_list(length=100)

Async pagination using cursors

@app.get("/users/page") 
async def paginated_users(last_id: str | None = None, limit: int = 20): 
  query = {"_id": {"$gt": ObjectId(last_id)}} if last_id else {} 
  cursor = users.find(query).sort("_id", 1).limit (limit) 
  results = await cursor.to_list(length=limit) 

  return { 
    "results": results,
    "next_cursor": str(results[-1]["_id"]) if results else None, 
  }

(Avoid skip() for large collections; use _id cursor pagination instead.)

Update and replace operations

Update one document example:

@app.patch("/users/{email}")
async def update_user(email: str, changes: dict):
  result = await users.update_one(
    {"email": email},
    {"$set": changes},
  ) 
  return {"matched": result.matched_count, "modified":
result.modified_count}

Replace one document example:

@app.put("/users/{email}")
async def replace_user(email: str, new_doc: dict):
  result = await users.replace_one({"email": email}, new_doc) 
  return {"matched": result.matched_count}

Delete one/many operations

Delete one document example:

@app.delete("/users/{email}") 
async def delete_user(email: str):
  result = await users.delete_one ({"email": email}) 
  return {"deleted": result.deleted_count}

Delete many documents example:

@app.delete("/users/inactive")
async def delete_inactive():
  result = await users.delete_many ({"active": False}) 
  return {"deleted": result.deleted_count}

Async aggregation pipelines

@app.get("/users/stats")
async def user_stats():
  pipeline = [
    {"$group": {"_id": "$country", ""count"": {"$sum": 1}}},
    {"$sort": {"count": -1}},
  ] 
cursor = await users.aggregate (pipeline)
return await cursor.to_list(length=100)

Error handling (duplicate keys, timeouts, network issues)

Duplicate keys:

from pymongo.errors import DuplicateKeyError
from fastapi import HTTPException

try: 
  await users.insert_one({"email": "a@b.com"})
except DuplicateKeyError:
  raise HTTPException(status_code=400, detail="User already exists")

Timeout/network issues:

from pymongo.errors import ExecutionTimeout, ConnectionFailure

try:
  await users.find_one({"email": "a@b.com"})
except ExecutionTimeout:
  raise HTTPException(status_code=504, detail="DB timeout")
except ConnectionFailure:
  raise HTTPException(status_code=503, detail="DB unreachable")

8. Performance Comparison: Sync PyMongo vs Async PyMongo

In this section, we break down the high-stakes battle between the traditional Synchronous (Sync) driver and the modern Asynchronous (Async) driver using the large-scale sample_airbnb dataset. This isn't just a syntax preference; it's the difference between a server that remains responsive under fire and one that collapses under its own weight.

Example endpoints and profiling results

To push these drivers to the limit, we implemented a Complex Aggregation task. Instead of a simple "Ping," we forced each request to filter high-rated Airbnb listings, sort them by price, and project complex nested fields.

Sync Logic: Utilizes list(users.aggregate(pipeline)), which occupies a physical thread for the duration of the database round-trip and data serialization.

Async Logic: Utilizes await (await users.aggregate(pipeline)).to_list().
The server yields control during the database "wait time," allowing the single-threaded event loop to process other incoming users.

Locust Benchmark setup

We subjected both implementations to a "Trial by Fire" using Locust:

User Load: Scaled up to 500 concurrent users.

Hardware Constraint: We intentionally limited the Sync thread pool to 1 thread to simulate extreme resource exhaustion and observe the "Queueing Effect."

Target: listingsAndReviews collection in MongoDB Atlas.

Understanding event-loop blocking

The results highlight a fundamental architectural divide. In the Sync model, when the thread pool is exhausted (in our case, limited to 1), every new user must wait for the person in front of them to finish their entire database operation. This creates a "Single Lane Bridge" bottleneck. In the Async model, even with thousands of users, the event loop acts as a high-speed scheduler. It sends the request to MongoDB and immediately moves to the next user, effectively "stacking" thousands of concurrent database operations without needing a thread for each one.

Graphs and metrics discussion (Locust latency plots)

Table: Sync stats showing the performance on one thread
Graph: Sync response times

Table: Async stats showing the performance

Graph: Async response times

Looking at our Locust metrics, the disparity is stark:
| Metric | Sync (1 Thread) | Async (Event Loop) |
|-------------------|-----------------|--------------------|
| Max RPS | ~48.8 | ~113.4 |
| Median Latency | ~8,700 ms | ~2,800 ms |
| 95th-Percentile | ~9,400 ms | ~4,100 ms |
| Throughput Driver | Thread-bound | DB / Network-bound |

Sync Graphs: Notice the staircase effect in response times.
As users increase, the latency climbs linearly because users are waiting in a queue. The RPS is "hard-capped" because the single thread can only process so many documents per second.

Async Graphs: The response times remain significantly lower and more stable.
Even as we ramped to 500 users, the 95th percentile hovered around 4 seconds compared to nearly 10 seconds for Sync. This proves that Async manages high-concurrency significantly better, even when the individual database query is complex.

Full working snippet for readers to run

Ready to see these results on your own machine?
We have provided a complete repository containing the FastAPI implementation, the AnyIO thread limiter configuration, and the Locust scripts used for these benchmarks.

Demo work - https://github.com/gouravmongodb/fastapi_pymongo_demo

9. Common Pitfalls & How to Avoid Them

Even with Async PyMongo, it is easy to introduce subtle performance or correctness issues if you are not careful. Below are the most common pitfalls developers encounter and how to avoid them.

Forgetting await on Database Calls

Because Async PyMongo methods are awaitable, forgetting await will not raise an immediate syntax error but will return a coroutine object instead of executing the operation. This can lead to silent failures or unexpected behavior in your API responses. Always double-check that every database operation inside an async def route is properly awaited.

Creating a New Client for Every Request

Instantiating AsyncMongoClient inside each request handler is a major anti-pattern. It creates unnecessary TCP connections, increases latency, and quickly exhausts resources under load. Instead, create a single shared client at application startup (using FastAPI lifespan or startup events) and reuse it across all requests.

Returning Non-JSON-Serializable Types

MongoDB types like ObjectId, datetime, or Decimal128 are not directly JSON serializable. Returning raw MongoDB documents from FastAPI will cause serialization errors. The recommended approach is to either convert these types to strings or map your documents to Pydantic models that handle serialization correctly.

Exhausting Connection Pools Under Heavy Load

Async applications can issue far more concurrent database operations than synchronous ones, which makes connection pool tuning critical. If maxPoolSize is too low, requests will start queuing inside the driver, increasing latency. Monitor Atlas metrics and adjust maxPoolSize, minPoolSize, and maxConnecting based on your workload.

Mixing Synchronous and Asynchronous Code

Using synchronous PyMongo calls inside async def routes (or vice versa) undermines FastAPI’s concurrency model and can block the event loop. Choose one model—preferably Async PyMongo for FastAPI—and stick to it consistently throughout your application.

10. Conclusion

Asynchronous database access is a cornerstone of modern, scalable API design. In an async-first framework like FastAPI, using a non-blocking database driver is not a luxury—it is a necessity for handling high concurrency, reducing latency, and making efficient use of resources.

With the stabilization of pymongo.async_ in PyMongo 4.13, Async PyMongo has become the recommended and officially supported way to use MongoDB with FastAPI. It provides a native, awaitable interface that aligns with FastAPI’s event loop, simplifies dependencies, and brings the future of the MongoDB Python ecosystem under a single driver.

Synchronous PyMongo still has a place in the ecosystem — particularly for simple scripts, batch jobs, or applications that are not built around async frameworks. However, for production web services, real-time applications, and microservices that must scale, Async PyMongo is the better choice.

Ultimately, performance depends not just on “sync vs async,” but on how you design your workload, tune your connection pool, index your collections, and scale your MongoDB cluster. When used thoughtfully, Async PyMongo with FastAPI offers a powerful, maintainable, and future-proof foundation for building high-performance Python APIs.

Research work

· https://pymongo.readthedocs.io/en/stable/changelog.html#changes-in-version-4-9-2024-09-18
· From Pymongo driver v4.9.0 introduced the beta version of async function.
· From Pymongo driver v4.13.0 - asynchronous API is now stable and no longer in beta.
· https://www.mongodb.com/docs/languages/python/pymongo-driver/current/reference/migration/#migrate-to-pymongo-async
· https://www.mongodb.com/docs/languages/python/pymongo-driver/current/reference/migration/#performance-benchmarks

DEV Community: MongoDB

Test Password Strength and Password History with TypeScript and MongoDB

The Prerequisites

Establish an API Foundation with Express and TypeScript

Create a Singleton MongoDB Class to Manage Database Connections

Test Password Strength with Regular Expressions and Zod

Store Passwords Safely in MongoDB with Password Change History

Testing the API Endpoints with cURL

Conclusion

Whistleblower for Database: Set up an internal informant who exposes performance

Whistleblower 1: Performance Informant

High CPU utilization?

Query <5 seconds?

Cache hit ratio <80%?

Whistleblower 2: Cost Informant

Storage >80%:

IOPS > Budget?

Unexpected auto scale?

Whistleblower 3: Reliability Informant

Replication lag >10 Seconds?

Oplog <24Hours?

Connection >80%?

Vacuum Seal Your MongoDB: Cuts Cost Down 50% Today

Online Archive

Table Compact

Resync Secondary:

TTL Index:

Query Optimization:

Flex Cluster:

Conclusion:

MongoDB Vector Search in Laravel: Finding the Unqueryable

Laravel vector search implementation

Prerequisites to run the repo

Connect the Laravel app to the MongoDB database

Connect to the database

3 steps to your MongoDB Vector Search in Laravel

Step 1: Generate vector embeddings for your data

Step 2: Create a vector index

Step 3: Perform a query

Use cases

Common questions

What happens if the model is updated, or if I change the embedding model?

Will vector search introduce high latency (slow down my queries) compared to my existing MySQL/full-text search?

Conclusion

My first “local” Vector Search: MongoDB community edition

How did I test this?

Real-Time Fraud Detection in Java with Kafka Streams and Vector Similarity

What We’re Building

The Challenge: Millisecond Decisions with Resilience Under Failure

The Strategy: Decoupling the System with Event-Driven Architecture

Stateful Checks Without a Database Round-Trip

Solving It with Kafka Streams

Stage 1: Guardrails, Fast Rule-Based Validation

Where Rule-Based Detection Falls Short

Stage 2: Vector Scoring, Behavioral Fraud Detection

From transactions to Vectors

Building the Baseline: Seeding the Fraud Pattern Collection

How This Collection Evolves

Running Vector Search with MongoDB

Performing the Search

Not All Matches Are Equal: Introducing the Threshold

The Full Architecture

Beyond Rules and Similarity

Source Code

How schema anti-patterns in MongoDB can cost you $$$$

Massive boundless array

Timeseries saved time

When multitenant became the villain

$lookup is designed badly

How MongoDB Executes a find() Query: A Complete Lifecycle Guide

The full journey of a find() query (quick overview)

Stage 1: The Client Sends a Query Over the Wire

Stage 2: Authentication Check

Stage 3: Authorization — Can You Read This Collection?

Stage 4: Query Parsing and BSON Validation

Stage 5: The Query Planner Enumerates Candidate Plans

Stage 6: Plan Cache Lookup — The Fast Path

Stage 7: Multi-Plan Trial — The Index Race

What about full collection scans (COLLSCAN)?

Stage 8: Index Scan (IXSCAN) or Collection Scan (COLLSCAN)

The full journey of a `find()` query (quick overview)

Overview of PyMongo’s new async API (`pymongo.async_`)