DEV Community: MongoDB

Streaming MongoDB Change Events to Amazon Redshift via Atlas Stream Processing and Kinesis

MongoDB Guests — Wed, 05 Aug 2026 13:13:42 +0000

This tutorial was written by Igor Alekseev.

Real-time data pipelines that move operational data into analytical systems are a cornerstone of modern architectures. In this post, I’ll walk through how to stream MongoDB change events into Amazon Redshift using MongoDB Atlas Stream Processing and AWS Kinesis Data Streams — with no S3 staging or Firehose in between.

Architecture Overview

The flow is straightforward:

MongoDB Atlas — Changes (inserts, updates, replaces) occur in a collection.
Atlas Stream Processing — A stream processor captures these changes via a change stream, filters and reshapes them into flat JSON, and emits records to Kinesis.
AWS Kinesis Data Stream — Acts as the durable, ordered buffer between the source and the analytical store.
Amazon Redshift — A streaming materialized view reads directly from Kinesis and exposes the data as queryable columns.

The key advantage here is simplicity: there’s no intermediate S3 bucket, no Firehose delivery stream, and no ETL job to manage. Data flows from operational writes to analytical queries in near real-time.

Why This Combination?

Atlas Stream Processing handles the hardest part of change data capture — it connects natively to MongoDB change streams, handles resume tokens, and provides a pipeline language (aggregation framework) for reshaping documents on the fly. Instead of writing and hosting a custom consumer, you declare a pipeline, and Atlas manages the execution.

Kinesis provides a durable, ordered stream that decouples the producer (Atlas) from the consumer (Redshift). If Redshift has a maintenance window or you need to replay data, Kinesis retains records for up to 365 days.

Redshift Streaming Ingestion eliminates the traditional pattern of landing data in S3 first. A materialized view reads directly from the Kinesis stream, making new records queryable within seconds of arrival.

Implementation

Step 1: Create the Kinesis Data Stream

aws kinesis create-stream \
  --stream-name atlas-change-events \
  --shard-count 1 \
  --region us-east-1

A single shard handles up to 1 MB/s or 1,000 records/s ingest. Scale shards based on your change event volume.

Step 2: Set Up IAM for Atlas

Atlas Stream Processing authenticates to Kinesis via IAM AssumeRole. Create a role with a trust policy that allows the Atlas AWS account to assume it, and attach minimal Kinesis write permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "kinesis:PutRecords",
        "kinesis:DescribeStreamSummary"
      ],
      "Resource": "arn:aws:kinesis:us-east-1:<YOUR_ACCOUNT>:stream/atlas-change-events"
    }
  ]
}

That’s it — just two permissions. Atlas uses the batch API (PutRecords) exclusively, so you don’t need the single-record API (PutRecord) or the full DescribeStream.

Step 3: Register the Kinesis Connection in Atlas

In the Atlas UI, navigate to Stream Processing → Connection Registry → Add Connection → Kinesis, and provide:

The stream name
The AWS region
The IAM role ARN

Step 4: Create the Stream Processor

The stream processor is an aggregation pipeline that runs continuously. Here’s the pattern:

sp.createStreamProcessor("atlasToKinesisProcessor", [
  // 1. Filter for relevant operation types
  { $match: { operationType: { $in: ["insert", "update", "replace"] } } },

  // 2. Flatten the change event — promote fullDocument fields to top level
  { $replaceRoot: {
      newRoot: { $mergeObjects: ["$fullDocument", { _operationType: "$operationType" }] }
  }},

  // 3. Project into a flat structure for Kinesis/Redshift
  { $project: {
      device_id: "$device_id",
      timestamp: "$timestamp",
      watts: "$obs.watts",
      temp: "$obs.temp",
      _operationType: 1
  }},

  // 4. Emit to Kinesis
  { $emit: {
      connectionName: "myKinesisConnection",
      stream: "atlas-change-events",
      region: "us-east-1",
      partitionKey: "$device_id"
  }}
]);

The $emit stage is Kinesis-specific. Note that stream, region, and partitionKey are all top-level fields — not nested inside a config object.

After creating the processor, start it in mongosh connected to your Stream Processing Instance (SPI):

sp.atlasToKinesisProcessor.start();

Step 5: Set Up Redshift Streaming Ingestion

On the Redshift side, create an external schema pointing to Kinesis:

CREATE EXTERNAL SCHEMA IF NOT EXISTS kinesis_schema
FROM KINESIS
IAM_ROLE 'arn:aws:iam::<YOUR_ACCOUNT>:role/<REDSHIFT_ROLE>';

Then create a streaming materialized view that parses the JSON records:

CREATE MATERIALIZED VIEW mdb_change_events AUTO REFRESH YES AS
SELECT
  approximate_arrival_timestamp,
  JSON_EXTRACT_PATH_TEXT(from_varbyte(kinesis_data, 'utf-8'), 'device_id') AS device_id,
  JSON_EXTRACT_PATH_TEXT(from_varbyte(kinesis_data, 'utf-8'), 'timestamp') AS event_timestamp,
  CAST(JSON_EXTRACT_PATH_TEXT(from_varbyte(kinesis_data, 'utf-8'), 'watts') AS DOUBLE PRECISION) AS watts,
  CAST(JSON_EXTRACT_PATH_TEXT(from_varbyte(kinesis_data, 'utf-8'), 'temp') AS DOUBLE PRECISION) AS temp,
  JSON_EXTRACT_PATH_TEXT(from_varbyte(kinesis_data, 'utf-8'), '_operationType') AS operation_type
FROM kinesis_schema."atlas-change-events";

With AUTO REFRESH YES, Redshift continuously ingests new records. You can also trigger a manual REFRESH MATERIALIZED VIEW mdb_change_events; if needed.

Step 6: Query Your Data

SELECT * FROM mdb_change_events
ORDER BY approximate_arrival_timestamp DESC
LIMIT 10;

That’s it. Operational writes in MongoDB are now queryable in Redshift within seconds.

Lessons Learned

Flatten before emitting. Redshift’s JSON_EXTRACT_PATH_TEXT is limited to 5 levels of nesting, has a 16MB record cap, and AWS recommends migrating to JSON_PARSE with SUPER type instead. By flattening in Atlas Stream Processing ($replaceRoot + $project), Kinesis receives simple flat JSON, and the Redshift materialized view stays clean.
The materialized view only captures new records. Records already in the stream when the view is created may not appear. Create the view before starting the processor, or insert fresh test data after creation.
External schema creation requires a superuser. The CREATE EXTERNAL SCHEMA statement must be run by the database owner or a user with the CREATE SCHEMA privilege. If your IAM-federated session doesn’t have these privileges, connect with the admin user and password instead.

When to Use This Pattern

This architecture works well when:

You need near-real-time analytics on operational MongoDB data
You want to avoid managing CDC infrastructure (Debezium, Kafka Connect, custom consumers)i
Your analytical queries run in Redshift, and you want columnar performance
You prefer a serverless, managed approach with minimal operational overhead

For higher throughput or more complex transformations, you might add more Kinesis shards and scale the Atlas Stream Processing instance accordingly. The architecture stays the same — just the capacity changes.

References

An agent is a graph

MongoDB Guests — Tue, 04 Aug 2026 16:04:18 +0000

This tutorial was written by Néstor Daza.

This is the fourth article in a series about building Claudius, my own Claude-based chatbot (Github). In the previous article, we scaffolded the core components of the app.

We already built the foundation: the data model, an identity the client cannot tamper with, and a health check that proves the application can reach both Atlas and Bedrock. We can now move to the streaming chat backbone, the phase where the conversation finally becomes real and Claudius turns into something you can use every day as your favorite AI tool.

We are going to build the live path, everything that happens between hitting send and tokens arriving on screen. The model runs as a small agent that can pause to search the web, stream its answer back word by word, and let you switch from a cheap model to a strong one in the middle of a thread without losing your place. The next article will be about where the conversation lives once the tokens stop, the persistence, the metering, and the guarded writing operations that keep the whole thing grounded. Here, the question is narrower and more immediate: what actually happens when you hit send?

The agent is not a prompt

The chat engine is not a single call to a model wrapped in a clever prompt; it’s a small state machine built with LangGraph, a design choice that lets the model do more than answer in one shot.

The graph has three nodes. load_context runs but does nothing yet, as a deliberate placeholder where memory retrieval will land in a later phase. agent is the model itself, bound to the tools it is allowed to call. tools is the executor that runs whatever the model asks for. A conditional edge sits after the agent; if the model requests a tool call, control passes to the tools node. If it returns a plain answer, the run ends. The tools node always loops back to the agent.

const builder = new StateGraph(MessagesAnnotation)
  .addNode("load_context", loadContext)
  .addNode("agent", agent)
  .addNode("tools", new ToolNode(tools))
  .addEdge(START, "load_context")
  .addEdge("load_context", "agent")
  .addConditionalEdges("agent", toolsCondition, ["tools", END])
  .addEdge("tools", "agent");

That loop keeps running until the model has what it needs, then returns a natural language answer instead of another tool call. A prompt answers once and stops, a graph keeps working until the task is finished.

Choose the model per run, not per graph

Here is the design decision that took the most thought, and it is invisible in the diagram above. The model is not compiled into the graph.

You might expect the agent node to close over a fixed model instance when the graph is constructed. Instead, the model is chosen per run and passed through LangGraph's configurable channel, the per-invocation configuration that travels alongside the graph state. The agent node reads the inference profile from that channel and builds its model client fresh on every turn.

async function agent(
  state: typeof MessagesAnnotation.State,
  config: RunnableConfig,
): Promise<typeof MessagesAnnotation.Update> {
  const { inferenceProfileId, maxTokens } = readConfigurable(config);
  const model = buildChatModel(
    inferenceProfileId,
    maxTokens !== undefined ? { maxTokens } : {},
  ).bindTools(tools);

  const response = await model.invoke(
    [new SystemMessage(SYSTEM_PROMPT), ...state.messages],
    config,
  );

  return { messages: [response] };
}

readConfigurable pulls the inference profile and token ceiling from the per-run config. buildChatModel constructs the ChatBedrockConverse client for that profile, and bindTools attaches the tool schema so the model knows what it is allowed to call (i.e, web_search). The node then invokes the model with the system prompt followed by the full message history, and returns the model's reply as the one update to the graph state.

Why go to this trouble? Because it is what makes switching models mid-conversation almost free. When you move from Haiku to Sonnet halfway through a thread, nothing about the stored conversation changes. The message history is untouched; on the next turn, we read a different inference profile from the config and build a different client using the same history. There is no graph to rebuild and no checkpoint to migrate. The model is a parameter of the turn, not a property of the conversation.

Model-switching appears in the request the client sends, not in the thread you see on screen. The rendered conversation carries no per-message model label, and the conversation row stores only the current model as a single field, so neither one records which model produced which turn. The request payload does, with each send carrying the conversation identifier, the chosen model, and the new text. Two consecutive sends in one conversation, sharing a conversationId and differing only in modelId, show up like this:

// Turn 3, sent while the selector was on Haiku
{
  "conversationId": "6770f1a2b3c4d5e6f7a8b9c0",
  "modelId": "claude-haiku",
  "text": "Summarize the tradeoffs we just covered."
}
// Turn 4, same conversation, selector now on Sonnet
{
  "conversationId": "6770f1a2b3c4d5e6f7a8b9c0",
  "modelId": "claude-sonnet",
  "text": "Now make the strongest case against that position."
}

The second turn still reads the full prior history that the server holds, so Sonnet answers turn four with turn three already in context, even though turn three ran on Haiku. The modelId is the public catalog identifier that the selector handed the client, paired with the display name on screen; the cross-region inference profile it resolves to remains on the server.

One tool, one predictable shape

The agent has just one tool in this phase, web_search, and it is a thin wrapper over Tavily. It returns exactly three fields per result: title, url, and snippet. Nothing else comes through.

The wrapper is hand-rolled rather than pulled from a library, to have a bit more control over the output shape. The model reads a compact JSON block, and the user interface renders it as a list of sources. Both sides depend on the shape staying fixed. Tavily returns its result text in a field called content, and the wrapper renames it to snippet at that boundary. Replace Tavily with another search service later, and the only code that changes is what's inside that wrapper.

This is a habit worth keeping past this one tool. A predictable shape at a boundary is cheap to build and prevents the rest of the system from developing a quiet dependency on a particular vendor's response format.

Bridging the event gap

The graph produces one kind of stream, the browser expects another; the route between them is a translation layer, the most intricate piece of this phase.

LangGraph emits a flat event log through its streamEvents interface. Model token chunks, tool starts, tool ends, and run boundaries all arrive as a sequence of typed events that you consume in order. The frontend uses the Vercel AI SDK, which expects something different, a typed stream of user-interface message parts. Natural text arrives as a text-start, then a run of text-delta events, then a text-end. Tool activity arrives as tool-input and tool-output parts. The two formats describe the same conversation, but they are not the same format. We need to close the gap between them.

A small state machine takes care of it. It walks the LangGraph event log and emits AI SDK parts as it goes.

for await (const ev of events) {
  switch (ev.event) {
    case "on_chat_model_stream": {
      const chunk = ev.data?.chunk as BaseMessage | undefined;
      const delta = chunk?.text ?? "";
      if (delta.length > 0) {
        if (textId === null) {
          textId = `text-${textSegment++}`;
          writer.write({ type: "text-start", id: textId });
        }
        writer.write({ type: "text-delta", id: textId, delta });
        assistantText += delta;
      }
      break;
    }
    case "on_chat_model_end": {
      endText();
      const output = ev.data?.output as
        | { usage_metadata?: UsageMetadata }
        | undefined;
      addUsage(totals, output?.usage_metadata);
      break;
    }
    case "on_tool_start": {
      writer.write({ type: "tool-input-start", toolCallId: ev.run_id, toolName: ev.name, dynamic: true });
      writer.write({ type: "tool-input-available", toolCallId: ev.run_id, toolName: ev.name, input: ev.data?.input ?? {}, dynamic: true });
      break;
    }
    case "on_tool_end": {
      writer.write({ type: "tool-output-available", toolCallId: ev.run_id, output: parseToolOutput(ev.data?.output), dynamic: true });
      break;
    }
  }
}

Read it event by event. On a model token chunk (on_chat_model_stream), if no text part is open yet, the machine opens one with a fresh identifier and a text-start, then writes the token as a text-delta and appends it to the running answer it keeps in assistantText. On the model turn ending (on_chat_model_end), endText closes the open part, emitting the text-end and resetting the identifier to null so the next token will open a new segment; the same handler pulls the token usage out of the event and adds it to the running totals, which the companion article picks up. On a tool starting (on_tool_start), the machine emits the tool-input parts. On a tool ending (on_tool_end), it parses the tool's JSON output back into a structured part that the renderer can show as sources.

Because the model finishes its turn before the tools it requires run, on_chat_model_end always fires ahead of on_tool_start. So each tool call sits cleanly between closed text segments, never inside an open one. The textSegment counter increments each time, so the answer the model writes after reading its search results opens as a fresh segment rather than reopening the one that came before the search. The renderer sees a clean sequence: natural text, then a tool call with its sources, then more text.

The tool parts are flagged dynamic. The web_search tool is not registered on the client with a schema, so as far as the AI SDK is concerned, its part is a dynamic tool, one whose shape is not known in advance. Marking it dynamic on the server lets the client assemble it into a cleanly typed part that the renderer can switch on, even though the client was never handed the tool's signature.

The conversation identifier rides back on a transient data part. When you send the first message in a brand-new conversation, the server creates the conversation row and needs to tell the client its identifier, without forcing a second request to get it. So it writes a one-off data-conversation part at the very top of the stream.

writer.write({ type: "start" });
writer.write({
  type: "data-conversation",
  data: { id: threadId, title: conversationTitle },
  transient: true,
});
writer.write({ type: "start-step" });

The client reads that part, adopts the identifier, updates the URL, and shows the new conversation in the sidebar, the part itself never becoming a message in the thread. transient marks it as out-of-band, a signal rather than content. No extra round trip, and the conversation is addressable from the first token onward.

The user sees what the agent did

The payoff of all that plumbing is visible on screen. Tokens animate in as they arrive, so the answer builds in front of you instead of appearing in one block after a wait. When the agent decides to search, a live line appears reading "Searching the web," and once results return, it collapses into an expandable list of the sources the tool found, each with its title and link.

That last behavior is a principle that runs through the whole product. Every tool call is visible and inspectable. The user should always be able to see what the agent did on their behalf, not only the polished answer that came out the other end. The event bridge is what makes this possible, because it carries tool activity through as first-class parts rather than burying it inside the model's turn.

There is one more thing to notice about the round trip, and it sets up the second half of the backbone. When you send a message, the client sends only the new text. It does not send back the transcript of everything said before. It cannot, really, because it does not hold the authoritative history. It holds only what it has rendered on screen. The full conversation lives elsewhere, and the model reads it from there on every turn.

Where it lives, and why one small identity decision makes reopening a week-old conversation as cheap as opening a fresh one, is the theme of the next article. The agent is a graph. The conversation, it turns out, is a database.

Building the foundation Claudius runs on

MongoDB Guests — Tue, 04 Aug 2026 15:21:07 +0000

This tutorial was written by Néstor Daza.

This is the third article in a series about building Claudius, my own Claude-based chatbot (Github). The previous article discussed the MongoDB data model to use for the app.

The previous article decided the shape of the data. None of it matters until the app around it is working, and getting it there is the unglamorous half of this phase. It comes down to three things: an identity system the client cannot tamper with, proof that Claudius can reach the two services it depends on, and the deployment realities that decide whether any of it runs at all. This is the boring work that quietly decides whether a project survives contact with production.

Identity: the client never gets a vote

Any Google account on Earth can sign into Claudius safely because a user's role is never something the client sends. It is decided on the server every time.

One piece of this lives outside the code. The Google provider needs an OAuth (Open Authorization) client that you register once in the Google Cloud Console, and the client identifier and secret from that registration are set in corresponding env variables. These setup steps live in the Auth.js and Google documentation, so I am not repeating them here.

Sign-in runs on Auth.js v5 with the Google provider and the MongoDB adapter. There are three roles, admin, member, and guest, and they resolve in exactly one place on the server, with a clear precedence:

export async function resolveRole(email: string | null | undefined): Promise<Role> {
  if (!email) return "guest";

  const normalized = email.toLowerCase();
  if (normalized === env.ADMIN_EMAIL.toLowerCase()) return "admin";

  const settings = await settingsCol();
  const allowlist = await settings.findOne({ _id: "allowlist" });
  if (allowlist && "emails" in allowlist) {
    const allowed = allowlist.emails.some((e) => e.toLowerCase() === normalized);
    if (allowed) return "member";
  }

  return "guest";
}

The bootstrap admin email wins outright, an allowlist of member emails in the settings collection comes next, and everyone else falls through to guest, the least-privileged default. The client supplies a Google identity and nothing more. It never names its own role, so there is no request a user can forge to promote themselves.

The session uses a JWT (JSON Web Token) rather than a database-backed session, so the resolved role is stored in the token and is available without a database read on every request. That choice has one small downside: the token caches the role until it refreshes, so adding someone to the member allowlist takes effect on their next sign-in, not the moment you edit the list. For a personal project, that is a fine trade.

The wiring is small: on first sign-in, the jwt callback calls provisionUser and stores the returned role on the token, and the session callback then copies it onto session.user for the app to read.

Provisioning is the other half of sign-in. The adapter has already written a bare user document, just the name, email, and image Google returns. It fills in the Claudius-specific fields with a single aggregation-pipeline update built around $ifNull, so each field defaults only when it is absent, and an admin's later change is never overwritten. The role is the exception, recomputed every time, since that is the field on which the whole security model rests:

export async function provisionUser(
  userId: string,
  email: string | null | undefined,
): Promise<Role> {
  const role = await resolveRole(email);
  const users = await usersCol();

  const resetsAt = new Date(Date.now() + 24 * 60 * 60 * 1000);

  await users.updateOne({ _id: new ObjectId(userId) }, [
    {
      $set: {
        role,
        allowedModels: { $ifNull: ["$allowedModels", null] },
        monthlyTokenBudget: { $ifNull: ["$monthlyTokenBudget", null] },
        status: { $ifNull: ["$status", "active"] },
        dailyMessageCount: {
          $ifNull: ["$dailyMessageCount", { count: 0, resetsAt }],
        },
      },
    },
  ]);

  return role;
}

One TypeScript detail in this layer was a bit of a pain. Augmenting the session type works through the next-auth module as you would expect, but the JWT type does not! It is only re-exported from next-auth/jwt and lives in @auth/core/jwt, so a declaration written against next-auth/jwt silently fails to merge, and the token field you were trying to type lands as unknown instead. You have to target the real module:

declare module "next-auth" {
  interface Session {
    user: {
      id: string;
      role: Role;
    } & DefaultSession["user"];
  }
}

// JWT lives in @auth/core/jwt; next-auth/jwt only re-exports it, so the
// augmentation must target the real module to merge rather than shadow.
declare module "@auth/core/jwt" {
  interface JWT {
    uid?: string;
    role?: Role;
  }
}

That silent failure is the worst kind: when there are no errors, the type just quietly goes wrong.

Connectivity: prove it reaches Atlas and Bedrock

The last piece of the skeleton is a single protected route that proves the app can talk to the things it depends on. Any signed-in role can call /api/health. It always pings Atlas, and given ?probe=bedrock it goes one step further and makes a one-token Converse call against Claude Haiku 4.5, to prove the credentials can invoke a real model. The probe is the interesting part: capped at a single output token, wrapped so no AWS internals leak back to the caller, and returning the counts read from the Converse usage metadata.

export async function bedrockHealthProbe(): Promise<BedrockHealth> {
  try {
    const llm = new ChatBedrockConverse({
      model: HAIKU_INFERENCE_PROFILE,
      region: env.AWS_REGION,
      credentials: {
        accessKeyId: env.AWS_ACCESS_KEY_ID,
        secretAccessKey: env.AWS_SECRET_ACCESS_KEY,
      },
      maxTokens: 1,
      temperature: 0,
    });

    const response = await llm.invoke([["human", "ping"]]);

    const usage = response.usage_metadata as TokenUsage | undefined;

    return {
      ok: true,
      inputTokens: usage?.input_tokens ?? null,
      outputTokens: usage?.output_tokens ?? null,
    };
  } catch {
    // Do not surface the underlying AWS/SDK error to callers.
    return { ok: false, error: "Bedrock probe failed" };
  }
}

On a live run, it comes back with the result you want:

{"ok":true,"inputTokens":8,"outputTokens":1}

Eight tokens in, one token out. That is the signal you want from a health check on an AI app, not only that the network is up, but that the credentials work and the meter is running.

The models, the probe, and the rest of the app are seeded into settings as a catalog. Each entry carries a real Bedrock cross-region inference profile ID rather than a bare model ID, which is the correct choice for cross-region routing:

[
{"id":"haiku","inferenceProfileId":"us.anthropic.claude-haiku-4-5-20251001-v1:0"},
  {"id":"sonnet","inferenceProfileId":"us.anthropic.claude-sonnet-4-6"},
  {"id":"opus","inferenceProfileId":"us.anthropic.claude-opus-4-8"}
]

Each entry also carries per-million-token input and output pricing, which a later phase uses to enforce tier costs. For now, those fields are a deliberate placeholder, to be confirmed against current Bedrock pricing before any billing math depends on them, so I am not quoting numbers here that would only mislead.

Env: validate only what this phase uses

As we work through this series, each development phase validates only the environment variables it uses, rather than declaring the project's entire surface up front. So far, we need just eight, for the database, Google auth, Bedrock, and the admin bootstrap, so the env schema gates exactly those. A missing or malformed value throws at startup, naming the offending variable and never printing its value:

const EnvSchema = z.object({
  MONGODB_URI: z.string().url(),
  AUTH_SECRET: z.string().min(1),
  AUTH_GOOGLE_ID: z.string().min(1),
  AUTH_GOOGLE_SECRET: z.string().min(1),
  AWS_ACCESS_KEY_ID: z.string().min(1),
  AWS_SECRET_ACCESS_KEY: z.string().min(1),
  AWS_REGION: z.string().min(1),
  ADMIN_EMAIL: z.string().email(),
});

const parsed = EnvSchema.safeParse(process.env);
if (!parsed.success) {
  // List the offending variable names only, never their values.
  const offenders = parsed.error.issues
    .map((issue) => issue.path.join("."))
    .join(", ");
  throw new Error(
    `Invalid environment configuration. Check these variables: ${offenders}`,
  );
}
export const env = parsed.data;

Other variables that will be introduced later, like Tavily for web search, Voyage for embeddings, Blob for file storage, and LangSmith for tracing, are not here yet.

The gotchas that cost real time

The skeleton works, but it did not the first time, and the failures are perhaps the most useful part of this build report.

The first was the most painful: the connection string has no database path, so anything that does not explicitly name the database falls back to test, which the Atlas credentials cannot touch. It bit twice, first in the index script:

MongoServerError: not authorized on test to execute command { listCollections: 1, ... $db: "test" }

Then again, after deployment, in production sign-in, because the Auth.js adapter does not go through the app's database helper and quietly defaults to test until it is told otherwise:

[auth][error] AdapterError
[auth][cause]: MongoServerError: not authorized on test to execute command
{ find: "accounts", ... $db: "test" }

The fix is to name the database explicitly in both places. The adapter gets its own databaseName:

// databaseName is set explicitly because the connection string has no default
// database; without it the adapter would target "test" (which the Atlas
// credentials cannot access).
export const { handlers, auth, signIn, signOut } = NextAuth({
  adapter: MongoDBAdapter(clientPromise, { databaseName: DB_NAME }),
  ...authConfig,
});

And the app reaches the database through one helper that names it the same way. That helper is also where the second gotcha lives. A naive new MongoClient() per request would leak a connection pool on every serverless invocation and every dev hot reload, so the client is cached on globalThis as a single instance, and DB_NAME is a constant rather than an env var because the database name is neither secret nor environment-specific:

const globalForMongo = globalThis as unknown as {
  _claudiusMongoClientPromise?: Promise<MongoClient>;
};

const client = new MongoClient(env.MONGODB_URI);

export const clientPromise: Promise<MongoClient> =
  globalForMongo._claudiusMongoClientPromise ??
  (globalForMongo._claudiusMongoClientPromise = client.connect());

export const DB_NAME = "claudius";

export async function getDb(): Promise<Db> {
  const connected = await clientPromise;
  return connected.db(DB_NAME);
}

The adapter is handed the same clientPromise, so auth, and the app shares one client and one pool rather than opening two. Making them agree on a single driver version took one line at the repo root, since the adapter and the app can otherwise resolve different ones:

"overrides": { "mongodb": "7.3.0" }

The third is a monorepo trap. The shared package is consumed as raw TypeScript and transpiled by the app's build through transpilePackages: ["@claudius/shared"], with no separate build step. That is convenient, but it means every runtime dependency the shared code imports has to resolve from the app workspace, because Vercel scopes its install to the app's root directory. The rule that keeps the build green is to declare a runtime dependency in every workspace that imports it, even when that looks like duplication.

The last is the kind of thing that only shows up on deploy. A lockfile generated on macOS omitted the Linux binaries that Tailwind's build needs, and Vercel's Linux install could not find them:

Error: Cannot find module '../lightningcss.linux-x64-gnu.node'

Declaring those binaries as optionalDependencies puts them in the lockfile for every platform, and the deploy goes green:

"optionalDependencies": {
  "@tailwindcss/oxide-linux-x64-gnu": "4.3.1",
  "lightningcss-linux-x64-gnu": "1.32.0"
}

What stands, and what a headless build cannot prove

No chat yet, but the foundation is real. Identity resolves on the server and rides safely in the token. The health route confirms Claudius can reach both Atlas and Bedrock, and the configuration fails loud and early when something is missing. The data model from the last article now has something solid underneath it.

The role-resolution logic is unit-tested, but the actual sign-in is an OAuth round trip, and that needs a real browser, not just this headless build. Confirming that an admin email lands as admin on the page and on the persisted user document is the one step I finish by hand.

Next is the streaming chat backbone, where the conversation finally gets a place to live, and I open the real checkpoint document I have been promising since the start.

Finally! MongoDB Atlas supports partial restore

MongoDB Guests — Mon, 03 Aug 2026 09:00:00 +0000

This article was written by Darshan Jayarama.

Reflecting on my tenure as a MongoDB TSE, I recall a specific incident where a DBA mistakenly dropped a production collection instead of a UAT one. At the time, we were compelled to restore the total cluster to a secondary instance, isolate the required data, and then re-import it into the live environment. The process was inefficient and expensive. We had to recover an entire 760GB cluster just to retrieve a single 50MB collection. You can imagine the massive resource drain involved.

One might wonder why we couldn't restore a single collection or select databases. That’s the new feature I am going to share with you today.

What is partial restore?

As explained in the MongoDB documentation, you can now simply restore any database or collection from the cloud backup; there is no need to restore the full cluster.

It means that if one collection is all you want, you can recover it without touching the entire dataset from the cloud backup.

Real-World Example

A production cluster will have a combination of multiple databases:

Live Production Environment

├── Customer_Profiles
├── Transaction_History
├── Stock_Management
├── Billing_Records
└── Data_Insights

If someone accidentally drops Billing_Records.invoices collection. To resolve this:

Before the Partial Restore feature:

Restore the entire cluster to a temporary Atlas cluster.
Wait for the restore to complete.
Export the required collection.
Import it back into production.
Delete the temporary cluster.

With the Partial Restore feature,simply choose:

Backup Snapshot
Database
Collection
Restore

Here is the workflow:
Click on the Backup tab, select the cluster, and select the backup snapshot as shown below.

Click the 3 dots per the selected snapshot, then click Restore.

In the next window, select the Restore Databases/Collection tab:

You will now have 2 options:

Create as new: With this option, you can choose not to remove the existing collection. If the restoring collection/DB name already exists, Atlas automatically renames the restoring collection to avoid a conflict.
Overwrite existing: Atlas replaces the existing collection with the restored version.

If you want to compare the existing and restored collection, then choose the "Create as new" option. If you want to replace the existing collection, then choose the "Overwrite existing" option. But please be aware that all the data written after the snapshot time will no longer exist.

After choosing your option, you can go ahead and select exactly the data you need. You have two ways to do this:

Using the UI: Just tick the checkboxes for the database or collection you want. Or, if you prefer, you can use the search bar to find what you need using a regex pattern.
Using List Entry: Type the database name or the database. collection name, one per line. For example, if you want everything from the Billing_records database, but only the NRI and HNI collections from the Customer_profile database, you just enter it like this: Billing_records Customer_profile.NRI Customer_profile.HNI

Keep one thing in mind —you cannot use a dot in your database name. If you use a dot, the system will treat it as a combined DB and collection name.

Once you are done with this part, just click Next. On the next screen, you will need to select the Destination details.

Developing a World Cup Stats application with Django MongoDB Extensions' MQL Panel

Abigail Afi Gbadago — Fri, 31 Jul 2026 13:32:12 +0000

Django MongoDB Extensions is a package that contains additional developer tools for the Django MongoDB Backend. One of these tools is the MongoDB Query Language (MQL) panel, and it shows MQL queries that your application runs during a request, including execution times and query plans. This tutorial shows you how to configure and use the panel.

We will highlight the functionality of Django Debug Toolbar by building a World Cup 2026 analytics application that queries match data using the Django ORM and raw aggregation pipelines. We'll use the Django Debug Toolbar to inspect the pipelines these queries generate as MQL.

There are two ways to follow this tutorial.
Option A: Clone the GitHub repo containing the finished code from this tutorial, set it up using the repo's README, and explore it on your own. Link to the GitHub project:django-mongodb-world-cup

Option B: Start the project from scratch by following the steps below.

Start the project from scratch

Prerequisites
NB: This tutorial uses config as the Django project name and world_cup as the app name throughout. If you use different names, you'll need to update every reference to them in the code examples that follow (imports, INSTALLED_APPS, and urls.py).

Before starting this tutorial, you will need to complete the following steps:

Download and install Django MongoDB Backend
Create a MongoDB deployment
Create a connection string
Create a Django project As a quick reference, you can create the Django project with the following command:

django-admin startproject config --template https://github.com/mongodb-labs/django-mongodb-project/archive/refs/heads/6.0.x.zip

(NB: I used config here, but you can also specify a Django project name of your choice, for example, world_cup_project)

Configure your MongoDB connection

These steps are explained in detail in the Get Started with Django MongoDB Backend tutorial.

Once these prerequisites are complete, the next step is installing Django MongoDB Extensions.

Install Django MongoDB Extensions

Navigate to your project's root directory; either the folder created by startproject, or the folder created when you clone the repository (i.e., django-mongodb-world-cup). This is also where you should create and activate your virtual environment, if you haven't already:

python3 -m venv venv
source venv/bin/activate

Create a requirements.txt file at the root of the project with the following contents:

#requirements.txt
django-mongodb-extensions
django-debug-toolbar
python-dotenv

Run the following command in your terminal to install the packages:
pip install -r requirements.txt
Create a .env file at the root of the project with the following contents and replace the placeholders with your own values.

For the SECRET_KEY, copy the value Django already generated in your settings.py when you ran startproject; it's normally hardcoded there by default, but we move it into .env to keep it out of version control, since it's used to sign session cookies, CSRF tokens, and other security-sensitive data.

#.env 
SECRET_KEY=[REPLACE_WITH_YOUR_SECRET_KEY]
MONGODB_URI="mongodb+srv://<username>:<password>@<cluster>.mongodb.net/?retryWrites=true&w=majority&appName=world_cup"
DB_NAME=[REPLACE_WITH_YOUR_DB_NAME]

Then update DATABASES and SECRET_KEY variables in settings.py to read from the environment instead of being hardcoded. Your settings.py should now include the following:

import os
from dotenv import load_dotenv
load_dotenv()
SECRET_KEY = os.getenv('SECRET_KEY')
DATABASES = {
    'default': {
        'ENGINE': 'django_mongodb_backend',
        'HOST': os.getenv('MONGODB_URI'),
        'NAME': os.getenv('DB_NAME'),
    },
}

Add Django MongoDB Extensions and Django Debug Toolbar to your installed apps: Navigate to your project's(or cloned repo) settings.py file and add "django_mongodb_extensions" to the INSTALLED_APPS setting, as shown in the following example:

INSTALLED_APPS = [
    'django_mongodb_extensions',
    'debug_toolbar',
]

Register the MQL panel

Replace your existing DEBUG_TOOLBAR_PANELS setting with the list below. This is Django Debug Toolbar's default panel list with one change: SQLPanel is swapped out for MQLPanel.

DEBUG_TOOLBAR_PANELS = [
    'debug_toolbar.panels.history.HistoryPanel',
    'debug_toolbar.panels.versions.VersionsPanel',
    'debug_toolbar.panels.timer.TimerPanel',
    'debug_toolbar.panels.settings.SettingsPanel',
    'debug_toolbar.panels.headers.HeadersPanel',
    'debug_toolbar.panels.request.RequestPanel',
    'django_mongodb_extensions.mql_panel.MQLPanel',
    'debug_toolbar.panels.staticfiles.StaticFilesPanel',
    'debug_toolbar.panels.templates.TemplatesPanel',
    'debug_toolbar.panels.alerts.AlertsPanel',
    'debug_toolbar.panels.cache.CachePanel',
    'debug_toolbar.panels.signals.SignalsPanel',
    'debug_toolbar.panels.community.CommunityPanel',
    'debug_toolbar.panels.redirects.RedirectsPanel',
    'debug_toolbar.panels.profiling.ProfilingPanel',
]

Configure Django Debug Toolbar Middleware

Add DebugToolbarMiddleware as the first item in MIDDLEWARE, and set INTERNAL_IPS:

MIDDLEWARE = [
    'debug_toolbar.middleware.DebugToolbarMiddleware',
    'django.middleware.security.SecurityMiddleware',
    # ... rest of middleware
]
INTERNAL_IPS = ['127.0.0.1']

Add Django Debug Toolbar’s URLs

In urls.py, add the following:

from django.conf import settings
from django.urls import include, path
urlpatterns = [
    # ... your app urls
]
if settings.DEBUG:
    urlpatterns += [path("__debug__/", include("debug_toolbar.urls"))]

Create an Application

From your project's root directory, run the following command to create a new Django app. This tutorial uses the name world_cup throughout, so if you use a different name, you'll need to update it in every file this tutorial references (INSTALLED_APPS, imports, urls.py):
python manage.py startapp world_cup

Next, include your app in your project by opening the settings.py file in config and editing your INSTALLED_APPS setting to resemble the following code:

INSTALLED_APPS = [
    'config.apps.MongoAdminConfig',
    'config.apps.MongoAuthConfig',
    'config.apps.MongoContentTypesConfig',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'django_mongodb_backend',
    'django_mongodb_extensions',
    'debug_toolbar',
    'world_cup',
]

Create models for Goal and Match data

The world_cup application stores the World Cup match data using two models: Match and Goal. The Goal model is embedded inside Match as an array of goal events because it is usually queried together with the corresponding matches.

Open the models.py file in the world_cup directory and replace its contents with the following code:

from django.db import models
from django_mongodb_backend.fields import EmbeddedModelArrayField
from django_mongodb_backend.managers import MongoManager
from django_mongodb_backend.models import EmbeddedModel

class Goal(EmbeddedModel):
    team = models.CharField(max_length=100)
    name = models.CharField(max_length=100)
    minute = models.CharField(max_length=10)     
    penalty = models.BooleanField(default=False)
    own_goal = models.BooleanField(default=False)

    def __str__(self):
        return f"{self.name} (({self.team}, {self.minute}')"
class Match(models.Model):
    num = models.IntegerField(null=True, blank=True)        
    round = models.CharField(max_length=50)
    date = models.DateField()
    kickoff_time = models.CharField(max_length=20)           
    group = models.CharField(max_length=20, blank=True)      
    ground = models.CharField(max_length=100)
    team1 = models.CharField(max_length=100)
    team2 = models.CharField(max_length=100)
#Fields ending in "1" refer to team1, fields ending in "2" refer to team2
    ft_score1 = models.IntegerField(null=True, blank=True)      #ft = full-time
    ft_score2 = models.IntegerField(null=True, blank=True)
    ht_score1 = models.IntegerField(null=True, blank=True)  #ht = half-time
    ht_score2 = models.IntegerField(null=True, blank=True)
    et_score1 = models.IntegerField(null=True, blank=True)  #et = extra-time
    et_score2 = models.IntegerField(null=True, blank=True)
    pen_score1 = models.IntegerField(null=True, blank=True) #penalty score
    pen_score2 = models.IntegerField(null=True, blank=True)
   goals = EmbeddedModelArrayField(Goal, default=list, blank=True)
objects = MongoManager()

    def __str__(self):
        return f"{self.team1} vs {self.team2} ({self.round})"

The Goal model represents the world_cup.world_cup_match collection and stores information about the goals. This model contains fields such as name, minute, penalty, and own goal. The Goal model contains the embedded document values stored in the Match model.

Run Migrations

Generate and apply the migration for the world_cup app:

python manage.py makemigrations world_cup
python manage.py migrate

Loading Data

mkdir -p world_cup/management/commands
mkdir -p world_cup/data
touch world_cup/management/__init__.py
touch world_cup/management/commands/__init__.py
touch world_cup/management/commands/load_matches.py

For the World Cup data, we use the JSON file from Open Football GitHub source and save it to the data folder under the world_cup folder (/world_cup/data/worldcup_2026.json).

After obtaining the data (JSON file), we create a script (world_cup/management/commands/load_matches.py) to load the matches. Replace the following content in the load_matches.py file.

# world_cup/management/commands/load_matches.py
import json
from datetime import datetime
from pathlib import Path

from django.core.management.base import BaseCommand

from world_cup.models import Goal, Match

WORLD_CUP_DATA = Path(__file__).resolve().parent.parent.parent / "data" / "worldcup_2026.json"


def build_goals(raw_goals, team_name):
    return [
        Goal(
            team=team_name,
            name=g["name"],
            minute=g["minute"],
            penalty=g.get("penalty", False),
            own_goal=g.get("owngoal", False),
        )
        for g in raw_goals
    ]
def to_match_kwargs(raw_match):
    score = raw_match.get("score", {})
    ft = score.get("ft", [None, None])
    ht = score.get("ht", [None, None])
    et = score.get("et", [None, None])
    p = score.get("p", [None, None])

    return dict(
        num=raw_match.get("num"),
        round=raw_match["round"],
        date=datetime.strptime(raw_match["date"], "%Y-%m-%d").date(),
        kickoff_time=raw_match["time"],
        group=raw_match.get("group", ""),
        ground=raw_match["ground"],
        team1=raw_match["team1"],
        team2=raw_match["team2"],
        ft_score1=ft[0],
        ft_score2=ft[1],
        ht_score1=ht[0],
        ht_score2=ht[1],
        et_score1=et[0],
        et_score2=et[1],
        pen_score1=p[0],
        pen_score2=p[1],
        goals=build_goals(raw_match.get("goals1", []), raw_match["team1"]) + build_goals(raw_match.get("goals2", []), raw_match["team2"]),
    )
class Command(BaseCommand):
    help = (
        "Load the curated World Cup 2026 matches (Groups A, B, I, J) into MongoDB. "
        "Source: worldcup_data/worldcup_2026.json — 24 matches total, "
        "16 completed with real goals, 8 unplayed (Matchday 14/17)."
    )

    def handle(self, *args, **options):
        with open(WORLD_CUP_DATA) as f:
            raw_matches = json.load(f)["matches"]

        self.stdout.write(f"Found {len(raw_matches)} matches to load.")

        batch_size = 5
        created = 0
        for i in range(0, len(raw_matches), batch_size):
            batch = raw_matches[i : i + batch_size]
            Match.objects.bulk_create([Match(**to_match_kwargs(m)) for m in batch])
            created += len(batch)
            self.stdout.write(f"Loaded {created}/{len(raw_matches)} matches...")

        self.stdout.write(self.style.SUCCESS(f"Done: {created} matches loaded."))

Next, we load the data by running:
python manage.py load_matches

Once this is successful, we can see the results either by querying the database via the Python shell or via MongoDB Atlas’s Data Explorer. For example, to confirm the data loaded correctly and inspect an embedded Goal array, run:

python manage.py shell

In this shell:

>>> from world_cup.models import Match
>>> Match.objects.count()
100
>>> Match.objects.get(team1="France", team2="Senegal").goals1

Match results queried via Python shell

World Cup match stats after upload in MDB Atlas

Create Views

These views provide three pages: a total goals counter using the Django ORM, a top scorers leaderboard using a raw aggregation pipeline, and an interactive query shell that allows you to run five different query types.

Open the views.py file in your world_cup directory and replace its contents with the following code:

# world_cup/views.py
from django.db import connections
from django.db.models import F, Sum
from django.shortcuts import render

from .models import Match


def tournament_goals(request):
    """Pipeline 1 — built-in aggregate: total goals across completed matches."""
    result = (
        Match.objects.filter(ft_score1__isnull=False)
        .annotate(total_goals=F("ft_score1") + F("ft_score2"))
        .aggregate(tournament_goals=Sum("total_goals"))
    )
    return render(request, "world_cup/goals.html", {
        "total_goals": result["tournament_goals"] or 0,
    })


def top_scorers(request):
    """Pipeline 2 — raw_aggregate: leaderboard via $unwind + $group."""
    pipeline = [
        {"$unwind": "$goals"},
        {"$match": {"goals.own_goal": {"$ne": True}}},
        {"$group": {"_id": "$goals.name", "total_goals": {"$sum": 1}}},
        {"$sort": {"total_goals": -1}},
        {"$limit": 10},
    ]
    results = Match.objects.raw_aggregate(pipeline)
    scorers = [{"name": r.pk, "goals": r.total_goals} for r in results]
    return render(request, "world_cup/scorers.html", {"scorers": scorers})


ALLOWED_QUERIES = {
    "team": {
        "description": "All matches for a team",
        "example": "team France",
    },
    "round": {
        "description": "Matches in a round (e.g. round Quarter-final)",
        "example": "round Quarter-final",
    },
    "top scorers": {
        "description": "Top 10 scorers via $unwind + $group pipeline",
        "example": "top scorers",
    },
    "goals by round": {
        "description": "Goals per round via $group pipeline",
        "example": "goals by round",
    },
    "goals": {
        "description": "Total tournament goals",
        "example": "goals",
    },
}


def _execute_query(query_text):
    """Parse and execute a user query, return result dict."""
    q = query_text.strip().lower()

    if q == "goals":
        result = (
            Match.objects.filter(ft_score1__isnull=False)
            .annotate(total_goals=F("ft_score1") + F("ft_score2"))
            .aggregate(tournament_goals=Sum("total_goals"))
        )
        return {
            "results": [{"tournament_goals": result["tournament_goals"] or 0}],
        }

    if q in ("top scorers", "scorers", "top"):
        pipeline = [
            {"$unwind": "$goals"},
            {"$match": {"goals.own_goal": {"$ne": True}}},
            {"$group": {"_id": "$goals.name", "total_goals": {"$sum": 1}}},
            {"$sort": {"total_goals": -1}},
            {"$limit": 10},
        ]
        results = Match.objects.raw_aggregate(pipeline)
        return {
            "results": [{"scorer": r.pk, "goals": r.total_goals} for r in results],
        }

    if q.startswith("team "):
        team = query_text.strip()[5:].strip()
        from django.db.models import Q
        matches = Match.objects.filter(Q(team1__iexact=team) | Q(team2__iexact=team)).order_by("date")
        return {
            "query": f'db.world_cup_match.aggregate([{{$match: {{$or: [{{team1: {{$regex: "^{team}$", $options: "i"}}}}, {{team2: {{$regex: "^{team}$", $options: "i"}}}}]}}}}, {{$sort: {{date: 1}}}}])',
            "results": [
                {
                    "date": str(m.date),
                    "round": m.round,
                    "team1": m.team1,
                    "team2": m.team2,
                    "score": f"{m.ft_score1}-{m.ft_score2}" if m.ft_score1 is not None else "TBD",
                }
                for m in matches
            ],
        }

    if q.startswith("round "):
        round_name = query_text.strip()[6:].strip()
        matches = Match.objects.filter(round__icontains=round_name).order_by("date")
        return {
            "query": f'db.world_cup_match.aggregate([{{$match: {{round: {{$regex: "{round_name}", $options: "i"}}}}}}, {{$sort: {{date: 1}}}}])',
            "results": [
                {
                    "date": str(m.date),
                    "team1": m.team1,
                    "team2": m.team2,
                    "score": f"{m.ft_score1}-{m.ft_score2}" if m.ft_score1 is not None else "TBD",
                }
                for m in matches
            ],
        }

    if q in ("goals by round", "goals per round"):
        pipeline = [
            {"$match": {"ft_score1": {"$ne": None}}},
            {"$group": {
                "_id": "$round",
                "total_goals": {"$sum": {"$add": ["$ft_score1", "$ft_score2"]}},
                "matches": {"$sum": 1},
            }},
            {"$sort": {"total_goals": -1}},
        ]
        results = Match.objects.raw_aggregate(pipeline)
        return {
            "results": [{"round": r.pk, "goals": r.total_goals, "matches": r.matches} for r in results],
        }

    return None


def query_shell(request):
    """Interactive query shell — form POST triggers full page reload for MQL panel."""
    context: dict = {"queries": ALLOWED_QUERIES}

    if request.method == "POST":
        query_text = request.POST.get("query", "").strip()
        context["last_query"] = query_text

        if query_text:
            result = _execute_query(query_text)
            if result is None:
                context["error"] = f"Unknown query. Try: {', '.join(ALLOWED_QUERIES.keys())}"
            else:
                context["mongo_query"] = result.get("query", "")
                context["results"] = result.get("results", [])

    return render(request, "world_cup/shell.html", context)

Configure URLs for your views

Create a new file called urls.py file in your world_cup directory. To map the views defined in the preceding step to URLs, paste the following code into urls.py:

from django.urls import path

from . import views

urlpatterns = [
    path("goals/total/", views.tournament_goals, name="tournament_goals"),
    path("goals/top-scorers/", views.top_scorers, name="top_scorers"),
    path("shell/", views.query_shell, name="query_shell"),
]
`

Then, navigate to the config/urls.py file and replace its contents with the following code:
`
from django.conf import settings
from django.shortcuts import render
from django.urls import include, path


def landing(request):
    return render(request, "world_cup/landing.html")


urlpatterns = [
    path("", landing, name="landing"),
    path("world_cup/", include("world_cup.urls")),
]

if settings.DEBUG:
    urlpatterns += [path("__debug__/", include("debug_toolbar.urls"))]

Create Templates

In your world_cup directory, create a subdirectory called templates.

Under templates, create five HTML files to display the base page, goals page(showing total goals by aggregation), landing page, scorers page(top scorers), and shell page(interactive shell to run queries and aggregation pipelines interactively).

base.html: This is the shared layout (nav, styles) that the other four templates extend.

#world_cup/templates/base.html
<!DOCTYPE html>
<html lang="en">
<head>
    <meta charset="UTF-8">
    <meta name="viewport" content="width=device-width, initial-scale=1.0">
    <title>{% block title %}World Cup 2026{% endblock %}</title>
    <style>
        :root {
            --nav-bg: #1C2D38;
            --page-bg: #f0f2f5;
            --card-bg: #ffffff;
            --text-primary: #1C2D38;
            --text-secondary: #5c6970;
            --border: #e2e5e9;
            --accent: #1a6b4b;
            --accent-light: #e8f5ef;
            --white: #ffffff;
        }

        * { margin: 0; padding: 0; box-sizing: border-box; }

        body {
            font-family: -apple-system, BlinkMacSystemFont, 'Segoe UI', Roboto, sans-serif;
            background: var(--page-bg);
            color: var(--text-primary);
            min-height: 100vh;
        }

        nav {
            background: var(--nav-bg);
            padding: 0.9rem 2rem;
            display: flex;
            align-items: center;
            justify-content: space-between;
            border-bottom: 3px solid var(--accent);
        }

        nav .brand {
            color: var(--white);
            font-size: 1.1rem;
            font-weight: 700;
            text-decoration: none;
        }

        nav .nav-links a {
            color: rgba(255,255,255,0.85);
            text-decoration: none;
            margin-left: 2rem;
            font-size: 0.9rem;
        }

        nav .nav-links a:hover {
            color: var(--white);
        }

        .container {
            max-width: 900px;
            margin: 2rem auto;
            padding: 0 1.5rem;
        }

        header {
            margin-bottom: 1.5rem;
        }

        header h1 {
            font-size: 1.6rem;
            font-weight: 700;
            color: var(--text-primary);
        }

        header p {
            color: var(--text-secondary);
            font-size: 0.9rem;
            margin-top: 0.25rem;
        }

        .card {
            background: var(--card-bg);
            border-radius: 8px;
            padding: 1.5rem;
            margin-bottom: 1rem;
            border: 1px solid var(--border);
        }

        .card:hover {
            border-color: var(--accent);
        }

        .card a {
            text-decoration: none;
            color: var(--text-primary);
            display: block;
        }

        .card h3 {
            font-size: 1rem;
            font-weight: 600;
            margin-bottom: 0.3rem;
        }

        .card p {
            color: var(--text-secondary);
            font-size: 0.85rem;
        }

        .badge {
            display: inline-block;
            background: var(--accent);
            color: var(--white);
            font-size: 0.7rem;
            font-weight: 600;
            padding: 0.2rem 0.6rem;
            border-radius: 4px;
            margin-bottom: 0.5rem;
            text-transform: uppercase;
        }

        table {
            width: 100%;
            border-collapse: collapse;
            margin-top: 0.5rem;
        }

        th, td {
            text-align: left;
            padding: 0.75rem 1rem;
            border-bottom: 1px solid var(--border);
        }

        th {
            color: var(--text-secondary);
            font-size: 0.75rem;
            font-weight: 600;
            text-transform: uppercase;
            letter-spacing: 0.03em;
        }

        td {
            font-size: 0.9rem;
        }

        .back-link {
            display: inline-block;
            margin-bottom: 1rem;
            color: var(--accent);
            text-decoration: none;
            font-size: 0.85rem;
        }

        .back-link:hover {
            text-decoration: underline;
        }

        .stat-value {
            font-size: 3rem;
            font-weight: 700;
            color: var(--accent);
        }

        .stat-label {
            color: var(--text-secondary);
            font-size: 0.9rem;
            margin-top: 0.25rem;
        }
    </style>
</head>
<body>
    <nav>
        <a href="/" class="brand">World Cup 2026 Stats</a>
        <div class="nav-links">
            <a href="{% url 'tournament_goals' %}">Goals</a>
            <a href="{% url 'top_scorers' %}">Scorers</a>
            <a href="{% url 'query_shell' %}">Shell</a>
        </div>
    </nav>
    <div class="container">
        {% block content %}{% endblock %}
    </div>
</body>
</html>

landing.html: This is the app's home page, linking out to the goals, scorers, and shell pages.
Endpoint: http://127.0.0.1:8000/

#world_cup/templates/landing.html
{% extends "world_cup/base.html" %}

{% block title %}World Cup 2026 — Home{% endblock %}

{% block content %}
<header>
    <h1>⚽ World Cup 2026 Stats</h1>
</header>

<div class="card">
    <a href="{% url 'tournament_goals' %}">
        <span class="badge">Aggregation</span>
        <h3>Total Goals</h3>
        <p>Sum of all goals scored — view the MQL query in the Debug Toolbar</p>
    </a>
</div>

<div class="card">
    <a href="{% url 'top_scorers' %}">
        <span class="badge">Pipeline</span>
        <h3>Top Scorers</h3>
        <p>$unwind + $group pipeline — view execution time in the MQL panel</p>
    </a>
</div>

<div class="card">
    <a href="{% url 'query_shell' %}">
        <span class="badge">Interactive</span>
        <h3>Query Shell</h3>
        <p>Run find queries and aggregation pipelines </p>
    </a>
</div>
{% endblock %}

goals.html: This displays the total goals count from the tournament_goals view's aggregation.
Endpoint: http://127.0.0.1:8000/world_cup/goals/total/

#world_cup/templates/goals.html
{% extends "world_cup/base.html" %}

{% block title %}Total Goals — World Cup 2026{% endblock %}

{% block content %}
<a href="/" class="back-link">← Back to Home</a>
<header>
    <h1>Tournament Goals</h1>
    <p>Total goals across all completed matches</p>
</header>

<div class="card" style="text-align: center; padding: 3rem;">
    <div class="stat-value">{{ total_goals }}</div>
    <div class="stat-label">goals scored</div>
</div>
{% endblock %}

scorers.html: This displays the top scorers leaderboard from the top_scorers view's raw pipeline.
Endpoint: http://127.0.0.1:8000/world_cup/goals/top-scorers/

#world_cup/templates/scorers.html
{% extends "world_cup/base.html" %}

{% block title %}Top Scorers — World Cup 2026{% endblock %}

{% block content %}
<a href="/" class="back-link">← Back to Home</a>
<header>
    <h1>Top Scorers</h1>
    <p>Leading goal scorers in the tournament</p>
</header>

<div class="card">
    <table>
        <thead>
            <tr>
                <th>#</th>
                <th>Player</th>
                <th>Goals</th>
            </tr>
        </thead>
        <tbody>
            {% for scorer in scorers %}
            <tr>
                <td>{{ forloop.counter }}</td>
                <td>{{ scorer.name }}</td>
                <td>{{ scorer.goals }}</td>
            </tr>
            {% empty %}
            <tr><td colspan="3">No scoring data available yet.</td></tr>
            {% endfor %}
        </tbody>
    </table>
</div>
{% endblock %}

shell.html: This is the interactive query shell which takes an input form plus a results table, for running any of the commands from ALLOWED_QUERIES and compares them in the MQL panel.

Endpoint: http://127.0.0.1:8000/world_cup/shell/

#/templates/shell.html
{% extends "world_cup/base.html" %}

{% block title %}Query Shell — World Cup 2026{% endblock %}

{% block content %}
<a href="/" class="back-link">← Back to Home</a>
<header>
    <h1>Query Shell</h1>
    <p>Each query reloads the page — check the MQL panel in the Debug Toolbar</p>
</header>

<div class="shell">
    <div class="shell-output" id="output">
        {% if not last_query %}
        <div class="shell-line system">Click an example or type a query:</div>
        <div class="shell-line system"></div>
        <div class="shell-line system" style="color: var(--mdb-gray); opacity: 0.6;">— Queries (find) —</div>
        <div class="shell-line example" data-query="team France">team France</div>
        <div class="shell-line example" data-query="team England">team England</div>
        <div class="shell-line example" data-query="round Quarter-final">round Quarter-final</div>
        <div class="shell-line system"></div>
        <div class="shell-line system" style="color: var(--mdb-gray); opacity: 0.6;">— Aggregation Pipelines —</div>
        <div class="shell-line example" data-query="top scorers">top scorers</div>
        <div class="shell-line example" data-query="goals by round">goals by round</div>
        <div class="shell-line example" data-query="goals">goals</div>
        {% endif %}

        {% if last_query %}
        <div class="shell-line query">{{ last_query }}</div>
        {% endif %}

        {% if error %}
        <div class="shell-line error">{{ error }}</div>
        {% endif %}

        {% if mongo_query %}
        <div class="shell-line mongo-query">{{ mongo_query }}</div>
        {% endif %}

        {% if results %}
        <div class="shell-line result">
            <table class="result-table">
                <thead>
                    <tr>
                        {% for key in results.0.keys %}
                        <th>{{ key }}</th>
                        {% endfor %}
                    </tr>
                </thead>
                <tbody>
                    {% for row in results %}
                    <tr>
                        {% for val in row.values %}
                        <td>{{ val|default:"—" }}</td>
                        {% endfor %}
                    </tr>
                    {% endfor %}
                </tbody>
            </table>
        </div>
        {% endif %}
    </div>

    <form method="post" action="{% url 'query_shell' %}" class="shell-input-row">
        {% csrf_token %}
        <span class="prompt">▶</span>
        <input type="text" name="query" id="query-input" value="" placeholder="e.g. team France, top scorers, round Quarter-final" autofocus>
    </form>
</div>

<style>
    .shell {
        background: var(--card-bg);
        border: 1px solid var(--border);
        border-radius: 8px;
        overflow: hidden;
        font-family: 'SF Mono', 'Fira Code', 'JetBrains Mono', monospace;
        font-size: 0.85rem;
    }

    .shell-output {
        padding: 1.5rem;
        max-height: 500px;
        overflow-y: auto;
    }

    .shell-line {
        margin-bottom: 0.4rem;
        line-height: 1.5;
    }

    .shell-line.system {
        color: var(--text-secondary);
    }

    .shell-line.query {
        color: var(--accent);
        font-weight: 600;
    }

    .shell-line.query::before {
        content: "▶ ";
        opacity: 0.5;
    }

    .shell-line.mongo-query {
        color: var(--text-secondary);
        font-size: 0.78rem;
        padding-left: 1rem;
        background: var(--page-bg);
        padding: 0.5rem 1rem;
        border-radius: 4px;
        margin: 0.3rem 0;
        overflow-x: auto;
    }

    .shell-line.result {
        color: var(--text-primary);
        padding-left: 0;
    }

    .shell-line.error {
        color: #c0392b;
    }

    .shell-line.example {
        color: var(--accent);
        cursor: pointer;
        padding-left: 1rem;
        opacity: 0.8;
    }

    .shell-line.example::before {
        content: "▶ ";
        opacity: 0.4;
    }

    .shell-line.example:hover {
        opacity: 1;
        text-decoration: underline;
    }

    .shell-input-row {
        display: flex;
        align-items: center;
        border-top: 1px solid var(--border);
        padding: 0.75rem 1.5rem;
        background: var(--page-bg);
    }

    .prompt {
        color: var(--accent);
        margin-right: 0.75rem;
        font-size: 0.8rem;
    }

    .shell-input-row input {
        flex: 1;
        background: transparent;
        border: none;
        outline: none;
        color: var(--text-primary);
        font-family: inherit;
        font-size: inherit;
    }

    .shell-input-row input::placeholder {
        color: var(--text-secondary);
        opacity: 0.5;
    }

    .result-table {
        width: 100%;
        border-collapse: collapse;
        margin: 0.5rem 0;
    }

    .result-table th {
        color: var(--text-secondary);
        font-size: 0.7rem;
        text-transform: uppercase;
        text-align: left;
        padding: 0.3rem 0.75rem;
        border-bottom: 1px solid var(--border);
    }

    .result-table td {
        padding: 0.3rem 0.75rem;
        font-size: 0.8rem;
        color: var(--text-primary);
    }
</style>

<script>
document.querySelectorAll('.shell-line.example').forEach(el => {
    el.addEventListener('click', () => {
        document.getElementById('query-input').value = el.dataset.query;
        el.closest('.shell').querySelector('form').submit();
    });
});
</script>
{% endblock %}

Build an Aggregation Pipeline

The tournament_goals view uses the Django ORM's aggregate() and Sum() to total up goals, and the MQL panel captures that query automatically. For more advanced analysis, like ranking top scorers, you can use MongoDB's aggregation pipelines. The top_scorers view in views.py does exactly this: it uses $concatArrays to merge embedded goal arrays, $unwind to flatten them into individual documents, and $group to count goals per player.

The top_scorers view uses raw_aggregate(), a method which accepts a raw MongoDB aggregation pipeline directly, including stages like $unwind, and returns Django model instances. See the QuerySet API reference for more details.

When you navigate to the /world_cup/goals/top-scorers/ and open the MQL panel in the Django Debug Toolbar. You'll see the full aggregation pipeline that was executed, along with its timing. The MQL panel captures both ORM-generated queries and raw pipelines, so you can inspect and optimize either approach from the same interface.

Verify the MQL panel

Start your development server by running the following command:
python manage.py runserver

Navigate to any page in your application that queries MongoDB. Open the Django Debug Toolbar and select the MQL panel to view the MongoDB queries executed during the request, including their execution times and explain plans.

View worldcup.goals query information

Now, navigate to http://127.0.0.1:8000/world_cup/goals/total/ to view MongoDB queries executed during the request.
The Django Debug Toolbar appears on the right side of the page. Select the MQL label to view the MongoDB query used to calculate the total goals, which resembles the following code:

After, you can explore the query information by clicking on the following buttons:
Select the + button to the left of the query to view the call stack, which resembles the following call trace:

Next, select the Query button to the top-right of the query, in the ACTION column, to view the pretty-printed MQL. This MQL resembles the following code:

Now, select the Explain button to the right of the query, in the ACTION column, to view the explain plan for the query. This plan resembles the following code:

View the top scorers aggregation pipeline

Next, navigate to http://127.0.0.1:8000/world_cup/goals/top-scorers/ to see the aggregation pipeline in the MQL panel. We see that this page runs a raw multi-stage pipeline ($concatArrays, $unwind, $group, $sort, $limit). Select the MQL panel to view the pipeline stages, execution time, and explain plans.

View query shell queries

Finally, access the endpoint http://127.0.0.1:8000/world_cup/shell/ and submit a query such as team France. The page reloads with the results, and the MQL panel shows the aggregation pipeline that was executed. Even though the view uses the Django ORM's filter() and order_by(), the MongoDB backend translates these into an aggregation pipeline.

Next Steps

Congratulations on getting this far! You now have a Django application that uses Django MongoDB Extensions to view and analyze MongoDB queries in the Django Debug Toolbar.

Here are some debugging exercises you can experiment with:

Run the "round" search in the query shell (e.g., round Group A) and check its query plan in the MQL panel. You'll see a COLLSCAN which is a full collection scan because round__icontains compiles to an unanchored regex, which can't use an index no matter what you add.
Compare it against round__iexact on the same field, and the panel shows IXSCAN instead. It is similar code but different queries underneath, and that is exactly what the Django Debug Toolbar makes evident.

To learn more about the Django MongoDB Extensions package, see django-mongodb-extensions and the django-mongodb-extensions repository on GitHub.
To learn how to query your data, see the Specify a Query guide.

Link to this GitHub Project: Django MongoDB World Cup project

Link to the Data: Open Football Dataset

What Is MongoDB Atlas Vector Search

MongoDB Guests — Tue, 28 Jul 2026 10:01:53 +0000

This article was written by Moses Anumadu.

Traditional search relies on keywords and tags to recommend search results. This system works and is in use across multiple platforms. However, it has a major limitation that has become more apparent in recent years due to increased AI adoption. The major limitation of traditional search is that it returns search results based solely on keyword matches. Traditional searches, as we have them today, do not consider semantic meaning. For example:

Imagine that we built a movie website using traditional search. Now, if a user wants to find The Matrix, but they can't remember the name. They type in "dystopian AI reality simulation." A traditional keyword database will return zero results because those exact words are not in the movie title or description. This is the major challenge for traditional search. It returns results strictly from keywords, in this case, either in the movie name or description. So, if a user does not type the exact keywords stored in your database, the correct or intended movie does not show up, even though it is in the database. The platform simply does not know that it was the intended movie. This can be limiting and produce search results that do not accurately match the users' intent.

A better approach for this use case is to have the search not rely on keywords to recommend movies. It should recommend movies based on semantic similarities. This is a very good example of when vector search should be used over traditional search.

In this tutorial, we will explore what vector search is, its importance, what MongoDB Vector Search is, and why you should consider it for your next vector search implementation. With that said, let's get started.

Prerequisites

To follow along with this article, you should have a basic understanding of the following concepts:

MongoDB Atlas
Database and collection
Can perform CRUD operations
Basic understanding of web development technologies

What Is Vector Search?

Vector search is a search technology that uses vectors to identify semantic similarity and recommend results that are accurate in terms of meaning and not keywords.

With vector search, text, images, and other media file types are stored as embeddings. Embeddings are numerical values generated by machine learning models that capture the context, intent, and semantic similarity of the stored data. Items with similar meanings have embeddings (numeric values) that are closer.

When a search is done, the vector search identifies similar values by comparing the closeness of the content embeddings. The closer the embeddings, the more similar the contents. This returns a more accurate search result that relies on meaning over a keyword search that relies on keywords.

The major difference to note here is that keyword searches rely on keyword matches to recommend results, while vector searches rely on meaning by comparing embeddings to recommend results.

Keyword Search vs. Vector Search

Keyword Search	Vector Search
Search relies on exact keyword matching to return results.	Search relies on a semantic understanding of search terms.
Results are not generated based on users' intentions. They rely on keyword accuracy.	Generates results with users' intent in consideration.
The use of synonyms in keyword searches does not often return the best search results. Eg, car and vehicle won't return the same result.	Synonyms in vector search generate far more accurate results. It factors in the actual meaning of the words to generate results.
Keyword search queries are not flexible to things like typos and long search queries.	Search keywords or phrases can be sentences. It can be flexible with misspelled words and still return accurate results.

How Embeddings Enable Vector Search

What Is an Embedding?

Think of embeddings as the foundation of vector searches. Embeddings are numerical values generated by artificial intelligence. These numeric values represent the meaning of text and media files, so vector search can compare them and return more accurate, similar results.

In a keyword search query, a traditional database does not treat words as symbols with meaning; it sees them as independent flat characters that don't have links to the sentence's meaning. So it only returns results that match exactly. An example can be a search keyword of "CAT" would return a CAT. Not because it understands it, but because the keywords match. But a search keyword of VEHICLE would not return a result of CAR.

How AI generates these similarities

Let's take a moment to examine how an AI embedding model generates the numbers in an embedding. Imagine you need to describe an apple using a scale of 1-10. Is an apple sweet? Yes. On a scale of 1 - 10, how sweet is an apple? Say 8. Is an apple crunchy? Yes. On a scale of 1 - 10, how crunchy is an apple? Say 6. Is an apple red in color? Yes. On a scale of 1 to 10, how red is an apple? Say 9. This would give us an embedding of [8, 6, 9]. A fruit like a banana can be [9, 1, 0].

The AI embedding model does something very similar to the example above; when it processes a sentence, it outputs a long list of numbers (called a vector). This list serves as a precise map coordinate for that specific piece of text in a giant, digital "map of meaning."
This results in words with similar meaning having vectors that are closely related or similar.

Why Embeddings Matter

Embeddings are like the building blocks of vectors. With them, vectors are generated, and similarity in terms of meaning can be measured in a search query.

How MongoDB Vector Search Works

The Vector Search Workflow

In a MongoDB vector search implementation, data is stored together with its generated vector in the same collection. This simplifies the implementation when compared to the steps required to implement this using different technologies. Let's examine the workflow for both storing a document and retrieving it during a search query.

Storing document

Storing record

This process starts with storing our normal documents in a MongoDB collection. A document can be in text or media format.

Embeddings are generated

After storing this document, embeddings are generated using any of the AI models that have this compatibility. Some popular examples are VoyageAI, OpenAI, Cohere, and Hugging Face.

Documents are updated to contain vectors.

The data collection in Atlas is updated to contain the generated vectors. This saves the vector as a field in the collection. Keeping both the data and its embeddings in one collection.

A vector search index is created

The final step in this flow is creating the vector search index. This boosts performance during searches.

The image below provides a visual explanation of the flow;

Search workflow

The MongoDB vector search workflow consists of a few steps; it starts with a user query. The query is converted into embeddings using an AI embedding model (for best results, use the same embedding model as the one used to embed the documents in the database is advised).

After the vectors are generated, MongoDB vector search calculates the semantic proximity between the search query vector and the vector of the documents stored in the collection. It then returns the results, which are semantically matching items to the user search query. The image below provides a more visual explanation of the process.

Common Use Cases

There is no doubt about the importance of vector searches. However, this does not make it the right tool for every search implementation. Vector search implementations require AI models to generate vectors. This can make the search implementation a bit more complex and costly. With that in mind, it is important to explore some real-world use cases where it can be a good fit and also where it is not. But in general, vector searches are a great solution for search results that want semantic similarity. Some examples include:

Semantic document search
AI knowledge bases
Product recommendation systems
Similar content discovery
FAQ search
AI assistants using business data

Benefits and Considerations

Benefits

Some of the key benefits include:

More relevant search results

Using vectors to measure similarity enables vector searches to return more relevant results. Unlike keyword searches, vector search can provide relevant results even if the search query contains misspelled or slang words.

Better user experience

Vector search implementation can improve the user experience. Vector search is less rigid; it provides some flexibility for users when typing their search. This improves the users' experience of the application.

Natural language querying

Vector search implementation allows users to type search queries in natural language, rather than having to target keywords, often explaining the expected result rather than targeting the keywords.

Native integration within MongoDB

Vector Search is part of the core features of MongoDB. This means that it is already integrated with your database and collections. If you have an already existing collection in MongoDB, you can easily use your existing data to implement vector searching.

Reduced need for a dedicated vector database

With MongoDB, you do not need a separate vector database. Data can be stored alongside its vector embeddings in the MongoDB collection.

Considerations

While vector search can provide more accurate results, it is worth noting that it is not a complete replacement for traditional searches. The following key points are worth noting before deciding to implement a vector search in an application:

The need for embedding models:

Vector search implementation requires the use of AI models to generate embeddings. These models can be expensive and provide an additional configuration layer.

Additional storage requirements for vectors

Vectors can be large in size. The vector for a sentence can have up to 8,192 dimensions. Imagine when you have a hundred blog posts. This can take up a huge amount of space and can lead to higher cloud bills.

Choosing an appropriate embedding model

There are several embedding models that are great for different industries. Choosing the right model for your particular use case is a factor that should be considered. For example, a model that was designed for the fashion industry might not give the best result for a model optimized for medical research.

When keyword search may still be the better choice

Vector search is not a complete replacement for keyword search. Keyword search can still be more effective for exact match searches. An example can be searching for an exact serial number, username, or any other situation that returns an exact keyword result.

When Should You Use MongoDB Vector Search?

A vector search implementation that does not use MongoDB requires data, which can be in the form of text, images, or audio, to be stored in a traditional database; the generated vector for the items will be stored in a different dedicated vector database. This results in repetition because two databases are used, increasing the complexity of keeping the two databases in sync and fragmenting data (text and vectors).

MongoDB vector search bridges that gap. It lets data and their generated vectors be stored in the same collection.

Another major advantage that MongoDB vector search brings is that it keeps data in sync. A very common issue with vector search integrations where one database is used for data and another for storing vectors is that the data is often not in sync. If a record is deleted from the regular database and the vector is not, search results can be recommended that are no longer available in the database. MongoDB vector search stores the record and keeps it in sync. If a record is deleted from its collection, its vector is deleted as well, keeping both the record and its vector in sync.

MongoDB vector search eliminates the need for new infrastructure by storing both records and vectors in the same collection.

Conclusion

In this article, we explored what vector search is, how it works, how AI comes into the picture, and what differentiates it from traditional keyword search. We also took it a step further and explored how MongoDB vector search works and what advantages it has over other vector databases.

By the end of this article, you should have a basic understanding of how:

Vector search retrieves information based on meaning rather than exact keywords.
Embeddings make semantic search possible.
MongoDB provides native vector search capabilities that integrate with your existing data.
Understanding these fundamentals is the first step toward building intelligent search experiences.

For a deeper dive on how to implement vector search with MongoDB, you can check out the MongoDB documentation here. If you prefer a step-by-step tutorial, check out this tutorial here.

Now you have the basics; go build something great. Cheers.

A Proven Application Layer for MongoDB

MongoDB Guests — Mon, 27 Jul 2026 11:48:12 +0000

The successful alternative I used before Atlas Device SDKs. This Article was written by Raschid Jimenez.

Atlas Device SDKs reached end-of-life in September 2025, and it’s easy to understand why so many MongoDB developers liked them: direct client access, authentication, sync, and security, all tied closely to MongoDB Atlas.

Long before Atlas Device SDKs, I had already found a different way to get many of those same benefits. Using Parse Server and Back4app, I built production apps for years without the burden of maintaining custom API layers across every client and language. This article is the recipe I wish more MongoDB developers knew: why this stack works, what it gives you, and why it’s such a strong fit on top of MongoDB.

I will walk you through three things that sold me on Parse Server and Back4app: native client SDKs across every platform, row-level security for multi-tenant apps, and managed schema with index control. Along the way, I'll cover what else comes in the package: authentication, real-time updates via LiveQuery, file storage, auto-generated GraphQL APIs, and offline support. Data never leaves MongoDB Atlas. Back4app is a gateway that translates client requests into operations against your cluster.

            ┌──────────────────────────────────────┐
            │  Client (SDKs / REST / GraphQL)      │
            └────────────────────┬─────────────────┘
                                 │
            ┌────────────────────▼─────────────────┐
            │     Application Layer                │
            │       (Back4app - Stateless Gateway) │ 
            └────────────────────┬─────────────────┘
                                 │
            ┌────────────────────▼─────────────────┐
            │     MongoDB Atlas (your cluster)     │
            └──────────────────────────────────────┘

Back4app Uses MongoDB Atlas as the Datastore

Back4app is a managed backend-as-a-service (BaaS) platform. Sign up, paste in your MongoDB connection string, and you get an application backend running in front of your cluster: APIs, SDKs, authentication, real-time sync, file storage, access control, and a dashboard. No provisioning servers, no configuring authorization, no maintaining the layer yourself.

The connection model is what should land for an Atlas audience: bring your own cluster. Back4app connects to an existing Atlas deployment via a standard MongoDB connection string. No migration, no copy of your data, no proprietary storage format, no lock-in at the storage tier. If you walked away from Back4app tomorrow, your Atlas cluster would still be there, untouched, with every native MongoDB primitive intact.

What Back4app adds is the application backend layer most teams end up building anyway: auto-generated REST and GraphQL APIs, authentication, real-time sync, file storage, ACLs and CLPs, and a visual data browser.

Back4app is a layer in front of Atlas, not a replacement for it.

Parse Server Under the Hood

Under the hood, Back4app runs Parse Server, an open-source backend framework with a mature SDK ecosystem across every major platform. That's where the Parse.Object, Parse.Query, Parse.ACL, and Parse.Schema APIs in the code samples throughout this article come from. They're Parse Server primitives. Back4app is the managed platform that runs Parse Server in production (like Atlas does for MongoDB), plus the pieces around it: hosted infrastructure, the connection-string flow into Atlas, the dashboard, the spreadsheet-like data browser, role-aware visual schema editing, monitoring, scaling, and backups.

The short version: code I write with a Parse.* prefix is Parse Server. The surfaces I interact with through the Back4app dashboard are Back4app.

Client SDKs Across Every Platform

With Atlas Device SDKs gone, the question for most teams is concrete: how do clients perform operations against MongoDB Atlas (CRUD, authentication, file uploads, real-time subscriptions, and access-controlled queries with relations) without building and maintaining a custom REST layer for each?

Back4app offers a set of native SDKs that talk to the gateway, which in turn talks to your Atlas cluster. Supported platforms:

iOS (Swift, Objective-C)
Android (Java, Kotlin)
Web (JavaScript, TypeScript)
React Native, Node.js
Flutter, Dart
PHP, .NET, C#
Edge runtimes

For anything not on that list, Back4app auto-generates REST and GraphQL endpoints from the same schema.

The interesting part isn't the language list. It's that the API shape is the same across all of them. Here are the four CRUD operations in JavaScript:

// CREATE - no REST endpoint, no schema migration, just save.
const order = new Parse.Object("Order");
order.set("total", 149.99);
await order.save();

// READ - ACLs are enforced server side; this user only sees their tenant's orders.
const query = new Parse.Query("Order");
query.greaterThan("total", 100);
query.include("customer"); // hydrate the pointer in one round trip
const orders = await query.find();

// UPDATE - same object, just mutate and save.
order.set("status", "shipped");
await order.save();

// DELETE - single call, ACL-checked at the data layer.
await order.destroy();

That include("customer") line is worth looking at. On raw MongoDB, resolving a reference to fetch the related document means writing an aggregation pipeline:

// Without Parse - manual aggregation pipeline:
db.orders.aggregate([
  { $match: { total: { $gt: 100 } } },
  { $lookup: {
      from: "users",
      localField: "customer",
      foreignField: "_id",
      as: "customer"
  }},
  { $unwind: "$customer" }
]);

// With Parse - one line:
await new Parse.Query("Order")
  .greaterThan("total", 100)
  .include("customer")
  .find();

Same result, one round trip, no pipeline to maintain. The include works identically across REST, GraphQL, and every SDK.

The same read in Swift, using the modern Parse-Swift SDK:

let query = Order.query("total" > 100)
  .include("customer")
let orders = try await query.find()

A note on API symmetry: query.include("customer") reads the same in JavaScript and Swift. The naming follows each language's conventions, but the mental model is identical across platforms. Same query model, same ACL enforcement, same relational include semantics. A team shipping iOS, Android, and web doesn't re-architect the data layer per client. They translate one mental model into three idioms.

If GraphQL is the API you want, the same schema produces a typed GraphQL endpoint automatically. No resolvers to write, and the same ACLs and CLPs that govern the SDKs govern those queries too.

For reactive UIs, LiveQuery lets you subscribe to any query and receive push updates on create, update, and delete over WebSockets. No polling loops, no reconnection logic, no homegrown change-stream plumbing. Subscribe to a dashboard to live order updates, and the dashboard updates itself when an order changes upstream.

Row-Level Security and Multi-Tenancy

Multi-tenant access control is application-layer work by design: the database holds the data, the application decides who sees what. In practice, that means every team building multi-tenant SaaS on Atlas writes the same middleware: tenant filters injected into every query, role checks scattered across controllers, field redaction in serializers. That creates a lot of security-critical logic to maintain.

Back4app moves that enforcement down to the data layer. Four layers of access control work together:

Layer	Granularity	When to use
Application Keys	Per client type	Separate public, client, and master contexts
Class-Level Permissions	Per class (collection)	"Who can create or read Orders at all"
Access Control Lists	Per document	"Who can read this Order"
Protected / Field-level	Per field	isAdmin: only the server can write

Here's what each layer replaces in a hand-rolled implementation:

// Without Parse — middleware you write and never get to delete:
app.get("/api/invoices", auth, async (req, res) => {
  const invoices = await db.collection("invoices").find({
    tenantId: req.user.tenantId,        // remember this on every endpoint
    $or: [
      { ownerId: req.user._id },
      { sharedWith: req.user._id }
    ]
  }).toArray();
  // ...strip fields the role shouldn't see...
  res.json(invoices);
});

// With Parse — write zero filter middleware:
const invoices = await new Parse.Query("Invoice").find();
// ACLs and CLPs enforce the same rules across REST, GraphQL, and every SDK.

Here's how you set those ACLs up for a multi-tenant invoice scoped to a tenant role:

const invoice = new Parse.Object("Invoice");
invoice.set("amount", 1200);

const acl = new Parse.ACL();
acl.setRoleReadAccess("tenant_acme", true);          // every user in this tenant
acl.setRoleWriteAccess("tenant_acme_admins", true);  // only admins can modify
invoice.setACL(acl);

await invoice.save();

Any query from a user outside tenant_acme will not see this document. Enforcement happens server-side at the gateway, not in hopeful client code. The same filter applies whether the request comes from the iOS SDK, the GraphQL endpoint, or an inbound webhook. There is no client-side path that can bypass it.

When per-object rules aren't expressive enough ("users can update their own row, but only during business hours, and only if the invoice isn't locked"), Cloud Code provides server-side functions for custom validation:

// Cloud Code — server side function, runs in Parse Server, ACL-aware.
Parse.Cloud.beforeSave("Invoice", async (req) => {
  if (req.object.get("locked") && !req.master) {
    throw "Locked invoices cannot be modified";
  }
});

Schema and Index Management

With Parse, the schema lives in the database itself. Every client SDK, REST endpoint, and GraphQL type derives from there automatically. One source of truth, propagated to every surface your app talks to.

Parse's ODM defines classes (collections) with typed fields: strings, numbers, dates, files, GeoPoints, pointers, and relations. Parse enforces types at the gateway across REST, GraphQL, and every SDK, so a malformed payload is rejected before it reaches Atlas. The underlying MongoDB collection and the indexes Parse needs are auto-created. Schema changes are tracked through the Parse schema API or the Back4app dashboard, and they propagate to every client without anyone regenerating models by hand.

Document stores don't usually feel relational, but Back4app adds pointers for one-to-one references and relations for many-to-many associations, with include queries that hydrate them in a single round trip:

const query = new Parse.Query("Order");
query.include("customer");   // hydrates the pointer
query.include("products");   // hydrates the relation
const orders = await query.find();
// One round trip. No manual $lookup pipeline.

When you want schemas in version control rather than clicked into a dashboard, schemas can be declared programmatically, and migrations become regular code:

const schema = new Parse.Schema("Invoice");
schema
  .addString("customerEmail")
  .addNumber("amount")
  .addPointer("tenant", "Tenant")
  .addIndex("by_email", { customerEmail: 1 }); // explicit MongoDB index
await schema.save();

Parse exposes MongoDB indexes directly. It creates defaults for ACL fields and pointers, but your hand-tuned indexes stay untouched. Parse only adds the ones it needs for ACL filtering and pointer fields. Your existing indexes for aggregation pipelines, Atlas Search, or hand-tuned queries stay exactly as you set them. The addIndex() call above creates a real MongoDB index. Parse doesn't abstract that away.

If you picked MongoDB for schemaless flexibility: Parse classes can stay flexible (allowAddField: true) during prototyping and tighten as the data model stabilizes. You pick per class. Strict where it matters, flexible where it helps.

Conclusion & Next Steps

Back4app is a stateless gateway in front of your existing MongoDB Atlas cluster. Connect it with a standard MongoDB connection string, and you get the dashboard, the spreadsheet-like data browser, SDKs across every major platform, auto-generated REST and GraphQL endpoints, LiveQuery for real-time UIs, and row-level security via CLPs and ACLs. Native MongoDB primitives (aggregations, transactions, indexes, Atlas Search) remain entirely available to you.

So my advice to you is: go to Back4App and sign up for free, create an app in your preferred language, and run your first operation on your data in under 10 minutes. Back4App offers a free tier (25,000 API requests per month), which is enough to try on a real app rather than a toy. My customers loved having their app backends hosted for free and paying only when they scaled.

Build Your AI-Powered Reading List Using DigitalOcean Functions, Mistral AI, and MongoDB

MongoDB Guests — Wed, 22 Jul 2026 10:24:29 +0000

This Tutorial was written by Diego Freniche.

Since I started browsing the Internet, I've used many different services, applications, and browser plugins (even the good old bookmarks) to mark articles and posts that seemed interesting and that I wanted to read. Only not now. I'll read them later, I promise. And then you hoard a bunch of links in one place. But when you go back to actually read them, you find yourself clicking again on them just to remember what those links where about and why did you store them in the first place...

Fast forward today, and I haven't changed much in how I interact with articles I want to read later. But nowadays we can use LLMs almost for free... and they are really good at writing text summaries. So, why not create a system that, given a link, generates a summary and then stores both the link and the summary in a fast database I can use to build my own UI later?

And this is now yet another read-it-later (maybe) system was born...

Prerequisites

Before you begin this guide, you'll need the following:

a DigitalOcean account, that we'll use to write a DigitalOcean Function. This function will receive our links and run the code that will generate the LLM-powered summary and store the results in a database.
a MongoDB database, also running in DigitalOcean. We will use it to store the links and summaries of the web pages. You can learn more about deploying a managed MongoDB instance in DigitalOcean here.
access to any LLM that can write a good summary. For this example, I'll use Mistral AI, so you'll need to register a free account there and get an API token. Take a note of that API key, we'll need it later!

Step 1 - Creating an Empty DigitalOcean Function

Let's start by creating the backbone of our system. We can write a backend app using Node, Spring Boot or any other web framework. But then we need to host it in a server, write the code, correctly size the server, update the operating system and dependencies, etc. Using a serverless function we focus on the task at hand. In this case, getting a URL via a parameter, producing a summary of the page, and storing the data in the database.

So head over to Functions and click on create > Functions. First, we need to choose where our serverless function will be hosted. Although it's a serverless function, our code will need a real server to run. Only we don't have to provide it. Choose a data center close to your location (or close to where the users of your function will be). Then give it a name, choosing a label. In my case, it will be read-it-later-i-pinly-promise.

After creating the namespace, you'll be in the Overview tab. Let's add some basic code to test our function! Just click on the Create Function button.

We need to select one of the supported runtimes (in our case, Nodejs 18), and we'll give the function a name: read_later. It's important to leave the Web Function box checked, as we want to call this function directly using HTTP.

Done! Now you should see the default source code of the newly created function. This just gets an argument name from the URL and returns it. As we're not configuring a web page or a JSON response, this will be printed out as plain text by our web browser.

function main(args) {
let name = args.name || 'stranger' let greeting = 'Hello ' + name + '!' console.log(greeting)
return {"body": greeting}
}

In the main function, we receive a JSON object with all the arguments passed in the function call, we search for name, and use it. If not found, we use stranger as name. Then we compose and print out a greeting which we also return from our function.

Step 2 - testing our function

As this is accessible via HTTP calls, we have several different options to test our function.

Using cURL

If your system includes curl (Microsoft Windows 10/11 and most UNIX systems, like Linux and macOS come with cURL preinstalled), you can copy the URL of your function and add the name parameter at the end. Then call curl with:

curl https://<redacted>.doserverless.co/api/v1/web/<redacted>/default/read_later? name=Diego

In your case, instead of <redacted>, you'll see the ids of your functions. If you call the function from a Terminal, you'll see the response from our function:

curl https://<redacted>.doserverless.co/api/v1/web/<redacted>/default/read_later? name=Diego
Hello Diego!%

Try it changing the name and the code in your function and experiment with it! It's super fun!

Testing directly in DigitalOcean

We can click on Test Parameters and add a JSON with our test data, in this case:

{ "name": "Diego" }

Then, we can hit the Run button and see the results in the Output and Logs windows.

Step 3 - Adding the LLM Code

First, we will change our variable names to reflect what we're doing: producing summaries from links. Instead of name we'll use a link variable. Also, we'll return a proper HTTP response, with
statusCode 200 in case we got a link.

export async function main(args) { const link = args.link;
if (!link) { return {
statusCode: 400,
headers: { "Content-Type": "application/json" }, body: { error: "Please send a link" }
};
}
const body = { URL: link }; return {
statusCode: 200,
headers: { "Content-Type": "application/json" }, body: body
};
}

After changing the code, hit the "Save" button to avoid losing your changes. You can now re-run your function and check the results.

While we're at it, let's also change the Test Parameters (if you're testing directly from the browser) to:

(
"link": "https://en.wikipedia.org/wiki/Seville"
)

Seems like we're about to travel to Seville!

After that, we'll add the code to interact with Mistral. We need to send a POST with the prompt to their endpoint. For this, we'll need an API key from Mistral AI (see prerequisites). Save the code, but don't run the function (yet) as we need to add the secret.

const API_KEY = process.env.MISTRAL_API_KEY; // secrets are stored in the function's Settings
const API_URL = 'https://api.mistral.ai/v1/chat/completions';

async function chatWithLeChat(prompt) { try {
// call the Mistral endpoint with POST const res = await fetch(API_URL, {
method: 'POST', headers: {
'Authorization': `Bearer ${API_KEY}`, 'Content-Type': 'application/json',
},
body: JSON.stringify({ model: 'mistral-tiny',
messages: [{ role: 'user', content: prompt }],
}),
});

const data = await res.json(); console.log(res.status, data);
return data?.choices?.[0]?.message?.content || null;
} catch (error) {
console.error('Error:', error?.message || error); return null;
}

Step 4 - Adding Secrets

As you can see, in our code, we don't embed the secrets (like API keys). Instead, we use the Functions environment to provide these:

const API_KEY = process.env.MISTRAL_API_KEY;

To add this secret MISTRAL_API_KEY we click on Settings. Then, we scroll down to Environment Variables. There, we can click on Edit

Now, click on Add and then Save. You should see your new environment variable now!

Step 5 - getting the LLM summary

It's time to call our chatWithLeChat function from main! To do that, we 1st create a system prompt that includes the link we received and instructs the LLM on what to do. Then, we'll call Mistral:

const prompt = 'I need a summary of this page: ' + link;

const summary = await chatWithLeChat(prompt);

We will extend our body to include the summary:

const body = { URL: link , summary: summary
};

This is all our code right now:

export async function main(args) { const link = args.link;
if (!link) { return {
statusCode: 400,
headers: { "Content-Type": "application/json" }, body: { error: "Please send a link" }
};
}

const prompt = 'I need a summary of this page: ' + link; const summary = await chatWithLeChat(prompt);
const body = { URL: link , summary: summary
};

return { statusCode: 200,
headers: { "Content-Type": "application/json" }, body: body
};
}

const API_KEY = process.env.MISTRAL_API_KEY; // secrets are stored in the function's Settings
const API_URL = 'https://api.mistral.ai/v1/chat/completions';

async function chatWithLeChat(prompt) { try {
// call the Mistral endpoint with POST const res = await fetch(API_URL, {
method: 'POST', headers: {
'Authorization': `Bearer ${API_KEY}`, 'Content-Type': 'application/json',
},
body: JSON.stringify({ model: 'mistral-tiny',
messages: [{ role: 'user', content: prompt }],
}),
});

const data = await res.json(); console.log(res.status, data);
return data?.choices?.[0]?.message?.content || null;
} catch (error) {
console.error('Error:', error?.message || error); return null;
}

If you save your function and run it, most probably you'll get an error message in LOGS:

2026-02-02T12:14:17.299547912Z stderr: The function exceeded its time limits of 3000 milliseconds. Logs might be missing.

We need to give our function more time to wait for the LLM to answer!

Setting Timeouts

Head to Settings again, and click Edit on Limits. By default you have 3 secs (3000 milliseconds) to run your function before you get a timeout. We'll set that to 10 secs. Save the change, go back to Source and run the function again.

After that change, you should get a summary of the page!

Persisting data into MongoDB

We're now ready to store our body object in a MongoDB collection. One of the nicest things about MongoDB is that you work with your application objects in memory, and when ready, you send them to MongoDB using the driver (a library that will interact directly with the database). No need for ORMs or complicated frameworks in this case.

We'll start by importing the MongoDB driver:

import { MongoClient } from "mongodb";

After that, we'll write a simple function that will:

receive our object as an argument, named page.
connect to MongoDB using a MongoClient instance and the connect method.
write it into MongoDB using insertOne

async function store_in_mongodb(page) {
const mongodb_URI = process.env.MONGODB_URI; // get the MongoDB URI from an environment variable

const client = new MongoClient(mongodb_URI); try {
// connect to MongoDB await client.connect();
const database = client.db('read_it_later'); const collection = database.collection('pages');

const result = await collection.insertOne(page); return result;
} finally {
await client.close();
}
};

As we did previously with our environment variable MISTRAL_API_KEY, we'll need to add a new environment variable in Settings called MONGODB_URI with the URI of our MongoDB installation. The URI looks like this:

mongodb+srv://user:password@cluster.something-here.mongodb.net/

The last step will be to call this new function in main:

await store_in_mongodb(greeting);

Once this is done, save your function and send some links to your function and see how the collection pages gets populated!

Conclusion

In this article we've explored how to write a serverless function that can interact with an LLM and store results in MongoDB. It's incredible how far a few lines of code can take us these days. Not so long ago you needed to spend days setting up web and database servers, hosting everything, exposing it to the Internet, writing your code, etc. Now, using DigitalOcean integrated suit of services and hosted applications you focus on what matters most: your application code. Our next step could be building a new function to read from MongoDB, that will be the base for our UI, or add Semantic Search using MongoDB's Vector Search capabilities.

Here you have the complete code listing for our function:

export async function main(args) { const link = args.link;
if (!link) { return {
statusCode: 400,
headers: { "Content-Type": "application/json" }, body: { error: "Please send a link" }
};
}

const prompt = 'I need a summary of this page: ' + link; const summary = await chatWithLeChat(prompt);
const body = { URL: link , summary: summary
};

await store_in_mongodb(body);

return { statusCode: 200,
headers: { "Content-Type": "application/json" }, body: body
};
}

const API_KEY = process.env.MISTRAL_API_KEY; // secrets are stored in the function's Settings
const API_URL = 'https://api.mistral.ai/v1/chat/completions';

async function chatWithLeChat(prompt) { try {
// call the Mistral endpoint with POST const res = await fetch(API_URL, {
method: 'POST', headers: {
'Authorization': `Bearer ${API_KEY}`, 'Content-Type': 'application/json',
},
body: JSON.stringify({ model: 'mistral-tiny',
messages: [{ role: 'user', content: prompt }],
}),
});

const data = await res.json(); console.log(res.status, data);
return data?.choices?.[0]?.message?.content || null;
} catch (error) {
console.error('Error:', error?.message || error); return null;
}
}
import { MongoClient } from "mongodb"; async function store_in_mongodb(page) {
const mongodb_URI = process.env.MONGODB_URI; // get the MongoDB URI from an environment variable

const client = new MongoClient(mongodb_URI); try {
// connect to MongoDB await client.connect();
const database = client.db('read_it_later'); const collection = database.collection('pages');

const result = await collection.insertOne(page); return result;
} finally {
await client.close();
}
};

Build a Movie/TV Show Tracker with MongoDB and TanStack Start

Didicodes — Fri, 17 Jul 2026 05:48:02 +0000

A few months ago, I found myself opening multiple different apps just to answer a simple question: "What episode was I on?"

I'd start a show, get distracted by work, switch to another series for a few weeks, then come back and have no idea where I'd stopped. Sometimes I'd scroll through episode lists trying to remember the last thing I watched. Other times, I'd accidentally rewatch an episode because I couldn't remember whether I'd already seen it. After doing this one too many times, I decided to build a small tracker for myself.

The idea was simple. Search for a show, add it to a watch list, and tap a button whenever you finish an episode. Behind that simple experience, though, there are a few interesting decisions to make. How should progress be stored? Should the show information be fetched from an external API every time, or stored in MongoDB? And how do you update both progress and status without introducing race conditions?

This tutorial explains how to build a movie/TV show tracker using MongoDB, Tanstack Start, and TMDB. You can find the finished project here in this GitHub repository.

What you will learn

How to design a data model for a TV show tracking application
How MongoDB aggregation pipelines can power both progress updates and application statistics
How TanStack Start server functions make it easy to work with MongoDB from a full-stack React application

Prerequisites

Before you start, you should have:

Node.js 18+ installed (npm is included by default)
A MongoDB Atlas account or a local MongoDB server
A free TMDB account, which gives you an API token
An understanding of React, TypeScript, and TanStack Start.

Setting up the project

Let’s set up our project for the application using TanStack Start's start-basic example, then add MongoDB. To do this, run the commands below on your terminal:

npx gitpick TanStack/router/tree/main/examples/react/start-basic tanstack-start-mongodb-tvshow-tracker
cd tanstack-start-mongodb-tvshow-tracker
npm install
npm install mongodb

gitpick copies only the start-basic folder from TanStack’s repo into a newly created directory named tanstack-start-mongodb-tvshow-tracker. npm install pulls down what the starter already uses (Zod is one of them), and npm install mongodb adds the official MongoDB nodejs driver.

At this point, the project has been created, but it still includes the files from the start-basic example. Before we start building the movie/TV show tracker, let's update the file structure to match the project. To make this easier, I've created a small setup script that removes the starter example files, updates the root route, and creates the folders and files we'll use throughout the rest of the tutorial. Run the command below from the root of your tanstack-start-mongodb-tvshow-tracker project:

curl -o setup.sh https://gist.githubusercontent.com/didicodes/e35a275795d3364ebb17e488724ca132/raw/7b403929ab93dd7f64ba2d701a0137277769f808/setup.sh && chmod +x setup.sh && ./setup.sh

After running the script, your project should look like the image below. Run npm run dev to confirm the project runs successfully. If you'd rather make the changes yourself, you can follow along using the project structure shown below:

We'll build out these files as we go, but setting up the structure now makes it easier to see where each piece of the application belongs.

Getting your credentials

The app needs a MongoDB connection string and a TMDB token. For MongoDB, create a free cluster in MongoDB Atlas, then open Connect, choose Drivers, and copy the Node.js connection string. It looks like mongodb+srv://<username>:<password>@<cluster-url>. For TMDB, sign in, open your API settings, request an API key if you don’t already have one, then copy the value labeled "API Read Access Token". Using your preferred IDE, open the tanstack-start-mongodb-tvshow-tracker project and add both credentials to the .env file as shown below

MONGODB_URI=your-mongodb-connection-string
TMDB_READ_ACCESS_TOKEN=your-tmdb-read-access-token

TanStack Start reads .env in development so that both credentials will be available as process.env values in server functions. They are read only on the server, so neither one reaches the browser.

Designing the show document

Before writing any database code, I wanted to decide what a TV show document should look like. As I started sketching out the data model, I caught myself thinking about features the app didn't even need yet. Things like, "Maybe one day I should track individual episodes," or "It would be nice to keep a viewing history," or even "What if I added personal notes?" Eventually, I realized that none of those ideas solved the problem I was trying to fix today. The only question this app really needed to answer was: How far through this show am I?

Once I focused on that question, the data model became much simpler. Before writing any MongoDB code, I wanted to capture that model in TypeScript so the rest of the application could share the same definitions. In this project, those shared types will live in src/lib/types.ts, so go there and add the code below:

// src/lib/types.ts
export interface ShowDocument {
  tmdbId: number;
  title: string;
  posterPath: string | null;
  totalSeasons: number;
  totalEpisodes: number;
  episodesWatched: number;
  status: "plan_to_watch" | "watching" | "completed";
  rating: number | null;
  createdAt: Date;
  updatedAt: Date;
}

Defining the fields was only part of the design. The next question was how to represent progress. My first instinct was to store every episode a user watches. That would make it possible to answer questions like "Which episodes have I seen?" or "What did I watch last week?". After sketching it out, though, storing individual episodes started to feel like overkill. If I'm halfway through a season, I don't really care whether the app knows I watched episode 4 or episode 5. What I care about is seeing that I've watched 12 out of 24 episodes. Once I looked at it that way, a single episodesWatched field felt like the simpler choice.

Progress wasn't the only tradeoff. I also went back and forth on how much information should come from TMDB versus how much should live in MongoDB. One option was to store only the TMDB ID and fetch everything else on page load. I decided against that approach. Every visit to the watch list would trigger another API request, even though details like a TV show's title or poster rarely change. Instead, I store a snapshot of the information I need when a TV show is first added, then read that snapshot directly from MongoDB from that point on. The tradeoff is that some values can become outdated over time. For example, if a TV show is renewed and a new season is released, the stored episode count won't update automatically. For a watch tracker, that's a tradeoff I'm happy to make in exchange for faster reads and fewer external API requests.

Connecting to MongoDB

Now that we know what a show document looks like, we need somewhere to store it. The next step is setting up a MongoDB connection that the rest of the application can reuse. Before opening a connection, I like to pull reusable values into a small config file. For this app, that's the database name, collection name, and a couple of connection settings. Go to src/config/mongodb.ts and add the code below:

// src/config/mongodb.ts
export const DB_NAME = "episode-tracker";
export const COLLECTION_NAME = "shows";

export const MONGODB_CONNECTION_CONFIG = {
  minPoolSize: 1,
  maxIdleTimeMS: 60000,
} as const;

The database and collection names are fairly straightforward. The connection settings are a little more interesting. Since TanStack Start server functions can run in short-lived environments, I don't want to create a new connection for every request. Keeping a connection around for a short period means later requests can reuse it instead of starting from scratch.

In this case, minPoolSize:1 keeps one connection ready to use, while maxIdleTimeMS:60000 allows idle connections to be cleaned up after about a minute. With the configuration in place, we can create a reusable MongoDB client. To do this, add the code below to src/lib/mongodb.ts:

// src/lib/mongodb.ts
import { MongoClient, type Collection, type Db } from "mongodb";
import {
  COLLECTION_NAME,
  DB_NAME,
  MONGODB_CONNECTION_CONFIG,
} from "../config/mongodb";
import type { ShowDocument } from "./types";

const MONGODB_URI = process.env.MONGODB_URI;

interface CachedConnection {
  client: MongoClient | null;
  db: Db | null;
  promise: Promise<{ client: MongoClient; db: Db }> | null;
}

const cached: CachedConnection = {
  client: null,
  db: null,
  promise: null,
};

export async function connectToDatabase(): Promise<{
  client: MongoClient;
  db: Db;
}> {
  if (cached.client && cached.db) {
    return { client: cached.client, db: cached.db };
  }

  if (cached.promise) {
    return cached.promise;
  }

  if (!MONGODB_URI) {
    throw new Error(
      "Missing MONGODB_URI environment variable. Add it to your .env file."
    );
  }

  cached.promise = MongoClient.connect(MONGODB_URI, {
    appName: "devrel-github-tanstack-tvshow-tracker",
    ...MONGODB_CONNECTION_CONFIG,
  })
    .then((client) => {
      const db = client.db(DB_NAME);

      cached.client = client;
      cached.db = db;
      cached.promise = null;

      return { client, db };
    })
    .catch((error) => {
      cached.promise = null;
      throw error;
    });

  return cached.promise;
}

export async function getShowsCollection(): Promise<Collection<ShowDocument>> {
  const { db } = await connectToDatabase();
  const collection = db.collection<ShowDocument>(COLLECTION_NAME);
  await collection.createIndex({ tmdbId: 1 }, { unique: true });
  return collection;
}

The goal of these helper functions is simple: avoid opening more MongoDB connections than necessary. Once a connection exists, we keep using it. If another request comes in while the connection is still being created, it waits for that same promise instead of starting a second connection attempt.

The tmdbId index is also created during this first connection step. That keeps getShowsCollection() focused on one job: returning the typed shows collection. It also means the server functions can call it without asking MongoDB to create the index each time.

The .catch() block clears the cached promise if the connection fails, so the next request can try again instead of reusing a failed connection attempt. From there, the rest of the app gets a typed collection that matches our ShowDocument shape.

Defining the application's data contract

Before moving on to the server functions, I wanted a single place to define the data shapes used throughout the application. By this point, we already know what a show document looks like in MongoDB. We also know that data will be flowing between the browser, our server functions, MongoDB, and TMDB. Rather than redefining those shapes in multiple places, we'll keep them together in src/lib/types.ts. This file will contain the application's statuses, Zod validation schemas, TypeScript interfaces, and a small helper for converting MongoDB documents into a browser-friendly format. Replace the code in the file with this:

// src/lib/types.ts
import { z } from "zod";
import type { ObjectId } from "mongodb";

export const WATCH_STATUSES = [
  "plan_to_watch",
  "watching",
  "completed",
] as const;

export type WatchStatus = (typeof WATCH_STATUSES)[number];

export const addShowSchema = z.object({
  tmdbId: z.number().int().positive(),
  status: z.enum(WATCH_STATUSES).default("plan_to_watch"),
});

export const updateShowSchema = z.object({
  id: z.string().min(1, "Show id is required"),
  status: z.enum(WATCH_STATUSES).optional(),
  rating: z.number().int().min(1).max(10).nullable().optional(),
  episodesWatched: z.number().int().min(0).optional(),
});

export const showIdSchema = z.object({
  id: z.string().min(1, "Show id is required"),
});

export const searchSchema = z.object({
  query: z.string().min(1, "Search query is required").max(120),
});

// How a show is stored in MongoDB
export interface ShowDocument {
  tmdbId: number;
  title: string;
  posterPath: string | null;
  totalSeasons: number;
  totalEpisodes: number;
  episodesWatched: number;
  status: WatchStatus;
  rating: number | null;
  createdAt: Date;
  updatedAt: Date;
}

// A stored show as it comes back from a query, including Mongo's _id
export interface ShowResponse extends ShowDocument {
  _id: ObjectId;
}

// How a show is sent to the browser. ObjectId and Date are not JSON-friendly, so this converts them to strings
export interface Show {
  id: string;
  tmdbId: number;
  title: string;
  posterPath: string | null;
  totalSeasons: number;
  totalEpisodes: number;
  episodesWatched: number;
  status: WatchStatus;
  rating: number | null;
  createdAt: string;
  updatedAt: string;
}

// Summary numbers for the stats bar
export interface LibraryStats {
  totalShows: number;
  episodesWatched: number;
  watching: number;
  completed: number;
  planToWatch: number;
}

export interface TmdbSearchResult {
  tmdbId: number;
  title: string;
  posterPath: string | null;
  year: string | null;
  overview: string;
}

export interface TmdbShowDetails {
  tmdbId: number;
  title: string;
  posterPath: string | null;
  totalSeasons: number;
  totalEpisodes: number;
}

export function documentToShow(doc: ShowResponse): Show {
  return {
    id: doc._id.toString(),
    tmdbId: doc.tmdbId,
    title: doc.title,
    posterPath: doc.posterPath,
    totalSeasons: doc.totalSeasons,
    totalEpisodes: doc.totalEpisodes,
    episodesWatched: doc.episodesWatched,
    status: doc.status,
    rating: doc.rating,
    createdAt: doc.createdAt.toISOString(),
    updatedAt: doc.updatedAt.toISOString(),
  };
}

Two separate concerns are being handled in this file:

The first is validation. The Zod schemas make sure the server only accepts the inputs we expect. For example, when a show is added, the client sends a TMDB ID and an optional status. Everything else comes directly from TMDB, so there's no reason to trust those values from the browser.
The second is keeping the different data shapes organized. A show exists in several forms as it moves through the application. Before a document is stored, there is no _id, so we use ShowDocument. Once MongoDB returns that document from a query, it now includes an _id, which is represented by ShowResponse. Finally, when the data is sent to the browser, the ObjectId and date fields need to become strings, which is represented by Show. Rather than repeating those conversions throughout the codebase, the documentToShow() helper handles them in one place.

Fetching show details from TMDB

When someone adds a show to their watch list, they aren't entering the title, poster URL, season count, or episode count manually. The only information we really need from them is the TMDB ID; everything else can come from TMDB. To keep that logic in one place, we'll create a TMDB client in src/lib/tmdb.ts that handles authentication, show search, and fetching the additional details we need before saving a document to MongoDB. Add the following code to src/lib/tmdb.ts:

// src/lib/tmdb.ts
import type { TmdbSearchResult, TmdbShowDetails } from "./types";

const TMDB_BASE_URL = "https://api.themoviedb.org/3";
const TMDB_TOKEN = process.env.TMDB_READ_ACCESS_TOKEN;

// Small wrapper around the TMDB REST API
async function tmdbFetch<T>(path: string): Promise<T> {
  if (!TMDB_TOKEN) {
    throw new Error(
      "Missing TMDB_READ_ACCESS_TOKEN environment variable. Add it to your .env file."
    );
  }

  const res = await fetch(`${TMDB_BASE_URL}${path}`, {
    headers: {
      Authorization: `Bearer ${TMDB_TOKEN}`,
      accept: "application/json",
    },
  });

  if (!res.ok) {
    throw new Error(`TMDB request failed with status ${res.status}`);
  }

  return res.json() as Promise<T>;
}

interface TmdbSearchResponse {
  results: Array<{
    id: number;
    name: string;
    poster_path: string | null;
    first_air_date?: string;
    overview?: string;
  }>;
}

interface TmdbDetailsResponse {
  id: number;
  name: string;
  poster_path: string | null;
  number_of_seasons: number;
  number_of_episodes: number;
}

// Search TV shows by title. Returns a trimmed list shaped for the UI
export async function searchShows(query: string): Promise<TmdbSearchResult[]> {
  const data = await tmdbFetch<TmdbSearchResponse>(
    `/search/tv?query=${encodeURIComponent(query)}&include_adult=false&page=1`
  );

  return (data.results ?? []).slice(0, 8).map((show) => ({
    tmdbId: show.id,
    title: show.name,
    posterPath: show.poster_path,
    year: show.first_air_date ? show.first_air_date.slice(0, 4) : null,
    overview: show.overview ?? "",
  }));
}

//Fetch the season and episode counts for a single show
export async function getShowDetails(tmdbId: number): Promise<TmdbShowDetails> {
  const data = await tmdbFetch<TmdbDetailsResponse>(`/tv/${tmdbId}`);

  return {
    tmdbId: data.id,
    title: data.name,
    posterPath: data.poster_path,
    totalSeasons: data.number_of_seasons ?? 0,
    totalEpisodes: data.number_of_episodes ?? 0,
  };
}

While building this, I ran into one small limitation of the TMDB API. The search endpoint is great for helping users find a TV show, but it doesn't include the season and episode totals our tracker needs. That's why there are two functions in this file: searchShows(), which returns the results displayed in the search UI, and getShowDetails(), which runs when a show is added to the library. That second request fetches the complete show details, including the season and episode counts we store in MongoDB. You'll also notice that the TMDB token never leaves the server, as it comes from an environment variable accessible only to the server-side code. The base URL for the poster image is different. Since it's just a constant rather than a secret, we can safely keep it in src/lib/constants.ts alongside the human-readable status labels:

//src/lib/constants.ts
import type { WatchStatus } from "./types";

// TMDB serves poster images from a few fixed sizes. w342 looks crisp on cards.
export const TMDB_POSTER_BASE = "https://image.tmdb.org/t/p/w342";

export const STATUS_LABELS: Record<WatchStatus, string> = {
  plan_to_watch: "Plan to watch",
  watching: "Watching",
  completed: "Completed",
};

At this point, we can search for TV shows and fetch all the information needed to save them. The next step is wiring these functions into our server so the application can start reading from and writing to MongoDB.

Building the server functions

Now that we can read data from MongoDB and fetch metadata from TMDB, we can start wiring the application together. TanStack Start uses createServerFn to define server-side functions. Each function validates its input with Zod before executing the handler, so by the time our code runs, we already know the data is valid and correctly typed. We'll start with the TMDB search function. Add the following code to src/server/tmdb.ts:

// src/server/tmdb.ts
import { createServerFn } from "@tanstack/react-start";
import { searchSchema, type TmdbSearchResult } from "~/lib/types";
import { searchShows } from "~/lib/tmdb";

export const searchTmdb = createServerFn({ method: "GET" })
  .validator(searchSchema)
  .handler(async ({ data }): Promise<TmdbSearchResult[]> => {
    try {
      return await searchShows(data.query);
    } catch (error) {
      console.error("TMDB search failed:", error);
      throw new Error("Failed to search TMDB");
    }
  });

Most of the application's behavior lives in shows.ts. Rather than pasting several hundred lines of code into this article, I've included the complete code in this repo at src/server/shows.ts, so add it to your project. Even though we aren't walking through every line, there are three functions worth understanding because they capture most of the application's behavior:

addShow checks whether a show with the same tmdbId already exists. If it doesn't, it fetches the show's details from TMDB, creates the MongoDB document, and returns the newly created show.
incrementProgress is where I ended up moving more logic into MongoDB. My first version updated the episode count in JavaScript after reading the document. Eventually, I realized the database already had everything it needed. The aggregation update increments the episode count, prevents it from going past the total number of episodes, and automatically updates the show's status when necessary.
getStats is where storing a dedicated status field starts paying off. Instead of loading every show into memory and counting them in JavaScript, MongoDB calculates the dashboard statistics with a single aggregation pipeline and returns only the numbers the UI needs.

With these server functions in place, the backend can search, store, update, and summarize data. The only missing piece is the user interface.

Building the user interface

Up to this point, everything we've built has lived on the server. The final step is putting a user interface on top of it. I kept the front end deliberately simple. The application lives in a single route that loads the watch list and statistics, renders them on the page, and refreshes itself whenever data changes. Instead of manually updating local state after every action, I simply invalidate the route so the loader fetches the latest data from MongoDB.

The route shown below is a shortened version of the final implementation that shows the overall pattern. The complete version is available in the GitHub repository at src/routes/index.tsx.

// src/routes/index.tsx
export const Route = createFileRoute("/")({
  loader: async () => {
    const [shows, stats] = await Promise.all([getShows(), getStats()]);
    return { shows, stats };
  },
  component: Home,
});

function Home() {
  const { shows, stats } = Route.useLoaderData();
  const router = useRouter();
  const [busyId, setBusyId] = useState<string | null>(null);

  // Re-run the loader so that the shows and stats reflect the latest database state
  const refresh = () => router.invalidate();

  async function handleIncrement(id: string) {
    setBusyId(id);
    try {
      await incrementProgress({ data: { id } });
      await refresh();
    } finally {
      setBusyId(null);
    }
  }

  // ...search, add, status, rating, and delete handlers follow the same shape
  return <main className="page">{/* stats bar, search, and the show list */}</main>;
}

I also moved the stylesheet to the repository (src/styles/app.css) to keep this article focused on the application architecture rather than on visual styling. Once those files are in place, the application is ready to run. Yay!

One lesson from building this project was that I didn't need much client-side state at all. Since the loader already fetches the latest data, I could treat the database as the source of truth and reload the route after each action. That approach kept the component small and avoided much of the bookkeeping in the browser.

Try it locally

Let’s test it out. To do this, run the npm run dev command on your terminal, then go to http://localhost:3000/, search for a show, and add it to your library. Try incrementing the episode count a few times and watch the progress update. The watch list, statistics, and progress tracking should all stay in sync as the route reloads after each change. At this point, the application is complete.

As you can see, on the app, you can now search for TV shows, add them to the library, track viewing progress, update ratings and status, and calculate dashboard statistics, all using TanStack Start server functions backed by MongoDB.

Final thoughts

This project started because I wanted a simple answer to a simple question: "How far through this show am I?"

Along the way, we built a small full-stack application using TanStack Start, MongoDB, TMDB, Zod, and server functions. The final result is intentionally simple. Search for a show, add it to your library, track your progress, and see a summary of your watching habits.

There are plenty of directions you could take this project next. User accounts, episode history, recommendations, and shared watch lists would all fit naturally on top of what we've built here. I stopped here because the app already does exactly what I wanted: it tells me where I am in a TV show without any extra complexity. If you'd like to keep building, the full source code is available in this repository.

Key Takeaways

Building a small TMDB client gave the application a single place to handle API requests, making future changes easier without affecting the rest of the codebase.
TanStack Start server functions, together with MongoDB, provide a clear separation between the user interface and the application's data and business logic.
MongoDB aggregation pipelines and atomic update operations simplify application code by moving data processing closer to where the data is stored.

Building a Vibe-Based Music Recommender with MongoDB and Voyage AI

Jordan Smith — Tue, 14 Jul 2026 14:08:37 +0000

Have you ever found yourself in the mood for a certain type of music but not any specific artists, songs, or genres in mind? Maybe you just had a long day and are looking for something with mellow and relaxing vibes. Or maybe it's a Friday afternoon, and you're ready for something to pump you up for the weekend. In times like these, traditional search experiences won't work. But today, we're going to build something that will.

In a traditional search experience, you would search for a term and get back a list of songs or artists that contain keywords from that search term somewhere in their name or description. But that's not what we want here. What we actually want is semantic search: a system that understands what you mean, not just what you typed. That's exactly what we're building. This music recommender takes a natural-language description of the music's “vibe” and finds songs that match that feel. And it's all powered by MongoDB Vector Search and Voyage AI.

In this tutorial, you’ll learn how to build a working web application where you can type something like "upbeat road trip with the windows down" and get back a ranked list of songs that fit that feeling. Let's get into it!

How It Actually Works

Before we write any code, let's go through a quick overview of how the search functionality in this app actually works.

The key to all of this is vector embeddings. A vector embedding is a way of representing text (or other data like images or audio) as an array of numbers, or vectors. Data with similar semantic meanings produces similar vectors, so you can compare all of your vectors mathematically to find data that's similar in meaning.

Here's how it actually works in this app:

Every song in our sample collection has an LLM-generated "vibe description". This is a 2-3 sentence description of what the song "feels" like to listen to. Vector embeddings were generated from these descriptions using Voyage AI and are stored with the songs.
When a user types a search query, it is converted into a vector using the same Voyage AI vector embedding model.
We use MongoDB Vector Search to find the songs in the collection whose vectors are closest to the query vector.
The results are returned to the user, ranked by how closely their vector matches the query vector.

Full Project Code

If you want to see the full project with all of the code already written, you can find it on GitHub. I'll explain all the elements throughout this tutorial, and you can clone the repository and follow along as well.

Prerequisites

Before getting started, you'll need a few things in place:

Node.js (v18 or later) installed on your machine
A MongoDB Atlas account with a free cluster
A Voyage AI API key - you can get one from the MongoDB Atlas UI

Project Setup

With that out of the way, let’s get going with the actual code.

Start by creating a new project directory and initializing it:

mkdir music-recommender
cd music-recommender
npm init -y

Then install the dependencies:

npm install express mongodb voyageai dotenv

Create a .env file in the root of your project with the following variables:

MONGODB_URI=<Your MongoDB Connection String>
MONGODB_DATABASE=song_vibe_search

VOYAGE_API_KEY=<Your Voyage AI API Key>

Next, you'll need to download the seed files. These files contain scripts that will initialize your database by using sample data. The scripts also contain code that you can use to generate your own descriptions and embeddings, but we won't get into that until the Deeper Dive: Generating Embeddings section at the end of the tutorial.

The sample data is a small sample of 500 songs from the Free Music Archive with pre-generated "vibe descriptions" and embeddings. You can download the seed files from the /seed directory in the GitHub repository for this tutorial.

After you have downloaded the seed folder, place it at the root of your project. Then run the following command to connect to your, insert the seed data, and create a vector search index:

npm run seed

This creates a new database called song_vibe_search and a new collection called songs. The song collection contains 500 documents, each representing a song with a "vibe description" and an embedding vector. It also creates a vector search index on the embedding field by running the following code:

await collection.createSearchIndex({
    name: VECTOR_INDEX_NAME,
    type: "vectorSearch",
    definition: {
    fields: [
        {
        type: "vector",
        path: "embedding",
        numDimensions: 512,
        similarity: "cosine",
        },
    ],
    },
});

This is what makes $vectorSearch possible. We need to tell MongoDB the shape of our embedding field so it can build the right index structure.

The index contains the type of index (in our case, vector), the path to the field, and the following parameters that tell MongoDB how to compare vectors:

numDimensions: 512 — This must match the output size of your embedding model. Voyage AI's voyage-3-lite\ model produces 512-dimensional vectors. If you're using a different model, you should be able to find the number of dimensions in the documentation for your model.
similarity: "cosine" — This is the distance metric used to compare vectors. Cosine similarity works well for text embeddings because it measures the angle between vectors rather than their raw distance, which makes it robust to differences in text length.

Now that we have our data and our vector search index, we are ready to build the main application.

Implementing Vector Search

In the root of your project, create a new directory called src and create a new file called server.ts in that directory. This is where we will implement the search logic that powers our recommendations.

First, we need to set our imports, our interfaces and constants, and establish our connection to MongoDB. Paste the following code into server.ts:

import "dotenv/config";
import { MongoClient } from "mongodb";
import { VoyageAIClient } from "voyageai";

const MONGODB_URI = process.env.MONGODB_URI;
const VOYAGE_API_KEY = process.env.VOYAGE_API_KEY;

if (!MONGODB_URI) {
  throw new Error(
    "Missing MONGODB_URI in environment. Ensure that you have created a .env file and added your connection string."
  );
}

if (!VOYAGE_API_KEY) {
  throw new Error(
    "Missing VOYAGE_API_KEY in environment. Ensure that you have created a .env file and added your Voyage AI key."
  );
}

export const client = new MongoClient(MONGODB_URI, {
  appName: "devrel-tutorial-javascript-music-recommender",
});

const voyage = new VoyageAIClient({ apiKey: VOYAGE_API_KEY });

const DB_NAME = process.env.MONGODB_DATABASE ?? "song_vibe_search";
const COLLECTION_NAME = "songs";
export const VECTOR_INDEX = "vibe_vector_index";
const EMBEDDING_MODEL = "voyage-3-lite";
export const NUM_RESULTS = 5;
export const NUM_CANDIDATES = 100;

export interface SearchResult {
  title: string;
  artist: string;
  genre_top: string;
  genres: string[];
  score: number;
}

export interface SearchResponse {
  results: SearchResult[];
  queryVector: number[];
}

If you're unfamiliar with vector search, some of the constants above may not make sense quite yet, but we'll explain them more as they're used.

Embedding the query

When a user submits a search, the first thing we need to do is embed that query by using the same Voyage AI model that was used to embed our song descriptions. Paste the following code into server.ts to create a search function and embed the query. Note the comment that shows where we will add the vector search query code in a later step.

export async function searchSongs(query: string): Promise<SearchResponse> {

  // Embed the user's query by using Voyage AI.
  const embedResponse = await voyage.embed({
    model: EMBEDDING_MODEL,
    input: [query],
    inputType: "query",
  });

  const queryVector = embedResponse.data![0].embedding!;

  // vector search query code here
}

This code specifies a few things. First, we specify the embedding model to use (in our case, we pass in the constant that we set earlier). We then pass the actual user query as an array to the input parameter. Finally, we specify the inputType as "query". Voyage AI distinguishes between the following two types of input:

"document" — used when embedding data that gets saved into your database (the song vibe descriptions, in our case)
"query" — used when embedding a search query at runtime

Using the right input type improves retrieval quality. It's a small but important thing to keep in mind when implementing vector search applications.

Now that we have our query embedded, it's time to use it to search for similar songs. Paste the following code to complete our search function. Ensure that this code is pasted inside the function that we defined in the previous code block.

  // ... previous embedding code here

  // vector search query code here 
  const collection = client.db(DB_NAME).collection(COLLECTION_NAME);

  const results = await collection
    .aggregate<SearchResult>([
      {
        $vectorSearch: {
          index: VECTOR_INDEX,
          path: "embedding",
          queryVector,
          numCandidates: NUM_CANDIDATES,
          limit: NUM_RESULTS,
        },
      },
      {
        $project: {
          _id: 0,
          title: 1,
          artist: 1,
          genre_top: 1,
          genres: 1,
          score: { $meta: "vectorSearchScore" },
        },
      },
    ])
    .toArray();

  return { results, queryVector };

To implement vector search, we use the $vectorSearch stage of the aggregation pipeline. The $vectorSearch stage takes the following parameters:

index — The name of the vector search index we created during seeding
path — The field in the document that holds the embedding (in this case, named "embedding").
queryVector — The vector we just generated from the user's query.
numCandidates — How many candidate documents to consider before ranking. A higher number means the search will be more thorough but slower.
limit — How many results to return.

After the search stage, a $project stage shapes our output, with { $meta: "vectorSearchScore" } pulling in the similarity score for each result — a number between 0 and 1, where higher means more similar.

And that's all we need for our search functionality! Next, we'll build out a simple server with an endpoint to call this function.

Building the Server

Create a new file called server.ts in the src directory. This is where we will create any endpoints we might need. In this tutorial, we will create a single endpoint to search for songs.

To start, paste the following code into server.ts, which specifies the imports and sets up an Express server:

import "dotenv/config";
import express from "express";
import path from "path";
import { client, searchSongs, VECTOR_INDEX, NUM_RESULTS, NUM_CANDIDATES } from "./search";

const app = express();
const PORT = process.env.PORT ?? 3000;

app.use(express.json());
app.use(express.static(path.join(__dirname, "../public")));

Now, we'll create our POST route that calls the searchSongs function we created earlier. Paste the following code into server.ts:

app.post("/api/search", async (req, res) => {
  const { query } = req.body;

  if (!query || typeof query !== "string" || query.trim() === "") {
    res.status(400).json({ error: "A search query is required." });
    return;
  }

  let results, queryVector;
  try {
    ({ results, queryVector } = await searchSongs(query.trim()));
  } catch (err) {
    console.error("Search failed:", err);
    res.status(500).json({ error: "Search failed. Please try again." });
    return;
  }

  res.json({
    results
  });
});

In the preceding code, it first checks that the query is valid. Then it sends the query to our searchSongs() function and returns the results. If there is an error, it returns an error message.

The last thing we need to do is write the code that actually starts the server and handles signals to close the client cleanly on shutdown. Paste the following code into server.ts:

async function startServer() {
  await client.connect();
  console.log("Connected to MongoDB Atlas.");

  app.listen(PORT, () => {
    console.log(`Server running at http://localhost:${PORT}`);
  });

  process.on("SIGINT", async () => {
    await client.close();
    process.exit(0);
  });

  process.on("SIGTERM", async () => {
    await client.close();
    process.exit(0);
  });
}

startServer();

Now that we have our server set up, let's create a simple front end to interact with it.

Building the Front End

Since the purpose of this tutorial is the search functionality itself, we'll keep the front end simple. Create a new directory called public in the root of your project (it's important to keep the name as public since our server is already set up to serve files from this directory). Then, create a new file called index.html in that folder.

Paste the following code into index.html:

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>Vibe Search</title>
    <link rel="stylesheet" href="style.css" />
  </head>
  <body>
    <div class="container">
      <header>
        <h1>Vibe Search</h1>
        <p>Describe a mood or feeling and we'll find songs that match.</p>
      </header>

      <form id="search-form">
        <input
          id="query-input"
          type="text"
          placeholder="e.g. melancholy rainy afternoon, upbeat road trip..."
          autocomplete="off"
        />
        <button type="submit">Search</button>
      </form>
      <p class="token-note">Note: Each search generates a Voyage AI embedding and consumes tokens.</p>

      <div id="results"></div>
    </div>

    <script src="app.js"></script>
  </body>
</html>

This HTML creates a simple page with a header, an input box, and a section for results.

Now we need to create the style.css and app.js files. Create a new file called style.css in the public directory and paste the following code into it:

*, *::before, *::after {
  box-sizing: border-box;
  margin: 0;
  padding: 0;
}

body {
  font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif;
  background: #0f0f0f;
  color: #e8e8e8;
  min-height: 100vh;
  display: flex;
  flex-direction: column;
  align-items: center;
  justify-content: center;
  padding: 2rem 1rem;
}

.container {
  width: 100%;
  max-width: 720px;
}

header {
  text-align: center;
  margin-bottom: 2.5rem;
}

header h1 {
  font-size: 2.5rem;
  font-weight: 700;
  letter-spacing: -0.5px;
  margin-bottom: 0.5rem;
}

header p {
  color: #777;
  font-size: 1rem;
}

/* ── Search form ── */

#search-form {
  display: flex;
  gap: 0.5rem;
  margin-bottom: 2.5rem;
}

#query-input {
  flex: 1;
  padding: 0.75rem 1rem;
  font-size: 1rem;
  background: #1a1a1a;
  border: 1px solid #333;
  border-radius: 8px;
  color: #e8e8e8;
  outline: none;
}

#query-input:focus {
  border-color: #555;
}

#query-input::placeholder {
  color: #555;
}

#search-form button {
  padding: 0.75rem 1.5rem;
  font-size: 1rem;
  font-weight: 600;
  background: #00ed64;
  color: #000;
  border: none;
  border-radius: 8px;
  cursor: pointer;
}

#search-form button:hover {
  background: #00c853;
}

/* ── Token note ── */

.token-note {
  font-size: 1rem;
  color: #989898;
  text-align: center;
  margin-top: 0.6rem;
  margin-bottom: 0;
}

/* ── Spinner ── */

.spinner {
  width: 28px;
  height: 28px;
  border: 3px solid #2a2a2a;
  border-top-color: #00ed64;
  border-radius: 50%;
  margin: 2rem auto;
  animation: spin 0.7s linear infinite;
}

@keyframes spin {
  to { transform: rotate(360deg); }
}

/* ── Result cards ── */

#results {
  display: flex;
  flex-direction: column;
  gap: 1rem;
}

.card {
  background: #1a1a1a;
  border: 1px solid #2a2a2a;
  border-radius: 12px;
  overflow: hidden;
}

.card-color-bar {
  height: 6px;
}

.card-body {
  padding: 1.25rem 1.5rem;
}

.card-title {
  font-size: 1.1rem;
  font-weight: 600;
  margin-bottom: 0.2rem;
}

.card-artist {
  color: #777;
  font-size: 0.9rem;
  margin-bottom: 0.85rem;
}

.tags {
  display: flex;
  flex-wrap: wrap;
  gap: 0.35rem;
  margin-bottom: 1rem;
}

.tag {
  font-size: 0.75rem;
  padding: 0.2rem 0.65rem;
  border-radius: 999px;
  background: #252525;
  color: #999;
}

.card-footer {
  display: flex;
  justify-content: space-between;
  align-items: center;
}

.score {
  font-size: 0.8rem;
  color: #555;
}

.fma-link {
  font-size: 0.8rem;
  color: #00ed64;
  text-decoration: none;
}

.fma-link:hover {
  text-decoration: underline;
}

.message {
  text-align: center;
  color: #555;
  padding: 2rem 0;
  font-size: 0.95rem;
}

.message.error {
  color: #e05252;
}

The CSS added here is outside of the scope of this tutorial, but it just makes the app easier to look at and use.

The last file we need to create is app.js, which contains the front-end JavaScript code. Create a new file called app.js in the public directory and paste the following code into it:

const DEFAULT_GRADIENT = "linear-gradient(135deg, #434343, #1a1a1a)";

function renderResults(results) {
  const container = document.getElementById("results");

  if (results.length === 0) {
    container.innerHTML = '<p class="message">No results found. Try a different vibe.</p>';
    return;
  }

  container.innerHTML = results.map((result) => {
    const score = Math.round(result.score * 100);
    const tags = result.genres.map((g) => `<span class="tag">${g}</span>`).join("");

    return `
      <div class="card">
        <div class="card-color-bar" style="background: ${DEFAULT_GRADIENT}"></div>
        <div class="card-body">
          <div class="card-title">${result.title}</div>
          <div class="card-artist">${result.artist}</div>
          <div class="tags">${tags}</div>
          <div class="card-footer">
            <span class="score">Match: ${score}%</span>
            <a class="fma-link" href="https://www.youtube.com/results?search_query=${encodeURIComponent(result.title + ' ' + result.artist)}" target="_blank" rel="noopener">Find on YouTube →</a>
          </div>
        </div>
      </div>
    `;
  }).join("");
}

document.getElementById("search-form").addEventListener("submit", async (e) => {
  e.preventDefault();

  const query = document.getElementById("query-input").value.trim();
  if (!query) return;

  const container = document.getElementById("results");
  container.innerHTML = '<div class="spinner"></div>';

  const response = await fetch("/api/search", {
    method: "POST",
    headers: { "Content-Type": "application/json" },
    body: JSON.stringify({ query }),
  });

  if (!response.ok) {
    container.innerHTML = '<p class="message error">Something went wrong. Please try again.</p>';
    return;
  }

  const { results } = await response.json();
  renderResults(results);
});

This JavaScript code adds interactivity to the page. It sends the search query to the server and displays the search results.

Running the App

At this point, you should have everything set up. Make sure you have already run the npm run seed\ command from earlier in the tutorial.

Then, run the following command to start the server:

npm run dev

Open http://localhost:3000 and try a few searches, such as:

- "melancholy rainy afternoon"

- "high energy workout, aggressive beats"

- "peaceful background music for focusing"

You should see the results ranked by similarity score, with a link to YouTube so you can listen to the songs (if they're uploaded there). In a real application, you would want to serve the audio files or link directly to a streaming service, but for the sample data we're using here, searching YouTube is the only free option without license limitations. If you want to expand on this with different music, check out APIs for streaming services like Spotify.

Next Steps

And that's the basics! You've now built a working music recommender that uses MongoDB Vector Search and Voyage AI to find songs based on their "vibe". There are lots of ways you can expand on this as well. You could modify this to use music from different sources, expand the search functionality to include other metadata or filtering options, or even implement user profiles so that you can create personalized playlists and recommendations based on their likes, dislikes, and listening habits.

In the last section of this tutorial, we'll go through how the data for this tutorial was embedded. This can be useful if you're interested in adapting this tutorial to use your own data, or if you just want to learn more about working with vector embeddings.

Deeper Dive: Generating Embeddings {#deeper-dive:-generating-embeddings}

If you look at the GitHub repo, you'll notice that there are a few files in the seed directory that we didn't use in this tutorial. The fetchTracks.ts file contains code for downloading metadata about the songs that were downloaded from the Free Music Archive. This file will be specific to the Free Music Archive data, so we won't go into depth on it here.

The generateVibeDescriptions.ts and generateEmbeddings.ts files contain the code used to generate the "vibe descriptions" and embeddings for the sample data. The generateVibeDescriptions.ts file uses the Gemini API to generate a text description of the "vibe" of each song. The generateEmbeddings.ts file uses the Voyage AI API to generate an embedding for each description.

Generating Vibe Descriptions

Before generating embeddings from our data, we need some kind of description to embed. Raw metadata like title, artist, and genre isn't rich enough to embed meaningfully on its own, so we use Google Gemini to generate a 2-3 sentence vibe description for each track. Then, we save these descriptions in a new field in our documents called vibe_description.

To run the code to generate the descriptions, you'll need to get a Gemini API key. You can do this by going to the AI Studio and creating a new project. Then, add your API key to the .env file:

GEMINI_API_KEY=<Your Gemini API Key>

Then, create a file in the seed directory called generateVibeDescriptions.ts and paste the following code into it:

import "dotenv/config";
import { GoogleGenAI } from "@google/genai";
import type { RawTrack } from "./fetchTracks";

const GEMINI_API_KEY = process.env.GEMINI_API_KEY;
const GEMINI_MODEL = "gemini-3-flash-preview";
const BATCH_SIZE = 50;

export interface TrackWithVibe extends RawTrack {
  vibe_description: string;
}

function buildPrompt(tracks: RawTrack[]): string {
  const trackList = tracks
    .map(
      (t, i) =>
        `${i + 1}. Title: "${t.title}" | Artist: "${t.artist}" | Album: "${t.album}" | Genres: ${t.genres.join(", ")}`
    )
    .join("\n");

  return `For each track listed below, write a 2-3 sentence vibe description in third person. Focus on mood, energy, atmosphere, and ideal listening context. Avoid simply restating the genre — describe how it feels to listen to it. Return a JSON array of strings, one description per track, in the same order as the input. Return only the JSON array with no additional text. Tracks: ${trackList}`;
}

function parseResponse(text: string): string[] {
  // Extract the JSON array by finding the outermost [ ... ] bounds.
  // This handles cases where the model adds explanatory text before or after the JSON.
  const start = text.indexOf("[");
  const end = text.lastIndexOf("]");
  if (start === -1 || end === -1 || end < start) {
    throw new Error(`No JSON array found in model response:\n${text}`);
  }
  return JSON.parse(text.slice(start, end + 1));
}

export async function generateVibeDescriptions(
  tracks: RawTrack[]
): Promise<TrackWithVibe[]> {
  if (!GEMINI_API_KEY) {
    throw new Error(
      "GEMINI_API_KEY is not set. Add it to your .env file.\n" +
      "Get a free key at: https://aistudio.google.com/apikey"
    );
  }

  const ai = new GoogleGenAI({ apiKey: GEMINI_API_KEY });
  const results: TrackWithVibe[] = [];
  const totalBatches = Math.ceil(tracks.length / BATCH_SIZE);

  console.log(`\nGenerating vibe descriptions via Gemini (${totalBatches} batches)...`);

  for (let i = 0; i < tracks.length; i += BATCH_SIZE) {
    const batch = tracks.slice(i, i + BATCH_SIZE);
    const batchNum = Math.floor(i / BATCH_SIZE) + 1;

    console.log(`  Batch ${batchNum}/${totalBatches}`);

    const response = await ai.models.generateContent({
      model: GEMINI_MODEL,
      contents: buildPrompt(batch),
    });

    const descriptions = parseResponse(response.text ?? "");

    if (descriptions.length !== batch.length) {
      throw new Error(
        `Batch ${batchNum}: expected ${batch.length} descriptions, got ${descriptions.length}`
      );
    }

    for (let j = 0; j < batch.length; j++) {
      results.push({ ...batch[j], vibe_description: descriptions[j] });
    }

  }

  console.log(`  ${results.length} vibe descriptions generated.`);
  return results;
}

In this code, the first function creates the prompt that we send to Gemini. The prompt is a list of all the tracks we want to generate descriptions for, along with instructions for how we want the descriptions formatted.

Then, the generateVibeDescriptions function takes our tracks and sends them to Gemini along with that prompt. Gemini will respond with a JSON array of strings, each of which is a vibe description for a track. We parse the response (and strip any extraneous text that Gemini might have added) and return the tracks with their new descriptions.

Generating Embeddings

Now that we have our vibe descriptions, we can generate embeddings for them. To do this, we use the same Voyage AI model that we used to embed our search queries earlier in the tutorial. This process is very similar to the process that we use to embed our search queries.

Create a file in the seed directory called generateEmbeddings.ts and paste the following code into it:

import "dotenv/config";
import { VoyageAIClient } from "voyageai";
import type { TrackWithVibe } from "./generateVibeDescriptions";

const EMBEDDING_MODEL = "voyage-3-lite";

export interface TrackWithEmbedding extends TrackWithVibe {
  embedding: number[];
}

export async function generateEmbeddings(
  tracks: TrackWithVibe[]
): Promise<TrackWithEmbedding[]> {
  const VOYAGE_API_KEY = process.env.VOYAGE_API_KEY;
  if (!VOYAGE_API_KEY) {
    throw new Error("Missing VOYAGE_API_KEY in environment");
  }

  const voyage = new VoyageAIClient({ apiKey: VOYAGE_API_KEY });

  console.log(`\nGenerating embeddings for ${tracks.length} tracks...`);

  const response = await voyage.embed({
    model: EMBEDDING_MODEL,
    input: tracks.map((t) => t.vibe_description),
    inputType: "document",
  });

  const results: TrackWithEmbedding[] = [];

  for (let i = 0; i < tracks.length; i++) {
    const embeddingData = response.data?.[i];
    if (!embeddingData?.embedding) {
      console.warn(`  Warning: no embedding returned for "${tracks[i].title}" — skipping`);
      continue;
    }
    results.push({ ...tracks[i], embedding: embeddingData.embedding });
  }

  console.log(`  ${results.length} embeddings generated.`);
  return results;
}

The main difference in embedding this data is that we specify the input type as "document" rather than "query". This distinction is important when embedding data with Voyage AI, as it embeds the data slightly differently depending on the type you specify.

Putting It All Together

Now that we have the code to generate descriptions and embeddings, we need to run it when seeding our data. To do so, we just need to modify the index.ts file in the seed directory and call the functions before inserting the data into the database. It will look something like this:

const tracksWithVibes = await generateVibeDescriptions(rawTracks);

const tracksWithEmbeddings = await generateEmbeddings(tracksWithVibes);

const documents = tracksWithEmbeddings.map((track) => ({
...track,
seeded_at: new Date(),
}));

console.log(`\nInserting ${documents.length} documents into MongoDB...`);
await collection.insertMany(documents);

Note that you will likely have to modify the code you created earlier so that the flow of the seed script doesn't insert multiple times (since we originally set it up to use pre-built data).

Now, run the seed command again, and you should see the script generate descriptions and embeddings for the data before inserting it into the database. Then you can run the app again to try searching through the data.

Conclusion

You have now built a fully functional application that uses MongoDB Vector Search and Voyage AI to recommend music based on their general “vibes.” While this app is pretty simple and provides only basic functionality, you can easily use it as a baseline to build much more sophisticated applications.

Hopefully, this gives you some insight into just how powerful MongoDB Vector Search can be. If you want to learn more, check out our documentation or leave a comment letting me know how you’d like to see this expanded on!

No messages table! The data model behind my own Claude-based chatbot

MongoDB Guests — Thu, 02 Jul 2026 15:47:31 +0000

This tutorial was written by Néstor Daza.

This is the second article in a series about building Claudius, my own Claude-based chatbot (Github). The prologue made the case for building it, and for choosing MongoDB as its foundation.

Open the conversations collection in Claudius’ database and you find the usual fields of a thread header but nothing else: a userId, a title, some timestamps, and so on, but no array of messages, no messages collection sitting beside it either! The text of every conversation lives somewhere else entirely, in the LangGraph checkpointer, which I wire up later in this series. This absence is a modeling decision, and how I came up with the database schema for my chatbot is the theme of this article.

If you come from a relational background, you're used to modeling the data first when designing a database. For a project like this, you would start by finding the entities and normalizing them, and the final schema would come out of the data's structure: a conversations table and a messages table with a foreign key between them, because that is what the data looks like.

Document modeling runs the other way. You start from how the application reads and writes, and the shape of the document follows the access patterns. Claudius never reads conversation messages without the agent's full working state wrapped around them, and that state is persisted using the LangGraph checkpointer. A separate messages table would add nothing, since the app would always have to join it back to that state on every read. The access pattern says the messages belong with the agent state, so that is where they go, and conversations are left as the lightweight header the list view actually needs.

That inversion, modeling around use rather than around the data, runs through everything below.

Schema-flexible is not schemaless

This is the misconception lots of people often carry, and it is worth killing on the way in. A document database does not mean no schema; it means a flexible, use-case-based data model. A relational engine holds the schema for you through column types and constraints; MongoDB will store documents in whatever shape your application gives it, and can validate that shape with a collection validator if you want. Claudius keeps the schema in code, as one Zod schema per collection, because that single definition gives both the runtime check and the TypeScript type.

import { z } from "zod";
import { zObjectId } from "./common";

export const ConversationSchema = z.object({
  _id: zObjectId.optional(),
  userId: zObjectId,
  title: z.string(),
  modelId: z.string(),
  createdAt: z.date(),
  updatedAt: z.date(),
  archived: z.boolean(),
  expiresAt: z.date().optional(),
});

export type Conversation = z.infer<typeof ConversationSchema>;

The reason is drift. Declare a TypeScript interface for a document and a separate validator for the data coming in, and the two will disagree eventually, while the type system will lie to you with complete confidence. Deriving the type from the validator collapses them into one artifact. One definition gives you the runtime check at the boundary and the compile-time type everywhere else. For a relational reader, this is your DDL (Data Definition Language) and your ORM (Object-Relational Mapping) models together in the same structure, which is something relational stacks rarely pull off.

Another advantage is that every external input, whether an incoming HTTP body or a model's structured output, is parsed through Zod at the boundary, so by the time data reaches the logic, it is already validated and typed.

A quick tour of the model

There are eight collections in this phase: users, conversations, memories, documents, chunks, usage_events, settings, and, arriving later, jobs. Two more, checkpoints and checkpoint_writes, belong entirely to the checkpointer and are never written to directly, so I do not model them. Most of these schemas are unremarkable once you see the access-pattern framing, but two modeling moves are worth pausing on.

The first is settings. It holds a handful of global singletons, each a different shape, each identified by a string _id: the member allowlist, the model catalog, the per-role tiers, and the guest spend circuit breaker. They share one collection, so the Zod schema is a discriminated union based on _id, and parsing a settings document narrows it to the correct shape from its _id literal.

export const SettingsSchema = z.discriminatedUnion("_id", [
  AllowlistSettingsSchema,        // _id: "allowlist"
  ModelCatalogSettingsSchema,     // _id: "modelCatalog"
  TiersSettingsSchema,            // _id: "tiers"
  GuestCircuitBreakerSettingsSchema, // _id: "guestCircuitBreaker"
]);
export type Settings = z.infer<typeof SettingsSchema>;

In relational terms, this is a single typed configuration table whose rows legitimately carry different columns, which is usually awkward to model relationally without resorting to a JSON column, and here comes out clean and readable.

The second move is to use document embedding as a data modeling technique. The users document carries its dailyMessageCount as an embedded object holding the count and its reset time together, rather than as two flat fields or a side table, because the application always reads and writes them together as a unit. That is the document modeling mantra: data used together is stored together.

Let the database keep the rule

The most satisfying part of this phase is how little policy I had to write. Three product rules that I would normally enforce in application code are carried by the storage engine instead, because I shaped the data so the engine could hold them.

Retention is a TTL index

Guest data in Claudius is ephemeral. Guest-created conversations and memories carry an expiresAt date; member and admin documents leave the field off entirely. A TTL (time-to-live) index with expireAfterSeconds set to 0 tells MongoDB to delete each document at the exact moment stored in its own expiresAt field; documents without this field are never touched.

// TTL on the guest-only expiresAt field. expireAfterSeconds:0 
// means "expire exactly at the date stored in the field"; 
// documents without the field are never touched, so member/admin
// data is permanent.

await db
  .collection(COLLECTIONS.conversations)
  .createIndex({ expiresAt: 1 }, { name: "expiresAt_ttl", expireAfterSeconds: 0 });

await db
  .collection(COLLECTIONS.memories)
  .createIndex({ expiresAt: 1 }, { name: "expiresAt_ttl", expireAfterSeconds: 0 });

So, guest ephemerality, a product rule, is enforced by the engine. There is no cron job and no cleanup endpoint to remember to run. The relational equivalent is a scheduled sweep; here, the policy lives in the index.

One detail ties the TypeScript compiler to the database behavior. With exactOptionalPropertyTypes turned on in tsconfig.json, an optional field means the key is absent, not present, and set to undefined. The compiler setting and the storage rule reinforce each other.

Telemetry is a time-series collection

Every billable model call writes one usage_events document, recording the token counts, the latency, and what the call was for. That data is append-only and gets queried along a few low-cardinality dimensions over a time window, which is the textbook shape for a MongoDB time-series collection. It has a timeField, timestamp, and a single metaField that holds the grouping dimensions (userId, modelId, purpose) nested alongside the measurements.

Two things follow from this choice. A time-series collection has to be created explicitly with a timeseries spec before the first insert (they cannot be created on first write, the way an ordinary collection does), so the index script creates it behind an existence check. Once it exists, the engine's own machinery becomes visible: list the collections and a system.buckets.usage_events sits next to usage_events, which is where the bucketed storage lives:

[
  {"name":"memories","type":"collection"},
  {"name":"usage_events","type":"timeseries"},
  {"name":"system.buckets.usage_events","type":"collection"},
  {"name":"settings","type":"collection"},
  {"name":"users","type":"collection"},
  {"name":"conversations","type":"collection"},
  {"name":"accounts","type":"collection"},
  {"name":"chunks","type":"collection"},
  {"name":"system.views","type":"collection"}
]

The relational analogy is a metrics or rollup table you would otherwise hand-roll and maintain, except the bucketing is automatic and the query you write against it stays simple.

A pre-filter field is a security control

The project's hardest rule is that a vector search pre-filters by userId and can never return another user's data. That rule is not bolted onto the search in the application code. It is a property of the index.

Both memories and chunks get an Atlas Vector Search index on the embedding field, with a filter on userId (and on chunks, a second filter on documentId).

await ensureVectorIndex(db, COLLECTIONS.memories, "memories_vector", [
  { type: "vector", path: "embedding", numDimensions: 1024, similarity: "cosine" },
  { type: "filter", path: "userId" },
], created, skipped);

await ensureVectorIndex(db, COLLECTIONS.chunks, "chunks_vector", [
  { type: "vector", path: "embedding", numDimensions: 1024, similarity: "cosine" },
  { type: "filter", path: "userId" },
  { type: "filter", path: "documentId" },
], created, skipped);

Pre-filtering means the userId condition is applied during the indexed search on the same documents being scored, not afterward to whatever the search returns. Post-filtering is not only slower, since you score documents you then throw away, but it is a correctness and security risk, since the wrong documents can slip through ranking and limits before your filter ever runs.

This is also why chunks denormalizes userId. A vector search has no notion of a join, so the filter has to sit on the very documents being searched. A chunk belongs to a document that belongs to a user, but the chunk has to carry the owner itself, or the index cannot enforce the boundary. That is the embed-versus-reference tension made concrete: a chunk references its parent document by id and duplicates the one field the security boundary depends on.

Two gotchas surfaced while creating these indexes, and both are worth saying out loud. A search index can only be built on a collection that already exists, and chunks had no classic index, so the collection did not exist yet the first time the script reached for its vector index:

MongoServerError: Error retrieving collection UUID and view info ::
caused by :: Collection 'claudius.chunks' does not exist. (NamespaceNotFound)

The fix is to add an existence check and create the collection first only if it is missing. The second gotcha is that Atlas builds search indexes asynchronously, so createSearchIndex returns when the build is queued, not when it is queryable! We also need a name check that keeps reruns safe while it builds.

This is the final function to build the indexes from code using the driver's createSearchIndex, with the checks that keep the script safe to run on every deploy:

async function ensureVectorIndex(
  db: Db,
  collectionName: string,
  indexName: string,
  fields: Array<Record<string, unknown>>,
  created: string[],
  skipped: string[],
): Promise<void> {
  // A vector search index can only be built on an existing
  // collection, so create it first if nothing else has (e.g.
  // chunks has no regular index).
  const collections = await db
    .listCollections({ name: collectionName })
    .toArray();
  if (collections.length === 0) {
    await db.createCollection(collectionName);
  }

  const collection = db.collection(collectionName);
  const present = await collection.listSearchIndexes().toArray();
  if (present.some((idx) => idx.name === indexName)) {
    skipped.push(`${collectionName}.${indexName}`);
    return;
  }

  // Atlas builds the index asynchronously; createSearchIndex
  // returns once the build is queued. The name check above 
  // keeps re-runs idempotent.
  await collection.createSearchIndex({
    name: indexName,
    type: "vectorSearch",
    definition: { fields },
  });
  created.push(`${collectionName}.${indexName} (vector)`);
}

The plumbing that keeps the code honest

Two small things hold the userId rule together in practice. First, userId is an ObjectId everywhere, matching the user’s _id, so filters and lookups line up without conversion. Second, no collection is ever queried through a raw driver call. Every access goes through a typed accessor, conversationsCol(), memoriesCol(), and so on, each returning a typed collection for the right document shape. Centralizing the collection names means a rename happens in one place, and typing each accessor keeps the userId filter visible to the compiler at every call site, which is how a rule that lives in prose stays honest in code.

The whole application and the Auth.js adapter share a single, serverless-safe MongoDB client to a single fixed database name, so there is exactly one connection pool. How that database name quietly caused two separate failures is a good story, and it belongs to the next article, where I stand the foundation up.

What this article bought

No chatbot yet, but the shape of the thing is already decided. Retention, the telemetry layout, and the tenant-isolation boundary are properties of the schema and its indexes now, not code I have to remember to run later. That is the payoff of modeling around the access patterns first: the database carries more of the load because you shaped it to. The data model is the foundational piece of the whole app.

Next, we'll build this foundation up, with identity the client never gets a vote in, and a health check that proves the app can reach Atlas and Bedrock, plus a view on the unglamorous errors that cost me real time. After that comes the streaming chat backbone, which is where the conversation finally gets a place to live, and I open a real checkpoint document to show you the messages that were never in their own collection.

Advanced API Authentication Strategies with Node.js and MongoDB (Part 3)

MongoDB Guests — Thu, 02 Jul 2026 09:00:00 +0000

This article was written by Moses Anumadu.

Secure Session Management with JWT and Refresh Tokens in Node.js and MongoDB

This is the final part of this 3-part series. In parts 1 and 2, we implemented email-based authentication. Then we went further to implement TOTP-based multi-factor authentication (MFA), QR code onboarding, and half-authentication login flow.

What we have built so far is an API that allows users to prove their identity using email and password. In this section, we will go beyond that. The application should not only be able to recognize users but also protect the user through concepts like secure session management, token expiration, and session revocation. We will implement these concepts through the following:

JWT access tokens
refresh tokens
MongoDB-backed sessions
refresh token rotation
protected routes
secure logout flows

By the end of this section, we should have an authentication system that can be deployed to production and improved upon for more advanced features.

Pre-requisites

To follow along with this tutorial, ensure you have prior experience working with the following technologies:

Node.js and Express
MongoDB and Mongoose
A MongoDB Atlas account
Postman
Ngrok
Parts one and two of this series

With that said, let's get started.

Understanding Access Tokens vs Refresh Tokens

An access token is a JWT token that grants a user access to a protected route. It often has a very short-lived life cycle. A refresh token is a token with a longer expiration time solely for requesting an access token. Without the refresh token, the user would need to log out and log back in to generate a new access token. The refresh token is stored in the database, enabling the system to recognize the user without terminating the session.

You are probably wondering why you need both, right? Let's look at an example. Imagine that a hacker steals a user's access token, and with this token, he (the hacker) can do malicious things. If this access token does not have a short-lived life cycle, the attacker would have a longer window to do more harm. To prevent this, the refresh token is required to generate a new access token from time to time. So, even if an access token is stolen, within 15 minutes, that token becomes invalid. The attacker can't do any harm.

This is an extra layer of security necessary for a secure authentication system and to improve the user experience. The flow looks like: a user logs in, access token and refresh token are granted, access token lasts for 15 minutes, refresh token lasts for 7 days, access token expires after 15 minutes, user does not need to log out, user generates a new access token using the refresh token saved in the database.

Creating Token Utilities

Now, we understand why we need both an access token and a refresh token. Let's proceed to the implementation. In part 2, when we created the MFA token, we created src/utils/tokenUtils.js file. We need to update that file to generate our access token, refresh token, and MFA token. Open src/utils/tokenUtils.js and replace the code with the code below:

const jwt = require("jsonwebtoken");

const crypto = require("crypto");

const generateAccessToken = (user) => {
  return jwt.sign(
    {
      userId: user._id,
      email: user.email,
    },
    process.env.JWT_ACCESS_SECRET,
    {
      expiresIn: process.env.ACCESS_TOKEN_EXPIRES_IN,
    }
  );
};

const generateRefreshToken = () => {
  return crypto.randomBytes(64).toString("hex");
};

const hashToken = (token) => {
  return crypto
    .createHash("sha256")
    .update(token)
    .digest("hex");
};

const generateMFAToken = (user) => {
  return jwt.sign(
    {
      userId: user._id,
      type: "mfa",
    },
    process.env.JWT_ACCESS_SECRET,
    {
      expiresIn: process.env.MFA_TOKEN_EXPIRES_IN,
    }
  );
};

module.exports = {
  generateAccessToken,
  generateRefreshToken,
  hashToken,
  generateMFAToken,
};

We added three methods generateAccessToken, generateRefreshToken, hashToken to the page. The hashToken will be used later. It will be used to encrypt our refresh token into a one-way SHA-256 hash before it is stored in the database or compared when validating. The generateAccessToken and generateRefreshToken generate an access token and a refresh token, respectively.

Why Refresh Tokens Are Not JWTs

Also, did you notice that our refresh tokens were generated using secure random bytes? This line of code here: crypto.randomBytes(64).toString("hex");. This simply gives us more control over it. The refresh token should be random, unpredictable, and most importantly, database-controlled. Refresh tokens, unlike access tokens, do not need readable claims.

So, this way, we can easily revoke it, rotate it, and control the session validation or invalidation.

Creating the Session Model

Our refresh token is database-controlled. That ultimately means we need to store and retrieve it from the database. To achieve this, we need to create a MongoDB model for storing user sessions. Create src/models/Session.js and update the page with the code below:

const mongoose = require("mongoose");

const sessionSchema = new mongoose.Schema(
  {
    userId: {
      type: mongoose.Schema.Types.ObjectId,
      ref: "User",
      required: true,
    },

    refreshTokenHash: {
      type: String,
      required: true,
    },

    userAgent: {
      type: String,
      default: null,
    },

    ipAddress: {
      type: String,
      default: null,
    },

    expiresAt: {
      type: Date,
      required: true,
    },

    revokedAt: {
      type: Date,
      default: null,
    },
  },
  {
    timestamps: true,
  }
);

module.exports = mongoose.model("Session", sessionSchema);

Notice that we are storing refreshTokenHash. Always remember that sensitive data at rest should also be encrypted. We created the fields we need to save for each user session. Notice fields like ipAddress, userAgent, expiresAt and revokedAt.

Issuing Tokens After MFA Verification

Now we can generate both a refresh token and an access token, and our MongoDB model is ready to save session values in the database. Next, we need to modify the authentication flow to include these. The flow should be: a user logs in, completes MFA, the application issues an access token and a refresh token to the user, and finally, the session values (including the refresh token) are stored in our MongoDB session collection. Let's proceed to the implementation.

!!!
Returning refresh tokens in JSON bodies is unsafe for browser clients and encourages XSS-exposed storage patterns. The safer pattern is HTTP-only cookie transport for refresh credentials. This is the approach we will follow in the next section.
!!!

Updating the Login Controller

In the next section, we will send refresh tokens using HTTP-only cookies transport as this is the safer option to prevent XSS-expose attacks.
The code in src/controllers/authController.js needs to be updated. Let's start by adding the import values and HTTP-only cookies configurations.

Open src/controllers/authController.js and add the following import statement to the top part of the page

const jwt = require("jsonwebtoken");
const Session = require("../models/Session");

const {
  generateAccessToken,
  generateRefreshToken,
  hashToken,
  generateMFAToken,
} = require("../utils/tokenUtils");

After the import statements, add the following HTTP-only cookie configurations. We will use these later in the code.

const getRefreshTokenExpiryMs = () => {
  const refreshDays = Number(
  process.env.REFRESH_TOKEN_EXPIRES_DAYS
  );

  return refreshDays * 24 * 60 * 60 * 1000;
};

const getRefreshTokenCookieOptions = () => {
  return {
    httpOnly: true,
    secure: process.env.NODE_ENV === "production",
    sameSite: "strict",
    path: "/api/auth",
    maxAge: getRefreshTokenExpiryMs(),
  };
};

const setRefreshTokenCookie = (res, refreshToken) => {
  res.cookie(
  "refreshToken",
  refreshToken,
  getRefreshTokenCookieOptions()
  );
};

const clearRefreshTokenCookie = (res) => {
  res.clearCookie("refreshToken", {
    ...getRefreshTokenCookieOptions(),
    maxAge: undefined,
  });
};

Updating the MFA Login Verification Flow

Next, replace the success section inside verifyMFALogin controller method with the code below:

Revoke this section

return res.status(200).json({
success: true,
message: "MFA login successful",
});

And replace with this:

// Issue access token
const accessToken = generateAccessToken(user);

// Generate refresh token
const refreshToken =
  generateRefreshToken();

// Hash refresh token
const refreshTokenHash =
  hashToken(refreshToken);

// Session expiry
const expiresAt = new Date();

expiresAt.setDate(
  expiresAt.getDate() +
    Number(
      process.env.REFRESH_TOKEN_EXPIRES_DAYS
    )
);

// Store session
await Session.create({
  userId: user._id,
  refreshTokenHash,
  userAgent: req.headers["user-agent"],
  ipAddress: req.ip,
  expiresAt,
});
setRefreshTokenCookie(res, refreshToken);
      return res.status(200).json({
       success: true,
       message: "Login successful",
       accessToken,
     });

From the additions we made, after password verification and MFA verification, a token is issued, and a session is created for the authenticated user.

Creating JWT Authentication Middleware

Now, we can issue an access token and a refresh token. Next, we need a way to protect our routes so that only users with the right access can access them. To do this, we need to implement middlewares.

A good example can be creating a middleware that protects routes and lets only authenticated users have access to it. Let's implement that. Create the following file src/middleware/authMiddleware.js, and update it with the code below:

const jwt = require("jsonwebtoken");

const User = require("../models/User");

const requireAuth = async (
  req,
  res,
  next
) => {
  try {
    const authHeader =
      req.headers.authorization;

    if (
      !authHeader ||
      !authHeader.startsWith("Bearer ")
    ) {
      return res.status(401).json({
        success: false,
        message: "Access token required",
      });
    }

    const token = authHeader.split(" ")[1];

    const decoded = jwt.verify(
      token,
      process.env.JWT_ACCESS_SECRET
    );

    const user = await User.findById(
      decoded.userId
    );

    if (!user) {
      return res.status(401).json({
        success: false,
        message: "Invalid user",
      });
    }

    req.user = user;

    next();
  } catch (error) {
    return res.status(401).json({
      success: false,
      message:
        "Invalid or expired access token",
    });
  }
};

module.exports = requireAuth;

Creating a Protected Route

At this point, the middleware is ready. We need to attach it to our routes to restrict access to only authenticated users. Let's create a controller to get the authenticated user getMe and a route for it /api/auth/me then protect it with the middleware we created above. To create the controller method, open src/controllers/authController.js and add the following controller method to the page.

const getMe = async (req, res) => {
  return res.status(200).json({
    success: true,
    user: {
      id: req.user._id,
      email: req.user.email,
      mfaEnabled: req.user.mfaEnabled,
    },
  });
};

Also, update the export block to contain the getMe method we just added, like so

module.exports = {
  register,
  login,
  setupMFA,
  verifyMFA,
  verifyMFALogin,
  getMe,
};

Adding the Protected Route

Next, let's create the route. Open src/routes/authRoutes.js and add the following to the page.

First, import the middleware by copying and pasting the code below to the top section of the page

const { getMe } = require("../controllers/authController");
const requireAuth = require("../middleware/authMiddleware");

Then add the route below to the route on the page.

router.get("/me", requireAuth, getMe);

Testing Protected Routes

We've made progress. Let's test what we've built so far. We will test with Postman on the web and Ngrok. Ngrok will be used to tunnel our localhost address to something on the internet. Check Part One of this article for more on the setup. With that said,

Login

To generate an access token, we need to go through a series of verifications. First, we need to log in and generate an MFA token to verify MFA. Send a post request to the route below to log in.

POST /api/auth/login

Add the following to the body

{ 
  "email": "test_user@example.com",
  "password": "password123"
}

Your response should look similar to the image below if everything is done right.

MFA verification and access key generation

We are partially logged in. We need to verify MFA to be fully logged in. Send a Post request to the route below:

POST /api/auth/mfa/login/verify

Also, add the following to the body of the request:

{ "token": "TOKEN_GENERATED_BY_AUTH_APP",
  "mfaToken": "MFA_TOKEN_GENERATED_FROM_LOGIN"
}

Protected route

Next, send a GET request to the route below

GET /api/auth/me

With the following Header values

Authorization: Bearer YOUR_ACCESS_TOKEN

If the token is valid, the request should return the authenticated user information, like the image below

Also, in MongoDB Atlas, you can see that user sessions were created and stored in our session collection.

Implementing Refresh Token Rotation

Let's take our authentication a step further by implementing refresh token rotations. What we want to achieve is simple. Every refresh request should generate a new access token, generate a new refresh token, and invalidate the old refresh tokens. Let's jump right into it.

Creating the Refresh Endpoint

Add the following controller method to src/controllers/authController.js.


const refreshAccessToken = async (req, res) => {
   try {
     const refreshToken = req.cookies?.refreshToken;
      if (!refreshToken) {
       return res.status(401).json({
         success: false,
         message: "Refresh token cookie required",
       });
     }
      const refreshTokenHash = hashToken(refreshToken);
      const session = await Session.findOne({
       refreshTokenHash,
       revokedAt: null,
     });
      if (!session) {
       clearRefreshTokenCookie(res);
       return res.status(401).json({
         success: false,
         message: "Invalid session",
       });
     }
      // Expired session
     if (session.expiresAt < new Date()) {
       clearRefreshTokenCookie(res);
       return res.status(401).json({
         success: false,
         message: "Session expired",
       });
     }
      const user = await User.findById(session.userId);
      if (!user) {
       return res.status(401).json({
         success: false,
         message: "Invalid user",
       });
     }
      // ROTATE REFRESH TOKEN
     const newRefreshToken = generateRefreshToken();
      const newRefreshTokenHash = hashToken(newRefreshToken);
      session.refreshTokenHash = newRefreshTokenHash;
      await session.save();
     setRefreshTokenCookie(res, newRefreshToken);
      // Generate new access token
     const accessToken = generateAccessToken(user);
      return res.status(200).json({
       success: true,
       accessToken,
     });
   } catch (error) {
     console.error(error);
      return res.status(500).json({
       success: false,
       message: "Failed to refresh token",
     });
   }
 };

Next, let's update the exports with the freshAccesstoken method like so:

module.exports = {
  register,
  login,
  setupMFA,
  verifyMFA,
  verifyMFALogin,
  getMe,
  refreshAccessToken,
};

Adding the Refresh Route and Testing

Let's proceed by adding a route to test the refresh token feature. Open src/routes/authRoutes.js and update the page with the code below

router.post(
  "/refresh",
  refreshAccessToken
);

Let's proceed to test. Send a POST request to the route below.

POST /api/auth/refresh

Request body:

{
  "refreshToken": "YOUR_REFRESH_TOKEN"
}

If everything was done right, your response should look like the screenshot above. And the previous tokens should no longer work.

Implementing Logout and Session Revocation

A complete logout system not only removes the access token from the browser in the front end. It should also invalidate the refresh token and revoke the session from the backend. That is exactly what we will do in this section.

Creating the Logout Endpoint

Add the following controller method to src/controllers/authController.js

const logout = async (req, res) => {
   try {
     const refreshToken = req.cookies?.refreshToken;
      if (!refreshToken) {
       return res.status(400).json({
         success: false,
         message: "Refresh token cookie required",
       });
     }
      const refreshTokenHash = hashToken(refreshToken);
      await Session.findOneAndUpdate(
       {
         refreshTokenHash,
         revokedAt: null,
       },
       {
         revokedAt: new Date(),
       }
     );
     clearRefreshTokenCookie(res);
      return res.status(200).json({
       success: true,
       message: "Logged out successfully",
     });
   } catch (error) {
     console.error(error);
      return res.status(500).json({
       success: false,
       message: "Logout failed",
     });
   }
 };

From the code, we did not delete the session records. We marked them as revoked instead, as we can see below.

await Session.findOneAndUpdate(
{
refreshTokenHash,
revokedAt: null,
},
{
revokedAt: new Date(),
}
);

This is a good practice as it helps preserve session history for audit and security visibility.

Implementing Logout From All Devices

Let's take it a step further by allowing users to revoke all active sessions simultaneously. This would let them log out from all devices at once. This is a nice feature to have; it helps a user protect their account in the case of a malicious login from an unknown device.

Let's proceed, modify src/controllers/authController.js file, and add the following controller method:

const logoutAll = async (req, res) => {
   try {
     await Session.updateMany(
       {
         userId: req.user._id,
         revokedAt: null,
       },
       {
         revokedAt: new Date(),
       }
     );
     clearRefreshTokenCookie(res);
      return res.status(200).json({
       success: true,
       message: "Logged out from all devices",
     });
   } catch (error) {
     console.error(error);
      return res.status(500).json({
       success: false,
       message: "Failed to logout all sessions",
     });
   }
 };

Update the export block to contain all the methods below

module.exports = {
  register,
  login,
  setupMFA,
  verifyMFA,
  verifyMFALogin,
  getMe,
  refreshAccessToken,
  logout,
  logoutAll,
};

Adding Logout Routes

To add the routes for logout, update src/routes/authRoutes.js and add the code below. Also ensure that you have imported both logout and logoutAll methods from authController.js.

router.post("/logout", logout);

router.post(
  "/logout-all",
  requireAuth,
  logoutAll
);

Testing Logout

Send a post request to the route below. Also, pass the body value in the body of your request.

Request:

POST /api/auth/logout

Request body:

{
  "refreshToken": "YOUR_REFRESH_TOKEN"
}

Your response should look similar to the screenshot below

You can also test the logout all feature the same way. Send a Post request to the route below with a valid access token.

Request:

POST /api/auth/logout-all

Also, add a header with a user's access token.

Authorization: Bearer YOUR_ACCESS_TOKEN

Expected response:

{
  "success": true,
  "message":
    "Logged out from all devices"
}

Did you notice that the logout-all requires an access token passed in as a request header, not a refresh token like the logout route? This is because the logout-all does not have an active user session or know the user sending the request. It needs a way to identify the user, so it requires a user's access token.

All refresh token sessions should now become invalid. To be sure, you can also try using the same refresh token. It should fail because this session has been revoked.

Conclusion

A big thumbs up to you for sticking around till the end of this three-part series. We built a complete authentication API using Node.js, Express, MongoDB Atlas, and tested using Postman on the web and Ngrok for localhost routing.

In this final part of this series (part three), we discussed the difference between JWT access tokens and refresh tokens, why we need both, and walked through the programmatic implementation of the authentication by implementing the following:

JWT access token authentication
Session storage
refresh token rotation
Middleware and protected routes
secure logout and session revocation

The complete flow of the authentication with all three (3) parts put together looks like:

User registration, password authentication, MFA setup and verification, JWT access tokens and refresh token issuance, refresh token sessions, middlewares and protected routes, session.

Security is more of layers than single features. For further reading, check out the following topics for additional layers that can be added to an authentication.

Rate limiting
Email verification
HTTPS enforcement
Secure HTTP-only cookies
Brute-force protection
Device monitoring
Anomaly detection

Awesome. You can find the complete code on GitHub.