DEV Community: Convex Champions

Scaling to 70 Modules: Building a Web, Mobile, and API backend on one Convex deployment

Hamza Saleem — Wed, 21 Jan 2026 06:31:53 +0000

Production Convex: What 70 Modules Looks Like

What I learned building one Convex backend for web, mobile, and API

When I started building Client Commander, I had one Next.js app and maybe ten tables. A year later, it's serving a web dashboard, a mobile app, a background sync service, and a full REST API. 50+ tables. 70+ modules. Same deployment.

I didn't plan for any of that. I just kept building, and the architecture held up. That surprised me.

Here's how it happened.

What We Built

Client Commander is a multi-tenant CRM. Companies sign up, add their team, manage contacts and deals. Nothing revolutionary about the domain — but the technical requirements add up fast: permissions, real-time sync, mobile, API access, background jobs.

Quick terminology so the rest makes sense:

Agents: The users (employees) using the system.
Contacts: The people/customers being tracked.
Companies: The tenants (customers of the SaaS).

Permissions That Update Themselves

Change someone's role, and their dashboard updates in real-time. They see different data instantly — no refresh, no logout, no waiting.

I didn't build that. I just wrote a normal permission check at the top of my queries, and it worked.

Took me a while to understand why. Convex queries are subscriptions, not one-off requests. They keep running — every time the underlying data changes, the query re-evaluates. So my permission check runs again when contacts change. It runs again when the user's role changes. Same code, always current.

const { agentId, companyId, role } = await getAuthContext(ctx);

if (!hasPermission(role, "contacts.view")) {
  throw new Error("Forbidden");
}

return await ctx.db.query("contacts")
  .withIndex("by_company", q => q.eq("companyId", companyId))
  .collect();

Adding new permission levels was just config after that. Owner sees everything. Team Leader sees their team. Agent sees only their own contacts. New roles? Add them to the config. Queries stay the same.

The part that impressed me: Convex tracks which data each query actually touches. My query reads the user's role, so the platform knows that query depends on that role record. Change the role, and only queries that care about it re-run — not everything. I didn't set up subscriptions or configure channels. It figured that out from the code.

With most backends, you'd need WebSocket infrastructure, cache invalidation, push logic — a whole thing. Here it's just... the default.

Adding Mobile Without Adding Backend

We shipped a mobile app. Zero new backend code.

But honestly, the time savings wasn't the main thing. The main thing was this: we stopped discovering "mobile is out of sync" from user bug reports. We started discovering it before the code compiles.

The Expo app imports the exact same API as the Next.js app. Same queries, same mutations, same types. Fix a bug in a query? Fixed on both. Change the schema? Both apps break until they handle it — during development, not after users complain.

// Same import, whether you're in Next.js or Expo
import { api } from "@workspace/backend/convex/_generated/api";

const contacts = useQuery(api.contacts.list);
const createContact = useMutation(api.contacts.create);

Convex generates typed APIs from your schema. That generated code is the contract. Write a query, and it creates types for the arguments and return value. Call it from React — web or mobile, doesn't matter — you get autocomplete and type checking.

The monorepo makes it work. Backend is a shared package. Both apps depend on it. Schema is the single source of truth.

Add a field? Both apps get type errors. Rename a query? TypeScript shows you every callsite that needs updating — across platforms, in one compile. I've caught so many things this way that would've been production bugs otherwise.

Most cross-platform setups have duplicated types, separate clients, manual syncing. Here, the generated API handles sync. Change the source, types change everywhere. No process to remember. It just breaks if you forget.

Real-Time for Users, REST for Everything Else

Real-time is great for dashboards. But external integrations don't speak WebSocket. They want REST. Webhooks need HTTP endpoints.

So we added a REST API — 40+ endpoints. No separate service. The HTTP layer just authenticates, rate limits, then calls the same functions that power the UI.

http.route({
  path: "/v1/contacts",
  method: "GET",
  handler: httpAction(async (ctx, request) => {
    const auth = await verifyApiKey(request);
    if (!auth.ok) return auth.response;

    const contacts = await ctx.runQuery(internal.contacts.list, {
      companyId: auth.companyId,
    });

    return new Response(JSON.stringify(contacts), {
      status: 200,
      headers: { "Content-Type": "application/json" },
    });
  }),
});

The nice part: ctx.runQuery inside an HTTP action runs the exact same code as the real-time subscriptions. I'm not maintaining two implementations — the REST endpoint is just a thin wrapper around stuff that already exists.

Webhooks work the same way. Payment provider sends a POST, I verify the signature, call a mutation. Same mutation the UI calls. One code path.

No separate API server. No connection pooling headaches. HTTP routes deploy with everything else. We went from "we need an API" to live endpoints in a day.

Workflows That Outlive Deployments

We have workflows that wait three days before executing the next step. Some wait a week.

Here's the thing: I don't lose them when I deploy. I don't wake up to half-finished workflows. I don't build state machines to track what step we're on.

They just continue. Server restarts, new deployment happens, doesn't matter. The delay finishes, the next step runs, picks up exactly where it left off.

If you've ever used scheduler.runAfter, you know it works for one-off delayed functions. But when you need a chain — wait, then do X, then wait again, then do Y — suddenly you're managing state. What if step 2 fails? How do you know step 1 finished? How do you retry?

The workflow component handles that. Each step gets recorded. If something restarts mid-execution, it replays from where it stopped — skipping steps that already ran.

const myWorkflow = workflow.define({
  args: { userId: v.id("users") },
  handler: async (step, { userId }) => {
    await step.runMutation(internal.users.markOnboardingStarted, { userId });

    await step.runMutation(internal.emails.sendFollowUp, 
      { userId }, 
      { runAfter: 3 * 24 * 60 * 60 * 1000 }  // 3 days
    );

    await step.runMutation(internal.users.checkEngagement, { userId });
  },
});

Each step is checkpointed. Server restarts after step 2 is scheduled? Fine. It recovers and picks up where it was.

I build automations that span weeks now without worrying about them. Onboarding sequences, trial expirations — they just run. No job queue to maintain. No polling for stuck jobs. No "what state is this in?" debugging at 2am.

Search Without the Infrastructure

We needed search. Full-text, on contacts. My first thought was "okay, time to figure out Elasticsearch."

Nope. Three lines in the schema.

contacts: defineTable({
  companyId: v.id("companies"),
  fullName: v.string(),
})
  .searchIndex("search_name", { 
    searchField: "fullName",
    filterFields: ["companyId"] 
  })

That's it. Search works. Filters by company. Ranks by relevance. Deploy, it's live.

Same pattern kept repeating. Need to find contacts by phone number? Add an index. Look up deals by stage? Index. Sort by next task due date? Store it denormalized and index it.

contactPhones: defineTable({
  contactId: v.id("contacts"),
  value: v.string(),
})
  .index("by_value", ["value"])

contacts: defineTable({
  // ...
  nextTaskDueAt: v.optional(v.number()),
})
  .index("by_company_nextTask", ["companyId", "nextTaskDueAt"])

Normally you'd be deciding: which search service, how to sync data, how to handle the lag between your database and search index. Here, search indexes update transactionally with your data. No sync. No eventual consistency weirdness.

The trade-off: I denormalize more than I would elsewhere. fullName gets computed and stored. nextTaskDueAt gets copied from tasks to contacts. Writes get a bit messier. But queries stay fast, and I don't manage infrastructure.

50+ tables, dozens of indexes. Every single one was a schema change, not a project.

That's what 70 modules looks like. One deployment. Web, mobile, REST, all hitting the same backend. Permissions that update in real-time. Types that catch drift before production. Workflows that survive restarts. Search that's three lines.

I didn't do anything clever to make this work. I just kept building, and the platform didn't get in the way.

If you're thinking about using Convex for something real — this is what happens when you do.

Hamza Saleem is the founder of Client Commander and a Convex Champion. Previously: Keeping Users in Sync with Convex.

Transactional Emails in Convex with Bluefox and AWS SES

Jamal — Wed, 23 Apr 2025 13:13:05 +0000

Introduction
Setting Up Bluefox with Convex
1. Create a Bluefox Account and Project
2. Configure Your Environment
3. Install and Run Convex Locally
Integrating Bluefox with Convex Actions
1. Setting Up the Bluefox Client
2. Sending Transactional Emails
3. Sending Triggered Emails
4. Error Handling Best Practices
5. Where to Trigger Emails (Mutations → Actions)
Advanced: Handling Webhooks
1. Features of Bluefox Webhooks
2. Setting Up Bluefox Webhooks
3. Verifying Webhook Requests
Production Deployment Best Practices for Bluefox
Troubleshooting Common Issues
Conclusion

Introduction

Bluefox is a modern email API built on top of Amazon SES, designed for developers and SaaS teams. Whether you’re sending transactional emails or integrating real-time webhooks in a serverless Convex backend, Bluefox simplifies the process. By wrapping a usage-based, developer-friendly API around Amazon SES, you get excellent deliverability and scalability, without the complexity of managing SES directly.

For Convexdevelopers, Bluefox integrates seamlessly with Convex actions, which make external calls to the Internet. This guide will walk you through setting up Bluefox with Convex, integrating it into your codebase, handling Bluefox webhooks, and outlining key considerations for deploying to production.

Setting Up Bluefox with Convex

1. Create a Bluefox Account and Project

Sign Up: Visit bluefox.email and create a new account.
Add a Project: Once inside your dashboard, create a new Project.
Connect AWS SES: Add a sending email address and link your AWS SES credentials (an IAM user with ses:SendEmail, ses:SendRawEmail, ses:GetSendQuota, ses:ListIdentities permissions).
Automatic Management: Bluefox handles sending queues, email rendering, and analytics for you.

2. Configure Your Environment

Obtain your API key from the Bluefox project settings. Then add BLUEFOX_API_KEY to your environment variables (via the Convex dashboard).

3. Install and Run Convex Locally

npm install convex
npx convex dev

This installs Convex locally, ensuring any changes sync with the Convex cloud.
(Note: Check npmjs.com for the latest bluefox-email version.)

Integrating Bluefox with Convex Actions

Convex organizes backend logic into queries, mutations, and actions. Only actions can perform external HTTP requests, making them ideal for sending emails via Bluefox.

Setting Up the Bluefox Client

Install the bluefox-email TypeScript library, which provides a typed interface, robust error handling, and webhook utilities:

npm install bluefox-email

Then create a reusable client (e.g., convex/bluefox.ts):

// convex/bluefox.ts
import { BluefoxClient } from "bluefox-email";

// Initialize the Bluefox client with your API key
export const bluefox = new BluefoxClient({
  apiKey: process.env.BLUEFOX_API_KEY!, // Ensure set in Convex dashboard
  debug: true // Optional debug logs
});

Sending Transactional Emails

Below is an example of sending a welcome email when a user signs up:

// convex/actions/sendWelcomeEmail.ts
import { internalAction } from "./_generated/server";
import { v } from "convex/values";
import { bluefox } from "../bluefox";

// Sends a welcome email using a transactional template
export const sendWelcomeEmail = internalAction({
  args: { email: v.string(), name: v.string() },
  handler: async (_ctx, { email, name }) => {
    const result = await bluefox.email.sendTransactional({
      to: email,
      transactionalId: "welcome-email", // Configured in the Bluefox dashboard
      data: { name }
    });

    if (!result.ok) {
      console.error("Failed to send welcome email:", result.error.message);
    }
  },
});

The transactionalId maps to a pre-configured template in your Bluefox dashboard, and the data object passes dynamic fields (e.g., {name}).

Sending Triggered Emails

Triggered emails are ideal for automated lifecycle messages, such as re-engagement:

// convex/actions/sendReengagementEmail.ts
import { internalAction } from "./_generated/server";
import { v } from "convex/values";
import { bluefox } from "../bluefox";

// Sends an automated re-engagement email based on user inactivity
export const sendReengagementEmail = internalAction({
  args: { email: v.array(v.string()), lastActiveDate: v.string() },
  handler: async (_ctx, { emails, lastActiveDate }) => {
    await bluefox.email.sendTriggered({
      emails,
      triggeredId: "reengagement-email", // Configured in Bluefox dashboard
      data: { lastActive: lastActiveDate }
    });
  },
});

Error Handling Best Practices

bluefox-email returns a Result type for safer handling:

const result = await bluefox.email.sendTransactional({ ... });

if (!result.ok) {
  switch (result.error.code) {
    case "VALIDATION_ERROR":
      console.error("Invalid email data:", result.error.message);
      break;
    case "RATE_LIMIT_ERROR":
      console.error("Too many requests, try again later");
      break;
    default:
      console.error("Unexpected error:", result.error.message);
  }
}

Where to Trigger Emails (Mutations → Actions)

To maintain data consistency, use Convex mutation functions to call email-sending actions:

// convex/mutations/createUser.ts
import { mutation, internal } from "./_generated/server";
import { v } from "convex/values";

// Creates a new user record and then sends a welcome email
export const createUser = mutation({
  args: { email: v.string(), name: v.string() },
  handler: async (ctx, { email, name }) => {
    await ctx.db.insert("users", { email, name });
    // Schedule the welcome email to be sent after user creation
    await ctx.scheduler.runAfter(0, internal.actions.sendWelcomeEmail, {
      email,
      name,
    });
  },
});

Advanced: Handling Webhooks

Bluefox supports webhooks for real-time event notifications (opens, clicks, bounces, complaints). This allows you to track user interactions directly within your Convex application.

Features of Bluefox Webhooks

Real-Time Updates: Receive immediate data about user activity.
Customizable Events: Choose which events you want to listen for.
Secure Communication: Validate incoming requests via API keys.
Easy Integration: Works seamlessly with Convex’s actions or httpAction endpoints.

Setting Up Bluefox Webhooks

Navigate to Webhooks: In your Bluefox Project Settings, find Webhooks.
Add Webhook URL: Provide a Convex httpAction endpoint URL to handle POST requests.
Select Events: Choose the events you want to track (opens, clicks, bounces, etc.).
Secure Your API Key: After saving, note the key you’ll use to validate requests.
Test Your Endpoint: Use the “Test Webhook” feature to confirm everything is working.

(For more details, see the Bluefox Webhook Docs.)

Verifying Webhook Requests

Webhook calls include an API key in the Authorization header (e.g., Bearer YOUR-SECRET-KEY). Verify that key against your stored keys.

// convex/httpActions/webhookHandler.ts
import { httpAction } from "./_generated/server";
import { bluefox } from "../bluefox";

// Handles incoming Bluefox webhook events
export const handleBluefoxWebhook = httpAction(async ({ request }) => {
  try {
    // Validate the webhook using one or more valid keys
    await bluefox.webhooks.validateWebhook({
      request,
      validApiKeys: ["primary-key", "secondary-key"]
    });

    // Parse the incoming event for further processing
    const event = await bluefox.webhooks.parseWebhookEvent(request);

    switch (event.type) {
      case "open":
        console.log(`Email opened by ${event.emailData?.to}`);
        break;
      case "click":
        console.log(`Link clicked: ${event.link}`);
        break;
      case "bounce":
        console.warn(`Email bounced: ${event.emailData?.to}`);
        break;
      // Handle other event types as needed
    }

    return new Response("OK", { status: 200 });
  } catch (error) {
    console.error("Webhook validation failed:", error);
    return new Response("Invalid webhook", { status: 400 });
  }
});

Production Deployment Best Practices for Bluefox

1. Configure Environment Variables per Deployment Stage

When deploying to different Convex environments (dev, staging, production), you’ll want to isolate your email configurations:

Use Distinct API Keys: Create a separate Bluefox project (and associated API key) for each environment. This prevents test emails from accidentally sending to live users and keeps each environment’s data, template settings, and analytics separate.
Secure Access in Convex: Store BLUEFOX_API_KEY and other sensitive credentials (like AWS Access Keys) in the Convex Dashboard’s environment settings. This avoids checking secrets into version control and ensures only authenticated project members can update them.

Proper isolation reduces risk. If your staging environment’s email-sending logic has a bug, you don’t risk spamming real users or impacting your production deliverability.

2. Leverage Bluefox's Multi-Environment Support

Bluefox makes it easy to manage multiple projects—one for each environment:

Separate Projects: Create a “Dev” project, a “Staging” project, and a “Production” project in the Bluefox dashboard. This way, each environment has its templates, sender identities, and analytics.
Template & Data Separation: By segregating templates, you can safely experiment with design or copy changes in a non-production environment without risking your live email campaigns.

This approach ensures a clear boundary between testing and live operations. It also simplifies debugging—if something fails in dev, you know exactly which key and project to inspect.

3. Ensure AWS SES Production Access

Bluefox relies on your AWS SES credentials, so you need to move out of the “sandbox” mode:

Domain Verification: Verify your domain in the AWS SES console. This adds DNS records (SPF, DKIM) that prove you own the domain.
Request Production Access: AWS requires a justification for sending emails outside the sandbox. Be clear about your use cases (transactional vs. marketing) and compliance practices in your support request.
Monitor Sending Limits: Even in production, AWS imposes sending limits based on your account’s trust level. Keep an eye on your quota to avoid hitting a cap mid-campaign.

Proper SES configuration is essential for deliverability. If you forget to verify your domain or finalize production access, your emails may never leave the sandbox—or worse, land in spam.

4. Implement Bounce and Complaint Handling

Maintaining a good sender reputation is critical for high delivery rates:

Real-Time Monitoring: Configure webhooks for bounce and complaint events in Bluefox. This allows your app to react immediately when an email fails or a user marks it as spam.
Automated Suppression: Whenever you receive a bounce or complaint event, mark the email address as “unreachable” in your database. Sending again to these addresses can harm your domain reputation.
User Feedback Loop: Consider notifying users with outdated or invalid emails to update their information.

Unchecked bounces and spam complaints degrade your sender reputation. Over time, this can lead to your emails being blocked or flagged as spam by major providers.

5. Authenticate Your Domain with SPF, DKIM, and DMARC

Proper domain authentication is crucial for email security and deliverability:

SPF (Sender Policy Framework): Specifies which servers can send email on behalf of your domain.
DKIM (DomainKeys Identified Mail): Uses cryptographic signatures to prove an email came from your domain.
DMARC (Domain-based Message Authentication, Reporting, and Conformance): Builds on SPF and DKIM, telling receiving mail servers how to handle unauthenticated messages and providing feedback via DMARC reports.

These protocols prevent spoofing and boost trust. Many email providers (Gmail, Outlook, etc.) use authentication checks to decide if an email goes to the inbox or spam folder.

6. Provide Unsubscribe and Preference Management

Even if you’re mostly sending transactional messages, giving users control over what they receive is beneficial:

Unsubscribe Links: For marketing or optional notifications, link users to a page where they can opt out.
Preference Center: Provide a central place where users can pause certain emails or change frequency.
Legal Compliance: Regulations like CAN-SPAM (in the US) and GDPR (in the EU) often require some form of opt-out for marketing emails.

Offering an easy way to unsubscribe or pause notifications reduces spam complaints and legal exposure. It also builds user trust by respecting their communication preferences.

7. Monitor and Optimize Your Email Metrics

Simply sending emails isn’t enough—continuous monitoring is key:

Track Delivery, Open, and Click Rates: Bluefox offers dashboards to view engagement metrics in near real-time.
Analyze Bounces and Complaints: A sudden spike can indicate deliverability issues or a poorly targeted list.
A/B Testing Templates: Experiment with subject lines, content, and send times. Then compare open/click rates to refine your approach.

Ongoing analytics help you spot red flags early (e.g., deliverability drops). They also guide iterative improvements, ensuring your email performance keeps pace with user needs and business goals.

Conclusion

Bluefox is an excellent choice for anyone building a SaaS on Convex who needs a robust, developer-friendly email platform. By abstracting away the complexity of Amazon SES, it lets you focus on crafting great user experiences while still benefiting from SES’s world-class deliverability. Set up your Bluefox project, follow these best practices for webhooks and production, and you’ll be sending modern, scalable emails in no time!

Beyond SQL: How Convex Reimagines the Modern Database — A ClarityText Startup Story

Matt Luo — Mon, 07 Apr 2025 15:26:00 +0000

Beyond SQL: How Convex Reimagines the Modern Database — A ClarityText Startup Story is a guest Stack Post from the community Convex Champion, Matt Luo.

To someone new to Convex, it might look like Convex is in the database category. And that’s a reasonable interpretation. It’s difficult to explain what Convex is, since it straddles multiple well-defined categories. If I were to describe Convex in one sentence, without any regard for removing technical jargon, I’d say that Convex’s primary focus is providing deterministic state for an app’s backend. But see, that sentence is difficult to communicate with.

Part of my motivation to write this article is to explain Convex’s value proposition without the constraint of oversimplifying it in order to fit it into a pithy social media post.

I’ll use my own app, ClarityText, to illustrate how Convex is used in a real world business use case. ClarityText is a group chat platform for business. A customer can turn their chat history into an AI-powered knowledge base and get discovered in Google search and AI responses.

Let me talk through some of the key pieces of the Convex offering that replaces my need to incorporate another framework or vendor into ClarityText’s tech stack. I think seeing the use of Convex in this way helps to learn about Convex’s value proposition through analogies to other vendors.

Database solutions

When I was initially shopping around for a database, I inadvertently looked for the hosting of databases, like cloud hosting for MySQL, and no further features. As a startup founder, this was a mistake. As I went through the shopping process and learned more, I realized that there are table stakes features my app would need to have any chance against today’s competition.

These days, buying a hosted solution for MySQL and PostgreSQL, with nothing else, is actually more time consuming than using a back end as a service would be.

With a database and nothing else, I would have to do all the work in my MVP of:

Ensuring type safety with an ORM.
Making sure I don't create a security vulnerability with publicly exposed endpoints.
Doing caching, since that's probably a table stakes requirement for a consumer app these days.
Make a realtime front end, which is also probably a table stakes requirement for a consumer app MVP these days So, I think the back end as a service category, i.e. Convex, Supabase, and to a lesser extent, Firebase, can get a founder like me started faster toward an MVP.

Let me expound on each of these items.

ORM (object relational manager)

Type safety is a key native value proposition for an app based on Convex. It was appealing to me that I didn’t need a standalone ORM library.

At the time I started ClarityText, which was the late summer of 2024, the ORM industry was in a transitional state between popular libraries. This would have caught my startup in a time without a great ORM.

The longstanding popular ORM library was Prisma. It was a mature product with a syntax for edge relations. At the time, Prisma had a lot of negative social media posts about it for being slow. It had just gone through a large re-write into Rust, but the performance seemed to have gotten worse to many developers. Popular developer influencers like Codedamn made lengthy videos about why Prisma was slow. It also had just raised a lot of VC capital to build a business that was fundamentally about database and code analytics, with ORM as more of a tool for them to get customers.

Drizzle was an up and coming ORM library that was rapidly gaining popularity. I used it in an early prototype to map my React app with MySQL and Postgres. It worked for its advertised scope of features. But I wasn’t really excited about it. There wasn’t any remarkable ergonomic aspect to it. It was a utilitarian tool just to map a table. It could do a subset of what SQL could do, and then let the developer execute raw SQL as well. It seemed that most of the developer excitement about it was that the syntax more resembled SQL than Prisma’s did, and that it did not have the performance problems that Prisma had.

Drizzle also was at an early stage in development, and so were the ancillary database startups that marketed it. At the time, a Postgres hosting company, Neon, had a connector for Drizzle, but it was in beta.

In the Convex experience, I got a much deeper end to end type safety experience. I did not need an ORM at all. The return from the query to Convex itself would map to the variables in my code. And the Convex documentation provided many various ways to do the mapping.

I think this end to end type safety contributes to what Convex calls, the “Zen of Convex”. It enabled ergonomics I did not anticipate, such as

Intellisense in VS Code
I can execute npx convex dev to start a Convex local development environment, which would give me error messages about how my database schema was not in sync with my functions and current data.
When new technologies came out, like the model context protocol, AI assistants had an easy time interpreting my repo.

Performant database configuration

One relief for me was not having to decide on a database server size. With a database hosting service like Neon, I had to choose a server size and would need define how to auto scale it in the future. For a database expert, it’s probably fine to tune and configure everything. But I know enough about databases to know the vast surface area of my ignorance about database management.

So Convex replaces the need for something like a database deployment service. One instructive vendor to learn all about the tech involved with that is PlanetScale. PlanetScale has done a lot of work managing the connection pool for its customers. The connection pool is often a point of failure for an application with a lot of simultaneous activity to serve. PlanetScale runs instances of MySQL over a Vitess cluster. It is like an extreme example of horizontal scaling.

Also, Convex replaces the need to go back and forth with a SQL query explainer and planner. When working with SQL, it’s important to figure out whether the query you wrote is indeed using the indexes you expect it to use. Sometimes, someone very proficient with SQL can explicitly call a particular index, but it’s not the focus of the SQL language design to make that easy.

With Convex, I use Typescript to define my business logic for my database queries. I imperatively invoke the database indexes. It’s great. I just assume it’s Convex responsibility to execute the query optimally. And I can tell the dev experience will keep getting better. The Convex dashboard recently released a function health dashboard. And I think it’s inevitable that there will be some AI powered validation and guidance when I define my functions in the future.

System Design related solutions

A significant reason why ClarityText is using Convex is because Convex has already solved a lot of System Design related problems. The way I think about it, I can flip through the book, System Design Interview - An Insider's Guide, by ByteByteGo (Alex Xu), and think oh, this is already part of the Convex product.

The most important aspect of this is the state management of my entire app, back and forth between the front end and the back end. The most palpable benefit I get as a developer is a realtime front end, by default. Earlier in my career, and during my evaluation of other database products, I always had to refresh the browser tab in order to see my changed results on the front end. It was a profound moment for me to just simply add a ConvexProvider component to my React code, and just change data in the database and see the front end automatically updated.

So, this “sync engine” value proposition replaces a high-effort, expensive vendor like Ably. For a messaging app built on Postgres, Ably might be a good solution to keep the front end in sync. But there’s a significant amount of documentation and expertise needed to make it happen.

Convex’s “deterministic state” design enables another important benefit: query caching. From my perspective as an engineering manager, it helps that developers shift the responsibility to Convex to do the query caching. It would be much more difficult to bring another big piece into the tech stack like Memcached. We’d have to figure out as a team how to invalidate the cache. And at the React level, with Convex we don’t have as strong a need to do complicated memoizations. It’s very satisfying to simply see the cached indicator for a query in the Convex dashboard.

Regarding application security, I admit I do not completely understand public function security, but I know that many Firebase applications have gotten hacked because of allowing too much database access on the client side. You can easily see examples by searching for “firepwn.” With Convex, there are ways to distinguish public versus private functions. And for public functions, I can add argument validators and return validators to enforce data types. This replaces the need for complicated API gateway vendors like Kong.

Common application needs

Rate limiting packages

Every app exposed to the Internet probably has some rate limiting features in it. Upstash has a popular package for rate limiting, based on Redis. The Convex rate limiter component replaces this package, and has better ergonomics because it is easier to associate user activity with a foreign key to my users table in Convex than it is with a mapping to some table in Redis.

Materialized counting libraries

Some new application developers are surprised to learn how slowly a SELECT(*) in SQL performs. The two counting Convex components, sharded counter and aggregate components, use computer science techniques to denormalize the counting under the hood.

Ci cd.

Convex preview deployments are really useful. The ecosystem of Convex, Vercel, and GitHub make preview deployments easy to use. In our team, each time we have a PR, we have an accompanying Convex preview deployment hosted on a Vercel preview deployment. Even if we don’t click into the preview deployment, it is important to know whether the PR will build in Vercel before we merge it into our main branch.

Bringing Convex Dashboard into my dev environment

Bobby Alv — Mon, 17 Mar 2025 18:43:13 +0000

Why I built convexpanel.dev and what it means for my workflow

When I first started using convex, I realized how awesome and easy it was to use the dashboard. It offers an amazing overview of logs, the ability to edit table data, and view your app health. However, as a developer who always has an insane amount of browser tabs open, I found myself constantly switching back and forth just to check on things, and with the massive amount of tabs, I always spent more time looking for the dashboard tab than I did looking at it. Even though I could use the Convex CLI to open the dashboard directly, there was still that nagging need for a more streamlined, integrated experience.

The Spark Behind Convex Panel

After countless times of having to hunt down my convex dashboard tab, I thought: Why not build something that brings the dashboard directly into my development environment? I wanted something that allowed me to do numerous things.

View logs without leaving my dev environment
Being able to edit tables directly within my dev environment
View my data analytics / function rates quickly

The desire to reduce wasted time looking for a browser tab (having a cluttered browser doesn’t help) led me to build this awesome tool. A tool designed to integrate the key features I love from the Convex Dashboard directly into my development environment.

The Initial Approach: A Chrome Extension

My first thought was to build this as a Chrome extension, I figured it would be nice to have something accessible directly in my browser. The idea was simple: have a side panel that could display logs, table data, and function analytics while I worked in the browser. This seemed like the perfect solution—until I ran into security issues and the realization that maybe not everyone uses Chrome.

Convex requires authentication and access tokens to gather data, and Chrome extensions are highly restrictive when it comes to handling sensitive credentials. Storing API keys in local storage or extension settings felt like a security risk, and I didn’t want to force users to manually authenticate each time they opened the extension. Additionally, interacting with the Convex API directly from a browser extension posed CORS limitations, making it more complex than I initially anticipated.

Pivoting to a React Component

After running into these roadblocks, I realized that the best approach would be to integrate directly into the developer’s existing workflow—right in the dev environment. Instead of a browser-based solution, I decided to build ConvexPanel as a React component, making it easy to embed with any Convex project.

This approach had several advantages:

Direct API Integration: No need to deal with CORS restrictions or browser security policies.
Better Authentication Handling: Credentials could be managed securely within the dev environment by accessing .env files.
Seamless Dev Experience: No more switching between browser tabs—just open the panel in your editor by clicking the convex icon!

How Convex Panel Works

At its core, Convex Panel taps into the same Convex APIs that power the dashboard. By leveraging modern practices, it allows you to interact with your Convex backend without ever leaving your dev environment.

Quick Start Code Snippet

Install the package via npm:

npm install convex-panel

Then you can easily integrate this into your project

import React from "react";
import ReactDOM from "react-dom/client";
import App from "./App";
import "./index.css";
import { ConvexProvider, ConvexReactClient } from "convex/react";
import { ConvexPanel } from 'convex-panel';

const convex = new ConvexReactClient(import.meta.env.VITE_CONVEX_URL as string);

ReactDOM.createRoot(document.getElementById("root")!).render(
  <React.StrictMode>
    <ConvexProvider client={convex}>
      <App />
      <ConvexPanel
        accessToken="YOUR_ACCESS_TOKEN"
        deployKey={process.env.CONVEX_DEPLOYMENT}
        convex={convex}
      />
    </ConvexProvider>
  </React.StrictMode>,
);

You can easily get your access token from your convex config file:

cat ~/.convex/config.json

This will give you a response similar to this:

{
  "accessToken": "eyJ2BeI6A....."
}%

Your deploy key can be created directly in your dashboard

Displayed Logs

The logs displayed show information needed for any application:

Timestamp: Each log entry is tied to a specific time, allowing you to track when each request was made.
Function Calls: The logs display calls to functions: Queries, Mutations, Actions, etc., helping to monitor backend operations.
Performance Metrics: Each request includes an execution time (in milliseconds or microseconds), which is useful for debugging performance issues.
Filtering & Debugging Tools: The UI provides filters for log types, success statuses, and request IDs, making it easier to find relevant logs.
Detailed Log Output: The expanded log view reveals function details, execution times, and logs related to filters, calls, and tasks, which can assist in diagnosing errors or understanding data flow.

All of this kind of logging is essential for real-time debugging, performance monitoring, and troubleshooting any interaction directly from your development environment.

Table View

Edit Table Cells: Easily update data directly within the table.
Browse All Tables: View a complete list of your tables in the left-hand panel.
Filter Tables & Columns: Quickly find relevant data by filtering tables and individual columns.

This streamlined experience makes it easy to manage and inspect your Convex data.

Health View

Data Analysis: Performance metrics for cache hit rates and failure rates in graphs
Scheduler Lag: A quick scheduler status and lag time metrics shown in a graph.
Convex Version: An indication of your current Convex version and an update button if your convex is out of date.

Under the Hood

Convex Panel is built with simplicity and efficiency in mind:

Code Practice: The tool leverages modern Typescript practices to keep the codebase clean, modular, and easy to extend.
Direct API Integration: By tapping directly into Convex’s APIs, it ensures that the data you see is always in sync with your backend.
Standard Styling: Instead of using Tailwind, this project uses regular CSS Styling.

Conclusion

The ambition to build this component was born out of a personal need to reduce wasted time and improve efficiency and development. If you're like me and are always juggling too many browser tabs, even when amazing tools like Convex dashboard and Convex CLI are available, convex panel might just help you too!

Try it out, contribute, or provide feedback!

Resources

Convex
Convex Panel
NPM Package
GitHub
Convex Discord

Authentication Best Practices: Convex, Clerk and Next.js

Jamal — Thu, 06 Mar 2025 19:53:30 +0000

Authentication is the backbone of any full-stack application, but it’s also one of the easiest places to introduce subtle, hard-to-debug security flaws. As developers, we aim to build secure, reliable systems, but scaling an app to a global audience presents unique challenges—especially regarding authentication.

At ClarityText, where we use Convex in production, we ran into unexpected authentication issues that affected a significant number of users. The problem stemmed from how we handled authentication checks in our client-side application.

Our app relies on the useConvexAuth() hook to verify user sessions on the client. The useConvexAuth() hook returns a promise that needs to be awaited—meaning other React hooks, including Convex queries, can start running before authentication is fully validated. The problem arose from the unpredictability of whether authentication would complete before a Convex query executed. This introduced a race condition where unauthenticated requests could slip through, causing security risks and a frustrating user experience.

In this post, I’ll share what we learned about Convex authentication, how it integrates with Clerk, and best practices for handling authentication effectively. We’ll break down race conditions—what they are, why they happen, and the risks they pose. Then, I’ll introduce three key rules to help you prevent these issues when using Convex and Clerk. To wrap things up, I’ll provide custom hooks that you can easily implement to create safer, more reliable authentication flows in your applications.

Where Does Authentication Belong in Your Application?

Authentication isn’t just about verifying users—it’s about ensuring security at every layer of your application.
When building with Convex, Next.js, and Clerk (or similar tools), authentication needs to be considered at three key levels:

Server-side Authentication – Traditionally, authentication is handled at the backend using middleware. In frameworks like Next.js, middleware runs before API requests are processed, allowing authentication checks to happen early. Clerk, for example, uses middleware to validate user sessions before they reach API routes.
Client-side Authentication – Many web apps require real-time interactions, meaning authentication must also be handled by the client. React hooks like useAuth() from Clerk help verify user sessions in the browser, but client-side authentication alone isn’t secure since frontend code can be manipulated. It’s useful for UI state but shouldn’t be solely relied on for access control.
Database Authentication (Convex) – In a typical backend setup, authentication checks happen at the API level before interacting with a database. But with Convex, your backend is a database and public API. That means authentication checks must be enforced within Convex functions to ensure unauthorized users can’t read or write data.

Each of these layers plays a crucial role in securing your application. Failing to authenticate users at any of
them can lead to vulnerabilities, especially in real-time applications where database queries might execute before
authentication is fully validated.

Implementing Authentication Correctly

When working with authentication, it’s easy to assume that once a user is authenticated at one layer of the application,
they’re secure everywhere. But with Next.js and Convex, authentication needs to be explicitly handled at multiple
points—especially in applications that mix server and client components.

The Hidden Authentication Risk in Next.js

Next.js makes it easy to secure API routes and server components using middleware. However, if a server component
renders a client component that doesn’t properly handle authentication, it can expose parts of your app to unauthorized users.

Consider this example:

Server.tsx

import { ClientComponent } from './client-component';

export const ServerComponent = () => {
  const user_session = { userId: 'user_123' };   // fetch this data from convex or a nextjs cookie

  return <ClientComponent session={user_session} />;
};

export default ServerComponent;

Client.tsx

'use client';

import { api } from '@convex/_generated/api';
import { useQuery } from 'convex/react';

interface Props {
  session: {
    userId: string;
  };
}

export const ClientComponent = ({ session }: Props) => {
  const user = useQuery(api.users.get, { userId: session.userId });

  return <div>{user.password}</div>;
};

export default ClientComponent;

At first glance, this looks fine—middleware ensures the user is authenticated before the server component runs.
However, authentication must still be handled explicitly in client components. If the client-side logic assumes
authentication has already been verified, it could mistakenly expose data or trigger unauthorized Convex queries.

Convex is a Public API by Default.

Unlike traditional databases where queries run directly on your backend server, Convex operates as a public API,
meaning queries are executed over the network. This makes it easy to call Convex functions from client components,
but it also means authentication checks must happen at multiple layers—once on the client and again within Convex’s backend functions.
Since your Next.js server is separate from the Convex backend, securing one doesn’t automatically secure the other.

Use Internal Functions for Increased Security

While public functions are accessible to client-side code, Convex also supports
internal functions that can only be called by other
functions within your Convex project. These are useful for protecting sensitive logic that shouldn't be directly
accessible from the client. By using internal functions, you can minimize the public surface area of your application,
reducing the risk of exposing vulnerable operations to malicious users.

Internal functions are ideal for situations where you need to run more secure, sensitive logic—such as handling
authentication, processing payments, or updating user data—without risking direct client access. They can be invoked
within actions, cron jobs, or other internal logic flows, ensuring that your sensitive operations are kept behind the
security of your backend. However, remember that even with internal functions, you should still validate user permissions
and data integrity to safeguard against potential security flaws.

Ensuring Authentication in Server Components

When using React Server Components, authentication can be enforced entirely on the server, eliminating the need for client-side checks.
This approach ensures that sensitive data is never exposed to the client before authentication is verified. It also allows
us to fetch user-specific data more efficiently, similar to traditional server-side rendering (SSR).

In a Next.js application, middleware can be used to enforce authentication across all server components, ensuring that only
authorized users access certain pages.

Securing Routes with Middleware

Next.js middleware runs before a request reaches a page, allowing us to validate authentication and modify requests.
Using Clerk, we can protect all authenticated routes by checking the user's session and redirecting them if they are not logged in.

Example from Clerk Nextjs docs

import { clerkMiddleware, createRouteMatcher } from '@clerk/nextjs/server'

const isProtectedRoute = createRouteMatcher(['/dashboard(.*)', '/forum(.*)'])

export default clerkMiddleware(async (auth, req) => {
  const { userId, redirectToSignIn } = await auth()

  if (!userId && isProtectedRoute(req)) {
    // Add custom logic to run before redirecting

    return redirectToSignIn()
  }
})

export const config = {
  matcher: [
    // Skip Next.js internals and all static files, unless found in search params
    '/((?!_next|[^?]*\\.(?:html?|css|js(?!on)|jpe?g|webp|png|gif|svg|ttf|wo==ff2?|ico|csv|docx?|xlsx?|zip|webmanifest)).*)',
    // Always run for API routes
    '/(api|trpc)(.*)',
  ],
}

With this middleware in place, any request to /dashboard or /forum will require authentication.
If a user is not logged in, they will be redirected to the sign-in page automatically.

Fetching User Data in a Server Component

Once authentication is enforced through middleware, we can safely fetch user data within a server component
without worrying about unauthorized access. You should still check authentication within your convex functions.

Example: Next.js Middleware for Authentication

import { api } from "@convex/_generated/api";
import { auth } from "@clerk/nextjs/server";
import { fetchQuery } from "convex/nextjs";

export const ServerComponent = async () => {
  const { userId } = await auth();

  const user = await fetchQuery(api.users.get, { userId });

  return <div>Hello, {user.name}!</div>;
};

export default ServerComponent;

Why This Approach Works?

Middleware handles authentication upfront, ensuring that only authenticated requests reach protected pages.
Server components fetch data securely without exposing queries to the client.
No client-side authentication checks are required, reducing frontend complexity and reducing chance of race conditions.

Consider Other Frameworks and Security Risks

While this example focuses on Next.js, authentication practices vary between frameworks. If you're using Vue, Svelte,
or another React meta-framework (e.g., Remix, Gatsby), you should research how authentication and data-fetching work
in that ecosystem. Some frameworks may have different security risks or best practices for handling authentication,
session management, and data validation. Understanding how your framework manages authentication will help you implement
the most secure and efficient approach for your Convex app.

Ensuring Authentication in Client Components

While we’ve secured our server components, client-side authentication is just as important when handling user interactions
and front-end data fetching. Client components in Next.js can directly interact with Convex, but we must ensure that queries
are only executed when the user is authenticated. This prevents unauthorized requests and protects sensitive user data.

Convex provides the useConvexAuth() hook to manage authentication
state within React components. This hook allows us to track whether the user is authenticated before making any
queries or displaying protected content.

Understanding `useConvexAuth()`

The useConvexAuth() hook provides two key values:

isLoading – A boolean indicating whether the authentication state is still being determined. This helps prevent flashing unauthorized content before authentication is confirmed.
isAuthenticated – A boolean that indicates whether the user is signed in. If true, you can safely proceed with rendering user-specific content and making database queries.

Example: Checking Authentication in a Client Component

'use client';

import { api } from '@convex/_generated/api';
import { useConvexAuth, useQuery } from 'convex/react';

export const ClientComponent = () => {
  const { isLoading, isAuthenticated } = useConvexAuth();

  if (isLoading) return <div>Loading...</div>;
  if (!isAuthenticated) return <div>Not allowed!</div>;

  return <div>Welcome, user!</div>;
};

Setting Up the Convex Authentication Provider

To use useConvexAuth(), you need to ensure that the authentication state is properly managed and passed down
through a provider. In a Next.js app using Clerk, this is handled by the ConvexProviderWithClerk.

Example: Wrapping the App with an Authentication Provider

'use client';

import { ReactNode } from 'react';
import { ConvexReactClient } from 'convex/react';
import { ConvexProviderWithClerk } from 'convex/react-clerk';
import { ClerkProvider, useAuth } from '@clerk/nextjs';

const convex = new ConvexReactClient(process.env.NEXT_PUBLIC_CONVEX_URL);

export default function ConvexClientProvider({ children }: { children: ReactNode }) {
  return (
    <ClerkProvider publishableKey={process.env.NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY}>
      <ConvexProviderWithClerk client={convex} useAuth={useAuth}>
        {children}
      </ConvexProviderWithClerk>
    </ClerkProvider>
  );
}

By wrapping the entire application with ConvexClientProvider, authentication data is available throughout the component tree.
This ensures that the authentication state is correctly passed down to all client components.

Skipping Queries When the User is Not Authenticated

Even with proper authentication checks in the UI, we must ensure that Convex queries do not execute before the user’s
authentication state is confirmed on the client side. A common mistake is triggering queries before checking authentication,
which can lead to unnecessary backend calls or even security vulnerabilities.

Convex allows queries to be conditionally skipped using "skip", preventing them from running until authentication is established.

Example: Skipping Queries Until Authentication is Confirmed

'use client';

import { api } from '@convex/_generated/api';
import { useConvexAuth, useQuery } from 'convex/react';

export const ClientComponent = ({ session }: { session: any }) => {
  const { isLoading, isAuthenticated } = useConvexAuth();

  const user = useQuery(
    api.users.get,
    isAuthenticated ? { userId: session.userId } : "skip"
  );

  if (isLoading) return <div>Loading...</div>;
  if (!isAuthenticated) return <div>Not allowed!</div>;

  return <div>{user?.password}</div>;
};

Why This Matters

Prevents unauthorized queries – Requests are only sent when the user is authenticated.
Optimizes performance – Skipping queries reduces unnecessary backend calls.
Avoids race conditions – Ensures authentication is fully resolved before fetching data.

Securing Your Convex Backend

While securing authentication on the client and server is crucial, the final layer of security lies in your Convex backend functions.
Even if a user bypasses frontend checks, your backend must enforce strict authorization to protect your data.

Convex provides built-in authentication methods that allow you to validate user identity before processing any database operations.
By correctly implementing these checks, you can ensure that only authenticated users can access or modify data.

Enforcing Authentication in Convex Functions

Every Convex function—queries, mutations, and actions—receives a context (ctx) object that contains an auth property.
This allows you to verify the authenticated user before executing any logic.

Example: Restricting Access to Authenticated Users

import { mutation } from "./_generated/server";
import { v, ConvexError } from "convex/values"

export const myMutation = mutation({
  args: {
    someData: v.any()
  },
  handler: async (ctx, { someData }) => {
    const identity = await ctx.auth.getUserIdentity();

    if (identity === null) {
      throw new ConvexError("Unauthenticated call to mutation");
    }

    // Proceed with database operations only for authenticated users...
  },
});

**Working with User Identity Fields

When getUserIdentity() is called, it returns a UserIdentity object containing important authentication details. At a minimum, it will always include:

tokenIdentifier – A unique identifier for the user across authentication providers.
subject – The user’s unique ID from the provider.
issuer – The authentication provider that issued the token.

For users authenticated through Clerk or Auth0, additional fields like name, email, and pictureUrl may be available.

Enforcing Authorization with User Records

Authentication alone isn't enough—you also need to verify whether the user has permission to access certain data.
A common approach is to store user records in your database and check their roles or permissions.

Example: Looking Up a User in the Database

export async function getCurrentUserOrThrow(ctx: QueryCtx) {
  const userRecord = await getCurrentUser(ctx);
  if (!userRecord) throw new ConvexError("Can't get current user");
  return userRecord;
}

export async function getCurrentUser(ctx: QueryCtx) {
  const identity = await ctx.auth.getUserIdentity();
  if (!identity) return null;
  return await userByTokenIdentifier(ctx, identity.tokenIdentifier);
}

export async function userByTokenIdentifier(ctx: QueryCtx, tokenIdentifier: string) {
  return await ctx.db
    .query('users')
    .withIndex('by_token', (q) => q.eq('tokenIdentifier', tokenIdentifier))
    .unique();
}

This function retrieves the authenticated user’s record from the database. If no record is found, access is denied.

Best Practices for Backend Security

Always validate ctx.auth.getUserIdentity() before performing database operations.
Restrict access by role or permissions when necessary.
Log authentication failures to detect unauthorized access attempts.

Considerations for Other Frameworks

If you’re using a different authentication provider or another framework like Vue or Svelte, be sure to understand how the user state is managed.
Different frameworks may expose authentication details differently, and security risks can vary based on implementation.
Always review your authentication provider’s documentation to ensure best practices.

Simplifying Development with Custom Hooks and Utility Functions

When working with Convex, writing safe, reusable utilities is essential for making backend logic easier to implement and reducing
the risk of misusing complex functions. By creating secure wrappers around common database operations and React hooks, developers
can avoid redundant code while ensuring consistent authentication and error handling.

Custom Backend Utilities: Queries, Mutations, and Actions

Convex allows you to define backend functions like queries, mutations, and actions. To ensure these operations enforce authentication
consistently, we can wrap them in custom utility functions.

Example: Secure Convex Functions with Authentication

import { action, mutation, query } from '../_generated/server';
import {
  customAction,
  customCtx,
  customMutation,
  customQuery,
} from 'convex-helpers/server/customFunctions';
import { AuthenticationRequired } from '../users/utils';

/** Custom query that requires authentication */
export const authQuery = customQuery(
  query,
  customCtx(async (ctx) => {
    await AuthenticationRequired({ ctx });
    return {};
  }),
);

/** Custom mutation that requires authentication */
export const authMutation = customMutation(
  mutation,
  customCtx(async (ctx) => {
    await AuthenticationRequired({ ctx });
    return {};
  }),
);

/** Custom action that requires authentication */
export const authAction = customAction(
  action,
  customCtx(async (ctx) => {
    await AuthenticationRequired({ ctx });
    return {};
  }),
);

/** Checks if the current user is authenticated. Throws if not */
export async function AuthenticationRequired({
  ctx,
}: {
  ctx: QueryCtx | MutationCtx | ActionCtx;
}) {
  const identity = await ctx.auth.getUserIdentity();
  if (identity === null) {
    throw new ConvexError('Not authenticated!');
  }
}

One of the major benefits of extending Convex's base functions is the ability to modify and extend the context.
This allows you to inject additional data or logic before a query, mutation, or action executes. If you have further questions,
I recommend joining the Convex Discord server for community support and discussions.

At a high level, a custom function wraps and calls the base function internally. This provides an opportunity to execute additional
logic before the function runs. In the example above, the return {} statement passes an empty object, meaning Convex will retain its default
context values. However, you can modify this to return additional data, such as a user object, which would allow all your database queries
to access user data without redundant queries.

By structuring your custom functions this way, they become drop-in replacements for your existing queries and mutations while
maintaining full compatibility with Convex's default behavior.

Making React Querying Easier with Custom Hooks

When working with Convex in React, fetching data can become repetitive. To improve developer experience, we can use custom hooks to
abstract complex logic, making queries easier to implement while enforcing authentication.

Using `useQueryWithStatus` for Enhanced Query Information

The convex-helpers package provides makeUseQueryWithStatus, a utility that extends Convex’s useQuery to include additional metadata such as:

status: 'pending' | 'error' | 'success'
isPending: Whether the request is still in progress
isSuccess: Whether the request was successful
isError: Whether an error occurred
error: The error object
data: The data returned from the convex function

Example: Importing and Using `useQueryWithStatus`

import { makeUseQueryWithStatus } from 'convex-helpers/react';
import { useQueries } from 'convex-helpers/react/cache/hooks';

export const useQueryWithStatus = makeUseQueryWithStatus(useQueries);

This utility makes working with queries more intuitive by giving immediate feedback about their status.

Authenticated Query Hooks: Simplifying Authentication Logic

Instead of manually checking if a user is authenticated before running a query in every component,
we can create custom hooks that automatically handle authentication for us.

`useAuthenticatedQueryWithStatus` – Handling Authentication Automatically

import { FunctionReference } from 'convex/server';
import {
  OptionalRestArgsOrSkip,
  useConvexAuth,
  useQueryWithStatus,
} from 'convex/react';

/**
 * A wrapper around useQueryWithStatus that automatically checks authentication state.
 * If the user is not authenticated, the query is skipped.
 */
export function useAuthenticatedQueryWithStatus<
  Query extends FunctionReference<'query'>,
>(query: Query, args: OptionalRestArgsOrSkip<Query>[0] | 'skip') {
  const { isAuthenticated } = useConvexAuth();
  return useQueryWithStatus(query, isAuthenticated ? args : 'skip');
}

Paginated Query Hook: Handling Pagination Securely

For paginated queries, we can extend the same authentication logic:

import {
  PaginatedQueryArgs,
  PaginatedQueryReference,
  useConvexAuth,
  usePaginatedQuery,
} from 'convex/react';

/**
 * A wrapper around usePaginatedQuery that automatically handles authentication state.
 * If the user is not authenticated, the query is skipped.
 */
export function useAuthenticatedPaginatedQuery<
  Query extends PaginatedQueryReference,
>(
  query: Query,
  args: PaginatedQueryArgs<Query> | 'skip',
  options: { initialNumItems: number },
) {
  const { isAuthenticated } = useConvexAuth();
  return usePaginatedQuery(query, isAuthenticated ? args : 'skip', options);
}

Why Use Custom Hooks and Utility Functions?

Good API design and well-structured utility functions are crucial in production environments.
They simplify complex logic, improve maintainability, and make debugging easier—especially when working in a team.
Here’s why investing in custom hooks and utilities pays off:

Encapsulate Complex Logic – Developers don’t need to understand every failure mode or edge case; they just call a function that "just works" without needing to dive into internal details.
Enforce Security by Default – Authentication and permission checks happen in a centralized place, ensuring every function follows security best practices and reducing the risk of human error.
Reduce Boilerplate Code – Avoid copy-pasting authentication and error-handling logic into every function. Instead, these checks happen automatically, keeping code clean and DRY.
Improve Team Collaboration – A well-designed API allows other developers to use your functions without needing deep knowledge of how they work. This makes onboarding easier and speeds up development.
Simplify Debugging and Logging – Wrapping key operations in utilities makes it easier to log errors consistently, trace issues, and update logic in one place without refactoring multiple files.

Flexible And Secure Development

In production, good API design isn’t just about convenience—it’s about preventing costly mistakes and improving long-term maintainability.
Investing in structured, reusable utilities will save time, reduce bugs, and keep your application secure in the long run.

One Last Commit Before We Ship

At the end of the day, good API design isn’t just about making life easier—it’s about making sure you don’t wake up at 2 AM because someone
forgot an authentication check. By structuring your backend utilities and React hooks properly, you’re not just writing code—you’re writing
code that others will thank you for.

A well-designed API keeps things predictable, maintainable, and secure, making collaboration smoother and debugging less of a nightmare.
It’s the difference between confidently shipping a feature and spending hours in a Slack thread trying to figure out why that one function call only works on Bob’s machine.

And remember: race conditions aren’t just a problem in concurrent programming. They also happen in real life—like when two developers both
deploy a fix at the same time and accidentally roll back the one that actually worked.

So take your time, build smart utilities, and design APIs that make everyone’s job easier. Your future self (and your teammates) will thank you. 🚀

Building Type-Safe Rust Applications with Convex: Introducing convex-typegen

Jamal — Wed, 26 Feb 2025 02:12:03 +0000

If you've been following the backend-as-a-service landscape, you've likely heard of Convex. This innovative platform has been turning heads by offering a unique combination of developer experience, serverless functions, and real-time subscriptions, all wrapped in a developer-friendly package. What makes Convex particularly interesting is that under the hood, it's powered by Rust – a language choice that speaks volumes about its commitment to performance and reliability.

Speaking of Rust, it's fascinating to see how this systems programming language has found its way into backend development. While traditionally associated with low-level programming, Rust has become increasingly popular for building backend services, and for good reason. Its zero-cost abstractions, memory safety guarantees, and fearless concurrency make it an excellent choice for building high-performance, reliable services. Companies like Discord, Dropbox, and Cloudflare have all embraced Rust for critical backend components, proving its worth in production environments.

The relationship between Convex and Rust runs deep. Not only is Convex's core written in Rust, but they also provide first-class support for Rust developers through their official client library. This makes perfect sense – if you're building applications that demand both real-time capabilities and high performance, the combination of Convex's infrastructure and Rust's efficiency is hard to beat.

However, there was one piece missing from this otherwise perfect puzzle: type safety between your Convex backend and Rust frontend. While TypeScript developers enjoyed seamless end-to-end type safety with Convex, Rust developers had to manually maintain their types. That's where our journey begins – creating convex-typegen, a tool that brings the same level of type safety and developer experience to the Rust ecosystem.

Pain Points

When building applications with Convex, type safety is crucial for catching errors early and providing a great developer experience. TypeScript developers have it easy – Convex's TypeScript SDK automatically generates type definitions that perfectly match their backend schema and functions. This means they get real-time type checking, autocomplete suggestions, and compile-time guarantees that their code will work as expected.

For Rust developers, however, the story was different. Despite Rust being one of the most type-safe languages available, integrating with Convex required manually defining and maintaining types that mirror the backend schema. This manual process was not only tedious but also error-prone. Imagine updating a field in your Convex schema and then having to track down every Rust type that needs updating – it's the kind of maintenance burden that can slow down development and introduce subtle bugs.

Let's look at a concrete example. Consider this simple Convex schema:

import { defineSchema, defineTable } from "convex/server";
import { v } from "convex/values";

export default defineSchema({
  messages: defineTable({
    body: v.string(),
    userId: v.id("users"),
  }),
  users: defineTable({
    name: v.string(),
    tokenIdentifier: v.string(),
  }).index("by_token", ["tokenIdentifier"]),
});

Before convex-typegen, Rust developers had to manually maintain corresponding types:

#[derive(Debug, Clone)]
struct MessageDocument
{
    body: String,
    userId: String,
}

#[derive(Debug, Clone)]
struct UserDocument
{
    name: String,
    tokenIdentifier: String,
}

This manual mapping becomes even more complex when dealing with:

Nested objects and arrays
Optional fields
Union types
Custom validators
Function arguments and return types

What we needed was a tool that could automatically generate these Rust types, keeping them in perfect sync with the Convex backend, while respecting Rust's strong type system and idioms. This would not only eliminate the manual maintenance burden but also provide the same level of type safety that TypeScript developers enjoy.

Options I Considered

Before building convex-typegen, I explored using a generic JSON approach to manage type maintenance. This method leverages the flexibility of JSON to dynamically handle data without predefined types. In Rust, this can be achieved using the Value type from the serde_json crate, which allows you to work with JSON data in a type-agnostic manner.

use serde_json::Value;

However, when working with Convex, you have the option to use Convex's own Value type, which is specifically designed to represent the various data types supported by Convex's API. This type provides a more seamless integration with Convex's functions and can be a more idiomatic choice when interacting with Convex services.

use convex::Value as ConvexValue;

Example: Building a Message with Convex's `Value` Type

Consider the following MessageDocument struct that matches your schema:

#[derive(Debug, Clone)]
struct MessageDocument
{
    body: String,
    userId: String,
}

To construct a message using Convex's Value type, you need to create a BTreeMap<String, ConvexValue>:

use convex::Value as ConvexValue;
use std::collections::BTreeMap;

let create_message_args = BTreeMap::from([
    ("body".to_string(), ConvexValue::String("Crabs can walk in all directions, but mostly walk and run sideways".to_string())),
    ("userId".to_string(), ConvexValue::String("j97b03kebvey8gnqmdzhvwf69974e6c6".to_string())),
]);

client.mutation("createMessage", create_message_args).await?;

This BTreeMap represents a structured message that aligns with the MessageDocument schema, allowing you to pass it to Convex functions seamlessly.

Trade-offs

While both approaches can simplify initial development by eliminating the need to define and update Rust types manually, they come with significant trade-offs:

Loss of Type Safety: By using generic JSON or Convex's Value, you forfeit Rust's strong type guarantees. Errors related to data structure mismatches will only surface at runtime, rather than being caught at compile time.
Reduced Developer Experience: Without explicit types, you lose the benefits of Rust's powerful type system, such as autocomplete suggestions and compile-time checks. This can lead to a more error-prone and less efficient development process.

In summary, while the generic JSON approach and Convex's Value type can be quick fixes for avoiding manual type maintenance, they sacrifice the robustness and efficiency that Rust is known for. This underscores the need for a tool like convex-typegen, which can automatically generate Rust types that are perfectly in sync with your Convex backend, preserving both type safety and developer productivity.

Building convex-typegen

Embarking on the development of convex-typegen was a thrilling adventure that took me deep into the realms of Abstract Syntax Trees (ASTs) and the art of code generation. My mission was clear: to craft a tool that seamlessly converts Convex's JavaScript schema definitions into robust Rust types. This would not only enhance type safety but also liberate developers from the tedium of manual type maintenance. Here's a glimpse into how I brought this vision to life:

Step 1: Parsing JavaScript with Oxc

The first step in my journey was to parse the JavaScript schema definitions into a format that I could work with programmatically. This is where Abstract Syntax Trees (ASTs) come into play. An AST is a tree representation of the abstract syntactic structure of source code. Each node in the tree denotes a construct occurring in the source code.

To parse JavaScript into an AST, I used the Oxc JavaScript Oxidation Compiler. Oxc is a high-performance toolchain written in Rust, known for its speed and efficiency. Here's a simplified example of how I used it, inspired by the Oxc documentation:

use oxc::{
    allocator::Allocator,
    parser::{Parser, ParserReturn},
    span::SourceType,
};

let source_text = r#"
import { defineSchema, defineTable } from "convex/server";
import { v } from "convex/values";

export default defineSchema({
  messages: defineTable({
    body: v.string(),
    userId: v.id("users"),
  }),
  users: defineTable({
    name: v.string(),
    tokenIdentifier: v.string(),
  }).index("by_token", ["tokenIdentifier"]),
});
"#;

// Memory arena where AST nodes are allocated.
let allocator = Allocator::default();
// Infer source type (TS/JS/ESM/JSX/etc) based on file extension
let source_type = SourceType::from_path("schema.js").unwrap();
let mut errors = Vec::new();

let parser_result = Parser::new(&allocator, source_text, source_type).parse();

// The program is the root of the AST.
let program = parser_result.program;

This AST provided a structured representation of the JavaScript code, which I could traverse and analyze to extract the necessary information for generating Rust types.

Step 2: Analyzing the AST

Once I had the AST, the next step was to analyze it to identify the schema definitions and their corresponding types. This involved traversing the AST and looking for specific patterns that matched Convex's schema definitions.

For example, I looked for defineSchema and defineTable calls and extracted the field names and types. This information was crucial for generating the corresponding Rust types.

fn find_define_schema(body: &[JsonValue]) -> Option<&JsonValue>
{
    for node in body {
        // Check if this is an export default declaration
        if let Some(declaration) = node.get("declaration") {
            // Check if this is a call expression
            if declaration["type"].as_str() == Some("CallExpression") {
                // Check if the callee is defineSchema
                if let Some(callee) = declaration.get("callee") {
                    if callee["type"].as_str() == Some("Identifier") && callee["name"].as_str() == Some("defineSchema") {
                        // Return the declaration node
                        return Some(declaration);
                    }
                }
            }
        }
    }
    None
}

Step 3: Constructing a Code Generator

With the AST parsed and analyzed, the next step was to save the schema and function information into Rust data types. This involved creating structured representations of the Convex schema, tables, columns, and functions.

use serde::Value as JsonValue;

/// The convex schema.
#[derive(Debug, Serialize, Deserialize)]
struct ConvexSchema
{
    tables: Vec<ConvexTable>,
}

/// A table in the convex schema.
#[derive(Debug, Serialize, Deserialize)]
struct ConvexTable
{
    name: String,
    columns: Vec<ConvexColumn>,
}

/// A column in the convex schema.
#[derive(Debug, Serialize, Deserialize)]
struct ConvexColumn
{
    name: String,
    data_type: JsonValue,
}

/// A list of all known convex functions.
type ConvexFunctions = Vec<ConvexFunction>;

/// Convex functions (Queries, Mutations, and Actions)
#[derive(Debug, Serialize, Deserialize)]
struct ConvexFunction
{
    name: String,
    params: Vec<ConvexFunctionParam>,
    type_: String,
    file_name: String,
}

/// A parameter in a convex function.
#[derive(Debug, Serialize, Deserialize)]
struct ConvexFunctionParam
{
    name: String,
    data_type: JsonValue,
}

These data structures allowed me to store information in a way that was organized and easy to manipulate. By serializing and deserializing these structures, I could seamlessly convert between the parsed AST and the Rust code I needed to generate.

Achieving Type Safety in Convex Functions

One key challenge was ensuring type safety in Convex functions. This required parsing the function arguments and generating corresponding Rust code that maintained type integrity. By analyzing the AST, I extracted function definitions, including their parameters and types, and mapped them to Rust's type system.

For each Convex function, I generated Rust code that defined the function's parameters and function query key, ensuring they matched the schema's expectations. This involved creating Rust structs and enums that mirrored the JavaScript types, providing compile-time guarantees of type safety.

Here's a conceptual example of how I approached this:

fn generate_function_code(function: ConvexFunction) -> String
{
    let mut code = String::new();

    // Generate the args struct name
    let struct_name = format!("{}Args", capitalize_first_letter(&function.name));

    // Generate struct with derive macros
    code.push_str("#[derive(Debug, Clone, Serialize, Deserialize)]\n");
    code.push_str(&format!("pub struct {} {{\n", struct_name));

    // Generate fields for each parameter
    for param in &function.params {
        let rust_type = convex_type_to_rust_type(&param.data_type, None, None);
        code.push_str(&format!("    pub {}: {},\n", param.name, rust_type));
    }

    code.push_str("}\n\n");

    // Add implementation block with static FUNCTION_PATH method
    code.push_str(&format!("impl {} {{\n", struct_name));
    code.push_str("    pub const FUNCTION_PATH: &'static str = ");
    code.push_str(&format!("\"{}:{}\";\n", function.file_name, function.name));
    code.push_str("}\n\n");

    // Generate From implementation to convert to BTreeMap
    code.push_str(&format!(
        "impl From<{}> for std::collections::BTreeMap<String, serde_json::Value> {{\n",
        struct_name
    ));
    code.push_str(&format!("    fn from(_args: {}) -> Self {{\n", struct_name));

    // Only create map and insert values if there are parameters
    if function.params.is_empty() {
        code.push_str("        std::collections::BTreeMap::new()\n");
    } else {
        code.push_str("        let mut map = std::collections::BTreeMap::new();\n");
        // Convert each field to a serde_json::Value and insert into map
        for param in &function.params {
            code.push_str(&format!(
                "        map.insert(\"{}\".to_string(), serde_json::to_value(_args.{}).unwrap());\n",
                param.name, param.name
            ));
        }
        code.push_str("        map\n");t6
    }

    code.push_str("    }\n");
    code.push_str("}\n\n");

    code
}

There are many more details to this process, but this gives you a good idea of how I approached it.

Putting it All Together

To start using convex-typegen, follow these steps:

Step 1: Setting Up `build.rs`

First, configure your build.rs file to generate types from your Convex schema. This setup ensures that your types are always up-to-date with any changes in your schema.

use std::path::PathBuf;
use convex_typegen::{generate, Configuration};

// Auto-rebuild the types if the schema or messages files change
println!("cargo:rerun-if-changed=convex/schema.ts");
println!("cargo:rerun-if-changed=convex/messages.ts");

let config = Configuration {
    // Tell the parser where to find the function definitions
    function_paths: vec![PathBuf::from("convex/messages.ts")],

    ..Default::default()
};

match generate(config) {
    Ok(_) => {}
    Err(e) => panic!("Typegen failed: {}", e),
};

This script leverages convex-typegen to parse your schema and generate corresponding Rust types, automatically rebuilding them whenever your schema changes. The default file generated is convex_types.rs, but you can change this in the output_file field of the Configuration struct. Make sure to add this file to your src/main.rs so cargo can find it.

Step 2: Writing Type-Safe Code

With the types generated, you can now write type-safe code that interacts with your Convex backend. Here's how you can perform the same message mutation with type safety:

use convex::ConvexClient;
use convex_typegen::convex::ConvexClientExt;

// Import the generated types
use convex_types::CreateMessageArgs;

let client = ConvexClient::new("your-convex-url").await?;

let message_args_map = client.prepare_args(CreateMessageArgs {
    body: "Crabs are omnivores, meaning they eat both plants and animals.".to_string(),
    userId: "j97b03kebvey8gnqmdzhvwf69974e6c6".to_string(),
});

let message_result = client.mutation(CreateMessageArgs::FUNCTION_PATH, message_args_map).await?;

In this version, CreateMessageArgs is a Rust struct generated by convex-typegen, ensuring the arguments you pass are type-checked at compile time. This not only prevents runtime errors but also enhances the developer experience with features like autocomplete and inline documentation. The FUNCTION_PATH constant is a string that points to the function you're calling, and it's generated by convex-typegen based on the function's location in your convex backend directory.

Closing Thoughts

Building convex-typegen was more than just solving a technical problem—it was an opportunity to embrace challenges, learn deeply, and reaffirm key principles of software development.

As developers, we often face scenarios where existing tools don't quite meet our needs. In these moments, the ability to create something new becomes crucial. Developing convex-typegen required not only a thorough understanding of Convex and Rust but also a willingness to innovate and tackle problems without a predefined path.

This project highlighted the critical role of type safety in modern software development. By ensuring type consistency across the backend and frontend, we can identify errors early, minimize runtime issues, and improve overall reliability. convex-typegen brings the robust type safety of TypeScript to the Rust ecosystem, providing compile-time guarantees that boost developer confidence.

Additionally, it underscores the transformative power of automation in streamlining workflows. Automating the generation of Rust types from Convex schemas saves time, reduces human error, and enables developers to focus on what truly matters—building features and solving complex problems. This focus not only enhances productivity but also leads to higher-quality software.

Ultimately, the journey of building convex-typegen reflects the importance of innovation, the power of type safety, the value of community collaboration, and the efficiency of automation. I hope this tool empowers other Rust developers to build more robust and reliable applications with ease.

Footnotes

GitHub Repository: View the source code and examples on GitHub.
YouTube Demo: Watch a demonstration of convex-typegen in action on YouTube.
Convex: Learn more about Convex and its capabilities on their official website.
Rust: Discover why Rust is a popular choice for backend development on the Rust official site.
Oxc JavaScript Oxidation Compiler: Explore the Oxc documentation for more details on parsing JavaScript with Rust.
serde_json: Check out the serde_json documentation for working with JSON in Rust.

Matrix: Building a real-time RPG game with Convex

jashwanth0712 — Sun, 23 Feb 2025 18:30:00 +0000

Problem I was trying to solve

This year, I moved to a new place where people speak a different language than my mother tongue. Although English can be a common language for communication, I really wanted to learn the local language as a fun experiment.

I found language learning apps too boring and soon realized that the only effective way to learn a new language is by speaking it frequently. Through conversations, we gradually build vocabulary and eventually become fluent. However, this process can be particularly challenging for introverts like me. That’s when I got the idea to create a real-time game where everyone can join a map augmenting real-world situations.

Game idea

Similar to Minecraft, it’s a real-time RPG game where you can talk to people you come close to on the map. This would help users talk or chat with new people or AI NPC bots in the map and, in the process, learn the language.

I wanted a backend infrastructure to store the real-time locations of players and update their coordinates on all devices with the game open. Additionally, whenever two players came close, the chat interface would open quickly to enable communication between them.

Features I wanted:

A real-time way to store the coordinates of players
No need to handle all synchronization issues manually
A free tier without requiring a credit card to start
Good integration with Next.js

Convex for reactive DB

Convex had all the features I wanted for the application, so I proceeded with Convex as the backend and Next.js for the frontend. The templates in Convex made it very easy to get started. I wanted to use Next.js, Convex, and Clerk, and the Informal template was a perfect starting point for me.

Building the Frontend for the game

With a basic UI idea and user flow in mind, I started with v0, using prompts to generate the frontend for the game scene and map selection scene. To my surprise, the outputs from v0 were so good that I had to write very little code for the frontend myself. For those who don’t know v0 is a frontend code generator website that works really well with shadcn , tailwindcss and Nextjs which are the exact techstack that we are currently using.

Building the Backend Logic

ChatGPT + Convex = 🔥. I observed that by providing a sample mutation and query, ChatGPT can generate the entire database plan, schema.ts, and all the mutations and queries required for the backend logic. There are three main tables that i have used

Map Table
Chat Table
Player Table

Schemas Used

Map Table
It’s just a table configuration to store the dimensions and name, but in the future, it can store map tile images, nature, weather, and much more.

I added an index for map_name so that anyone searching for a particular map name can quickly retrieve the map details.

maps: defineTable({
    dimensions: v.object({
      height: v.float64(),
      width: v.float64(),
    }),
    map_name: v.string(),
  }).index("map_name", ["map_name"]),

Player Table
Any player should be present in only one map at a time. To fix this, instead of storing players in the map table, I have added x_coordinate, y_coordinate, and map_id to the player table itself. To make the search faster, I have added an index to the player's email ID

players: defineTable({
    img_url: v.string(),
    player_mail: v.string(),
    present_map_id: v.optional(v.id("maps")),
    x_coordinate: v.float64(),
    y_coordinate: v.float64(),
  })
    .index("player_mail", ["player_mail"])
    .index("present_map_id", ["present_map_id"]),
});

Chat Table

The chat table stored the details of each message and its corresponding sender-receiver pair. Encryption should have been added, but for the current implementation, I have left it as it is.

chat: defineTable({
    messages: v.array(
      v.object({
        message: v.string(),
        receiver: v.id("players"),
        sender: v.id("players"),
        timestamp: v.float64(),
      })
    ),
    receiver: v.id("players"),
    sender: v.id("players"),
  }).index("sender_receiver", ["sender", "receiver"]),

One thing I really like about Convex is that it offers amazing analytics and the ability to manage both production and development environments as easily as changing environment variables.

After the schema, the next most important aspects are the mutations and queries. I personally love the syntax of their mutations and queries. Unlike ORMs, they have a simple, code-like syntax that makes the developer experience truly great.

Whenever two players come close to each other, we enable the chat option between them. If this were any other traditional database, it would have taken a lot of effort to create this real-time functionality. But with Convex, it’s as simple as writing a better query.

Fetching Realtime player proximity

The following code is the query I used to present the chat options for players who are close to the current player:

    export const nearPlayer = query({
  args: {
    player_mail: v.string(),
    map_name: v.string(),
  },
  handler: async ({ db }, { player_mail, map_name }) => {
    const map = await db.query("maps")
      .withIndex("map_name", q => q.eq("map_name", map_name))
      .first();
    if (!map) {
      throw new Error("Map not found");
    }

    const player = await db.query("players")
      .withIndex("player_mail", q => q.eq("player_mail", player_mail))
      .first();
    if (!player) {
      throw new Error("Player not found");
    }

    const playersOnMap = await db.query("players")
      .withIndex("present_map_id", (q) => q.eq("present_map_id", map._id))
      .collect();

    const { x_coordinate, y_coordinate } = player;

    const nearbyPositions = [
      { dx: 0, dy: 0 }, 
      { dx: 1, dy: 0 },  
      { dx: -1, dy: 0 }, 
      { dx: 0, dy: 1 },  
      { dx: 0, dy: -1 }, 
    ];

    const nearbyPlayers = playersOnMap.filter(otherPlayer => {
      if (otherPlayer._id === player._id) return false;

      return nearbyPositions.some(pos => {
        const targetX = x_coordinate + pos.dx;
        const targetY = y_coordinate + pos.dy;
        return otherPlayer.x_coordinate === targetX && otherPlayer.y_coordinate === targetY;
      });
    });

    return nearbyPlayers.map(player => ({
      _id: player._id,
      player_mail: player.player_mail,
      img_url: player.img_url
    }));
  },
});

Writing similar database interactions with real-time features in other databases could have been a daunting task. Thankfully, Convex is here to save us with its intuitive syntax. For any player, I searched for all the players who are on the same map and are just one block away from the player.

Bringing Magic of AI into the application

Now that the basic implementation of the map, movements, and chat is done, we have added AI functionality to convert the message text into the language specified by the user. OpenAI perfectly handles the translation and provides accurate results.

Do you wanna build a RPG game? use this template

Following is the github repo of the Game and this has the entire boiler plate code to maintain maps, skins , movements and authentication. its easy to install it as well, looking forward for more fun games to be built with this template.

How I Wrote a Script to Migrate Data from MongoDB to Convex (and Lived to Tell the Tale)

Bobby Alv — Fri, 21 Feb 2025 18:22:31 +0000

Ah, data migration. The unsung hero of the developer’s life. It’s like moving houses, but instead of furniture, you’re hauling bits and bytes from one database to another. And just like moving, it’s always more complicated than you think. In this article, I’ll walk you through how I wrote a script to migrate data from MongoDB to Convex (shoutout to convex.dev), and hopefully, you’ll learn something — or at least get a good laugh.

The Setup: MongoDB and Convex Sitting in a Tree
First, let’s set the stage. I had a MongoDB database with over 30 tables and over 2M documents that relate to one another. But MongoDB, while great for many things, wasn’t cutting it for our new app’s real-time needs. Enter Convex, a backend-as-a-service that promised to make my life easier with its real-time capabilities and serverless architecture. The only problem? Getting all that data from MongoDB into Convex.

So, I rolled up my sleeves, brewed a pot of coffee, and got to work.

The Code: A Tale of Two Databases
Here’s the script I wrote to migrate the data. It’s a bit of a beast, but I’ll break it down for you. (You can find the full code at the end of this article, but let’s walk through the highlights.)

| Step 1: Connecting to MongoDB and Convex
First things first: I needed to connect to both databases. MongoDB was easy — I just used the MongoClient from the mongodb package. Convex, on the other hand, required a bit more setup, including an auth token.

const mongoClient = new MongoClient(MONGODB_URI);
const convexClient = new ConvexHttpClient(CONVEX_URL);

await mongoClient.connect();
convexClient.setAuth(authToken);

Pro tip: If your auth token is invalid, Convex will throw an error faster than you can say “unauthorized.” So, make sure you’ve got the right token.

| Step 2: Filtering and Counting Documents
Next, I needed to filter the team/organization/workspace object in MongoDB that I wanted to migrate. I created a filter only include documents from specific teams and to make sure that the object was not deleted.

const filter = {
  team: { $in: [
    new ObjectId('628...'),
    new ObjectId('628...')
  ]},
  isDeleted: { $ne: true }
};

const contactsCount = await contactsCollection.countDocuments(filter);
const taskCount = await tasksCollection.countDocuments(filter);
// ...and so on

I console logged This gave me a sense of how much data I was dealing with. Spoiler: It was a lot.

| Step 3: The Migration Loop
Now, the fun part: the migration loop. I processed the data in batches of 100 to avoid overwhelming the system (Convex only allows for a little over 16k). For each contact, I checked if it had already been merged (using a Set to track processed IDs) and skipped it if it had.

for (let skip = skipCount; ; skip += batchSize) {
  const batch = await contactsCollection.find(filter).skip(skip).limit(batchSize).toArray();
  if (batch.length === 0) break;

  for (const oldContact of batch) {
    if (processedMongoIds.has(oldContact._id.toString())) {
      console.log(`Contact already merged: ${oldContact.firstName} ${oldContact.lastName}`);
      continue;
    }
    // ... process the contact ...
  }
}

This loop is the heart of the script. It’s where the magic (and the headaches) happen.

| Step 4: Formatting Data for Convex
I had changed my schema in Convex, so I had to reformat the MongoDB data to fit. This included mapping phone numbers, emails, and addresses into my new structure.

const phoneNumbers = [];
if (oldContact.phoneNumbers?.length > 0) {
  phoneNumbers.push(...oldContact.phoneNumbers.map(p => ({
    label: p.phoneLabel || "Other",
    number: p.phone,
    isBad: p.isBadNumber || false,
    isPrimary: p.isPrimary || false
  })));
}

I also had to handle edge cases, like contacts with no first or last name. (Yes, those exist. No, I don’t know why.)

| Step 5: Creating Contacts and Tasks in Convex
Once the data was formatted, I used Convex mutations to create new records, tasks, tags, activities… and so on.

const result = await convexClient.mutation('contacts:create', newContact);

If a contact had associated tasks, tags or anything that was related to it, I created those too. This part of the script was like assembling IKEA furniture — tedious, but satisfying when it worked. (Also I never buy from IKEA anymore).

| Step 6: Error Handling and Retries
Of course, nothing ever goes perfectly. I added error handling and a retry mechanism to deal with hiccups like network issues or cursor timeouts.

async function retryOperation(operation, maxRetries = 3, delay = 1000) {
  for (let attempt = 1; attempt <= maxRetries; attempt++) {
    try {
      return await operation();
    } catch (error) {
      if (attempt === maxRetries) throw error;
      console.log(`Attempt ${attempt} failed, retrying in ${delay}ms...`);
      await new Promise(resolve => setTimeout(resolve, delay));
      delay *= 2; // Exponential backoff
    }
  }
}

The Aftermath: Lessons Learned
After hours of debugging, coffee, and the occasional existential crisis, the migration was complete. Here’s what I learned:

Batch processing is your friend. It keeps things manageable and prevents your script from crashing under its own weight.
Error handling is non-negotiable. Things will go wrong, so plan for it.
Log everything. When something breaks (and it will), you’ll want to know where and why.
Convex is awesome. Once the data was in, Convex made real-time updates a breeze.

The Code
Here’s the full script for your reading pleasure. Feel free to adapt it for your own migrations — just don’t forget the coffee.

import { MongoClient, ObjectId } from 'mongodb';
import { ConvexHttpClient } from 'convex/browser';

const MONGODB_URI = 'YOUR MONGO URI';
const CONVEX_URL = 'CONVEX CLOUD URL';
const createdContacts = new Map();
async function migrateProperties() {
  const mongoClient = new MongoClient(MONGODB_URI);
  const convexClient = new ConvexHttpClient(CONVEX_URL);
  try {
    console.log("Connecting to MongoDB...");
    await mongoClient.connect();
    const db = mongoClient.db('propbear');
    console.log("Connected to MongoDB.");
    console.log("Authenticating with Convex...");

    const authToken = "YOUR AUTH TOKEN";
    convexClient.setAuth(authToken);
    const user = await convexClient.query('users:viewer');
    if (!user) {
      throw new Error("Authentication failed");
    }
    console.log("Authenticated with Convex.");
    const propertiesCollection = db.collection('propertydetails');
    const contactsCollection = db.collection('contactdetails');
    const tasksCollection = db.collection('tasks');
    const linkedPropertyDetailsCollection = db.collection('linkedpropertydetails');
    const filter = {
      team: { $in: [
        new ObjectId('628...'),
        new ObjectId('627...')
      ]}
    };
    const propertyCount = await propertiesCollection.countDocuments(filter);
    const contactCount = await contactsCollection.countDocuments(filter);
    const taskCount = await tasksCollection.countDocuments(filter);
    const linkedPropertyDetailsCount = await linkedPropertyDetailsCollection.countDocuments(filter);

    console.log("Mongo Properties:", propertyCount);
    console.log("Mongo Contacts:", contactCount);
    console.log("Mongo Tasks:", taskCount);
    console.log("Mongo Linked Property Details:", linkedPropertyDetailsCount);
    console.log("Starting migration...");
    let migratedCount = 0;
    let errorCount = 0;
    let skippedCount = 0;
    const batchSize = 100;
    let lastId = null;
    const skipCount = 6800;
    const skipCursor = propertiesCollection.find(filter)
      .sort({ _id: 1 })
      .skip(skipCount)
      .limit(1);

    const lastSkippedDoc = await skipCursor.next();
    if (lastSkippedDoc) {
      lastId = lastSkippedDoc._id;
    }
    while (true) {
      const query = lastId ? { ...filter, _id: { $gt: lastId } } : filter;
      const cursor = propertiesCollection.find(query).sort({ _id: 1 }).limit(batchSize);
      let batch = await retryOperation(async () => await cursor.toArray());
      if (batch.length === 0) {
        console.log("No more properties to process. Migration completed.");
        break;
      }
      let allSkipped = true;
      for (const oldProperty of batch) {
        try {
          async function recordExistsByMongoId(client, table, mongoId) {
            try {
              const result = await client.query(`${table}:getByMongoId`, { mongoId });
              return result ? result._id : null;
            } catch (error) {
              console.error(`Error checking if ${table} record exists:`, error);
              return null;
            }
          }
          try {
            const existingPropertyId = await recordExistsByMongoId(convexClient, 'properties', oldProperty._id.toString());

            if (existingPropertyId) {
              console.log(`Property already exists: ${oldProperty.name}`, existingPropertyId);
              skippedCount++;
              continue;
            }
            allSkipped = false;
            let contactId = null;
            if (oldProperty.owner) {
              const oldContact = await contactsCollection.findOne({ _id: oldProperty.owner });
              if (oldContact) {
                contactId = await recordExistsByMongoId(convexClient, 'contacts', oldContact._id.toString());
                if (!contactId) {
                  const newContact = {
                    mongoId: oldContact._id.toString(),
                    recordType: "contacts",
                    firstName: oldContact.firstName || "",
                    lastName: oldContact.lastName || "",
                    fullName: `${oldContact.firstName || ""} ${oldContact.lastName || ""}`.trim(),
                    orgId: mapOrgId(oldContact.team?.toString()) || "nd70p3ngxkh8n7chaddb81tjpx7166mr",
                    phone: (() => {
                      let phoneNumbers = [];
                      if (oldContact.phoneNumbers && oldContact.phoneNumbers.length > 0) {
                        phoneNumbers = oldContact.phoneNumbers.map(p => ({
                          label: p.phoneLabel || "Other",
                          number: p.phone,
                          isPrimary: p.isPrimary || false,
                          isBad: oldContact.phoneDetails?.isBadNumber || false
                        }));
                      }
                      if (oldContact.phone && !phoneNumbers.some(p => p.number === oldContact.phone)) {
                        phoneNumbers.push({
                          label: oldContact.phoneDetails?.phoneLabel || "Other",
                          number: oldContact.phone,
                          isPrimary: oldContact.phoneDetails?.isPrimary || phoneNumbers.length === 0,
                          isBad: oldContact.phoneDetails?.isBadNumber || false
                        });
                      }
                      return phoneNumbers.length > 0 ? phoneNumbers : undefined;
                    })(),
                    email: (() => {
                      let emails = [];
                      if (oldContact.emails && oldContact.emails.length > 0) {
                        emails = oldContact.emails.map(e => ({
                          label: e.emailLabel || "Other",
                          address: e.email,
                          isPrimary: e.isPrimary || false,
                          isBad: oldContact.emailDetails?.isBadEmail || false
                        }));
                      }
                      if (oldContact.email && !emails.some(e => e.address === oldContact.email)) {
                        emails.push({
                          label: oldContact.emailDetails?.emailLabel || "Other",
                          address: oldContact.email,
                          isPrimary: oldContact.emailDetails?.isPrimary || emails.length === 0,
                          isBad: oldContact.emailDetails?.isBadEmail || false
                        });
                      }
                      return emails.length > 0 ? emails : [{ label: "Other", address: "", isPrimary: true, isBad: false }];
                    })(),
                    address: oldContact.address ? [{
                      label: "Home",
                      street: oldContact.address,
                      city: oldContact.city,
                      state: oldContact.state,
                      zip: oldContact.zipCode ? parseFloat(oldContact.zipCode) : undefined,
                    }] : undefined,
                    summary: oldContact.stickyNote,
                  };
                  Object.keys(newContact).forEach(key => newContact[key] === undefined && delete newContact[key]);
                  try {
                    let contactId;
                    if (createdContacts.has(oldContact._id.toString())) {
                      contactId = createdContacts.get(oldContact._id.toString());
                      await convexClient.mutation('contacts:updateContact', {
                        id: contactId,
                      });
                    } else {
                      contactId = await convexClient.mutation('contacts:createContactWithoutEnrichment', newContact);
                      createdContacts.set(oldContact._id.toString(), contactId);
                    }
                    if (oldContact.tags && oldContact.tags.length > 0) {
                      for (const tag of oldContact.tags) {
                        if (tag !== "Import") {
                          try {
                            const tagId = await createOrGetTag(
                              convexClient, 
                              tag.name || tag,
                              "contacts", 
                              contactId, 
                              newContact.orgId, 
                              await mapUserId(oldContact.createdBy?.toString()),
                              tag._id ? tag._id.toString() : undefined
                            );
                            if (tagId) {
                              console.log(`Created or got tag: ${tag.name || tag}`, tagId);
                            } else {
                              console.log(`Failed to create or get tag: ${tag.name || tag}`);
                            }
                          } catch (error) {
                            console.error(`Error processing tag ${tag.name || tag} for contact:`, error);
                          }
                        }
                      }
                    }
                    console.log(`Created contact: ${newContact.firstName} ${newContact.lastName}`, contactId);
                    // Create tasks for the contact
                    if (oldContact.tasks && oldContact.tasks.length > 0) {
                      for (const taskId of oldContact.tasks) {
                        const oldTask = await tasksCollection.findOne({ _id: taskId });
                        if (oldTask) {
                          const existingTaskId = await recordExistsByMongoId(convexClient, 'tasks', oldTask._id.toString());
                          if (!existingTaskId) {
                            const newTask = {
                              mongoId: oldTask._id.toString(),
                              orgId: newContact.orgId,
                              title: oldTask.task,
                              description: oldTask.description,
                              dueDate: oldTask.endDate ? new Date(oldTask.endDate).toISOString() : undefined,
                              linkedRecord: contactId,
                              assignedTo: await mapUserId(oldTask.createdForUserId?.toString()),
                              priority: mapPriority(oldTask.taskPriority),
                              column: mapStatus(oldTask.status),
                            };
                            Object.keys(newTask).forEach(key => newTask[key] === undefined && delete newTask[key]);
                            try {
                              const taskResult = await convexClient.mutation('tasks:createTask', newTask);
                              console.log(`Created contact task: ${newTask.title}`, taskResult);
                            } catch (error) {
                              console.error(`Error creating contact task ${newTask.title}:`, error);
                            }
                          } else {
                            console.log(`Task already exists: ${oldTask.task}`, existingTaskId);
                          }
                        }
                      }
                    }
                    // Create related contacts
                    if (oldContact.relatedContacts && oldContact.relatedContacts.length > 0) {
                      for (const relatedContact of oldContact.relatedContacts) {
                        const newRelatedContact = {
                          label: "Related Contact",
                          firstName: relatedContact.name.split(' ')[0] || "",
                          lastName: relatedContact.name.split(' ').slice(1).join(' ') || "",
                          email: [{
                            label: "Other",
                            address: relatedContact.email,
                            isBad: false,
                            isPrimary: true
                          }],
                          phone: [{
                            label: "Other",
                            number: relatedContact.phone,
                            isBad: false,
                            isPrimary: true
                          }],
                          orgId: newContact.orgId,
                          recordId: contactId
                        };
                        try {
                          const relatedContactResult = await convexClient.mutation('relatedContacts:createRelatedContact', newRelatedContact);
                          console.log(`Created related contact: ${newRelatedContact.firstName} ${newRelatedContact.lastName}`, relatedContactResult);
                        } catch (error) {
                          console.error(`Error creating related contact ${newRelatedContact.firstName} ${newRelatedContact.lastName}:`, error);
                        }
                      }
                    }
                  } catch (error) {
                    console.error(`Error creating contact ${newContact.firstName} ${newContact.lastName}:`, error);
                  }
                } else {
                  console.log(`Contact already exists: ${oldContact.firstName} ${oldContact.lastName}`, contactId);
                }
              }
            }
            // Create the property
            const newProperty = {
              mongoId: oldProperty._id.toString(),
              name: oldProperty.name,
              recordType: "properties",
              image: oldProperty.propertyImages && oldProperty.propertyImages.length > 0 ? oldProperty.propertyImages[0] : undefined,
              orgId: mapOrgId(oldProperty.team?.toString()) || "nd70p3ngxkh8n7chaddb81tjpx7166mr",
              propertyType: oldProperty.type || undefined,
              address: {
                street: oldProperty.address ? toProperCase(oldProperty.address) : toProperCase(oldProperty.name), 
                city: oldProperty.city ? toProperCase(oldProperty.city) : undefined,
                state: oldProperty.state ? oldProperty.state.toUpperCase() : undefined,
                zip: oldProperty.zipCode ? parseFloat(oldProperty.zipCode) : undefined,
              },
              location: oldProperty.coordinates && oldProperty.coordinates.lng && oldProperty.coordinates.lat ? {
                type: "Point",
                coordinates: [
                  parseFloat(oldProperty.coordinates.lng),
                  parseFloat(oldProperty.coordinates.lat)
                ],
              } : { type: "Point", coordinates: [0, 0] }, // Default coordinates if not available
              isDeleted: false,
              tags: oldProperty.tags ? oldProperty.tags.map(tag => tag._id?.toString()) : [],
              yearBuilt: oldProperty.propertyData?.yearBuilt ? parseFloat(oldProperty.propertyData.yearBuilt) : undefined,
              squareFootage: oldProperty.propertyData?.acres ? oldProperty.propertyData.acres * 43560 : undefined,
              units: oldProperty.units ? parseFloat(oldProperty.units) : undefined,
              price: oldProperty.propertyData?.forsale?.price ? parseFloat(oldProperty.propertyData.forsale.price) : undefined,
              parcelNumber: oldProperty.parcelNo || oldProperty.propertyData?.parcelNo || undefined,
              saleDate: oldProperty.lastSold ? new Date(parseFloat(oldProperty.lastSold)).toISOString() : undefined,
              salePrice: oldProperty.propertyData?.soldPrice ? parseFloat(oldProperty.propertyData.soldPrice) : undefined,
              landValue: oldProperty.landValue ? parseFloat(oldProperty.landValue) : undefined,
              buildingValue: oldProperty.propertyData?.bldgValue ? parseFloat(oldProperty.propertyData.bldgValue) : undefined,
              status: oldProperty.status,
              primaryUse: oldProperty.propertyData?.occupancy || undefined,
              construction: oldProperty.propertyData?.construction || undefined,
              lotSize: oldProperty.propertyData?.acres ? parseFloat(oldProperty.propertyData.acres) : undefined,
              zoning: oldProperty.propertyData?.zoning || undefined,
              meterType: oldProperty.propertyData?.meterType || undefined,
              class: oldProperty.propertyData?.class || undefined,
              structures: oldProperty.propertyData?.structures ? parseInt(oldProperty.propertyData.structures) : undefined,
              parking: oldProperty.propertyData?.parking || undefined,
            };
            // Remove createdBy field as it's not allowed
            delete newProperty.createdBy;
            Object.keys(newProperty).forEach(key => newProperty[key] === undefined && delete newProperty[key]);
            // Ensure location is always present
            if (!newProperty.location || isNaN(newProperty.location.coordinates[0]) || isNaN(newProperty.location.coordinates[1])) {
              newProperty.location = { type: "Point", coordinates: [0, 0] };
            }
            try {
              const result = await convexClient.mutation('properties:createPropertyWithoutEnrichment', newProperty);
              console.log(`Created property: ${newProperty.name}`, result);
              // Process tags for the property
              if (oldProperty.tags && oldProperty.tags.length > 0) {
                for (const tag of oldProperty.tags) {
                  if (tag !== "Import") {
                    try {
                      const tagId = await createOrGetTag(
                        convexClient, 
                        tag.name || tag,
                        "properties", 
                        result, 
                        newProperty.orgId, 
                        await mapUserId(oldProperty.creator?.toString()),
                        tag._id ? tag._id.toString() : undefined
                      );
                      if (tagId) {
                        console.log(`Created or got tag: ${tag.name || tag}`, tagId);
                      } else {
                        console.log(`Failed to create or get tag: ${tag.name || tag}`);
                      }
                    } catch (error) {
                      console.error(`Error processing tag ${tag.name || tag} for property:`, error);
                    }
                  }
                }
              }
              // Create tasks for the property
              if (oldProperty.tasks && oldProperty.tasks.length > 0) {
                for (const taskId of oldProperty.tasks) {
                  const oldTask = await tasksCollection.findOne({ _id: taskId });
                  if (oldTask) {
                    const existingTaskId = await recordExistsByMongoId(convexClient, 'tasks', oldTask._id.toString());
                    if (!existingTaskId) {
                      const newTask = {
                        mongoId: oldTask._id.toString(),
                        orgId: newProperty.orgId,
                        title: oldTask.task,
                        description: oldTask.description,
                        dueDate: oldTask.endDate ? new Date(oldTask.endDate).toISOString() : undefined,
                        linkedRecord: result,
                        assignedTo: await mapUserId(oldTask.createdForUserId?.toString()),
                        priority: mapPriority(oldTask.taskPriority),
                        column: mapStatus(oldTask.status),
                      };
                      Object.keys(newTask).forEach(key => newTask[key] === undefined && delete newTask[key]);
                      try {
                        const taskResult = await convexClient.mutation('tasks:createTask', newTask);
                        console.log(`Created property task: ${newTask.title}`, taskResult);
                      } catch (error) {
                        console.error(`Error creating property task ${newTask.title}:`, error);
                      }
                    } else {
                      console.log(`Task already exists: ${oldTask.task}`, existingTaskId);
                    }
                  }
                }
              }
              if (contactId) {
                await convexClient.mutation('contactLinkedProperties:linkPropertyFromMongo', {
                  contactId: contactId,
                  propertyId: result,
                  relation: 'Owner',
                  orgId: newProperty.orgId,
                });
                console.log(`Linked contact ${contactId} as Owner to property ${result}`);
              }
              // After creating the main property and owner contact
              await processLinkedProperties(convexClient, oldProperty, result, newProperty.orgId, {
                linkedPropertyDetailsCollection,
                contactsCollection
              }, createdContacts);
              migratedCount++;
            } catch (error) {
              console.error(`Error migrating property ${newProperty.name}:`, error);
              errorCount++;
            }
          } catch (error) {
            console.error(`Error migrating property ${oldProperty.name}:`, error);
            errorCount++;
          }
        } catch (error) {
          if (error.code === 43 && error.codeName === 'CursorNotFound') {
            console.error('Cursor not found, restarting migration from the last successful point');
            // You might want to implement a way to track the last successfully migrated property
            // and restart from there instead of from the beginning
            break;
          }
          throw error; // Rethrow other errors
        }
        // Update lastId after processing each property
        lastId = oldProperty._id;
      }
      if (allSkipped) {
        console.log(`All ${batchSize} properties in this batch were already migrated. Moving to next batch.`);
      }
      console.log(`Processed batch. Total migrated: ${migratedCount}, Skipped: ${skippedCount}, Errors: ${errorCount}`);
    }
    console.log(`Migration completed. Total migrated: ${migratedCount}, Skipped: ${skippedCount}, Errors: ${errorCount}`);
  } catch (error) {
    console.error('Error during migration:', error);
  } finally {
    await mongoClient.close();
  }
}
async function processLinkedProperties(convexClient, oldProperty, propertyId, orgId, collections, createdContacts) {
  const { linkedPropertyDetailsCollection, contactsCollection } = collections;
  const batchSize = 10; // Reduced batch size
  let skip = 0;
  while (true) {
    const linkedPropertyDetails = await linkedPropertyDetailsCollection.find({ property: oldProperty._id })
      .skip(skip)
      .limit(batchSize)
      .toArray();
    if (linkedPropertyDetails.length === 0) break;
    for (const link of linkedPropertyDetails) {
      await processLinkedProperty(convexClient, link, propertyId, orgId, contactsCollection, createdContacts);
      await new Promise(resolve => setTimeout(resolve, 100)); // 100ms delay between each link
    }
    skip += batchSize;
    await new Promise(resolve => setTimeout(resolve, 1000)); // 1 second delay between batches
  }
}
async function processLinkedProperty(convexClient, link, propertyId, orgId, contactsCollection, createdContacts) {
  try {
    const oldContact = await contactsCollection.findOne({ _id: link.contact });
    if (oldContact && oldContact._id) {
      let linkedContactId;
      if (createdContacts.has(oldContact._id.toString())) {
        linkedContactId = createdContacts.get(oldContact._id.toString());
      } else {
        // Create the linked contact
        const newLinkedContact = {
          mongoId: oldContact._id.toString(),
          firstName: oldContact.firstName || "",
          lastName: oldContact.lastName || "",
          fullName: `${oldContact.firstName || ""} ${oldContact.lastName || ""}`.trim(),
          email: oldContact.email ? [{
            label: "Other",
            address: oldContact.email,
            isBad: false,
            isPrimary: true
          }] : [{ label: "Other", address: "", isPrimary: true, isBad: false }],
          orgId: orgId,
          recordType: "contacts",
        };
        // Add company name if it exists
        if (oldContact.company) {
          delete oldContact.company;
        }
        // Use company name as fullName if no individual name is provided
        if (!newLinkedContact.fullName && newLinkedContact.company) {
          newLinkedContact.fullName = newLinkedContact.company;
        }
        // Skip truly empty contacts
        if (!newLinkedContact.fullName.trim() && !newLinkedContact.email[0].address.trim() && !newLinkedContact.company) {
          console.log(`Skipping empty linked contact for property ${propertyId}`);
          return;
        }
        try {
          linkedContactId = await convexClient.mutation('contacts:createContactWithoutEnrichment', newLinkedContact);
          console.log(`Created linked contact: ${newLinkedContact.fullName || newLinkedContact.company}`, linkedContactId);
          await convexClient.mutation('contactLinkedProperties:linkPropertyFromMongo', {
            contactId: linkedContactId,
            propertyId: propertyId,
            relation: link.relation || "Other",
            orgId: orgId,
          });
          console.log(`Linked contact ${linkedContactId} as ${link.relation || "Other"} to property ${propertyId}`);
          createdContacts.set(oldContact._id.toString(), linkedContactId);
        } catch (error) {
          console.error(`Error creating linked contact ${newLinkedContact.fullName || newLinkedContact.company}:`, error);
          return;
        }
      }
    } else {
      console.error(`Invalid linked contact found for property ${propertyId}:`, link);
    }
  } catch (error) {
    console.error(`Error processing linked property for property ${propertyId}:`, error);
  }
}
/**
 * @param {string | undefined} oldId
 * @returns {string | null}
 */
function mapOrgId(oldId) {
  if (!oldId) return "nd70p3ngxkh8n7chaddb81tjpx7166mr";
  const orgIdMap = { 
    '628..': 'nd7b...', // first is mongoId second is convex _id
    '627...': 'nd7d...'
  };
  return orgIdMap[oldId] || null;
}
function mapUserId(oldId) {
  if (!oldId) return null;

  const userIdMap = {
    '628c...': 'jx7...', // first is mongoId second is convex _id
    ...
  };
  return userIdMap[oldId] || null;
}
function mapPriority(oldPriority) {
  const priorityMap = {
    'Low': 'low',
    'Medium': 'medium',
    'High': 'high'
  };
  return priorityMap[oldPriority] || 'no-priority';
}
function mapStatus(oldStatus) {
  const statusMap = {
    'In Progress': 'in-progress',
    'Completed': 'done',
    'Not Started': 'todo'
  };
  return statusMap[oldStatus] || 'todo';
}
function toProperCase(str) {
  return str.replace(/\w\S*/g, function(txt) {
    if (isNaN(txt)) {
      return txt.charAt(0).toUpperCase() + txt.substr(1).toLowerCase();
    } else {
      return txt;
    }
  });
}
async function retryOperation(operation, maxRetries = 3, delay = 1000) {
  for (let attempt = 1; attempt <= maxRetries; attempt++) {
    try {
      return await operation();
    } catch (error) {
      if (attempt === maxRetries) throw error;
      console.log(`Attempt ${attempt} failed, retrying in ${delay}ms...`);
      await new Promise(resolve => setTimeout(resolve, delay));
      delay *= 2; // Exponential backoff
    }
  }
}
async function createOrGetTag(convexClient, tagName, recordType, recordId, orgId, createdBy, mongoId = undefined) {
  try {
    let existingTag;

    // Try to get the tag by mongoId if it exists
    if (mongoId) {
      existingTag = await convexClient.query('tags:getTagByMongoId', { mongoId });
    }
    if (existingTag) {
      // If the tag exists, return its ID
      return existingTag._id;
    } else {
      // If the tag doesn't exist, create a new one
      const newTag = {
        name: tagName,
        recordType,
        orgId,
        createdBy,
        mongoId
      };
      const tagId = await convexClient.mutation('tags:createTag', newTag);
      // Link the tag to the record
      await convexClient.mutation('tags:linkTagToRecord', {
        tagId,
        recordId
      });
      return tagId;
    }
  } catch (error) {
    console.error(`Error creating or getting tag ${tagName}:`, error);
    return null;
  }
}

migrateProperties().catch(console.error);

Final Thoughts
Data migration isn’t glamorous, but it’s a necessary evil. With the right tools and a bit of patience, you can move mountains of data without losing your sanity. And if all else fails, remember: there’s always more coffee.

More Examples: https://github.com/robertalv/merge-mongodb-convex

Happy coding! 🚀

Keeping Users in Sync: Building Real-time Collaboration with Convex

Hamza Saleem — Sun, 09 Feb 2025 17:24:33 +0000

Sticky my startup, is inspired by the idea of making collaboration as natural as doing it in person. Building features for Stick was both challenging and exciting experience. As I worked through it, I gained alot of insights on how to keep users in sync, handle data efficiently and to make sure everything works smoothly.

Why Real Time Collaboration?

Realtime collaboration is important for modern workflow. Either for brainstorming some ideas, editing documents or debugging your code. Immediate feedback keeps the momentum alive. It lets you bounce ideas of off each other just like you will do in person.

Apps like Google Docs or Figma have done a great job making this possible. I tried to bring the same effortless feel to brainstorming ideas using sticky notes,

Challenges making Real Time Collaboration work:

Its not just about syncing data but also about creating a seamless and engaging experience. Here's how i approached some of the problems using Convex:

Sync and Optimistic Updates

Syncing data in real time is a common challenge in a collaborative app. In Sticky, each user's change needs to be reflected across all the active sessions instantly the traditional approach would require creating a complex system using Pollling and WebSockets. Which requires ton of custom logic and backend setup.

How Convex Solves This:

Convex simplifies this with a native real-time data sync mechanism. By using the useQuery hook, you can easily fetch and sync data, like sticky notes, with minimal setup. Convex handles the underlying complexities of WebSocket connections, keeping all clients in sync automatically. This lets developers focus on creating features instead of managing the socket connections to keep everything connected in real time

Optimistic Updates:

When collaborating in real time, you'd expect changes to appear instantly, however network latency can cause delays in reflecting these updates making the user experience sluggish. to address this properly. I used optimistic updates to instantly reflect changes in the UI even before server save changes in the backend.

What Optimistic Updates Help Solve:

Optimistic updates solve the problem of UI delay caused by network latency. When user creates a new note the change is immediately shown in the Board even though the mutation maybe still be pending. The change is rolled back if mutation fails ensuring data consistency without effecting the user experience.

const createNote = useMutation(api.notes.createNote).withOptimisticUpdate(
  (localStore, args) => {
    const existingNotes =
      localStore.getQuery(api.notes.getNotes, { boardId: actualBoardId }) || [];
    const tempId = `temp_${Date.now()}` as Id<"notes">;
    const now = Date.now();
    localStore.setQuery(api.notes.getNotes, { boardId: actualBoardId }, [
      ...existingNotes,
      { _id: tempId, _creationTime: now, ...args },
    ]);
  }
);

By immediately showing the new note on the board, users can continue collaborating without waiting for the server's response. Only if the mutation completes with a different result will the UI be updated accordingly.

Real-time Presence Tracking Tracking who is online and where they are on the board is essential for real-time collaboration. In Sticky, I needed to show users cursor positions, indicating where others are working on the board. Real-time presence tracking also ensures that the team knows who is active and engaged.

What I Learned:

Implementing real-time presence required handling frequent updates of user cursor positions, which can overwhelm the system if not managed efficiently.

How Convex Solves This:

I used debounced updates to efficiently manage the frequency of presence updates. This function ensures the database isn't overloaded with too many requests by limiting the frequent updates.

Here’s a quick explanation of debouncing:

debouncing is a technique where the function only triggers after a specified delay, avoiding redundant or unnecessary calls. In this case, it helps prevent sending excessive updates for small, rapid movements of the cursor.

export function usePresence(boardId: Id<"boards">, isShared: boolean) {
  const updatePresence = useMutation(api.presence.updatePresence);
  const removePresence = useMutation(api.presence.removePresence);
  const activeUsers = useQuery(api.presence.getActiveUsers, { boardId });
  const cursorPositionRef = useRef({ x: 0, y: 0 });
  const [localCursorPosition, setLocalCursorPosition] = useState({ x: 0, y: 0 });

  const debouncedUpdatePresence = useCallback(
    debounce((position: { x: number; y: number }) => {
      if (isShared) {
        updatePresence({
          boardId,
          cursorPosition: position,
          isHeartbeat: false
        });
      }
    }, PRESENCE_UPDATE_INTERVAL, { maxWait: PRESENCE_UPDATE_INTERVAL * 2 }),
    [boardId, updatePresence, isShared]
  );

  useEffect(() => {
    if (!isShared) return;

    const heartbeatInterval = setInterval(() => {
      updatePresence({
        boardId,
        cursorPosition: cursorPositionRef.current,
        isHeartbeat: true
      });
    }, HEARTBEAT_INTERVAL);

    return () => {
      clearInterval(heartbeatInterval);
      removePresence({ boardId });
    };
  }, [boardId, updatePresence, removePresence, isShared]);

  return {
    activeUsers: isShared ? activeUsers : [],
    updateCursorPosition,
    localCursorPosition
  };
}

By debouncing the updates, the system can efficiently tracks user activity without overwhelming the backend or causing performance issues.

Schema Design and Indexing As real-time data grows, efficient database design becomes crucial. I had to ensure that data could be retrieved quickly as the board and user base expanded.

How Convex Helps:

Convex's schema design allows for easy definition of tables and indexes.

For example, the presence table schema enables efficient queries based on user and board, as well as tracking the last update time for each user.

presence: defineTable({
  userId: v.id("users"),
  boardId: v.id("boards"),
  lastUpdated: v.number(),
  cursorPosition: v.object({
    x: v.number(),
    y: v.number(),
  }),
})
  .index("by_board", ["boardId"])
  .index("by_user_and_board", ["userId", "boardId"])
  .index("by_board_and_lastUpdated", ["boardId", "lastUpdated"]);

These indexes ensure that as the database grows, data retrieval remains fast, even with a large number of active users or boards.

End-to-End Type Safety A major advantage of using Convex is its seamless integration with TypeScript. The type safety that TypeScript offers across the stack is a huge benefit when building real-time systems. With type-safe queries, mutations, and schema definitions, I was able to catch potential bugs during development, making the entire process smoother and more predictable.

Lesson Learned

I learned was how important is is for state management to be simple. Real time systems are naturally complex but with right tools it can be make a huge difference. With Convex, I can focus on creating a great user experience instead of worrying about the technicals.

A Final Thought

Building Sticky was a rewarding experience. The process taught me a lot about real-time systems and the value of tools that simplify complex tasks. For anyone looking to implement similar features, my advice is to start small, leverage existing tools, and iterate as you learn.

Check out Sticky to see these features in action or explore the source code for more implementation details.

DEV Community: Convex Champions

Scaling to 70 Modules: Building a Web, Mobile, and API backend on one Convex deployment

Production Convex: What 70 Modules Looks Like

What We Built

Permissions That Update Themselves

Adding Mobile Without Adding Backend

Real-Time for Users, REST for Everything Else

Workflows That Outlive Deployments

Search Without the Infrastructure

Transactional Emails in Convex with Bluefox and AWS SES

Table of Contents

Introduction

Setting Up Bluefox with Convex

1. Create a Bluefox Account and Project

2. Configure Your Environment

3. Install and Run Convex Locally

Integrating Bluefox with Convex Actions

Setting Up the Bluefox Client

Sending Transactional Emails

Sending Triggered Emails

Error Handling Best Practices

Where to Trigger Emails (Mutations → Actions)

Advanced: Handling Webhooks

Features of Bluefox Webhooks

Setting Up Bluefox Webhooks

Verifying Webhook Requests

Production Deployment Best Practices for Bluefox

1. Configure Environment Variables per Deployment Stage

2. Leverage Bluefox's Multi-Environment Support

3. Ensure AWS SES Production Access

4. Implement Bounce and Complaint Handling

5. Authenticate Your Domain with SPF, DKIM, and DMARC

6. Provide Unsubscribe and Preference Management

7. Monitor and Optimize Your Email Metrics

Conclusion

Beyond SQL: How Convex Reimagines the Modern Database — A ClarityText Startup Story

Database solutions

ORM (object relational manager)

Performant database configuration

System Design related solutions

Common application needs

Rate limiting packages

Materialized counting libraries

Ci cd.

Bringing Convex Dashboard into my dev environment

The Spark Behind Convex Panel

The Initial Approach: A Chrome Extension

Pivoting to a React Component

How Convex Panel Works

Quick Start Code Snippet

Displayed Logs

Table View

Health View

Under the Hood

Conclusion

Resources

Authentication Best Practices: Convex, Clerk and Next.js

Where Does Authentication Belong in Your Application?

Implementing Authentication Correctly

The Hidden Authentication Risk in Next.js

Convex is a Public API by Default.

Use Internal Functions for Increased Security

Ensuring Authentication in Server Components

Securing Routes with Middleware

Fetching User Data in a Server Component

Why This Approach Works?

Consider Other Frameworks and Security Risks

Ensuring Authentication in Client Components

Understanding useConvexAuth()

Setting Up the Convex Authentication Provider

Skipping Queries When the User is Not Authenticated

Why This Matters

Securing Your Convex Backend

Enforcing Authentication in Convex Functions

**Working with User Identity Fields

Enforcing Authorization with User Records

Example: Looking Up a User in the Database

Best Practices for Backend Security

Considerations for Other Frameworks

Simplifying Development with Custom Hooks and Utility Functions

Understanding `useConvexAuth()`

Using `useQueryWithStatus` for Enhanced Query Information

Example: Importing and Using `useQueryWithStatus`

`useAuthenticatedQueryWithStatus` – Handling Authentication Automatically

Example: Building a Message with Convex's `Value` Type

Step 1: Setting Up `build.rs`