Handling AI Traffic Spikes: Decoupling Express APIs with Redis and BullMQ

This is a submission for DEV's Summer Bug Smash: Smash Stories powered by Sentry.

Executive Summary & Roadmap

Imagine 100 students hitting "Submit" on a final exam at exactly 11:00 AM.

In a traditional CRUD app, this is a standard traffic spike. But when every submission requires evaluating multiple short-answer questions using an AI model, that standard spike turns into a bottleneck that can easily take down your entire backend.

At MOD-U-GO, an online examination platform, we wanted to automate the grading of text-based answers using Small Language Models (SLMs). We chose the Groq LLM API for its incredibly fast inference speeds. However, we quickly realized that bolting an LLM call directly into our synchronous HTTP request lifecycle was a recipe for disaster.

Here is how we solved this by decoupling our AI grading process from the submission lifecycle—designing an Asynchronous Architecture using Node.js, Bull, and Redis to grade exams at scale without ruining the user experience.

---

Topics to look forward to:

• The Pitfalls of Synchronous LLM Calls: Why outbound socket exhaustion and provider-side queueing trigger 504 Gateway Timeouts under load.

• The "Fire-and-Forget" Pattern: Decoupling user submissions from heavy background processing using Node.js, Bull, and Redis.

• Architectural Trade-offs & Zero Data Loss: Handling API timeouts safely with a pending_review state so no student exam is ever lost or scored zero.

• Scaling Strategies: Queue-level global rate limiting and horizontal worker scaling to handle large-scale exam traffic.

---

Table of Contents

1. The Problem: Why Synchronous LLM Grading Fails

2. Architecture & Data Flow: The "Fire-and-Forget" Pattern

3. Queue Configuration & System Resilience

4. Edge Cases & Architectural Trade-offs

5. Future Enhancements & Scalability

1. The Problem: Why Synchronous LLM Grading Fails

When building AI features, the most common anti-pattern is treating an LLM call like a database query. If we bolt an LLM directly into a synchronous HTTP request lifecycle, the math works against us.

The Mathematics of HTTP Timeouts

Let’s break down the exact numbers. A single call to an SLM to evaluate a short-answer response takes ~1.5 seconds on average. Assume we have an examination of 10 short-answer questions.

• 10 questions × 1.5 seconds = 15 seconds just to grade a single exam.

On paper, 15 seconds is under the standard 30-second Cloudflare/AWS ALB gateway timeout. But what happens when 30 students hit "Submit" at the exact same moment when the exam timer ends?

In a synchronous setup, 30 concurrent submissions create a burst of 300 outbound HTTP requests to the LLM provider. This triggers two immediate bottlenecks:

1. Outbound Socket Queuing: Node.js limits outbound concurrent sockets to prevent host memory exhaustion. As 300 requests fight for socket availability, later requests would wait in our server's local connection pool before ever reaching the network.

2. Provider Latency Degradation: Flooding an LLM gateway with 300 burst requests forces the provider to queue our jobs, degrading per-call latency from 1.5s to 3.5s+.

Because the questions are processed sequentially, an increase of just 2 seconds per inference call inflates total execution time to 10 * 3.5 = 35 seconds.

Hence the connection drops, returning a 504 Gateway Timeout.

The API Rate Limit Wall

Beyond timeouts, it immediately hit Limit. Those 30 concurrent exams generate 300 simultaneous API calls (30 students × 10 questions).

LLM providers strictly enforce RPM (Requests Per Minute) and TPM (Tokens Per Minute) quotas. Flooding an API with 300 rapid-fire inference requests will instantly trigger a wave of 429 Too Many Requests errors. Because the backend is processing this synchronously, those failures bubble straight up to the user, completely breaking the exam submission flow.

To prevent this cascading failure, we must decouple the submission from the evaluation.

2. Architecture & Data Flow: The "Fire-and-Forget" Pattern

To survive massive traffic spikes, we shifted to an Event-Driven Architecture backed by a job queue using Bull and Redis. The golden rule of this architecture is the Fire-and-Forget pattern: secure the data, acknowledge the request, and keep the heavy lifting for future.

Text-based questions are marked for Evaluation. The submission is saved to the database with a grading status, and the API immediately fires a 201 Created response back to the client. As far as the student is concerned, the exam is safely submitted with near-zero latency.

The End-to-End Submission Journey

1. Submission: The student submits their exam via a POST request.

2. Synchronous Split: The backend grades all MCQs instantly.

3. Immediate Client Response: The API returns a success response to the student without waiting for the LLM.(Near - Zero Latency)

4. Fire-and-Forget Enqueueing: The API pushes a lightweight job containing only the submissionId into the Bull queue, keeping Redis memory footprint small.

5. Worker Processing: A background worker process pops the job from Redis, fetches the full exam context from the database, and iterates through the text answers.

6. SLM Inference: The worker makes independent HTTP requests to the Groq API for each answer, asking the SLM to evaluate semantic similarity against a baseline model answer.

7. Persistence & Notification: The returned SLM scores (0.0 to 1.0) are normalized into actual marks. The database is updated, and the student receives a push notification that their final grade is ready.

System Architecture Diagram and Bull working

To understand more about Redis and it's working you can read - https://dev.to/socket_a64c6489500ee7/demystifying-redis-how-i-built-a-custom-in-memory-store-in-c-5fg4

3. Queue Configuration & System Resilience

Networks are uncertain, and third-party APIs go down. A grading system cannot afford to lose student data or assign a score of zero due to a dropped connection. We built resilience into the pipeline at two distinct levels: the queue configuration and the application layer.

Exponential Backoff & Retry Strategies

We configured our Bull queue with an Exponential Backoff strategy. If the Groq API drops a request, the worker doesn't immediately hammer the API again. It retries up to 3 times, waiting 2 seconds, then 4 seconds, and finally 8 seconds before giving up. We also keep a tight limit on job history (removeOnComplete: 100, removeOnFail: 50) to avoid Redis memory bloat.

// Bull queue robust default options
const queueOptions = {
  defaultJobOptions: {
    attempts: 3,
    backoff: {
      type: "exponential",
      delay: 2000, // Starts at 2s, exponentially increases
    },
    removeOnComplete: 100,
    removeOnFail: 50,
  },
};

In-Memory Graceful Fallback (Operating Without Redis)

In a rigid architecture, the grading pipeline would break entirely in case redis conenction failure We built an In-Memory Graceful Fallback directly into the queue producer. If the redisAvailable flag is false during enqueueing, the system catches the failure and immediately falls back to processing the grading job in the background memory of the Node.js process.

// Gracefully fallback if Redis is down
async function enqueueGrading(submissionId) {
  if (redisAvailable && gradingQueue) {
    try {
      await gradingQueue.add("grade-submission", { submissionId });
      return { async: true };
    } catch (error) {
      console.warn("Queue add failed (falling back to background memory)");
    }
  }

  // Background fallback: process without Redis
  processGradingJob(submissionId).catch((err) =>
    console.error("Background grading error:", err)
  );
  return { async: true };
}

While processing heavy tasks on the main Node event loop isn't ideal for massive scale, this fallback ensures that MOD-U-GO stays alive and student exams continue to be graded even during a total Redis outage.

4. Edge Cases & Architectural Trade-offs

Moving to an asynchronous, queue-driven pipeline solved our scale and timeout problems, but it introduced new challenges in user experience and edge-case handling.

The UX Dilemma: Immediate Feedback vs. Eventual Consistency(Trade-off)

In an ideal world, a student clicks "Submit" and sees their final grade instantly. By decoupling the grading pipeline, we had to embrace Eventual Consistency. We explicitly traded instant feedback for guaranteed system stability.

Zero Data Loss: The pending_review Fallback Strategy(Edge Case)

What happens when the worst-case scenario occurs? Suppose the Groq API goes down completely, and our Bull worker exhausts all 3 of its exponential backoff retries. In a naive implementation, the student might recieve score of zero . We engineer Zero Data Loss strategy. If the respone of SLM is a timeout or error , we gracefully handle it

} catch (error) {
  const isTimeout = error.code === "ECONNABORTED";

  // Gracefully fail this specific question, not the whole exam
  return {
    isCorrect: false,
    slmScore: null,
    marksAwarded: 0,
    gradingStatus: "pending_review", // Hand over to human teacher
    gradingMethod: "slm_semantic",
    error: isTimeout ? "SLM inference timeout (>30s)" : `SLM error: ${error.message}`,
  };
}

The status will be updated to pending review and alert the teacher that manual intervention is needed.

4. Future Enhancements & Scalability

Our asynchronous architecture solved our immediate bottleneck, but engineering is an iterative process. As MOD-U-GO scales to support larger institutions, it would require updates.

Queue-Level Rate Limiting

Currently, our Bull workers process jobs as fast as possible. If we scale out horizontally by adding more Node.js instances, our concurrent API calls will multiply, risking RPM limit. The next step is implementing Queue-Level Rate Limiting. By configuring Bull’s internal rate limiter (e.g., limiter: { max: 50, duration: 60000 }), which will act as a shared atomic clock and counter then limiting request overall the workers.

Horizontal Worker Auto-Scaling

Because our worker logic is completely decoupled from the main API servers, we can scale it independently. By monitoring Redis for Queue Depth (the number of pending jobs), we can configure Kubernetes (or our cloud provider) to dynamically spin up additional worker instances during peak exam periods (like midterms and finals), and scale back down to zero when the queue is empty to save compute costs.

Batch Prompting to Minimize Network Overhead

Right now, evaluating a 5-question exam requires 5 separate HTTP calls to the Groq API. This introduces significant network latency overhead. To optimize this, we can implement Batch Prompting. By aggregating all 5 text answers into a single, structured JSON prompt, we can evaluate the entire submission in a single API round-trip. This dramatically reduces network overhead and minimizes our total API request volume.

Conclusion

Building software with Third-Party AI integrations requires a fundamental mindset shift: never trust external API latencies inside your synchronous request paths.

By decoupling MOD-U-GO's exam submission endpoint from the SLM evaluation engine using Bull and Redis, we guaranteed near-zero submission latency for students, shielded our infrastructure from API rate-limit walls, and implemented graceful fallback states to ensure zero data loss—even when external services fail.

Architecting for scale isn't about preventing failures; it's about making sure your system degrades gracefully when they inevitably happen.

---

Thanks for reading! If you’ve built similar event-driven pipelines or have thoughts on queue optimization strategies with Node.js and Redis, I’d love to hear your feedback in the comments below!