DEV Community: Angelina Gupta

I ran Keploy on my MERN app. The scariest result was a green one.

Angelina Gupta — Sun, 27 Sep 2026 10:17:26 +0000

Nine of my API tests were getting 401 Unauthorized instead of 200. Keploy said the run passed. Exit code 0.

That was the last thing I found. Here's how I got there.

Why I tried this

Honestly, the first reason is that I'm applying for the DevRel internship at Keploy. The second is that I wanted to see how API testing actually works on a real app.

TaskFlow is my MERN project management app. It has an eval harness for its AI features, but the Express API itself had no tests at all. So I pointed Keploy at it and wrote down everything that happened, including the parts that went wrong.

Setup

App: TaskFlow backend (Express, MongoDB, JWT auth, and AI endpoints that call Groq)
Keploy: 3.8.44, Free, running natively on my MacBook Air (Apple Silicon), no Docker
Database: a throwaway local MongoDB, not my real Atlas database
Traffic: 12 requests covering register, login, workspace, project, task, read, update, delete, one AI call, and one request with a bad token Recording is one command:

keploy record -c "node server.js"

Use node, not nodemon. A restarting process confuses the recorder.

My first attempt just hung for five and a half minutes with no output. It turned out my Desktop folder syncs to iCloud, and most of node_modules had been offloaded, so Node was waiting on iCloud to download files one by one. That's not Keploy's fault. But Keploy also never told me that nothing was listening.

First run: 5 out of 12

When I stopped recording, Keploy immediately replayed what it had just recorded. I didn't ask it to; that's the default. 7 of 12 tests failed.

I did not expect 7 of my endpoints to fail. Then I read the diffs, and none of them were bugs. They were values that are different every time:

MongoDB _ids. Mongoose generates them inside the app, so they can't come from the mocks.
JWTs and the refresh-token cookie. They're freshly signed, with new timestamps.
Etag headers. Express hashes the response body, so if anything in the body changes, the Etag changes too.
joinedAt. A timestamp. The fix was nine lines of globalNoise in keploy.yml:

test:
  globalNoise:
    global:
      body:
        project._id: ['^[0-9a-f]{24}$']
        task._id: ['^[0-9a-f]{24}$']
        workspace._id: ['^[0-9a-f]{24}$']
        workspace.members._id: ['^[0-9a-f]{24}$']
        user.id: ['^[0-9a-f]{24}$']
        workspace.members.joinedAt: ['^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3}Z$']
        accessToken: ['^eyJ[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+\.[A-Za-z0-9_-]+$']
      header:
        Etag: ['^W/"[0-9a-f]+-[A-Za-z0-9+/]{27}"$']
        Set-Cookie: ['^refreshToken=eyJ[A-Za-z0-9_.-]+; Max-Age=604800; Path=/; Expires=[^;]+; HttpOnly; SameSite=Lax$']

I used regex patterns instead of empty arrays on purpose. An empty array ignores the field completely. A pattern still checks that the new value looks right, so a null or a malformed ID would still fail.

What Keploy showed me about my own app

This was the part I didn't expect. Keploy records every database and network call, so reading the mocks felt like reading my backend's diary.

1. A background write I'd forgotten about. When a task is created, TaskFlow computes an embedding and saves it to MongoDB, but it doesn't wait for that before responding. Keploy caught these writes landing in the next test's time window. When they failed, my app just logged embedding failed and moved on. No error, nothing visible to the user.

2. PUT /api/tasks/:id makes 33 database calls. The next-highest endpoint makes 9. I have no idea why yet, and I'm going to find out.

3. A 37 MB download hiding at runtime. The first time a task is created, TaskFlow downloads an embedding model from HuggingFace. Keploy captured all of it, and that one download made up about 98% of mocks.yaml. After I pre-downloaded the model outside Keploy, mocks.yaml went from 37.7 MB to 286 KB.

Green: 12 out of 12

With the noise config and the model cached, I re-recorded and ran:

keploy test -c "node server.js" --delay 10

12 out of 12 passed.

The part I'd actually use every day: the Groq call is mocked too. The AI endpoint test replays the recorded LLM response instead of calling Groq. So it's deterministic, it's free, and it doesn't break because the model phrased something differently today.

One honest caveat: the background embedding write from point 1 is still a race. It only lined up on this run because the cached model made everything faster. The test is green, but the race is still there.

The green run that wasn't

My access tokens expire after 15 minutes. Keploy replays the exact token it recorded. So I waited and ran the tests again, 16 minutes after recording.

Nine tests got 401 Not authorized instead of the recorded 200/201. Here's what Keploy reported:

)
Default run, 16 minutes after recording.

The other nine were marked obsolete, not failed. The report said PASSED, and the process exited with code 0. In CI, that's a green check on top of nine broken endpoints.

Keploy's summary said the mocks were probably stale and suggested re-recording, or running with --update-test-mapping. But the real cause was the 401. Updating the mappings would have quietly rewritten the tests to match the broken behavior.

Here's why it happens:

My auth middleware rejects the expired token before touching the database.
So the test uses none of the database mocks it recorded.
Keploy sees "different mocks than expected" and marks the test obsolete.
By default, obsolete doesn't count as a failure. This is documented. The --help text for --assert-dependencies says such a test is "demoted to OBSOLETE today and the run still exits 0." I just hadn't read it until I went looking.

That's what stuck with me: not all bugs are easy to see, and they can still be fatal. A test suite that can't go red isn't protecting you.

What about --freezeTime? It's meant for exactly this: it pins the app's clock to the recording time. But the docs list it as an Enterprise feature for Linux, WSL and Docker. On native macOS, Keploy tried to inject a Linux library, logged an error saying time freezing couldn't be verified, and ran the tests anyway. Same result.

The fix: one flag

Same recording, same expired tokens:

Run	Passed	Failed	Obsolete	Report	Exit code
Default	3	0	9	PASSED	0
`--strict-failure`	3	9	0	FAILED	1
`--assert-dependencies`	3	9	0	FAILED	1

)
Same tests, same expired tokens, with --strict-failure.

--strict-failure fails a test when its response is wrong.
--assert-dependencies fails a test when its expected database or network calls never happened. That catches cases where the response still matches but the logic behind it has vanished. With --strict-failure, Keploy's summary also named the real cause (expected=200 got=401) instead of blaming stale mocks.

If you run Keploy in CI, add one of these. I'd use both.

Watch your secrets

A short checklist, because I learned this the hard way:

Outbound calls are recorded with their headers. My Groq API key ended up in mocks.yaml in plain text. Check your mocks before you commit or share anything.
Report upload and mock upload are separate settings. I turned off report upload, but my mocks were still uploaded to Keploy's registry on the green run. If you don't want that, set test.disableMockUpload: true as well.
mocks.yaml is gitignored by default. That's good for git, but a search tool that respects .gitignore will skip the one file most likely to contain your keys. I rotated my Groq key. Twice.

Would I recommend it?

Yes, to check your code's reliability before it hits production. Recording real traffic and getting tests plus mocks back, including for LLM calls, is genuinely useful, and the logs explained almost everything I ran into.

My one warning: don't trust the default green. Run with --strict-failure, and read your mocks for secrets.

My last big bug was a token cap silently truncating JSON in 26% of TaskFlow's AI requests, and the lesson here was the same: a green check isn't proof that things work. I wrote about that one here.

My AI feature was failing 26% of the time. Nothing looked broken.

Angelina Gupta — Sat, 26 Sep 2026 13:44:43 +0000

I built TaskFlow, a project management app with an AI feature called Quick Add. You type "assign the API docs to Priya by Friday" and it creates a task with a title, an assignee, and a due date.

For twelve days in August, it quietly stopped doing that for about a quarter of requests. Users got a task titled "assign the API docs to Priya by Friday" — no assignee, no date. No error. No crash. Just a worse product.

My test suite caught it on day one. I didn't look until day twelve. This post is about both of those things.

Why LLM features need their own tests

A normal unit test checks that a function returned. It can't tell you whether the model got the date right. LLM output looks fluent even when it's wrong, so "it didn't crash" means very little.

So I built an eval harness: 71 test cases across 4 prompt suites — quick-add (50), extract-tasks (9), decompose (6), and today (6). Each case is an input plus the fields the model should return.

A few decisions that turned out to matter:

It imports the real production prompts, not copies. If a prompt changes, the tests test the change.
Scoring is deterministic. No LLM grading another LLM — dates and assignees either match or they don't.
The clock is pinned, so "Friday" means the same date on every run.
It mirrors production's token limits. This one ends up being the whole story.
It runs in GitHub Actions: a small subset on every pull request, the full suite nightly.
It separates a wrong answer from a failed request. Rate-limit errors get retried, so a 429 doesn't show up as a quality failure.

What happened

Groq retired the model I was using, llama-3.3-70b-versatile, on August 16. I switched to openai/gpt-oss-120b on August 24.

The nightly suite went red that same night. It stayed red for twelve consecutive nights before I looked properly.

Two things kept it hidden:

The fallback worked too well. When Quick Add couldn't parse the model's output, it used the raw text as the task title. That's the right design — the app stays usable — but it meant there was no error to notice.
The error looked like someone else's problem. 13 of the 15 failures were 400 Failed to validate JSON. That reads like an infrastructure issue, so my first instinct was to blame the provider.

Finding the cause

The error message was a dead end. The distribution wasn't.

Suite	Result	`max_tokens`
quick-add	35/50	300
extract-tasks	9/9	1500
decompose	6/6	1800
today	6/6	900

Every failure was in the one suite with the smallest token limit.

The token data confirmed it. Of the quick-add cases that returned, the five largest completions were 283, 284, 288, 298 and 298 tokens — jammed right up against the 300 limit. The old model's largest was 64.

The new model is a reasoning model. It spends tokens thinking before it writes any JSON. With a 300-token cap, it was running out mid-object and returning truncated JSON, which the provider then rejected.

The fix — and what it uncovered

The first fix was one number: max_tokens from 300 to 900, in two places that must always match — the production controller and the eval harness.

That exposed three more bugs the 400s had been hiding. Once the model could finish its answers, three date cases returned due: null:

"by end of week"
"by July 20"
"in two weeks"

The prompt gave the model a 10-day calendar and told it never to compute weekdays itself. So anything outside that window came back empty. The proof: "in 3 days" and "by July 9" already passed, because both fall inside it.

I split the rule. Weekday phrases still resolve from the calendar; absolute dates and offsets are computed forward from today. The prompt now says the calendar is "a 10-day window, not a limit on what you may answer."

Priority had a similar gap — it only recognised words like "asap" or "critical", so "this is blocking the release" came back as unstated. It now judges described impact, not tone.

Quick-add went from 35/50 to 50/50. The full suite went from 78.9% to 98.6%.

Where it stands today

As of the latest nightly, it's 69/71 — 97.2%. Two cases fail.

One is a genuine quality miss. The "today" planner includes a task it should filter out. I haven't fixed it because it's a ranking judgment, not a rule I can state cleanly — and I'd rather keep one honest failure than overfit the prompt to a single test case.

The other is the same bug coming back. Same truncated-JSON signature, now in the "today" suite at its 900-token cap. My fix was too narrow: I raised the limit where it was failing, instead of asking which other endpoints had limits sized for a non-reasoning model. One endpoint is still at 400 tokens with no eval coverage at all.

What I'd tell other developers

Graceful fallbacks hide bugs. Make your fallback path loud — log it, count it, alert on it.
Read the distribution, not the error. The error text pointed at the provider. The pattern across suites pointed at the real cause.
When you fix a limit, audit every place with the same assumption. A model change breaks assumptions everywhere at once.
A red test nobody reads isn't a test. The harness did its job on night one. The gap was mine.

TaskFlow is live at taskflow-dpsa.vercel.app. Code: github.com/angelina10504/taskflow.