DEV Community: Rahul

Red, then green: 11 bugs TestSprite found in my NASA video research agent

Rahul — Mon, 26 Oct 2026 23:39:32 +0000

TL;DR

Ephemeris is a research agent over 87 NASA clips. I put 85 TestSprite tests behind it (34 browser, 51 API), and they found 11 real bugs.
The centre of the story is PR #13: the first commit is only a regression test and goes red, the second carries the fixes and goes green.
Every pull request gets a preview of its own and a TestSprite check before it can merge. Below: how I set that up, what the tests caught, and the steps to copy it.

This is my write-up for TestSprite Season 4.

The problem: an archive in the wrong order

NASA published almost nothing online before the year 2000. So a clip uploaded in 2015 is routinely discussing results from 1976. Sort the video library by its own metadata and you get the order things were posted, which is not the order anything happened.

That matters if you want to understand how a field changed its mind. The real value of an archive like this is somebody on camera in 1965 saying what they thought Mars was, and being wrong, and somebody in 2015 saying what we know now. Titles and tags do not capture that. The truth is in the spoken words and in what was printed on the screen behind them.

Today you can find a clip called "Mars report". You cannot ask how our understanding of water on Mars changed and get back the exact moments that answered it. That is what I built Ephemeris to do.

What Ephemeris does

You ask a question in plain English and get three things:

A cited answer, ordered by when events actually happened, not by when NASA uploaded the file. Every claim carries a citation number that maps to a moment in a clip.
An evidence reel, one video cut from the exact seconds that support the answer, with the citation number, the year, how that year was determined, the mission and the NASA identifier burned into the frame so it survives an export.
A 3D solar system whose camera follows the reel. Each moment stands on the body it concerns, and the camera moves through the decades as the reel plays.

The corpus is 87 real NASA public-domain clips: 240 minutes, 1,484 scenes, 1957 to 2025. Nothing is synthetic and nothing is mocked. The live app is at https://ephemeris-taupe.vercel.app, and the code is at https://github.com/crypticsaiyan/Ephemeris. The four preset questions on the landing page are real pipeline output and need no key.

How it works, briefly

The tests only make sense if you know what they are guarding, so here is the short version.

Teaching the archive to read

The footage is ingested into VideoDB, which gives each clip a chained analyzer graph. Spoken words and on-screen text (OCR) both feed a vision-language pass that describes each scene while reading what was said and printed during it. Fusion happens at ingestion, not as a merge of three searches at query time. The OCR is load-bearing: NASA footage carries burned-in lower-thirds and mission clocks, and where narration is sparse it is the only textual signal there is.

The result is five indexes: transcript (what was said), scene_semantic (what is visible), scene_facets (what kind of scene it is), ocr (what is printed on screen) and mission_meta (who, where and when). One vision-language pass backs two of them, so the second surface costs no extra inference.

Four ways to know a date

Publication date is useless for this archive, so every scene carries a date and a record of how it was known:

Source	Meaning	Trust
`scene`	the scene states a year, in speech or on screen	highest
`mission`	the stated year was impossible for the mission, so its operating window decided	corrected
`video`	the clip as a whole is anchored to an era	medium
`published`	NASA's publication date	always correct, rarely interesting

The correction matters. A compilation dated once at the top hands that year to every scene beneath it, and one Curiosity segment came back as 1990, twenty-one years before launch. Across the archive, 20.7% of dated scenes with a known mission fell outside its operating window, and 15.7% of the archive is re-dated. A date the scene stated itself is never overruled, because archive footage is full of retrospect.

Answering a question

A search box takes a query and returns clips. Ephemeris takes a question, decomposes it into sub-questions and alternate phrasings, and runs two gates before any retrieval: is the question answerable at all, and does the corpus hold the world it names. Then it retrieves from each index separately, joins runs of touching cells into passages, caps the evidence at two moments per clip, snaps each window to sentence boundaries using word-level timings, orders the result by the era each moment discusses, and asks the model for a cited answer with a chronology and caveats. Last, it compiles the reel.

Several of those choices were forced by measurement:

Query expansion is load-bearing. "Twin rovers landing in 2004" retrieves nothing, because the archive says "the first of two rovers". That phrasing hits rank 1 at 0.783.
Indexes are queried one at a time. Scores are not comparable across indexes, and a combined call dropped a 0.6899 transcript hit out of 20 results.
Diversity is enforced. Without a per-clip cap the chronology comes from one dense clip.
Refusing is a feature. Given gibberish, an early version split it into plausible sub-questions and answered with confidence. Asked about Venus, it once reasoned from Artemis reentry footage while conceding no clip showed Venus. There are zero Venus scenes, and the coverage gate now ends that run in 19 seconds with what the archive does cover.

Retrieval is checked against ground truth built from NASA's own published caption files, so the evaluation does not grade the pipeline on its own output: 35 of 36 cases, mean recall 0.909.

The scene that follows the reel

In the browser, one activeEvidenceIndex is the entire sync bus. Inline [n] markers, timeline needles, shot rows, the craft standing in the 3D scene and the reel's playhead all read from it, so clicking any one of them moves all of them. That single shared index is also why so much of the interface is testable: if one thing selects a moment, you can assert that the other four followed.

Making every pull request testable

The first change was architecture, not tests.

The app ran as one container, and the Next.js route spawned the Python agent as a child process. That rules out a preview deployment per pull request, and a pull request check should test the pull request's own code, not main. In PR #7 I split the Python agent into its own small HTTP service on Render's free tier, behind a shared token, and moved the web app to Vercel. The web route now validates, rate-limits, and forwards the question to the service, streaming the events back unchanged. Now every pull request gets a Vercel preview with its own URL, and that URL is what TestSprite tests.

Setting up TestSprite

This is what I actually did, in order.

1. Generate a suite from the live app. TestSprite explores the URL with several agents and proposes a test plan. Mine proposed 62 tests across 8 use cases, with a lot of repetition: "switch between presets" appeared five times and "chronology is in order" three. I trimmed it to 20. A pull request run costs roughly one credit per browser test, so a deliberate suite is also a cheaper one.

2. Write assertions about what a user can see. I uploaded a short product requirements document first, with the real labels, the preset names, and what counts as a pass. Then I edited the generated steps so each test passes for a reason. The reel test, for example, passes if the player plays, or says the stream is slow, or says the browser cannot play it, with a link; it fails only on a bare "loading" spinner forever.

3. Run it once before wiring it up. A trigger re-runs an existing suite. If the suite has never been generated and run, the trigger has nothing to start. Generate it, run it, and only then add the trigger.

4. Connect the GitHub App and add a trigger. The pull request trigger listens for the Vercel commit status, so it fires when the preview is live, not when the build starts. Its target URL pattern is the preview hostname with a branch placeholder, for example https://my-app-git-{branch-slug}-my-team.vercel.app. Then switch on the merge gate, so a red check blocks the merge.

5. Drive it from the command line. The TestSprite CLI lets me read a failure the same way I would read a stack trace. testsprite test failure summary <id> prints the failing step and a root-cause hypothesis. testsprite test plan put <id> replaces a test's steps from a file. That made the loop quick: read the failure, decide whether the app or the test is wrong, fix one of them, push.

6. Add an API project. A second TestSprite project plans its tests from docs/openapi.yaml and from probes of the live API. It started with 7 tests and grew to 51, and runs against production through a push trigger.

The pull request that proves it: PR #13

PR #13 is built so the history proves the tests can fail. I had written 26 new tests and ran them against production before touching any code. Two failed for real reasons. So the pull request opens with a commit that contains only a regression test, and the fixes come in the second commit.

Commit	Python CI	TestSprite browser suite
1: regression test only	red	16 of 18 pass
2: the fixes	green	18 of 18 pass

I left the red run in the history on purpose. A test you have never seen fail is a test you cannot fully trust, and a red run followed by a green one is the cheapest evidence that it is real. TestSprite rewrites its comment on the pull request as each run finishes, but the red run stays on the first commit's check, which links to its report.

Three bugs were behind the red:

The ask service corrupted the next request after a refusal. The service checks the auth token before it reads the request body, on purpose, so an unauthorised caller costs nothing. That meant a 401 went out with the JSON body still unread. On a connection that stays open, those bytes are parsed as the next request line, and a proxy that reuses upstream connections hands that failure to a different caller. TestSprite's API test sends three refusals in a row and hung on the second. The service now closes the connection after any refusal, and a Python regression test fails without that line. That test is why commit 1 turns Python CI red as well as TestSprite.

The Venus and Mercury panels said "reading corpus summary" forever. The 3D scene lets you focus either planet, but the corpus has no footage of them, so the summary file has no entry for those bodies. The panel could not tell "no footage" from "still loading". Now a loaded summary with no entry for that body says there is no footage of it.

The first question after the free Render host slept failed with a 502. The service sleeps when idle and takes about 40 seconds to wake, and the route gave up on the first 502. I had only ever tested it warm. The route now retries 502, 503, 504 and refused connections for up to 75 seconds, and the trace tells the visitor the host is starting up.

Climbing from 13 to 18: PR #8

PR #8 is the other one worth reading, because the suite went 13 of 18, 16 of 18, then 18 of 18 across three commits, and every step was a real fix.

The planet names could not be clicked. The names floating over the 3D scene were plain text sitting over an invisible picker sphere, so clicking a name did nothing. I turned them into buttons. The test then failed again, differently: the buttons were visible but no click reached them. I reproduced it in headless Chrome. The button was there, the click timed out, and elementFromPoint at its centre returned the wrapper, not the button. The label rule set pointer-events: none so floating names never block a drag on the scene, and the buttons inherited it. A more specific rule fixed it, and a direct check showed the body panel opening with its counts.

Typing stalled. Two tests were blocked because the page was too slow to type into. I measured it. On a browser drawing the 3D scene on the CPU, the orrery at device pixel ratio 1.75 cost 53 ms a frame, and every keystroke in the Ask box waited behind a frame, about 110 ms a key. The scene now checks once whether the renderer is software and, if so, draws at half resolution without antialiasing: 17 ms a frame, about 25 ms a key. I had never noticed it on a machine with a GPU.

A browser without H.264 sat on "loading stream" forever. The player now says it cannot play the reel, or that the stream is slow, and gives a link. A spinner that never ends is the worst of the three possible outcomes, and the test treats it as a failure.

The Asked list vanished when empty, so a first visit showed no sign of where runs go. An empty row now says so.

An over-limit question was refused only after submit. The page now counts characters live, disables submit and says "question too long".

The others

A value that vanished (PR #11). A long value set from code disappeared from the Ask box. It was a controlled React input, and React discards a value that arrives without an input event, which is how some assistive tools, password managers and test drivers fill a field. The box is now uncontrolled, and the length is read from the field on submit, so an over-limit question is refused however it got there.
A 500 where a 400 belonged (PR #16). /api/ask answered 500 with an empty body when the question was a number, an array, an object or a boolean, because the route called .trim() on whatever the JSON held. A visitor cannot cause it from the page, but any API client can. I wrote one API test per JSON type and ran them against production first, so the red run is in the portal's history. The route now answers 400 "question must be text", before the quota is touched.

A suite that grew from 24 to 85

I started with 24 tests. By the end there were 85, and the growth came from asking what each area promises:

Area	Tests	What is asserted
Browser, presets and answers	34 in total	each preset loads its own answer, chronology and reel; citations select their evidence row; the timeline ruler has a needle per moment
Browser, scene		focusing a body opens its panel; the camera strip flips from following the reel to manual orbit; hide notes clears the side panels and show notes restores them
Browser, live ask		one real question through the whole agent, saved to the browser, surviving a reload
API, preset files	51 in total	every preset is a complete result: eight evidence rows in ascending era order, one reel shot per row, citations that point at rows, a numbered trace, a reachable stream
API, refusals		invalid JSON, empty, whitespace-only, over-length and non-text questions each get the exact documented 400, as JSON, with no path or key in the body
API, ask service		health check; no token or a wrong token is a 401; no other routes
API, the site		the page contains no key names or service address

Alongside those, eight offline Python suites run in CI on every push, plus the browser-side checks. The one test that spends VideoDB credits is the live ask, and the visitor quota of five questions an hour bounds it.

What I took from it

Give every pull request a preview of its own. Splitting the agent into a service is the change that made a pull request check meaningful, because the tests ran against the pull request's code and not against main.
Write the regression test as its own first commit. Seeing the red run before the fix, as in PR #13, is how I know the test is real.
Assert what a user can see, and let the test fail on a missing explanation. "Plays, or says why not" is a stronger contract than "plays", because it forbids the silent spinner.
Measure before you fix. The 53 ms a frame came from a measurement, not a hunch, and the fix could be checked against it.
Run new tests against production before you fix anything. The red run is the evidence that the test would have caught the bug.
Keep branch names short. The Vercel preview hostname is built from the branch name and is limited to 63 characters. A longer name will not match the URL pattern the trigger expects, so the check never starts.

Copy this setup

Make sure every pull request gets its own preview URL, and that the app on it can run without a local backend.
Write a short requirements document with the real labels and what counts as a pass, and generate a suite from the live URL.
Trim duplicates, edit the steps, and run the suite once by hand.
Add a pull request trigger on the Vercel commit status, with the preview hostname as the target pattern, and turn the merge gate on.
Open a pull request that contains only a failing test, watch it go red, then add the fix and watch it go green.
Plan an API project from your OpenAPI file, and run new tests against production before you fix what they find.

I Built a Real-Time Multilingual Dubbing Platform and Used TestSprite MCP to Test It

Rahul — Fri, 17 Apr 2026 19:54:14 +0000

What if you could speak, and everyone listening heard you in their own language, with no noticeable delay?

That question turned into PolyDub.

What It Does

Three modes:

Live Broadcast: one speaker, listeners worldwide, each hearing a dubbed stream in their language
Multilingual Rooms: everyone speaks their own language, everyone hears everyone else in theirs
VOD Dubbing: upload a video, download a dubbed MP4 with SRT subtitles

The real-time pipeline:

Mic -> WebSocket -> Deepgram Nova-2 (STT) -> Google Translate (~300ms) -> Deepgram Aura-2 (TTS) -> Speaker

Perceived latency is around 1.2 to 1.5 seconds. Fast enough for a real conversation.

A Few Decisions Worth Explaining

Why Google Translate instead of Lingo.dev for real-time? Lingo.dev is LLM-based, which means 5 to 8 seconds of latency. Fine for batch work, not for live speech. Google's gtx endpoint runs at 250 to 350ms warm. Lingo.dev is still in the project, compiling UI strings at build time across 15 locales.

Why Deepgram Aura-2? Aura v1 only shipped English voices regardless of the language param. Aura-2 ships genuinely native-accent voices: Japanese prosody, Spanish regional variation, German intonation. Using an English voice mispronouncing another language defeats the entire product.

Why a per-listener TTS queue? In a room with multiple speakers, audio chunks from different people arrive at the same socket in parallel. Without serialization they interleave into noise. A per-socket promise chain fixes this, and the queue depth is capped at 1 so stale utterances get dropped rather than building an 8-second backlog.

Screenshots

Broadcast mode: pick source and target languages, hit Start, share the listener link.

Rooms: each participant sets their own language and voice. The server handles translation per-person.

VOD: upload a video, pick a language, get a dubbed MP4 and SRT file back.

Testing With TestSprite MCP

The project was built under hackathon pressure. Third-party APIs can fail in specific ways. Frontend validation is easy to break quietly. Writing full test coverage by hand would have eaten most of the remaining build time.

TestSprite MCP plugs into Claude Code as an MCP server. It reads the codebase, generates a test plan, and writes runnable test code. I ran it twice: once for a baseline, and again after a round of fixes.

Backend tests generated (5/5 passing):

Test	What it checks
TC001	`POST /api/dub` with valid file returns `{ srt, mp3 }`
TC002	`POST /api/dub` with missing params returns 400
TC003	`POST /api/dub` with broken third-party API returns 500
TC004	`POST /api/mux` with valid inputs returns `video/mp4` stream
TC005	`POST /api/mux` with missing inputs returns 400

The generated code is more thorough than what you'd write in a hurry. TC001 builds a minimal valid WAV file inline, validates the base64 response actually decodes, and checks the SRT string is non-empty:

mp3_bytes = base64.b64decode(json_data["mp3"], validate=True)
assert len(mp3_bytes) > 0
assert "srt" in json_data and len(json_data["srt"].strip()) > 0

Frontend tests generated (12 cases): broadcast start and validation, room create/join/leave/rejoin, language and voice change in-session, VOD upload validation, and landing-to-mode navigation flows.

What the first run caught:

/api/dub was returning a plain string in some error paths instead of a consistent JSON shape. TC003 found it.
The room ID field was letting through malformed IDs before hitting the server. TC009 found it.

Fixed both, reran, all clean. The dashboard keeps a full run history so you can diff before and after. That is the actual useful part: not a single passing run, but a record of what broke, what changed, and whether the fix held.

Running It

git clone https://github.com/your-username/polydub
cd polydub
pnpm install
cp .env.example .env
# set DEEPGRAM_API_KEY and LINGO_API_KEY in .env

pnpm dev     # terminal 1: Next.js on :3000
pnpm server  # terminal 2: WebSocket server on :8080

Github: https://github.com/crypticsaiyan/PolyDub

AutoFinance: Your AI Financial Command Center, right inside the terminal!

Rahul — Mon, 16 Feb 2026 06:22:07 +0000

This is a submission for the GitHub Copilot CLI Challenge

Why Leave the Terminal?

As developers, we live in the terminal. So why do we keep switching to 20 different browser tabs just to check market volatility, technical indicators, or portfolio drift?

I wanted to see if I could build a professional-grade financial research desk that lives exactly where I do. The result is AutoFinance: a heavy-duty orchestration of 13 specialized MCP (Model Context Protocol) servers that turn your command line into a high-speed financial command center.

Demo

The 13-Server Engine Room

To power these agents, I stood up 13 microservices using the Model Context Protocol. Each server handles a specific domain:

Port	Server	Responsibility
9001	Market	Real-time data via Yahoo Finance & Binance
9002	Risk	Automated trade validation and position sizing
9005	Technical	Crunches RSI, MACD, and Bollinger Bands
9006	Fundamental	P/E ratios, ROE, and growth metrics
9008	News	Local sentiment analysis on market headlines
9012	Simulation	Monte Carlo scenarios for stress testing

Key Features

📈 Real-Time Market Data: Connects directly to Yahoo Finance and Binance (via WebSockets) to stream live prices. No staleness—just raw, real-time data.
📡 Multi-Channel Global Notifications: Never miss a beat. The system pushes critical alerts to where you actually live:
- Slack & Discord: Get trade confirmations, price spikes, and agent debates in your channels.
- SMS & WhatsApp: Urgent margin calls or crash alerts sent directly to your phone.
- Email: Daily portfolio summaries and audit reports.
🛡️ Automated Risk Guardrails: The Risk Server enforces strict position limits (max 5%) and stop-losses automatically. It rejects any trade that violates your pre-set safety policy.
💬 Natural Language Interface: Deep copilot CLI integration lets you "chat" with your portfolio. "Hey, sell half my Bitcoin if RSI > 80."
📊 Comprehensive Analysis:
- Technical: Real-time RSI, MACD, and Bollinger Bands.
- Fundamental: P/E ratios, ROE, and growth metrics.
- Sentiment: Scans news headlines to gauge the current market mood.

How GitHub Copilot CLI Saved My Sanity

It helped me build this project in less than 36 hours!!

AutoFinance/
├── .github/                       # COPILOT BRAIN 🧠
│   ├── copilot-instructions.md    # The "Bible" (Context & Rules)
│   ├── mcp-config.json            # Local MCP Server Config
│   ├── agents/                    # Specialized Personas
│   │   ├── mcp-server-developer.md
│   │   ├── financial-analyst.md
│   │   ├── cli-dashboard-dev.md
│   │   └── test-engineer.md
│   ├── skills/                    # Reusable Knowledge
│   │   ├── financial-data-validation/
│   │   ├── mcp-server-debugging/
│   │   └── technical-analysis-impl/
│   ├── instructions/              # Path-Specific Context
│   │   ├── mcp-servers.md
│   │   └── tests.md
│   └── hooks/                     # Automation Scripts
│       └── hooks.json             # Pre/Post-execution logic
├── mcp-servers/                   # THE 13 MICROSERVICES
│   ├── market/                    # Yahoo Finance Integration
│   ├── technical/                 # RSI/MACD/Bollinger Logic
│   ├── sentiment/                 # NewsAPI + LLM Analyzer
│   ├── risk/                      # Compliance & Limits Engine
│   └── ... (10 others)
└── cli/                           # The Textual Dashboard UI

(I specifically organized .github/ to leverage Copilot's ability to read configuration files. This ensures every suggestion respects the project's architecture.)

Building a system with 13 microservices is not easy. Honestly, if I had to manually manage the boilerplate for every server, I’d still be writing imports.

Here is how Copilot CLI helped me:

1. The @mcp-server-developer Persona

I defined custom instructions so Copilot knew exactly how I wanted my MCP servers structured. When I needed the Volatility Server, I didn't write code; I just talked to my terminal:

copilot --agent=mcp-server-developer "Create a volatility server on port 9010 using our standard SSE transport."
It generated the Pydantic models and error handling that matched the rest of my stack perfectly.

2. Local MCP Integration

I configured Copilot CLI to actually talk to my running AutoFinance servers. By mapping my local tools in ~/.copilot/mcp-config.json, I could ask the CLI real-time questions:

Me: "What's the current RSI for NVDA?"
Copilot: "The current RSI for NVDA is 72.4 (Overbought). Recommendation: Wait for a pullback."
It bridged the gap between a coding assistant and a production console.

3. Living Documentation

Using .github/copilot-instructions.md, I gave Copilot the knowledge of the project—things like port ranges and "no hardcoded API keys" policy. This meant I never had to correct it on the small stuff. It just knew the architecture.

Screenshots

Landing UI

Dashboard

Setup in 30 Seconds

git clone https://github.com/crypticsaiyan/AutoFinance.git
pip install -r mcp-servers/requirements.txt
pip install -r cli/requirements.txt
./start_sse_servers.sh  # Fire up the 13-server swarm
python cli/main.py      # Launch the Command Center
# see the repo for more advanced setup

I really enjoyed and learnt a lot making this project (plus burnt a lot of credits too :P)

made by: crypticsaiyan
GitHub Repository: crypticsaiyan/AutoFinance

DEV Community: Rahul

Red, then green: 11 bugs TestSprite found in my NASA video research agent

The problem: an archive in the wrong order

What Ephemeris does

How it works, briefly

Teaching the archive to read

Four ways to know a date

Answering a question

The scene that follows the reel

Making every pull request testable

Setting up TestSprite

The pull request that proves it: PR #13

Climbing from 13 to 18: PR #8

The others

A suite that grew from 24 to 85

What I took from it

Copy this setup

Links

I Built a Real-Time Multilingual Dubbing Platform and Used TestSprite MCP to Test It

What It Does

A Few Decisions Worth Explaining

Screenshots

Testing With TestSprite MCP

Running It

AutoFinance: Your AI Financial Command Center, right inside the terminal!

Why Leave the Terminal?

Demo

The 13-Server Engine Room

Key Features

How GitHub Copilot CLI Saved My Sanity

1. The @mcp-server-developer Persona

2. Local MCP Integration

3. Living Documentation

Screenshots

Landing UI

Dashboard

Setup in 30 Seconds