Most AI tools fall apart at the same point: making sure the output is grounded in what's happening right now, not what the model was trained on months ago.
Bright Data built an open-source demo that solves this. It's called the Signal Terminal, a financial research tool built around that problem.
Why This Tool Exists
Meir Kadosh is an AI Engineer at Bright Data, and he kept noticing the same thing happening with financial customers. They would pull Google search results, pass them to an LLM, scrape the relevant pages for deeper context, and try to piece together why a stock had moved. The pipeline was almost identical every time. Different teams, same use case, everyone rebuilding from scratch.
So Meir built the canonical version — an open-source reference architecture that financial customers can use directly and developers can build on.
What Happens When You Ask a Question
Type "Why did Nvidia's stock drop today?" and the app runs a structured pipeline:
plan → search → scrape → extract → link → cluster → render
Each stage is visible and each source is attributed.
The planning stage comes first. The model generates specific search angles based on the question: what to look for, how recent, which categories of signals matter, things like macro events, earnings surprises, analyst actions, and news catalysts.
Then Bright Data's SERP API fires across Google and returns structured JSON for the most relevant sources in under a second. For pages that need deeper analysis, Bright Data's Web Unlocker fetches the full content as clean Markdown, handling the sites that block standard scrapers.
From there the LLM pulls named entities, causal relationships, and key figures from the content, those entities get connected into an evidence graph, related signals cluster into narrative themes, and everything surfaces in a live multi-panel dashboard.
The evidence map shows exactly which sources contributed which claims, how they connect to each other, and which narratives the evidence actually supports.
Two Paths, Depending on What You Need
Not every query needs the same depth.
The fast path uses SERP results only. Bright Data's SERP API returns structured JSON for each result (URL, description, position) and that passes directly to the LLM. Standard SERP delivers in 1-2 seconds, and with Bright Data's premium infrastructure, complete results come back in under one second. For real-time chat interfaces where latency is the whole point, this is the right path.
The deep path, which Meir calls "search and extract" internally, adds a second layer. After SERP identifies the relevant sources, Web Unlocker fetches each page's full content as clean Markdown, which then goes to the LLM for extraction and analysis. It takes longer, but for a full research run where completeness matters more than speed, that tradeoff makes sense.
The SERP route maps the incoming format param to Bright Data's format string and fires the request:
// app/api/serp/route.ts
const results = await brightDataSerpGoogle({
query: parsed.data.q,
format:
format === 'full' ? 'full_json_google' :
format === 'markdown' ? 'markdown' :
'light_json_google',
vertical, // 'web' | 'news'
recency, // 'h' | 'd' | 'w' | 'm' | 'y'
});
The token is BRIGHTDATA_API_TOKEN (with API_TOKEN as fallback). The SERP zone resolves through BRIGHTDATA_SERP_ZONE then BRIGHTDATA_SERP_ZONE_NAME, falling back to the Web Unlocker zone if neither is set. The Web Unlocker zone has its own resolution chain, defaulting to mcp_unlocker if nothing is configured.
Why Bright Data and Not Something Else
If you've built a scraper before, you already know how this story goes. It works locally. You deploy it. It gets blocked. You add proxy rotation and now some sites work, but LinkedIn doesn't, and Reddit doesn't, and when you look up how to handle TLS fingerprinting you're suddenly two weeks deep into a rabbit hole that has nothing to do with your actual product.
Getting clean data from LinkedIn, Reddit, X, and TikTok means solving years of adversarial engineering: TLS fingerprinting, browser fingerprinting, behavioral analysis, rotating CAPTCHAs. Web Unlocker handles all of that. The platforms that reliably block everything else.
Coverage spans seven major search engines and 195 countries with city-level targeting, so the results reflect what someone in a specific location actually sees.
The Part Most Developers Skip
There's a failure mode that shows up constantly in LLM pipelines built on web data. You fetch a full page, pass it to the model, and the output is worse than you expected. The instinct is to blame the model. Usually the model is fine. The context is the problem.
A full webpage is mostly noise. Navigation menus, cookie banners, author bios, related articles, advertisement slots. All of that lands in the context window alongside the three paragraphs that actually matter to your query. Flooding the model with irrelevant tokens makes the output worse, drives up token costs, and gives the model more surface area to hallucinate from.
The Signal Terminal handles this with a reflection stage that runs between fetching and analysing. Before the main analysis call, an extraction step strips everything irrelevant from the Markdown. Only the facts, named entities, causal statements, and figures directly related to the original query reach the analysis model.
Each stage has a dedicated prompt-builder function. The summarization stage looks like this:
// lib/prompts/signal-terminal.ts
export function buildSignalTerminalSummariesPrompt({ topic, evidenceExcerpts }) {
const system = [
'You are a market news summarizer for active traders.',
'Use only the provided excerpts; do not invent facts.',
'If an excerpt is low-signal (navigation/menus/price page), omit it.',
'Return strict JSON only.',
].join('\n');
const user = [
`Topic: ${topic}`,
'Evidence excerpts (JSON):',
JSON.stringify(evidenceExcerpts),
'',
'Return JSON: { items: [{ id, bullets: string[2..5], entities?, catalysts?, sentiment?, confidence? }] }',
'- Write for traders: prefer concrete nouns, numbers, and named actors.',
'- Sentiment: "bullish" | "bearish" | "mixed" | "neutral" — reflect the excerpt, not your opinion.',
'- Confidence: 0..1 based on excerpt specificity (dates/numbers/attribution increases confidence).',
].join('\n');
return { system, user };
}
Every stage has its own function like this: planning, impact graph expansion, chat. The model never gets a freeform prompt and a wall of raw text. It gets a tightly scoped system instruction, a JSON payload of curated evidence, and a strict output schema. The analysis model cannot fabricate beyond what it's been given.
Where Convex Comes In
Once Bright Data has acquired the evidence, it needs somewhere to go that can stream each pipeline stage to the UI as it completes, persist the results for follow-up queries, and stay fast enough that the interface actually feels live.
Meir’s first choice for this layer was Supabase. He switched to Convex because the latency difference in the real-time chat experience was noticeable enough to change the stack.
Relational databases have no built-in reactivity. If you want to know when something changes, you build it yourself — polling, pub/sub servers, WebSocket management layered on top of your existing stack.
Convex tracks the data dependencies of every active query, and the moment any of those dependencies change, it reruns the affected query and pushes the updated result to every subscribed client over a persistent WebSocket. For a pipeline that executes in sequential stages, you want the frontend to reflect each stage as it lands, not on the next poll cycle.
Beyond reactivity, Convex brought two things that would have required separate infrastructure elsewhere: full-text search built into the database, and a transactional scheduler for TTL cleanup. No cron jobs, no external queue, no search service.
Convex ends up serving three distinct roles in the Signal Terminal.
Role 1: Streaming Pipeline Events in Real Time
Each stage of the pipeline writes its results to Convex the moment it completes, and the frontend subscribes with a single useQuery hook.
// convex/schema.ts
export default defineSchema({
sessions: defineTable({
sessionId: v.string(),
topic: v.string(),
status: v.string(),
step: v.string(),
progress: v.float64(),
meta: v.optional(v.any()),
})
.index("by_sessionId", ["sessionId"])
.searchIndex("search_topic", { searchField: "topic" }),
sessionEvents: defineTable({
sessionId: v.string(),
type: v.string(),
payload: v.any(),
}).index("by_sessionId", ["sessionId"]),
});
The server writes an event every time a stage completes: search.partial, evidence, graph, clusters. The client sees it the moment it lands. No polling interval to tune, no missed transitions.
Role 2: Topic-Indexed Search for Follow-Up Chat (RAG Without Vectors)
After a research run completes, users can ask follow-up questions like "What did the Reuters article say about supply chain risk?" The app retrieves relevant past evidence without re-running the full Bright Data pipeline.
Convex handles this by indexing the topic field and enabling text-based search across historical sessions. No embeddings, no vector database, no additional infrastructure.
// convex/sessions.ts — search past sessions by topic keyword
export const list = query({
args: {
limit: v.optional(v.float64()),
status: v.optional(v.string()),
q: v.optional(v.string()),
},
handler: async (ctx, args) => {
const limit = (args.limit ?? 50) as number;
if (args.q && args.q.trim()) {
// Full-text search across the topic field via searchIndex
let results = await ctx.db
.query("sessions")
.withSearchIndex("search_topic", (q) => q.search("topic", args.q!.trim()))
.take(limit);
if (args.status) results = results.filter((s) => s.status === args.status);
return results;
}
let results = await ctx.db.query("sessions").order("desc").take(limit);
if (args.status) results = results.filter((s) => s.status === args.status);
return results;
},
});
No separate index on topic. Convex's searchIndex handles full-text retrieval.
The same mechanism that powers the follow-up chat also powers session history browsing. And because this is a Convex query, it's reactive: if a new session for the same topic completes while the user is in chat, the context updates automatically.
Role 3: Scheduled Cleanup
Convex's built-in scheduler handles cleanup without a cron job. When a session is created, a cleanup task is scheduled to run exactly 24 hours later.
// convex/sessions.ts — TTL scheduled at creation time
const TTL_MS = 24 * 60 * 60 * 1000;
export const create = mutation({
args: { sessionId: v.string(), topic: v.string(), /* ... */ },
handler: async (ctx, args) => {
await ctx.db.insert("sessions", args);
// Schedule cleanup for exactly this session, 24h from now
await ctx.scheduler.runAfter(TTL_MS, internal.sessions.deleteExpired, {
sessionId: args.sessionId,
});
},
});
export const deleteExpired = internalMutation({
args: { sessionId: v.string() },
handler: async (ctx, { sessionId }) => {
const session = await ctx.db
.query("sessions")
.withIndex("by_sessionId", (q) => q.eq("sessionId", sessionId))
.first();
if (session) await ctx.db.delete(session._id);
const events = await ctx.db
.query("sessionEvents")
.withIndex("by_sessionId", (q) => q.eq("sessionId", sessionId))
.collect();
for (const ev of events) await ctx.db.delete(ev._id);
},
});
Each session cleans up after itself. The scheduler fires once per session, 24 hours after creation, and deletes that session and all its events. Because scheduling from a mutation is atomic, if the session creation succeeds, the cleanup is guaranteed to be scheduled, and because it's a mutation, it's guaranteed to execute exactly once.
On the Model Choice
The app runs on Gemini 3.0 Flash via OpenRouter. The architecture supports full model configurability: different models per stage, per speed mode. Each stage (planning, summaries, artifacts, chat) can be independently configured with a base model, a fast variant, and a deep variant.
At runtime, the selection logic works down a priority chain: explicit caller override first, then the stage-specific fast or deep model, then the stage base, then the provider default.
// lib/ai/model-selector.ts
export function selectStageModel({ stage, mode, requestedModel }) {
const explicit = firstNonEmpty(requestedModel);
if (explicit) return explicit; // caller override wins
const profile = env.ai.openrouter;
const stageProfile = stageModels(profile, stage); // plan | summaries | artifacts | chat
const modeFallback = mode === 'fast' ? profile.modelFast : profile.modelDeep;
if (mode === 'fast') {
return firstNonEmpty(stageProfile.fast, stageProfile.base, modeFallback, profile.model);
}
return firstNonEmpty(stageProfile.deep, stageProfile.base, modeFallback, profile.model);
}
This matters in multi-stage pipelines because latency compounds. Planning, summarization, graph expansion, and chat all run in sequence, and a slow model at each step multiplies into a slow tool. A fast model at planning and summarization with a deeper model only for chat is a meaningful architectural choice.
What Bright Data and Convex Have in Common
Both solve the same class of problem in their respective layers. You could build what either one does yourself: proxy rotation and bot evasion on the data side, WebSocket management and pub/sub infrastructure on the persistence side.
Both are years of engineering work that exist so you don't have to do them. Bright Data delivers structured, fresh web data at the moment you ask for it. Convex delivers that data to every subscribed client the moment it lands in the database. The application code gets to be about the product, not the infrastructure.
The Architecture You Can Take With You
The Signal Terminal is a financial research tool, but the pattern applies anywhere you need an AI that reasons about what's happening now: competitive intelligence, news monitoring, lead research, price tracking, sentiment analysis, supply chain surveillance.
Bright Data handles the outside world — fresh, compliant web data at scale, with unlocking across platforms that block everything else. Convex handles the inside world: reactive persistence, streaming to the UI, querying without a vector database, housekeeping on a schedule. The LLM reasons only from what it's been given, because the reflection stage ensures it only receives what's relevant.
Try It, Fork It, Build With It
Go to demos.brightdata.com/market-terminal and ask it a real question about a company or market event you already know something about. Watch each pipeline stage complete. See the evidence map build. Ask a follow-up in the chat and see whether the output matches what you already know.
The code is at github.com/brightdata/market-terminal. Open source, contributions welcome.
The Convex community Discord is a strong resource for questions about the reactive database layer.
Thanks to Meir Kadosh at Bright Data for walking me through all of this and building something worth writing about.
Top comments (0)