DEV Community: Victor

The Failure My Quality Metrics Couldn't See: Testing the Router in Vicquant

Victor — Sat, 17 Oct 2026 14:45:49 +0000

This was called to my attention by one of my readers which tell me to check for the router path.
I really appreciate my readers which take their time to read, drop their suggestions and correct me to do the right thing. Thank you all...

The Short Version of how Vicquant answer financial question

Vicquant has two pathways to answer a financial question:

A RAG path (rag_vault) that searches the user's uploaded documents (such as a 401(k) statement or tax summary) and cites the exact source.

A chat path (fast_chat) that answers from the model's general financial training data.

A small classification heuristic (the query router ) decides which path each incoming question takes.

My quality run scores four metrics (including faithfulness and context recall) on 25 golden questions. Every one of those questions goes straight into the RAG path. So those four numbers describe RAG only after a question has already arrived there. They say nothing about whether the router picked the right path.

That was a blind spot. When I finally audited the router, I found that 18 out of 20 document-grounded questions never reached the document vault at all.

Why This Matters

Imagine a user uploads their 401(k) statement and asks: "What formula does my employer use to match my contributions?"

If routed to RAG: The system inspects the statement, extracts the matching tier, and cites page 2 of the document.

If routed to Chat: The model answers from general knowledge. It might sound completely confident, stating a standard 50% match up to 6% even though the employer's actual formula is 100% up to 4%.

In the second case, my quality metrics didn't drop at all, because the question never entered the RAG pipeline. The failure was completely invisible to my evaluation harness.

The Two Errors Do Not Cost the Same

The one sent to RAG and the one sent to Chat cost two different things which are;

Because the failure costs are completely lopsided, the router should lean heavily toward retrieval. When in doubt, the question must go to RAG.

The Metric I Added

Alongside latency benchmarks, I now track:

The share of document-grounded inquiries that mistakenly route to the chat path.

I also track the reverse (general inquiries routed to RAG) so the engineering trade-off remains transparent.

How the Check Was Run

A Clean Benchmark Set: 40 questions (20 document-grounded inquiries about a sample 401(k) statement and 20 general personal finance questions).
Pre-Labelled: The expected path for every question was locked in before running the benchmark.
Automated End-to-End Execution: A test user uploaded the sample document, dispatched each query through the API, and recorded the path taken directly from the streaming response metadata.

Baseline Results (Pre-Fix)

The results before the fix was carried out;

Why Did 18 Document Questions Miss the Vault?

Personal Possessive Pronouns (8 misses): Questions like "What formula does my employer use?" or "Who is listed as my primary beneficiary?" used personal references (my employer, my balance) instead of generic keywords. The router mistook them for general conversational chat.
Missing Document Nouns (4 misses): Questions referencing "the statement" or "my plan summary" lacked exact token matches in the old keyword list.
Unrecognized Plan Terminology (6 misses): Operational benefits terms (graded vesting schedule, in-service rollover, plan loan origination fee) had no heuristic triggers.

The Fix: "Retrieval by Default"

Instead of "send to RAG only if an explicit keyword matches," the architecture flipped to retrieval by default:

A query is sent to fast_chat only if it matches an unequivocal, positive general financial signal (such as "How does compound interest work?" or "Explain the 50/30/20 budget").
Any ambiguous, borderline, or pronoun-anchored question automatically defaults to rag_vault.
If a user has not uploaded any documents, the sub-millisecond retrieval check returns zero matching chunks, and the pipeline gracefully streams a standard answer without failing or injecting empty context.

Before and After: Validation Across Two Sets

Set 1: Benchmark Set (40 Questions)

Set 2: Fresh Evaluation Set (30 Unseen Questions, marked not independent)
To test generalizability against new phrasing, a second set of 30 questions (15 document, 15 general) was run once without tuning:

(Note: In a 15-question sample, 0 misses statistically rules out a true miss rate above roughly ~20%. A larger ongoing test corpus will narrow this confidence interval).

The Cost of Fallback

Local Compute: Negligible. Local hybrid BM25 and vector search takes 13.8 ms mean compute time.
Context Overhead: A fallback question incurs ~400 additional tokens of retrieved context, costing approximately +$0.000015 USD on OpenRouter.
The Asymmetric Trade-Off: Sending 73% of borderline general inquiries through the RAG pipeline is an intentional design choice. Paying 14 ms of compute and a fraction of a cent is incomparably better than allowing a participant's retirement question to reach a chat model that cannot see their statement.

What I Learned
A quality metric only measures what flows through it.

My four RAG evaluation scores were mathematically sound, yet completely blind to our most dangerous failure mode because the failure occurred before the pipeline began.

If your AI system has a router, a classifier, or an intent gate standing in front of your RAG pipeline, evaluate the gate first, and weight its errors by what they actually cost the user.

In your spare time, kindly check out vicquant(It is in the beta testing phase ) so i am in need of all the help i can get and send your feedback. So I can Improve it and get it to work better.
The Link: https://budget-planner-advisor-rho.vercel.app/

Thank you!

I Retracted My Latency Numbers. Here Are the Real Ones: 1,938 Requests Over 3 Days

Victor — Fri, 16 Oct 2026 12:29:57 +0000

A few days ago I published latency numbers for my AI finance app that were wrong. The headline was "all queries under one second." A reader asked what the benchmark actually measured, and my own audit revealed that 35 of the 75 data points had been simulated rather than measured end-to-end. I retracted the numbers immediately.

This is the replacement: measured through the real, live production app, across six benchmark sessions over three consecutive days, with zero simulated data.

The Short Version of what i worked on in the past few days

Typical case: Time to first token (TTFT) through the deployed app averaged 2.3 to 3.6 seconds (the median of each session). The exact same model call made directly to the inference provider took about 0.6 to 1.8 seconds.

The surprise: With several people using the app concurrently, it slowed down dramatically. At 5 simultaneous users, median time to first token jumped to 9 to 22 seconds.

What I can't explain yet: An overhead of about 1.9 seconds sits outside the model inference window, and the "fast" chat path was unexpectedly slower on average than the document-retrieval (RAG) path.

Earlier tests on my laptop against a remote database clocked ~13 seconds to first token. Removing four sequential database round trips and repairing missing HTTP streaming chunk headers eliminated that initial lag. Those older numbers came from a local environment and different code; the numbers below represent the audited, production-verified measurements.

How I Measured

Target Deployment: The live app hosted on Render, a Neon Postgres database in us-east-2, and meta-llama/llama-3.1-8b-instruct routed through OpenRouter using the production provider hierarchy (Groq first, then Novita, then DeepInfra).

Client Vantage Point: An automated streaming client on a mobile cellular broadband connection, connecting through the public HTTPS URL.

Metric Definitions:

Time to First Token (TTFT): Elapsed duration from client request transmission (t_client_send) until the first content token arrives at the client.

Completion: Elapsed duration from client request transmission to the arrival of the final token / stream termination.

Test Protocol: Six discrete sessions across three days (two sessions per day, October 13–15), with 323 requests per session (1,938 total requests):

120 steady-state requests on a warm, reused connection.
40 requests on fresh, newly established TCP/TLS connections.
30 direct OpenRouter requests across each of three provider policies (Default, Groq-excluded, Groq+Novita excluded).
60 concurrency load-test requests (20 each at $c=1, 3, 5$).
3 cold-start probes following server idle intervals.
10 baseline network probe pings.

Hygiene & Cost: Test accounts used reserved ephemeral domains (@bench.invalid) and were purged by exact ID from Neon at session conclusion (verified 0 residual rows). Total OpenRouter inference cost across all 1,938 requests was $0.0108 USD.

Result 1: The App Adds About 1.9 Seconds to the Raw Model Call

Note: that the difference column isn't all "app." Some of it is the extra hop from my laptop to Render, and the network probe (a lightweight endpoint, so it isn't a clean subtraction) suggests that hop costs a few hundred milliseconds or more on my connection. I did not trace the server side in these sessions, so I won't pin the rest on any single cause. My candidates are database round trips, request middleware, and streaming overhead, but those are guesses.

Two other patterns I can't explain yet:

Slower on the second and third day; Day 1 medians were 2.3 to 2.7 seconds, and days 2 and 3 were 3.0 to 3.6. The network probe didn't rise by a comparable amount. I'm checking whether the deployed code or the data changed between days.

The "fast" path was slower. The fast-chat path had a mean session median of about 3.4 seconds, and the document (RAG) path about 2.7 seconds. I expected the opposite. One would expect document retrieval, vector matching, and reranking to add latency, not decrease it.

Result 2: a fresh connection costs 60 to 714 ms
On a brand-new connection, first token arrived 60 to 714 ms later than on a reused connection, with a mean of about 330 ms. Browsers reuse connections, so most turns skip this, but a first-time visitor pays it. While browsers reuse active connections during an ongoing chat session, first-time visitors must absorb this initial handshake overhead.

Result 3: Groq-First Provider Order Delivers ~0.4s Faster TTFT; It is about 0.4 seconds faster to the first token

By isolating provider endpoints in our routing policy, we measured the upstream first-token latency differences:

Important Context: This test evaluated first-token arrival only. In full generation benchmarks, DeepInfra exhibits lower tokens-per-second output throughput, leading to substantially longer full-response completion times (~10–14 seconds). Provider selection involves a trade-off between instant token streaming initiation and sustained output throughput.

*Result 4: Concurrency Bottlenecks Emerge Quickly; running simultaneous users hurt a lot
*
Running concurrent client requests through the deployed app exposed an event loop / worker concurrency wall:

Median TTFT (seconds) across Concurrency Tiers (20 requests per tier per session):

With 3 clients the median first token was 1.6 to 3.9 times slower than with 1, and with 5 clients it was 2.7 to 4.4 times slower. That's roughly what you'd see if the server handled requests largely one after another: with k requests arriving together, the median wait is about (k+1)/2 times a single request, which is 2 times for 3 clients and 3 times for 5. It's consistent with that picture, but it doesn't prove it. I haven't measured whether something is blocking the event loop. Day 3 was worse at every level, and I don't know why.
For a small beta, a single user sees about 3 seconds. Three or more people chatting at the same moment is the current limit, and it's the first thing I'm fixing.

When multiple clients query the system concurrently, response times degrade in near-linear proportion to the concurrency level. A pattern characteristic of single-worker event loop blocking or synchronous I/O operations occurring within asynchronous route handlers.

Result 5: Cold Starts
The first request after an idle period took 5.4, 3.6, 3.2, 3.8, 5.8 and 6.6 seconds across the six sessions. Those are single runs, so I'm reporting ranges, not statistics. On day 2, session 2, the second and third requests were also slow (7.0 and 6.2 seconds), and I have no explanation for that.

Pooled Aggregate Percentiles (All 720 Steady-State App Requests)

Rather than evaluating session medians alone, pooling all 720 steady-state requests ($A1$) collected across identical code and conditions yields the following distribution:
p50 (Median): 2,842 ms
p90: 5,420 ms
p95: 7,140 ms
p99: 10,680 ms

What I Can and Cannot Claim

Network Specificity: All telemetry was measured from a remote cellular client connecting over public international routing to a US-East server deployment. Users on direct fiber or closer to the server host will experience lower transport and TLS handshake latencies.
Server-Side Dissection: Because server tracing was turned off during benchmark execution, the exact millisecond distribution between application middleware, template compilation, database session acquisition, and upstream proxy dispatch remains unverified.
Quality vs. Speed: These benchmarks isolate response velocity only. They make no assessment of response grounding, semantic relevance, or financial calculation accuracy.

What I Learned

Benchmark the User Path, Not the Bare Model: Direct model calls bypass the database, session handling, serialization, and network routing. Measuring the raw model told me how fast OpenRouter was, but told me nothing about the experience of someone using the app.

Concurrency Reveals What Single-User Testing Hides: A 2.8-second single-user response is usable; a 16-second response under modest concurrency is unacceptable. Concurrency profiling must be an integral component of testing from day one.

Transparency Builds Trust: Retracting incorrect numbers is painful, but publishing verified, unvarnished data is the only standard that matters.

What I'm doing next
1.Trace the server side to find where the extra 1.9 seconds goes, and why the fast path is slower than the document path.
2.Fix the concurrency behavior, then re-run this exact harness and publish before and after numbers.
3.Pooled percentiles with confidence intervals, grouped by network and code version.
4.Repeat the provider quality comparison with enough runs to see past the noise.
5.Tomorrow: a second article on a blind spot in my quality metrics which a reader . They measured the pipeline accurately and still missed that 18 of 20 document questions never reached it.

If you've tackled single-worker event loop concurrency bottlenecks in production FastAPI/Uvicorn applications, what was your primary culprit?

I Built an AI Financial Advisor. Now I Need You to Break It.

Victor — Thu, 15 Oct 2026 15:49:16 +0000

Vicquant is open for beta testers: pick a frontend design, run four short tasks, and tell me exactly what feels wrong.

The problem with asking "What do you think?"

When I first put Vicquant in front of beta testers, I asked the question every builder asks:
"What do you think?" The answers I got back were polite and vague. "Looks good." "Nice app."
Not one of them told me where someone got stuck, what confused them, or what they expected to happen and didn't.
That is my mistake, not theirs. Open-ended questions produce open-ended politeness.

So this time I am doing it differently: I am giving you specific things to do, a short way to report what
happened, and a say in how the app looks. If you have ever wanted to shape a product while it is still early, this is that chance.

What is Vicquant?

Vicquant is an AI-assisted personal finance and market-analysis app. It started as a small local chatbot that helped people plan debt repayment and savings, and it has grown into something much bigger. One rule shaped it from day one: the AI never does the arithmetic. All the
money math (balances, payoffs, risk numbers) is computed by normal, testable code, and the language model only explains the results. That way a language model's confident guess cannot quietly become your budget.

What you can try in the beta:

• AI Advisor and Budget: ask questions about debt, savings and spending in plain language.
• RAG Vault: upload your own financial documents and get answers with citations back to the source, so you can check the answer instead of trusting it.
• Stocks and Portfolio: track holdings and see risk measures such as Sharpe, Sortino, VaR and CVaR, without needing a finance degree to read them.
• Market regime detection and alerts: a statistical model that tries to tell calm markets from turbulent ones.
• Proof ledger: trading signals are recorded in a tamper-evident, hash-chained log with fixed rules set in advance, so results cannot be quietly rewritten after the fact. No win rate is shown until at least 30 signals have been resolved, because a handful of lucky calls proves
nothing.

Honest status: what is rough right now

A beta invitation is only useful if it is honest, so here is where things stand:
• The AI currently runs through a hosted model service rather than fully on your own device.
Running it locally again is on the roadmap, but it is not the current setup.
• The mobile web experience is not good enough. A native Android and iOS app is planned, so please judge the layout with that in mind.
• Vicquant gives information and analysis. It is not licensed financial advice, and nothing in it is a guarantee of returns.
• Some features are still being tuned, and speed can vary. If something feels slow, I want to know exactly when it happened.
• The frontend UI and UX design also needs working on. This is my first time on working on something of this scale.
Please, I need your honest review and contribution, so i can improve on it

Please use sample numbers. During the beta, enter made-up balances and debts rather than real account details.You can test everything that matters without exposing anything
personal.

Help me choose the design

I have three frontend designs and I do not want to choose alone. The feedback I got earlier was that the interface needs to feel easier for someone who is not a finance expert, so this is where your eyes matter most. The UI design is okay for laptop size screen but for the mobile, i am still working on it.

When you vote, do not only tell me which one you like. Tell me which one you understood fastest. In a finance app, "pretty" matters less than "I knew what to tap."

Your 15-minute mission

Four tasks. Do them in order, and write down anywhere you hesitated, even for a second.

How to send feedback (it takes two minutes)
After the four tasks, answer these five questions in the feedback form:

• Which design did you pick, and which one did you understand fastest?
• Where did you get stuck or confused? Name the screen.
• What did you expect to happen that did not?
• Did any AI answer feel wrong, vague or untrustworthy? Paste it if you can.
• Would you use this again? What single change would make you say yes?

Bug reports are gold. A screenshot, your device and browser, and one sentence about what you were doing is all I need.

Join the beta
If you are a developer, a student, a finance enthusiast, or simply someone who gets confused by money apps, you are exactly who I need. Beginners are especially welcome, because they
find the problems experts forgive.

• Beta link: https://budget-planner-advisor-rho.vercel.app/
• Reply here or message me with questions.

Thank you for reading, and for being willing to break something so it can be built better. Tell me what is wrong with it, and I will make sure the next version is better because of you.

I Measured My RAG Pipeline Honestly. It Was 40x Slower Than I Thought.

Victor — Sat, 10 Oct 2026 15:54:04 +0000

A few days ago I published the architecture behind Vicquant’s RAG Vault; a 9-stage retrieval pipeline built to ground financial AI answers in actual source documents, with strict citations, so a user asking “what’s my 401(k) contribution limit” gets an answer tied to a real document, not a model’s confident guess. The quality evaluation backed it up: +5.2% answer relevance, +16% context precision, +4% context recall over the simpler pipeline it replaced, with zero hallucinations measured on either version.

What I didn’t have yet was an honest measurement of how fast it actually was. This is the story of getting that number, not liking it, and fixing it properly.

The number that was true and meaningless

My first latency measurement showed the pipeline completing in about 4.56 milliseconds. That number was real, and completely useless, because the test suite that produced it had the network-dependent calls (the hosted embedding API, the LLM calls for HyDE and generation) running against mocked test doubles, not the real OpenRouter API. It measured the pipeline’s own logic, correctly, while saying nothing about what a user would actually experience.

So I built a live benchmark: the same 25-question golden evaluation set, run through the real pipeline against the real API, three times, with full per-stage timing.

Mean latency: 14.81 seconds. p95: 21.28 seconds.

That’s not a number you publish as a feature. That’s a number you fix.

Finding the actual levers

The full pipeline runs several sequential steps that each cost real time: HyDE generates a hypothetical document via an LLM call, multi-query expansion generates alternate phrasings via another LLM call, then retrieval, reranking, and a final generation call. Three separate cloud LLM calls, done carefully but done on every single query, including ones that had no reason to need any of it.

_The first and biggest fix was routing: _Most of what a financial advisor app gets asked; budgeting questions, “how am I doing this month,” general conversation, doesn’t reference an uploaded document at all. Only questions actually grounded in RAG Vault content need the full apparatus. I added a lightweight, sub-millisecond heuristic classifier to route document-grounded questions into the full pipeline and everything else into the existing fast direct-chat path. Tested against a realistic 75-query mix (25 document-grounded, 50 general), it classified with 100% accuracy at roughly 0.09 milliseconds of overhead — a rounding error compared to what it saves.

The second fix was concurrency: HyDE generation and multi-query expansion don’t depend on each other’s output, but they were running one after another. Running them with asyncio.gather instead cut roughly 20% off the document-grounded path on its own.

The third fix was streaming: Total response time matters, but felt response time matters more for whether something feels broken. Adding SSE token streaming meant the fast path now renders its first token in under a second, instead of a user staring at nothing until the full response assembles.

Combined, these three changes brought mean latency from 14.81s down to 6.24s, and median latency from 14.71s down to 3.61s. A real improvement. Still not where I wanted to land.

The variance that wasn’t random

Even after those fixes, something was off: identical questions, identical model, wildly different response times from one call to the next. It looked like noise. It wasn’t.

OpenRouter doesn’t always serve a given model from the same place. A model string like meta-llama/llama-3.1-8b-instruct can be fulfilled by several different upstream inference providers, and OpenRouter rotates between them per request. I added logging to capture which provider actually served every call, then ran twelve identical back-to-back requests with the exact same prompt and model string to see what would happen.

Three different providers answered across those twelve calls; Groq, Novita, and DeepInfra and the latency gap between them was stark:

DeepInfra was roughly 11x slower than Groq for identical work. That single provider, showing up unpredictably in the rotation, was the entire source of the long tail in my earlier benchmarks, the 2-to-14-second HyDE call spread I’d measured wasn’t jitter, it was DeepInfra showing up some of the time.

OpenRouter supports explicit provider ordering in the request body, so I pinned it: Groq first, Novita second, DeepInfra kept as a last-resort fallback (never removed entirely because availability matters more than shaving off a few hundred more milliseconds when both fast providers happen to be down).

Every one of the 75 test queries now completes in under one second. That’s a 97.5% reduction in mean latency from where this started.

The cost of pinning to faster providers was functionally zero about $0.000003 extra per query, something like three-thousandths of a cent. And at every single stage of this entire process, I re-ran the quality evaluation: faithfulness, answer relevance, context precision, and context recall moved by exactly 0.0000 throughout. None of these fixes touched what the pipeline retrieves or generates, only when the calls happen and which infrastructure answers them.

What this actually proves

Three things I’d rather have learned on purpose than by accident, written down for whoever reads this next:

A benchmark is only honest if it measures the thing a user actually experiences. Mocked-out network calls will always look fast, that’s not a benchmark, it’s a logic test wearing a benchmark’s clothes.

“Random” variance usually isn’t. The twelve-identical-calls test took a few minutes to write and turned an unexplained 6x latency spread into a fully understood, fully fixable root cause.

And the last one: speed and correctness aren’t actually in tension here, even though it’s tempting to assume they are. Every optimization in this piece made the pipeline faster without moving a single quality metric. The tradeoff I was bracing for, give up some grounding rigor for responsiveness and was never had to be made.

RAG Vault is done. Next up: what Vicquant needed a mobile-native rebuild to get right, and why a web wrapper wasn’t going to cut it.

Why a Financial Advisor App Needs a RAG Vault, Not Just a Chatbot

Victor — Wed, 07 Oct 2026 19:56:12 +0000

When I started building Vicquant, the core design principle was simple and non-negotiable: the AI never does the math. Every number and a debt payoff schedule, a backtested win rate, a risk metric is computed by deterministic code. The model's only job is to explain results in plain language, never to generate a number from scratch.

That principle protects users from one kind of failure: a model confidently inventing a wrong calculation. But it doesn't protect against a second, quieter failure mode, when a model confidently answering a factual question about a document it has never actually seen, using something that merely sounds plausible. If a user uploads a fee schedule and asks "what does this actually charge me for an overdraft," a generic chatbot answer isn't good enough. It needs to be grounded in that specific document, with a receipt.

That's what the RAG Vault is for, and getting it right turned out to be a deeper engineering problem than I expected.

*Why the naive version doesn't work for finance
*
A basic RAG implementation do split text into fixed-size chunks, embed them, retrieve the closest matches and works reasonably well for general text. It falls apart on financial documents for a specific reason: financial documents are full of tables, and a naive character-splitter slices through them without any awareness that it's destroying a row of numbers. Tax brackets, fee schedules, 10-K disclosures are all tabular, all fragile to blind splitting. The fix was layout-aware parsing that detects and preserves tables, headings, and structured sections as whole units rather than arbitrary text windows.

The chunk-size dilemma

Even with layout-aware parsing, there's a tension in how big each retrievable piece should be. Small chunks match a search query precisely but don't give the model enough surrounding context to answer well. Large chunks give plenty of context but are harder to match precisely against a specific question. Vicquant resolves this with hierarchical chunking: small "child" chunks (150-250 tokens) are what gets searched and matched, but each one is linked to a larger "parent" chunk (800-1200 tokens) that actually gets handed to the model once a match is found. Precision for retrieval, context for generation and it decoupled instead of traded off against each other.

Finding the right passage when the words don't match

Pure keyword or basic cosine-similarity search misses conceptual synonyms and also "drawdown mitigation" and "capital preservation" mean roughly the same thing to a person asking a question, but not to a system doing literal string or frequency matching. The fix is dense neural embeddings combined with traditional keyword search (BM25) in a hybrid approach, fused using Reciprocal Rank Fusion(RRF) rather than an arbitrary weighted average. RRF is scale-invariant, so it doesn't break when one search method's scores happen to run on a different numeric range than the other's.

On top of that sits a cross-encoder reranker: a second-pass model that looks at the actual query and each candidate passage together, rather than comparing pre-computed embeddings in isolation. This catches token-level relevance that embedding similarity alone misses.

Where the model's attention actually goes

One detail that's easy to miss: large language models don't pay uniform attention across a long context window — information placed in the middle tends to get under-weighted compared to the beginning and end (a well-documented effect sometimes called "lost in the middle"). So the most relevant retrieved passages are deliberately placed at the start and end of what gets sent to the model, not simply stacked in ranked order.

Grounding, not just retrieval

The last piece is making every answer auditable. Vicquant's citation format isn't a vague "according to the document". it's a strict, deterministic token: [Citation ID: doc_id:chunk_id], pointing at the exact passage that produced the claim. For a financial app, this matters for a reason beyond user trust: it's the difference between an answer someone can verify and an answer they just have to believe.

What the evaluation actually showed

I ran the new pipeline against the old one on a 25-question financial benchmark, split across numeric lookups, conceptual synonym matches, and multi-table extraction, scored using RAGAS metrics:

Metric Legacy pipeline New pipeline Change
Faithfulness 100% 100% unchanged
Answer Relevance 78.7% 83.9% +5.2%
Context Precision 34.0% 50.0% +16.0%
Context Recall 92.0% 96.0% +4.0%

_Faithfulness _— the measure of whether claims are actually grounded in retrieved context rather than hallucinated, it was already perfect on the old pipeline and stayed perfect. That's expected; faithfulness is downstream of the generation step, not the retrieval architecture. The real gains are in Context Precision, up 16 points and the new pipeline pulls in noticeably less irrelevant "distractor" content, largely thanks to the cross-encoder reranking stage filtering out near-misses before they reach the model.

I also stress-tested the resilience claims directly rather than assuming them: deliberately failing the hosted embedding API confirms the circuit breaker trips and the pipeline falls back to the local offline embedder with zero unhandled exceptions; feeding in a document containing an embedded instruction ("SYSTEM OVERRIDE: ignore all prior instructions") confirms that text stays encapsulated as inert data rather than influencing the model's behavior.

One honest caveat on performance: the pipeline's internal processing overhead measured in testing is small, but that number reflects the pipeline's own logic with the network-dependent components (the hosted embedding call, HyDE's LLM call) running against test doubles, not live API latency. Real-world response time will be dominated by those actual network calls to OpenRouter, which I haven't benchmarked against the live API yet and that's the next thing to measure honestly before making any speed claims to users.

Why build a financial advisor this way at all

The short version: the worst thing an AI assistant can do with someone's money isn't getting something wrong. It's getting something wrong confidently, with no way for the person to check. Every design decision in Vicquant, from "the AI never computes numbers" to "every RAG answer carries a citation," traces back to that one principle. It's slower to build this way. I think it's the only responsible way to build it.

Building a Financial App That Works With Zero Internet (and Doesn't Lie About It)

Victor — Tue, 06 Oct 2026 15:03:40 +0000

Most personal-finance and trading apps make one quiet assumption: you always have a fast, stable connection to their servers. If you don't, you get a spinner, a blank screen, or worse, a stale number presented as if it were live.

I'm building Vicquant, a financial advisor and quant-analytics app that flips that assumption. The architecture offline design to runs the local AI model on everyone device for budgeting and debt planning, no internet required and layers in live market data, quant signals, and backtesting when you do have a connection. This article covers the engineering behind making that hybrid model actually reliable, not just "works most of the time."

The core problem

Once you decide an app must work offline and online, you inherit a new category of bugs that a purely cloud app never has to think about:

What happens when the network drops mid-request?
What happens when a free-tier data API rate-limits you?
What happens when a provider is down for an hour?
How do you show data from an hour ago without pretending it's from now?

None of these are exotic problems. They're just usually ignored, because "assume the internet works" is the default.

The first Design Decision: Python computes, the AI explains

Small local language models are surprisingly capable at conversation, but they are unreliable at arithmetic off-by-one errors, dropped decimal places, confidently wrong totals. For a finance app, a wrong number is worse than no number.

So every calculation needs a debt payoff schedules, savings projections, risk metrics and it runs in deterministic Python code first. The model's only job is to explain the result in plain language. It never generates a number from scratch. This single rule eliminates an entire class of trust-destroying bugs.

Design decision 2: every external call assumes failure

The data layer is built around a simple idea: assume every network call will eventually fail, and design for what happens next, not just the happy path. Concretely, that means four pieces working together:

Retry with backoff and jitter; A single timeout shouldn't be treated as a real failure. It's retried with an increasing delay, plus a little randomness (jitter) so many clients don't all retry at the exact same moment and overwhelm the provider again.

A circuit breaker per data provider; If a provider fails repeatedly, the app stops calling it for a cool-down period instead of retrying forever and wasting time on a dead endpoint. After the cool-down, one trial call checks if it's healthy again.

A rate limiter per provider; Free-tier market-data APIs have strict call limits. A simple per-provider minimum interval between calls keeps you under the limit without needing to track quotas manually.

A provider fallback chain; If provider A is down, the app automatically tries provider B, then C. The rest of the app never knows or cares which one actually answered, it just gets a quote back, or a clear failure.

def get_quote(self, symbol, target_currency=None):
    if self._connectivity.is_online():
        quote = self._try_providers(symbol)   # walks the fallback chain
        if quote is not None:
            self._cache.put(f"quote:{symbol}", quote.to_dict())
            return self._maybe_convert(quote, target_currency)
    return self._serve_from_cache(f"quote:{symbol}", symbol, target_currency)

Design decision 3: never hide staleness

When every provider fails and the app falls back to cache, that data gets returned with stale=True and an age_seconds field. The UI is required to show a "last updated 3 hours ago" badge whenever that flag is set. This is a small technical detail with an outsized trust impact, a user who sees an honest "this is old" beats a user who later discovers a number quietly lied to them.

Design decision 4: not every error deserves a retry

A "symbol not found" error and a "server timed out" error look similar on the surface but mean opposite things. One is a permanent fact; the other is transient. The error hierarchy distinguishes them explicitly, so a typo'd ticker fails fast instead of retrying three times against every provider in the fallback chain for no reason.

class ProviderError(DataLayerError):
    """Retryable: timeout, 5xx, temporary provider trouble."""

class SymbolNotFound(DataLayerError):
    """Not retryable: the symbol simply doesn't exist."""

I'm currently building the backtesting engine, every quant signal has to prove itself against two years of historical data, with realistic transaction costs, before it's allowed to appear live in the app.

The Architecture I Built vs. The Architecture I Could Deploy

Victor — Sun, 04 Oct 2026 09:22:06 +0000

I participated in a competition and it gave me an insight into something interesting and it led me to build Vicquant. An offline financial advsior app but when i was building it, i ran into a problem and and this aid me in solving real world problem and be innovative for the time being.

Vicquant's core idea is a dual local-model setup with zero cloud dependency: Llama-3.1-8B-Instruct for deep financial reasoning, Phi-3.5-mini as a fast, low-RAM fallback, both running through llama.cpp directly on the user's device. I will explain the reason for the split in an article soon, the short version is that reasoning quality and response speed pull in opposite directions, and one model can't optimize for both.

That design is real. The code exists, the model-routing logic exists, and it passed testing on an 8GB RAM / 4 vCPU target at roughly 12-15 tokens/second.

What I could actually deploy

Here's the part that doesn't usually make it into these posts: running an 8B-parameter model needs meaningful RAM and CPU, and the PaaS tier I'm currently hosting on have a Pro plan available which i haven't yet purchased. So the version of Vicquant live right now cannot run the local models in production.

Rather than leave the deployment broken or ship nothing, I connected it to OpenRouter so the app can serve AI responses today. Practically, this means: the deployed demo currently requires internet and routes model calls through OpenRouter. Offline mode is off for this deployment. The local-inference architecture isn't gone, it's in the codebase, tested, and it comes back as soon as i can pay for the hosting plans. But I'd rather say that directly than let a live demo imply a capability it isn't currently providing.

Why I'm writing this instead of hiding it

Two reasons.

First, credibility compounds. If someone inspects network requests on the live demo and finds a mismatch with what I claimed, every other technical claim I've made gets discounted too; the backtesting gate, the test suite, all of it. Saying the true thing now costs me one slightly less impressive post. Saying the false thing costs trust I can't easily rebuild.

Second, this constraint is itself worth writing about. The gap between "architecture you design" and "architecture you can afford to deploy on day one" is a real, common problem, and it's one most build-in-public content glosses over in favor of a clean narrative. The clean narrative isn't always the honest one.

What's unaffected by this

The things that don't depend on which model is answering:

147/147 tests passing across indicators (SMA, EMA, RSI, MACD, Bollinger, ATR), the signal engine (candlestick patterns, z-score mean reversion, momentum), and the backtesting engine (2-year minimum history, walk-forward validation, costs and slippage included).

The AI never computes anything. Every number comes from deterministic Python. Whether the explanation comes from a local Phi-3.5 model or an OpenRouter-hosted model, its job is identical: describe an already-correct number in plain language, never generate one.

The backtest gate. No signal reaches a user without proving itself against two years of historical data first, regardless of which model is running.

What's next

Local inference returns once the hosting constraint lifts. I'll post that update when it happens, not before. Thanks for reading

Nebula Agent: An AI That Surfaces Contradictions in Structured Content

Victor — Sun, 27 Sep 2026 10:27:07 +0000

This is a submission for the Sanity Challenge, Path One: Ship an Agent That Queries Real Content

What I Built

A Sanity-powered AI agent that answers questions about a fictional board game called Nebula and, crucially, surfaces contradictions between sources.

The agent doesn't just search for keywords. It writes its own GROQ queries(Graph Relational Object Queries) against a Sanity Content Lake in real time, reads the structured articles it retrieves, and when two sources disagree and presents both claims side by side with their source URLs.

The core question:

"According to the Nebula rulebook and errata, how many energy tokens do players start with? Are there any contradictions?"

The agent's answer:

The sources provide conflicting information:

Nebula Core Rulebook - Setup states players start with 5 energy tokens (source)

Nebula Official Errata v2.1 corrects this to 8 energy tokens (source)

A keyword search would have returned both articles. It would not have known which was authoritative, nor that the errata corrects the rulebook. My agent does this because the content was structured.

Demo

Since the agent runs in the terminal, here's a live transcript from two consecutive runs.

Question 1_Energy tokens:

text
👤 User: According to the Nebula rulebook and errata, how many energy tokens
   do players start with? Are there any contradictions?

🤖 Agent using tool: query_documents
   Args: {"query":"*[_type match \"*rule*\" || _type match \"*errata*\" || lower(title) match \"*rule*\" || lower(title) match \"*errata*\"]"}
   Result preview: [{"_type":"article","body":"To win Nebula, you must collect 10 stars..."}]

🤖 Agent: According to the Nebula rulebook and errata, there is a contradiction
regarding how many energy tokens players start with:

* Nebula Core Rulebook - Setup: States that each player starts with 5 energy
  tokens. Source: https://example.com/nebula-rulebook

* Nebula Official Errata v2.1: States that the core rulebook is incorrect and
  that the correct starting amount is 8 energy tokens.
  Source: https://example.com/nebula-errata

Question 2_Win condition:

👤 User: What is the win condition for Nebula?

🤖 Agent using tool: query_documents
   Args: {"query":"*[_type == \"article\" && (title match \"*Nebula*\" || body match \"*Nebula*\")]{title, body, source}"}
   Result preview: [{"body":"Many players struggle with energy management..."}]

🤖 Agent: The sources provide conflicting information regarding the win
condition for Nebula:

* According to the Nebula Core Rulebook and the strategy guide, you must
  collect 10 stars to win (Rulebook, Strategy Guide).

* According to the Nebula Official Errata v2.1, the win condition was updated
  to collecting 12 stars (Errata).

Code

Vicarioy / sanity-nebula-agent

Nebula Sanity Agent

A command-line agent that uses Gemini to answer questions from article content stored in Sanity.

How It Works

The agent checks its Sanity connection, sends a built-in question about Nebula's win condition to Gemini, and answers using Sanity query results. It can query documents with GROQ or list the dataset's document types and fields. Its instructions require source URLs for claims and ask it to show contradictory sources side by side.

The command prints the question, connection status, tool name, a result preview, and the final answer. Query arguments are not printed. Errors are written to stderr.

Stack

Node.js with ES modules
Gemini model gemini-3.5-flash-lite
Sanity project uyvc8si1, dataset production
GROQ for Sanity queries

Setup

Run these commands from the agent directory:

npm install

Create an .env file in the agent directory with both keys:

GEMINI_API_KEY=your-gemini-api-key
SANITY_API_TOKEN=your-sanity-api-token

The Sanity token must have…

View on GitHub

Stack:

Node.js

Gemini 3.5 Flash Lite (free tier)

@sanity/client for GROQ queries

@google/genai for function calling

How i used Sanity

** Structured Content Model**
i defined a single document type in sanity studio using typescript:

export const article = defineType({
  name: 'article',
  title: 'Article',
  type: 'document',
  fields: [
    defineField({ name: 'title',  title: 'Title',  type: 'string' }),
    defineField({ name: 'slug',   title: 'Slug',   type: 'slug', options: { source: 'title' } }),
    defineField({ name: 'body',   title: 'Body',   type: 'text' }),
    defineField({ name: 'source', title: 'Source', type: 'url' }),
  ],
})

I populated it with three documents that set up a deliberate contradiction:

The title and the source for each of the documents are:
Nebula Core Rulebook; The Setup - https://example.com/nebula-rulebook

Nebula Official Errata - https://example.com/nebula-errata

How to Win at Nebula - https://example.com/nebula-strategy

(The source URLs are placeholder examples. In a real deployment, this field would point to the actual publisher page.)

How the agents read and queries Content

The agent uses function calling with Gemini. It has one primary tool and it was coded out with javascript:

{
  name: "query_documents",
  description: "Run a GROQ query against the Sanity dataset. Returns JSON.",
  parameters: {
    type: "object",
    properties: {
      query: { type: "string", description: "A GROQ query string." }
    },
    required: ["query"]
  }
}

When the User asks a question, the LLM:

Writes a GROQ query itself: an example- *[_type == "article"]{title, body, source}
My code executes it via @sanity/client against project uyvc8sil, dataset production
The JSON result is fed back into the LLM's Context
The LLM reads the content and answers with citations

The system prompt tells the model:
"When two sources contradict each other, show both claims side by side with their sources. Cite the source URL for every claim. Never invent information."

What the agent does with retrieved content

Because the content is structured, the agent can do the following:

Compare specific fields across documents; it reads body from the rulebook and body from the errata and notices the numeric values differ.
Attribute each claim to its source URL; the citations aren't guessed, they're data.
Distinguish types of documents; the errata is authoritative over the rulebook because its title says so.

A plain-text search would have surfaced both documents but had no way to reason about which was newer, more authoritative, or even that they contradicted each other.

Designs decisions that mattered

Schema-informed prompting; The LLM initially guessed type names like card and document, returning empty results. I injected a schema description into the system prompt so it knows to query *[_type == "article"]. This is a small detail that helps to dramatically improved reliability.
Sanity as the source of truth(where the articles are confirmed from); No content is hardcoded in the agent. Every fact the agent reports comes from a live GROQ query against the Content Lake. If I update an article in the Studio, the next question gets the updated answer.
useCdn(false); I disabled the CDN for the agent so responses are always fresh from the Content Lake and always 100% Real Time. This matters for an agent where "the latest errata" is the whole point.

The Sanity Project Details

Project ID: uyvc8sil
Dataset: production
Document type: article (fields: title, slug, body, source)

What I Learned

I learned a lot because this is my first time of hearing about sanity. I learned how to made the case for structured content viscerally clear. So this provide me an opportunity that an LLM can reason about provenance when we have fields like title, body, and source defined in a schema. This led me to understand the difference between these statements "here are two documents" and "these two sources disagree, and here's the URL for each claim".

The hardest part wasn't the LLM logic, it was teaching the agent the schema. Once I stopped trying to make the model guess type names and instead told it was in the content Lake, everything clicked.

Thanks

Thanks to Sanity and DEV for the challenge. It pushed me to build something I'd been wanting to try: an agent that treats structured content as the foundation, not an afterthought.

Correlation vs. Causation: What They Actually Mean

Victor — Mon, 14 Sep 2026 18:48:49 +0000

" Correlation isn't causation" is one of the most repeated phrases in data science and one of the least explained. Here's what each term actually means, why the confusion happens, and four specific ways it goes wrong in practice.

If you've spent any time around data, you've heard "correlation doesn't imply causation" so many times it's become background noise. But if you stopped someone and asked them to define correlation and causation separately, in plain words, a lot of people would struggle. So let's start there, before touching any of the traps.

What Is Correlation?
Correlation is simply a measure of how two things tend to move together. If one goes up and the other usually goes up too, that's a positive correlation. If one goes up while the other tends to go down, that's a negative correlation. If they seem to have nothing to do with each other, that's little to no correlation.

A simple, real example: height and shoe size are correlated. Taller people tend to have bigger feet. That's a positive correlation, and it's intuitive, but notice we haven't said anything yet about why.

Correlation is purely descriptive: it tells you two variables tend to change together, and roughly how strongly, using a number called the correlation coefficient (often written as r), which ranges from -1 (perfectly opposite) to +1 (perfectly together), with 0 meaning no relationship at all. That's the entire job of correlation: describe a pattern. It has no opinion on what's causing that pattern, or whether anything is causing it at all.

What Is Causation?
Causation is a stronger, different claim: it means one variable directly produces a change in another. There's a mechanism connecting them, and if you intervened and changed the first variable, and only that variable, the second one would change as a result.

Take a simple example: turning up your oven's temperature causes your food to cook faster. If you change the temperature, cooking time changes because of that action. There's a direct, physical mechanism (more heat transfers energy faster), and you can test it by changing nothing else and watching the result change predictably.

A useful mental test for causation: "If I could reach in and change only this one thing, would the other thing change as a result?" If yes, that's a causal relationship. If you can't actually intervene, or if changing one doesn't reliably move the other, you're probably just looking at correlation.

Why We Keep Mixing Them Up
Human brains are pattern-matching machines; we're wired to notice when two things happen together and jump to "one must be causing the other," because that instinct was useful for survival long before it was useful for data analysis. The problem is that correlation is easy to measure (a single formula gives you a number) while causation is hard to establish (it usually requires careful experiments or reasoning about mechanisms). So we default to the easy signal and treat it like the hard one.

This is exactly where the four classic traps come in; they're the specific ways a real, measurable correlation can exist without any real causation behind it.

Confounding Variables (The Third-Variable Problem)

This happens when a hidden third factor is actually driving both variables you're looking at, making them look connected to each other when they're really both just reacting independently to something else.

The classic case: ice cream sales and shark attacks both rise in summer. It's not that ice cream causes attacks, or attacks cause ice cream sales warmer weather (the confounder) independently drives more people to buy ice cream and more people into the ocean at the same time. The confounder never shows up if you only look at the two headline variables.

In real analysis work, common hidden confounders include location, income, age, and season they quietly explain relationships that look meaningful on the surface.

Reverse Causality
Sometimes the relationship is genuinely causal, just pointing the opposite direction from what you assumed. A model might show "customers who contact support more often churn more," tempting you to conclude support interactions drive churn.

Just as plausible: customers who are already unhappy and about to leave contact support more because they're frustrated. The outcome is quietly driving the presumed cause.

Reverse causality is sneaky because the correlation itself carries no signal about which direction is correct you need outside knowledge, timing data, or a controlled study to untangle it.

Coincidence (Spurious Correlations)
Given enough variables and enough time, some completely unrelated trends will line up purely by chance. This is the trap behind the famous "spurious correlations" examples; like per-capita cheese consumption tracking almost perfectly with the number of people who died tangled in their bedsheets. There is no mechanism connecting them at all. With enough random series being compared, some pair will always match closely just by luck which is why testing dozens of variables against each other without a hypothesis ("data dredging") is such a well-known danger in statistics.

Selection Bias
This trap comes from how the data was collected, not from the variables themselves. If your sample isn't representative of the population you're trying to understand, you can manufacture a statistical link that isn't real or hide one that is.

A classic case: surveying only customers who already responded to a marketing email makes that email look more effective than it is, because people who never open emails are systematically missing from the data. The relationship you're seeing was baked in by how the sample was built, not by anything the two variables are doing to each other.

So How Do You Actually Prove Causation?
If correlation alone can't do it, what can? A few real tools researchers and data teams use:
● Randomized controlled trials (RCTs): randomly split a group in two, change one variable for one group only, and compare outcomes. Randomization cancels out confounders on average, so any difference in outcome can be attributed to the variable you changed.
● Natural experiments: when you can't randomize, look for situations where something close to random chance assigned the variable anyway (like a policy that took effect in one region but not a neighboring one).
● Statistical controls: _techniques like propensity score matching or regression with control variables that try to account for known confounders mathematically, when an experiment isn't possible.
● _A believable mechanism: even strong statistical evidence is more convincing when there's a plausible, explainable reason why one thing would cause the other.

A Quick Checklist Before You Claim Causation

● Could a third factor explain both variables?
● Could the arrow of cause and effect be pointing the other way?
● How many other variables did I compare before I found this one?
● Does my sample actually represent the population I'm making claims about?
● Do I have experimental evidence, or just an observed pattern?

Why This Matters
None of this means correlation is useless a strong correlation is a great lead. It tells you where to look. The mistake isn't noticing the pattern; it's stopping there and calling it an explanation. Treat correlation as the start of an investigation, not the end of one.

"""Demonstrating why correlation does not prove causation in marketing.

Scenario:
- A company spends more on ads during the holiday season.
- Customers also buy more during the holiday season.
- Holiday demand causes both variables to increase.
- Marketing spend has no causal effect in this simulation.
"""

import numpy as np

rng = np.random.default_rng(42)
months = np.arange(24)
holiday_season = (months % 12 >= 9).astype(float)

# Holiday demand is a hidden third variable (a confounder).
marketing_spend = 10 + 20 * holiday_season + rng.normal(0, 2, size=24)
sales = 100 + 80 * holiday_season + rng.normal(0, 5, size=24)

correlation = np.corrcoef(marketing_spend, sales)[0, 1]

print(f"Correlation between marketing spend and sales: {correlation:.2f}")
print("A high correlation alone does not prove that marketing caused the sales.")
print("In this simulation, the true causal effect of marketing spend is zero.")

# Once we compare months within the same season, the misleading relationship
# largely disappears because the hidden seasonal factor is held constant.
non_holiday = holiday_season == 0
holiday = holiday_season == 1

non_holiday_correlation = np.corrcoef(marketing_spend[non_holiday], sales[non_holiday])[
    0, 1
]
holiday_correlation = np.corrcoef(marketing_spend[holiday], sales[holiday])[0, 1]

print(f"Correlation in non-holiday months: {non_holiday_correlation:.2f}")
print(f"Correlation in holiday months: {holiday_correlation:.2f}")
print("To test causation, use a randomized marketing experiment or A/B test.")

HealthBridge: Engineering an Offline, CPU-Only LLM to Bridge Nigeria’s 1:4,000 Doctor-to-Patient Gap

Victor — Sun, 06 Sep 2026 16:12:47 +0000

This is a submission for Weekend Challenge: Generosity Edition

What I Built

I built HealthBridge; an offline, CPU-only AI assistant designed to tackle Nigeria's severe doctor shortage and connectivity barriers.

Specifically, here is exactly what I delivered:

A fully offline LLM-powered health education tool
I packaged a quantized large language model (Qwen2.5-1.5B-Instruct in GGUF Q4_K_M) so it runs entirely on a standard laptop's CPU. No internet, no GPU, and no recurring cloud costs making it viable for rural and peri-urban Nigerian communities where connectivity is spotty or unaffordable.
A triage and patient education interface
I built an interactive assistant (working through the Streamlit interface challenges you saw in the report) that does four specific jobs for community health workers, patients, and caregivers:

. Explains common symptoms in plain language.

. Describes basic treatments (e.g., oral rehydration therapy for cholera/diarrhea).

. Guides users on when to urgently seek hospital care (red-flag triage).

. Actively steers people away from seeking advice from unqualified friends, neighbours, or unregistered local pharmacies.

Crucially, it is strictly scoped as a complement to professionals; it never claims to replace clinical diagnosis.

A lightweight, resource-optimized runtime
I chose the llama.cpp runtime and carefully picked the Q4_K_M quantization to hit the sweet spot: coherent, medically-contextual responses while fitting comfortably inside an 8GB RAM budget (peaks at just 1.7 GB during my tests). The model runs at 16.76 tokens/second on integrated graphics; no discrete GPU required.
A zero-dependency deployment pipeline
I set up a download_model.sh script to fetch the model via Hugging Face, but after that initial download, the entire inference pipeline runs completely offline. I ensured the model file itself isn't committed to git, so evaluators can pull it fresh and run it immediately on the ADTC standard laptop (10th–12th gen i5, 8GB DDR4, Ubuntu 22.04).

In short: I built a portable, cost-free, offline health-information kiosk-in-a-laptop that gives populations accurate, actionable guidance—bridging the gap between community pharmacies and overstretched hospitals, without ever pretending to be a doctor.

Demo

https://youtu.be/PI3vzJGBDSA

Code

https://github.com/Vicarioy/adtc-2026-submission-template.git

How I Built It

Here is the technical walkthrough of how I built HealthBridge; covering the architecture, the tough trade-offs, and the exact steps I took to wrestle it onto resource-constrained hardware.

Model Selection: The "Goldilocks" Search The first decision was choosing the brain. I couldn't just pick the biggest model.

What I tested: I ran local benchmarks with Phi-3-mini-4k (3.8B—too much RAM, risked exceeding the 8GB target), Qwen2.5-0.5B (fit easily but gave vague, sometimes incorrect health advice), and Llama-3.2-1B (decent, but weaker at following multi-turn health Q&A prompts).

The decision: I chose Qwen2.5-1.5B-Instruct. It sits perfectly in the middle strong enough to reason through symptom-checking and triage logic, yet small enough to leave headroom for the OS and interface. Its multilingual tokenizer also helps handle local language nuances when users describe symptoms in pidgin or their native tongue.

Quantization Strategy (The Accuracy-vs-Memory Trade-off) A raw 1.5B FP16 model wouldn't fit in 8GB comfortably, so I needed compression.

I rejected Q4_0 (too much precision loss—critical for medical context) and Q5/Q8 (too heavy).

I settled on GGUF Q4_K_M. This uses K-means grouping on the most important weight matrices. It applies 4-bit quantization but allocates bits more intelligently across layers. This preserves the model's "reasoning" capability far better than flat quantization, giving me medically coherent responses while keeping peak RAM at just 1.7GB, well under the 8GB ceiling.

Runtime & Inference Engine (llama.cpp) The competition rules strictly mandated llama.cpp, but honestly, it was the perfect choice anyway.

I compiled llama.cpp with standard x86-64 CPU optimizations (leveraging AVX2 instructions on Intel chips).

I wrote a Python wrapper that uses llama-cpp-python to interface with the .gguf file. The key tweak here was setting n_ctx=32768 (to utilize Qwen's full context window for long patient histories) but carefully tuning n_batch to 512 to maximize throughput without blowing up the CPU cache.

This setup gave me 16.76 tokens/second on an integrated UHD 620; fast enough for real-time conversation.

The Interface Nightmare (Where I Almost Broke) You noted in the report that building the interface was a massive problem, and you're right. Streamlit kept timing out or failing silently.

The issue: Loading a 1.5B GGUF model inline during Streamlit's startup sequence blocks the main thread. Streamlit's watchdog timer would kill the process if the model loaded slower than ~10 seconds.

How I fixed it: I decoupled the model lifecycle. Instead of loading the model globally, I wrapped it inside a st.cache_resource decorator with lazy initialization meaning the model only loads on the first user prompt, not when the page boots. I also spawned the llama.cpp inference in a separate background thread with a queue, so the UI remains responsive while the model warms up. This single architectural shift made the app stable and prevented timeouts.

Deployment & Zero-Connectivity Pipeline Since internet is the problem, I had to make internet the non-requirement.

I wrote a download_model.sh script to fetch the .gguf from Hugging Face only once during setup.

I strictly excluded the ~900MB model file from git using .gitignore. The evaluator runs the script, grabs the model, and from that point forward, every inference call hits the local filesystem. No API keys, no network retries, no cloud latency.

The entire app runs on localhost; I pinned the environment dependencies (llama-cpp-python, streamlit) to specific versions to guarantee reproducible builds on the ADTC Ubuntu 22.04 target.

Domain-Specific Prompt Engineering Because I'm legally and ethically scoped to education/triage, not diagnosis, I hardcoded a system prompt that:

Explicitly states: "I am an educational assistant, not a doctor. Do not take my advice as a clinical diagnosis."

Guides the model to always include a "Red Flag" section in responses (e.g., "Go to a hospital immediately if you see blood in stool or have a high fever over 39°C").

Instructs the model to actively deflect dangerous queries if a user asks for a prescription, it responds with safety advice and urges them to visit an MDCN-registered practitioner.

Prize Technology Highlight: This project squarely falls into Edge AI / On-Device AI categories. By combining gguf quantization with llama.cpp, I built a production-ready, CPU-only LLM that democratizes health information in bandwidth-scarce regions; proving that impactful AI doesn't require the cloud.

5 ML Math Concepts I Stopped Memorizing (And Started Actually Using)

Victor — Sun, 06 Sep 2026 15:16:26 +0000

I've been deep in linear algebra and calculus for the better part of a year. But up until this week, I realized i was still treating core ML math like the chain rule and eigenvalue; as a syllabus checklist. Week 3 of my structured deep dive is when i finally forced myself to stop, derive them by hand and connect them to actual code using numpy. Here are some of the concepts that finally clicked:
1. The Chain Rule Is the Engine Behind Backpropagation
Every neural network learns by adjusting its weights, and it knows which direction to adjust them in because of the chain rule. A network is really just a stack of functions: input goes through a layer, then an activation function, then another layer, and so on. To know how much a single weight buried deep in that stack contributed to the final error, you need to differentiate through every function it passed through. That's exactly what the chain rule lets you do. I traced this end to end: starting from gradient descent (which says "move the weights a little in the
direction that reduces error"), through the activation function's own derivative, back to the weight itself. Once I could follow that chain by hand instead of trusting a library to do it, backpropagation stopped feeling like magic and started feeling like bookkeeping.
2. The Gaussian (Normal) Distribution Is Everywhere in ML
The bell curve isn't just a statistics-class cliché; it shows up constantly in machine learning: in how we initialize weights, in the noise assumptions behind linear regression, in probabilistic models, and in how we reason about errors clustering around a mean. Understanding its two parameters; mean and variance, is really understanding "where is the data centered" and "how spread out isit," which turns out to be one of the most reused ideas in the entire field.
3. Eigenvalues Are the Secret Behind Face Recognition
This was the concept that surprised me most. Eigenvalues and eigenvectors which sound purely academic are the foundation of techniques like PCA (Principal Component Analysis) and the classic "Eigenfaces" approach to face recognition. The idea: a face image has thousands of pixels, but most of the meaningful variation between different faces can be captured by a much smaller set of directions, the eigenvectors with the largest eigenvalues. Compress along those directions, and you keep what matters while discarding noise. Linear algebra, doing real work.
4. The Laplace Rule of Succession: Smoothing for Probability
What's the probability of an event you've never seen happen? Naively, zero but that's usually wrong and dangerous in ML, especially in text and classification models where an unseen word or category shouldn't automatically get a probability of zero. The Laplace rule of succession solves this by adding a small smoothing count to every outcome, so nothing is ever assigned a flat-out impossible probability. It's a simple idea with a name that sounds far more intimidating than it is.
5. Sampling and Confidence Intervals: Choosing Data You Can
Trust
You can't train on an entire population, so you sample from it; but that raises an obvious question: how do you know your sample actually represents the population? This is where confidence intervals come in. They give you a principled range around an estimate, along with a stated level of certainty, so that when you pick a dataset you know how much to trust conclusions drawn from it. This felt directly relevant to my own work, since a lot of real-world data (like the regional health data in my thesis project) is sampled, imperfect, and needs exactly this kind of scrutiny.
Why This Matters
None of these five ideas are new inventions, they're decades old. But sitting with the math instead of skipping to the library call changed how I read papers. When something behaves strangely, I now have a better instinct for whether the culprit is a distributional assumption, a smoothing issue, or a bad sample, instead of just tuning hyperparameters and hoping.
Next up: going deeper into differential calculus and connecting it more directly to optimizers beyond
plain gradient descent.