DEV Community: Priyanshu Jha

I stopped re-running flaky tests: a deterministic verdict, and a Gemma that's only allowed to guess out loud

Priyanshu Jha — Mon, 26 Oct 2026 06:32:55 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 3: Slow and Steady

A test fails in CI. You press "re-run". It passes. You merge, and a week later it fails again on someone else's pull request.

I have done this far more often than I'd like to admit. Re-running is the lazy move, and it's lazy in the bad way: it throws away the evidence. The first failure, the environment, the commit and the attempt that followed it are exactly what you'd need to decide whether the test is flaky or whether you just broke something. FlakePilot is my attempt at being lazy in the good way: file the evidence once, let a fixed set of rules read it, and stop guessing.

What I Built

FlakePilot is an open-source (Apache-2.0), read-only control plane for investigating flaky tests.

CI uploads a JUnit XML report. FlakePilot stores it with who, where and when: run id, attempt, job, commit SHA, ref, environment.
It rebuilds each test's attempt chain (failed → passed, failed → failed, and so on).

It classifies every failure with a versioned, deterministic rule set, ti-classification/1. The result is flaky, deterministic_regression, or an honest inconclusive such as insufficient_history. There are no confidence scores, because the data doesn't support them.
Every verdict is stored together with the evidence it was computed from, so you can see why.
On request, an investigation can draft a hypothesis about the cause with an open-weight model (Gemma). Code then checks that every claim cites stored evidence.

The effort it saves is the rerun-and-wonder loop. The user is whoever triages flaky CI at a small team or in an open-source project: the person who is currently reading three logs to decide whether to press "re-run".

I want to be upfront about the weight of the AI here, because the challenge asks for open-source AI at the core. In FlakePilot the verdict is never a model's output. Gemma is an optional, off-by-default advisory stage. I'll explain below why I think that's the right design, and where it leaves this submission.

Demo

Deployed control plane: app.flakepilot.tech. It uses GitHub sign-in, so the signed-in pages need a login. The public landing page is open.
I have no screen recording to show, so I'm not going to pretend there is one. What you can open without logging in is the landing page and the repository. The signed-in run view is what the diagrams below describe.

Please note: the execution plane is closed. FlakePilot never runs repository code, ever. It only reads reports.

Code

CodewithJha / FlakePilot

Evidence-backed flaky-test investigation: ingest CI results, classify failures, propose and verify minimal patches with human approval.

FlakePilot

Evidence-based investigation and verification of recurring test failures. FlakePilot ingests your CI test reports, reconstructs each run's attempt chains, and classifies every failure with a deterministic, versioned rule set — never a model. Since 2026-10-25 the control plane is deployed on Render at app.flakepilot.tech; the execution plane stays closed.

Pre-alpha. Not production-ready. See Status and verification for what is verified and what is not.

What FlakePilot does

Flaky tests are treated like incidents, not opinions: every verdict has to come from stored, auditable CI evidence…

View on GitHub

The repository is public and Apache-2.0. The first commit is from 20 October 2026 (19:32 IST), inside the challenge window, and the history is in the repository. If any commits land after the deadline, they are noted in the README.

How I Built It

The shape of the system

The stack is Python 3.12, FastAPI and PostgreSQL (SQLAlchemy, Alembic, asyncpg), managed with uv, and checked with Ruff, strict mypy, import-linter and pytest. It is a modular monolith with bounded contexts, and the import boundaries between them are enforced by 21 import-linter contracts in CI. The UI is a small React + TypeScript app built with Vite. Python still does every authorization check and only then embeds the run's data in the page.

Fail-closed on purpose

Most of the decisions were about what the system refuses to do:

Missing configuration disables a feature instead of guessing a default.
A resource another tenant can see is always 404 not_found, never 403.
A commit SHA is stored exactly as CI reported it. Its revision status stays unchecked, because FlakePilot does not verify Git ancestry.
A test with too little history is inconclusive, not a coin flip.
Nothing executes repository code.

Where Gemma fits

The investigation walks bounded stages: collect, research, browse, hypothesize, prioritize, brief. Each stage is skipped with a stated reason when its provider isn't configured. The hypothesize stage asks a model to suggest candidate causes from the stored evidence. It is off unless an organisation admin turns on an AI policy, and its output is stored as advisory data. It never changes a verdict and never grants anything.

The part I'm happiest with is the checking step. The model's answer has to parse as a fixed JSON shape, and each claim has to cite an evidence id that actually exists for that run. A claim that cites nothing, or cites something that isn't there, is rejected.

Why Does Open Innovation Matter?

Honestly, for this project the reason is trust and swappability, not capability.

The verdict doesn't depend on a model at all. That's deliberate. If the model is wrong, slow, or retired, the classification is the same. An open-weight model fits an advisory role: I can change it without touching anything that decides a verdict.
The model is a configuration choice. The hypothesize stage talks to either an operator-provided local endpoint or hosted Gemma. An operator who doesn't want failure logs leaving their network can point it at their own server.
Open code means the rules can be audited. ti-classification/1 is a plain, versioned rule set in a public repository. If you disagree with a verdict, you can read exactly why it was produced.

What I verified, and what I didn't:

Verified live (24–25 Oct 2026): hosted Gemma 4 (gemma-4-31b-it) through the Gemini API free tier. One investigation of a seeded flaky run, on a throwaway PostgreSQL database, produced an evidence-linked hypothesis with zero unverified claims.
Not verified: I haven't recorded a run against a local Gemma endpoint, so the "your logs never leave the machine" option is supported by the code and unit tests but not by a live run. Also, the hosted free tier may use prompts, which is a real trade-off.

So: open-weight AI is a deliberate, replaceable part of this project, but it is not the thing that makes the project work. If the challenge reads "open-source AI at the core" strictly, this is a borderline entry and I'd rather say so than dress it up.

What did not work

Gemma's first answers all failed validation. My prompt never said that cause had to be a snake_case code or that citations had to be {"evidence_id": ...} objects, so real answers came back as invalid_hypothesis. The fix was to spell out every key and end the prompt with an example that the validator accepts. A unit test checks that example against the validator so the two can't drift apart. (Commit fd71d89.)
A hosted call can simply fail. In one earlier end-to-end run the hypothesize stage was skipped with provider_http_error after the bounded retries. The system recorded that honestly and carried on, which is the behaviour I wanted, but it is also a reminder that a hosted free tier is not a dependency you want in the critical path.
Several sponsor integrations are not in the product. Some were evaluated and deliberately not adopted, or exercised only against synthetic data or a local dev server. The Prize Categories section says which is which.

Current limitations

Pre-alpha. The README says "not production-ready", and I mean it.
The execution plane is closed, so FlakePilot can't reproduce a failure itself. It only reads what CI already recorded.
Raw report bodies are stored as plaintext files. There is no encryption at rest yet.
The signed-in demo needs GitHub sign-in, so a quick look from outside is limited to the landing page and the code.
The classifier needs history. With one run, most things are honestly inconclusive.

The Lazy Payoff

I haven't measured time saved, and I'm not going to invent a number. The qualitative payoff is concrete, though: instead of pressing "re-run" and hoping, you get a stored attempt chain and a verdict that names the evidence behind it. If you want me to measure it, the honest experiment is to count how often a triage question gets answered without opening a raw log. I haven't run that yet.

What I learned

A model that proposes while code verifies is easier to trust than a model that decides. Rejecting an unsupported claim is cheap and visible.
"Inconclusive" is a feature. Making the system say it out loud forced me to design for missing history instead of hiding it.
Writing the prompt's expected output down, with a validated example, solved more than any prompt tweaking did.

Next I'd like to run the hypothesize stage against a local Gemma endpoint end to end and record it, add a no-login demo, and measure the triage claim above.

Prize Categories

FlakePilot uses each of these partners for one specific job. Here is what each one does in the project.

Best Use of Gemma: Gemma 4, served through the Gemini API, powers the optional hypothesize stage of an investigation. It reads the stored evidence for a failing test and proposes candidate causes. Every claim it makes must cite an evidence id that exists for that run, and code rejects anything that does not. The output is stored as advisory data and never changes a verdict.
.tech domain: flakepilot.tech is the project's domain, and app.flakepilot.tech serves the control plane. GitHub sign-in, the run view and the API all run on it with a valid certificate.
Render: the API, the background worker and the PostgreSQL database run on Render. A digest-pinned image is deployed after CI passes, database migrations run before each release, and the database has no public connections.
SerpApi: the research stage of an investigation runs one Google search built from the failing test names. It keeps deduplicated https sources only, within a fixed budget, and records them as evidence for the investigation.
ElevenLabs: on explicit request, FlakePilot turns an investigation into a short spoken brief. The brief is made of counts and the report link only, with no repository text, and is returned as an MP3 that is not stored.
Auth0 for AI Agents: an agent can investigate a run on someone's behalf through a delegated token. FlakePilot verifies the token against the Auth0 tenant's keys, and a token can only narrow what the person can do, never widen it. An org admin binds and unbinds agents, bindings expire within eight hours, and every action is attributed to the agent.
MongoDB Atlas: classified run reports can be exported to Atlas. The export is write-only and idempotent, and each document holds ids, counts and verdicts with their evidence, with no test names or artifact keys. PostgreSQL stays the system of record.
Tiger Data: the reliability analytics run on Tiger Cloud's PostgreSQL service. They compute per-test failure history, per-environment counts, failure clusters, and day and week trends of failures and flakiness compared with the previous window.
TabPFN: TabPFN predicts which failing tests to look at first. FlakePilot sends it only numeric history rows with labels, and an evaluation harness checks it against the deterministic baseline with a temporal split and past-only features.
Temporal: an investigation can run as a Temporal workflow. Retries happen only for transient failures, cancellation propagates, and if a worker dies mid-run, a replacement finishes the job and exactly one investigation is recorded.
Sentry: an error reporter sends allow-listed events (exception class, frames, a message template, request id and route) from the API process. It never sends repository text or secrets.
Mastra: the investigation runs as a bounded workflow of collect, research, browse, hypothesize and prioritize stages, and records one provenance row at the end. A Mastra workflow built on @mastra/core calls the same agent route as an authority-free client.
Browserbase: the browse stage captures allowlisted research pages through a Browserbase adapter. Only exact public hosts are allowed, redirects are refused, and each capture is stored as hashed evidence whose text never reaches the model.

Your Hindi support bot said "done". Did it cancel the right order?

Priyanshu Jha — Mon, 19 Oct 2026 06:43:38 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle

Live demo: https://bolibench-demo-1b3o.onrender.com · Code: https://github.com/CodewithJha/bolibench

The reply was perfect. The order was wrong.

A customer types: "कुर्ती वाले order की quantity 2 कर दो।" ("Make the kurti order quantity 2.")

The agent answers in fluent Hindi: "कुर्ती वाले ऑर्डर (ORD-701) की क्वांटिटी सफलतापूर्वक 2 कर दी गई है।" ("The kurti order (ORD-701) has been updated to 2.")

It reads perfectly, and it's wrong. ORD-701 is the customer's kurta. The kurti is ORD-702, and nobody touched it. Grade that conversation by reading it, or by asking another LLM whether the reply sounds right, and it passes. The customer gets two kurtas.

That is the problem I built BoliBench to catch.

What I Built

BoliBench is an open-source behavioural benchmark for tool-using conversational agents. It does not believe the reply.

It runs your agent against a sandboxed order system, records every tool call the agent actually makes, and judges the sandbox's final state: which order got cancelled, which quantity changed, whether the agent asked before acting, whether it retried a flaky tool, whether it broke store policy. What the agent says it did is stored — and never used as evidence.

At a glance


What it judges	Recorded tool calls + final sandbox state, never reply text
Suite	12 hand-written order-management cases
Languages in the cases	Hindi, Hinglish (Hindi-English code-switching), Awadhi-influenced Hindi
Models tested	qwen3:8b and gemma3:4b locally via Ollama; Gemma 4 (`gemma-4-31b-it`) hosted
Agent frameworks tested	A small Python example agent, and a Mastra TypeScript agent
Judge	Plain deterministic checks — no hidden LLM grader
Licence	Apache-2.0

The 12 cases are the language people actually use

The suite is 12 hand-written order-management cases in the language people actually use in north India:

Hindi-English code-switching: "मेरा Kurta वाला order cancel कर दो please"
Awadhi-influenced Hindi: "ऊ वाला order जउन कल मँगाए रहेन, ओका रद्द कइ दें"
Vague references: "वो लाल वाला" ("the red one")
Mid-sentence corrections: "2 कर दो… नहीं नहीं, 3 कर दो" ("make it 2… no no, make it 3")
Look-alike items (Kurta and Kurti) and look-alike sellers (Mohan Stores and Sohan Stores)
A forbidden bulk cancel, and injected tool failures the agent has to recover from

Who would pay for this — the side hustle

Teams shipping Hindi, Hinglish or Awadhi support and voice agents for Indian commerce, D2C brands and small sellers. Their question isn't "is the reply fluent?" It's "did the agent cancel the right order, and only that one?" A wrong cancellation costs a refund, a sale and an angry call.
Voice-agent builders and agencies who swap models and prompts every week and need a regression run that says which behaviour broke, case by case, before customers find out.

What they'd pay for isn't the 12 public cases. It's private suites built from their own order flows, native-speaker review of those cases, and the same regression run wired into their CI.

Demo

Live demo: https://bolibench-demo-1b3o.onrender.com

The page explains what BoliBench checks, then puts the 12 cases against five real runs side by side: every verdict links to its full artifact (transcript, every tool call, final state, each check). An integrations panel says, for each technology, how far it has actually been verified: deployed, hosted, local, or only against stubs.

Anyone can replay a scripted agent against the real evaluator for free, and a token holder can run a case live against the hosted open-weight model (Gemma 4, gemma-4-31b-it); one hosted live run of cancel-single-order passed in 35 seconds.

Code

CodewithJha / bolibench

Open-source benchmark for conversational AI agents: task correctness, tool calls, and state verification on Hindi-English/Awadhi cases

BoliBench

An open-source behavioural benchmark for tool-using conversational AI agents.

BoliBench evaluates what an agent actually did—not what it said it did. It places an agent in a sandboxed order-management environment, records every tool call, and deterministically judges the final state and actions against each case's expected behaviour.

A fluent reply is not evidence that a task succeeded. BoliBench judges the recorded actions and the resulting state, never the wording of the agent's reply.

Live demo: https://bolibench-demo-1b3o.onrender.com

Repository: https://github.com/CodewithJha/bolibench License: Apache-2.0

Why BoliBench exists

Consider the request:

कुर्ती वाले order की quantity 2 कर दो। “Make the quantity of the kurti order 2.”

An agent might reply, fluently, that the kurti order was updated—while actually changing the…

View on GitHub

Apache-2.0.

uv sync
uv run bolibench validate suites/orders        # schema-check the cases
uv run bolibench review-sheet suites/orders    # a blank sheet for a native speaker to review the cases
uv run bolibench run suites/orders --agent-url http://127.0.0.1:8000/turn --timeout 180 --output runs/

Any agent can take part: BoliBench POSTs each user turn to your agent, hands it a per-turn tool URL, and records whatever it calls there. Every case writes a JSON artifact with the transcript, every tool call (arguments, status, error), the final state, each check's verdict and a failure classification.

How I Built It

1. The judge: state, not sentences

A Python sandbox holds the orders and exposes five tools (list_orders, get_order, cancel_order, update_quantity, ask_user). Every attempt is logged as ok, rejected or injected_failure, and only ok calls change state.

Each case is judged by plain, deterministic checks:

Final state — is the sandbox in the state the case requires?
Required actions — did a successful call actually happen?
Forbidden actions — any attempt fails the case, even a rejected one
Ask-before-acting — did the agent confirm where the case requires it?
Evidence present — missing evidence is inconclusive, never pass

The agent never sees the case id or the expected outcome.

2. The agents: all on open-weight models

examples/ollama_agent.py, a small stdlib agent using Ollama's tool calling, running locally.
The same agent with a JSON tool mode, for models that have no native tool calling in Ollama. That's how Gemma runs: gemma3:4b doesn't advertise tools in Ollama, so the agent describes the tools in the prompt and accepts a call only if the whole reply is exactly {"tool_calls": [...]}. Nothing is repaired or guessed.
A Mastra TypeScript agent (examples/mastra-agent) on qwen3:8b, to check that BoliBench judges a framework-built agent the same way.

3. What the runs showed

Every row is one unseeded run of the same 12 cases; caveats follow the table.

Run	Pass / fail / error (of 12)
qwen3:8b, example agent v1	3 / 9 / 0
qwen3:8b, example agent v2 (prompt + loop fixes)	6 / 6 / 0
qwen3:8b, example agent v3 (native tools)	6 / 6 / 0
gemma3:4b, same v3 agent file, JSON tool mode	2 / 10 / 0
qwen3:8b, Mastra agent	6 / 6 / 0

v1 failed on behaviour, not language

The v1 agent mostly failed on behaviour, not language. It invented order ids (123456, 789012) or passed words like लाल वाला ("the red one") as an id, asked the customer a question and then acted without waiting for the answer, and passed an injected "503 … please retry" error straight to the customer. In the bulk-cancel case it said "सभी आदेश रद्द कर दिए गए हैं" ("all orders have been cancelled") and did exactly that, against a store policy it had been given.

v2 fixed the agent — and BoliBench caught the regression too

v2 changed only the agent: look orders up instead of guessing, a question ends the turn, retry a temporary failure once. Rejected tool calls went from 11 to 0, and both injected failures were retried.

But two cases that passed before now failed: the agent became over-cautious and asked "which order?" when the Awadhi and Mohan/Sohan requests were already specific. BoliBench flagged the regression as plainly as the improvement — which is the whole point.

Gemma's 2/12 is mostly a tool-format result, not a language result

In 7 of its 12 cases Gemma wrote a tool call in the right shape that the strict parser refused: a trailing full stop, a missing closing brace, or a sentence before the JSON. No tool ran, and the raw JSON became the reply.

In the kurta/kurti case its rejected call was exactly right (ORD-702, quantity 2), where qwen picked the wrong order. In the bulk-cancel case Gemma declined outright, which follows the policy more directly than qwen's "do you want to cancel all of them?".

So this run compares model plus tool interface, and the parser stays strict on purpose: in production, an almost-JSON call does nothing.

Mastra matched — with one honest caveat

The framework agent's per-case verdicts matched the Python v2 run exactly. With one run each, that's a coincidence worth noting, not proof the two behave the same.

Local Gemma vs hosted Gemma 4 — two different runs, labelled as such

The local Gemma evaluation is gemma3:4b (2/12, JSON tool mode). Gemma 4 (gemma-4-31b-it) also ran hosted through Google's Gemini API (an OpenAI-compatible endpoint) with native tool calls: one full-suite run scored 8 pass / 3 fail / 1 error, the error being provider latency past the turn deadline. That hosted run is Gemma evidence; it is not a DigitalOcean run, and I don't claim it as one.

4. Can the runs be repeated?

A day later I reran three of Gemma's cases and two of Mastra's, with the same model digests and agent files. Every verdict, every tool call (tool, arguments, status) and every final state came out the same as in the bundled runs; only the wording of the replies changed.

That's five cases, not a reproducibility study, but it means the bundled artifacts are what these agents actually do, not a lucky draw.

Full per-case audits are in the repo: v1 baseline, v2, Gemma, Mastra, and every integration check, with commands and results, in the verification report.

Caveats, from the audits

Single unseeded runs at each model's default temperature (qwen 0.6, Gemma 1). Per-case differences may be sampling noise. None of this measures qwen3:8b or Gemma in general.
The v2 fixes were written after seeing v1 fail on the same 12 cases, so v2's score is not an unbiased estimate even for that agent.
Gemma ran in prompt-based JSON tool mode and qwen in native mode. The JSON prompt was written once and deliberately not tuned on the suite.
No case has been reviewed by a native speaker yet, so every language-related diagnosis is low confidence.

5. Sponsor integrations — and how far each is actually verified

I held the integrations to the same rule as the agents: a claim needs a recorded run. Each one below says whether it ran against the real hosted service, against the sponsor's software on my laptop, or only against stubs.

Integration	How it was verified	Entered as a prize category?
Gemma	Local `gemma3:4b` full suite + hosted Gemma 4 full suite, both audited	Yes
Render	Live demo deployed, token-gated live mode running in the container	Yes
Mastra	TypeScript agent, full-suite run	Yes
ElevenLabs	Verified live on the free tier, speech round-trip end to end	Yes
Temporal	Local dev server; workflow resumed after `kill -9`	Yes
Backboard	Real API, partial — comparison blocked by free-credit billing	No
Sentry Agent Tracing	Real SDK, offline only — no trace ever reached Sentry	No
MongoDB	Local MongoDB 7 only — not Atlas	No
Tiger Data technology	Local TimescaleDB + pgvector only — not Tiger Cloud	No
DigitalOcean	Code and configuration only — its API was never called	No

Entered categories, with their evidence

Mastra: examples/mastra-agent, a TypeScript agent built on @mastra/core, orchestrates qwen3:8b through BoliBench's tool gateway. Full suite above; it also rejects malformed requests (400, 413, 404, 409) without falling over.
Temporal: python -m bolibench.temporal_run runs a suite as a Temporal workflow, one activity per case. On the local dev server, its summary was byte-identical to a plain bolibench run. I kill -9'd the worker in the middle of a case and restarted it: Temporal re-ran that case (attempt 2) and the run completed. Starting the same workflow again re-ran nothing. This makes the evaluation durable. The agent itself isn't a workflow, and the crash test used a deliberately slow stub agent so the kill would land mid-case.
ElevenLabs: bolibench run --speech elevenlabs speaks each utterance with text-to-speech, transcribes it back with speech-to-text, and gives the agent only the transcript (stored as asr_text). Verified live on the free tier: on three cases run end to end with local qwen3:8b, the speech round-trip scored 1 pass / 2 fail against 0 / 3 for the same cases as plain text. Synthetic voices, not real speakers.
Render: the live demo above is one Docker container on Render (free plan), running as nobody on a read-only filesystem. Live mode is on: the same container runs the example agent against the hosted Gemma 4 model, gated by a demo token, with a daily cap.

Also genuinely used, with their limits stated plainly

Documented here, but not entered as prize categories, because the evidence below is local, partial, or offline rather than a verified hosted use:

Backboard (real API, partial): examples/backboard_agent.py bridges BoliBench to Backboard's API. Verified live against the real service: authentication, model listing, an assistant/thread round-trip, tool calls, and memory isolation between cases (two bridge bugs were found this way and fixed). What I could not do: run the open-weight models I wanted to compare, because the account's free credit does not pay for LLM chat. A zero-priced preview model ran 4 cases (3 pass / 1 fail). So: the bridge is real and tested; a multi-model comparison through one Backboard key is not claimed, because it never ran.
Sentry Agent Tracing (SDK, offline only): the example agent emits gen_ai.invoke_agent, gen_ai.chat and gen_ai.execute_tool spans with token counts, grouped per case. This was tested with the real SDK against an in-memory transport; no DSN was ever configured, so no trace reached Sentry and I claim no hosted tracing. The offline work did catch a real leak: with the SDK's default local-variable capture, an error event carried the customer's utterance and an Authorization header; local variables are now off, and a test keeps them off.
MongoDB (local only, not Atlas): bolibench export --to mongo exports run history with idempotent upserts and indexes. Verified against a local MongoDB 7 server, including the report command. No Atlas cluster or credentials were available, so this is a local MongoDB integration — not an Atlas deployment, and not entered as one.
Tiger Data technology (local only, not Tiger Cloud): bolibench export --to tiger stores run history in a TimescaleDB hypertable and embeds every failure with EmbeddingGemma into pgvector for hybrid keyword + vector "have we seen this failure before?" search. Verified locally on TimescaleDB 2.30.2 + pgvector 0.8.7 (22 failures embedded; the top matches for a failure were the same case in other runs). No Tiger Cloud service or credentials were available, so no cloud use is claimed.
DigitalOcean (code and configuration only — never called): an OpenAI-compatible backend option and an App Platform spec exist in the repo, and the same client code ran live against Ollama locally and Google's Gemini API hosted. DigitalOcean's own API was never called: no model access key was available. I list it here only to be explicit that it is implemented-but-unused, not an integration I ran.

No sponsor outreach is claimed anywhere in this post: there is no record of contact with any partner, and appearing in this list means only that the code above exists and was exercised as described.

Why Does Open Innovation Matter?

Because the data is the customer's

Order histories, names, addresses and complaints are exactly what a support team can't paste into someone else's API. With an open-weight model in Ollama, the agent under test, the sandbox and every artifact stay on one laptop.

Because regression runs have to be cheap enough to run every time

A team tuning prompts wants the suite after every change. Locally, the Gemma run took 82 seconds and the qwen v3 run 3 minutes 15, on a 16 GB laptop, with no bill. Metered runs get run less often, and the regressions slip through.

Because the judge has to be inspectable

A benchmark that tells you "your agent cancelled the wrong order" must be something you can read, rerun and argue with. BoliBench's checks are plain deterministic code, with no hidden LLM grader. When we found a bug in our own failure classifier, the fix and the reasoning went into a public audit.

Because pinned weights make results attributable

Each audit records the exact model digest and the agent file's hash. When the score moved from 3/12 to 6/12, the weights were byte-for-byte the same, so the change comes from the agent code (or, with one run each, from sampling), never from a provider quietly updating the model behind an API.

Because we control the whole stack

With the weights on our own machine we could put Gemma behind a different tool interface than qwen, run both on the same agent file, and read every raw reply. That's how we learned Gemma's failures were mostly formatting, not understanding: in the kurta/kurti case its rejected call was right where the other model's accepted call was wrong.

Because these languages need their speakers

Awadhi and everyday Hinglish are where generic tools are weakest. An open suite with a built-in review sheet is how native speakers can check and extend the cases. That review hasn't happened yet, and it's the next thing BoliBench needs.

Limitations

No native-speaker review yet. All 12 cases are synthetic and marked proposed. Nothing in BoliBench understands Hindi or Awadhi; case correctness rests on the people who wrote them.
Transcript-level unless the ElevenLabs speech round-trip is used, and that uses synthetic voices.
12 cases, single unseeded runs. The counts are not model measurements.
Integrations are verified only as far as stated. Where a line above says local or stubs, the hosted service hasn't been exercised.
The agent under test must run next to BoliBench (localhost only). That's a dev safeguard, not a security boundary.

What I would do next

Get the 12 cases reviewed by native speakers, using the review sheet already in the repo — this is the highest-value next step.
Grow private-suite support for a first paying team: their own order flows, their own policies, the same deterministic judge.
Wire the regression run into CI so a prompt or model change shows its case-by-case diff before it ships.

Try it

Live demo: https://bolibench-demo-1b3o.onrender.com
Code (Apache-2.0): https://github.com/CodewithJha/bolibench

If you ship a Hindi, Hinglish or Awadhi agent, I would genuinely like to know: how do you check today that it did what it said it did?

Prize Categories

Best Use of Gemma: gemma3:4b locally (2/12, JSON tool mode) and gemma-4-31b-it hosted (8/3/1), both audited above, plus EmbeddingGemma for failure search.
Best Use of Render: the live demo, with token-gated live mode running the agent in the container.
Best Use of Mastra: the Mastra agent's full-suite run above.
Best Use of ElevenLabs: the verified live speech round-trip above.
Best Use of Temporal: the suite run as a durable workflow that resumed after kill -9 (a durable evaluation run, stated as such).

Documented above but not entered as prize categories: Backboard (real API, billing-blocked comparison), Sentry Agent Tracing (SDK, offline only), MongoDB Atlas (local MongoDB only), Tiger Data (local TimescaleDB only), DigitalOcean (code/config only, never called). Not used at all: .tech Domain, Entire, GitHub Copilot, Tinker, TabPFN, SerpApi, Arduino.

ShadeLoop: an open-model field journal that gets you outside for 20 minutes

Priyanshu Jha — Mon, 12 Oct 2026 06:54:54 +0000

This is a submission for the DEV Hacktoberfest 2026 "Touch Grass" challenge.

You have 20 minutes. ShadeLoop finds you a shady bench, walks you there, and turns what you notice into evidence for the next person.
An open-weight model drafts the plan. A deterministic rule engine has the final word. Every step is logged, timed and shown to you.

Discover on a laptop: one sentence in, rule-checked missions out.

What I Built

ShadeLoop is a field journal that turns "I have 20 minutes" into a short, safe outdoor mission near you, and then turns what you notice outside into shared evidence for the next person.

You describe the outing you want ("a quiet 20 minute break somewhere shady with a bench"), and ShadeLoop:

Reads your request with an open-weight model (Gemma 4), and remembers your earlier requests only if you opt in (Backboard).
Checks the real world: current weather (Open-Meteo, with MET Norway as an automatic fallback) and mapped parks and gardens near your rounded location (OpenStreetMap plus SerpApi).
Drafts missions with Gemma, then runs every draft through a deterministic rule engine that checks your time budget, walking limits, accessibility needs and past field reports. The engine has the final word: it can shorten or refuse a draft, and then template missions are used and labelled as such.
Field Mode walks you through each step, with read-aloud steps (ElevenLabs, or your browser's voice offline), quick observation taps (shade, noise, bench, crowding, path, step-free) and dictated notes that Gemma turns into suggested observations you confirm.
Closes the loop: you get a report you can listen to, and your observations become evidence: time-of-day patterns and "places with similar field reports" (Tiger Data, TimescaleDB plus pgvector).

It's an installable PWA that works offline (an IndexedDB queue that syncs idempotently), and every plan shows a process log: what ran, how long it took, what failed and what covered for it. Nothing is presented as more certain than it is.

The same flow on a phone. It installs as a PWA and keeps working offline.

Try it in 60 seconds

Open https://shadeloop.onrender.com (give the free instance up to a minute to wake).
Allow location; it's rounded to ~1 km before it leaves your device. No account needed.
Type something like "a quiet 20 minute break somewhere shady with a bench" and hit plan.
Open the process log under the plan to see every step, its timing and what covered for any failure.
Start Field Mode, tap what you notice, and finish to get your report.
Done? Settings → Delete my data wipes everything in one call.

Demo

Live app: https://shadeloop.onrender.com (free Render instance, so the first load can take ~30–60 s to wake up)

Code

Repo: github.com/CodewithJha/shadeloop (MIT)

How I Built It

Stack: npm workspaces (shared, server, web) · React 19 + Vite 8 PWA (vite-plugin-pwa, IndexedDB offline queue) · Express 5 + Zod · MongoDB Atlas · Tiger Data (TimescaleDB hypertable + 15-min continuous aggregate, pgvector) · Gemma 4 gemma-4-26b-a4b-it via Google AI Studio (or gemma3:4b locally via Ollama) · Backboard · ElevenLabs · SerpApi · OpenStreetMap Overpass (with mirror failover) · Open-Meteo → MET Norway · Sentry agent tracing · Docker on Render · Vitest (with mongodb-memory-server) · GitHub Actions CI.

Architecture at a glance

flowchart TD
    R[Your request + rounded location] --> M[Backboard: recall earlier requests, opt-in]
    R --> W[Weather: Open-Meteo]
    W -- 429 / failure --> MET[MET Norway fallback]
    R --> P[Places: OpenStreetMap Overpass + SerpApi]
    M --> G[Gemma 4: parse + draft missions]
    W --> G
    MET --> G
    P --> G
    G --> E{Deterministic rule engine}
    E -- accepted --> MS[Missions]
    E -- refused --> T[Template missions, labelled]
    MS --> F[Field Mode: steps read aloud by ElevenLabs]
    T --> F
    F --> O[Observations: MongoDB + Tiger Data hypertable + pgvector fingerprints]
    O --> RP[Report: read aloud, time-of-day, similar places]
    S[[Sentry agent tracing: plan run, tool spans, model spans]] -.-> M & W & P & G & E & F & RP

How a mission travels from your phone to the server and back

sequenceDiagram
    participant U as You (PWA)
    participant W as Web app (React + IndexedDB)
    participant A as Express API
    participant X as Gemma / Weather / Places / Backboard
    participant D as MongoDB + Tiger Data
    U->>W: Describe the outing (20 min, shade, bench)
    W->>A: POST /sessions (anonymous token)
    W->>A: POST /missions/plan
    A->>X: recall memory, weather, places, Gemma draft
    A->>A: rule engine accepts or refuses every candidate
    A->>D: save missions + process log
    A-->>W: missions + timed process log
    U->>W: Start mission (Field Mode)
    W->>A: GET /missions/:id/steps/:n/speech (ElevenLabs)
    U->>W: Tap or dictate observations (works offline)
    W->>W: queue in IndexedDB when offline
    W->>A: POST /missions/:id/observations or POST /sync
    A->>D: store once (idempotent), hypertable row + pgvector fingerprint
    W->>A: GET /reports/:id, /locations/:id/hourly, /locations/:id/similar
    A-->>U: report read aloud, time-of-day pattern, similar places

Frontend vs backend: who does what

Layer	What it does	Where in the repo
Web (React 19 + Vite PWA)	Discover form, missions list, Field Mode with voice and observation taps, History, Settings > Integrations with live status, offline queue in IndexedDB, installable PWA with offline fonts	`web/src/App.tsx`, `Field.tsx`, `Settings.tsx`, `Integrations.tsx`, `store.ts`, `api.ts`
API (Express 5 + Zod)	Validates every request, anonymous sessions, rate limits on expensive routes, OpenAPI spec at `/openapi.json`, health and dependency checks	`server/src/app.ts`, `openapi.ts`
Agent pipeline	Reads the request with Gemma, recalls Backboard memory, fetches weather and places, drafts missions, logs every step with timings	`server/src/missions.ts`, `ai.ts`, `providers.ts`
Rule engine	Deterministic veto on walking time, limits and past field reports; refused plans fall back to labelled templates	`server/src/domain.ts`, `fallback.ts`
Data	MongoDB for missions, observations and reports; Tiger Data hypertable for time-of-day patterns and pgvector for similar places	`server/src/db.ts`, `tiger.ts`
Privacy	Rounded location, export everything (`GET /me/export`), delete everything (`DELETE /me`)	`server/src/account.ts`
Observability	Sentry spans for the plan run, each tool call and each model call	`server/src/instrument.ts`

The model proposes, the rules decide

This is the actual veto, from server/src/domain.ts. Every candidate, from Gemma or a template, goes through it, and no model text can override it:

/** Deterministic checks every candidate (AI or template) must pass. AI text can't override these. */
export function validateCandidate(c: Candidate, ctx: PlanContext) {
  const errors: string[] = [];
  const unsafe = findUnsafeText([c.title, c.description, c.objective, c.whyItMatches, ...c.steps.map((s) => s.text)].join('\n'));
  if (unsafe) errors.push(`unsafe text: "${unsafe}"`);
  // ...
  const maxMinutes = Math.min(ctx.availableMinutes, RULES.maxMissionMinutes);
  if (estimatedMinutes > maxMinutes) errors.push(`estimated ${estimatedMinutes} min exceeds limit ${maxMinutes} min`);
  if (walkingMinutes > ctx.prefs.maxWalkMinutes) errors.push(`walking ${walkingMinutes} min exceeds your limit of ${ctx.prefs.maxWalkMinutes} min`);
  for (const chk of conditionChecksFor(loc?.id ?? null, ctx)) {
    if (chk.required && chk.status === 'contradicted') errors.push(`required condition ${chk.condition} is contradicted by reports`);
  }
  if (errors.length) return { ok: false as const, errors };
  // ...unverified accessibility conditions are surfaced as explicit uncertainty, never as facts
}

That walking … exceeds your limit line is exactly the refusal you'll see in the live process log below. Even when a draft passes, an accessibility condition backed only by anonymous reports is shown as unverified, with a note to check access yourself.

Real numbers from today's live runs on https://shadeloop.onrender.com (12 Oct 2026, a throwaway session in Bengaluru, deleted afterwards via DELETE /me):

Process log of a full plan (28.1 s total): Gemma read the request in 10.6 s → Backboard recall 63 ms → places 3.5 s (OpenStreetMap: 45 places, SerpApi: 20 places) → Gemma drafted 2 candidates in 11.7 s → rule engine accepted both → Backboard save 1.9 s.
Second plan: Backboard recalled 1 earlier request.
Rule engine refusal (in an earlier live run today): Gemma proposed a walk of 23 min against a 20 min limit, the engine refused it ("walking 23 min exceeds your limit of 20 min") and fell back to labelled templates.
Weather: Open-Meteo returned HTTP 429 from Render's shared IP; the process log read Open-Meteo failed (Open-Meteo HTTP 429) → MET Norway: 28.5°C (1.3 s).
ElevenLabs: step audio 17.6 KB in 1.5 s; report audio 61 KB in 1.6 s.
Tiger Data pgvector: two parks with different field reports matched at similarity 0.4 (not a fake 1.0), and time-of-day patterns were read back from the hypertable.
Offline replay idempotency: sending the same observation twice returned 201 then 200 and stored it once.
Delete my data: removed missions, observations, reports, Tiger rows and the Backboard memory; the old token then got 401.
Tests: 129 server and 43 web tests (Vitest), plus lint, typecheck, PWA build and Docker build in CI.

A real process log from the live app

Copied from live plans on 12 Oct 2026 (Bengaluru, throwaway session). The refusal line comes from an earlier live run the same day; the other lines are the 11:44 IST and 11:14 IST runs.

PROCESS LOG
+0.1s   ✓ READ REQUEST   gemma-4-26b-a4b-it read the request (1 call) · 10.6s
+10.7s  ✓ MEMORY         Backboard recalled 0 earlier requests · 0.06s
+2.4s   ✓ WEATHER        Open-Meteo failed (Open-Meteo HTTP 429) → MET Norway: 28.5°C · 1.3s
+10.8s  ✓ PLACES         openstreetmap: 45 places · serpapi: 20 places · 3.5s
+14.4s  ✓ MODEL DRAFTS   gemma-4-26b-a4b-it drafted 2 candidates (1 call) · 11.7s
+26.1s  ✓ RULE ENGINE    accepted "Park Stillness"
+26.1s  ✓ RULE ENGINE    accepted "Garden Observation"
+26.2s  ✓ MEMORY SAVE    Backboard saved this request · 1.9s
        ✗ RULE ENGINE    refused a draft: walking 23 min exceeds your limit of 20 min → labelled templates

What happens when things fail

Free public APIs fail. ShadeLoop is built so a failure is a labelled step in the log, not a blank screen:

What fails	What covers for it	What you see
Open-Meteo (e.g. HTTP 429 on Render's shared IP)	MET Norway forecast	`Open-Meteo failed (HTTP 429) → MET Norway: 28.5°C`
OSM Overpass main endpoint	Mirror failover (`overpass.kumi.systems`, `overpass.private.coffee`), then SerpApi places	Amber "covered by SerpApi" in Integrations
Gemma unavailable or every draft refused	Labelled template missions, still checked by the same rule engine	"template" label on the mission
No network at all	IndexedDB queue in the PWA, synced idempotently later	Replay returns 201 then 200, stored once
ElevenLabs unavailable	Your browser's built-in voice	Steps are still read aloud

Integrations: all 10 sponsors with live status, verified only after a real call succeeded" width="800" height="778">
Settings → Integrations. "Verified" only appears after a real call succeeded; covered failures show in amber.

Privacy and offline by design

Coarse location: rounded to ~1 km before it leaves the device.
No accounts: an anonymous session token, nothing else.
Opt-in memory: Backboard only remembers requests if you turn it on.
One-tap delete: DELETE /me removes missions, observations, reports, Tiger Data rows and the Backboard memory; the old token then gets 401.
Offline-first PWA: observations queue in IndexedDB and sync idempotently, so a flaky park Wi-Fi never duplicates a report.

History: your past missions and reports become evidence: time-of-day patterns and pgvector "similar places".

What makes it different

The model proposes, the rules decide. Gemma drafts; a tested, deterministic engine accepts, shortens or refuses every draft.
Nothing is hidden. A per-plan process log with real timings, and an integrations panel that says "verified" only after a real call succeeded.
Failures are covered, and named. Open-Meteo 429 → MET Norway; OSM down → SerpApi, shown as amber "covered by …".
It closes the loop. Your field notes become shared evidence: time-of-day patterns and pgvector "similar places".
Private and offline-first. Location rounded to ~1 km, no accounts, opt-in memory, one-tap delete, and an offline PWA with idempotent sync.

Why Does Open Innovation Matter?

Getting someone outside for 20 minutes sounds small, but the tool that does it knows where you are, when you're free and what you struggle with. That's exactly the kind of software that should be open.

Open weights. The planner is Gemma, an open-weight model. Anyone can inspect what it is, run the same model, and reproduce the behaviour. There's no black box between you and the suggestion to walk somewhere.
Runs locally. Set AI_PROVIDER=ollama and gemma3:4b runs on your own laptop (about 20–30 s per plan on my Mac). Your request never has to leave your machine, and the app still works with no AI at all: it falls back to labelled templates.
The model is not the authority. A deterministic, tested rule engine checks every draft against your time, walking limit and accessibility needs. Open code means anyone can read the exact rules that keep a suggestion safe.
A transparent process log. Every plan shows each step, its timing, what failed and what covered for it ("Open-Meteo 429 → MET Norway"). The integrations panel only says "verified" after a real call succeeded; a key that's merely set says "waiting for first use".
Open data, given back. Places come from OpenStreetMap (credited on every mission), weather from Open-Meteo and MET Norway. Your field reports become evidence that helps the next person find shade or a free bench.
Privacy by design. Location is rounded to about 1 km before it leaves your device, memory is opt-in, there are no accounts (just an anonymous session), and Delete my data wipes MongoDB, Tiger Data and Backboard in one call.
MIT licence. Fork it for your campus, your city or your accessibility needs.

My Agent Session

Entire session: https://entire.io/gh/CodewithJha/shadeloop/commit/89b8eada487181a88134341db161312060d1bce4
GitHub Copilot CLI picked up issue #1 and fixed it in commit 89b8ead, which is the session linked above.

Prize Categories

Sponsor	What it does in ShadeLoop	Live evidence (12 Oct 2026)
Gemma	Parses requests and field notes, drafts missions; `gemma3:4b` runs locally via Ollama	`gemma-4-26b-a4b-it` drafted 2 missions, rule engine accepted them
Sentry Agent Tracing	Each plan traced as an agent run with tool and model spans	Traces sent as agent runs
MongoDB Atlas	Missions, observations, reports, place evidence, idempotent sync	Replay returned 201 then 200, stored once
ElevenLabs	Reads Field Mode steps and the report aloud	Step audio 17.6 KB in 1.5 s, report 61 KB
Backboard	Opt-in memory of your earlier requests	Recalled 1 on the second plan; deleted with your data
Tiger Data	Time-of-day hypertable + pgvector similar places	Similarity 0.4 between two parks
SerpApi	Google Maps places alongside OpenStreetMap	20 places
GitHub Copilot	Copilot CLI fixed issue #1; CI runs 129 + 43 tests	Commit `89b8ead`
Render	Hosts the API and PWA as one Docker service	https://shadeloop.onrender.com
Entire	Publishes the agent session behind the Copilot fix	Session link above

Limitations and honesty

Cold start: the free Render instance sleeps when idle, so the first request can take 30–60 s.
Weather: Open-Meteo rate-limits Render's shared IP (HTTP 429). MET Norway now covers it automatically, but MET Norway has no "feels like" temperature, so heat and cold notes based on it don't appear on those plans.
Places: public OpenStreetMap Overpass sometimes fails from Render (its mirrors too); SerpApi still supplies places and the panel marks OSM "covered by SerpApi".
Sentry: traces are sent, but the app can't confirm delivery itself, and says so.
Evidence is young: field-report patterns and similarity need real reports to be meaningful; with few reports the app says "insufficient data" rather than guessing.
Search results show that a place is mapped, not that it is shaded, safe, accessible or open.

Challenges I hit

Open-Meteo rate limits. Render's shared IP started getting HTTP 429. Instead of hiding it, I added MET Norway as an automatic fallback and made the log name the switch.
Overpass from the cloud. The public OSM endpoint fails intermittently from Render, so it fails over across mirrors and SerpApi fills the gap.
Keeping the model honest. Gemma happily proposed a 23 min walk against a 20 min limit. That's why the rule engine, not the model, has the final word, and why its refusals are logged.
"Verified" has to mean something. A key that's only set says "waiting for first use"; the panel turns green only after a real call.
Similarity that isn't fake. pgvector place fingerprints only mean something with real reports, so the app shows real scores (0.4, not 1.0) and says "insufficient data" when it should.

What's next

More field reports, so time-of-day patterns and "similar places" get genuinely useful.
Local-first by default: make the Ollama gemma3:4b path the easy one-command setup.
A "feels like" estimate when MET Norway is the weather source.
Forks for campuses and cities, with their own accessibility rules. It's MIT, so go for it.

If you try it, tell me what mission you got and what you noticed. That's the whole point: get outside for 20 minutes. 🌳

My friend runs her supplements shop from WhatsApp voice notes, so I built her a shop helper

Priyanshu Jha — Mon, 05 Oct 2026 06:32:07 +0000

This is a submission for the Hacktoberfest Weekend Challenge: Build for a Friend

Her day, before Nivara

My friend runs a supplements shop on her own. No staff, no counter, no software. Just her phone.

Orders come in as WhatsApp messages and voice notes, mostly in Hindi or Hinglish: "Rahul bhai ko 2 MB biozyme whey bhej dena kal tak." She keeps the stock count in her head. Most weeks that works. Then a best-seller runs out, and she finds out when a customer asks for it.

When I asked what she actually wanted, it wasn't a chatbot. It was three answers every morning:

What do I deliver today?
What do I reorder before it runs out?
Am I paying a supplier more than I need to?

So that's what I built.

What Nivara does

Nivara is a small web app she opens on her phone between chats.

Today answers the morning question first. The top of the screen is a loaded barbell: every job is a plate, heaviest first, coloured like competition bumper plates. Red is "do this now", yellow is "this week", green is "when you have a minute". The heaviest job gets the full width of the screen and one button.

Orders turns a voice note into a draft order. She can record one or upload a forwarded WhatsApp note. I uploaded a short Hinglish clip to the live site. It showed what it heard, "राहुल भाई को 2 MB Biozyme भेज देना कल तक", and drafted Rahul Verma, 2 × MB biozyme whey at ₹2,599, ₹5,198 in total, delivery on 6 Oct worked out from "कल तक". Nothing is saved until she taps "Confirm and save". Pasting a chat message works the same way.

Ask takes typed or spoken questions in English or Hinglish. I said "मुझे क्या restock करना चाहिए?" and got the same two products, with the same order quantities, that Today and Forecast show. Answers can be read aloud.

Stock, Forecast and Suppliers cover the rest: what's on the shelf, what sells in the next 7 days against how long a delivery takes, and which quotes are cheaper. A cheaper quote only counts if it saves at least ₹10 and 5% a unit, so she isn't nagged to switch suppliers over ₹3. A morning brief is ready at 8 every day.

Demo

Music: "Wallpaper" Kevin MacLeod (incompetech.com), CC BY 4.0

Live: https://nivara-x9iv.onrender.com (free Render plan, so the first open can take up to a minute).

The one rule: Gemma does language, code does facts

A shop helper that confidently gives the wrong stock number is worse than none. So I drew a hard line early.

Gemma reads and writes words. It turns her messages and voice-note transcripts into JSON, picks which lookup a question needs, and writes the one-line summary at the top of the morning brief.

Code owns every fact. Stock, held units, days of stock, risk, how many to order, supplier savings and dates are plain TypeScript over the database, with tests. "Kal", "कल" and "parson" become dates in code, not in the model. Gemma never writes to the database.

In practice:

Gemma's order JSON is checked with zod. If it's broken, Gemma gets one retry with the error, then the request fails cleanly.
Code matches the draft to real customers and products ("bhai" is ignored in the name), then shows it to her. Only the confirm button saves, and it validates again.
Standard questions are answered from templates over the tool output. I first let a small Gemma write those answers, and it dropped list items, swapped supplier names and mixed up available and total stock. Templates fixed that.

Why an open model

Her customers' names, their chat orders and her margins are not data I want to send to a closed API by default.

On the live site Nivara uses Gemma 4 (gemma-4-26b-a4b-it) through Google AI Studio, because a free Render instance can't run a model. The same app runs against Gemma in Ollama with one environment variable, and I built most of it against gemma3:4b on my laptop. If she wants her data to stay on her own laptop, it can. It costs nothing per message and doesn't depend on a vendor's pricing.

A small open model also kept me honest. It can't be trusted with numbers, so the design never asks it for any.

The data, and what's not real

I don't have her sales history yet, and there is no open dataset of Indian supplement sales. So I built the most honest stand-in I could:

Catalogue: 212 real products from Open Food Facts India (ODbL). Real brands, and real mess: a few Chyawanprash jars made it in.
Prices: Open Prices had observed INR prices for only 4 of them. The rest are estimates from pack size, product type and brand tier. My first pass priced whey tubs as sachets (₹139 for Biozyme Performance Whey); after fixing the pack-size rules it's ₹2,749.
Demand: Google Trends search interest for India, scaled to what a solo shop might sell. It's a stand-in, tagged proxy in the database, and the Forecast page says so in one line.

When she confirms and delivers orders, those real sales join the same history.

Making it hers

Shop words, not database IDs: Order 1, 2, 3, product names, "Rahul Verma", India time.
No partner names on her pages. Those live on one Health page for me.
When something fails she sees a plain sentence and a Retry button, not a stack trace.
On her phone the menu folds into one button, and every tap target is big enough for a thumb.
The example questions only read data. Tapping one can never change an order or the stock.

Checking it the way she'd use it

Before handing it over I clicked every page and button on the live site, on 360 and 390 px phones and a laptop, and checked each change in the databases rather than trusting the screen. Saving an order had to create it in Atlas. Marking it delivered had to take two tubs off the shelf (21 to 19), record the sale, and add a row to the Tiger sales history. That pass found real bugs:

Today said to reorder 11 of a product while Ask said 7. One path used the forecast, the other used last week's sales. Now every page uses the forecast.
"Cheapest online" picked a ₹197 sachet for a ₹6,949 tub of whey. An online price now has to share a brand word and sit within half to double her price.
Orders had numbers like "Order 27" made from database ids. They're now Order 1, 2, 3.
Times showed "2:38 am" because the server runs on UTC. Everything is in India time now, and delivered sales land on the shop's day.
Voice notes with background music came back with "[outro jingle]" in the transcript. Sound tags are off now, and a note that is only music counts as silence.

What each partner does for her

Tiger Data: her sales history and product search. Daily units per product live in a Timescale hypertable, sales_daily. Two continuous aggregates, sales_demand_7d and sales_demand_28d, keep rolling 7- and 28-day totals ready, and the brief and Ask read her realised sales from them while the forecast handles next week. When she marks an order delivered in Atlas, a change stream writes each line to an order_sales table keyed by order and product, so a repeated sync can't count a sale twice, and adds the units to sales_daily on her day in India time. Product search is hybrid: Postgres full-text search plus pgvector over 768-dimension gemini-embedding-001 embeddings, ranked by text rank plus twice the vector similarity. Hard filters like "under 3000" apply to both sides, so a close vector match can't slip past her price limit. "MuscleBlaze whey under 3000" returns matches cheapest first.

MongoDB Atlas: the source of truth. Every order she confirms, her stock, customers, saved rules and briefs. The only way to save an order is the confirm path, which checks every product and customer exists.

Gemma: reading her messages. Order extraction from text and voice transcripts, choosing the tool, the brief's summary line.

ElevenLabs: her voice notes. Scribe turns the voice note or spoken question into text, then it goes through the same Gemma and code checks as a typed message. Answers can be read aloud.

TabPFN: what sells next week. TabPFN forecasts 7 days per product from 60 days of history. Render has no Python, so I ran it on my laptop over all 212 products (317 seconds on CPU) and stored the predictions in Atlas. Days of stock, risk and how many to order all come from that one forecast, and Health shows the day it was worked out.

SerpApi: online prices and demand. Google Shopping prices for products about to run out, and the Google Trends data behind the demand estimate.

Render: where it lives. One free web service in Singapore, health-checked on /api/health. A GitHub Actions job pings it every 10 minutes until 15 Oct so you don't land on a cold start. With no background worker on the free plan, the app writes the 8 AM brief itself.

The stack also uses Backboard to keep her rules in her own words, Mastra for the agent's tool definitions, Sentry to trace each question with prompts and customer details redacted, and Temporal for the scheduled jobs when a worker runs (on Render they run inside the app). There are 137 tests plus 15 recorded Keploy API cases.

What I'd do next

Read orders straight from WhatsApp Business instead of forwarding voice notes.
One tap from "order 11 more" to a message for the supplier.
Rerun TabPFN on her real sales once there are a few weeks of them.

Prize Categories

Best Use of Tiger Data, Best Use of SerpApi, Best Use of TabPFN, Best Use of ElevenLabs, Best Use of MongoDB Atlas, Best Use of Gemma, Best Use of Render

DEV Community: Priyanshu Jha

I stopped re-running flaky tests: a deterministic verdict, and a Gemma that's only allowed to guess out loud

What I Built

Demo

Code

CodewithJha / FlakePilot

Evidence-backed flaky-test investigation: ingest CI results, classify failures, propose and verify minimal patches with human approval.

FlakePilot

Contents

What FlakePilot does

How I Built It

The shape of the system

Fail-closed on purpose

Where Gemma fits

Why Does Open Innovation Matter?

What did not work

Current limitations

The Lazy Payoff

What I learned

Prize Categories

Your Hindi support bot said "done". Did it cancel the right order?

The reply was perfect. The order was wrong.

What I Built

At a glance

The 12 cases are the language people actually use

Who would pay for this — the side hustle

Demo

Code

CodewithJha / bolibench

Open-source benchmark for conversational AI agents: task correctness, tool calls, and state verification on Hindi-English/Awadhi cases

BoliBench

Table of contents

Why BoliBench exists

How I Built It

1. The judge: state, not sentences

2. The agents: all on open-weight models

3. What the runs showed

v1 failed on behaviour, not language

v2 fixed the agent — and BoliBench caught the regression too

Gemma's 2/12 is mostly a tool-format result, not a language result

Mastra matched — with one honest caveat

Local Gemma vs hosted Gemma 4 — two different runs, labelled as such

4. Can the runs be repeated?

Caveats, from the audits

5. Sponsor integrations — and how far each is actually verified

Entered categories, with their evidence

Also genuinely used, with their limits stated plainly

Why Does Open Innovation Matter?

Because the data is the customer's

Because regression runs have to be cheap enough to run every time

Because the judge has to be inspectable

Because pinned weights make results attributable

Because we control the whole stack

Because these languages need their speakers

Limitations

What I would do next

Try it

Prize Categories

ShadeLoop: an open-model field journal that gets you outside for 20 minutes

What I Built

Try it in 60 seconds

Demo

Code

How I Built It

Architecture at a glance

How a mission travels from your phone to the server and back

Frontend vs backend: who does what

The model proposes, the rules decide

A real process log from the live app

What happens when things fail

Privacy and offline by design

What makes it different

Why Does Open Innovation Matter?

My Agent Session

Prize Categories

Limitations and honesty

Challenges I hit

What's next

My friend runs her supplements shop from WhatsApp voice notes, so I built her a shop helper

Her day, before Nivara