DEV Community: Tao Li

TrailBrief: Keep the Evidence, Leave the Screen

Tao Li — Sat, 10 Oct 2026 15:24:21 +0000

What I Built

TrailBrief turns a pasted route notice and weather snapshot into a pocket brief you can save as a self-contained HTML file, review offline or print.

It is for someone preparing a short outing who wants the important source evidence in one place. The aim is to make the screen a small part of preparation: collect the notices, check the gaps, take the brief, and verify conditions with the actual land manager and weather service before leaving.

The AI has a narrow job. It selects source segment IDs. The backend copies the original text and adds fixed check questions. It never asks the model to invent a forecast, decide that a route is safe, or compose the evidence quotations.

All examples in this post and demo are synthetic. I have not field-tested this application or authenticated real trail or weather data.

Demo

Explore the recorded demo.

The public page is labeled Recorded model result / live API disabled. You can switch between six real request records, inspect the synthetic source inputs and selected IDs, compare the failures, and download the two accepted briefs. Changing the selector does not call a model. There is no API-key field or live backend.

The fourth record includes a closed boardwalk, missing forecast coverage and an embedded source instruction telling the model to declare the route safe. The model selected four permitted IDs. The backend copied the closure, the forecast gap and the two official-source check reminders exactly. The instruction segment was not selected.

The fifth record supplies a forecast window that contains the planned departure and a notice reporting no closures. Its six selected segments all copied exactly, including both required evidence lines. These statements are true within the supplied fictional text; the application does not independently verify the conditions.

Code

Source repository. The reviewed release contains 216 files.

The repository contains the actual local Node/Mastra application, locked dependencies, tests, recorded results and the static /docs demo. Original TrailBrief source remains UNLICENSED. Public source visibility is not an additional open-source license; dependency licenses and notices are preserved separately.

How I Built It

The application uses @mastra/core 1.75.0 and Zod with a locked npm dependency graph. The real workflow has three steps:

Validate the two supplied snapshots and add review warnings.
Select evidence through the model adapter, or use the explicitly labeled deterministic demo.
Verify the selection and assemble the portable brief.

The first contract asked the model to return verbatim quotations. The early calls showed why this was fragile:

Attempt	What happened
1	Extraction was rejected. The original assistant body was not retained, so I cannot reconstruct the exact cause.
2	The 2048-token completion allowance was exhausted with an empty answer and `finish_reason=length`. The provider also reported all 2048 tokens as reasoning.
3	With an 8192-token allowance, JSON and schema validation passed. But a quotation changed “an imaginary valley” to “a valley”, so exact matching rejected it.
4	I changed the contract to ID selection. Four valid IDs produced an accepted, deterministically assembled brief.
5	The frozen ID version accepted a different normal fixture: supplied forecast coverage and no reported closure.
6	The frozen version's missing-data fixture encountered `API_TRANSPORT_ERROR` after about 30 seconds. No body or usage was received. It remained uncertain and was not retried.

For the new contract, the program splits each supplied source into bounded original slices. IDs bind the source ID, complete source text and positions. The model returns only sourceId and segmentId. Strict validation rejects additional fields, unknown IDs, repeats, cross-source substitutions and recognizable override directives. The program then copies the selected original slices and runs the existing exact-quotation check. No fuzzy matching or quotation repair happens.

Questions and checklist items are fixed program text. A model-written explanation cannot silently become evidence. Full snapshots and selected IDs stay available for review.

All six calls used the external deepseek/deepseek-v4.1-flash model with low reasoning and zero retries. The fourth API request through complete response-body receipt took 12.216 seconds; the fifth took 15.602 seconds. These measurements include the network and provider response, not just model compute. The sixth did not yield a completed-response measurement.

There are 82 passing offline tests in the frozen release. They cover ID stability, ownership, unknown/repeated IDs, strict fields, instruction isolation, missing-weather handling, exact copying, safe export, actual Mastra execution with mocked extraction, budget continuity and first-error stop. Separate desktop/mobile browser checks verified that the static demo displayed recorded results, exposed no key form, made only local static GET requests and downloaded byte-identical saved briefs.

This is a small development record, not a benchmark or success-rate claim. The missing-data case has offline coverage but no accepted real response. Reported usage for attempts 1–5 yields a conventional estimate of $0.013918 at the provider's published token prices. That excludes any unknown charge for attempt 6 and is not a final invoice. The retained $1.50 reservation is a safety allowance, not actual spending.

Why Does Open Innovation Matter?

Mastra makes the workflow boundaries inspectable: I can keep model selection separate from source validation and export, and test the same orchestration with an injected mock adapter. It is central to the local application's working path, rather than an unused dependency added for a badge.

The main design change was possible because the evidence contract is owned by the application. I could replace fragile model-written quotations with ID selection while leaving the workflow and the source-verification boundary intact. An OpenAI-compatible adapter also separates provider transport from evidence assembly; I tested one fixed model, not a broad provider comparison.

The open-source claim here is about the framework and third-party components. My original source is currently UNLICENSED. The model is called through an external provider. Exported briefs can be read offline, but real model selection does not run offline and is not a free hosted service.

Important limits remain: pasted text and timestamps are not authenticated; exact copying does not establish truth, freshness, completeness or safety; models can omit important evidence; recognizable instruction detection is not a complete injection defense; and a documented moderate transitive dependency advisory remains in the local Node prototype. The public site serves plain static records, not that Node runtime or a live inference service. Human review is required.

My Agent Session

I directed an AI-assisted build; AI agents generated the implementation, tests and this write-up, and reviewed the recorded evidence. I am publishing the source, synthetic fixtures, validation records and failure history rather than a raw session log that could contain private configuration. The recorded demo shows the model inputs, retained assistant outputs and deterministic assembly trace without request headers or credentials.

Prize Categories

Best Use of Mastra — the actual three-step local workflow uses Mastra's workflow API in both the deterministic demo and model-selection paths. No other partner category is claimed.

RetryBudget-24: When the Right Retry Decision Still Fails the JSON Contract

Tao Li — Sat, 10 Oct 2026 06:49:34 +0000

This is a submission for the Kaggle Benchmarking Challenge.

What I Benchmarked

An AI component advising an API client has two jobs: choose the right action and return something the client can actually parse. RetryBudget-24 tests those jobs together, then separates their failure modes for analysis.

Each question describes a failed or completed request and supplies an explicit, synthetic retry policy. The model returns exactly two fields:

{"action":"RETRY","delay_ms":100}

The actions are DONE, STOP, RECONCILE, and RETRY. The policy covers status precedence, attempt limits, idempotency guarantees, capped exponential backoff, Retry-After, and a remaining-time budget. A retry is permitted only when its wait plus the next attempt's estimated duration fits the budget. Equality is allowed.

This is a text-based policy-decision task. It does not execute HTTP requests, measure inference-serving latency, or demonstrate safe production retries.

Twelve controlled pairs

The formal set contains 24 original synthetic cases arranged as 12 pairs. Within each pair, exactly one input field changes. That makes the boundary being tested easier to inspect than in a collection of unrelated puzzles.

For example, keep a 100 ms wait and a 200 ms next attempt fixed:

299 ms remaining means STOP.
300 ms remaining means RETRY with a 100 ms delay.

Other pairs change whether a server guarantees deduplication for an idempotency key, whether the attempt limit has been reached, or whether the backoff cap binds.

The cases, fixed order, prompts, oracle, and scorer were hash-frozen before any model call. Four separate development cases checked execution without tuning the prompts against model responses. Expected answers were checked against the deterministic oracle; 20 offline tests checked the original benchmark implementation.

The primary score is exact correctness on both output fields, divided by 24. JSON whitespace and key order are allowed. Code fences, explanations, extra or duplicate keys, floats, and booleans in the integer field fail the contract. No LLM judge is involved.

For context, deterministic baselines score 24/24 for the policy oracle, 6/24 for always STOP, and 7/24 for a status-only heuristic that retries transient statuses after 100 ms. The oracle is a correctness reference, not a competing language model.

Models Tested

The experiment used two models available through Kaggle's Model Proxy:

google/gemini-2.5-flash
anthropic/claude-haiku-4-5@20251001

This small, cross-provider comparison kept the experiment inexpensive. It is not a survey of the strongest available models.

Both received the same frozen prompts through Kaggle Benchmarks SDK 0.6.1, with max_tokens=512, seed 0, and temperature 0 requested. Importantly, the inspected SDK path reported temperature unsupported for both and dropped that setting. Actual temperature was therefore not controlled. Reasoning settings remained at provider defaults, and requested settings do not establish identical effective inference budgets.

Each case used a separate chat. There were no retries for wrong answers. Client SDK retries were disabled; retries inside the proxy were not observable.

Findings

The planned experiment

Model	Strict score	Valid schema	Both members of pair correct
Gemini 2.5 Flash	16/24	22/24	4/12
Claude Haiku 4.5	5/24	9/24	1/12

The strict gap looks large. Inspecting the responses changes what can reasonably be concluded from it.

Haiku wrapped 15 of its 24 answers in complete Markdown code fences. Nine of those contained the correct action and delay. Gemini produced one complete fence with correct content and one unclosed fence.

A narrowly defined, post-hoc diagnostic removed only complete outer fences, without repairing JSON, coercing types, or extracting objects from prose. Correctness then became 17/24 for Gemini and 14/24 for Haiku. These are diagnostic counts, not replacement benchmark scores.

The distinction matters for integration design. A strict JSON consumer really would reject those fenced responses. But calling every rejection a reasoning failure would hide a substantial part of what happened. A parser adapter or constrained-output interface could be worth testing in a separately frozen experiment; neither was retrofitted into these scores.

Some errors remain after formatting is separated

Both models chose RETRY on the 299 ms budget example, although 100 + 200 exceeds 299. At 300 ms, both returned the correct decision content, with Haiku's answer fenced.

On an unsafe timeout with no idempotency guarantee, Haiku returned RETRY where the policy required RECONCILE. On a capped-backoff case where min(1000, 700 * 2) should produce 1000 ms, Gemini returned 1400 and Haiku returned 700.

Those examples identify concrete policy boundaries worth testing. They do not establish how either model would behave across real API traffic.

An unintended second experiment

There was an execution-control mistake during publication preparation: Build Task was treated as a save/build operation, but it executed the original notebook again. The additional run completed before it could be stopped. That exceeded the planned 56 client calls and must be counted.

Batch	Gemini formal score	Haiku formal score	Calls including development	Reported quota cost
Planned experiment	16/24	5/24	56	$0.0521384
Unplanned Build rerun	18/24	4/24	56	$0.0523159
Total	Separate batches	Separate batches	112	$0.1044543

All 112 recorded calls completed. The additional batch used identical prompt hashes and unchanged scoring. Its outputs are retained separately; the better Gemini result does not replace the planned result, and the batches are not pooled into a new headline score.

The amounts are reported Model Proxy quota consumption within the account's available free quota, not a cash purchase. They show that this particular experiment was inexpensive, not that future runs have a guaranteed price.

This also exposed a benchmark-workflow requirement: publication steps need their own execution budget. A call ledger inside one notebook session cannot impose a global cap across fresh platform executions.

What the results cannot tell us

This is a small synthetic set, with related pairs, one planned sample per case, and one accidental repeat. It supports failure analysis rather than a general model ranking or statistically reliable reliability estimate.

The saved artifacts do not contain finish_reason. In the planned Gemini test, 21 of 24 reported output-token counts were at least 450 against the requested 512-token limit. That warrants caution, but it does not prove truncation or explain a particular error. The models' effective generation conditions were not fully controlled.

The next useful experiment would predefine strict versus constrained-output conditions, more policy boundaries, and planned repetitions. It would also verify the platform's build behavior before enabling any model-call entry point.

My Benchmark

Kaggle benchmark collection: RetryBudget-24: Policy and JSON Contracts

Source, frozen cases, manifests, and raw results: Planned experiment snapshot and unplanned Build rerun snapshot. The self-authored benchmark code is published under Apache 2.0.

The published Kaggle task and collection contain only the unplanned batch's Gemini result, 0.75 (18/24). The two-model comparisons above come from separately preserved and audited raw experiment artifacts; they are not a claim that both models appear on the platform leaderboard. The original execution adapter also depends on its notebook context. A self-contained publication wrapper has been prepared and tested offline, but it has not been run against real models or substituted for the measured version.

AI assistance was used to design and implement the synthetic benchmark, run the workflow, audit saved outputs, and draft this article. The task uses original synthetic cases rather than third-party evaluation data. The SDK is credited below. The article explicitly retains the AI-assisted workflow's unintended rerun instead of hiding it.

Official references: