I built an AI agent that diagnoses cars — and asks before it touches anything
Built for the TrueForge Agent Harness Hackathon (Aug 24–30, 2026).
"Something expensive broke. Figure out why and prove it. Ask a human before you touch anything."
That was basically the idea behind FaultTrace.
I wanted to build something that felt more like an actual investigation than another chatbot with a few tools attached. The result is an autonomous vehicle-forensics agent that can gather evidence, test competing explanations, run actual analysis, and stop when it reaches a physical-world action that needs a human.
And honestly, the interesting part wasn't getting the first version working.
It was getting the whole thing to keep working reliably.
This is how I built it, and what went wrong along the way.
Why diagnose a car with an agent?
A car throws:
P0171 — System Too Lean
A chatbot can explain what P0171 means in a few seconds.
But that's not really the hard part.
A technician still has to figure out why the car thinks it's running lean. It could be a vacuum leak, a dirty MAF sensor, a fuel-delivery problem, or even an O2 sensor that's giving misleading information.
So the real problem isn't:
"What does P0171 mean?"
It's:
"Which of several possible causes actually explains the evidence?"
That's an investigation.
That distinction is what led me to build FaultTrace.
Given a vehicle failure event, FaultTrace:
- gathers evidence from multiple sources,
- creates competing root-cause hypotheses,
- predicts what each hypothesis should look like in the data,
- runs actual analysis in a sandbox,
- uses deterministic Bayesian calculations to rank the hypotheses,
- calculates which additional test would reduce uncertainty the most,
- and stops for human approval before taking a physical-world action.
For the hackathon, I kept the scope deliberately concrete: vehicle diagnostic forensics.
The hero scenario is a cracked brake-booster vacuum hose on a 2003 Honda Accord, resulting in P0171 + P0300.
It's a small enough problem to demonstrate end-to-end, but complicated enough to make the agent actually investigate rather than just look up a DTC.
The architecture
The vehicle domain is the implemented MVP. The underlying investigation pattern is intended to generalize to other safety-sensitive physical systems later.
Here's the high-level architecture:
flowchart TB
U["User / Technician"] --> AG
subgraph TF["TrueForge Harness"]
direction TB
AG["Investigator Agent<br/>(faulttrace-investigator)"]
SUB["Dynamic Subagents<br/>(per-hypothesis fan-out)"]
SBX["Harness sandbox<br/>(optional agent-generated checks)"]
RANK["Bayesian ranking<br/>prior × likelihood → posterior"]
SES["Persistent session"]
end
subgraph MCP["faulttrace-vehicle MCP server"]
R1["get_dtcs · get_freeze_frame"]
R2["get_sensor_log · get_compact_telemetry"]
R3["lookup_dtc_knowledge · get_vehicle_info"]
RA["run_analysis"]
G2["request_measurement — Tier 2"]
G3["clear_codes · order_part — Tier 3"]
end
AG --> R1
AG --> R2
AG --> R3
AG --> SES
AG -- "hypothesis fan-out" --> SUB
SUB -- "supporting / contradictory evidence" --> AG
AG -- "run analysis" --> RA
RA --> FIXED["fixed analyze.py (server-side, deterministic)"]
FIXED --> RANK
RANK --> AG
SUB -- "optional custom checks" --> SBX
AG -- "propose physical action" --> AP["Human approval gate"]
AP -- "approved → invoke" --> G2
AP -- "approved → invoke" --> G3
AP -- "rejected → cancel" --> XL["no tool call"]The important thing here isn't the number of boxes.
It's the loop.
The investigation loop
Failure event
(DTC + freeze-frame + sensor conditions)
↓
Observe
(read-only evidence via MCP)
↓
Form competing root-cause hypotheses
(each with a predicted signature)
↓
Run computed analysis
(real computation, not LLM text math)
↓
Evaluate supporting vs contradictory evidence
↓
Bayesian update
prior × likelihood → posterior differential
↓
Identify remaining uncertainty
↓
Choose the next diagnostic
(expected information gain + cost)
↓
STOP for human approval
before Tier 2 / Tier 3 actions
↓
New evidence arrives
↓
Continue investigation
↓
Defensible root-cause conclusion
That is the part I wanted to get right.
The model isn't supposed to just produce a diagnosis and call it a day. It needs to figure out what it knows, what it doesn't know, and what it should do next.
What makes this different from a chatbot?
This was one of the design questions I kept coming back to.
If I could replace the whole system with:
"Paste your DTC into ChatGPT"
then I hadn't really built an agent.
FaultTrace needs to:
- retrieve evidence from multiple tools,
- delegate specialized investigation,
- form competing hypotheses,
- determine what each hypothesis predicts,
- execute computational analysis,
- compare evidence,
- identify missing evidence,
- choose a useful next diagnostic,
- pause for human approval,
- receive the result,
- and continue the same investigation.
So the distinction is pretty simple:
A chatbot gives you an answer. FaultTrace runs an investigation.
TrueForge isn't just there for the hackathon points
This was important to me.
I didn't want to build a normal chatbot and then bolt TrueForge onto it just so I could say I used the sponsor's technology.
The harness is actually doing a lot of the work.
Agent runtime
FaultTrace runs as a TrueForge agent, defined by a manifest containing the model, instructions, and MCP configuration.
MCP
The agent talks to a real vehicle MCP server over HTTP.
The tools aren't simulated function descriptions sitting inside the prompt. The agent actually reaches the server and gets data back.
Dynamic subagents
FaultTrace can fan out the investigation by hypothesis.
For example:
FaultTrace
│
Bayesian differential
│
┌─────────────┼─────────────┐
↓ ↓ ↓
Vacuum leak Misfire / Sensor
investigator ignition plausibility
investigator investigator
Each subagent gets its own thread and sandbox and reports back supporting and contradictory evidence.
Sandbox
The model can propose analysis, but the important computation happens in code.
The fixed analyze.py library performs the deterministic diagnostic calculations, including the Bayesian differential and expected information gain.
Persistent sessions
An investigation isn't just one request/response.
The TrueForge session can pause for an approval, reconnect, and continue the same investigation. In the demo, the investigation actually resumes mid-flow.
Human approval
This is probably my favorite part.
If the agent decides it needs a physical measurement or wants to clear codes/order a part, the harness pauses the action and puts the decision in front of a human.
Nothing executes until it's approved.
The MCP tools
The vehicle MCP server currently exposes 13 tools across the investigation and safety workflow.
Some of the important ones are:
| Tool | What it does |
|---|---|
list_vehicles |
Discover available vehicles |
get_vehicle_info |
Retrieve vehicle metadata |
get_dtcs |
Retrieve diagnostic trouble codes |
get_freeze_frame |
Retrieve the fault-state snapshot |
get_pid_list |
Discover available telemetry PIDs |
get_compact_telemetry |
Retrieve bounded current telemetry |
get_sensor_log |
Retrieve historical sensor telemetry |
lookup_dtc_knowledge |
Retrieve scenario-specific diagnostic knowledge |
run_analysis |
Run the deterministic computed differential |
request_measurement |
Request an additional diagnostic measurement |
clear_codes |
Clear diagnostic codes |
order_part |
Request a replacement part |
The last two are intentionally gated.
That distinction matters because the agent can be autonomous without being allowed to do whatever it wants.
Letting the data actually decide
One thing I didn't want was a model looking at a bunch of numbers and then casually saying:
"I'm 91% confident this is a vacuum leak."
That's not very convincing.
So FaultTrace has a deterministic analysis layer.
The model supplies the investigation context, but the actual analysis code calculates the differential.
Conceptually:
P(hypothesis | evidence)
∝
P(evidence | hypothesis) × P(hypothesis)
The analyzer takes scenario-specific priors and telemetry-derived likelihoods, performs the Bayesian update, and returns a normalized posterior ranking.
It also calculates expected information gain for the available diagnostic tests.
That gives the agent something more useful than "try another test":
Current uncertainty
↓
Evaluate available tests
↓
Calculate expected information gain
↓
Consider test cost
↓
Select the most useful next test
The result is reproducible because the computation is deterministic and seeded.
That separation is important to the architecture:
The LLM decides what to investigate. The analysis code calculates the numbers.
The predicted-signature idea
Every hypothesis isn't just a label.
It comes with a prediction:
"If this hypothesis is actually true, what should I see in the data?"
For example, a vacuum leak should produce a different pattern from a MAF fault or an ignition problem.
That gives the agent something concrete to test.
Instead of:
Hypothesis: Vacuum leak
we have:
Hypothesis:
Vacuum leak
Predicted signature:
- elevated fuel trims
- stronger effect at idle
- abnormal airflow relationship
- correlation with misfire behavior
The analysis then checks the telemetry against those signatures.
This also makes the final explanation much more useful because we can show both:
why a hypothesis fits
and
why another hypothesis doesn't.
Supporting evidence isn't enough
I wanted the agent to actively look for evidence against its own hypotheses too.
So a final differential isn't just:
Vacuum leak
✓ Fuel trims support this
✓ MAF relationship supports this
It should look more like:
VACUUM LEAK
Supporting evidence
✓ Positive fuel trim at idle
✓ Airflow relationship matches predicted behavior
✓ Misfire pattern is consistent
Contradictory evidence
⚠ Idle instability is weaker than expected
Missing evidence
? Fuel pressure under load
And for another hypothesis:
MAF FAULT
Supporting evidence
✓ Some airflow irregularity
Contradictory evidence
✕ Fuel-trim behavior is more consistent with
unmetered air
Missing evidence
? Independent airflow measurement
That "why not?" reasoning is a big part of making the result feel forensic rather than classificatory.
Active diagnosis: knowing what to test next
This is another part I really wanted to avoid making into a hard-coded flowchart.
Suppose the agent has narrowed the problem down to two plausible causes:
1. Vacuum leak
2. Weak fuel delivery
The agent shouldn't just say:
"More data is needed."
It should ask:
"What measurement would actually separate these two explanations?"
That's where expected information gain comes in.
The analysis evaluates the available tests and returns something like:
Recommended test:
fuel_pressure_under_load
Expected information gain:
X.XX bits
Cost:
Low
Reason:
The result is expected to distinguish the two leading hypotheses.
And then the agent stops.
It doesn't execute the physical test automatically.
The three-tier safety model
This is deliberately simple.
Tier 1 — Investigate
The agent can do these autonomously:
- read DTCs,
- retrieve telemetry,
- inspect history,
- look up diagnostic knowledge,
- generate hypotheses,
- run analysis,
- use the sandbox.
Tier 2 — Diagnose physically
Human approval required:
- request a measurement,
- run a guided diagnostic procedure,
- collect data that requires an active diagnostic action.
Tier 3 — Change the vehicle / external world
Human approval required:
- clear codes,
- order a part,
- modify vehicle state,
- perform a physical repair.
The rule is:
Investigate freely. Act carefully.
There are two layers of protection here. TrueForge provides the actual approval gate, and the MCP server independently refuses gated calls that don't contain the required approval.
So even if something goes wrong in the agent layer, the server has another line of defense.
The part that almost killed the project: models are not interchangeable
This was probably the biggest practical lesson I got from the build.
I initially assumed that if a model was good enough at reasoning, I could just swap it into the same agent and everything would behave roughly the same.
Nope.
The behavior around tools and subagents can be dramatically different.
First problem: GLM
I initially used:
openrouter/z-ai-glm-5.3-flash
because it was cheap and fast.
It did something interesting: it actually spawned real create_sub_agent threads.
I could see three separate thread.created events with different thread IDs.
So, great?
Not quite.
The subagents then hit a wall because the harness's local sandbox is macOS/Linux only, while I was running the project on Windows. They ended up trying to execute their Python in a cloud sandbox and timing out.
So I had:
beautiful fan-out
↓
three real subagents
↓
sandbox timeout
↓
zero useful results
That wasn't exactly the demo I wanted.
Then Gemini
I switched the primary model to Gemini 2.5 Flash.
Now I had the opposite problem.
Gemini would talk about subagents without actually creating them.
I'd see things like:
Sub-agent: investigating vacuum leak...
Sub-agent: checking ignition...
but there were no real thread.created events.
It was essentially role-playing the delegation.
That's a surprisingly important distinction when you're building an agent system.
Tightening the instructions
I eventually had to make delegation explicit and give the subagents very concrete instructions.
Things like:
- use the VIN from the actual failure event,
- use the exact PID names,
- don't invent telemetry,
- write Python to a file before running it,
- use the sandbox for actual computation,
- return supporting and contradictory evidence.
One small example caused a ridiculous amount of pain:
rpm
was the actual PID.
The subagent would sometimes assume:
engine_rpm
and the whole analysis would fail.
Another lesson: when running Python through the shell, I had much better results writing it to a file and executing the file than trying to pipe complex Python through echo.
Sometimes the "AI problem" is just shell quoting.
Another Gemini problem: "Should I proceed?"
The other thing that bit me was the approval flow.
I needed the model to produce the actual gated tool call so TrueForge could surface the approval UI.
Instead, Gemini would sometimes do this:
The next step would be to request
a fuel-pressure measurement.
Would you like me to proceed?
Looks reasonable to a human.
But it completely bypasses the actual approval mechanism.
There was no approval button because there was no tool call.
So I added an explicit rule:
If the agent has decided on a gated action, the turn must end with the gated tool call rather than a prose question.
That was one of those tiny prompt changes that made a huge difference.
Qodo caught some things I would have missed
I also ran Qodo code review on the project's pull requests.
It wasn't just a checkbox for the hackathon. A few findings actually changed the implementation.
Single source of truth
Qodo flagged that run_analysis could be treated as one source of posterior probabilities while the orchestrator had already derived another view from subagent evidence.
That was a dangerous design.
I changed the architecture so:
run_analysisis the single authoritative source of posterior probabilities.
Subagents contribute evidence and narrative analysis, but they don't modify the posterior.
Ground truth leaking through the tool schema
Qodo also caught a mismatch where a tool description exposed scenario information that the response itself intentionally kept hidden.
I aligned the MCP contract with the actual allow-listed response.
The VIN bug
This one was particularly important.
A subagent recipe had accidentally hard-coded Scenario A's VIN.
That meant Scenario B or C could have caused the subagent to investigate the wrong vehicle.
Qodo caught it.
The fix was simple: every subagent now receives and reuses the VIN from the actual failure event.
And sometimes the reviewer is wrong
Qodo also suggested converting a smoke test to CommonJS.
I pushed back on that one.
The repository uses ESM with "type": "module", and the actual tests follow that convention. So I dismissed the suggestion with the reasoning recorded in the PR.
That's a useful lesson too:
Code review tools are extremely useful, but they aren't infallible. You still need to understand the code you're reviewing.
The feature I'm most proud of isn't the Bayesian math
It's the fact that FaultTrace knows when to stop.
The agent can investigate a problem for as long as the work is read-only.
But eventually it might conclude:
The next useful step is a physical measurement.
At that point:
┌─────────────────────────────────────────────┐
│ HUMAN APPROVAL REQUIRED │
│ │
│ Request fuel-pressure measurement? │
│ │
│ [ Approve ] [ Reject ] │
└─────────────────────────────────────────────┘
The agent waits.
If the human rejects it, nothing happens.
If the human approves it, the action executes and the same investigation can continue with the new evidence.
That's the boundary I wanted:
Autonomous investigation. Human-controlled action.
What's real and what's mocked?
A hackathon project should be honest about this.
Real
- TrueForge agent orchestration
- MCP communication
- Dynamic subagent delegation
- Sandbox analysis
- Bayesian differential
- Expected information gain
- Persistent investigation sessions
- Human approval gates
- Vehicle diagnostic workflow
- Dashboard
Simulated
- Vehicle telemetry is synthetic and seeded.
- There is no physical OBD-II connection.
- Parts ordering is mocked; no real money moves.
- The vehicle itself is simulated.
Scenarios B (dirty MAF) and C (stuck O2) exist for regression coverage.
Scenario A (vacuum leak) is the hero.
The broader architecture could eventually apply to industrial machinery, robotics, energy systems, and other physical systems, but those are future applications, not claims about what this MVP already implements.
Running it
The basic setup is:
npm install
npm start
cd mcp-server
npm install
npm run start:http
The TrueForge harness runs on:
http://localhost:8790
The repository contains the remaining configuration and environment setup required to run the demo.
The demo
I've kept the main demo focused on one investigation rather than trying to show every feature.
The flow is:
DTC event
↓
MCP evidence collection
↓
Competing hypotheses
↓
Dynamic subagents
↓
Sandbox analysis
↓
Bayesian differential
↓
Expected information gain
↓
Recommended diagnostic
↓
Human approval
↓
Investigation resumes
↓
Root-cause report
What I learned
If I had to summarize the whole project in one sentence:
The hard part of an agent isn't getting a model to reason — it's making the entire loop reliable.
Getting an LLM to say:
"I think this is a vacuum leak"
is easy.
Getting it to:
- retrieve the correct vehicle,
- call the right MCP tools,
- form competing hypotheses,
- delegate real work,
- execute that work in a sandbox,
- use deterministic computations,
- choose the next useful test,
- stop at the physical-world boundary,
- get human approval,
- execute the action,
- resume the same session,
- and produce a defensible conclusion
is a very different problem.
That's what made FaultTrace interesting to build.
And that's probably the biggest thing I took away from the hackathon:
The model is only one component of an agent. The orchestration around it is where the real engineering starts.
Built for the TrueForge Agent Harness Hackathon. AI coding assistants were used during development, and the code was reviewed throughout the project, including with Qodo.
Top comments (0)